rsync with overlay tree

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

rsync with overlay tree

tomr
I maintain a directory structure containing dirs and files that I regularly push to ~50 hosts, which are divided into 3 groups that have slightly different needs (minor mods in a couple of files).

So ideally I would have 4 directories:
    /path/to/sync/common/   <- common files
    /path/to/sync/group1/   <- group1 specific only
    /path/to/sync/group2/   <- group2 specific only
    /path/to/sync/group3/   <- group3 specific only

Then I'd run an rsync like:
    rsync -av --overlay /path/to/sync/groupN \
     /path/to/sync/common remotehost:

Thinking in terms of a list of files to be transferred, I would like:
- Anything present in common/ added to the file list; then
- Anything present in groupN/ added to the list, clobbering if applicable (regardless of mtime)
- The destination directory to show no sign of the common / overlay structure

I suspect I could populate the list myself and use '--files-from=<LIST>' but I would rather have rsync work it out if possible.  If all else fails, I will do the merge locally first.

    TEMPDIR=$(mktemp -d)
    cp -r /path/to/sync/common/* $TEMPDIR
    cp -r /path/to/sync/groupN/* $TEMPDIR
    rsync -av $TEMPDIR/* remotehost:
    rm -r $TEMPDIR

Thanks in advance,
tom



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync with overlay tree

Marian Marinov
On 03/31/2016 07:40 AM, tomr wrote:

> I maintain a directory structure containing dirs and files that I regularly push to ~50 hosts, which are divided into 3 groups that have slightly different needs (minor mods in a couple of files).
>
> So ideally I would have 4 directories:
>     /path/to/sync/common/   <- common files
>     /path/to/sync/group1/   <- group1 specific only
>     /path/to/sync/group2/   <- group2 specific only
>     /path/to/sync/group3/   <- group3 specific only
>
> Then I'd run an rsync like:
>     rsync -av --overlay /path/to/sync/groupN \
>      /path/to/sync/common remotehost:
>
> Thinking in terms of a list of files to be transferred, I would like:
> - Anything present in common/ added to the file list; then
> - Anything present in groupN/ added to the list, clobbering if applicable (regardless of mtime)
> - The destination directory to show no sign of the common / overlay structure
I really like the idea, however I have one question. How is rsync going to decide which files to be present in the final list?
I mean, should rsync always prefer the files from the overlay dir instead of the files in the common dir, no matter is they differ or not?


>
> I suspect I could populate the list myself and use '--files-from=<LIST>' but I would rather have rsync work it out if possible.  If all else fails, I will do the merge locally first.
>
>     TEMPDIR=$(mktemp -d)
>     cp -r /path/to/sync/common/* $TEMPDIR
>     cp -r /path/to/sync/groupN/* $TEMPDIR
>     rsync -av $TEMPDIR/* remotehost:
>     rm -r $TEMPDIR
>
> Thanks in advance,
> tom
>
>
>

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: rsync with overlay tree

tomr

> On 31 Mar 2016, at 16:22, Marian Marinov <[hidden email]> wrote:
>
> On 03/31/2016 07:40 AM, tomr wrote:
>> I maintain a directory structure containing dirs and files that I regularly push to ~50 hosts, which are divided into 3 groups that have slightly different needs (minor mods in a couple of files).
>>
>> So ideally I would have 4 directories:
>>    /path/to/sync/common/   <- common files
>>    /path/to/sync/group1/   <- group1 specific only
>>    /path/to/sync/group2/   <- group2 specific only
>>    /path/to/sync/group3/   <- group3 specific only
>>
>> Then I'd run an rsync like:
>>    rsync -av --overlay /path/to/sync/groupN \
>>     /path/to/sync/common remotehost:
>>
>> Thinking in terms of a list of files to be transferred, I would like:
>> - Anything present in common/ added to the file list; then
>> - Anything present in groupN/ added to the list, clobbering if applicable (regardless of mtime)
>> - The destination directory to show no sign of the common / overlay structure
>
> I really like the idea, however I have one question. How is rsync going to decide which files to be present in the final list?
> I mean, should rsync always prefer the files from the overlay dir instead of the files in the common dir, no matter is they differ or not?
Yes :-)

In my thinking rsync should never know or even check whether they differ: if a filename (or directory?) is present in the overlay path, it should clobber any matching filename (or directory?) in the common path.

An example for clarity.... I start with:

    common/config/settings.conf
    common/config/include.d/names.conf
    common/config/include.d/numbers.conf
    common/config/include.d/plugins.conf
    common/bin/someapp
    common/bin/someplugin
    common/lib/somelib
    common/someapp-release

Almost all of the config of someapp has remained unchanged in the latest major release.  I don't want to fork all that config, and maintain in two places until we're fully upgraded, and I want to be able to provision servers with the old and the new config easily in the mean time.  So I create (minimally):

    group1/config/include.d/plugins.conf
    group1/config/include.d/newfeature.conf
    group1/bin/someplugin
    group1/lib/somelib
    group1/someapp-release

An rsync with eg '--overlay-path group1/' would sync plugins.conf, newfeature.conf, someplugin, somelib and someapp-release from under group1 regardless of any fact other than that they exist.  The automation that produces names.conf and numbers.conf can continue running without having to know about the arrangement.

If I eventually want to collapse group1 into common I can just run 'cp -r group1/* common && rm -r group1/*'

I think ;)

best,
tom



>> I suspect I could populate the list myself and use '--files-from=<LIST>' but I would rather have rsync work it out if possible.  If all else fails, I will do the merge locally first.
>>
>>    TEMPDIR=$(mktemp -d)
>>    cp -r /path/to/sync/common/* $TEMPDIR
>>    cp -r /path/to/sync/groupN/* $TEMPDIR
>>    rsync -av $TEMPDIR/* remotehost:
>>    rm -r $TEMPDIR
>>
>> Thanks in advance,
>> tom
>>
>>
>>
>

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

signature.asc (859 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: rsync with overlay tree

Steven Levine
In reply to this post by tomr
In <[hidden email]>, on 03/31/16
   at 03:40 PM, tomr <[hidden email]> said:

Hi,


>Then I'd run an rsync like:
>    rsync -av --overlay /path/to/sync/groupN \
>     /path/to/sync/common remotehost:

>Thinking in terms of a list of files to be transferred, I would like: -
>Anything present in common/ added to the file list; then - Anything
>present in groupN/ added to the list, clobbering if applicable
>(regardless of mtime) - The destination directory to show no sign of the
>common / overlay structure

The best solution will depend on the number of files in the special
groups.

Since you say the number is small, I suspect that something simple like:

  rsync -hia /path/to/sync/common remotehost:
  rsync -hia --ignore-times /path/to/sync/groupN remotehost:

will be close to optimal.

Another option is to build an --exclude-from list for the files in the
special group and apply it to the first rsync.  Then you can run the
second rsync without the --ignore-times, which may result in faster
overall throughput.

FWIW, I think you will find the -hi output more useful than the -v output.

Steven

--
----------------------------------------------------------------------
"Steven Levine" <[hidden email]>  Warp/DIY/BlueLion etc.
www.scoug.com www.arcanoae.com www.warpcave.com
----------------------------------------------------------------------


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync with overlay tree

Fabian Cenedese

>Since you say the number is small, I suspect that something simple like:
>
>  rsync -hia /path/to/sync/common remotehost:
>  rsync -hia --ignore-times /path/to/sync/groupN remotehost:
>
>will be close to optimal.
>
>Another option is to build an --exclude-from list for the files in the
>special group and apply it to the first rsync.  Then you can run the
>second rsync without the --ignore-times, which may result in faster
>overall throughput.

Another way could be to build the specific directories and in there
create links to the common files if there aren't too many. Then on
rsyncing you need to dereference the links. That way you only
have one directory (structure) which rsync can already handle.

bye  Fabi


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync with overlay tree

Perry Hutchison
In reply to this post by tomr
tomr <[hidden email]> wrote:
> Almost all of the config of someapp has remained unchanged
> in the latest major release.

Distributing config files really sounds more like a job for
sysutils/puppet than for rsync :)  Teach facter to return the
version of someapp that's installed and you don't even have
to keep track of which machines need to have the overlay tree
included in their rsync invocations.

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync with overlay tree

Henri Shustak
If you are looking at overlays specifically have you checked out RADMIND?

http://rsug.itd.umich.edu/software/radmind/macosx.html

Note : The link above is more specific to the MacOSX release but
       it is a good starting point

Hope that helps!

--------------------------------------------------------------------
This email is protected by LBackup, an open source backup solution
http://www.lbackup.org

On 1/04/2016, at 3:57 PM, Perry Hutchison <[hidden email]> wrote:

> tomr <[hidden email]> wrote:
>> Almost all of the config of someapp has remained unchanged
>> in the latest major release.
>
> Distributing config files really sounds more like a job for
> sysutils/puppet than for rsync :)  Teach facter to return the
> version of someapp that's installed and you don't even have
> to keep track of which machines need to have the overlay tree
> included in their rsync invocations.
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html