Quantcast

which rsync command?

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

which rsync command?

David Epstein
All the files and directories I want to talk about are on the same machine (a Mac).  I have a directory that I will call SOURCE. This contains a a number of files, some of which are directories containing further files. I want to copy these files to another directory, which I will call TARGET. Please assume that SOURCE and TARGET include absolute path-names. I think they should also end in a slash (subject to correction).

If a top-level file/directory is present in TARGET, but not in SOURCE, I do not want it to be disturbed.
If a top-level file/directory is not present in TARGET, but is present in SOURCE, I want it to be recursively copied into TARGET
If a top-level file/directory is present in TARGET and also present in SOURCE, I want it to overwrite, so that the ccorresponding file/directory in TARGET at the end of the rsync job is a copy of what is in SOURCE, with the same permissions and date-stamps.

Since it’s on a Mac, I also want to transfer extended attributes.

I would be grateful for the recommended recipe.

Would dry-run tell me what rsync proposed to delete as well as what it proposed to install. It’s because I don’t understand the option dry-run fully that I’m reluctant to use trial-and-error to get to the right command.

I think I would have to use “sudo” since some of the files are not readable by me when I am an ordinaty user.

Thanks



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: which rsync command?

Kevin Korb
You should be fine as long as you don't add --delete.

I would start with rsync -vai --dry-run SOURCE/ TARGET/
Add whatever the OSX specific option is (I think -E)

Yes, --dry-run shows you what it would have done without --dry-run and
yes, if you can't read all the files then you need to run it as root.

On 02/22/2017 08:57 AM, David Epstein wrote:

> All the files and directories I want to talk about are on the same machine (a Mac).  I have a directory that I will call SOURCE. This contains a a number of files, some of which are directories containing further files. I want to copy these files to another directory, which I will call TARGET. Please assume that SOURCE and TARGET include absolute path-names. I think they should also end in a slash (subject to correction).
>
> If a top-level file/directory is present in TARGET, but not in SOURCE, I do not want it to be disturbed.
> If a top-level file/directory is not present in TARGET, but is present in SOURCE, I want it to be recursively copied into TARGET
> If a top-level file/directory is present in TARGET and also present in SOURCE, I want it to overwrite, so that the ccorresponding file/directory in TARGET at the end of the rsync job is a copy of what is in SOURCE, with the same permissions and date-stamps.
>
> Since it’s on a Mac, I also want to transfer extended attributes.
>
> I would be grateful for the recommended recipe.
>
> Would dry-run tell me what rsync proposed to delete as well as what it proposed to install. It’s because I don’t understand the option dry-run fully that I’m reluctant to use trial-and-error to get to the right command.
>
> I think I would have to use “sudo” since some of the files are not readable by me when I am an ordinaty user.
>
> Thanks
>
>
>
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
        Kevin Korb Phone:    (407) 252-6853
        Systems Administrator Internet:
        FutureQuest, Inc. [hidden email]  (work)
        Orlando, Florida [hidden email] (personal)
        Web page: http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

signature.asc (197 bytes) Download Attachment
Joe
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: which rsync command?

Joe
I'm probably missing something, but it looks like your first requirement
rules out using --delete,
but your third requirement seems to imply the need for it because you
want the source and target to be identical - and it won't be if there
are any files on the target which are not on the source.

If this is the case, then you may have to use more than pass - the first
at the top-level without the --delete and one or more at a lower level
with --delete, avoiding those top-level directories/files which do not
exist on the source.

It depends on exactly what you want to achieve.

--dry-run is your friend.

On 02/22/2017 10:36 AM, Kevin Korb wrote:

> You should be fine as long as you don't add --delete.
>
> I would start with rsync -vai --dry-run SOURCE/ TARGET/
> Add whatever the OSX specific option is (I think -E)
>
> Yes, --dry-run shows you what it would have done without --dry-run and
> yes, if you can't read all the files then you need to run it as root.
>
> On 02/22/2017 08:57 AM, David Epstein wrote:
>> All the files and directories I want to talk about are on the same machine (a Mac).  I have a directory that I will call SOURCE. This contains a a number of files, some of which are directories containing further files. I want to copy these files to another directory, which I will call TARGET. Please assume that SOURCE and TARGET include absolute path-names. I think they should also end in a slash (subject to correction).
>>
>> If a top-level file/directory is present in TARGET, but not in SOURCE, I do not want it to be disturbed.
>> If a top-level file/directory is not present in TARGET, but is present in SOURCE, I want it to be recursively copied into TARGET
>> If a top-level file/directory is present in TARGET and also present in SOURCE, I want it to overwrite, so that the ccorresponding file/directory in TARGET at the end of the rsync job is a copy of what is in SOURCE, with the same permissions and date-stamps.
>>
>> Since it’s on a Mac, I also want to transfer extended attributes.
>>
>> I would be grateful for the recommended recipe.
>>
>> Would dry-run tell me what rsync proposed to delete as well as what it proposed to install. It’s because I don’t understand the option dry-run fully that I’m reluctant to use trial-and-error to get to the right command.
>>
>> I think I would have to use “sudo” since some of the files are not readable by me when I am an ordinaty user.
>>
>> Thanks
>>
>>
>>
>
>


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: which rsync command?

David Epstein
Thanks for the helpful advice.

In view of the conflicting advice from Joe and Kevin I tried creating a couple of very small directories test1 and test2, to see what happens when the rsync command
rsync -avi —dry-run test1/ test2/
is given. I was pleased to see that —dry-run does explicitly say when a file is being deleted.
Joe’s email is correct and it seems that Kevin’s version would not satisfy my third requirement.

Joe says “It depends on exactly what you want to achieve”. I think my objectives were very clearly stated, but I’m happy to clarify if necessary.

Could someone possibly elaborate on Joe’s advice, with more detail? I would be very grateful.
On the first pass, it seems that one should just do
rsync -avi -E SOURCE/ TARGET/
Howevr t’s not clear to me what the rsync command should be on the second pass. I could easily generate a list of files/directories in TARGET/ that should not be deleted or changed, and put these into a file “somewhere". Then I could give an rsync command with the option —delete, but how do I protect the entries in my “somewhere” file from deletion?

thanks for any help
David

> On 23 Feb 2017, at 01:25, Joe <[hidden email]> wrote:
>
> I'm probably missing something, but it looks like your first requirement rules out using --delete,
> but your third requirement seems to imply the need for it because you want the source and target to be identical - and it won't be if there are any files on the target which are not on the source.
>
> If this is the case, then you may have to use more than pass - the first at the top-level without the --delete and one or more at a lower level with --delete, avoiding those top-level directories/files which do not exist on the source.
>
> It depends on exactly what you want to achieve.
>
> --dry-run is your friend.
>
> On 02/22/2017 10:36 AM, Kevin Korb wrote:
>> You should be fine as long as you don't add --delete.
>>
>> I would start with rsync -vai --dry-run SOURCE/ TARGET/
>> Add whatever the OSX specific option is (I think -E)
>>
>> Yes, --dry-run shows you what it would have done without --dry-run and
>> yes, if you can't read all the files then you need to run it as root.
>>
>> On 02/22/2017 08:57 AM, David Epstein wrote:
>>> All the files and directories I want to talk about are on the same machine (a Mac).  I have a directory that I will call SOURCE. This contains a a number of files, some of which are directories containing further files. I want to copy these files to another directory, which I will call TARGET. Please assume that SOURCE and TARGET include absolute path-names. I think they should also end in a slash (subject to correction).
>>>
>>> If a top-level file/directory is present in TARGET, but not in SOURCE, I do not want it to be disturbed.
>>> If a top-level file/directory is not present in TARGET, but is present in SOURCE, I want it to be recursively copied into TARGET
>>> If a top-level file/directory is present in TARGET and also present in SOURCE, I want it to overwrite, so that the ccorresponding file/directory in TARGET at the end of the rsync job is a copy of what is in SOURCE, with the same permissions and date-stamps.
>>>
>>> Since it’s on a Mac, I also want to transfer extended attributes.
>>>
>>> I would be grateful for the recommended recipe.
>>>
>>> Would dry-run tell me what rsync proposed to delete as well as what it proposed to install. It’s because I don’t understand the option dry-run fully that I’m reluctant to use trial-and-error to get to the right command.
>>>
>>> I think I would have to use “sudo” since some of the files are not readable by me when I am an ordinaty user.
>>>
>>> Thanks
>>>
>>>
>>>
>>
>>
>
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: which rsync command?

Kevin Korb
I hate to say it because it goes against my normal advice but this is
one instance where using a * in the source parameter would help...

rsync -vaiE --delete --dry-run source/* target/

This would ignore any top level directory that is on the target but not
the source while rsyncing with --delete any top level directory that is
on the source.

It is normally recommended to not use a * in the source because it
causes --delete to behave this way which is not normally desirable and
because * does not include dot files (or directories).

On 02/23/2017 10:53 AM, David Epstein wrote:

> Thanks for the helpful advice.
>
> In view of the conflicting advice from Joe and Kevin I tried creating a couple of very small directories test1 and test2, to see what happens when the rsync command
> rsync -avi —dry-run test1/ test2/
> is given. I was pleased to see that —dry-run does explicitly say when a file is being deleted.
> Joe’s email is correct and it seems that Kevin’s version would not satisfy my third requirement.
>
> Joe says “It depends on exactly what you want to achieve”. I think my objectives were very clearly stated, but I’m happy to clarify if necessary.
>
> Could someone possibly elaborate on Joe’s advice, with more detail? I would be very grateful.
> On the first pass, it seems that one should just do
> rsync -avi -E SOURCE/ TARGET/
> Howevr t’s not clear to me what the rsync command should be on the second pass. I could easily generate a list of files/directories in TARGET/ that should not be deleted or changed, and put these into a file “somewhere". Then I could give an rsync command with the option —delete, but how do I protect the entries in my “somewhere” file from deletion?
>
> thanks for any help
> David
>
>> On 23 Feb 2017, at 01:25, Joe <[hidden email]> wrote:
>>
>> I'm probably missing something, but it looks like your first requirement rules out using --delete,
>> but your third requirement seems to imply the need for it because you want the source and target to be identical - and it won't be if there are any files on the target which are not on the source.
>>
>> If this is the case, then you may have to use more than pass - the first at the top-level without the --delete and one or more at a lower level with --delete, avoiding those top-level directories/files which do not exist on the source.
>>
>> It depends on exactly what you want to achieve.
>>
>> --dry-run is your friend.
>>
>> On 02/22/2017 10:36 AM, Kevin Korb wrote:
>>> You should be fine as long as you don't add --delete.
>>>
>>> I would start with rsync -vai --dry-run SOURCE/ TARGET/
>>> Add whatever the OSX specific option is (I think -E)
>>>
>>> Yes, --dry-run shows you what it would have done without --dry-run and
>>> yes, if you can't read all the files then you need to run it as root.
>>>
>>> On 02/22/2017 08:57 AM, David Epstein wrote:
>>>> All the files and directories I want to talk about are on the same machine (a Mac).  I have a directory that I will call SOURCE. This contains a a number of files, some of which are directories containing further files. I want to copy these files to another directory, which I will call TARGET. Please assume that SOURCE and TARGET include absolute path-names. I think they should also end in a slash (subject to correction).
>>>>
>>>> If a top-level file/directory is present in TARGET, but not in SOURCE, I do not want it to be disturbed.
>>>> If a top-level file/directory is not present in TARGET, but is present in SOURCE, I want it to be recursively copied into TARGET
>>>> If a top-level file/directory is present in TARGET and also present in SOURCE, I want it to overwrite, so that the ccorresponding file/directory in TARGET at the end of the rsync job is a copy of what is in SOURCE, with the same permissions and date-stamps.
>>>>
>>>> Since it’s on a Mac, I also want to transfer extended attributes.
>>>>
>>>> I would be grateful for the recommended recipe.
>>>>
>>>> Would dry-run tell me what rsync proposed to delete as well as what it proposed to install. It’s because I don’t understand the option dry-run fully that I’m reluctant to use trial-and-error to get to the right command.
>>>>
>>>> I think I would have to use “sudo” since some of the files are not readable by me when I am an ordinaty user.
>>>>
>>>> Thanks
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>> --
>> Please use reply-all for most replies to avoid omitting the mailing list.
>> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
>> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
        Kevin Korb Phone:    (407) 252-6853
        Systems Administrator Internet:
        FutureQuest, Inc. [hidden email]  (work)
        Orlando, Florida [hidden email] (personal)
        Web page: http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

signature.asc (197 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: which rsync command?

Francis.Montagnac-2

Hi.

On Thu, 23 Feb 2017 11:07:16 -0500 Kevin Korb wrote:

> I hate to say it because it goes against my normal advice but this is
> one instance where using a * in the source parameter would help...

I totally agree.

I thing that using a protect filter achieves this goal (without using
a * thus).

Example:

    mkdir SOURCE/{a,b,c}
    mkdir TARGET/d

    ## FAIL: delete d
    rsync --dry-run -ai --delete SOURCE/ TARGET/
    *deleting   d/
    .d..t...... ./
    cd+++++++++ a/
    cd+++++++++ b/
    cd+++++++++ c/

    ## OK: for this trivial test
    rsync --dry-run -ai --delete --filter 'protect /*' SOURCE/ TARGET/
    .d..t...... ./
    cd+++++++++ a/
    cd+++++++++ b/
    cd+++++++++ c/

You may need to read carefully the FILTER RULES section of the rsync
manual. I didn't count the number of time I read it :-)

---
Francis

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: which rsync command?

David Epstein
This is looking good and very helpful to an rsync novice. I will try the modification

rsync --dry-run -avi --delete --filter 'protect /*’ --filter ‘protect /.*’  SOURCE/ TARGET/

and see what it produces. I do have a number of directories and files beginning with a dot in TARGET and these need to be protected.
Unfortunately, the output from —dry-run is still likely to be sufficiently extensive that looking over it won’t be a completely certain test.

I therefore ask
"Does the above modification seem right to the experts?”

Two supplementary points, if I may. Do you advise -ai or -avi? And why do you prefer one over the other?
I don’t understand the line of rsync output that reads

 .d..t...... ./

David

> On 23 Feb 2017, at 17:02, <[hidden email]> <[hidden email]> wrote:
>
>
> Hi.
>
> On Thu, 23 Feb 2017 11:07:16 -0500 Kevin Korb wrote:
>
>> I hate to say it because it goes against my normal advice but this is
>> one instance where using a * in the source parameter would help...
>
> I totally agree.
>
> I thing that using a protect filter achieves this goal (without using
> a * thus).
>
> Example:
>
>    mkdir SOURCE/{a,b,c}
>    mkdir TARGET/d
>
>    ## FAIL: delete d
>    rsync --dry-run -ai --delete SOURCE/ TARGET/
>    *deleting   d/
>    .d..t...... ./
>    cd+++++++++ a/
>    cd+++++++++ b/
>    cd+++++++++ c/
>
>    ## OK: for this trivial test
>    rsync --dry-run -ai --delete --filter 'protect /*' SOURCE/ TARGET/
>    .d..t...... ./
>    cd+++++++++ a/
>    cd+++++++++ b/
>    cd+++++++++ c/
>
> You may need to read carefully the FILTER RULES section of the rsync
> manual. I didn't count the number of time I read it :-)
>
> ---
> Francis


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: which rsync command?

Kevin Korb
Either solution should work fine.  Your dry run results should be the same.

-v and -i are only output modifying options they don't actually change
the way that rsync works.  The extra string there is explained in the
--itemize-changes section of the man page but that particular one means
that the times tamp on the directory is different.

On 02/23/2017 01:07 PM, David Epstein wrote:

> This is looking good and very helpful to an rsync novice. I will try the modification
>
> rsync --dry-run -avi --delete --filter 'protect /*’ --filter ‘protect /.*’  SOURCE/ TARGET/
>
> and see what it produces. I do have a number of directories and files beginning with a dot in TARGET and these need to be protected.
> Unfortunately, the output from —dry-run is still likely to be sufficiently extensive that looking over it won’t be a completely certain test.
>
> I therefore ask
> "Does the above modification seem right to the experts?”
>
> Two supplementary points, if I may. Do you advise -ai or -avi? And why do you prefer one over the other?
> I don’t understand the line of rsync output that reads
>
>  .d..t...... ./
>
> David
>
>> On 23 Feb 2017, at 17:02, <[hidden email]> <[hidden email]> wrote:
>>
>>
>> Hi.
>>
>> On Thu, 23 Feb 2017 11:07:16 -0500 Kevin Korb wrote:
>>
>>> I hate to say it because it goes against my normal advice but this is
>>> one instance where using a * in the source parameter would help...
>>
>> I totally agree.
>>
>> I thing that using a protect filter achieves this goal (without using
>> a * thus).
>>
>> Example:
>>
>>    mkdir SOURCE/{a,b,c}
>>    mkdir TARGET/d
>>
>>    ## FAIL: delete d
>>    rsync --dry-run -ai --delete SOURCE/ TARGET/
>>    *deleting   d/
>>    .d..t...... ./
>>    cd+++++++++ a/
>>    cd+++++++++ b/
>>    cd+++++++++ c/
>>
>>    ## OK: for this trivial test
>>    rsync --dry-run -ai --delete --filter 'protect /*' SOURCE/ TARGET/
>>    .d..t...... ./
>>    cd+++++++++ a/
>>    cd+++++++++ b/
>>    cd+++++++++ c/
>>
>> You may need to read carefully the FILTER RULES section of the rsync
>> manual. I didn't count the number of time I read it :-)
>>
>> ---
>> Francis
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
        Kevin Korb Phone:    (407) 252-6853
        Systems Administrator Internet:
        FutureQuest, Inc. [hidden email]  (work)
        Orlando, Florida [hidden email] (personal)
        Web page: http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

signature.asc (197 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: which rsync command?

Francis.Montagnac-2
In reply to this post by David Epstein


On Thu, 23 Feb 2017 18:07:20 +0000 David Epstein wrote:

> I will try the modification

> rsync --dry-run -avi --delete --filter 'protect /*’ --filter ‘protect /.*’  SOURCE/ TARGET/

The 'protect /.*’ is useless: unlike the shell, rsync interprets *
simply as a path component, regardless thus if it starts with a dot or
not. From the man:

   a '*' matches any path component, but it stops at slashes.

> Unfortunately, the output from —dry-run is still likely to be
> sufficiently extensive that looking over it won’t be a completely
> certain test.

Then redirect to a file and grep deleting in this file.

---
Francis

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: which rsync command?

Francis.Montagnac-2


On Fri, 24 Feb 2017 07:51:56 +0100 [hidden email] wrote:

>> Unfortunately, the output from —dry-run is still likely to be
>> sufficiently extensive that looking over it won’t be a completely
>> certain test.

> Then redirect to a file and grep deleting in this file.

Given the format of --itemize-changes, you can also use awk to
restrict the output to the direct subdirs.

Example;

    mkdir -p SOURCE/a/x
    mkdir TARGET/.z

    rsync --dry-run -ai --delete  SOURCE/ TARGET/
    *deleting   d/
    *deleting   .z/
    .d..t...... ./
    cd+++++++++ a/
    cd+++++++++ a/x/
    cd+++++++++ b/
    cd+++++++++ c/

    ## FAIL
    rsync --dry-run -ai --delete  SOURCE/ TARGET/ | awk -F/ 'NF == 2 && $1 ~ /deleting/ {print}'
    *deleting   d/
    *deleting   .z/

    ## OK: no output
    rsync --dry-run -ai --delete --filter 'protect /*'  SOURCE/ TARGET/ | awk -F/ 'NF == 2 && $1 ~ /deleting/ {print}'

--
Francis

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Loading...