rsync keeps writing files over

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

rsync keeps writing files over

McDowell, Blake

Hi,

 

At my work we use rsync to move files between drives and to LTO among other things.

 

I’m having an issue using rsync to move material between and external drive and an internal drive.

 

We run “rsync –avvPh <src> <dest>” and most of the files keep writing every time I run this. It appears that the modification times are not being carried through to the destination resulting in the files continually wanting to re-write.

 

I’m hoping someone can help me figure this out. Any information you need from me (logs, etc) I’m happy to try and provide.

 

Many thanks,

Blake  


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync keeps writing files over

Kevin Korb
Instead of the second -v (or even the first) add --itemize-changes.  It
will tell you why each file is being copied.  If the file timestamps are
not correct then perhaps the underlying storage doesn't allow arbitrary
mtime changes.

On 06/02/2016 05:23 PM, McDowell, Blake wrote:

> Hi,
>
>  
>
> At my work we use rsync to move files between drives and to LTO among
> other things.
>
>  
>
> I’m having an issue using rsync to move material between and external
> drive and an internal drive.
>
>  
>
> We run “rsync –avvPh <src> <dest>” and most of the files keep writing
> every time I run this. It appears that the modification times are not
> being carried through to the destination resulting in the files
> continually wanting to re-write.
>
>  
>
> I’m hoping someone can help me figure this out. Any information you need
> from me (logs, etc) I’m happy to try and provide.
>
>  
>
> Many thanks,
>
> Blake  
>
>
>
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
        Kevin Korb Phone:    (407) 252-6853
        Systems Administrator Internet:
        FutureQuest, Inc. [hidden email]  (work)
        Orlando, Florida [hidden email] (personal)
        Web page: http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

signature.asc (188 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: rsync keeps writing files over

McDowell, Blake
Thanks Kevin! I¹m unclear how to read the ‹itemize-changes output. Can you
provide some insight?

This is a local transfer from an external drive to an internal drive all
attached to one computer.


rsync -aPh --itemize-changes -n
/Volumes/shuttle_05/2012_79_1_14_1__1199_Workprint /Volumes/3TB_LTO/LT003A/
sending incremental file list

>f..t.......
>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Derivatives
>/2012_79_1_14_1_DER_01.mov
>f..t.......
>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Derivatives
>/2012_79_1_14_1_DER_02.mov
>f..t.......
>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/119
>9WP_00086400.dpx
>f..t.......
>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/119
>9WP_00086401.dpx
>f..t.......
>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/119
>9WP_00086402.dpx
>f..t.......
>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/119
>9WP_00086403.dpx
>f..t.......
>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/119
>9WP_00086404.dpx
>f..t.......
>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/119
>9WP_00086405.dpx
>f..t.......
>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/119
>9WP_00086406.dpx


~Blake



On 6/2/16, 5:44 PM, "rsync on behalf of Kevin Korb"
<[hidden email] on behalf of [hidden email]> wrote:

>Instead of the second -v (or even the first) add --itemize-changes.  It
>will tell you why each file is being copied.  If the file timestamps are
>not correct then perhaps the underlying storage doesn't allow arbitrary
>mtime changes.
>
>On 06/02/2016 05:23 PM, McDowell, Blake wrote:
>> Hi,
>>
>>  
>>
>> At my work we use rsync to move files between drives and to LTO among
>> other things.
>>
>>  
>>
>> I¹m having an issue using rsync to move material between and external
>> drive and an internal drive.
>>
>>  
>>
>> We run ³rsync ­avvPh <src> <dest>² and most of the files keep writing
>> every time I run this. It appears that the modification times are not
>> being carried through to the destination resulting in the files
>> continually wanting to re-write.
>>
>>  
>>
>> I¹m hoping someone can help me figure this out. Any information you need
>> from me (logs, etc) I¹m happy to try and provide.
>>
>>  
>>
>> Many thanks,
>>
>> Blake  
>>
>>
>>
>
>--
>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> Kevin Korb Phone:    (407) 252-6853
> Systems Administrator Internet:
> FutureQuest, Inc. [hidden email]  (work)
> Orlando, Florida [hidden email] (personal)
> Web page: http://www.sanitarium.net/
> PGP public key available on web site.
>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync keeps writing files over

Kevin Korb
It is saying the timestamp is wrong and that it is copying the file and
changing the timestamp.  If it does that every time then either the
timestamps are changing on the source or the target isn't storing them.

On 06/02/2016 06:13 PM, McDowell, Blake wrote:

> Thanks Kevin! I¹m unclear how to read the ‹itemize-changes output. Can you
> provide some insight?
>
> This is a local transfer from an external drive to an internal drive all
> attached to one computer.
>
>
> rsync -aPh --itemize-changes -n
> /Volumes/shuttle_05/2012_79_1_14_1__1199_Workprint /Volumes/3TB_LTO/LT003A/
> sending incremental file list
>> f..t.......
>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Derivatives
>> /2012_79_1_14_1_DER_01.mov
>> f..t.......
>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Derivatives
>> /2012_79_1_14_1_DER_02.mov
>> f..t.......
>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/119
>> 9WP_00086400.dpx
>> f..t.......
>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/119
>> 9WP_00086401.dpx
>> f..t.......
>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/119
>> 9WP_00086402.dpx
>> f..t.......
>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/119
>> 9WP_00086403.dpx
>> f..t.......
>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/119
>> 9WP_00086404.dpx
>> f..t.......
>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/119
>> 9WP_00086405.dpx
>> f..t.......
>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/119
>> 9WP_00086406.dpx
>
>
> ~Blake
>
>
>
> On 6/2/16, 5:44 PM, "rsync on behalf of Kevin Korb"
> <[hidden email] on behalf of [hidden email]> wrote:
>
>> Instead of the second -v (or even the first) add --itemize-changes.  It
>> will tell you why each file is being copied.  If the file timestamps are
>> not correct then perhaps the underlying storage doesn't allow arbitrary
>> mtime changes.
>>
>> On 06/02/2016 05:23 PM, McDowell, Blake wrote:
>>> Hi,
>>>
>>>  
>>>
>>> At my work we use rsync to move files between drives and to LTO among
>>> other things.
>>>
>>>  
>>>
>>> I¹m having an issue using rsync to move material between and external
>>> drive and an internal drive.
>>>
>>>  
>>>
>>> We run ³rsync ­avvPh <src> <dest>² and most of the files keep writing
>>> every time I run this. It appears that the modification times are not
>>> being carried through to the destination resulting in the files
>>> continually wanting to re-write.
>>>
>>>  
>>>
>>> I¹m hoping someone can help me figure this out. Any information you need
>>> from me (logs, etc) I¹m happy to try and provide.
>>>
>>>  
>>>
>>> Many thanks,
>>>
>>> Blake  
>>>
>>>
>>>
>>
>> --
>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>> Kevin Korb Phone:    (407) 252-6853
>> Systems Administrator Internet:
>> FutureQuest, Inc. [hidden email]  (work)
>> Orlando, Florida [hidden email] (personal)
>> Web page: http://www.sanitarium.net/
>> PGP public key available on web site.
>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>>
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
        Kevin Korb Phone:    (407) 252-6853
        Systems Administrator Internet:
        FutureQuest, Inc. [hidden email]  (work)
        Orlando, Florida [hidden email] (personal)
        Web page: http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

signature.asc (188 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: rsync keeps writing files over

McDowell, Blake
OK. Thanks. Where can I find information regarding how to interpret
—itemize-changes?

The timestamps aren’t changing, so the target must not be storing them,
which I have no idea why. The directory I’m writing to is 777.

What is the flag to tell rsync to ignore the timestamps?

Thanks,
Blake

On 6/2/16, 6:18 PM, "rsync on behalf of Kevin Korb"
<[hidden email] on behalf of [hidden email]> wrote:

>It is saying the timestamp is wrong and that it is copying the file and
>changing the timestamp.  If it does that every time then either the
>timestamps are changing on the source or the target isn't storing them.
>
>On 06/02/2016 06:13 PM, McDowell, Blake wrote:
>> Thanks Kevin! I¹m unclear how to read the ‹itemize-changes output. Can
>>you
>> provide some insight?
>>
>> This is a local transfer from an external drive to an internal drive all
>> attached to one computer.
>>
>>
>> rsync -aPh --itemize-changes -n
>> /Volumes/shuttle_05/2012_79_1_14_1__1199_Workprint
>>/Volumes/3TB_LTO/LT003A/
>> sending incremental file list
>>> f..t.......
>>>
>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Derivativ
>>>es
>>> /2012_79_1_14_1_DER_01.mov
>>> f..t.......
>>>
>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Derivativ
>>>es
>>> /2012_79_1_14_1_DER_02.mov
>>> f..t.......
>>>
>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/1
>>>19
>>> 9WP_00086400.dpx
>>> f..t.......
>>>
>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/1
>>>19
>>> 9WP_00086401.dpx
>>> f..t.......
>>>
>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/1
>>>19
>>> 9WP_00086402.dpx
>>> f..t.......
>>>
>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/1
>>>19
>>> 9WP_00086403.dpx
>>> f..t.......
>>>
>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/1
>>>19
>>> 9WP_00086404.dpx
>>> f..t.......
>>>
>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/1
>>>19
>>> 9WP_00086405.dpx
>>> f..t.......
>>>
>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/1
>>>19
>>> 9WP_00086406.dpx
>>
>>
>> ~Blake
>>
>>
>>
>> On 6/2/16, 5:44 PM, "rsync on behalf of Kevin Korb"
>> <[hidden email] on behalf of [hidden email]> wrote:
>>
>>> Instead of the second -v (or even the first) add --itemize-changes.  It
>>> will tell you why each file is being copied.  If the file timestamps
>>>are
>>> not correct then perhaps the underlying storage doesn't allow arbitrary
>>> mtime changes.
>>>
>>> On 06/02/2016 05:23 PM, McDowell, Blake wrote:
>>>> Hi,
>>>>
>>>>  
>>>>
>>>> At my work we use rsync to move files between drives and to LTO among
>>>> other things.
>>>>
>>>>  
>>>>
>>>> I¹m having an issue using rsync to move material between and external
>>>> drive and an internal drive.
>>>>
>>>>  
>>>>
>>>> We run ³rsync ­avvPh <src> <dest>² and most of the files keep writing
>>>> every time I run this. It appears that the modification times are not
>>>> being carried through to the destination resulting in the files
>>>> continually wanting to re-write.
>>>>
>>>>  
>>>>
>>>> I¹m hoping someone can help me figure this out. Any information you
>>>>need
>>>> from me (logs, etc) I¹m happy to try and provide.
>>>>
>>>>  
>>>>
>>>> Many thanks,
>>>>
>>>> Blake  
>>>>
>>>>
>>>>
>>>
>>> --
>>>
>>>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>>> Kevin Korb Phone:    (407) 252-6853
>>> Systems Administrator Internet:
>>> FutureQuest, Inc. [hidden email]  (work)
>>> Orlando, Florida [hidden email] (personal)
>>> Web page: http://www.sanitarium.net/
>>> PGP public key available on web site.
>>>
>>>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>>>
>
>--
>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> Kevin Korb Phone:    (407) 252-6853
> Systems Administrator Internet:
> FutureQuest, Inc. [hidden email]  (work)
> Orlando, Florida [hidden email] (personal)
> Web page: http://www.sanitarium.net/
> PGP public key available on web site.
>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync keeps writing files over

Kevin Korb
The man page has a section on what all the itemize-changes flags do.

There is a --ignore-times but the result is what you have now, re-copy
everything even if the timestamp matches.

The best you can really do with storage that can't handle timestamps is
to use --update.  But if you do that you need to get rid of --partial
(part of -P) or else rsync will never complete a file that was partially
copied since the partial copy is newer that the source file.

OTOH, are you using rsync to copy files to a tape drive?  If so then I
would think rsync is not the right tool for dealing with linear storage.

On 06/02/2016 06:25 PM, McDowell, Blake wrote:

> OK. Thanks. Where can I find information regarding how to interpret
> —itemize-changes?
>
> The timestamps aren’t changing, so the target must not be storing them,
> which I have no idea why. The directory I’m writing to is 777.
>
> What is the flag to tell rsync to ignore the timestamps?
>
> Thanks,
> Blake
>
> On 6/2/16, 6:18 PM, "rsync on behalf of Kevin Korb"
> <[hidden email] on behalf of [hidden email]> wrote:
>
>> It is saying the timestamp is wrong and that it is copying the file and
>> changing the timestamp.  If it does that every time then either the
>> timestamps are changing on the source or the target isn't storing them.
>>
>> On 06/02/2016 06:13 PM, McDowell, Blake wrote:
>>> Thanks Kevin! I¹m unclear how to read the ‹itemize-changes output. Can
>>> you
>>> provide some insight?
>>>
>>> This is a local transfer from an external drive to an internal drive all
>>> attached to one computer.
>>>
>>>
>>> rsync -aPh --itemize-changes -n
>>> /Volumes/shuttle_05/2012_79_1_14_1__1199_Workprint
>>> /Volumes/3TB_LTO/LT003A/
>>> sending incremental file list
>>>> f..t.......
>>>>
>>>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Derivativ
>>>> es
>>>> /2012_79_1_14_1_DER_01.mov
>>>> f..t.......
>>>>
>>>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Derivativ
>>>> es
>>>> /2012_79_1_14_1_DER_02.mov
>>>> f..t.......
>>>>
>>>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/1
>>>> 19
>>>> 9WP_00086400.dpx
>>>> f..t.......
>>>>
>>>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/1
>>>> 19
>>>> 9WP_00086401.dpx
>>>> f..t.......
>>>>
>>>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/1
>>>> 19
>>>> 9WP_00086402.dpx
>>>> f..t.......
>>>>
>>>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/1
>>>> 19
>>>> 9WP_00086403.dpx
>>>> f..t.......
>>>>
>>>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/1
>>>> 19
>>>> 9WP_00086404.dpx
>>>> f..t.......
>>>>
>>>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/1
>>>> 19
>>>> 9WP_00086405.dpx
>>>> f..t.......
>>>>
>>>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/1
>>>> 19
>>>> 9WP_00086406.dpx
>>>
>>>
>>> ~Blake
>>>
>>>
>>>
>>> On 6/2/16, 5:44 PM, "rsync on behalf of Kevin Korb"
>>> <[hidden email] on behalf of [hidden email]> wrote:
>>>
>>>> Instead of the second -v (or even the first) add --itemize-changes.  It
>>>> will tell you why each file is being copied.  If the file timestamps
>>>> are
>>>> not correct then perhaps the underlying storage doesn't allow arbitrary
>>>> mtime changes.
>>>>
>>>> On 06/02/2016 05:23 PM, McDowell, Blake wrote:
>>>>> Hi,
>>>>>
>>>>>  
>>>>>
>>>>> At my work we use rsync to move files between drives and to LTO among
>>>>> other things.
>>>>>
>>>>>  
>>>>>
>>>>> I¹m having an issue using rsync to move material between and external
>>>>> drive and an internal drive.
>>>>>
>>>>>  
>>>>>
>>>>> We run ³rsync ­avvPh <src> <dest>² and most of the files keep writing
>>>>> every time I run this. It appears that the modification times are not
>>>>> being carried through to the destination resulting in the files
>>>>> continually wanting to re-write.
>>>>>
>>>>>  
>>>>>
>>>>> I¹m hoping someone can help me figure this out. Any information you
>>>>> need
>>>>> from me (logs, etc) I¹m happy to try and provide.
>>>>>
>>>>>  
>>>>>
>>>>> Many thanks,
>>>>>
>>>>> Blake  
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>>>> Kevin Korb Phone:    (407) 252-6853
>>>> Systems Administrator Internet:
>>>> FutureQuest, Inc. [hidden email]  (work)
>>>> Orlando, Florida [hidden email] (personal)
>>>> Web page: http://www.sanitarium.net/
>>>> PGP public key available on web site.
>>>>
>>>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>>>>
>>
>> --
>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>> Kevin Korb Phone:    (407) 252-6853
>> Systems Administrator Internet:
>> FutureQuest, Inc. [hidden email]  (work)
>> Orlando, Florida [hidden email] (personal)
>> Web page: http://www.sanitarium.net/
>> PGP public key available on web site.
>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>>
>
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
        Kevin Korb Phone:    (407) 252-6853
        Systems Administrator Internet:
        FutureQuest, Inc. [hidden email]  (work)
        Orlando, Florida [hidden email] (personal)
        Web page: http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

signature.asc (188 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: rsync keeps writing files over

McDowell, Blake
Cool Thanks!

Specifically, the timestamps on both <src> and <dest> match for "ls -l"
but do not match for "ls -lu" or "ls -lc”

The storage is just an regular HDD in a mac pro tower. I can’t imagine why
it wouldn’t handle timestamps. Also of note - this problem doesn’t exist
for every file, just the vast majority. So, that just makes it more
confusing.

Yes, I imagine rsync is not the best for linear tape but give the choice
between cp (which is faster and causes less problems but offers almost
zero verbosity) and rsync, I’ll choose rsync. If people know of other
options, I’d be very happy to know of them.

On 6/2/16, 6:33 PM, "Kevin Korb" <[hidden email]> wrote:

>The man page has a section on what all the itemize-changes flags do.
>
>There is a --ignore-times but the result is what you have now, re-copy
>everything even if the timestamp matches.
>
>The best you can really do with storage that can't handle timestamps is
>to use --update.  But if you do that you need to get rid of --partial
>(part of -P) or else rsync will never complete a file that was partially
>copied since the partial copy is newer that the source file.
>
>OTOH, are you using rsync to copy files to a tape drive?  If so then I
>would think rsync is not the right tool for dealing with linear storage.
>
>On 06/02/2016 06:25 PM, McDowell, Blake wrote:
>> OK. Thanks. Where can I find information regarding how to interpret
>> —itemize-changes?
>>
>> The timestamps aren’t changing, so the target must not be storing them,
>> which I have no idea why. The directory I’m writing to is 777.
>>
>> What is the flag to tell rsync to ignore the timestamps?
>>
>> Thanks,
>> Blake
>>
>> On 6/2/16, 6:18 PM, "rsync on behalf of Kevin Korb"
>> <[hidden email] on behalf of [hidden email]> wrote:
>>
>>> It is saying the timestamp is wrong and that it is copying the file and
>>> changing the timestamp.  If it does that every time then either the
>>> timestamps are changing on the source or the target isn't storing them.
>>>
>>> On 06/02/2016 06:13 PM, McDowell, Blake wrote:
>>>> Thanks Kevin! I¹m unclear how to read the ‹itemize-changes output. Can
>>>> you
>>>> provide some insight?
>>>>
>>>> This is a local transfer from an external drive to an internal drive
>>>>all
>>>> attached to one computer.
>>>>
>>>>
>>>> rsync -aPh --itemize-changes -n
>>>> /Volumes/shuttle_05/2012_79_1_14_1__1199_Workprint
>>>> /Volumes/3TB_LTO/LT003A/
>>>> sending incremental file list
>>>>> f..t.......
>>>>>
>>>>>
>>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Derivat
>>>>>iv
>>>>> es
>>>>> /2012_79_1_14_1_DER_01.mov
>>>>> f..t.......
>>>>>
>>>>>
>>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Derivat
>>>>>iv
>>>>> es
>>>>> /2012_79_1_14_1_DER_02.mov
>>>>> f..t.......
>>>>>
>>>>>
>>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX
>>>>>/1
>>>>> 19
>>>>> 9WP_00086400.dpx
>>>>> f..t.......
>>>>>
>>>>>
>>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX
>>>>>/1
>>>>> 19
>>>>> 9WP_00086401.dpx
>>>>> f..t.......
>>>>>
>>>>>
>>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX
>>>>>/1
>>>>> 19
>>>>> 9WP_00086402.dpx
>>>>> f..t.......
>>>>>
>>>>>
>>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX
>>>>>/1
>>>>> 19
>>>>> 9WP_00086403.dpx
>>>>> f..t.......
>>>>>
>>>>>
>>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX
>>>>>/1
>>>>> 19
>>>>> 9WP_00086404.dpx
>>>>> f..t.......
>>>>>
>>>>>
>>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX
>>>>>/1
>>>>> 19
>>>>> 9WP_00086405.dpx
>>>>> f..t.......
>>>>>
>>>>>
>>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX
>>>>>/1
>>>>> 19
>>>>> 9WP_00086406.dpx
>>>>
>>>>
>>>> ~Blake
>>>>
>>>>
>>>>
>>>> On 6/2/16, 5:44 PM, "rsync on behalf of Kevin Korb"
>>>> <[hidden email] on behalf of [hidden email]> wrote:
>>>>
>>>>> Instead of the second -v (or even the first) add --itemize-changes.
>>>>>It
>>>>> will tell you why each file is being copied.  If the file timestamps
>>>>> are
>>>>> not correct then perhaps the underlying storage doesn't allow
>>>>>arbitrary
>>>>> mtime changes.
>>>>>
>>>>> On 06/02/2016 05:23 PM, McDowell, Blake wrote:
>>>>>> Hi,
>>>>>>
>>>>>>  
>>>>>>
>>>>>> At my work we use rsync to move files between drives and to LTO
>>>>>>among
>>>>>> other things.
>>>>>>
>>>>>>  
>>>>>>
>>>>>> I¹m having an issue using rsync to move material between and
>>>>>>external
>>>>>> drive and an internal drive.
>>>>>>
>>>>>>  
>>>>>>
>>>>>> We run ³rsync ­avvPh <src> <dest>² and most of the files keep
>>>>>>writing
>>>>>> every time I run this. It appears that the modification times are
>>>>>>not
>>>>>> being carried through to the destination resulting in the files
>>>>>> continually wanting to re-write.
>>>>>>
>>>>>>  
>>>>>>
>>>>>> I¹m hoping someone can help me figure this out. Any information you
>>>>>> need
>>>>>> from me (logs, etc) I¹m happy to try and provide.
>>>>>>
>>>>>>  
>>>>>>
>>>>>> Many thanks,
>>>>>>
>>>>>> Blake  
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>>
>>>>>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._
>>>>>.,
>>>>> Kevin Korb Phone:    (407) 252-6853
>>>>> Systems Administrator Internet:
>>>>> FutureQuest, Inc. [hidden email]  (work)
>>>>> Orlando, Florida [hidden email] (personal)
>>>>> Web page: http://www.sanitarium.net/
>>>>> PGP public key available on web site.
>>>>>
>>>>>
>>>>>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._
>>>>>.,
>>>>>
>>>
>>> --
>>>
>>>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>>> Kevin Korb Phone:    (407) 252-6853
>>> Systems Administrator Internet:
>>> FutureQuest, Inc. [hidden email]  (work)
>>> Orlando, Florida [hidden email] (personal)
>>> Web page: http://www.sanitarium.net/
>>> PGP public key available on web site.
>>>
>>>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>>>
>>
>
>--
>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> Kevin Korb Phone:    (407) 252-6853
> Systems Administrator Internet:
> FutureQuest, Inc. [hidden email]  (work)
> Orlando, Florida [hidden email] (personal)
> Web page: http://www.sanitarium.net/
> PGP public key available on web site.
>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync keeps writing files over

Kevin Korb
Rsync only cares about the modification time.  The ls command usually
abbreviates the timestamp so it is better to use the stat command on one
of the problem files to see the full thing.

On 06/02/2016 06:42 PM, McDowell, Blake wrote:

> Cool Thanks!
>
> Specifically, the timestamps on both <src> and <dest> match for "ls -l"
> but do not match for "ls -lu" or "ls -lc”
>
> The storage is just an regular HDD in a mac pro tower. I can’t imagine why
> it wouldn’t handle timestamps. Also of note - this problem doesn’t exist
> for every file, just the vast majority. So, that just makes it more
> confusing.
>
> Yes, I imagine rsync is not the best for linear tape but give the choice
> between cp (which is faster and causes less problems but offers almost
> zero verbosity) and rsync, I’ll choose rsync. If people know of other
> options, I’d be very happy to know of them.
>
> On 6/2/16, 6:33 PM, "Kevin Korb" <[hidden email]> wrote:
>
>> The man page has a section on what all the itemize-changes flags do.
>>
>> There is a --ignore-times but the result is what you have now, re-copy
>> everything even if the timestamp matches.
>>
>> The best you can really do with storage that can't handle timestamps is
>> to use --update.  But if you do that you need to get rid of --partial
>> (part of -P) or else rsync will never complete a file that was partially
>> copied since the partial copy is newer that the source file.
>>
>> OTOH, are you using rsync to copy files to a tape drive?  If so then I
>> would think rsync is not the right tool for dealing with linear storage.
>>
>> On 06/02/2016 06:25 PM, McDowell, Blake wrote:
>>> OK. Thanks. Where can I find information regarding how to interpret
>>> —itemize-changes?
>>>
>>> The timestamps aren’t changing, so the target must not be storing them,
>>> which I have no idea why. The directory I’m writing to is 777.
>>>
>>> What is the flag to tell rsync to ignore the timestamps?
>>>
>>> Thanks,
>>> Blake
>>>
>>> On 6/2/16, 6:18 PM, "rsync on behalf of Kevin Korb"
>>> <[hidden email] on behalf of [hidden email]> wrote:
>>>
>>>> It is saying the timestamp is wrong and that it is copying the file and
>>>> changing the timestamp.  If it does that every time then either the
>>>> timestamps are changing on the source or the target isn't storing them.
>>>>
>>>> On 06/02/2016 06:13 PM, McDowell, Blake wrote:
>>>>> Thanks Kevin! I¹m unclear how to read the ‹itemize-changes output. Can
>>>>> you
>>>>> provide some insight?
>>>>>
>>>>> This is a local transfer from an external drive to an internal drive
>>>>> all
>>>>> attached to one computer.
>>>>>
>>>>>
>>>>> rsync -aPh --itemize-changes -n
>>>>> /Volumes/shuttle_05/2012_79_1_14_1__1199_Workprint
>>>>> /Volumes/3TB_LTO/LT003A/
>>>>> sending incremental file list
>>>>>> f..t.......
>>>>>>
>>>>>>
>>>>>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Derivat
>>>>>> iv
>>>>>> es
>>>>>> /2012_79_1_14_1_DER_01.mov
>>>>>> f..t.......
>>>>>>
>>>>>>
>>>>>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Derivat
>>>>>> iv
>>>>>> es
>>>>>> /2012_79_1_14_1_DER_02.mov
>>>>>> f..t.......
>>>>>>
>>>>>>
>>>>>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX
>>>>>> /1
>>>>>> 19
>>>>>> 9WP_00086400.dpx
>>>>>> f..t.......
>>>>>>
>>>>>>
>>>>>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX
>>>>>> /1
>>>>>> 19
>>>>>> 9WP_00086401.dpx
>>>>>> f..t.......
>>>>>>
>>>>>>
>>>>>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX
>>>>>> /1
>>>>>> 19
>>>>>> 9WP_00086402.dpx
>>>>>> f..t.......
>>>>>>
>>>>>>
>>>>>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX
>>>>>> /1
>>>>>> 19
>>>>>> 9WP_00086403.dpx
>>>>>> f..t.......
>>>>>>
>>>>>>
>>>>>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX
>>>>>> /1
>>>>>> 19
>>>>>> 9WP_00086404.dpx
>>>>>> f..t.......
>>>>>>
>>>>>>
>>>>>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX
>>>>>> /1
>>>>>> 19
>>>>>> 9WP_00086405.dpx
>>>>>> f..t.......
>>>>>>
>>>>>>
>>>>>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX
>>>>>> /1
>>>>>> 19
>>>>>> 9WP_00086406.dpx
>>>>>
>>>>>
>>>>> ~Blake
>>>>>
>>>>>
>>>>>
>>>>> On 6/2/16, 5:44 PM, "rsync on behalf of Kevin Korb"
>>>>> <[hidden email] on behalf of [hidden email]> wrote:
>>>>>
>>>>>> Instead of the second -v (or even the first) add --itemize-changes.
>>>>>> It
>>>>>> will tell you why each file is being copied.  If the file timestamps
>>>>>> are
>>>>>> not correct then perhaps the underlying storage doesn't allow
>>>>>> arbitrary
>>>>>> mtime changes.
>>>>>>
>>>>>> On 06/02/2016 05:23 PM, McDowell, Blake wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>>  
>>>>>>>
>>>>>>> At my work we use rsync to move files between drives and to LTO
>>>>>>> among
>>>>>>> other things.
>>>>>>>
>>>>>>>  
>>>>>>>
>>>>>>> I¹m having an issue using rsync to move material between and
>>>>>>> external
>>>>>>> drive and an internal drive.
>>>>>>>
>>>>>>>  
>>>>>>>
>>>>>>> We run ³rsync ­avvPh <src> <dest>² and most of the files keep
>>>>>>> writing
>>>>>>> every time I run this. It appears that the modification times are
>>>>>>> not
>>>>>>> being carried through to the destination resulting in the files
>>>>>>> continually wanting to re-write.
>>>>>>>
>>>>>>>  
>>>>>>>
>>>>>>> I¹m hoping someone can help me figure this out. Any information you
>>>>>>> need
>>>>>>> from me (logs, etc) I¹m happy to try and provide.
>>>>>>>
>>>>>>>  
>>>>>>>
>>>>>>> Many thanks,
>>>>>>>
>>>>>>> Blake  
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>>
>>>>>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._
>>>>>> .,
>>>>>> Kevin Korb Phone:    (407) 252-6853
>>>>>> Systems Administrator Internet:
>>>>>> FutureQuest, Inc. [hidden email]  (work)
>>>>>> Orlando, Florida [hidden email] (personal)
>>>>>> Web page: http://www.sanitarium.net/
>>>>>> PGP public key available on web site.
>>>>>>
>>>>>>
>>>>>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._
>>>>>> .,
>>>>>>
>>>>
>>>> --
>>>>
>>>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>>>> Kevin Korb Phone:    (407) 252-6853
>>>> Systems Administrator Internet:
>>>> FutureQuest, Inc. [hidden email]  (work)
>>>> Orlando, Florida [hidden email] (personal)
>>>> Web page: http://www.sanitarium.net/
>>>> PGP public key available on web site.
>>>>
>>>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>>>>
>>>
>>
>> --
>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>> Kevin Korb Phone:    (407) 252-6853
>> Systems Administrator Internet:
>> FutureQuest, Inc. [hidden email]  (work)
>> Orlando, Florida [hidden email] (personal)
>> Web page: http://www.sanitarium.net/
>> PGP public key available on web site.
>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>>
>
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
        Kevin Korb Phone:    (407) 252-6853
        Systems Administrator Internet:
        FutureQuest, Inc. [hidden email]  (work)
        Orlando, Florida [hidden email] (personal)
        Web page: http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

signature.asc (188 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: rsync keeps writing files over

Steven Levine
In reply to this post by McDowell, Blake
In <D3762D63.17A7%[hidden email]>, on 06/02/16
   at 10:42 PM, "McDowell, Blake" <[hidden email]> said:

Hi Blake,

>The storage is just an regular HDD in a mac pro tower. I can t imagine
>why it wouldn t handle timestamps. Also of note - this problem doesn t
>exist for every file, just the vast majority. So, that just makes it more
>confusing.

Are the file systems the same on the source and the destination
partitions?

Check out

  --modify-window

in the help.  If the source and destination file systems have different
timestamp precision, this is the usual solution.

You can also try

  rsync /path-to-foo

where path-to-foo is a directory or file.  This will list the file
timestamps as rsync interprets them.

BTW, what version of rsync are you running?  It might matter.

Steven

--
----------------------------------------------------------------------
"Steven Levine" <[hidden email]>  Warp/DIY/BlueLion etc.
www.scoug.com www.arcanoae.com www.warpcave.com
----------------------------------------------------------------------


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync keeps writing files over

Perry Hutchison
In reply to this post by McDowell, Blake
"McDowell, Blake" <[hidden email]> wrote:

> The storage is just an regular HDD in a mac pro tower. I can't
> imagine why it wouldn't handle timestamps. Also of note - this
> problem doesn't exist for every file, just the vast majority.
> So, that just makes it more confusing.

The filesystem format (MacOS native?  FAT?) might be a factor.

> Yes, I imagine rsync is not the best for linear tape but give the
> choice between cp (which is faster and causes less problems but
> offers almost zero verbosity) and rsync, I???ll choose rsync. If
> people know of other options, I???d be very happy to know of them.

Best choice for magtape is probably something like tar, cpio, or pax
(for a file-oriented backup), or the appropriate variant of dump(8)
(to back up an entire filesystem -- but not all FS formats have a
dump/restore suite available).

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync keeps writing files over

Simon Hobson-2
In reply to this post by McDowell, Blake
"McDowell, Blake" <[hidden email]> wrote:

> The storage is just an regular HDD in a mac pro tower.

Ah, is this the version of rsync that comes with OS X ? Are these HFS+ filesystems ?

I vaguely recall that the OS X version is "hacked" to handle the file semantics of HFS+ filesystems. Hopefully someone else actually knows the details, I could be a bit wrong here, but IIRC it's something like :
On a "*nix" filsystem, a file is a file - a chunk of data and some filesystem metadata. On HFS+, a file is comprised of up to 3 parts - the data fork, the resource fork (I don't believe this is widely used these days), and a chunk more metadata. "Regular" rsync only copies the data fork and that part of the metadata that maps to *nix filesystem semantics, the OS X version of rsync copies the whole file by way of quite a kludge - can't remember if it needs an extra cmd line switch to do this.
The kludge is to treat the data fork as one file, and the resource fork plus metadata as another file. I vaguely recall that this means it does something like :

1) copy/sync data fork as one file
2) copy/sync resource fork as another file - put the bits together at the destination

From memory (it's a while since I last used rsync for doing backups), in step 1 the files don't match because the destination file was modified during step 2 of the last copy - thus the file gets synced again. Again from memory, I think I used to see that every file with a resource fork would be copied in it's entirety every time.

This could be a complete red herring of course, but it's something I've come across in the past/



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync keeps writing files over

Simon Hobson-2
In reply to this post by Perry Hutchison
[hidden email] (Perry Hutchison) wrote:

> Best choice for magtape is probably something like tar, cpio, or pax
> (for a file-oriented backup), or the appropriate variant of dump(8)
> (to back up an entire filesystem -- but not all FS formats have a
> dump/restore suite available).

I wouldn't have thought rsync a good choice either. I've used both tar and cpio in the past - I don't think there's much to choose between them, "personal comfort" with the tool is probably as big a factor as anything. Used them with QIC, DAT, DLT, and SLR over the years.

If you want "ease of use" and aren't averse to paid for/proprietary programs, then on a Mac I think Retrospect has a lot going for it. I've been using it for home and business use for something like 25 years - oh, that makes me feel old now ! It's exceedingly good for incremental/differential backups - I've tried a few others but always managed to find situations where they would miss copying files (not a good attribute for a backup system), an area Dantz has some patents on.


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync keeps writing files over

McDowell, Blake
In reply to this post by Simon Hobson-2
Thanks for that info Simon.

The files system is Journaled HFS+ and I¹m running rsync  version 3.1.2
protocol version 31.

I run rsync exclusively through the CLI/Terminal, so I¹m not sure what the
version that comes with OSX is, but it is not the GUI version.


On 6/3/16, 5:17 AM, "rsync on behalf of Simon Hobson"
<[hidden email] on behalf of [hidden email]> wrote:

>"McDowell, Blake" <[hidden email]> wrote:
>
>> The storage is just an regular HDD in a mac pro tower.
>
>Ah, is this the version of rsync that comes with OS X ? Are these HFS+
>filesystems ?
>
>I vaguely recall that the OS X version is "hacked" to handle the file
>semantics of HFS+ filesystems. Hopefully someone else actually knows the
>details, I could be a bit wrong here, but IIRC it's something like :
>On a "*nix" filsystem, a file is a file - a chunk of data and some
>filesystem metadata. On HFS+, a file is comprised of up to 3 parts - the
>data fork, the resource fork (I don't believe this is widely used these
>days), and a chunk more metadata. "Regular" rsync only copies the data
>fork and that part of the metadata that maps to *nix filesystem
>semantics, the OS X version of rsync copies the whole file by way of
>quite a kludge - can't remember if it needs an extra cmd line switch to
>do this.
>The kludge is to treat the data fork as one file, and the resource fork
>plus metadata as another file. I vaguely recall that this means it does
>something like :
>
>1) copy/sync data fork as one file
>2) copy/sync resource fork as another file - put the bits together at the
>destination
>
>From memory (it's a while since I last used rsync for doing backups), in
>step 1 the files don't match because the destination file was modified
>during step 2 of the last copy - thus the file gets synced again. Again
>from memory, I think I used to see that every file with a resource fork
>would be copied in it's entirety every time.
>
>This could be a complete red herring of course, but it's something I've
>come across in the past/
>
>
>
>--
>Please use reply-all for most replies to avoid omitting the mailing list.
>To unsubscribe or change options:
>https://lists.samba.org/mailman/listinfo/rsync
>Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync keeps writing files over

McDowell, Blake
In reply to this post by Perry Hutchison
Hi Perry,

The issue I¹m having is not writing to the LTO tape - rsync does that ok,
but we have just switch over to gcp followed by rsync as a double-check.
It has certainly helped with our LTO writing.

The problem I¹m having with the files continually re-writing is for a
local transfer from external HDD to internal HDD, both HFS+

Thanks,
Blake


On 6/3/16, 4:14 AM, "Perry Hutchison" <[hidden email]> wrote:

>"McDowell, Blake" <[hidden email]> wrote:
>
>> The storage is just an regular HDD in a mac pro tower. I can't
>> imagine why it wouldn't handle timestamps. Also of note - this
>> problem doesn't exist for every file, just the vast majority.
>> So, that just makes it more confusing.
>
>The filesystem format (MacOS native?  FAT?) might be a factor.
>
>> Yes, I imagine rsync is not the best for linear tape but give the
>> choice between cp (which is faster and causes less problems but
>> offers almost zero verbosity) and rsync, I???ll choose rsync. If
>> people know of other options, I???d be very happy to know of them.
>
>Best choice for magtape is probably something like tar, cpio, or pax
>(for a file-oriented backup), or the appropriate variant of dump(8)
>(to back up an entire filesystem -- but not all FS formats have a
>dump/restore suite available).


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync keeps writing files over

McDowell, Blake
In reply to this post by Steven Levine
Hi Steven,

Yes, both file systems are the same.

rsync -nri --modify-window=1 <src> <dest>

Gives me the following for most files >f..T.......
2015_167_1_1__Boy_What_A_Girl_R2/2015_167_1_1__Boy_What_A_Girl__UHD_DPX_R2/
BWAG_R2_00138428.dpx
Although a few have  >f..T......n
2015_167_1_1__Boy_What_A_Girl_R2/2015_167_1_1__Boy_What_A_Girl__UHD_DPX_R2/
BWAG_R2_00135909.dpx

Strangely, no matter what HDD I¹m transferring from it is always the same
sequence of files that have the n - "A n means the create time (newness)
is different and is being updated to the sender's value (requires
--crtimes).
(I¹m not quite sure I completely understand ‹-modify-window)

Here is a <dest> file example of timestamps as rsync interprets them:
-rwxrwxrwx     24,839,552 2016/06/08 13:13:19 BWAG_R2_00086400.dpx

Here is a <src> file example of timestamps as rsync interprets them:
-rwxrwxrwx     24,839,552 2016/05/27 13:43:32 BWAG_R2_00086400.dpx



So, yeah, that is the problem, apparently. The timestamps are
transferring.

But, for the files that have the ³n² the timestamps appear to be fine and
those ones don¹t continually transfer. A mystery to me as to why the
timestamps work on these particular files.

<src>
-rwxrwxrwx     24,839,552 2016/05/27 14:00:57 BWAG_R2_00130798.dpx
<dest>


-rwxrwxrwx     24,839,552 2016/05/27 14:00:57 BWAG_R2_00130798.dpx

I¹m running version 3.1.2 protocol 31.


Thanks,
Blake


On 6/2/16, 7:26 PM, "rsync on behalf of Steven Levine"
<[hidden email] on behalf of [hidden email]> wrote:

>In <D3762D63.17A7%[hidden email]>, on 06/02/16
>   at 10:42 PM, "McDowell, Blake" <[hidden email]> said:
>
>Hi Blake,
>
>>The storage is just an regular HDD in a mac pro tower. I can t imagine
>>why it wouldn t handle timestamps. Also of note - this problem doesn t
>>exist for every file, just the vast majority. So, that just makes it more
>>confusing.
>
>Are the file systems the same on the source and the destination
>partitions?
>
>Check out
>
>  --modify-window
>
>in the help.  If the source and destination file systems have different
>timestamp precision, this is the usual solution.
>
>You can also try
>
>  rsync /path-to-foo
>
>where path-to-foo is a directory or file.  This will list the file
>timestamps as rsync interprets them.
>
>BTW, what version of rsync are you running?  It might matter.
>
>Steven
>
>--
>----------------------------------------------------------------------
>"Steven Levine" <[hidden email]>  Warp/DIY/BlueLion etc.
>www.scoug.com www.arcanoae.com www.warpcave.com
>----------------------------------------------------------------------
>
>
>--
>Please use reply-all for most replies to avoid omitting the mailing list.
>To unsubscribe or change options:
>https://lists.samba.org/mailman/listinfo/rsync
>Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync keeps writing files over

McDowell, Blake
In reply to this post by Kevin Korb
Hi Kevin,

Stat is pretty cool! But, I can’t quite figure out some of the stuff it is
telling me right now.

I’ll try using —update but I’m hoping to figure out why the timestamps are
not transferring on the majority of my files.

Thanks,
Blake

On 6/2/16, 6:58 PM, "rsync on behalf of Kevin Korb"
<[hidden email] on behalf of [hidden email]> wrote:

>Rsync only cares about the modification time.  The ls command usually
>abbreviates the timestamp so it is better to use the stat command on one
>of the problem files to see the full thing.
>
>On 06/02/2016 06:42 PM, McDowell, Blake wrote:
>> Cool Thanks!
>>
>> Specifically, the timestamps on both <src> and <dest> match for "ls -l"
>> but do not match for "ls -lu" or "ls -lc”
>>
>> The storage is just an regular HDD in a mac pro tower. I can’t imagine
>>why
>> it wouldn’t handle timestamps. Also of note - this problem doesn’t exist
>> for every file, just the vast majority. So, that just makes it more
>> confusing.
>>
>> Yes, I imagine rsync is not the best for linear tape but give the choice
>> between cp (which is faster and causes less problems but offers almost
>> zero verbosity) and rsync, I’ll choose rsync. If people know of other
>> options, I’d be very happy to know of them.
>>
>> On 6/2/16, 6:33 PM, "Kevin Korb" <[hidden email]> wrote:
>>
>>> The man page has a section on what all the itemize-changes flags do.
>>>
>>> There is a --ignore-times but the result is what you have now, re-copy
>>> everything even if the timestamp matches.
>>>
>>> The best you can really do with storage that can't handle timestamps is
>>> to use --update.  But if you do that you need to get rid of --partial
>>> (part of -P) or else rsync will never complete a file that was
>>>partially
>>> copied since the partial copy is newer that the source file.
>>>
>>> OTOH, are you using rsync to copy files to a tape drive?  If so then I
>>> would think rsync is not the right tool for dealing with linear
>>>storage.
>>>
>>> On 06/02/2016 06:25 PM, McDowell, Blake wrote:
>>>> OK. Thanks. Where can I find information regarding how to interpret
>>>> —itemize-changes?
>>>>
>>>> The timestamps aren’t changing, so the target must not be storing
>>>>them,
>>>> which I have no idea why. The directory I’m writing to is 777.
>>>>
>>>> What is the flag to tell rsync to ignore the timestamps?
>>>>
>>>> Thanks,
>>>> Blake
>>>>
>>>> On 6/2/16, 6:18 PM, "rsync on behalf of Kevin Korb"
>>>> <[hidden email] on behalf of [hidden email]> wrote:
>>>>
>>>>> It is saying the timestamp is wrong and that it is copying the file
>>>>>and
>>>>> changing the timestamp.  If it does that every time then either the
>>>>> timestamps are changing on the source or the target isn't storing
>>>>>them.
>>>>>
>>>>> On 06/02/2016 06:13 PM, McDowell, Blake wrote:
>>>>>> Thanks Kevin! I¹m unclear how to read the ‹itemize-changes output.
>>>>>>Can
>>>>>> you
>>>>>> provide some insight?
>>>>>>
>>>>>> This is a local transfer from an external drive to an internal drive
>>>>>> all
>>>>>> attached to one computer.
>>>>>>
>>>>>>
>>>>>> rsync -aPh --itemize-changes -n
>>>>>> /Volumes/shuttle_05/2012_79_1_14_1__1199_Workprint
>>>>>> /Volumes/3TB_LTO/LT003A/
>>>>>> sending incremental file list
>>>>>>> f..t.......
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Deriv
>>>>>>>at
>>>>>>> iv
>>>>>>> es
>>>>>>> /2012_79_1_14_1_DER_01.mov
>>>>>>> f..t.......
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Deriv
>>>>>>>at
>>>>>>> iv
>>>>>>> es
>>>>>>> /2012_79_1_14_1_DER_02.mov
>>>>>>> f..t.......
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_D
>>>>>>>PX
>>>>>>> /1
>>>>>>> 19
>>>>>>> 9WP_00086400.dpx
>>>>>>> f..t.......
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_D
>>>>>>>PX
>>>>>>> /1
>>>>>>> 19
>>>>>>> 9WP_00086401.dpx
>>>>>>> f..t.......
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_D
>>>>>>>PX
>>>>>>> /1
>>>>>>> 19
>>>>>>> 9WP_00086402.dpx
>>>>>>> f..t.......
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_D
>>>>>>>PX
>>>>>>> /1
>>>>>>> 19
>>>>>>> 9WP_00086403.dpx
>>>>>>> f..t.......
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_D
>>>>>>>PX
>>>>>>> /1
>>>>>>> 19
>>>>>>> 9WP_00086404.dpx
>>>>>>> f..t.......
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_D
>>>>>>>PX
>>>>>>> /1
>>>>>>> 19
>>>>>>> 9WP_00086405.dpx
>>>>>>> f..t.......
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_D
>>>>>>>PX
>>>>>>> /1
>>>>>>> 19
>>>>>>> 9WP_00086406.dpx
>>>>>>
>>>>>>
>>>>>> ~Blake
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 6/2/16, 5:44 PM, "rsync on behalf of Kevin Korb"
>>>>>> <[hidden email] on behalf of [hidden email]>
>>>>>>wrote:
>>>>>>
>>>>>>> Instead of the second -v (or even the first) add --itemize-changes.
>>>>>>> It
>>>>>>> will tell you why each file is being copied.  If the file
>>>>>>>timestamps
>>>>>>> are
>>>>>>> not correct then perhaps the underlying storage doesn't allow
>>>>>>> arbitrary
>>>>>>> mtime changes.
>>>>>>>
>>>>>>> On 06/02/2016 05:23 PM, McDowell, Blake wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>>  
>>>>>>>>
>>>>>>>> At my work we use rsync to move files between drives and to LTO
>>>>>>>> among
>>>>>>>> other things.
>>>>>>>>
>>>>>>>>  
>>>>>>>>
>>>>>>>> I¹m having an issue using rsync to move material between and
>>>>>>>> external
>>>>>>>> drive and an internal drive.
>>>>>>>>
>>>>>>>>  
>>>>>>>>
>>>>>>>> We run ³rsync ­avvPh <src> <dest>² and most of the files keep
>>>>>>>> writing
>>>>>>>> every time I run this. It appears that the modification times are
>>>>>>>> not
>>>>>>>> being carried through to the destination resulting in the files
>>>>>>>> continually wanting to re-write.
>>>>>>>>
>>>>>>>>  
>>>>>>>>
>>>>>>>> I¹m hoping someone can help me figure this out. Any information
>>>>>>>>you
>>>>>>>> need
>>>>>>>> from me (logs, etc) I¹m happy to try and provide.
>>>>>>>>
>>>>>>>>  
>>>>>>>>
>>>>>>>> Many thanks,
>>>>>>>>
>>>>>>>> Blake  
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,
>>>>>>>._
>>>>>>> .,
>>>>>>> Kevin Korb Phone:    (407) 252-6853
>>>>>>> Systems Administrator Internet:
>>>>>>> FutureQuest, Inc. [hidden email]  (work)
>>>>>>> Orlando, Florida [hidden email] (personal)
>>>>>>> Web page: http://www.sanitarium.net/
>>>>>>> PGP public key available on web site.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,
>>>>>>>._
>>>>>>> .,
>>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>>
>>>>>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._
>>>>>.,
>>>>> Kevin Korb Phone:    (407) 252-6853
>>>>> Systems Administrator Internet:
>>>>> FutureQuest, Inc. [hidden email]  (work)
>>>>> Orlando, Florida [hidden email] (personal)
>>>>> Web page: http://www.sanitarium.net/
>>>>> PGP public key available on web site.
>>>>>
>>>>>
>>>>>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._
>>>>>.,
>>>>>
>>>>
>>>
>>> --
>>>
>>>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>>> Kevin Korb Phone:    (407) 252-6853
>>> Systems Administrator Internet:
>>> FutureQuest, Inc. [hidden email]  (work)
>>> Orlando, Florida [hidden email] (personal)
>>> Web page: http://www.sanitarium.net/
>>> PGP public key available on web site.
>>>
>>>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>>>
>>
>
>--
>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> Kevin Korb Phone:    (407) 252-6853
> Systems Administrator Internet:
> FutureQuest, Inc. [hidden email]  (work)
> Orlando, Florida [hidden email] (personal)
> Web page: http://www.sanitarium.net/
> PGP public key available on web site.
>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync keeps writing files over

Kevin Korb
In reply to this post by McDowell, Blake
the T means that the timestamp is wrong and rsync is not fixing it
because you don't have --times or --archive in your command line.

On 06/08/2016 08:17 PM, McDowell, Blake wrote:

> Hi Steven,
>
> Yes, both file systems are the same.
>
> rsync -nri --modify-window=1 <src> <dest>
>
> Gives me the following for most files >f..T.......
> 2015_167_1_1__Boy_What_A_Girl_R2/2015_167_1_1__Boy_What_A_Girl__UHD_DPX_R2/
> BWAG_R2_00138428.dpx
> Although a few have  >f..T......n
> 2015_167_1_1__Boy_What_A_Girl_R2/2015_167_1_1__Boy_What_A_Girl__UHD_DPX_R2/
> BWAG_R2_00135909.dpx
>
> Strangely, no matter what HDD I¹m transferring from it is always the same
> sequence of files that have the n - "A n means the create time (newness)
> is different and is being updated to the sender's value (requires
> --crtimes).
> (I¹m not quite sure I completely understand ‹-modify-window)
>
> Here is a <dest> file example of timestamps as rsync interprets them:
> -rwxrwxrwx     24,839,552 2016/06/08 13:13:19 BWAG_R2_00086400.dpx
>
> Here is a <src> file example of timestamps as rsync interprets them:
> -rwxrwxrwx     24,839,552 2016/05/27 13:43:32 BWAG_R2_00086400.dpx
>
>
>
> So, yeah, that is the problem, apparently. The timestamps are
> transferring.
>
> But, for the files that have the ³n² the timestamps appear to be fine and
> those ones don¹t continually transfer. A mystery to me as to why the
> timestamps work on these particular files.
>
> <src>
> -rwxrwxrwx     24,839,552 2016/05/27 14:00:57 BWAG_R2_00130798.dpx
> <dest>
>
>
> -rwxrwxrwx     24,839,552 2016/05/27 14:00:57 BWAG_R2_00130798.dpx
>
> I¹m running version 3.1.2 protocol 31.
>
>
> Thanks,
> Blake
>
>
> On 6/2/16, 7:26 PM, "rsync on behalf of Steven Levine"
> <[hidden email] on behalf of [hidden email]> wrote:
>
>> In <D3762D63.17A7%[hidden email]>, on 06/02/16
>>   at 10:42 PM, "McDowell, Blake" <[hidden email]> said:
>>
>> Hi Blake,
>>
>>> The storage is just an regular HDD in a mac pro tower. I can t imagine
>>> why it wouldn t handle timestamps. Also of note - this problem doesn t
>>> exist for every file, just the vast majority. So, that just makes it more
>>> confusing.
>>
>> Are the file systems the same on the source and the destination
>> partitions?
>>
>> Check out
>>
>>  --modify-window
>>
>> in the help.  If the source and destination file systems have different
>> timestamp precision, this is the usual solution.
>>
>> You can also try
>>
>>  rsync /path-to-foo
>>
>> where path-to-foo is a directory or file.  This will list the file
>> timestamps as rsync interprets them.
>>
>> BTW, what version of rsync are you running?  It might matter.
>>
>> Steven
>>
>> --
>> ----------------------------------------------------------------------
>> "Steven Levine" <[hidden email]>  Warp/DIY/BlueLion etc.
>> www.scoug.com www.arcanoae.com www.warpcave.com
>> ----------------------------------------------------------------------
>>
>>
>> --
>> Please use reply-all for most replies to avoid omitting the mailing list.
>> To unsubscribe or change options:
>> https://lists.samba.org/mailman/listinfo/rsync
>> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
>
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
        Kevin Korb Phone:    (407) 252-6853
        Systems Administrator Internet:
        FutureQuest, Inc. [hidden email]  (work)
        Orlando, Florida [hidden email] (personal)
        Web page: http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

signature.asc (188 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: rsync keeps writing files over

McDowell, Blake
I ran the original transfer with the -a flag. It doesn't fix the problem.

-----Original Message-----
From: rsync [mailto:[hidden email]] On Behalf Of Kevin Korb
Sent: Wednesday, June 08, 2016 8:26 PM
To: [hidden email]
Subject: Re: rsync keeps writing files over

the T means that the timestamp is wrong and rsync is not fixing it because you don't have --times or --archive in your command line.

On 06/08/2016 08:17 PM, McDowell, Blake wrote:

> Hi Steven,
>
> Yes, both file systems are the same.
>
> rsync -nri --modify-window=1 <src> <dest>
>
> Gives me the following for most files >f..T.......
> 2015_167_1_1__Boy_What_A_Girl_R2/2015_167_1_1__Boy_What_A_Girl__UHD_DP
> X_R2/
> BWAG_R2_00138428.dpx
> Although a few have  >f..T......n
> 2015_167_1_1__Boy_What_A_Girl_R2/2015_167_1_1__Boy_What_A_Girl__UHD_DP
> X_R2/
> BWAG_R2_00135909.dpx
>
> Strangely, no matter what HDD I¹m transferring from it is always the
> same sequence of files that have the n - "A n means the create time
> (newness) is different and is being updated to the sender's value
> (requires --crtimes).
> (I¹m not quite sure I completely understand ‹-modify-window)
>
> Here is a <dest> file example of timestamps as rsync interprets them:
> -rwxrwxrwx     24,839,552 2016/06/08 13:13:19 BWAG_R2_00086400.dpx
>
> Here is a <src> file example of timestamps as rsync interprets them:
> -rwxrwxrwx     24,839,552 2016/05/27 13:43:32 BWAG_R2_00086400.dpx
>
>
>
> So, yeah, that is the problem, apparently. The timestamps are
> transferring.
>
> But, for the files that have the ³n² the timestamps appear to be fine
> and those ones don¹t continually transfer. A mystery to me as to why
> the timestamps work on these particular files.
>
> <src>
> -rwxrwxrwx     24,839,552 2016/05/27 14:00:57 BWAG_R2_00130798.dpx
> <dest>
>
>
> -rwxrwxrwx     24,839,552 2016/05/27 14:00:57 BWAG_R2_00130798.dpx
>
> I¹m running version 3.1.2 protocol 31.
>
>
> Thanks,
> Blake
>
>
> On 6/2/16, 7:26 PM, "rsync on behalf of Steven Levine"
> <[hidden email] on behalf of [hidden email]> wrote:
>
>> In <D3762D63.17A7%[hidden email]>, on 06/02/16
>>   at 10:42 PM, "McDowell, Blake" <[hidden email]> said:
>>
>> Hi Blake,
>>
>>> The storage is just an regular HDD in a mac pro tower. I can t
>>> imagine why it wouldn t handle timestamps. Also of note - this
>>> problem doesn t exist for every file, just the vast majority. So,
>>> that just makes it more confusing.
>>
>> Are the file systems the same on the source and the destination
>> partitions?
>>
>> Check out
>>
>>  --modify-window
>>
>> in the help.  If the source and destination file systems have
>> different timestamp precision, this is the usual solution.
>>
>> You can also try
>>
>>  rsync /path-to-foo
>>
>> where path-to-foo is a directory or file.  This will list the file
>> timestamps as rsync interprets them.
>>
>> BTW, what version of rsync are you running?  It might matter.
>>
>> Steven
>>
>> --
>> ---------------------------------------------------------------------
>> - "Steven Levine" <[hidden email]>  Warp/DIY/BlueLion etc.
>> www.scoug.com www.arcanoae.com www.warpcave.com
>> ---------------------------------------------------------------------
>> -
>>
>>
>> --
>> Please use reply-all for most replies to avoid omitting the mailing list.
>> To unsubscribe or change options:
>> https://lists.samba.org/mailman/listinfo/rsync
>> Before posting, read:
>> http://www.catb.org/~esr/faqs/smart-questions.html
>
>

--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
        Kevin Korb Phone:    (407) 252-6853
        Systems Administrator Internet:
        FutureQuest, Inc. [hidden email]  (work)
        Orlando, Florida [hidden email] (personal)
        Web page: http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync keeps writing files over

Steven Levine
In reply to this post by McDowell, Blake
In <D37E24FA.120D%[hidden email]>, on 06/09/16
   at 12:17 AM, "McDowell, Blake" <[hidden email]> said:

Hi Blake,

Please reply to the list.

>rsync -nri --modify-window=1 <src> <dest>

As others mentioned, you need need to use --times.  This is needed so that
we can see use output from --itemize-changes.

>Gives me the following for most files >f..T.......
>2015_167_1_1__Boy_What_A_Girl_R2/2015_167_1_1__Boy_What_A_Girl__UHD_DPX_R2/
>BWAG_R2_00138428.dpx
>Although a few have  >f..T......n
>2015_167_1_1__Boy_What_A_Girl_R2/2015_167_1_1__Boy_What_A_Girl__UHD_DPX_R2/
>BWAG_R2_00135909.dpx

At a certain level this makes sense.  Without --times, the timestamps are
not used to determine whether or not a file needs to be transferred and,
in addition, the receiver will set the timestamp to the current time on
the receiver.

This is what the docs mean when they say

"Note that if this option is not used, the optimization that excludes
files that have not been modified  cannot  be  effective;"

>(I¹m not quite sure I completely understand  -modify-window)

You should not need --modify-window.  Modify window is needed when the
source and destination file systems have differing timestamp resolutions.
For example if transferring to a Window's filesystem that had 2-second
timestamp resolution from a *ix system with 1-second resolution, you would
need to use --modify-window=2 to avoid spurious transfers.

>Here is a <dest> file example of timestamps as rsync interprets them:
>-rwxrwxrwx     24,839,552 2016/06/08 13:13:19 BWAG_R2_00086400.dpx

>Here is a <src> file example of timestamps as rsync interprets them:
>-rwxrwxrwx     24,839,552 2016/05/27 13:43:32 BWAG_R2_00086400.dpx

Without --times, this is the expected behavior.  The timestamps differ, so
rsync will transfer the file.  Because the file content is the same, the
transfer will be quick, but a tranfer will happen.

If you cannot use --times, you many need to use some combination of
--update and --ignore-times an possibly --size-only to avoid selecting
these files for transfer.  Exactly which options will be approriate will
depend on the content of your files and how the content changes.

FWIW, if --times cannot set the timestamp correctly on the receiver, I
would suspect an issue with the filesystem or your rsync build.  Rsync
uses the standard platform APIs for setting the timestamps so this should
just work.

>But, for the files that have the ³n²

What "n" do you mean?

Another FWIW, when testing, --dry-run (i.e. -n) is useful to understand
which files will be transferred, especially when working with complex
filters, but you need to run without -n to see that true results of the
transfer.

Steven

--
----------------------------------------------------------------------
"Steven Levine" <[hidden email]>  Warp/DIY/BlueLion etc.
www.scoug.com www.arcanoae.com www.warpcave.com
----------------------------------------------------------------------


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync keeps writing files over

McDowell, Blake
Hi Steve,

I run my first transfer always as rsync -avvPh. I now add in -i based on
what I’ve learned here.

My understanding from the rsync man page is that -a or ―archive implicitly
includes -t or ―times. Am I mistaken there? Do I need to run ―times
separately?


The output of my first transfer is:
>f++++++++++ 2012_79_1_79_1__New_Ark__Derivatives/2012_79_1_79_1_DER_01.mov
         12.09G 100%  132.21MB/s    0:01:27 (xfr#4, to-chk=13489/13497)

When I immediately re-run the same rsync command in dry-run mode I get:
>f..t....... 2012_79_1_79_1__New_Ark__Derivatives/2012_79_1_79_1_DER_01.mov

I am running about 13,000 files on this transfer and that is what is case
for every file except for a very small amount that output:
.f           2012_79_1_79_1__New_Ark__UHD_DPX/NewArk_00086558.dpx


It is a mystery to me as to why these particular files are transferring
their timestamps correctly.

Re-running the rysnc command without dry run does not make only the
timestamps transfer; the whole file retransfers and it takes the same
length of time to transfer as the first time. This is continually the
case, the timestamps never update.

I can’t imagine what would be wrong with my rsync build - I pull it
straight from homebrew/github

/Users/medialab
?:? brew install rsync
Warning: homebrew/dupes/rsync-3.1.2 already installed

Maybe the file system… Any suggested troubleshooting for that or rsync?

Thanks,
Blake
 





 

On 6/10/16, 11:43 PM, "rsync on behalf of Steven Levine"
<[hidden email] on behalf of [hidden email]> wrote:

>In <D37E24FA.120D%[hidden email]>, on 06/09/16
>   at 12:17 AM, "McDowell, Blake" <[hidden email]> said:
>
>Hi Blake,
>
>Please reply to the list.
>
>>rsync -nri --modify-window=1 <src> <dest>
>
>As others mentioned, you need need to use --times.  This is needed so that
>we can see use output from --itemize-changes.
>
>>Gives me the following for most files >f..T.......
>>2015_167_1_1__Boy_What_A_Girl_R2/2015_167_1_1__Boy_What_A_Girl__UHD_DPX_R
>>2/
>>BWAG_R2_00138428.dpx
>>Although a few have  >f..T......n
>>2015_167_1_1__Boy_What_A_Girl_R2/2015_167_1_1__Boy_What_A_Girl__UHD_DPX_R
>>2/
>>BWAG_R2_00135909.dpx
>
>At a certain level this makes sense.  Without --times, the timestamps are
>not used to determine whether or not a file needs to be transferred and,
>in addition, the receiver will set the timestamp to the current time on
>the receiver.
>
>This is what the docs mean when they say
>
>"Note that if this option is not used, the optimization that excludes
>files that have not been modified  cannot  be  effective;"
>
>>(I¹m not quite sure I completely understand  -modify-window)
>
>You should not need --modify-window.  Modify window is needed when the
>source and destination file systems have differing timestamp resolutions.
>For example if transferring to a Window's filesystem that had 2-second
>timestamp resolution from a *ix system with 1-second resolution, you would
>need to use --modify-window=2 to avoid spurious transfers.
>
>>Here is a <dest> file example of timestamps as rsync interprets them:
>>-rwxrwxrwx     24,839,552 2016/06/08 13:13:19 BWAG_R2_00086400.dpx
>
>>Here is a <src> file example of timestamps as rsync interprets them:
>>-rwxrwxrwx     24,839,552 2016/05/27 13:43:32 BWAG_R2_00086400.dpx
>
>Without --times, this is the expected behavior.  The timestamps differ, so
>rsync will transfer the file.  Because the file content is the same, the
>transfer will be quick, but a tranfer will happen.
>
>If you cannot use --times, you many need to use some combination of
>--update and --ignore-times an possibly --size-only to avoid selecting
>these files for transfer.  Exactly which options will be approriate will
>depend on the content of your files and how the content changes.
>
>FWIW, if --times cannot set the timestamp correctly on the receiver, I
>would suspect an issue with the filesystem or your rsync build.  Rsync
>uses the standard platform APIs for setting the timestamps so this should
>just work.
>
>>But, for the files that have the ³n²
>
>What "n" do you mean?
>
>Another FWIW, when testing, --dry-run (i.e. -n) is useful to understand
>which files will be transferred, especially when working with complex
>filters, but you need to run without -n to see that true results of the
>transfer.
>
>Steven
>
>--
>----------------------------------------------------------------------
>"Steven Levine" <[hidden email]>  Warp/DIY/BlueLion etc.
>www.scoug.com www.arcanoae.com www.warpcave.com
>----------------------------------------------------------------------
>
>
>--
>Please use reply-all for most replies to avoid omitting the mailing list.
>To unsubscribe or change options:
>https://lists.samba.org/mailman/listinfo/rsync
>Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html