-H option causes unneccessary transfers

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

-H option causes unneccessary transfers

Albert Berger
Greetings,

when the daily system backup with rsync is performed, thousands of files (e.g. from /usr directory),
which are not modified are uselessly transferrred to the backup storage every day.
The rsync command is the following:

rsync --update -DHAErlptgo --relative --atimes --delete-during ${SRCPATH} ${BACKUPDIR}

Quick investigation shown, that these files are hard links, and the command option related to this
behaviour is -H: without it the unmodified files are not copied. Seemingly this situation is described
in the rsync manual:

"If  incremental recursion is active (see --recursive), rsync may transfer a missing
hard-linked file before it finds that another link for that contents  exists  else‐
where  in  the  hierarchy.... One way to avoid this inefficiency  is  to  disable  
incremental  recursion  using the --no-inc-recursive option."

Adding --no-inc-recursive to the aforementioned command line doesn't help (are --recursive and
--no-inc-recursive supposed to be used in one option set?). And I use --recursive to copy entire
directory tree, including subdirectories. Could someone advise please how one can prevent copying
unmodified hard-linked files?

Thanks,
A. Berger.

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: -H option causes unneccessary transfers

Kevin Korb
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

What does --itemize-changes say about these files?

Also, what is --atimes?  Is this the Apple modified rsync for OSX?

On 03/28/2016 07:51 AM, Albert Berger wrote:

> Greetings,
>
> when the daily system backup with rsync is performed, thousands of
> files (e.g. from /usr directory), which are not modified are
> uselessly transferrred to the backup storage every day. The rsync
> command is the following:
>
> rsync --update -DHAErlptgo --relative --atimes --delete-during
> ${SRCPATH} ${BACKUPDIR}
>
> Quick investigation shown, that these files are hard links, and the
> command option related to this behaviour is -H: without it the
> unmodified files are not copied. Seemingly this situation is
> described in the rsync manual:
>
> "If  incremental recursion is active (see --recursive), rsync may
> transfer a missing hard-linked file before it finds that another
> link for that contents  exists  else‐ where  in  the  hierarchy....
> One way to avoid this inefficiency  is  to  disable incremental
> recursion  using the --no-inc-recursive option."
>
> Adding --no-inc-recursive to the aforementioned command line
> doesn't help (are --recursive and --no-inc-recursive supposed to be
> used in one option set?). And I use --recursive to copy entire
> directory tree, including subdirectories. Could someone advise
> please how one can prevent copying unmodified hard-linked files?
>
> Thanks, A. Berger.
>

- --
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
        Kevin Korb Phone:    (407) 252-6853
        Systems Administrator Internet:
        FutureQuest, Inc. [hidden email]  (work)
        Orlando, Florida [hidden email] (personal)
        Web page: http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlb5H1sACgkQVKC1jlbQAQe0LwCg0yhgY81WywKO4NYioxCfbF3J
qVcAn2P9hJiW9pXwMZutAmuOjIvdvvNZ
=/Enu
-----END PGP SIGNATURE-----

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: -H option causes unneccessary transfers

Albert Berger
On Mon, Mar 28, 2016 at 08:11:07AM -0400, Kevin Korb wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> What does --itemize-changes say about these files?
>
> Also, what is --atimes?  Is this the Apple modified rsync for OSX?
>
> On 03/28/2016 07:51 AM, Albert Berger wrote:
> > Greetings,
> >
> > when the daily system backup with rsync is performed, thousands of
> > files (e.g. from /usr directory), which are not modified are
> > uselessly transferrred to the backup storage every day. The rsync
> > command is the following:
> >
> > rsync --update -DHAErlptgo --relative --atimes --delete-during
> > ${SRCPATH} ${BACKUPDIR}
> >
> > Quick investigation shown, that these files are hard links, and the
> > command option related to this behaviour is -H: without it the
> > unmodified files are not copied. Seemingly this situation is
> > described in the rsync manual:
> >
> > "If  incremental recursion is active (see --recursive), rsync may
> > transfer a missing hard-linked file before it finds that another
> > link for that contents  exists  else‐ where  in  the  hierarchy....
> > One way to avoid this inefficiency  is  to  disable incremental
> > recursion  using the --no-inc-recursive option."
> >
> > Adding --no-inc-recursive to the aforementioned command line
> > doesn't help (are --recursive and --no-inc-recursive supposed to be
> > used in one option set?). And I use --recursive to copy entire
> > directory tree, including subdirectories. Could someone advise
> > please how one can prevent copying unmodified hard-linked files?
> >
> > Thanks, A. Berger.
> >

As an example of copying hard-linked files, the /usr/bin contains the following
two hard-linked sets for two inodes:

inode 1:
ld
ld.bfd

inode 2:
g++
i686-pc-linux-gnu-c++
i686-pc-linux-gnu-g++
c++

Below is the --itemized-changes output for those two sets which is produced when
/usr/bin doesn't have any modifications and is synced with the aforementioned command
to its backup copy:

hf......u.. /usr/bin/ld.bfd
hf......u.. /usr/bin/i686-pc-linux-gnu-c++
hf......u.. /usr/bin/i686-pc-linux-gnu-g++
hf......u.. /usr/bin/g++

As can bee seen, one file in each set is not listed at all, and all other hard-linked
files are marked for updating. Is this behaviour normal?

--atimes is an option for preserving access times. This option is added by patching
rsync source with a patch from rsync-patches-3.1.1.tar.gz file from rsync download page.

The system is ArchLinux, kernel 4.4.5.


Regards,
A.Berger.

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: -H option causes unneccessary transfers

Kevin Korb
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

My guess is that it is the atimes patch causing the problem.  Normally
rsync would never say "u" in the itemized output as it is "reserved
for future use".  Plus copying the file would change the atime.

On 03/28/2016 08:44 AM, Albert Berger wrote:

> On Mon, Mar 28, 2016 at 08:11:07AM -0400, Kevin Korb wrote:
>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
>>
>> What does --itemize-changes say about these files?
>>
>> Also, what is --atimes?  Is this the Apple modified rsync for
>> OSX?
>>
>> On 03/28/2016 07:51 AM, Albert Berger wrote:
>>> Greetings,
>>>
>>> when the daily system backup with rsync is performed, thousands
>>> of files (e.g. from /usr directory), which are not modified
>>> are uselessly transferrred to the backup storage every day. The
>>> rsync command is the following:
>>>
>>> rsync --update -DHAErlptgo --relative --atimes --delete-during
>>> ${SRCPATH} ${BACKUPDIR}
>>>
>>> Quick investigation shown, that these files are hard links, and
>>> the command option related to this behaviour is -H: without it
>>> the unmodified files are not copied. Seemingly this situation
>>> is described in the rsync manual:
>>>
>>> "If  incremental recursion is active (see --recursive), rsync
>>> may transfer a missing hard-linked file before it finds that
>>> another link for that contents  exists  else‐ where  in  the
>>> hierarchy.... One way to avoid this inefficiency  is  to
>>> disable incremental recursion  using the --no-inc-recursive
>>> option."
>>>
>>> Adding --no-inc-recursive to the aforementioned command line
>>> doesn't help (are --recursive and --no-inc-recursive supposed
>>> to be used in one option set?). And I use --recursive to copy
>>> entire directory tree, including subdirectories. Could someone
>>> advise please how one can prevent copying unmodified
>>> hard-linked files?
>>>
>>> Thanks, A. Berger.
>>>
>
> As an example of copying hard-linked files, the /usr/bin contains
> the following two hard-linked sets for two inodes:
>
> inode 1: ld ld.bfd
>
> inode 2: g++ i686-pc-linux-gnu-c++ i686-pc-linux-gnu-g++ c++
>
> Below is the --itemized-changes output for those two sets which is
> produced when /usr/bin doesn't have any modifications and is synced
> with the aforementioned command to its backup copy:
>
> hf......u.. /usr/bin/ld.bfd hf......u..
> /usr/bin/i686-pc-linux-gnu-c++ hf......u..
> /usr/bin/i686-pc-linux-gnu-g++ hf......u.. /usr/bin/g++
>
> As can bee seen, one file in each set is not listed at all, and all
> other hard-linked files are marked for updating. Is this behaviour
> normal?
>
> --atimes is an option for preserving access times. This option is
> added by patching rsync source with a patch from
> rsync-patches-3.1.1.tar.gz file from rsync download page.
>
> The system is ArchLinux, kernel 4.4.5.
>
>
> Regards, A.Berger.
>

- --
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
        Kevin Korb Phone:    (407) 252-6853
        Systems Administrator Internet:
        FutureQuest, Inc. [hidden email]  (work)
        Orlando, Florida [hidden email] (personal)
        Web page: http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlb5KEUACgkQVKC1jlbQAQeWQACg2dJZ/rSs7z1UnsLMZ1ZZa98B
4yMAnR8zOH80RpyWUMB/OWecC2sCE/2W
=XQ6i
-----END PGP SIGNATURE-----

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: -H option causes unneccessary transfers

Albert Berger
Thanks!

On Mon, Mar 28, 2016 at 08:49:09AM -0400, Kevin Korb wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> My guess is that it is the atimes patch causing the problem.  Normally
> rsync would never say "u" in the itemized output as it is "reserved
> for future use".  Plus copying the file would change the atime.
>
> On 03/28/2016 08:44 AM, Albert Berger wrote:
> > On Mon, Mar 28, 2016 at 08:11:07AM -0400, Kevin Korb wrote:
> >> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
> >>
> >> What does --itemize-changes say about these files?
> >>
> >> Also, what is --atimes?  Is this the Apple modified rsync for
> >> OSX?
> >>
> >> On 03/28/2016 07:51 AM, Albert Berger wrote:
> >>> Greetings,
> >>>
> >>> when the daily system backup with rsync is performed, thousands
> >>> of files (e.g. from /usr directory), which are not modified
> >>> are uselessly transferrred to the backup storage every day. The
> >>> rsync command is the following:
> >>>
> >>> rsync --update -DHAErlptgo --relative --atimes --delete-during
> >>> ${SRCPATH} ${BACKUPDIR}
> >>>
> >>> Quick investigation shown, that these files are hard links, and
> >>> the command option related to this behaviour is -H: without it
> >>> the unmodified files are not copied. Seemingly this situation
> >>> is described in the rsync manual:
> >>>
> >>> "If  incremental recursion is active (see --recursive), rsync
> >>> may transfer a missing hard-linked file before it finds that
> >>> another link for that contents  exists  else‐ where  in  the
> >>> hierarchy.... One way to avoid this inefficiency  is  to
> >>> disable incremental recursion  using the --no-inc-recursive
> >>> option."
> >>>
> >>> Adding --no-inc-recursive to the aforementioned command line
> >>> doesn't help (are --recursive and --no-inc-recursive supposed
> >>> to be used in one option set?). And I use --recursive to copy
> >>> entire directory tree, including subdirectories. Could someone
> >>> advise please how one can prevent copying unmodified
> >>> hard-linked files?
> >>>
> >>> Thanks, A. Berger.
> >>>
> >
> > As an example of copying hard-linked files, the /usr/bin contains
> > the following two hard-linked sets for two inodes:
> >
> > inode 1: ld ld.bfd
> >
> > inode 2: g++ i686-pc-linux-gnu-c++ i686-pc-linux-gnu-g++ c++
> >
> > Below is the --itemized-changes output for those two sets which is
> > produced when /usr/bin doesn't have any modifications and is synced
> > with the aforementioned command to its backup copy:
> >
> > hf......u.. /usr/bin/ld.bfd hf......u..
> > /usr/bin/i686-pc-linux-gnu-c++ hf......u..
> > /usr/bin/i686-pc-linux-gnu-g++ hf......u.. /usr/bin/g++
> >
> > As can bee seen, one file in each set is not listed at all, and all
> > other hard-linked files are marked for updating. Is this behaviour
> > normal?
> >
> > --atimes is an option for preserving access times. This option is
> > added by patching rsync source with a patch from
> > rsync-patches-3.1.1.tar.gz file from rsync download page.
> >
> > The system is ArchLinux, kernel 4.4.5.
> >
> >
> > Regards, A.Berger.
> >
>
> - --
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> Kevin Korb Phone:    (407) 252-6853
> Systems Administrator Internet:
> FutureQuest, Inc. [hidden email]  (work)
> Orlando, Florida [hidden email] (personal)
> Web page: http://www.sanitarium.net/
> PGP public key available on web site.
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2
>
> iEYEARECAAYFAlb5KEUACgkQVKC1jlbQAQeWQACg2dJZ/rSs7z1UnsLMZ1ZZa98B
> 4yMAnR8zOH80RpyWUMB/OWecC2sCE/2W
> =XQ6i
> -----END PGP SIGNATURE-----
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html