Memory consumption for rsync -axv --delete

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Memory consumption for rsync -axv --delete

John Long
Hi,

I have been using rsync for many years and never had any kind of problem.
Lately I am running out of RAM trying to do an incremental backup to a box
that only has 2G of RAM. The entire directory structure I'm mirroring is
about 200G of files. A minority of subdirectories have many files.

Is there a way to do an incremental backup with --delete option that does
not use as much memory? Is there a way to tell rsync to use a tempfile
instead of RAM for keeping tracking of whatever it does?

And would it be useful to add ignores for the subdirectories I know have
many files and back them up separately? Is --delete safe to use in this
case, as in does --delete with --ignore somedir/ not delete files in other
target dirs that are not in the ignore path?

Thanks,

/jl

--
ASCII ribbon campaign ( ) Powered by Lemote Fuloong
 against HTML e-mail   X  Loongson MIPS and OpenBSD
   and proprietary    / \    http://www.mutt.org
     attachments     /   \  Code Blue or Go Home!
 Encrypted email preferred  PGP Key 2048R/DA65BC04

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: Memory consumption for rsync -axv --delete

Paul Slootman-5
On Fri 25 Mar 2016, John Long wrote:
>
> I have been using rsync for many years and never had any kind of problem.
> Lately I am running out of RAM trying to do an incremental backup to a box
> that only has 2G of RAM. The entire directory structure I'm mirroring is
> about 200G of files. A minority of subdirectories have many files.
>
> Is there a way to do an incremental backup with --delete option that does
> not use as much memory? Is there a way to tell rsync to use a tempfile
> instead of RAM for keeping tracking of whatever it does?

No to the last question; you could consider adding (more) swapspace to
the system, which is effectively like using a tempfile.

> And would it be useful to add ignores for the subdirectories I know have
> many files and back them up separately? Is --delete safe to use in this
> case, as in does --delete with --ignore somedir/ not delete files in other
> target dirs that are not in the ignore path?

There's no --ignore, you probably mean --exclude.
I don't really understand what you're asking in your last question...
Why should --exclude somedir/ affect what --delete does elsewhere?
--delete will still delete stuff elsewhere if necessary.
Also look at the description of --delete and --delete-excluded, if you
have any questions about what's in the manpage then feel free to ask
those here; but for now I get the impression you haven't spent much time
reading the manpage.


Paul

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: Memory consumption for rsync -axv --delete

John Long
In reply to this post by John Long
On Fri, Mar 25, 2016 at 09:54:14AM +0000, John Long wrote:

> Hi,
>
> I have been using rsync for many years and never had any kind of problem.
> Lately I am running out of RAM trying to do an incremental backup to a box
> that only has 2G of RAM. The entire directory structure I'm mirroring is
> about 200G of files. A minority of subdirectories have many files.
>
> Is there a way to do an incremental backup with --delete option that does
> not use as much memory? Is there a way to tell rsync to use a tempfile
> instead of RAM for keeping tracking of whatever it does?
>
> And would it be useful to add ignores for the subdirectories I know have
> many files and back them up separately? Is --delete safe to use in this
> case, as in does --delete with --ignore somedir/ not delete files in other
> target dirs that are not in the ignore path?

I didn't phrase this part very well. Is --delete safe to use with --ignore,
meaning will rsync avoid deleting files in the ignore path on the target
side? I think the answer is probably yes but since I'm crashing the target
box with --delete I don't want to have to try this too many times.

Really I'm looking for a workaround to the high memory consumption so I can
sync up the file trees without exceeding the small RAM capacity of the
target box. Any suggestions appreciated.

Thanks,

/jl

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: Memory consumption for rsync -axv --delete

Kevin Korb
In reply to this post by John Long
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

If you were using --link-dest to make multiple backups you wouldn't
need --delete because the target is always a new empty directory (with
- --link-dest pointing to the previous backup run).

So, you get the benefit of having multiple backups to restore from and
rsync doesn't have to --delete.  When you run low on space you just rm
- -rf some old backups (takes a while but doesn't need much RAM).

On 03/25/2016 05:54 AM, John Long wrote:

> Hi,
>
> I have been using rsync for many years and never had any kind of
> problem. Lately I am running out of RAM trying to do an incremental
> backup to a box that only has 2G of RAM. The entire directory
> structure I'm mirroring is about 200G of files. A minority of
> subdirectories have many files.
>
> Is there a way to do an incremental backup with --delete option
> that does not use as much memory? Is there a way to tell rsync to
> use a tempfile instead of RAM for keeping tracking of whatever it
> does?
>
> And would it be useful to add ignores for the subdirectories I know
> have many files and back them up separately? Is --delete safe to
> use in this case, as in does --delete with --ignore somedir/ not
> delete files in other target dirs that are not in the ignore path?
>
> Thanks,
>
> /jl
>

- --
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
        Kevin Korb Phone:    (407) 252-6853
        Systems Administrator Internet:
        FutureQuest, Inc. [hidden email]  (work)
        Orlando, Florida [hidden email] (personal)
        Web page: http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlb1Vl8ACgkQVKC1jlbQAQccMQCfR+5LfpqH9to3D1QDBDScZOBX
RIQAn1BB2vFbf7eDgyy7HSS2SITBTWCM
=BYru
-----END PGP SIGNATURE-----

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: Memory consumption for rsync -axv --delete

John Long
Hi,

On Fri, Mar 25, 2016 at 11:16:47AM -0400, Kevin Korb wrote:

> If you were using --link-dest to make multiple backups you wouldn't
> need --delete because the target is always a new empty directory (with
> - --link-dest pointing to the previous backup run).

The source is around 200G and the target box only has 500G total and some of
it is used for other data. What I want to do is mirror the source on the
target and be able to prune the files that get deleted from the source from
the target also. I don't have enough space to back up the whole thing and
that is very time consuming anyway over 100M/b link which is why I was using
--delete. For a long time it was ok, but now I don't have enough RAM.

There is one giant directory that is probably problematic because it has a
huge number of files. I suspect this is the one that's causing me problems
but it is relatively static. I suppose it could be backed up and cleaned up
separately.

Is there any way to reduce RAM consumption on the target box while still
getting the benefit of the --delete function? I am thinking of trying to
back up everything but the gigantic directory with a large number of files,
and then backing up only that directory. Is this a reasonable strategy? I
just couldn't understand if --delete with --exclude would delete files from
the target outside the --exclude path. I guess the answer is no but it would
be a very time consuming mistake. I'm trying to make sure before I try it.

Thanks for your help and I'm sorry for my poorly worded post(s).

/jl

--
ASCII ribbon campaign ( ) Powered by Lemote Fuloong
 against HTML e-mail   X  Loongson MIPS and OpenBSD
   and proprietary    / \    http://www.mutt.org
     attachments     /   \  Code Blue or Go Home!
 Encrypted email preferred  PGP Key 2048R/DA65BC04

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: Memory consumption for rsync -axv --delete

Kevin Korb
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

You miss-understand the purpose of --link-dest.  Yes, it gives you
multiple complete backups, but each only consumes the disk space
needed to store files that are unique to that backup.  Files that are
the same in 2 backup runs are actually the same file in multiple
directories requiring only 1 to actually be stored.

On 03/27/2016 02:39 AM, John Long wrote:

> Hi,
>
> On Fri, Mar 25, 2016 at 11:16:47AM -0400, Kevin Korb wrote:
>
>> If you were using --link-dest to make multiple backups you
>> wouldn't need --delete because the target is always a new empty
>> directory (with - --link-dest pointing to the previous backup
>> run).
>
> The source is around 200G and the target box only has 500G total
> and some of it is used for other data. What I want to do is mirror
> the source on the target and be able to prune the files that get
> deleted from the source from the target also. I don't have enough
> space to back up the whole thing and that is very time consuming
> anyway over 100M/b link which is why I was using --delete. For a
> long time it was ok, but now I don't have enough RAM.
>
> There is one giant directory that is probably problematic because
> it has a huge number of files. I suspect this is the one that's
> causing me problems but it is relatively static. I suppose it could
> be backed up and cleaned up separately.
>
> Is there any way to reduce RAM consumption on the target box while
> still getting the benefit of the --delete function? I am thinking
> of trying to back up everything but the gigantic directory with a
> large number of files, and then backing up only that directory. Is
> this a reasonable strategy? I just couldn't understand if --delete
> with --exclude would delete files from the target outside the
> --exclude path. I guess the answer is no but it would be a very
> time consuming mistake. I'm trying to make sure before I try it.
>
> Thanks for your help and I'm sorry for my poorly worded post(s).
>
> /jl
>

- --
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
        Kevin Korb Phone:    (407) 252-6853
        Systems Administrator Internet:
        FutureQuest, Inc. [hidden email]  (work)
        Orlando, Florida [hidden email] (personal)
        Web page: http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlb3gl4ACgkQVKC1jlbQAQdZ7gCfXzhoaTSU2eW8fFUZQgA/8pjN
JQIAn3fQ6ESTB1gm1LT2AsO9kK4MV4ow
=Y4bU
-----END PGP SIGNATURE-----

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: Memory consumption for rsync -axv --delete

John Long
Thanks I'll look this up. There is still the issue of how to get the target
box cleaned up since I can no longer run --delete.

/jl

On Sun, Mar 27, 2016 at 02:49:02AM -0400, Kevin Korb wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> You miss-understand the purpose of --link-dest.  Yes, it gives you
> multiple complete backups, but each only consumes the disk space
> needed to store files that are unique to that backup.  Files that are
> the same in 2 backup runs are actually the same file in multiple
> directories requiring only 1 to actually be stored.
>
> On 03/27/2016 02:39 AM, John Long wrote:
> > Hi,
> >
> > On Fri, Mar 25, 2016 at 11:16:47AM -0400, Kevin Korb wrote:
> >
> >> If you were using --link-dest to make multiple backups you
> >> wouldn't need --delete because the target is always a new empty
> >> directory (with - --link-dest pointing to the previous backup
> >> run).
> >
> > The source is around 200G and the target box only has 500G total
> > and some of it is used for other data. What I want to do is mirror
> > the source on the target and be able to prune the files that get
> > deleted from the source from the target also. I don't have enough
> > space to back up the whole thing and that is very time consuming
> > anyway over 100M/b link which is why I was using --delete. For a
> > long time it was ok, but now I don't have enough RAM.
> >
> > There is one giant directory that is probably problematic because
> > it has a huge number of files. I suspect this is the one that's
> > causing me problems but it is relatively static. I suppose it could
> > be backed up and cleaned up separately.
> >
> > Is there any way to reduce RAM consumption on the target box while
> > still getting the benefit of the --delete function? I am thinking
> > of trying to back up everything but the gigantic directory with a
> > large number of files, and then backing up only that directory. Is
> > this a reasonable strategy? I just couldn't understand if --delete
> > with --exclude would delete files from the target outside the
> > --exclude path. I guess the answer is no but it would be a very
> > time consuming mistake. I'm trying to make sure before I try it.
> >
> > Thanks for your help and I'm sorry for my poorly worded post(s).
> >
> > /jl
> >
>
> - --
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> Kevin Korb Phone:    (407) 252-6853
> Systems Administrator Internet:
> FutureQuest, Inc. [hidden email]  (work)
> Orlando, Florida [hidden email] (personal)
> Web page: http://www.sanitarium.net/
> PGP public key available on web site.
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2
>
> iEYEARECAAYFAlb3gl4ACgkQVKC1jlbQAQdZ7gCfXzhoaTSU2eW8fFUZQgA/8pjN
> JQIAn3fQ6ESTB1gm1LT2AsO9kK4MV4ow
> =Y4bU
> -----END PGP SIGNATURE-----
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: Memory consumption for rsync -axv --delete

Kevin Korb
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

You will have an old backup dir and a new backup dir.  The new one
will contain all the current stuff.  The old one will contain what was
current the last time you ran rsync.  Just rm -rf the old one.  Or
keep a few.  Or a few dozen.

On 03/27/2016 02:54 AM, John Long wrote:

> Thanks I'll look this up. There is still the issue of how to get
> the target box cleaned up since I can no longer run --delete.
>
> /jl
>
> On Sun, Mar 27, 2016 at 02:49:02AM -0400, Kevin Korb wrote: You
> miss-understand the purpose of --link-dest.  Yes, it gives you
> multiple complete backups, but each only consumes the disk space
> needed to store files that are unique to that backup.  Files that
> are the same in 2 backup runs are actually the same file in
> multiple directories requiring only 1 to actually be stored.
>
> On 03/27/2016 02:39 AM, John Long wrote:
>>>> Hi,
>>>>
>>>> On Fri, Mar 25, 2016 at 11:16:47AM -0400, Kevin Korb wrote:
>>>>
>>>>> If you were using --link-dest to make multiple backups you
>>>>> wouldn't need --delete because the target is always a new
>>>>> empty directory (with - --link-dest pointing to the
>>>>> previous backup run).
>>>>
>>>> The source is around 200G and the target box only has 500G
>>>> total and some of it is used for other data. What I want to
>>>> do is mirror the source on the target and be able to prune
>>>> the files that get deleted from the source from the target
>>>> also. I don't have enough space to back up the whole thing
>>>> and that is very time consuming anyway over 100M/b link which
>>>> is why I was using --delete. For a long time it was ok, but
>>>> now I don't have enough RAM.
>>>>
>>>> There is one giant directory that is probably problematic
>>>> because it has a huge number of files. I suspect this is the
>>>> one that's causing me problems but it is relatively static. I
>>>> suppose it could be backed up and cleaned up separately.
>>>>
>>>> Is there any way to reduce RAM consumption on the target box
>>>> while still getting the benefit of the --delete function? I
>>>> am thinking of trying to back up everything but the gigantic
>>>> directory with a large number of files, and then backing up
>>>> only that directory. Is this a reasonable strategy? I just
>>>> couldn't understand if --delete with --exclude would delete
>>>> files from the target outside the --exclude path. I guess the
>>>> answer is no but it would be a very time consuming mistake.
>>>> I'm trying to make sure before I try it.
>>>>
>>>> Thanks for your help and I'm sorry for my poorly worded
>>>> post(s).
>>>>
>>>> /jl
>>>>
>
>>
>> -- Please use reply-all for most replies to avoid omitting the
>> mailing list. To unsubscribe or change options:
>> https://lists.samba.org/mailman/listinfo/rsync Before posting,
>> read: http://www.catb.org/~esr/faqs/smart-questions.html
>

- --
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
        Kevin Korb Phone:    (407) 252-6853
        Systems Administrator Internet:
        FutureQuest, Inc. [hidden email]  (work)
        Orlando, Florida [hidden email] (personal)
        Web page: http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlb3hKgACgkQVKC1jlbQAQdqCgCdGPKy5BJGKP5oqwHFIt+I6zvw
dgcAoJGkHyaai+LWPIWXiCDwTxz8V21+
=Gpd0
-----END PGP SIGNATURE-----

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: Memory consumption for rsync -axv --delete

John Long
Thank you.

On Sun, Mar 27, 2016 at 02:58:49AM -0400, Kevin Korb wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> You will have an old backup dir and a new backup dir.  The new one
> will contain all the current stuff.  The old one will contain what was
> current the last time you ran rsync.  Just rm -rf the old one.  Or
> keep a few.  Or a few dozen.
>
> On 03/27/2016 02:54 AM, John Long wrote:
> > Thanks I'll look this up. There is still the issue of how to get
> > the target box cleaned up since I can no longer run --delete.
> >

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: Memory consumption for rsync -axv --delete

Dan Stromberg-2
In reply to this post by John Long

You could probably use CIFS, NFS or sshfs.  It wouldn't be as fast, but the memory requirements should be less.


Sent from my T-Mobile 4G LTE Device


-------- Original message --------
From: John Long <[hidden email]>
Date:2016/03/25 04:10 (GMT-08:00)
Cc:
Subject: Re: Memory consumption for rsync -axv --delete

On Fri, Mar 25, 2016 at 09:54:14AM +0000, John Long wrote:

> Hi,
>
> I have been using rsync for many years and never had any kind of problem.
> Lately I am running out of RAM trying to do an incremental backup to a box
> that only has 2G of RAM. The entire directory structure I'm mirroring is
> about 200G of files. A minority of subdirectories have many files.
>
> Is there a way to do an incremental backup with --delete option that does
> not use as much memory? Is there a way to tell rsync to use a tempfile
> instead of RAM for keeping tracking of whatever it does?
>
> And would it be useful to add ignores for the subdirectories I know have
> many files and back them up separately? Is --delete safe to use in this
> case, as in does --delete with --ignore somedir/ not delete files in other
> target dirs that are not in the ignore path?

I didn't phrase this part very well. Is --delete safe to use with --ignore,
meaning will rsync avoid deleting files in the ignore path on the target
side? I think the answer is probably yes but since I'm crashing the target
box with --delete I don't want to have to try this too many times.

Really I'm looking for a workaround to the high memory consumption so I can
sync up the file trees without exceeding the small RAM capacity of the
target box. Any suggestions appreciated.

Thanks,

/jl

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: Memory consumption for rsync -axv --delete

John Long
Hi,

On Sun, Mar 27, 2016 at 06:39:53PM -0700, drsalists wrote:
>
> You could probably use CIFS, NFS or sshfs. ??It wouldn't be as fast, but the memory requirements should be less.

Sorry, I don't understand how to sync over NFS or sshfs without rsync. Can
you explain this please? I don't want one copy of the data- I want a backup,
but I don't want the backup to have extraneous files that I have deleted
from the source.

Thanks,

/jl

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html