Quantcast

[Bug 12769] New: error allocating core memory buffers (code 22) depending on source file system

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 12769] New: error allocating core memory buffers (code 22) depending on source file system

Samba - rsync mailing list
https://bugzilla.samba.org/show_bug.cgi?id=12769

            Bug ID: 12769
           Summary: error allocating core memory buffers (code 22)
                    depending on source file system
           Product: rsync
           Version: 3.1.0
          Hardware: All
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P5
         Component: core
          Assignee: [hidden email]
          Reporter: [hidden email]
        QA Contact: [hidden email]

We run an openSuSE Leap 42.2 and an Ubuntu 14.04.5 on two servers. Copying a
large number of files (in this case about 28 million) leads to different
results depending on the source file system.
We copy with rsync -rlptgoDAxHnP --info=progress2 --delete
--link-dest=$LINK_DEST root@$SERVER:/$FOLDER /backup/rsynctest/ . Replacing
--delete by --delete-delay doesn't change the behaviour as expected. The error
occurs with and without the option -n, in this case it is just for testing
reasons included.
In case the source is located on an Ext4 file system we run into the following
error message after about 26 million files copied:
ERROR: out of memory in hashtable_node [sender]
rsync error: error allocating core memory buffers (code 22) at util2.c(102)
[sender=3.1.0]
In case the source is located on an XFS file system the above command copies
all files without error.
Both of the file systems hold the same data as the one is the backup copy of
the other. The behaviour appears as well when we use rsync via an rsync server
and not via SSH and as well when we copy locally on one of the two machines.
And it appears regardless of the operating system (openSuSE 42.2 or Ubuntu
14.04.5).
I did not try in the last time but replaying the backup from an Ext4 showed
this error at least one year ago as well. With the change to XFS on the source
file system the error suddenly disappeared.
As the error appears even if just doing a --dry-run it seems to be related to
the way rsync handles metadata. The data size seems to be unimportant.

--
You are receiving this mail because:
You are the QA Contact for the bug.

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 12769] error allocating core memory buffers (code 22) depending on source file system

Samba - rsync mailing list
https://bugzilla.samba.org/show_bug.cgi?id=12769

--- Comment #1 from Roland Haberkorn <[hidden email]> ---
If you want me to run further testings with other file systems I am totally
willing to produce fake data and run tests. I just haven't done yet because of
my lack of knowledge about the underlying mechanisms and because I am not
totally sure whether this is a rsync's problem or a kernel issue.
To add two more things: We saw this issue also when having mounted the data
with NFSv3 or v4. The target file system does not matter, we've had this issue
with btrfs, Ext4 and XFS.

--
You are receiving this mail because:
You are the QA Contact for the bug.

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 12769] error allocating core memory buffers (code 22) depending on source file system

Samba - rsync mailing list
In reply to this post by Samba - rsync mailing list
https://bugzilla.samba.org/show_bug.cgi?id=12769

--- Comment #2 from Roland Haberkorn <[hidden email]> ---
I did some further investigation...
First thing to add: The ext4 file systems are hard-linked differential rsync
backups of the real data on XFS.
I changed the testcase by deleting the --link-dest option.
When rsyncing from an XFS, the rsync process on the client uses about 3% RAM
(of total 8GB). When rsyncing from an ext4, it uses up to about 50% RAM.
This picture totally changes when I delete the option -H. In this case also
copying from an ext4 uses only less than 2% RAM.
My guess would be that perhaps -H breaks the incremental recursion when copying
from an ext4.

--
You are receiving this mail because:
You are the QA Contact for the bug.

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Loading...