rsync 3.0.9 segmentation fault

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

rsync 3.0.9 segmentation fault

Aron Rotteveel
Hi,

I am experiencing segfaults when transferring files via rsync though sudo.
Setup:

- Backupserver initiates the rsync command with --delete -vvv --no-inc-recursive --numeric-ids --delete-excluded --relative --rsync-path=/home/backupuser/rsync-wrapper.sh
- rsync-wrapper.sh (on the client) contains /usr/bin/sudo /usr/bin/rsync "$@";
- user "backupuser" has sudo access to the rsync command
- Both host and client are running 3.0.9

The transfer starts and some files are actually transferred. Once a certain file is reached (plain PHP file, no special characters or any other peculiarities) it segfaults.

rsync host output:

[sender] make_file(<redacted>/libraries/phputf8/mbstring/strlen.php,*,2)
rsync: connection unexpectedly closed (51261222 bytes received so far) [Receiver]
rsync error: unexplained error (code 139) at io.c(605) [Receiver=3.0.9]
[Receiver] _exit_cleanup(code=12, file=io.c, line=605): about to call exit(139)
----------------------------------------------------------------------------
rsnapshot encountered an error! The program was invoked with these options:
/usr/bin/rsnapshot -c \
    /home/remotebackup/hosts/<redacted>/rsnapshot.conf sync
----------------------------------------------------------------------------
ERROR: /usr/bin/rsync returned 139 while processing backupuser@<redacted>:/backup/

Client output when using gdb to debug the coredump:

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff015fd000
Core was generated by `/usr/bin/rsync --server --sender -vvvlogDtprRe.Lsf --numeric-ids . /backup'.
Program terminated with signal 11, Segmentation fault.
#0  0x00000035cda7b441 in memcpy () from /lib64/libc.so.6

Any help would be greatly appreciated. Please let me know if additional info is required to properly debug this issue.
--
Best regards / Met vriendelijke groet,

Aron Rotteveel

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Aw: rsync 3.0.9 segmentation fault

devzero
Hi,
 
rsync 3.0.9 is quite ancient, more than 3 years old. A lot of bugs have been fixed since then.
 
Is there a chance to update to the latest rsync version and retry with that ?
 
regards
Roland
 
Gesendet: Dienstag, 17. März 2015 um 11:51 Uhr
Von: "Aron Rotteveel" <[hidden email]>
An: [hidden email]
Betreff: rsync 3.0.9 segmentation fault
Hi,
 
I am experiencing segfaults when transferring files via rsync though sudo.
Setup:
 
- Backupserver initiates the rsync command with --delete -vvv --no-inc-recursive --numeric-ids --delete-excluded --relative --rsync-path=/home/backupuser/rsync-wrapper.sh
- rsync-wrapper.sh (on the client) contains /usr/bin/sudo /usr/bin/rsync "$@";
- user "backupuser" has sudo access to the rsync command
- Both host and client are running 3.0.9
 
The transfer starts and some files are actually transferred. Once a certain file is reached (plain PHP file, no special characters or any other peculiarities) it segfaults.
 
rsync host output:
 
[sender] make_file(<redacted>/libraries/phputf8/mbstring/strlen.php,*,2)
rsync: connection unexpectedly closed (51261222 bytes received so far) [Receiver]
rsync error: unexplained error (code 139) at io.c(605) [Receiver=3.0.9]
[Receiver] _exit_cleanup(code=12, file=io.c, line=605): about to call exit(139)
----------------------------------------------------------------------------
rsnapshot encountered an error! The program was invoked with these options:
/usr/bin/rsnapshot -c \
    /home/remotebackup/hosts/<redacted>/rsnapshot.conf sync
----------------------------------------------------------------------------
ERROR: /usr/bin/rsync returned 139 while processing backupuser@<redacted>:/backup/
 
Client output when using gdb to debug the coredump:
 
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff015fd000
Core was generated by `/usr/bin/rsync --server --sender -vvvlogDtprRe.Lsf --numeric-ids . /backup'.
Program terminated with signal 11, Segmentation fault.
#0  0x00000035cda7b441 in memcpy () from /lib64/libc.so.6
 
Any help would be greatly appreciated. Please let me know if additional info is required to properly debug this issue.
--
Best regards / Met vriendelijke groet,

Aron Rotteveel
-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Aw: Re: rsync 3.0.9 segmentation fault

devzero
Hi Aron,
 
i hope it`s ok for you if i bring this back on-list. Your issue and the way or possible fix to resolve it may be interesting for others too (that may include future searches etc)
 
so with 3.1.1 we are a step further....
 
i don`t really have a clue what`s happening here but my next step would be taking a closer look on how the memory usage of rsync on the client and server grows.
 
you could log it like this:
while true;do ps -eo vsz,rss,sz,rsync|grep cron;sleep 10;done >logfile
 
does it grow continuously? does the oom situation reproducibly happen at a certain size ?
what`s the client and server platform?
how many files? (-> https://rsync.samba.org/FAQ.html#5 ! )
 
regards
roland
 
Gesendet: Donnerstag, 19. März 2015 um 12:24 Uhr
Von: "Aron Rotteveel" <[hidden email]>
An: [hidden email]
Betreff: Re: rsync 3.0.9 segmentation fault
In addition to my last message:
  • Client (sender) has 16GB's or RAM, of which only 6.5GB is used during peak.
  • I tried using --no-inc-recursive, but it does not solve the issue.
What currrently is puzzling me is the question of why I am receiving these errors when my server seems to have plenty of memory to spare.
 
--
Best regards / Met vriendelijke groet,

Aron Rotteveel
 
2015-03-19 11:52 GMT+01:00 Aron Rotteveel <[hidden email]>:
Hi Roland,
 
I just upgrade both the client and host to 3.1.1 and seem to memory related issues now:
 
ERROR: out of memory in make_file [sender]
rsync error: error allocating core memory buffers (code 22) at util2.c(102) [sender=3.1.1]
[sender] _exit_cleanup(code=22, file=util2.c, line=102): about to call exit(22)
[Receiver] _exit_cleanup(code=22, file=io.c, line=1633): about to call exit(22)
----------------------------------------------------------------------------
rsnapshot encountered an error! The program was invoked with these options:
/usr/bin/rsnapshot -c \
    /home/remotebackup/hosts/<redacted>/rsnapshot.conf sync
----------------------------------------------------------------------------
 
--
Best regards / Met vriendelijke groet,

Aron Rotteveel
 
2015-03-18 23:43 GMT+01:00 <[hidden email]>:
Hi,
 
rsync 3.0.9 is quite ancient, more than 3 years old. A lot of bugs have been fixed since then.
 
Is there a chance to update to the latest rsync version and retry with that ?
 
regards
Roland
 
Gesendet: Dienstag, 17. März 2015 um 11:51 Uhr
Von: "Aron Rotteveel" <[hidden email]>
An: [hidden email]
Betreff: rsync 3.0.9 segmentation fault
Hi,
 
I am experiencing segfaults when transferring files via rsync though sudo.
Setup:
 
- Backupserver initiates the rsync command with --delete -vvv --no-inc-recursive --numeric-ids --delete-excluded --relative --rsync-path=/home/backupuser/rsync-wrapper.sh
- rsync-wrapper.sh (on the client) contains /usr/bin/sudo /usr/bin/rsync "$@";
- user "backupuser" has sudo access to the rsync command
- Both host and client are running 3.0.9
 
The transfer starts and some files are actually transferred. Once a certain file is reached (plain PHP file, no special characters or any other peculiarities) it segfaults.
 
rsync host output:
 
[sender] make_file(<redacted>/libraries/phputf8/mbstring/strlen.php,*,2)
rsync: connection unexpectedly closed (51261222 bytes received so far) [Receiver]
rsync error: unexplained error (code 139) at io.c(605) [Receiver=3.0.9]
[Receiver] _exit_cleanup(code=12, file=io.c, line=605): about to call exit(139)
----------------------------------------------------------------------------
rsnapshot encountered an error! The program was invoked with these options:
/usr/bin/rsnapshot -c \
    /home/remotebackup/hosts/<redacted>/rsnapshot.conf sync
----------------------------------------------------------------------------
ERROR: /usr/bin/rsync returned 139 while processing backupuser@<redacted>:/backup/
 
Client output when using gdb to debug the coredump:
 
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff015fd000
Core was generated by `/usr/bin/rsync --server --sender -vvvlogDtprRe.Lsf --numeric-ids . /backup'.
Program terminated with signal 11, Segmentation fault.
#0  0x00000035cda7b441 in memcpy () from /lib64/libc.so.6
 
Any help would be greatly appreciated. Please let me know if additional info is required to properly debug this issue.
--
Best regards / Met vriendelijke groet,

Aron Rotteveel
-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: Aw: Re: rsync 3.0.9 segmentation fault

Kevin Korb
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Using --no-inc-recursive would increase memory usage not reduce it.
That option forces rsync to keep the entire tree in memory.

On 03/19/2015 03:10 PM, [hidden email] wrote:

> Hi Aron,
>
> i hope it`s ok for you if i bring this back on-list. Your issue and
> the way or possible fix to resolve it may be interesting for others
> too (that may include future searches etc)
>
> so with 3.1.1 we are a step further....
>
> i don`t really have a clue what`s happening here but my next step
> would be taking a closer look on how the memory usage of rsync on
> the client and server grows.
>
> you could log it like this: while true;do ps -eo
> vsz,rss,sz,rsync|grep cron;sleep 10;done >logfile
>
> does it grow continuously? does the oom situation reproducibly
> happen at a certain size ? what`s the client and server platform?
> how many files? (-> https://rsync.samba.org/FAQ.html#5 ! )
>
> regards roland
>
> *Gesendet:* Donnerstag, 19. März 2015 um 12:24 Uhr *Von:* "Aron
> Rotteveel" <[hidden email]> *An:* [hidden email]
> *Betreff:* Re: rsync 3.0.9 segmentation fault In addition to my
> last message:
>
> * Client (sender) has 16GB's or RAM, of which only 6.5GB is used
> during peak. * I tried using --no-inc-recursive, but it does not
> solve the issue.
>
> What currrently is puzzling me is the question of why I am
> receiving these errors when my server seems to have plenty of
> memory to spare.
>
> -- Best regards / Met vriendelijke groet,
>
> Aron Rotteveel
>
> 2015-03-19 11:52 GMT+01:00 Aron Rotteveel
> <[hidden email]>:
>
> Hi Roland,
>
> I just upgrade both the client and host to 3.1.1 and seem to
> memory related issues now:
>
> ERROR: out of memory in make_file [sender] rsync error: error
> allocating core memory buffers (code 22) at util2.c(102)
> [sender=3.1.1] [sender] _exit_cleanup(code=22, file=util2.c,
> line=102): about to call exit(22) [Receiver] _exit_cleanup(code=22,
> file=io.c, line=1633): about to call exit(22)
> ----------------------------------------------------------------------------
>
>
rsnapshot encountered an error! The program was invoked with these

> options: /usr/bin/rsnapshot -c \
> /home/remotebackup/hosts/<redacted>/rsnapshot.conf sync
> ----------------------------------------------------------------------------
>
>  -- Best regards / Met vriendelijke groet,
>
> Aron Rotteveel
>
> 2015-03-18 23:43 GMT+01:00 <[hidden email]>:
>
> Hi,
>
> rsync 3.0.9 is quite ancient, more than 3 years old. A lot of bugs
> have been fixed since then.
>
> Is there a chance to update to the latest rsync version and retry
> with that ?
>
> regards Roland
>
> *Gesendet:* Dienstag, 17. März 2015 um 11:51 Uhr *Von:* "Aron
> Rotteveel" <[hidden email]> *An:* [hidden email]
> *Betreff:* rsync 3.0.9 segmentation fault Hi,
>
> I am experiencing segfaults when transferring files via rsync
> though sudo. Setup:
>
> - Backupserver initiates the rsync command with --delete -vvv
> --no-inc-recursive --numeric-ids --delete-excluded --relative
> --rsync-path=/home/backupuser/rsync-wrapper.sh - rsync-wrapper.sh
> (on the client) contains /usr/bin/sudo /usr/bin/rsync "$@"; - user
> "backupuser" has sudo access to the rsync command - Both host and
> client are running 3.0.9
>
> The transfer starts and some files are actually transferred. Once a
> certain file is reached (plain PHP file, no special characters or
> any other peculiarities) it segfaults.
>
> rsync host output:
>
> [sender]
> make_file(<redacted>/libraries/phputf8/mbstring/strlen.php,*,2)
> rsync: connection unexpectedly closed (51261222 bytes received so
> far) [Receiver] rsync error: unexplained error (code 139) at
> io.c(605) [Receiver=3.0.9] [Receiver] _exit_cleanup(code=12,
> file=io.c, line=605): about to call exit(139)
> ----------------------------------------------------------------------------
>
>
rsnapshot encountered an error! The program was invoked with
> these options: /usr/bin/rsnapshot -c \
> /home/remotebackup/hosts/<redacted>/rsnapshot.conf sync
> ----------------------------------------------------------------------------
>
>
ERROR: /usr/bin/rsync returned 139 while processing

> backupuser@<redacted>:/backup/
>
> Client output when using gdb to debug the coredump:
>
> warning: no loadable sections found in added symbol-file
> system-supplied DSO at 0x7fff015fd000 Core was generated by
> `/usr/bin/rsync --server --sender -vvvlogDtprRe.Lsf --numeric-ids .
> /backup'. Program terminated with signal 11, Segmentation fault. #0
> 0x00000035cda7b441 in memcpy () from /lib64/libc.so.6
>
> Any help would be greatly appreciated. Please let me know if
> additional info is required to properly debug this issue. -- Best
> regards / Met vriendelijke groet,
>
> Aron Rotteveel -- Please use reply-all for most replies to avoid
> omitting the mailing list. To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync Before posting,
> read: http://www.catb.org/~esr/faqs/smart-questions.html
>
>
>

- --
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
        Kevin Korb Phone:    (407) 252-6853
        Systems Administrator Internet:
        FutureQuest, Inc. [hidden email]  (work)
        Orlando, Florida [hidden email] (personal)
        Web page: http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlULH/cACgkQVKC1jlbQAQcs2wCfWcwjC5Ljed22UAwjkrcf1p8s
vyIAoJ8AkX6OqxdNJFZjkNMjccMFc65Y
=PP+Q
-----END PGP SIGNATURE-----
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: Re: rsync 3.0.9 segmentation fault

Aron Rotteveel
In reply to this post by devzero
Hi Roland,

Thanks for the reply. Memory usage on both machines seem fine. The server has 4GB's of RAM, of which about 3GB is used during the file list build and about 1.5GB is used during the actual transfer. The client has 16GB of RAM with a peak usage of 8.5GB. 

I just tried three transfers in a row and it consistently breaks at a certain point, after which I get the "ERROR: out of memory in flist_expand [sender]" error. There is not much special to mention regarding the file on which it breaks: it's a 22KB JPEG file with no special attributes.

The backup server is running Debian 7.8, the client runs on CentOS 5.11.

A `find . | wc -l` in the backup directory results in 7434013 files.

--
Best regards / Met vriendelijke groet,

Aron Rotteveel

2015-03-19 20:10 GMT+01:00 <[hidden email]>:
Hi Aron,
 
i hope it`s ok for you if i bring this back on-list. Your issue and the way or possible fix to resolve it may be interesting for others too (that may include future searches etc)
 
so with 3.1.1 we are a step further....
 
i don`t really have a clue what`s happening here but my next step would be taking a closer look on how the memory usage of rsync on the client and server grows.
 
you could log it like this:
while true;do ps -eo vsz,rss,sz,rsync|grep cron;sleep 10;done >logfile
 
does it grow continuously? does the oom situation reproducibly happen at a certain size ?
what`s the client and server platform?
how many files? (-> https://rsync.samba.org/FAQ.html#5 ! )
 
regards
roland
 
Gesendet: Donnerstag, 19. März 2015 um 12:24 Uhr
Von: "Aron Rotteveel" <[hidden email]>
An: [hidden email]
Betreff: Re: rsync 3.0.9 segmentation fault
In addition to my last message:
  • Client (sender) has 16GB's or RAM, of which only 6.5GB is used during peak.
  • I tried using --no-inc-recursive, but it does not solve the issue.
What currrently is puzzling me is the question of why I am receiving these errors when my server seems to have plenty of memory to spare.
 
--
Best regards / Met vriendelijke groet,

Aron Rotteveel
 
2015-03-19 11:52 GMT+01:00 Aron Rotteveel <[hidden email]>:
Hi Roland,
 
I just upgrade both the client and host to 3.1.1 and seem to memory related issues now:
 
ERROR: out of memory in make_file [sender]
rsync error: error allocating core memory buffers (code 22) at util2.c(102) [sender=3.1.1]
[sender] _exit_cleanup(code=22, file=util2.c, line=102): about to call exit(22)
[Receiver] _exit_cleanup(code=22, file=io.c, line=1633): about to call exit(22)
----------------------------------------------------------------------------
rsnapshot encountered an error! The program was invoked with these options:
/usr/bin/rsnapshot -c \
    /home/remotebackup/hosts/<redacted>/rsnapshot.conf sync
----------------------------------------------------------------------------
 
--
Best regards / Met vriendelijke groet,

Aron Rotteveel
 
2015-03-18 23:43 GMT+01:00 <[hidden email]>:
Hi,
 
rsync 3.0.9 is quite ancient, more than 3 years old. A lot of bugs have been fixed since then.
 
Is there a chance to update to the latest rsync version and retry with that ?
 
regards
Roland
 
Gesendet: Dienstag, 17. März 2015 um 11:51 Uhr
Von: "Aron Rotteveel" <[hidden email]>
An: [hidden email]
Betreff: rsync 3.0.9 segmentation fault
Hi,
 
I am experiencing segfaults when transferring files via rsync though sudo.
Setup:
 
- Backupserver initiates the rsync command with --delete -vvv --no-inc-recursive --numeric-ids --delete-excluded --relative --rsync-path=/home/backupuser/rsync-wrapper.sh
- rsync-wrapper.sh (on the client) contains /usr/bin/sudo /usr/bin/rsync "$@";
- user "backupuser" has sudo access to the rsync command
- Both host and client are running 3.0.9
 
The transfer starts and some files are actually transferred. Once a certain file is reached (plain PHP file, no special characters or any other peculiarities) it segfaults.
 
rsync host output:
 
[sender] make_file(<redacted>/libraries/phputf8/mbstring/strlen.php,*,2)
rsync: connection unexpectedly closed (51261222 bytes received so far) [Receiver]
rsync error: unexplained error (code 139) at io.c(605) [Receiver=3.0.9]
[Receiver] _exit_cleanup(code=12, file=io.c, line=605): about to call exit(139)
----------------------------------------------------------------------------
rsnapshot encountered an error! The program was invoked with these options:
/usr/bin/rsnapshot -c \
    /home/remotebackup/hosts/<redacted>/rsnapshot.conf sync
----------------------------------------------------------------------------
ERROR: /usr/bin/rsync returned 139 while processing backupuser@<redacted>:/backup/
 
Client output when using gdb to debug the coredump:
 
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff015fd000
Core was generated by `/usr/bin/rsync --server --sender -vvvlogDtprRe.Lsf --numeric-ids . /backup'.
Program terminated with signal 11, Segmentation fault.
#0  0x00000035cda7b441 in memcpy () from /lib64/libc.so.6
 
Any help would be greatly appreciated. Please let me know if additional info is required to properly debug this issue.
--
Best regards / Met vriendelijke groet,

Aron Rotteveel
-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync 3.0.9 segmentation fault

Aron Rotteveel
Yes, I removed "--no-inc-recursive", without success.

--
Best regards / Met vriendelijke groet,

Aron Rotteveel

2015-03-27 12:24 GMT+01:00 Kevin Korb <[hidden email]>:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Have you tried removing --no-inc-recursive yet?

On 03/27/2015 07:19 AM, Aron Rotteveel wrote:
> Hi Roland,
>
> Thanks for the reply. Memory usage on both machines seem fine. The
> server has 4GB's of RAM, of which about 3GB is used during the file
> list build and about 1.5GB is used during the actual transfer. The
> client has 16GB of RAM with a peak usage of 8.5GB.
>
> I just tried three transfers in a row and it consistently breaks at
> a certain point, after which I get the "ERROR: out of memory in
> flist_expand [sender]" error. There is not much special to mention
> regarding the file on which it breaks: it's a 22KB JPEG file with
> no special attributes.
>
> The backup server is running Debian 7.8, the client runs on CentOS
> 5.11.
>
> A `find . | wc -l` in the backup directory results in 7434013
> files.
>
> -- Best regards / Met vriendelijke groet,
>
> Aron Rotteveel
>
> 2015-03-19 20:10 GMT+01:00 <[hidden email]
> <mailto:[hidden email]>>:
>
> Hi Aron,
>
> i hope it`s ok for you if i bring this back on-list. Your issue
> and the way or possible fix to resolve it may be interesting for
> others too (that may include future searches etc)
>
> so with 3.1.1 we are a step further....
>
> i don`t really have a clue what`s happening here but my next step
> would be taking a closer look on how the memory usage of rsync on
> the client and server grows.
>
> you could log it like this: while true;do ps -eo
> vsz,rss,sz,rsync|grep cron;sleep 10;done >logfile
>
> does it grow continuously? does the oom situation reproducibly
> happen at a certain size ? what`s the client and server platform?
> how many files? (-> https://rsync.samba.org/FAQ.html#5 ! )
>
> regards roland
>
> *Gesendet:* Donnerstag, 19. März 2015 um 12:24 Uhr *Von:* "Aron
> Rotteveel" <[hidden email]
> <mailto:[hidden email]>> *An:* [hidden email]
> <mailto:[hidden email]> *Betreff:* Re: rsync 3.0.9 segmentation
> fault In addition to my last message:
>
> * Client (sender) has 16GB's or RAM, of which only 6.5GB is used
> during peak. * I tried using --no-inc-recursive, but it does not
> solve the issue.
>
> What currrently is puzzling me is the question of why I am
> receiving these errors when my server seems to have plenty of
> memory to spare.
>
> -- Best regards / Met vriendelijke groet,
>
> Aron Rotteveel
>
> 2015-03-19 11:52 GMT+01:00 Aron Rotteveel
> <[hidden email]>:
>
> Hi Roland,
>
> I just upgrade both the client and host to 3.1.1 and seem to memory
> related issues now:
>
> ERROR: out of memory in make_file [sender] rsync error: error
> allocating core memory buffers (code 22) at util2.c(102)
> [sender=3.1.1] [sender] _exit_cleanup(code=22, file=util2.c,
> line=102): about to call exit(22) [Receiver] _exit_cleanup(code=22,
> file=io.c, line=1633): about to call exit(22)
> ----------------------------------------------------------------------------
>
>
rsnapshot encountered an error! The program was invoked with
> these options: /usr/bin/rsnapshot -c \
> /home/remotebackup/hosts/<redacted>/rsnapshot.conf sync
> ----------------------------------------------------------------------------
>
>  -- Best regards / Met vriendelijke groet,
>
> Aron Rotteveel
>
> 2015-03-18 23:43 GMT+01:00 <[hidden email]>:
>
> Hi,
>
> rsync 3.0.9 is quite ancient, more than 3 years old. A lot of bugs
> have been fixed since then.
>
> Is there a chance to update to the latest rsync version and retry
> with that ?
>
> regards Roland
>
> *Gesendet:* Dienstag, 17. März 2015 um 11:51 Uhr *Von:* "Aron
> Rotteveel" <[hidden email]> *An:* [hidden email]
> *Betreff:* rsync 3.0.9 segmentation fault Hi,
>
> I am experiencing segfaults when transferring files via rsync
> though sudo. Setup:
>
> - Backupserver initiates the rsync command with --delete -vvv
> --no-inc-recursive --numeric-ids --delete-excluded --relative
> --rsync-path=/home/backupuser/rsync-wrapper.sh - rsync-wrapper.sh
> (on the client) contains /usr/bin/sudo /usr/bin/rsync "$@"; - user
> "backupuser" has sudo access to the rsync command - Both host and
> client are running 3.0.9
>
> The transfer starts and some files are actually transferred. Once a
> certain file is reached (plain PHP file, no special characters or
> any other peculiarities) it segfaults.
>
> rsync host output:
>
> [sender]
> make_file(<redacted>/libraries/phputf8/mbstring/strlen.php,*,2)
> rsync: connection unexpectedly closed (51261222 bytes received so
> far) [Receiver] rsync error: unexplained error (code 139) at
> io.c(605) [Receiver=3.0.9] [Receiver] _exit_cleanup(code=12,
> file=io.c, line=605): about to call exit(139)
> ----------------------------------------------------------------------------
>
>
rsnapshot encountered an error! The program was invoked with
> these options: /usr/bin/rsnapshot -c \
> /home/remotebackup/hosts/<redacted>/rsnapshot.conf sync
> ----------------------------------------------------------------------------
>
>
ERROR: /usr/bin/rsync returned 139 while processing
> backupuser@<redacted>:/backup/
>
> Client output when using gdb to debug the coredump:
>
> warning: no loadable sections found in added symbol-file
> system-supplied DSO at 0x7fff015fd000 Core was generated by
> `/usr/bin/rsync --server --sender -vvvlogDtprRe.Lsf --numeric-ids .
> /backup'. Program terminated with signal 11, Segmentation fault. #0
> 0x00000035cda7b441 in memcpy () from /lib64/libc.so.6
>
> Any help would be greatly appreciated. Please let me know if
> additional info is required to properly debug this issue. -- Best
> regards / Met vriendelijke groet,
>
> Aron Rotteveel -- Please use reply-all for most replies to avoid
> omitting the mailing list. To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync Before posting,
> read: http://www.catb.org/~esr/faqs/smart-questions.html
>
>
>
>

- --
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
        Kevin Korb                      Phone:    (407) 252-6853
        Systems Administrator           Internet:
        FutureQuest, Inc.               [hidden email]  (work)
        Orlando, Florida                [hidden email] (personal)
        Web page:                       http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlUVPe8ACgkQVKC1jlbQAQdJfQCdFpMSsACi2df9P8eJZQAi4qeB
xA8Amwbpj9PGpzeQfrd4VnAytU9dN1uc
=1Ayf
-----END PGP SIGNATURE-----


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Aw: Re: Re: rsync 3.0.9 segmentation fault

devzero
In reply to this post by Aron Rotteveel
>build and about 1.5GB is used during the actual transfer. The client has 16GB of RAM with a peak usage of 8.5GB. 
 
1.5GB + 8.5GB of systems memory, including buffers etc?
give it a closer look to the rsync process with ps (as mentioned below)
 
also have a look at:
 
but what is mentioned there does not really fit to your problem with <8M files - as that theoretically should sum up to <1GB rsync memory requirement (if things mentioned in the faq are still valid)
 
are your rsync binares 32 or 64 bit ?
 
regards
roland
 
 
 
Gesendet: Freitag, 27. März 2015 um 12:19 Uhr
Von: "Aron Rotteveel" <[hidden email]>
An: [hidden email]
Cc: [hidden email]
Betreff: Re: Re: rsync 3.0.9 segmentation fault
Hi Roland,
 
Thanks for the reply. Memory usage on both machines seem fine. The server has 4GB's of RAM, of which about 3GB is used during the file list build and about 1.5GB is used during the actual transfer. The client has 16GB of RAM with a peak usage of 8.5GB. 
 
I just tried three transfers in a row and it consistently breaks at a certain point, after which I get the "ERROR: out of memory in flist_expand [sender]" error. There is not much special to mention regarding the file on which it breaks: it's a 22KB JPEG file with no special attributes.
 
The backup server is running Debian 7.8, the client runs on CentOS 5.11.
 
A `find . | wc -l` in the backup directory results in 7434013 files.
 
--
Best regards / Met vriendelijke groet,

Aron Rotteveel
 
2015-03-19 20:10 GMT+01:00 <devzero@...>:
Hi Aron,
 
i hope it`s ok for you if i bring this back on-list. Your issue and the way or possible fix to resolve it may be interesting for others too (that may include future searches etc)
 
so with 3.1.1 we are a step further....
 
i don`t really have a clue what`s happening here but my next step would be taking a closer look on how the memory usage of rsync on the client and server grows.
 
you could log it like this:
while true;do ps -eo vsz,rss,sz,rsync|grep cron;sleep 10;done >logfile
 
does it grow continuously? does the oom situation reproducibly happen at a certain size ?
what`s the client and server platform?
how many files? (-> https://rsync.samba.org/FAQ.html#5 ! )
 
regards
roland
 
Gesendet: Donnerstag, 19. März 2015 um 12:24 Uhr
Von: "Aron Rotteveel" <rotteveel.aron@...>
An: devzero@...
Betreff: Re: rsync 3.0.9 segmentation fault
In addition to my last message:
  • Client (sender) has 16GB's or RAM, of which only 6.5GB is used during peak.
  • I tried using --no-inc-recursive, but it does not solve the issue.
What currrently is puzzling me is the question of why I am receiving these errors when my server seems to have plenty of memory to spare.
 
--
Best regards / Met vriendelijke groet,

Aron Rotteveel
 
2015-03-19 11:52 GMT+01:00 Aron Rotteveel <[hidden email]>:
Hi Roland,
 
I just upgrade both the client and host to 3.1.1 and seem to memory related issues now:
 
ERROR: out of memory in make_file [sender]
rsync error: error allocating core memory buffers (code 22) at util2.c(102) [sender=3.1.1]
[sender] _exit_cleanup(code=22, file=util2.c, line=102): about to call exit(22)
[Receiver] _exit_cleanup(code=22, file=io.c, line=1633): about to call exit(22)
----------------------------------------------------------------------------
rsnapshot encountered an error! The program was invoked with these options:
/usr/bin/rsnapshot -c \
    /home/remotebackup/hosts/<redacted>/rsnapshot.conf sync
----------------------------------------------------------------------------
 
--
Best regards / Met vriendelijke groet,

Aron Rotteveel
 
2015-03-18 23:43 GMT+01:00 <[hidden email]>:
Hi,
 
rsync 3.0.9 is quite ancient, more than 3 years old. A lot of bugs have been fixed since then.
 
Is there a chance to update to the latest rsync version and retry with that ?
 
regards
Roland
 
Gesendet: Dienstag, 17. März 2015 um 11:51 Uhr
Von: "Aron Rotteveel" <[hidden email]>
An: [hidden email]
Betreff: rsync 3.0.9 segmentation fault
Hi,
 
I am experiencing segfaults when transferring files via rsync though sudo.
Setup:
 
- Backupserver initiates the rsync command with --delete -vvv --no-inc-recursive --numeric-ids --delete-excluded --relative --rsync-path=/home/backupuser/rsync-wrapper.sh
- rsync-wrapper.sh (on the client) contains /usr/bin/sudo /usr/bin/rsync "$@";
- user "backupuser" has sudo access to the rsync command
- Both host and client are running 3.0.9
 
The transfer starts and some files are actually transferred. Once a certain file is reached (plain PHP file, no special characters or any other peculiarities) it segfaults.
 
rsync host output:
 
[sender] make_file(<redacted>/libraries/phputf8/mbstring/strlen.php,*,2)
rsync: connection unexpectedly closed (51261222 bytes received so far) [Receiver]
rsync error: unexplained error (code 139) at io.c(605) [Receiver=3.0.9]
[Receiver] _exit_cleanup(code=12, file=io.c, line=605): about to call exit(139)
----------------------------------------------------------------------------
rsnapshot encountered an error! The program was invoked with these options:
/usr/bin/rsnapshot -c \
    /home/remotebackup/hosts/<redacted>/rsnapshot.conf sync
----------------------------------------------------------------------------
ERROR: /usr/bin/rsync returned 139 while processing backupuser@<redacted>:/backup/
 
Client output when using gdb to debug the coredump:
 
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff015fd000
Core was generated by `/usr/bin/rsync --server --sender -vvvlogDtprRe.Lsf --numeric-ids . /backup'.
Program terminated with signal 11, Segmentation fault.
#0  0x00000035cda7b441 in memcpy () from /lib64/libc.so.6
 
Any help would be greatly appreciated. Please let me know if additional info is required to properly debug this issue.
--
Best regards / Met vriendelijke groet,

Aron Rotteveel
-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync 3.0.9 segmentation fault

Kevin Korb
In reply to this post by Aron Rotteveel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Try also removing --delete-excluded.  Without those two options there
should be no reason for rsync to require gigs of RAM.  Well, unless
the other system has rsync 2.x.

On 03/27/2015 07:29 AM, Aron Rotteveel wrote:

> Yes, I removed "--no-inc-recursive", without success.
>
> -- Best regards / Met vriendelijke groet,
>
> Aron Rotteveel
>
> 2015-03-27 12:24 GMT+01:00 Kevin Korb <[hidden email]
> <mailto:[hidden email]>>:
>
> Have you tried removing --no-inc-recursive yet?
>
> On 03/27/2015 07:19 AM, Aron Rotteveel wrote:
>> Hi Roland,
>
>> Thanks for the reply. Memory usage on both machines seem fine.
>> The server has 4GB's of RAM, of which about 3GB is used during
>> the file list build and about 1.5GB is used during the actual
>> transfer. The client has 16GB of RAM with a peak usage of 8.5GB.
>
>> I just tried three transfers in a row and it consistently breaks
>> at a certain point, after which I get the "ERROR: out of memory
>> in flist_expand [sender]" error. There is not much special to
>> mention regarding the file on which it breaks: it's a 22KB JPEG
>> file with no special attributes.
>
>> The backup server is running Debian 7.8, the client runs on
>> CentOS 5.11.
>
>> A `find . | wc -l` in the backup directory results in 7434013
>> files.
>
>> -- Best regards / Met vriendelijke groet,
>
>> Aron Rotteveel
>
>> 2015-03-19 20:10 GMT+01:00 <[hidden email]
>> <mailto:[hidden email]> <mailto:[hidden email]
>> <mailto:[hidden email]>>>:
>
>> Hi Aron,
>
>> i hope it`s ok for you if i bring this back on-list. Your issue
>> and the way or possible fix to resolve it may be interesting for
>> others too (that may include future searches etc)
>
>> so with 3.1.1 we are a step further....
>
>> i don`t really have a clue what`s happening here but my next
>> step would be taking a closer look on how the memory usage of
>> rsync on the client and server grows.
>
>> you could log it like this: while true;do ps -eo
>> vsz,rss,sz,rsync|grep cron;sleep 10;done >logfile
>
>> does it grow continuously? does the oom situation reproducibly
>> happen at a certain size ? what`s the client and server
>> platform? how many files? (-> https://rsync.samba.org/FAQ.html#5
>> ! )
>
>> regards roland
>
>> *Gesendet:* Donnerstag, 19. März 2015 um 12:24 Uhr *Von:* "Aron
>> Rotteveel" <[hidden email]
>> <mailto:[hidden email]>
>> <mailto:[hidden email]
> <mailto:[hidden email]>>> *An:* [hidden email]
> <mailto:[hidden email]>
>> <mailto:[hidden email] <mailto:[hidden email]>> *Betreff:* Re:
> rsync 3.0.9 segmentation
>> fault In addition to my last message:
>
>> * Client (sender) has 16GB's or RAM, of which only 6.5GB is used
>> during peak. * I tried using --no-inc-recursive, but it does not
>> solve the issue.
>
>> What currrently is puzzling me is the question of why I am
>> receiving these errors when my server seems to have plenty of
>> memory to spare.
>
>> -- Best regards / Met vriendelijke groet,
>
>> Aron Rotteveel
>
>> 2015-03-19 11:52 GMT+01:00 Aron Rotteveel
>> <[hidden email] <mailto:[hidden email]>>:
>
>> Hi Roland,
>
>> I just upgrade both the client and host to 3.1.1 and seem to
>> memory related issues now:
>
>> ERROR: out of memory in make_file [sender] rsync error: error
>> allocating core memory buffers (code 22) at util2.c(102)
>> [sender=3.1.1] [sender] _exit_cleanup(code=22, file=util2.c,
>> line=102): about to call exit(22) [Receiver]
>> _exit_cleanup(code=22, file=io.c, line=1633): about to call
>> exit(22)
>
>
>

- --
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
        Kevin Korb Phone:    (407) 252-6853
        Systems Administrator Internet:
        FutureQuest, Inc. [hidden email]  (work)
        Orlando, Florida [hidden email] (personal)
        Web page: http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlUVWZIACgkQVKC1jlbQAQfQ4wCeOjzVgtBt0t9LQ4Mf9X3kOhjF
tEcAoJAh158PF51O3Vnn8alkd7q0iSHQ
=pQeg
-----END PGP SIGNATURE-----
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync 3.0.9 segmentation fault

Aron Rotteveel
I am now running with --delete --numeric-ids --relative but the problem still persists.

--
Best regards / Met vriendelijke groet,

Aron Rotteveel

2015-03-27 14:22 GMT+01:00 Kevin Korb <[hidden email]>:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Try also removing --delete-excluded.  Without those two options there
should be no reason for rsync to require gigs of RAM.  Well, unless
the other system has rsync 2.x.

On 03/27/2015 07:29 AM, Aron Rotteveel wrote:
> Yes, I removed "--no-inc-recursive", without success.
>
> -- Best regards / Met vriendelijke groet,
>
> Aron Rotteveel
>
> 2015-03-27 12:24 GMT+01:00 Kevin Korb <[hidden email]
> <mailto:[hidden email]>>:
>
> Have you tried removing --no-inc-recursive yet?
>
> On 03/27/2015 07:19 AM, Aron Rotteveel wrote:
>> Hi Roland,
>
>> Thanks for the reply. Memory usage on both machines seem fine.
>> The server has 4GB's of RAM, of which about 3GB is used during
>> the file list build and about 1.5GB is used during the actual
>> transfer. The client has 16GB of RAM with a peak usage of 8.5GB.
>
>> I just tried three transfers in a row and it consistently breaks
>> at a certain point, after which I get the "ERROR: out of memory
>> in flist_expand [sender]" error. There is not much special to
>> mention regarding the file on which it breaks: it's a 22KB JPEG
>> file with no special attributes.
>
>> The backup server is running Debian 7.8, the client runs on
>> CentOS 5.11.
>
>> A `find . | wc -l` in the backup directory results in 7434013
>> files.
>
>> -- Best regards / Met vriendelijke groet,
>
>> Aron Rotteveel
>
>> 2015-03-19 20:10 GMT+01:00 <[hidden email]
>> <mailto:[hidden email]> <mailto:[hidden email]
>> <mailto:[hidden email]>>>:
>
>> Hi Aron,
>
>> i hope it`s ok for you if i bring this back on-list. Your issue
>> and the way or possible fix to resolve it may be interesting for
>> others too (that may include future searches etc)
>
>> so with 3.1.1 we are a step further....
>
>> i don`t really have a clue what`s happening here but my next
>> step would be taking a closer look on how the memory usage of
>> rsync on the client and server grows.
>
>> you could log it like this: while true;do ps -eo
>> vsz,rss,sz,rsync|grep cron;sleep 10;done >logfile
>
>> does it grow continuously? does the oom situation reproducibly
>> happen at a certain size ? what`s the client and server
>> platform? how many files? (-> https://rsync.samba.org/FAQ.html#5
>> ! )
>
>> regards roland
>
>> *Gesendet:* Donnerstag, 19. März 2015 um 12:24 Uhr *Von:* "Aron
>> Rotteveel" <[hidden email]
>> <mailto:[hidden email]>
>> <mailto:[hidden email]
> <mailto:[hidden email]>>> *An:* [hidden email]
> <mailto:[hidden email]>
>> <mailto:[hidden email] <mailto:[hidden email]>> *Betreff:* Re:
> rsync 3.0.9 segmentation
>> fault In addition to my last message:
>
>> * Client (sender) has 16GB's or RAM, of which only 6.5GB is used
>> during peak. * I tried using --no-inc-recursive, but it does not
>> solve the issue.
>
>> What currrently is puzzling me is the question of why I am
>> receiving these errors when my server seems to have plenty of
>> memory to spare.
>
>> -- Best regards / Met vriendelijke groet,
>
>> Aron Rotteveel
>
>> 2015-03-19 11:52 GMT+01:00 Aron Rotteveel
>> <[hidden email] <mailto:[hidden email]>>:
>
>> Hi Roland,
>
>> I just upgrade both the client and host to 3.1.1 and seem to
>> memory related issues now:
>
>> ERROR: out of memory in make_file [sender] rsync error: error
>> allocating core memory buffers (code 22) at util2.c(102)
>> [sender=3.1.1] [sender] _exit_cleanup(code=22, file=util2.c,
>> line=102): about to call exit(22) [Receiver]
>> _exit_cleanup(code=22, file=io.c, line=1633): about to call
>> exit(22)
>
>
>

- --
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
        Kevin Korb                      Phone:    (407) 252-6853
        Systems Administrator           Internet:
        FutureQuest, Inc.               [hidden email]  (work)
        Orlando, Florida                [hidden email] (personal)
        Web page:                       http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlUVWZIACgkQVKC1jlbQAQfQ4wCeOjzVgtBt0t9LQ4Mf9X3kOhjF
tEcAoJAh158PF51O3Vnn8alkd7q0iSHQ
=pQeg
-----END PGP SIGNATURE-----
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync 3.0.9 segmentation fault

Kevin Korb
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Try it without any --delete options.

On 03/27/2015 09:31 AM, Aron Rotteveel wrote:

> I am now running with --delete --numeric-ids --relative but the
> problem still persists.
>
> -- Best regards / Met vriendelijke groet,
>
> Aron Rotteveel
>
> 2015-03-27 14:22 GMT+01:00 Kevin Korb <[hidden email]
> <mailto:[hidden email]>>:
>
> Try also removing --delete-excluded.  Without those two options
> there should be no reason for rsync to require gigs of RAM.  Well,
> unless the other system has rsync 2.x.
>
> On 03/27/2015 07:29 AM, Aron Rotteveel wrote:
>> Yes, I removed "--no-inc-recursive", without success.
>
>> -- Best regards / Met vriendelijke groet,
>
>> Aron Rotteveel
>
>> 2015-03-27 12:24 GMT+01:00 Kevin Korb <[hidden email]
>> <mailto:[hidden email]> <mailto:[hidden email]
>> <mailto:[hidden email]>>>:
>
>> Have you tried removing --no-inc-recursive yet?
>
>> On 03/27/2015 07:19 AM, Aron Rotteveel wrote:
>>> Hi Roland,
>
>>> Thanks for the reply. Memory usage on both machines seem fine.
>>> The server has 4GB's of RAM, of which about 3GB is used during
>>> the file list build and about 1.5GB is used during the actual
>>> transfer. The client has 16GB of RAM with a peak usage of
>>> 8.5GB.
>
>>> I just tried three transfers in a row and it consistently
>>> breaks at a certain point, after which I get the "ERROR: out of
>>> memory in flist_expand [sender]" error. There is not much
>>> special to mention regarding the file on which it breaks: it's
>>> a 22KB JPEG file with no special attributes.
>
>>> The backup server is running Debian 7.8, the client runs on
>>> CentOS 5.11.
>
>>> A `find . | wc -l` in the backup directory results in 7434013
>>> files.
>
>>> -- Best regards / Met vriendelijke groet,
>
>>> Aron Rotteveel
>
>>> 2015-03-19 20:10 GMT+01:00 <[hidden email]
>>> <mailto:[hidden email]> <mailto:[hidden email]
>>> <mailto:[hidden email]>>
> <mailto:[hidden email] <mailto:[hidden email]>
>>> <mailto:[hidden email] <mailto:[hidden email]>>>>:
>
>>> Hi Aron,
>
>>> i hope it`s ok for you if i bring this back on-list. Your
>>> issue and the way or possible fix to resolve it may be
>>> interesting for others too (that may include future searches
>>> etc)
>
>>> so with 3.1.1 we are a step further....
>
>>> i don`t really have a clue what`s happening here but my next
>>> step would be taking a closer look on how the memory usage of
>>> rsync on the client and server grows.
>
>>> you could log it like this: while true;do ps -eo
>>> vsz,rss,sz,rsync|grep cron;sleep 10;done >logfile
>
>>> does it grow continuously? does the oom situation reproducibly
>>> happen at a certain size ? what`s the client and server
>>> platform? how many files? (->
>>> https://rsync.samba.org/FAQ.html#5 ! )
>
>>> regards roland
>
>>> *Gesendet:* Donnerstag, 19. März 2015 um 12:24 Uhr *Von:*
>>> "Aron Rotteveel" <[hidden email]
>>> <mailto:[hidden email]>
>>> <mailto:[hidden email]
>>> <mailto:[hidden email]>>
>>> <mailto:[hidden email]
>>> <mailto:[hidden email]>
>> <mailto:[hidden email]
>> <mailto:[hidden email]>>>> *An:*
> [hidden email] <mailto:[hidden email]>
>> <mailto:[hidden email] <mailto:[hidden email]>>
>>> <mailto:[hidden email] <mailto:[hidden email]>
> <mailto:[hidden email] <mailto:[hidden email]>>> *Betreff:* Re:
>> rsync 3.0.9 segmentation
>>> fault In addition to my last message:
>
>>> * Client (sender) has 16GB's or RAM, of which only 6.5GB is
>>> used during peak. * I tried using --no-inc-recursive, but it
>>> does not solve the issue.
>
>>> What currrently is puzzling me is the question of why I am
>>> receiving these errors when my server seems to have plenty of
>>> memory to spare.
>
>>> -- Best regards / Met vriendelijke groet,
>
>>> Aron Rotteveel
>
>>> 2015-03-19 11:52 GMT+01:00 Aron Rotteveel
>>> <[hidden email] <mailto:[hidden email]>
> <mailto:[hidden email]
> <mailto:[hidden email]>>>:
>
>>> Hi Roland,
>
>>> I just upgrade both the client and host to 3.1.1 and seem to
>>> memory related issues now:
>
>>> ERROR: out of memory in make_file [sender] rsync error: error
>>> allocating core memory buffers (code 22) at util2.c(102)
>>> [sender=3.1.1] [sender] _exit_cleanup(code=22, file=util2.c,
>>> line=102): about to call exit(22) [Receiver]
>>> _exit_cleanup(code=22, file=io.c, line=1633): about to call
>>> exit(22)
>
>
>
>
> -- Please use reply-all for most replies to avoid omitting the
> mailing list. To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync Before posting,
> read: http://www.catb.org/~esr/faqs/smart-questions.html
>
>

- --
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
        Kevin Korb Phone:    (407) 252-6853
        Systems Administrator Internet:
        FutureQuest, Inc. [hidden email]  (work)
        Orlando, Florida [hidden email] (personal)
        Web page: http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlUVW/0ACgkQVKC1jlbQAQeKNQCgkZF+PmmIgXUCrFWGRIBgTNI0
WTIAoNfGiD91ubrz/StpXNmMxZ86gzZe
=RIfE
-----END PGP SIGNATURE-----
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync 3.0.9 segmentation fault

Aron Rotteveel
Hi Kevin,

Just did: same result.

--
Best regards / Met vriendelijke groet,

Aron Rotteveel

2015-03-27 14:32 GMT+01:00 Kevin Korb <[hidden email]>:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Try it without any --delete options.

On 03/27/2015 09:31 AM, Aron Rotteveel wrote:
> I am now running with --delete --numeric-ids --relative but the
> problem still persists.
>
> -- Best regards / Met vriendelijke groet,
>
> Aron Rotteveel
>
> 2015-03-27 14:22 GMT+01:00 Kevin Korb <[hidden email]
> <mailto:[hidden email]>>:
>
> Try also removing --delete-excluded.  Without those two options
> there should be no reason for rsync to require gigs of RAM.  Well,
> unless the other system has rsync 2.x.
>
> On 03/27/2015 07:29 AM, Aron Rotteveel wrote:
>> Yes, I removed "--no-inc-recursive", without success.
>
>> -- Best regards / Met vriendelijke groet,
>
>> Aron Rotteveel
>
>> 2015-03-27 12:24 GMT+01:00 Kevin Korb <[hidden email]
>> <mailto:[hidden email]> <mailto:[hidden email]
>> <mailto:[hidden email]>>>:
>
>> Have you tried removing --no-inc-recursive yet?
>
>> On 03/27/2015 07:19 AM, Aron Rotteveel wrote:
>>> Hi Roland,
>
>>> Thanks for the reply. Memory usage on both machines seem fine.
>>> The server has 4GB's of RAM, of which about 3GB is used during
>>> the file list build and about 1.5GB is used during the actual
>>> transfer. The client has 16GB of RAM with a peak usage of
>>> 8.5GB.
>
>>> I just tried three transfers in a row and it consistently
>>> breaks at a certain point, after which I get the "ERROR: out of
>>> memory in flist_expand [sender]" error. There is not much
>>> special to mention regarding the file on which it breaks: it's
>>> a 22KB JPEG file with no special attributes.
>
>>> The backup server is running Debian 7.8, the client runs on
>>> CentOS 5.11.
>
>>> A `find . | wc -l` in the backup directory results in 7434013
>>> files.
>
>>> -- Best regards / Met vriendelijke groet,
>
>>> Aron Rotteveel
>
>>> 2015-03-19 20:10 GMT+01:00 <[hidden email]
>>> <mailto:[hidden email]> <mailto:[hidden email]
>>> <mailto:[hidden email]>>
> <mailto:[hidden email] <mailto:[hidden email]>
>>> <mailto:[hidden email] <mailto:[hidden email]>>>>:
>
>>> Hi Aron,
>
>>> i hope it`s ok for you if i bring this back on-list. Your
>>> issue and the way or possible fix to resolve it may be
>>> interesting for others too (that may include future searches
>>> etc)
>
>>> so with 3.1.1 we are a step further....
>
>>> i don`t really have a clue what`s happening here but my next
>>> step would be taking a closer look on how the memory usage of
>>> rsync on the client and server grows.
>
>>> you could log it like this: while true;do ps -eo
>>> vsz,rss,sz,rsync|grep cron;sleep 10;done >logfile
>
>>> does it grow continuously? does the oom situation reproducibly
>>> happen at a certain size ? what`s the client and server
>>> platform? how many files? (->
>>> https://rsync.samba.org/FAQ.html#5 ! )
>
>>> regards roland
>
>>> *Gesendet:* Donnerstag, 19. März 2015 um 12:24 Uhr *Von:*
>>> "Aron Rotteveel" <[hidden email]
>>> <mailto:[hidden email]>
>>> <mailto:[hidden email]
>>> <mailto:[hidden email]>>
>>> <mailto:[hidden email]
>>> <mailto:[hidden email]>
>> <mailto:[hidden email]
>> <mailto:[hidden email]>>>> *An:*
> [hidden email] <mailto:[hidden email]>
>> <mailto:[hidden email] <mailto:[hidden email]>>
>>> <mailto:[hidden email] <mailto:[hidden email]>
> <mailto:[hidden email] <mailto:[hidden email]>>> *Betreff:* Re:
>> rsync 3.0.9 segmentation
>>> fault In addition to my last message:
>
>>> * Client (sender) has 16GB's or RAM, of which only 6.5GB is
>>> used during peak. * I tried using --no-inc-recursive, but it
>>> does not solve the issue.
>
>>> What currrently is puzzling me is the question of why I am
>>> receiving these errors when my server seems to have plenty of
>>> memory to spare.
>
>>> -- Best regards / Met vriendelijke groet,
>
>>> Aron Rotteveel
>
>>> 2015-03-19 11:52 GMT+01:00 Aron Rotteveel
>>> <[hidden email] <mailto:[hidden email]>
> <mailto:[hidden email]
> <mailto:[hidden email]>>>:
>
>>> Hi Roland,
>
>>> I just upgrade both the client and host to 3.1.1 and seem to
>>> memory related issues now:
>
>>> ERROR: out of memory in make_file [sender] rsync error: error
>>> allocating core memory buffers (code 22) at util2.c(102)
>>> [sender=3.1.1] [sender] _exit_cleanup(code=22, file=util2.c,
>>> line=102): about to call exit(22) [Receiver]
>>> _exit_cleanup(code=22, file=io.c, line=1633): about to call
>>> exit(22)
>
>
>
>
> -- Please use reply-all for most replies to avoid omitting the
> mailing list. To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync Before posting,
> read: http://www.catb.org/~esr/faqs/smart-questions.html
>
>

- --
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
        Kevin Korb                      Phone:    (407) 252-6853
        Systems Administrator           Internet:
        FutureQuest, Inc.               [hidden email]  (work)
        Orlando, Florida                [hidden email] (personal)
        Web page:                       http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlUVW/0ACgkQVKC1jlbQAQeKNQCgkZF+PmmIgXUCrFWGRIBgTNI0
WTIAoNfGiD91ubrz/StpXNmMxZ86gzZe
=RIfE
-----END PGP SIGNATURE-----
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync 3.0.9 segmentation fault

Aron Rotteveel
Anyone have any other ideas I could try to debug this issue? :)

--
Best regards / Met vriendelijke groet,

Aron Rotteveel

2015-03-27 16:02 GMT+01:00 Aron Rotteveel <[hidden email]>:
Hi Kevin,

Just did: same result.

--
Best regards / Met vriendelijke groet,

Aron Rotteveel

2015-03-27 14:32 GMT+01:00 Kevin Korb <[hidden email]>:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Try it without any --delete options.

On 03/27/2015 09:31 AM, Aron Rotteveel wrote:
> I am now running with --delete --numeric-ids --relative but the
> problem still persists.
>
> -- Best regards / Met vriendelijke groet,
>
> Aron Rotteveel
>
> 2015-03-27 14:22 GMT+01:00 Kevin Korb <[hidden email]
> <mailto:[hidden email]>>:
>
> Try also removing --delete-excluded.  Without those two options
> there should be no reason for rsync to require gigs of RAM.  Well,
> unless the other system has rsync 2.x.
>
> On 03/27/2015 07:29 AM, Aron Rotteveel wrote:
>> Yes, I removed "--no-inc-recursive", without success.
>
>> -- Best regards / Met vriendelijke groet,
>
>> Aron Rotteveel
>
>> 2015-03-27 12:24 GMT+01:00 Kevin Korb <[hidden email]
>> <mailto:[hidden email]> <mailto:[hidden email]
>> <mailto:[hidden email]>>>:
>
>> Have you tried removing --no-inc-recursive yet?
>
>> On 03/27/2015 07:19 AM, Aron Rotteveel wrote:
>>> Hi Roland,
>
>>> Thanks for the reply. Memory usage on both machines seem fine.
>>> The server has 4GB's of RAM, of which about 3GB is used during
>>> the file list build and about 1.5GB is used during the actual
>>> transfer. The client has 16GB of RAM with a peak usage of
>>> 8.5GB.
>
>>> I just tried three transfers in a row and it consistently
>>> breaks at a certain point, after which I get the "ERROR: out of
>>> memory in flist_expand [sender]" error. There is not much
>>> special to mention regarding the file on which it breaks: it's
>>> a 22KB JPEG file with no special attributes.
>
>>> The backup server is running Debian 7.8, the client runs on
>>> CentOS 5.11.
>
>>> A `find . | wc -l` in the backup directory results in 7434013
>>> files.
>
>>> -- Best regards / Met vriendelijke groet,
>
>>> Aron Rotteveel
>
>>> 2015-03-19 20:10 GMT+01:00 <[hidden email]
>>> <mailto:[hidden email]> <mailto:[hidden email]
>>> <mailto:[hidden email]>>
> <mailto:[hidden email] <mailto:[hidden email]>
>>> <mailto:[hidden email] <mailto:[hidden email]>>>>:
>
>>> Hi Aron,
>
>>> i hope it`s ok for you if i bring this back on-list. Your
>>> issue and the way or possible fix to resolve it may be
>>> interesting for others too (that may include future searches
>>> etc)
>
>>> so with 3.1.1 we are a step further....
>
>>> i don`t really have a clue what`s happening here but my next
>>> step would be taking a closer look on how the memory usage of
>>> rsync on the client and server grows.
>
>>> you could log it like this: while true;do ps -eo
>>> vsz,rss,sz,rsync|grep cron;sleep 10;done >logfile
>
>>> does it grow continuously? does the oom situation reproducibly
>>> happen at a certain size ? what`s the client and server
>>> platform? how many files? (->
>>> https://rsync.samba.org/FAQ.html#5 ! )
>
>>> regards roland
>
>>> *Gesendet:* Donnerstag, 19. März 2015 um 12:24 Uhr *Von:*
>>> "Aron Rotteveel" <[hidden email]
>>> <mailto:[hidden email]>
>>> <mailto:[hidden email]
>>> <mailto:[hidden email]>>
>>> <mailto:[hidden email]
>>> <mailto:[hidden email]>
>> <mailto:[hidden email]
>> <mailto:[hidden email]>>>> *An:*
> [hidden email] <mailto:[hidden email]>
>> <mailto:[hidden email] <mailto:[hidden email]>>
>>> <mailto:[hidden email] <mailto:[hidden email]>
> <mailto:[hidden email] <mailto:[hidden email]>>> *Betreff:* Re:
>> rsync 3.0.9 segmentation
>>> fault In addition to my last message:
>
>>> * Client (sender) has 16GB's or RAM, of which only 6.5GB is
>>> used during peak. * I tried using --no-inc-recursive, but it
>>> does not solve the issue.
>
>>> What currrently is puzzling me is the question of why I am
>>> receiving these errors when my server seems to have plenty of
>>> memory to spare.
>
>>> -- Best regards / Met vriendelijke groet,
>
>>> Aron Rotteveel
>
>>> 2015-03-19 11:52 GMT+01:00 Aron Rotteveel
>>> <[hidden email] <mailto:[hidden email]>
> <mailto:[hidden email]
> <mailto:[hidden email]>>>:
>
>>> Hi Roland,
>
>>> I just upgrade both the client and host to 3.1.1 and seem to
>>> memory related issues now:
>
>>> ERROR: out of memory in make_file [sender] rsync error: error
>>> allocating core memory buffers (code 22) at util2.c(102)
>>> [sender=3.1.1] [sender] _exit_cleanup(code=22, file=util2.c,
>>> line=102): about to call exit(22) [Receiver]
>>> _exit_cleanup(code=22, file=io.c, line=1633): about to call
>>> exit(22)
>
>
>
>
> -- Please use reply-all for most replies to avoid omitting the
> mailing list. To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync Before posting,
> read: http://www.catb.org/~esr/faqs/smart-questions.html
>
>

- --
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
        Kevin Korb                      Phone:    (407) 252-6853
        Systems Administrator           Internet:
        FutureQuest, Inc.               [hidden email]  (work)
        Orlando, Florida                [hidden email] (personal)
        Web page:                       http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlUVW/0ACgkQVKC1jlbQAQeKNQCgkZF+PmmIgXUCrFWGRIBgTNI0
WTIAoNfGiD91ubrz/StpXNmMxZ86gzZe
=RIfE
-----END PGP SIGNATURE-----
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync 3.0.9 segmentation fault

Paul Slootman-5
On Tue 07 Apr 2015, Aron Rotteveel wrote:

> Anyone have any other ideas I could try to debug this issue? :)

You could try using the latest version of rsync? 3.1.1 was released some
time ago.


Paul
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync 3.0.9 segmentation fault

Aron Rotteveel

Both server and client are running 3.1.1 now but the problem still persists.

Op 7 apr. 2015 10:35 AM schreef "Paul Slootman" <[hidden email]>:
On Tue 07 Apr 2015, Aron Rotteveel wrote:

> Anyone have any other ideas I could try to debug this issue? :)

You could try using the latest version of rsync? 3.1.1 was released some
time ago.


Paul
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html