rsync buffer overflow detected

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

rsync buffer overflow detected

Samba - rsync mailing list
Hello!

I use rsync from python on my Debian Jessie amd64 and get this error:
*** buffer overflow detected ***: <snip>/rsync terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x731af)[0x7ffff78971af]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7ffff791caa7]
/lib/x86_64-linux-gnu/libc.so.6(+0xf6cc0)[0x7ffff791acc0]
/lib/x86_64-linux-gnu/libc.so.6(+0xf8a17)[0x7ffff791ca17]
<snip>/rsync(+0x30c78)[0x555555584c78]
<snip>/rsync(+0x31cfe)[0x555555585cfe]
<snip>/rsync(+0x31ef6)[0x555555585ef6]
<snip>/rsync(+0x336ed)[0x5555555876ed]
<snip>/rsync(+0x22417)[0x555555576417]
<snip>/rsync(+0x2395e)[0x55555557795e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7ffff7845b45]
<snip>/rsync(+0x7f89)[0x55555555bf89]

I guess that problem is about too many open fds
STR for this is a small script on python:
import os
import subprocess

F = 'test'
OPENS = 1600

cmd = [
    #'gdb', '--args',
    './rsync',
    '-aviH',
    '/etc/passwd',
    '/tmp/passwd'
]

for i in xrange(OPENS):
    fd = os.open(F, os.O_WRONLY | os.O_CREAT)
print(cmd)
subprocess.check_call(cmd)

I rebuild rsync-3.1.1 from Debian source with debug and -O1 and get bt from gdb:
(gdb) bt
#0  0x00007ffff7859067 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff785a448 in __GI_abort () at abort.c:89
#2  0x00007ffff78971b4 in __libc_message (do_abort=do_abort@entry=2,
fmt=fmt@entry=0x7ffff7989cb3 "*** %s ***: %s terminated\n")
    at ../sysdeps/posix/libc_fatal.c:175
#3  0x00007ffff791caa7 in __GI___fortify_fail
(msg=msg@entry=0x7ffff7989c4a "buffer overflow detected") at
fortify_fail.c:31
#4  0x00007ffff791acc0 in __GI___chk_fail () at chk_fail.c:28
#5  0x00007ffff791ca17 in __fdelt_chk (d=d@entry=1606) at fdelt_chk.c:25
#6  0x0000555555584c78 in safe_read (fd=fd@entry=1606,
buf=buf@entry=0x7fffffffa810 "\037", len=len@entry=4) at io.c:245
#7  0x0000555555585cfe in read_buf (f=f@entry=1606,
buf=buf@entry=0x7fffffffa810 "\037", len=len@entry=4) at io.c:1815
#8  0x0000555555585ef6 in read_int (f=f@entry=1606) at io.c:1711
#9  0x00005555555876ed in setup_protocol (f_out=1605, f_in=1606) at compat.c:158
#10 0x0000555555576417 in client_run (f_in=1606, f_out=1605,
pid=24793, argc=1, argv=0x5555557d5240) at main.c:1128
#11 0x000055555557795e in start_client (argv=0x5555557d5240, argc=1)
at main.c:1423
#12 main (argc=2, argv=0x5555557d5240) at main.c:1651

It looks like a bug, but I'm not sure)

--
Boris

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Aw: rsync buffer overflow detected

Samba - rsync mailing list
What's the value of "i" when this happens and what are the system ulimit values for the user running that?

Roland



> Gesendet: Freitag, 14. April 2017 um 19:22 Uhr
> Von: "Boris Savelev via rsync" <[hidden email]>
> An: [hidden email]
> Betreff: rsync buffer overflow detected
>
> Hello!
>
> I use rsync from python on my Debian Jessie amd64 and get this error:
> *** buffer overflow detected ***: <snip>/rsync terminated
> ======= Backtrace: =========
> /lib/x86_64-linux-gnu/libc.so.6(+0x731af)[0x7ffff78971af]
> /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7ffff791caa7]
> /lib/x86_64-linux-gnu/libc.so.6(+0xf6cc0)[0x7ffff791acc0]
> /lib/x86_64-linux-gnu/libc.so.6(+0xf8a17)[0x7ffff791ca17]
> <snip>/rsync(+0x30c78)[0x555555584c78]
> <snip>/rsync(+0x31cfe)[0x555555585cfe]
> <snip>/rsync(+0x31ef6)[0x555555585ef6]
> <snip>/rsync(+0x336ed)[0x5555555876ed]
> <snip>/rsync(+0x22417)[0x555555576417]
> <snip>/rsync(+0x2395e)[0x55555557795e]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7ffff7845b45]
> <snip>/rsync(+0x7f89)[0x55555555bf89]
>
> I guess that problem is about too many open fds
> STR for this is a small script on python:
> import os
> import subprocess
>
> F = 'test'
> OPENS = 1600
>
> cmd = [
>     #'gdb', '--args',
>     './rsync',
>     '-aviH',
>     '/etc/passwd',
>     '/tmp/passwd'
> ]
>
> for i in xrange(OPENS):
>     fd = os.open(F, os.O_WRONLY | os.O_CREAT)
> print(cmd)
> subprocess.check_call(cmd)
>
> I rebuild rsync-3.1.1 from Debian source with debug and -O1 and get bt from gdb:
> (gdb) bt
> #0  0x00007ffff7859067 in __GI_raise (sig=sig@entry=6) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> #1  0x00007ffff785a448 in __GI_abort () at abort.c:89
> #2  0x00007ffff78971b4 in __libc_message (do_abort=do_abort@entry=2,
> fmt=fmt@entry=0x7ffff7989cb3 "*** %s ***: %s terminated\n")
>     at ../sysdeps/posix/libc_fatal.c:175
> #3  0x00007ffff791caa7 in __GI___fortify_fail
> (msg=msg@entry=0x7ffff7989c4a "buffer overflow detected") at
> fortify_fail.c:31
> #4  0x00007ffff791acc0 in __GI___chk_fail () at chk_fail.c:28
> #5  0x00007ffff791ca17 in __fdelt_chk (d=d@entry=1606) at fdelt_chk.c:25
> #6  0x0000555555584c78 in safe_read (fd=fd@entry=1606,
> buf=buf@entry=0x7fffffffa810 "\037", len=len@entry=4) at io.c:245
> #7  0x0000555555585cfe in read_buf (f=f@entry=1606,
> buf=buf@entry=0x7fffffffa810 "\037", len=len@entry=4) at io.c:1815
> #8  0x0000555555585ef6 in read_int (f=f@entry=1606) at io.c:1711
> #9  0x00005555555876ed in setup_protocol (f_out=1605, f_in=1606) at compat.c:158
> #10 0x0000555555576417 in client_run (f_in=1606, f_out=1605,
> pid=24793, argc=1, argv=0x5555557d5240) at main.c:1128
> #11 0x000055555557795e in start_client (argv=0x5555557d5240, argc=1)
> at main.c:1423
> #12 main (argc=2, argv=0x5555557d5240) at main.c:1651
>
> It looks like a bug, but I'm not sure)
>
> --
> Boris
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync buffer overflow detected

Samba - rsync mailing list
ulimit:
time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        8192
coredump(blocks)     0
memory(kbytes)       unlimited
locked memory(kbytes) 64
process              64098
nofiles              65536
vmemory(kbytes)      unlimited
locks                unlimited


Rsync call at the end, so i = 1600

modified test script:

---8<---
import os
import subprocess
#import psutil

F = 'test'
OPENS = 1600

cmd = [
    #'gdb', '--args',
    #'strace', '-f', '-olog',
    './rsync',
    '-aviH',
    '/etc/passwd',
    '/tmp/passwd'
]

for i in xrange(OPENS):
    fd = os.open(F, os.O_WRONLY | os.O_CREAT)

os.system('ulimit -a')
#proc = psutil.Process()
#print len(proc.open_files())
subprocess.check_call(cmd)
---8<---

strace:
https://pastebin.com/Xu9Pq4R9

If system limit excided it may fall with EMFILE, I think

2017-04-16 9:43 GMT+03:00  <[hidden email]>:

> What's the value of "i" when this happens and what are the system ulimit values for the user running that?
>
> Roland
>
>
>
>> Gesendet: Freitag, 14. April 2017 um 19:22 Uhr
>> Von: "Boris Savelev via rsync" <[hidden email]>
>> An: [hidden email]
>> Betreff: rsync buffer overflow detected
>>
>> Hello!
>>
>> I use rsync from python on my Debian Jessie amd64 and get this error:
>> *** buffer overflow detected ***: <snip>/rsync terminated
>> ======= Backtrace: =========
>> /lib/x86_64-linux-gnu/libc.so.6(+0x731af)[0x7ffff78971af]
>> /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7ffff791caa7]
>> /lib/x86_64-linux-gnu/libc.so.6(+0xf6cc0)[0x7ffff791acc0]
>> /lib/x86_64-linux-gnu/libc.so.6(+0xf8a17)[0x7ffff791ca17]
>> <snip>/rsync(+0x30c78)[0x555555584c78]
>> <snip>/rsync(+0x31cfe)[0x555555585cfe]
>> <snip>/rsync(+0x31ef6)[0x555555585ef6]
>> <snip>/rsync(+0x336ed)[0x5555555876ed]
>> <snip>/rsync(+0x22417)[0x555555576417]
>> <snip>/rsync(+0x2395e)[0x55555557795e]
>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7ffff7845b45]
>> <snip>/rsync(+0x7f89)[0x55555555bf89]
>>
>> I guess that problem is about too many open fds
>> STR for this is a small script on python:
>> import os
>> import subprocess
>>
>> F = 'test'
>> OPENS = 1600
>>
>> cmd = [
>>     #'gdb', '--args',
>>     './rsync',
>>     '-aviH',
>>     '/etc/passwd',
>>     '/tmp/passwd'
>> ]
>>
>> for i in xrange(OPENS):
>>     fd = os.open(F, os.O_WRONLY | os.O_CREAT)
>> print(cmd)
>> subprocess.check_call(cmd)
>>
>> I rebuild rsync-3.1.1 from Debian source with debug and -O1 and get bt from gdb:
>> (gdb) bt
>> #0  0x00007ffff7859067 in __GI_raise (sig=sig@entry=6) at
>> ../nptl/sysdeps/unix/sysv/linux/raise.c:56
>> #1  0x00007ffff785a448 in __GI_abort () at abort.c:89
>> #2  0x00007ffff78971b4 in __libc_message (do_abort=do_abort@entry=2,
>> fmt=fmt@entry=0x7ffff7989cb3 "*** %s ***: %s terminated\n")
>>     at ../sysdeps/posix/libc_fatal.c:175
>> #3  0x00007ffff791caa7 in __GI___fortify_fail
>> (msg=msg@entry=0x7ffff7989c4a "buffer overflow detected") at
>> fortify_fail.c:31
>> #4  0x00007ffff791acc0 in __GI___chk_fail () at chk_fail.c:28
>> #5  0x00007ffff791ca17 in __fdelt_chk (d=d@entry=1606) at fdelt_chk.c:25
>> #6  0x0000555555584c78 in safe_read (fd=fd@entry=1606,
>> buf=buf@entry=0x7fffffffa810 "\037", len=len@entry=4) at io.c:245
>> #7  0x0000555555585cfe in read_buf (f=f@entry=1606,
>> buf=buf@entry=0x7fffffffa810 "\037", len=len@entry=4) at io.c:1815
>> #8  0x0000555555585ef6 in read_int (f=f@entry=1606) at io.c:1711
>> #9  0x00005555555876ed in setup_protocol (f_out=1605, f_in=1606) at compat.c:158
>> #10 0x0000555555576417 in client_run (f_in=1606, f_out=1605,
>> pid=24793, argc=1, argv=0x5555557d5240) at main.c:1128
>> #11 0x000055555557795e in start_client (argv=0x5555557d5240, argc=1)
>> at main.c:1423
>> #12 main (argc=2, argv=0x5555557d5240) at main.c:1651
>>
>> It looks like a bug, but I'm not sure)
>>
>> --
>> Boris
>>
>> --
>> Please use reply-all for most replies to avoid omitting the mailing list.
>> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
>> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>>



--
Boris

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync buffer overflow detected

Samba - rsync mailing list
In reply to this post by Samba - rsync mailing list
On Fri, Apr 14, 2017 at 10:22 AM, Boris Savelev via rsync <[hidden email]> wrote:
#8  0x0000555555585ef6 in read_int (f=f@entry=1606) at io.c:1711
#9  0x00005555555876ed in setup_protocol (f_out=1605, f_in=1606) at compat.c:158

Since rsync is just trying to setup the protocol, I'd imagine that your shell is outputting extraneous characters. You can try running something manually, such as "ssh host echo hello", which should output only "hello", and "ssh host rsync --server . ." (note the 2 trailing dot args) which should output nothing, and require a Ctrl-C to kill it.  You might be able to work around the issue with something like -e'ssh -q'.

..wayne..

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync buffer overflow detected

Samba - rsync mailing list
In reply to this post by Samba - rsync mailing list
On Fri, Apr 14, 2017 at 08:22:29PM +0300, Boris Savelev via rsync wrote:
> I use rsync from python on my Debian Jessie amd64 and get this error:
> *** buffer overflow detected ***: <snip>/rsync terminated

> I rebuild rsync-3.1.1 from Debian source with debug and -O1 and get bt from gdb:
> (gdb) bt

> #5  0x00007ffff791ca17 in __fdelt_chk (d=d@entry=1606) at fdelt_chk.c:25
> #6  0x0000555555584c78 in safe_read (fd=fd@entry=1606, buf=buf@entry=0x7fffffffa810 "\037", len=len@entry=4) at io.c:245

That is FD_SET(fd, &r_fds); with fd >= FD_SETSIZE, which is 1024.
You cannot use select with file descriptor numbers >= FD_SETSIZE (or < 0),
and glibc is catching that.

The "buffer" that would overflow is the fd_set.

Maybe rsync could simply close all inherited file descriptors,
first things first, before it does anything else,
possibly after making sure fds 0,1,2 are open to somewhere,
to avoid any output to "supposedly" stdout/stderr to clobber
fds opened only later.  Similar to what lvm tools do in their
_check_standard_fds() and _close_stray_fds()?

But of course rsync could also say: not my problem, *you* (whatever
entity was spawning rsync) leaked file descriptors, learn to use
O_CLOEXEC resp. set FD_CLOEXEC, so only 0,1,2 will be inherited.

quick and dirty workaround:
use a wrapper script, close all fds >= 3 "just in case",
then exec rsync.

> It looks like a bug, but I'm not sure)

Thanks,

    Lars Ellenberg


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html