[Bug 10312] New: Rsync times out during deletion on big folders

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[Bug 10312] New: Rsync times out during deletion on big folders

samba-bugs
https://bugzilla.samba.org/show_bug.cgi?id=10312

           Summary: Rsync times out during deletion on big folders
           Product: rsync
           Version: 3.0.9
          Platform: x64
        OS/Version: Linux
            Status: NEW
          Severity: normal
          Priority: P5
         Component: core
        AssignedTo: [hidden email]
        ReportedBy: [hidden email]
         QAContact: [hidden email]


Hello,

I am using rsync to run daily backups on many circumstances and it always works
like a charm, so, first of all, thank you for creating this indispensable tool.

Unfortunately, it looks like I've found a bug: when I use rsync on a very big
folder with the delete and timeout options, it times out during deletion even
if the process is running normally, giving the following error:

rsync error: timeout in data send/receive (code 30) at io.c(137)
[receiver=3.0.9]
rsync: connection unexpectedly closed (9 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605)
[sender=3.0.9]
 -1

Steps to reproduce:
1. Create a destination folder with about 10.000 files (more if your PC is
faster) so that deletion of all the files will require more --timeout to
complete
2. Rsync from an empty folder to your destination folder using the --delete and
--timeout options

What happens:
Deletion of the files starts and goes on quite fast, until it unexpectedly ends
with the error above.

What I would expect to happen:
Rsync deleting all files without timing out.

More info:
This is just a guess: maybe the timeout counter is not reset during deletion?

My test command line:
ionice -c 3 nice -n 19 rsync -v -a -x --timeout=300 --delete /tmp/test
root@host:/mnt/slow/test

--
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

[Bug 10312] Rsync times out during deletion on big folders

samba-bugs
https://bugzilla.samba.org/show_bug.cgi?id=10312

--- Comment #1 from Ben RUBSON <[hidden email]> 2013-12-19 13:54:52 UTC ---
Hello,

I encounter same issue, I have sometimes a huge amount of files to delete.
Rsync daemon can take a lot of time to delete them, leading Rsync client into a
timeout...

Another similar situation encountered is when the daemon is hard-linking a very
huge amount of files (when the client uses --link-dest) and takes a very long
time to perform this task.

According to Wayne : both sides of the transfer try to periodically send a
keep-alive message if too much time has elapsed since some data was
transferred.  However, Rsync does not always check this enough in some
circumstances. Version 3.1.0 has an improved keep-alive setup that may help.

Note that Wayne has improved keepalive process even after 3.1.0 release :
https://www.mail-archive.com/rsync@.../msg29431.html

I have not tested it yet, but I plan to do it to know whether or not it
resolves timeout problems during deletion and hard-linking.

Ben

--
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

[Bug 10312] Rsync times out during deletion on big folders

samba-bugs
In reply to this post by samba-bugs
https://bugzilla.samba.org/show_bug.cgi?id=10312

--- Comment #2 from Ben RUBSON <[hidden email]> ---
Regarding --link-dest, I made further testing with rsync daemon 3.1.2.
I added a usleep(5000) at the beginning of try_dests_reg() in generator.c, so
that hard-linking all the files takes more than 2 minutes instead of some
seconds only.
Client, with a very low --timeout=10, does not timeout on this.
So it's OK, no more issue with --link-dest !

Same experience with --delete (--delete-after in my tests), adding some
usleep() so that deletion takes around 4 minutes did not make client with
--timeout=10 to abort.

Perfect !

--
You are receiving this mail because:
You are the QA Contact for the bug.

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html