Quantcast

Rsync Auto Resume

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Rsync Auto Resume

Shane Uys

I am trying to copy a 5GB file over the internet with two Win2003 servers. The internet on ServerA occasionally drops the connection and rsync terminates with error “connection reset by peer”.  Is there a way to keep alive, extend timeout or auto resume transfer? I tried using the --partial but it just starts over with another partial. I tried using the --timeout=1000 but still terminates the process after 60s. I have installed cwrsync which uses rsync 3.03. Here is the current command.

 

rsync -e "ssh" file1.x user@server:


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Rsync Auto Resume

Matt McCutchen-7
On Wed, 2008-08-13 at 13:28 -0500, Shane Uys wrote:
> I am trying to copy a 5GB file over the internet with two Win2003
> servers. The internet on ServerA occasionally drops the connection and
> rsync terminates with error “connection reset by peer”.  Is there a
> way to keep alive, extend timeout or auto resume transfer? I tried
> using the --partial but it just starts over with another partial.

--partial does allow the receiver to use the already transmitted data in
the delta-transfer algorithm on the next run, avoiding the need to send
it over the network again, but I guess the disk I/O of rewriting that
data to the new temporary file is the problem.  Try --partial --append;
the --append option tells rsync to make the (correct) assumption that
the destination file left by --partial matches the beginning of the
source file and just append to the existing file.

Matt

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

signature.asc (204 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Rsync Auto Resume

Shane Uys
That works really well. After retesting, the --partial by itself also works. I must have overlooked it the first time. Is there any option for auto resume? I tried --timeout=1000 but it still terminates after 60s (when the internet drops). Thanks


-----Original Message-----
From: Matt McCutchen [mailto:[hidden email]]
Sent: Wednesday, August 13, 2008 5:09 PM
To: Shane Uys
Cc: [hidden email]
Subject: Re: Rsync Auto Resume

On Wed, 2008-08-13 at 13:28 -0500, Shane Uys wrote:
> I am trying to copy a 5GB file over the internet with two Win2003
> servers. The internet on ServerA occasionally drops the connection and
> rsync terminates with error “connection reset by peer”.  Is there a
> way to keep alive, extend timeout or auto resume transfer? I tried
> using the --partial but it just starts over with another partial.

--partial does allow the receiver to use the already transmitted data in the delta-transfer algorithm on the next run, avoiding the need to send it over the network again, but I guess the disk I/O of rewriting that data to the new temporary file is the problem.  Try --partial --append; the --append option tells rsync to make the (correct) assumption that the destination file left by --partial matches the beginning of the source file and just append to the existing file.

Matt

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Rsync Auto Resume

Matt McCutchen-7
On Wed, 2008-08-13 at 23:53 -0500, Shane Uys wrote:
> Is there any option for auto resume? I tried --timeout=1000 but it
> still terminates after 60s (when the internet drops).

No.  If the OS tells rsync that its network connection has broken, there
is no way rsync can reestablish the connection, so it exits.  If you
want to stop the connection from breaking, you might need to raise a TCP
timeout (I'm not familiar with this) or tunnel rsync through another
tool that maintains a virtual connection across network disruptions.
The rsync timeout is designed to make rsync exit when the connection (or
the remote process) becomes unresponsive, even if the OS doesn't
officially break the connection; it has nothing to do with preventing
the connection from breaking.

Another option would be to write a shell loop that runs rsync repeatedly
until it exits other than with a "connection reset by peer" error.

Matt

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

signature.asc (204 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Rsync Auto Resume

Shane Uys
I don't have programming experience except for a little in DOS commands.
Is there something simple you have in mind for a shell loop? Thanks.

-----Original Message-----
From: Matt McCutchen [mailto:[hidden email]]
Sent: Thursday, August 14, 2008 10:41 AM
To: Shane Uys
Cc: [hidden email]
Subject: RE: Rsync Auto Resume

On Wed, 2008-08-13 at 23:53 -0500, Shane Uys wrote:
> Is there any option for auto resume? I tried --timeout=1000 but it
> still terminates after 60s (when the internet drops).

No.  If the OS tells rsync that its network connection has broken, there
is no way rsync can reestablish the connection, so it exits.  If you
want to stop the connection from breaking, you might need to raise a TCP
timeout (I'm not familiar with this) or tunnel rsync through another
tool that maintains a virtual connection across network disruptions.
The rsync timeout is designed to make rsync exit when the connection (or
the remote process) becomes unresponsive, even if the OS doesn't
officially break the connection; it has nothing to do with preventing
the connection from breaking.

Another option would be to write a shell loop that runs rsync repeatedly
until it exits other than with a "connection reset by peer" error.

Matt
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Loading...