Is it possible to transfer a large, dynamic file in a piecemeal fashion?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Is it possible to transfer a large, dynamic file in a piecemeal fashion?

Samba - rsync mailing list
Greetings,
 
Is it possible to use rsync to transmit a large, dynamic 2TB file in a
piecemeal fashion during daylight hours over the course of a dozen days?
On a good day, about 200GB of data can be transferred before rsync times
out to enable a nightly local backup to complete. The local backup
changes a few 512 byte blocks in the 2TB file. That's why the 2TB file
is dynamic.

As an experiment, rsync was started on the client to sync the 2TB file.
At that point, rsyncd on the backup server created a temporarily file
with the 2TB file name appended with a dot followed by six characters.

After about 100MB was transferred the rsync client was interrupted and
stopped in order to simulate the nightly shutdown. When rsync was
restarted it erased the original 100MB temporary file and started over
with a new, empty temporarily file.

In summary:

* rsync apparently erases temporarily files upon restart

* it takes about a dozen days to transfer the 2TB file

Unless there's an option to have rsync pick up where it left off it
seems impossible to transmit a large file in a piecemeal fashion.
Because the (nearly) same 200GB at the beginning of the 2TB file gets
retransmitted over and over with each restart.

Thank you,

--
Don Kuenz KB7RPU

Genius is nothing but a greater aptitude for patience. - de Buffon


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: Is it possible to transfer a large, dynamic file in a piecemeal fashion?

Samba - rsync mailing list
On Mon, Sep 04, 2017 at 10:45:26PM +0000, Don Kuenz via rsync wrote:

> Greetings,
>  
> Is it possible to use rsync to transmit a large, dynamic 2TB file in a
> piecemeal fashion during daylight hours over the course of a dozen days?
> On a good day, about 200GB of data can be transferred before rsync times
> out to enable a nightly local backup to complete. The local backup
> changes a few 512 byte blocks in the 2TB file. That's why the 2TB file
> is dynamic.
>
> As an experiment, rsync was started on the client to sync the 2TB file.
> At that point, rsyncd on the backup server created a temporarily file
> with the 2TB file name appended with a dot followed by six characters.
>
> After about 100MB was transferred the rsync client was interrupted and
> stopped in order to simulate the nightly shutdown. When rsync was
> restarted it erased the original 100MB temporary file and started over
> with a new, empty temporarily file.
>
> In summary:
>
> * rsync apparently erases temporarily files upon restart
>
> * it takes about a dozen days to transfer the 2TB file
>
> Unless there's an option to have rsync pick up where it left off it
> seems impossible to transmit a large file in a piecemeal fashion.
> Because the (nearly) same 200GB at the beginning of the 2TB file gets
> retransmitted over and over with each restart.

Would --partial work for you?
Possibly combine with --partial-dir, --temp-dir,
--delay-updates, and/or --inplace.

--
: Lars Ellenberg
: LINBIT | Keeping the Digital World Running
: DRBD -- Heartbeat -- Corosync -- Pacemaker
: R&D, Integration, Ops, Consulting, Support

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html