Re: Is it possible to transfer a large, dynamic file in a piecemeal fashion? Yes it is.

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Is it possible to transfer a large, dynamic file in a piecemeal fashion? Yes it is.

Samba - rsync mailing list
Subject: Re: Is it possible to transfer a large, dynamic file in a piecemeal fashion? Yes it is.

Don Kuenz via rsync <[hidden email]> wrote:

> Greetings,
>
> Is it possible to use rsync to transmit a large, dynamic 2TB file in a
> piecemeal fashion during daylight hours over the course of a dozen days?
> On a good day, about 200GB of data can be transferred before rsync times
> out to enable a nightly local backup to complete. The local backup
> changes a few 512 byte blocks in the 2TB file. That's why the 2TB file
> is dynamic.
>
> As an experiment, rsync was started on the client to sync the 2TB file.
> At that point, rsyncd on the backup server created a temporarily file
> with the 2TB file name appended with a dot followed by six characters.
>
> After about 100MB was transferred the rsync client was interrupted and
> stopped in order to simulate the nightly shutdown. When rsync was
> restarted it erased the original 100MB temporary file and started over
> with a new, empty temporarily file.
>
> In summary:
>
> * rsync apparently erases temporarily files upon restart
>
> * it takes about a dozen days to transfer the 2TB file
>
> Unless there's an option to have rsync pick up where it left off it
> seems impossible to transmit a large file in a piecemeal fashion.
> Because the (nearly) same 200GB at the beginning of the 2TB file gets
> retransmitted over and over with each restart.

The rsync --partial argument retains the partial file between an
interruption and a restart. It works for --timelimit interruptions as
well as ctrl-c interruptions on the client side.    

Thank you,

--
Don Kuenz KB7RPU

Genius is nothing but a greater aptitude for patience. - de Buffon


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: Is it possible to transfer a large, dynamic file in a piecemeal fashion? Yes it is.

Samba - rsync mailing list
You probably want --inplace for this rather than partial.  The upside of
--partial is that the existing file isn't replaced until there is a
complete file to replace it with.  The downside is building up that
entire new file.

On 09/05/2017 12:01 AM, Don Kuenz via rsync wrote:

> Subject: Re: Is it possible to transfer a large, dynamic file in a piecemeal fashion? Yes it is.
>
> Don Kuenz via rsync <[hidden email]> wrote:
>> Greetings,
>>
>> Is it possible to use rsync to transmit a large, dynamic 2TB file in a
>> piecemeal fashion during daylight hours over the course of a dozen days?
>> On a good day, about 200GB of data can be transferred before rsync times
>> out to enable a nightly local backup to complete. The local backup
>> changes a few 512 byte blocks in the 2TB file. That's why the 2TB file
>> is dynamic.
>>
>> As an experiment, rsync was started on the client to sync the 2TB file.
>> At that point, rsyncd on the backup server created a temporarily file
>> with the 2TB file name appended with a dot followed by six characters.
>>
>> After about 100MB was transferred the rsync client was interrupted and
>> stopped in order to simulate the nightly shutdown. When rsync was
>> restarted it erased the original 100MB temporary file and started over
>> with a new, empty temporarily file.
>>
>> In summary:
>>
>> * rsync apparently erases temporarily files upon restart
>>
>> * it takes about a dozen days to transfer the 2TB file
>>
>> Unless there's an option to have rsync pick up where it left off it
>> seems impossible to transmit a large file in a piecemeal fashion.
>> Because the (nearly) same 200GB at the beginning of the 2TB file gets
>> retransmitted over and over with each restart.
>
> The rsync --partial argument retains the partial file between an
> interruption and a restart. It works for --timelimit interruptions as
> well as ctrl-c interruptions on the client side.    
>
> Thank you,
>
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
        Kevin Korb Phone:    (407) 252-6853
        Systems Administrator Internet:
        FutureQuest, Inc. [hidden email]  (work)
        Orlando, Florida [hidden email] (personal)
        Web page: http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

signature.asc (231 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Is it possible to transfer a large, dynamic file in a piecemeal fashion? Yes it is.

Samba - rsync mailing list
In reply to this post by Samba - rsync mailing list
Kevin Korb via rsync <[hidden email]> wrote:

> [-- multipart/signed, encoding 7bit, 98 lines --]
>
>    [-- multipart/mixed, encoding 7bit, 72 lines --]
>
>        [-- text/plain, encoding quoted-printable, charset: utf-8, 65 lines --]
>
> You probably want --inplace for this rather than partial.  The upside of
> --partial is that the existing file isn't replaced until there is a
> complete file to replace it with.  The downside is building up that
> entire new file.

Agreed. --inplace definitely works better for me.

Thank you,

--
Don Kuenz KB7RPU

Genius is nothing but a greater aptitude for patience. - de Buffon


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html