Solution for rsync overall progress data display

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Solution for rsync overall progress data display

Samba - rsync mailing list
Hello,

I looking for a solution to display overall rsync progress on an LCD
display as a bargraph.
I have found 2 parameters:

   --progress
               This  option  tells  rsync  to  print  information  
showing  the
               progress of the transfer. This gives a bored user  
something  to
               watch.  Implies --verbose if it wasn't already specified.

               While  rsync  is  transferring  a  regular  file,  it  
updates a
               progress line that looks like this:

                     782448  63%  110.64kB/s    0:00:04

But they are not showing the overall progress during the transfer what I
need.

All I would need is to get data out of rsync either the completed data's
bytes size or completed percentage what I could pipe into another C
program or a text file what my app could periodically read to construct
this graph.

One solution would be to run a dry rsync, get the number of overall
bytes going to be transferred and keep running du -s /datadir to get the
directories size where the rsync goes into but this is not feasible with
over 1TB data and lot of small files.

Thank you.

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: Solution for rsync overall progress data display

Samba - rsync mailing list
--info=progress2

On 11/26/2017 09:27 AM, netbsd--- via rsync wrote:

> Hello,
>
> I looking for a solution to display overall rsync progress on an LCD
> display as a bargraph.
> I have found 2 parameters:
>
>   --progress
>               This  option  tells  rsync  to  print  information 
> showing  the
>               progress of the transfer. This gives a bored user 
> something  to
>               watch.  Implies --verbose if it wasn't already specified.
>
>               While  rsync  is  transferring  a  regular  file,  it 
> updates a
>               progress line that looks like this:
>
>                     782448  63%  110.64kB/s    0:00:04
>
> But they are not showing the overall progress during the transfer what I
> need.
>
> All I would need is to get data out of rsync either the completed data's
> bytes size or completed percentage what I could pipe into another C
> program or a text file what my app could periodically read to construct
> this graph.
>
> One solution would be to run a dry rsync, get the number of overall
> bytes going to be transferred and keep running du -s /datadir to get the
> directories size where the rsync goes into but this is not feasible with
> over 1TB data and lot of small files.
>
> Thank you.
>
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
        Kevin Korb Phone:    (407) 252-6853
        Systems Administrator Internet:
        FutureQuest, Inc. [hidden email]  (work)
        Orlando, Florida [hidden email] (personal)
        Web page: http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

signature.asc (231 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Solution for rsync overall progress data display

Samba - rsync mailing list
In reply to this post by Samba - rsync mailing list
> I looking for a solution to display overall rsync progress on an LCD display as a bargraph.
> I have found 2 parameters:
>
> --progress
>             This  option  tells  rsync  to  print  information  showing  the
>             progress of the transfer. This gives a bored user  something  to
>             watch.  Implies --verbose if it wasn't already specified.
>
>             While  rsync  is  transferring  a  regular  file,  it  updates a
>             progress line that looks like this:
>
>                   782448  63%  110.64kB/s    0:00:04
>
> But they are not showing the overall progress during the transfer what I need.

Bear in mind that until the sync is almost finished, rsync does NOT know how much is left to do. AIUI, one thread is running a compare, working down the directory tree and building a list of files that aren't up to date on the target. Another thread is then taking files from this list and syncing them.
So at any point in time, there is a queue of files to be synced which is NOT complete, and a process that's syncing those files one at a time. Until the first thread is done, there isn't even a list of files, and until the sync is running, there isn't information on how much needs to be transferred for each of those files.

It's well worth reading Andrew Tridgell's PHD thesis where the algorithm is detailed. It's quite readable and gives a good insight into how rsync works.
https://www.samba.org/~tridge/phd_thesis.pdf


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: Solution for rsync overall progress data display

Samba - rsync mailing list
with --no-i-r you at least get some idea of the # of files to check.


/kc


On Sun, Nov 26, 2017 at 09:34:25PM +0000, Simon Hobson via rsync said:
  >> I looking for a solution to display overall rsync progress on an LCD display as a bargraph.
  >> I have found 2 parameters:
  >>
  >> --progress
  >>             This  option  tells  rsync  to  print  information  showing  the
  >>             progress of the transfer. This gives a bored user  something  to
  >>             watch.  Implies --verbose if it wasn't already specified.
  >>
  >>             While  rsync  is  transferring  a  regular  file,  it  updates a
  >>             progress line that looks like this:
  >>
  >>                   782448  63%  110.64kB/s    0:00:04
  >>
  >> But they are not showing the overall progress during the transfer what I need.
  >
  >Bear in mind that until the sync is almost finished, rsync does NOT know how much is left to do. AIUI, one thread is running a compare, working down the directory tree and building a list of files that aren't up to date on the target. Another thread is then taking files from this list and syncing them.
  >So at any point in time, there is a queue of files to be synced which is NOT complete, and a process that's syncing those files one at a time. Until the first thread is done, there isn't even a list of files, and until the sync is running, there isn't information on how much needs to be transferred for each of those files.
  >
  >It's well worth reading Andrew Tridgell's PHD thesis where the algorithm is detailed. It's quite readable and gives a good insight into how rsync works.
  >https://www.samba.org/~tridge/phd_thesis.pdf
  >
  >
  >--
  >Please use reply-all for most replies to avoid omitting the mailing list.
  >To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
  >Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

--
Ken Chase - Heavy Computing Inc. Guelph Canada

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html