DO NOT REPLY [Bug 7120] New: Variable bandwidth limit .. bwlimit

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

DO NOT REPLY [Bug 7120] New: Variable bandwidth limit .. bwlimit

samba-bugs
https://bugzilla.samba.org/show_bug.cgi?id=7120

           Summary: Variable bandwidth limit .. bwlimit
           Product: rsync
           Version: 3.1.0
          Platform: Other
        OS/Version: Linux
            Status: NEW
          Severity: enhancement
          Priority: P3
         Component: core
        AssignedTo: [hidden email]
        ReportedBy: [hidden email]
         QAContact: [hidden email]


I used rsync with the --bwlimit option quite a bit so that I can rsync without
slowing down the network for others.

Often, I'd use rsync without any options to see the maximum bandwidth, then
kill it,and restart with a percentage of the speed reported.

Given this situation, I was wondering if it would be possible (and you would be
willing to) add an option:

--bwlimitpercentage=
or allow for:
--bwlimit=[0-9][0-9]%

that periodically (say 1 seconds every 30 seconds) doesn't throttle the
transfer to obtain the maximum speed, then throttles down to a percentage of
that measured.

Pierre


--
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

DO NOT REPLY [Bug 7120] Variable bandwidth limit .. bwlimit

samba-bugs
https://bugzilla.samba.org/show_bug.cgi?id=7120


[hidden email] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]




------- Comment #1 from [hidden email]  2010-11-23 03:54 CST -------
Speaking about variable bandwidth, it would also be sweet if rsync could
somehow be signalled to use a different bwlimit than the one it was started
with.

The use case is: on huge rsync jobs (think multiple weeks) one would like to
throttle the bandwidth rather stringently during daytime (when the internet
connection and/or server performance is needed for work), and speed up the
rsync job during night time.
Currently this would mean killing and restarting rsync every time you'd want to
change the throttle. With a huge job (think at least a couple of hours to
send/receive file lists) this results in a highly inefficient usage of
resources.

It would be awesome to be able to signal rsync in some way to use a different
bandwidth throttle without otherwise disturbing the ongoing transfer.


--
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 7120] Variable bandwidth limit .. bwlimit

samba-bugs
In reply to this post by samba-bugs
https://bugzilla.samba.org/show_bug.cgi?id=7120

--- Comment #2 from It is me <[hidden email]> 2013-03-05 15:15:51 UTC ---
Hi,
just a note, due to the fact I am also looking forward for this feature.

I see two options regarding usability and scritablility. :)

1# Piping (not used up to now)

mkfifo /tmp/bla
echo "1000" > /tmp/bla

cat /tmp/bla | rsync ... --bwlimit-stdin ...

changing by just write to the fifo
echo "50" > /tmp/bla


2# file and signal

mktemp > /tmp/bla2
echo "1000" > /tmp/bla2
rsync ... -bwlimit-from=/tmp/bla2 .... &
RSYNCPID=$!
echo "50" > /tmp/bla2
kill -USR1 ${RSYNCPID}

hope the thinking helps.
Me.

--
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Bug 7120] Variable bandwidth limit .. bwlimit

devzero
mhh - interesting question......

what about combining the power of throttle ( http://linux.die.net/man/1/throttle ) or similar tools (there are some more like this) with rsync ?

via this hint: http://lists.samba.org/archive/rsync/2006-February/014623.html i got a clue how to combine rsync and throttle and gave it a try :

cat throttle-wrap
#!/bin/bash
throttle -k 1 -s 1| "$@"

rsync --rsh='/tmp/throttle-wrap ssh' -avz /src  user@host:/dest

seems to work fine for me, but mind that throttle needs -s 1 as this seems to circumvent the buffering problem.

not sure about the maximum bandwidth you can reach with that and how good you really can adjust the bandwith, so please report your findings......

If that is a useful option, can someone please put this into the bugzilla entry?

regards
Roland



List:       rsync
Subject:    [Bug 7120] Variable bandwidth limit .. bwlimit
From:       samba-bugs () samba ! org
Date:       2013-03-05 15:15:53
Message-ID: E1UCtar-0010tB-0s () samba-bugzilla ! samba ! org
[Download message RAW]

https://bugzilla.samba.org/show_bug.cgi?id=7120

--- Comment #2 from It is me <[hidden email]> 2013-03-05 15:15:51 UTC ---
Hi,
just a note, due to the fact I am also looking forward for this feature.

I see two options regarding usability and scritablility. :)

1# Piping (not used up to now)

mkfifo /tmp/bla
echo "1000" > /tmp/bla

cat /tmp/bla | rsync ... --bwlimit-stdin ...

changing by just write to the fifo
echo "50" > /tmp/bla


2# file and signal

mktemp > /tmp/bla2
echo "1000" > /tmp/bla2
rsync ... -bwlimit-from=/tmp/bla2 .... &
RSYNCPID=$!
echo "50" > /tmp/bla2
kill -USR1 ${RSYNCPID}

hope the thinking helps.
Me.

--
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[prev in list] [next in list] [prev in thread] [next in thread]



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Bug 7120] Variable bandwidth limit .. bwlimit

devzero
In reply to this post by samba-bugs
or better try pipe viewer. it seems less buggy (kill -SIGUSR1 $PIDOFTHROTTLE doesn't work for me) , has realtime progress and there is a homepage and maintainer ( http://www.ivarch.com/programs/pv.shtml )

linux-rqru:/tmp # cat /tmp/pv-wrapper
#!/bin/bash
pv -L 1000 | "$@"

Adjust Transfer-Rate:

pv -R $PIDOFPV -L RATE

linux-rqru:/tmp # rsync --rsh='/tmp/pv-wrapper ssh' -a /proc root@localhost:/tmp/test
Password:    4 B 0:00:01 [3.68 B/s] [ <=>                                                                                                ]
file has vanished: "/proc/10/exe"
file has vanished: "/proc/10/task/10/exe"
file has vanished: "/proc/11/exe"
file has vanished: "/proc/11/task/11/exe"
file has vanished: "/proc/12/exe"=>                                                                                            ]
4.88kiB 0:00:05 [1002 B/s] [         <=>  

regards
roland






mhh - interesting question......

what about combining the power of throttle ( http://linux.die.net/man/1/throttle ) or similar tools (there are some more like this) with rsync ?

via this hint: http://lists.samba.org/archive/rsync/2006-February/014623.html i got a clue how to combine rsync and throttle and gave it a try :

cat throttle-wrap
#!/bin/bash
throttle -k 1 -s 1| "$@"

rsync --rsh='/tmp/throttle-wrap ssh' -avz /src  user@host:/dest

seems to work fine for me, but mind that throttle needs -s 1 as this seems to circumvent the buffering problem.

not sure about the maximum bandwidth you can reach with that and how good you really can adjust the bandwith, so please report your findings......

If that is a useful option, can someone please put this into the bugzilla entry?

regards
Roland



List:       rsync
Subject:    [Bug 7120] Variable bandwidth limit .. bwlimit
From:       samba-bugs () samba ! org
Date:       2013-03-05 15:15:53
Message-ID: E1UCtar-0010tB-0s () samba-bugzilla ! samba ! org
[Download message RAW]

https://bugzilla.samba.org/show_bug.cgi?id=7120

--- Comment #2 from It is me <[hidden email]> 2013-03-05 15:15:51 UTC ---
Hi,
just a note, due to the fact I am also looking forward for this feature.

I see two options regarding usability and scritablility. :)

1# Piping (not used up to now)

mkfifo /tmp/bla
echo "1000" > /tmp/bla

cat /tmp/bla | rsync ... --bwlimit-stdin ...

changing by just write to the fifo
echo "50" > /tmp/bla


2# file and signal

mktemp > /tmp/bla2
echo "1000" > /tmp/bla2
rsync ... -bwlimit-from=/tmp/bla2 .... &
RSYNCPID=$!
echo "50" > /tmp/bla2
kill -USR1 ${RSYNCPID}

hope the thinking helps.
Me.

--
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[prev in list] [next in list] [prev in thread] [next in thread]



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 7120] Variable bandwidth limit .. bwlimit

samba-bugs
In reply to this post by samba-bugs
https://bugzilla.samba.org/show_bug.cgi?id=7120

--- Comment #3 from roland <[hidden email]> 2013-03-09 12:06:15 UTC ---
you can do that with pipe-viewer or similar pipe transfer visualization tools
with throttling feature.

http://www.ivarch.com/programs/pv.shtml 

first we need to create a little wrapper to put pv into the transfer chain:

linux:/tmp # cat /tmp/pv-wrapper
#!/bin/bash
pv -L 1000 | "$@"

example:

linux:/tmp # rsync --rsh='/tmp/pv-wrapper ssh' -a /proc
root@localhost:/tmp/test
Password: 4 B 0:00:01 [3.68 B/s] [ <=> ]
file has vanished: "/proc/10/exe"
file has vanished: "/proc/10/task/10/exe"
file has vanished: "/proc/11/exe"
file has vanished: "/proc/11/task/11/exe"
file has vanished: "/proc/12/exe"=> ]
4.88kiB 0:00:05 [1002 B/s] [ <=>

You can even adjust the transfer rate at runtime, as pv can communicate with a
running instance of itself - you will just need the appropriate PID. This would
even make cron based tuning of transfer rates possible.

pv -R $PIDOFPV -L RATE

--
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 7120] Variable bandwidth limit .. bwlimit

samba-bugs
In reply to this post by samba-bugs
https://bugzilla.samba.org/show_bug.cgi?id=7120

--- Comment #4 from Pol Hallen <[hidden email]> 2014-05-05 19:15:42 UTC ---
Hi all and thanks for replies, any "stable" way to do rsync with variable
bandwidht?

Thanks

Pol

--
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 7120] Variable bandwidth limit .. bwlimit

samba-bugs
In reply to this post by samba-bugs
https://bugzilla.samba.org/show_bug.cgi?id=7120

--- Comment #5 from roland <[hidden email]> 2014-05-06 18:11:03 UTC ---
what`s "unstable" in the way i described it?

why build every feature in every tool when the power of unix is to concatenate
tools in a way to handle requirements?

--
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 7120] Variable bandwidth limit .. bwlimit

samba-bugs
In reply to this post by samba-bugs
https://bugzilla.samba.org/show_bug.cgi?id=7120

Wayne Davison <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |WONTFIX

--- Comment #6 from Wayne Davison <[hidden email]> 2014-07-21 15:59:09 UTC ---
I like the use of the pv tool for dynamic control.  Nice suggestion!

--
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Bug 7120] Variable bandwidth limit .. bwlimit

samba-bugs
In reply to this post by samba-bugs
https://bugzilla.samba.org/show_bug.cgi?id=7120

--- Comment #7 from roland <[hidden email]> ---
please mind the note from this posting when using pv:
http://superuser.com/questions/778066/using-pv1-to-limit-rsync-speed


--snipp--
It seems that pv is waiting for data from rsync, and rsync is waiting for data
too (stuck in select()) and not closing the input to pv. So it's a deadlock.
Same happens when you substitute pv with something else (like dd). It seems
that those commands just don't behave like rsync expects them to.

Haven't found a workaround short of killing everything:

export RSYNC_RSH="sh -c 'pv -qL10k | ssh \"\$@\" | (pv -qL11k; kill \$\$)' ssh"

kill is not a solution I'd be happy with. But I haven't found another.

Of course using $$ was the lazy way out, should kill pv instead but how to
obtain the pid in a oneliner? pidof just gives any pv pids. Querying their
respective parent pids seemed a bit odd for a oneliner.

Looking forward to a real solution... ;)
--snipp--

--
You are receiving this mail because:
You are the QA Contact for the bug.

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Bug 7120] Variable bandwidth limit .. bwlimit

Lars Ellenberg
On Wed, Mar 01, 2017 at 05:38:15PM +0000, [hidden email] wrote:

> https://bugzilla.samba.org/show_bug.cgi?id=7120
>
> --- Comment #7 from roland <[hidden email]> ---
> please mind the note from this posting when using pv:
> http://superuser.com/questions/778066/using-pv1-to-limit-rsync-speed
>
>
> --snipp--
> It seems that pv is waiting for data from rsync, and rsync is waiting for data
> too (stuck in select()) and not closing the input to pv. So it's a deadlock.
> Same happens when you substitute pv with something else (like dd). It seems
> that those commands just don't behave like rsync expects them to.
>
> Haven't found a workaround short of killing everything:
>
> export RSYNC_RSH="sh -c 'pv -qL10k | ssh \"\$@\" | (pv -qL11k; kill \$\$)' ssh"

The trick is to avoid any remaining parent- or sub-shell,
achieved by using exec (the inner execs are not always needed,
that depends on the bash version. but they don't hurt, either).

this runs both pv on the local box:

RSYNC_RSH="bash -xc '\
        exec 3< <(exec pv -L10k); \
        exec 4< <(exec \"\$@\" <&3 ) 3<&-; \
        exec pv -L11k <&4 4<&-' -- ssh"

feel free to add -N IN -c, -N OUT -c, or similar...

Or runs outgoing limit on local box,
incomming limit on remote box
(each box limits the stream that leaves it):

O_LIM="bash -c 'exec 3< <(exec pv -L10k); exec \"\$@\" <&3 3<&-' -- ssh"
I_LIM="bash -c 'exec 3< <(exec \"\$@\"); exec pv -L11k <&3 3<&-' -- rsync"
rsync --rsh "$O_LIM" --rsync-path "$I_LIM" ...

Or the other way around,
if you want each box limit their respective incoming stream instead.

Cheers,

        Lars


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Bug 7120] Variable bandwidth limit .. bwlimit

Samba - rsync mailing list
In reply to this post by samba-bugs
[hidden email] wrote:
> --snipp--
> It seems that pv is waiting for data from rsync, and rsync is waiting for data
> too (stuck in select()) and not closing the input to pv. So it's a deadlock.
> Same happens when you substitute pv with something else (like dd). It seems
> that those commands just don't behave like rsync expects them to.
>  
---
    Would a use of "stdbuf" (coreutils) help?  It allows one to
change the  input and/or output buffering of the tools to
from full buffered to line-buffered to unbuffered for tools
normally connected via a pipe.


> Haven't found a workaround short of killing everything:
>
> export RSYNC_RSH="sh -c 'pv -qL10k | ssh \"\$@\" | (pv -qL11k; kill \$\$)' ssh"
> kill is not a solution I'd be happy with. But I haven't found another.
>  
---
    Maybe a suspend/continue would be more gentle than killing things?



--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Loading...