Quantcast

time variable throttling rsync traffic

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

time variable throttling rsync traffic

Eric S. Johansson
I am using rsync to back up across a VPN.  Unfortunately, every so often the
home office miscreants drop a big block of data into the backup and that
particular backup cycle takes many hours.  These same people also complain when
net pipe is filled during the day.

What I need is an ability to change the bwlimit based on time of day.  Any
suggestions as to how to do this?

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: time variable throttling rsync traffic

Matt McCutchen-7
On Sun, 2009-09-13 at 21:02 -0400, Eric S. Johansson wrote:
> I am using rsync to back up across a VPN.  Unfortunately, every so often the
> home office miscreants drop a big block of data into the backup and that
> particular backup cycle takes many hours.  These same people also complain when
> net pipe is filled during the day.
>
> What I need is an ability to change the bwlimit based on time of day.  Any
> suggestions as to how to do this?

How about the suggestions you were given on the rsnapshot list?

http://sourceforge.net/mailarchive/forum.php?thread_name=1252620930.2041.30.camel%40localhost&forum_name=rsnapshot-discuss

--
Matt

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: time variable throttling rsync traffic

Andrew Gideon
On Sun, 13 Sep 2009 21:20:01 -0400, Matt McCutchen wrote:

> How about the suggestions you were given on the rsnapshot list?

Assuming that you're using Linux somewhere in the mix, its ability to put
different network traffic into different pools for purposes of rate
management is (1) admittedly poorly documented, but (2) quite potent.  
You could put the rsync traffic into a low priority queue, where it gets
all the bandwidth not being used by something else.  Or you could set a
minimum, so that the rsync cannot be completely starved out.

        - Andrew
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: time variable throttling rsync traffic

Eric S. Johansson
In reply to this post by Matt McCutchen-7
On 9/13/2009 9:20 PM, Matt McCutchen wrote:

> On Sun, 2009-09-13 at 21:02 -0400, Eric S. Johansson wrote:
>> I am using rsync to back up across a VPN.  Unfortunately, every so often the
>> home office miscreants drop a big block of data into the backup and that
>> particular backup cycle takes many hours.  These same people also complain when
>> net pipe is filled during the day.
>>
>> What I need is an ability to change the bwlimit based on time of day.  Any
>> suggestions as to how to do this?
>
> How about the suggestions you were given on the rsnapshot list?

they're fine as far as they go.  I have some serious objections to a fair number
of them, most from operational or understandability perspective.  I mostly posed
the question here to see if new ideas would come to the top. the rsnapshot
community is much smaller than that of rsync and I thought it wouldn't be a bad
idea to give the idea a wider audience.

As for the general ideas proposed for using things like firewall rate limiting
and the like, they are all error prone.  One of the reasons I liked modifying
bwlimit is that it's easy for any twit to understand and complicated enough for
a smarty to hurt themselves.  It is self-contained.  It is all within one tool
and there's no way you can hurt or damage anyone else through its use.

The QOS solution is good because if the firewall gets clogged, in theory, rsync
should back off but I would have to wrapper the firewall commands such that a
well-meaning idiot couldn't hurt anyone.  This comes back to the all in one
package around some version of rsync.

Unfortunately, the QOS solution only works for the platform you develop it for.
  On the other hand, the bwlimit solution works for almost every platform but
doesn't behave well if there are multiple rsync clients talking to one hopes.
The only way around that is to have the clients tell the host and let the host
do the rate limiting for all transactions.

So yeah, all solutions suck in different ways including those we don't know yet.
  I'm probably going to go with the QOS solution because it's generally a better
solution but I'm a wee bit nervous about diving into the  firewall instructions
because it's been a couple of years.  No problems though, I think I'll get it
working with only a few reboots necessary.  :-)  The hard part will be a good
user interface.


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: time variable throttling rsync traffic

Andrew Gideon
On Sun, 13 Sep 2009 22:22:34 -0400, Eric S. Johansson wrote:

>  It is all within one tool and there's no way you can hurt or damage
> anyone else through its use.

It is also within one instance of the tool.  What if two of your remote
users rsync at the same time?  Twenty?  What if someone has a need for
low-latency communication at some "odd" hour during which much rsync-ing
is occurring?

Operating at the level of the network or server provide proper
allocation, and protection of non-rsync bandwidth, regardless of the
number of rsync operations occurring at a given time.

        - Andrew
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: time variable throttling rsync traffic

Nathan Ward-7
In reply to this post by Eric S. Johansson
On 14/09/2009, at 2:22 PM, Eric S. Johansson wrote:

> Unfortunately, the QOS solution only works for the platform you  
> develop it for.  On the other hand, the bwlimit solution works for  
> almost every platform but doesn't behave well if there are multiple  
> rsync clients talking to one hopes. The only way around that is to  
> have the clients tell the host and let the host do the rate limiting  
> for all transactions.

Unless you do it properly, and do your QoS on routers in the middle.

Classify by port, put rsync traffic in to a low priority queue, and  
everything else in to a higher priority queue.

The assumption is that you *can* classify by port - if you're using  
ssh or some other shared service as the transport then that might not  
work so well.

There are some things you can do to lessen the impact of using  
something shared, especially when the service is used for something  
interactive - ie. classify large packets differently to smaller  
packets, etc.

--
Nathan Ward
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: time variable throttling rsync traffic

Andrew Gideon
On Mon, 14 Sep 2009 14:45:02 +1200, Nathan Ward wrote:

> Unless you do it properly, and do your QoS on routers in the middle.

This is true.  But there are considerations.  I became curious about
this, so I did some reading to refresh my memory.

First, keep in mind that we're talking about controlling the packets of a
remote backup.  That is, the largest volume of traffic is inbound to the
network/server in question.  There's limited ability to control the
traffic, therefore.  That is one advantage of bwlimit: it exercise
control exactly where it does the most good.

On the other hand, that control is clearly *not* where the most
information is available.

If a router is involved, it can do egress shaping on the local side.  
That's best.  If a router is not involved, then the server must do
ingress filtering.  I did some reading about this last night, and
assuming that what I read is not out of date or otherwise inaccurate, one
cannot attach class-full queues to the ingress filter.  Given that, one
can police the inbound traffic but not prioritize it.  That means that
one cannot do as I originally wrote: let the backup traffic use as much
bandwidth as available, but let anything else steal traffic from it.  To
do that requires shaping at the router.

Shaping is best on the router also because it knows more.  It can balance
traffic between inbound traffic to multiple servers.  The receiving rsync
server can only balance amongst traffic to that server.  The server knows
more than a single sending rsync, but less than the router.

As mentioned above, the router and server have limited ability to
actually control the flow of remote backup traffic.  What they can do is
delay the return of TCP ACK packets.  In theory, this slows the inbound
traffic.  But it is not ideal, as it can also cause "excess" traffic via
retries.  Since the idea is to let non-backup traffic have use of the
inbound link, this is less than ideal.

So control is most effective at the sending rsync, which suggests that
bwlimit is a good approach.  But the most information is available at the
receiving router, suggesting that shaping at the router is also a good
approach.

Interesting.

        - Andrew
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: time variable throttling rsync traffic

Eric S. Johansson
On 9/14/2009 9:25 AM, Andrew Gideon wrote:

> So control is most effective at the sending rsync, which suggests that
> bwlimit is a good approach.  But the most information is available at the
> receiving router, suggesting that shaping at the router is also a good
> approach.
>
> Interesting.

Exactly what I meant by "suck in different ways including those we don't know
yet".  I think we have a couple of "low-cost options".  As we discussed on the
rsnapshot list, you could use the remote shell option to create a bandwidth
limited remote shell.  It's crude, it costs extra because of all the data
copying but it should be a relatively cheap way to assess whether or not bwlimit
modifications would be worthwhile.

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: time variable throttling rsync traffic

Andrew Gideon
On Mon, 14 Sep 2009 13:09:41 -0400, Eric S. Johansson wrote:

> On 9/14/2009 9:25 AM, Andrew Gideon wrote:
>
>> So control is most effective at the sending rsync, which suggests that
>> bwlimit is a good approach.  But the most information is available at
>> the receiving router, suggesting that shaping at the router is also a
>> good approach.
>>
>> Interesting.
>
> Exactly what I meant by "suck in different ways including those we don't
> know yet".  I think we have a couple of "low-cost options".  

I don't think you're quite getting my point; I'm sorry I'm not being
clear.  At the moment, I see *no* good option.  I see a few different
partial solutions, but nothing that really (1) accurately controls the
flow while (2) taking other uses of bandwidth into account.


> As we
> discussed on the rsnapshot list, you could use the remote shell option
> to create a bandwidth limited remote shell.  

I don't see how this is any better than the --bwlimit feature.  It has
more control than what can be done at the router or server, but it
doesn't have the router's or server's awareness of the other uses of the
bandwidth.  So while it can replace bwlimit, it cannot shape the traffic
in any intelligent way.

For example, if there are two, twenty, or two hundred rsync users, the
"bandwidth limited remote shell" wouldn't know to slow traffic down.

Or have I misunderstood you or missed something about the "bandwidth
limited remote shell" solution?

Even if the "bandwidth limited remote shell" were built to work with an
interceding server process to coordinate bandwidth use amongst rsync
users, this would still only be dealing with rsyncs to that server.  It
ignores other servers on the same network, and it ignores other users of
the bandwidth in question.

I wonder: Is there something "smarter" than just withholding ACKs that
the receiving router (or server) could do to slow the incoming traffic?

        - Andrew

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: time variable throttling rsync traffic

Phil Vandry
In reply to this post by Andrew Gideon
On Mon, 14 Sep 2009 13:25:55 +0000 (UTC), Andrew Gideon wrote:
> If a router is involved, it can do egress shaping on the local side.  
> That's best.  If a router is not involved, then the server must do

I don't think that's best at all. I would think priority queuing is
better than shaping in this case. Shaping is much more
resource-intensive, many routers (including Linux) do a poor job of it
or don't support it at all, and it's not attacking the problem
directly anyway.

> can police the inbound traffic but not prioritize it.  That means that
> one cannot do as I originally wrote: let the backup traffic use as much
> bandwidth as available, but let anything else steal traffic from it.  To
> do that requires shaping at the router.

If there is one or more bottleneck link in the network (places where
traffic feeds from one or more links with aggregate larger capacity
into a link with smaller capacity) then it makes sense that this work
has to be done on the router that is on the sending side of the
bottleneck link(s), not on the receiving side. That's the only place
where the output queue will build up. On the receiving side, it's too
late: the other router has already made a bad decision and chosen low
priority traffic over high priority traffic from its outbound queue.

> Shaping is best on the router also because it knows more.  It can balance

Yes, the router that sits in front of the bottleneck knows the most:
it's the only one that knows, based on the size of its output queue,
whether or not congestion is present. But shaping doesn't take advantage
of that information. Priority queuing does.

-Phil
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: time variable throttling rsync traffic

Andrew Gideon

> I would think priority queuing is
> better than shaping in this case.

I'm afraid I'm not following you here.  As I've learned it, priority
queuing is one of several tools available to achieve shaping.

No?

[...]

>
> If there is one or more bottleneck link in the network (places where
> traffic feeds from one or more links with aggregate larger capacity
> into a link with smaller capacity) then it makes sense that this work
> has to be done on the router that is on the sending side of the
> bottleneck link(s), not on the receiving side.

I've been assuming that this isn't possible for a couple of reasons.

Since this is a remote backup application, I'm assuming that senders are
"all over".  That is, there is no one "sending router"; there are
instead many sending routers.  The only routers common to all these data
streams would be those close to the receiving server.

Next, most random users have no control over their routing
infrastructure.

> That's the only place
> where the output queue will build up.

I did mention that delaying ACKs is less than accurate in terms of
bandwidth control, but - as far as I know - this is all the receiving
routers/server can possibly do.  In theory, at least, this will serve to
control the rate at which data is sent because it can force the sender
to pause its transmission.

I've used this myself, but only to control traffic at a very short
distance (ie. ingress policing on a VLAN's gateway interface).  In that
case, at least, it works rather well.  But I don't believe I've ever
tried this to throttle traffic inbound over a WAN link.

        - Andrew


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: time variable throttling rsync traffic

Eric S. Johansson
On 9/14/2009 3:55 PM, Andrew Gideon wrote:

>> If there is one or more bottleneck link in the network (places where
>> traffic feeds from one or more links with aggregate larger capacity
>> into a link with smaller capacity) then it makes sense that this work
>> has to be done on the router that is on the sending side of the
>> bottleneck link(s), not on the receiving side.
>
> I've been assuming that this isn't possible for a couple of reasons.
>
> Since this is a remote backup application, I'm assuming that senders are
> "all over".  That is, there is no one "sending router"; there are
> instead many sending routers.  The only routers common to all these data
> streams would be those close to the receiving server.
>
> Next, most random users have no control over their routing
> infrastructure.

In theory, you could have graph of rsync traffic between multiple hosts.   but
for right now, let's just consider two hosts. The total bandwidth available
between any two machines is the lesser of the two bwlimits.  It would need to be
negotiated periodically as bandwidth limits change over time.  For example, at 8
a.m. local time, my bandwidth limit my drop to 100 kbytes a second whereas the
other end may still be available to run full tilt.  I need to drop that link at
8 a.m. without necessarily restarting the transfer.

It's not a very good model in that you can't take into account and with
limitations at various chokepoints but, it's better than nothing.

When you expand it to a graph network, you'll need some form of a rendezvous
point and allocation scheme to allocate bandwidth to all the others.  It would
get messy because a client to or three hops away could influence the bandwidth
allocated to a local client.

Unfortunately, I have an idea for experimentation. Tell me how gross is this?

run rsync till a given time deadline, killing off the original program instance
and then restart with a new bandwidth limit.  I would probably use a small
program invoking rsync and then sending a signal when "it's Time" then starting
a new rsync run.

This could only be client-side as server-side couldn't trigger proper restarts I
don't think.  I'd need to look into that.  My main question would be is there
any way to tell if rsync has been shut down abruptly by external signal or has
terminated on its own (with or without errors) ?

I know, this is probably ugly as all get out but, it gives me a chance to
experiment.

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: time variable throttling rsync traffic

Matthias Schniedermeyer
On 15.09.2009 12:11, Eric S. Johansson wrote:

> On 9/14/2009 3:55 PM, Andrew Gideon wrote:
>
>>> If there is one or more bottleneck link in the network (places where
>>> traffic feeds from one or more links with aggregate larger capacity
>>> into a link with smaller capacity) then it makes sense that this work
>>> has to be done on the router that is on the sending side of the
>>> bottleneck link(s), not on the receiving side.
>>
>> I've been assuming that this isn't possible for a couple of reasons.
>>
>> Since this is a remote backup application, I'm assuming that senders are
>> "all over".  That is, there is no one "sending router"; there are
>> instead many sending routers.  The only routers common to all these data
>> streams would be those close to the receiving server.
>>
>> Next, most random users have no control over their routing
>> infrastructure.
>
> run rsync till a given time deadline, killing off the original program
> instance and then restart with a new bandwidth limit.  I would probably
> use a small program invoking rsync and then sending a signal when "it's
> Time" then starting a new rsync run.

I had a similar problem with "wget" some time ago, i needed a dynamic
"limit-rate". What i did was patch in a "limite-rate-file"-option that
changes the "limit-rate"-variable with the contents of the file whenever
the mtime of the file changes. It even works when the file is missing
initally (no initial "limit-rate" then)


Just an idea, it sucks in some ways but "Works for me(tm)".



Bis denn

--
Real Programmers consider "what you see is what you get" to be just as
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated,
cryptic, powerful, unforgiving, dangerous.

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: time variable throttling rsync traffic

Andrew Gideon
In reply to this post by Eric S. Johansson
On Tue, 15 Sep 2009 12:11:03 -0400, Eric S. Johansson wrote:

> run rsync till a given time deadline, killing off the original program
> instance and then restart with a new bandwidth limit.  I would probably
> use a small program invoking rsync and then sending a signal when "it's
> Time" then starting a new rsync run.

I'd recommend that you use what someone else has suggested: the --rsh
option to invoke some utility which uses SSH but which adds throttling.  
This would give you much more flexibility in testing, including the
ability (for example) to reread a configuration file upon receipt of a
USR1 signal.

No restarts of rsync necessary, therefore, to change the bw consumption.

It can also potentially be extended in other directions.  For one crazy
example, the utility (or some other utility that modifies the first
utilities configuration) could listen on a port for messages from -
presumably - the receiving server.  That would be a way for the receiving
server to tell the sender what bandwidth to consume.

I'm not clear that this is an improvement over withholding ACKs, mind
you, but at least it is a possibility with which you can experiment.

        - Andrew
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: time variable throttling rsync traffic

Andrew Gideon
On Tue, 15 Sep 2009 22:01:04 +0000, Andrew Gideon wrote:

> It can also potentially be extended in other directions.  For one crazy
> example, the utility (or some other utility that modifies the first
> utilities configuration) could listen on a port for messages from -
> presumably - the receiving server.  That would be a way for the
> receiving server to tell the sender what bandwidth to consume.

Having given this a little more thought, I see issues.  The largest is
that it requires that the receiving device be able to connect to a port
on the sending device.  In general, I'd expect firewalls to be a problem.

There's a work around that I think started with the gaming community.  
The sending device needs to send a message to the receiving device which
causes the sending device's firewall(s) to create an opening through
which the receiver can then send messages.  

All this is just a work around, avoiding the "real" solution: some way
for the two intermediary processes - the one invoked via --rsh on the
client and the one presumably invoked via --rsync-path on the server - to
communicate over the same SSH stream used by the two rsync processes.

Hmm...perhaps a middle ground is for the SSH process on the sending side
to permit the receiving device to "talk back" to the sending side via use
of SSH's -R port forwarding option?

        - Andrew
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: time variable throttling rsync traffic

Eric S. Johansson
On 9/16/2009 11:11 AM, Andrew Gideon wrote:

> On Tue, 15 Sep 2009 22:01:04 +0000, Andrew Gideon wrote:
>
>> It can also potentially be extended in other directions.  For one crazy
>> example, the utility (or some other utility that modifies the first
>> utilities configuration) could listen on a port for messages from -
>> presumably - the receiving server.  That would be a way for the
>> receiving server to tell the sender what bandwidth to consume.
>
> Having given this a little more thought, I see issues.  The largest is
> that it requires that the receiving device be able to connect to a port
> on the sending device.  In general, I'd expect firewalls to be a problem.
>
> There's a work around that I think started with the gaming community.
> The sending device needs to send a message to the receiving device which
> causes the sending device's firewall(s) to create an opening through
> which the receiver can then send messages.
>
> All this is just a work around, avoiding the "real" solution: some way
> for the two intermediary processes - the one invoked via --rsh on the
> client and the one presumably invoked via --rsync-path on the server - to
> communicate over the same SSH stream used by the two rsync processes.
>
> Hmm...perhaps a middle ground is for the SSH process on the sending side
> to permit the receiving device to "talk back" to the sending side via use
> of SSH's -R port forwarding option?


Lee Winter and I had a conversation last night by phone about this very topic.
Turns out that a fairly small set of changes enables all whole bunch of control.
  For example, if you have a single file (one to an rsync process) in which you
place the bandwidth limit of that  rsync process and you sample that file on a
regular basis for change, you can modulate the data rate for that given process
by changing the contents of that file.

The next level of control requires an independent demon which allocates
bandwidth on a per process basis according to some set maximum.  As rsync
processes come or go, the bandwidth allocation would shift among the remaining
processes.

If you allow the control processes to talk to each other and perform "speed
tests" you can allocate according to the link and/or limits on both sides.

If you let an rsync process fill a second say this file with how much is left to
be transferred, you could "finish up" almost on processes at a higher bandwidth
allocation than those that will be running much longer.  Yet, seems
counterintuitive but it lets you complete short running tasks faster and let
them get out of the way which may (or may not) provide benefits.

by adding relatively simple signals, bandwidth can be modulated by various
mechanisms of varying complexity ranging from the very simple to the very complex.

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Loading...