why rsync update large directory so slow?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

why rsync update large directory so slow?

风形
Hi:
  I have a large directory with about 0 Milillion files in it to backup by network. Server and client both windows.
  rsync server config:
------------------------------------------------------------------------------------------------------
strict modes = false
hosts allow = 10.200.121.104
hosts allow = 10.171.209.87
hosts deny = *
log file = /cygdrive/d/log/rsyncd.log
port=8730
uid = 0
gid = 0
# Module definitions
# Remember cygwin naming conventions : c:\work becomes /cygwin/c/work
#
[d_hst]
path = /cygdrive/d/hst
read only = false
transfer logging = yes
auth users = admin
secrets file = etc/user.pass
list = no
ignore errors
------------------------------------------------------------------------------------------------------
 
client bat script:
------------------------------------------------------------------------------------------------------
@echo off
set RSYNC_HOME="c:\Program Files (x86)\cwRsync\bin\"
cd /d %RSYNC_HOME%
:: Source root directory
set src_dir=rsync://admin@10.160.27.48:8730/d_hst
:: Destination root directory
set dst_dir=/cygdrive/e/filetest
:: Make sure path without space
set pwd_file="/cygdrive/c/user.pass"
:: Log file
set log_dir=/cygdrive/d/log
:: Synchronize Files, with 16 connections according to 16 directories(0~f)
for %%i in (0 1 2 3 4 5 6) do (
setlocal enabledelayedexpansion
set log_file=%log_dir%/%date:~3,4%%date:~8,2%%date:~11,2%%time:~0,2%%time:~3,2%%time:~6,2%_%%i.txt
:: Remove space in log_file
set "log_file=!log_file: =!"
start cmd /c rsync -avvu --delete %src_dir%/Files/%%i %dst_dir%/Files --password-file=%pwd_file% --log-file=!log_file! --log-file-format=""
)
------------------------------------------------------------------------------------------------------
 
In order to make backup faster, I created many rsync intances with "-avvu" options. It seems worked, and only take about 5 hours to complete backup job. 
When I restarted the client bat script, and expect very little time to complete backup, but it still cost about 3 hours.
It seems that concurrence makes no effect, I can not figure out what happend, the file checking is so slow?
Does anybody encounted cases like this? Any help will be thankful.
 
 

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: why rsync update large directory so slow?

Fabian Cenedese
At 14:22 24.11.2015, =?gb18030?B?t+fQzg==?= wrote:
>Hi:
>  I have a large directory with about 0 Milillion files in it to backup by network. Server and client both windows.
>
>In order to make backup faster, I created many rsync intances with "-avvu" options. It seems worked, and only take about 5 hours to complete backup job.
>When I restarted the client bat script, and expect very little time to complete backup, but it still cost about 3 hours.
>It seems that concurrence makes no effect, I can not figure out what happend, the file checking is so slow?
>Does anybody encounted cases like this? Any help will be thankful.

If this is just one directory on one hard disk then the disk head has to
move constantly from one place to another to another to another for
each process. It's possible that it will run even faster if you have
just one process. On the other hand Windows will cache file info
in RAM so a second run could be faster then the first one. This
depends one your RAM, how much have you got? Have you tried
with just one process?

bye  Fabi


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: Slow backup

Fabian Cenedese

>>>  I have a large directory with about 0 Milillion files in it to backup by network. Server and client both windows.
>>>
>>>In order to make backup faster, I created many rsync intances with "-avvu" options. It seems worked, and only take about 5 hours to complete backup job.
>>>When I restarted the client bat script, and expect very little time to complete backup, but it still cost about 3 hours.
>>>It seems that concurrence makes no effect, I can not figure out what happend, the file checking is so slow?
>>>Does anybody encounted cases like this? Any help will be thankful.
>>
>>If this is just one directory on one hard disk then the disk head has to
>>move constantly from one place to another to another to another for
>>each process. It's possible that it will run even faster if you have
>>just one process. On the other hand Windows will cache file info
>>in RAM so a second run could be faster then the first one. This
>>depends one your RAM, how much have you got? Have you tried
>>with just one process?
>
>Thanks for your reply. I have tried one process, but have no improvement in essence. So the question is
>why the file comparison consumes so much time? In my mind, building file list may consumes some time,
>but comparison should very fast, what is the problem?

Please keep it on the mailing list. I'm not an expert, others might
know more than me.

rsync has to stat each file and directory. I don't know what exactly
takes much time. But maybe you can give more information about
your setup: What OS, what file system, how many files and dirs,
small/big files, connection speed, full command line with all
rsync options, RAM, CPU etc.

Are you using cwRsync? With the CygWin emulation? This
is known to be slower. But even then 5h seems slow.

Just as an example: Here's a log from a backup running on a
slow NAS to another NAS in the local LAN not doing much
as the files are already the same. But both are running Linux
or actually BusyBox:

Number of files: 3445179
Number of files transferred: 0
Total file size: 349083766943 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 71253503
File list generation time: 0.016 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 71953972
Total bytes received: 700464

sent 71953972 bytes  received 700464 bytes  66079.52 bytes/sec
total size is 349083766943  speedup is 4804.71

real    18m18.760s
user    0m55.883s
sys     2m10.016s

So over 3 million files in 20 minutes. What are your --stats?

bye  Fabi


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html