Summary: Sparse (-S) option doesn't work with new files
AssignedTo: [hidden email] ReportedBy: [hidden email] QAContact: [hidden email]
Excuse me, this seems to be a known issue, but I couldn't find any related
existing bug for it.
The 'handle sparse files efficiently' option (--sparse/-S) doesn't work with
new files, i.e. files with doesn't exist on the destination. In this case all
sparse blocks are transmitted as zero blocks which slows down the sync process
heavily in some application.
I detected this today when I was scripting something to sync some large video
files in mkv format (around 1.1GB) with the last MB transfered first in order
to watch the incomplete video while it was transfered. (BTW: An option in rsync
for this would be awesome!!). To do this I created a sparse copy of the video
with only the last MB holding real data using 'dd if=... of=.sparse/... count=0
bs=1M seek=$SIZE_IN_MB_MINUS_ONE' and then rsyncing the .sparse dir first using
-S. To my surprise this was as slow as the rsyncing of the real file.
What |Removed |Added
Summary|Sparse (-S) option doesn't |Speed up delta-transfer of
|work with new files |zeros in new files with --
------- Comment #1 from [hidden email] 2008-10-01 20:17 CST -------
As currently designed, the --sparse option only affects writing of the
destination file, not the transfer of the data, so this is an enhancement
request. -z may help. It may be worth adding a special case for zero blocks
to the delta-transfer algorithm, as suggested in the Debian bug report, to
improve the behavior for sparse files without the full CPU-time penalty of
------- Comment #2 from [hidden email] 2009-10-05 09:16 CST -------
I would like to repeat my interest in such a feature. I frequently copying
large sparse files over a not-so-fast network connections. At the moment I have
to pre-create the destination files as large sparse files using an extra
In my opinion this should be done by rsync itself.
There, the further suggestion is made that the receiver should be able to skip
long runs of zeros without reconstructing them in memory. A simple approach
would be to skip one delta-transfer block at a time. Or, if the protocol is
changed (rather than just having the generator fabricate an all-zero block), it
would be easy to add a token representing a run of zeros of arbitrary length.