DO NOT REPLY [Bug 5801] New: Sparse (-S) option doesn't work with new files

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

DO NOT REPLY [Bug 5801] New: Sparse (-S) option doesn't work with new files

samba-bugs
https://bugzilla.samba.org/show_bug.cgi?id=5801

           Summary: Sparse (-S) option doesn't work with new files
           Product: rsync
           Version: 3.1.0
          Platform: Other
        OS/Version: Linux
            Status: NEW
          Severity: normal
          Priority: P3
         Component: core
        AssignedTo: [hidden email]
        ReportedBy: [hidden email]
         QAContact: [hidden email]


Excuse me, this seems to be a known issue, but I couldn't find any related
existing bug for it.

The 'handle sparse files efficiently' option (--sparse/-S) doesn't work with
new files, i.e. files with doesn't exist on the destination. In this case all
sparse blocks are transmitted as zero blocks which slows down the sync process
heavily in some application.

This issue was discussed already:
http://lists.samba.org/archive/rsync/2003-August/006998.html
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=337827



OT:
I detected this today when I was scripting something to sync some large video
files in mkv format (around 1.1GB) with the last MB transfered first in order
to watch the incomplete video while it was transfered. (BTW: An option in rsync
for this would be awesome!!). To do this I created a sparse copy of the video
with only the last MB holding real data using 'dd if=... of=.sparse/... count=0
bs=1M seek=$SIZE_IN_MB_MINUS_ONE' and then rsyncing the .sparse dir first using
-S. To my surprise this was as slow as the rsyncing of the real file.


--
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

DO NOT REPLY [Bug 5801] Speed up delta-transfer of zeros in new files with --sparse

samba-bugs
https://bugzilla.samba.org/show_bug.cgi?id=5801


[hidden email] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|normal                      |enhancement
            Summary|Sparse (-S) option doesn't  |Speed up delta-transfer of
                   |work with new files         |zeros in new files with --
                   |                            |sparse




------- Comment #1 from [hidden email]  2008-10-01 20:17 CST -------
As currently designed, the --sparse option only affects writing of the
destination file, not the transfer of the data, so this is an enhancement
request.  -z may help.  It may be worth adding a special case for zero blocks
to the delta-transfer algorithm, as suggested in the Debian bug report, to
improve the behavior for sparse files without the full CPU-time penalty of
compression.


--
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

DO NOT REPLY [Bug 5801] Speed up delta-transfer of zeros in new files with --sparse

samba-bugs
In reply to this post by samba-bugs
https://bugzilla.samba.org/show_bug.cgi?id=5801





------- Comment #2 from [hidden email]  2009-10-05 09:16 CST -------
I would like to repeat my interest in such a feature. I frequently copying
large sparse files over a not-so-fast network connections. At the moment I have
to pre-create the destination files as large sparse files using an extra
script.
In my opinion this should be done by rsync itself.


--
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

DO NOT REPLY [Bug 5801] Speed up delta-transfer of zeros in new files with --sparse

samba-bugs
In reply to this post by samba-bugs
https://bugzilla.samba.org/show_bug.cgi?id=5801





------- Comment #3 from [hidden email]  2009-11-21 00:00 CST -------
Fedora has a similar RFE:

https://bugzilla.redhat.com/show_bug.cgi?id=525545

There, the further suggestion is made that the receiver should be able to skip
long runs of zeros without reconstructing them in memory.  A simple approach
would be to skip one delta-transfer block at a time.  Or, if the protocol is
changed (rather than just having the generator fabricate an all-zero block), it
would be easy to add a token representing a run of zeros of arbitrary length.


--
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

[Bug 5801] Speed up delta-transfer of zeros in new files with --sparse

samba-bugs
In reply to this post by samba-bugs
https://bugzilla.samba.org/show_bug.cgi?id=5801

--- Comment #4 from [hidden email] 2012-03-10 22:53:06 UTC ---
Maybe related:
https://bugzilla.samba.org/show_bug.cgi?id=7854

--
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

[Bug 5801] Speed up delta-transfer of zeros in new files with --sparse

samba-bugs
In reply to this post by samba-bugs
https://bugzilla.samba.org/show_bug.cgi?id=5801

Björn Jacke <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #5 from Björn Jacke <[hidden email]> ---
*** Bug 7854 has been marked as a duplicate of this bug. ***

--
You are receiving this mail because:
You are the QA Contact for the bug.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html