Rsync to just upload new/updated files without comparing source/destination

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Rsync to just upload new/updated files without comparing source/destination

Samba - rsync mailing list
Hi,

I have huge data on my destination system (>5 TB, > 400,000 files, nested
folderss) and i am trying to publish new files from our shared location to
destination system. I am looking forward to use rsync but want it to just
push new/updated files uploaded on shared server to destination. I don't
intent to keep all data to remain in sync on source server and want to
delete the source file as soon as it's uploaded to destination.

Essentially i want to keep same folder structure on source and destination
(with ability for users to add new folders) but don't want to keep files on
source. As per my understanding rsync compares files on source and
destination and then performs the sync. I am worried that this comparison
can take large amount of time due to data size.

Is there a way by which i can just upload new/updated files and new folders
to respective location in destination using rsync without doing any
comparisons? I read about --ignore-existing flag but not sure if
checksum/comparison between source and destination is still performed and
will we run into performance issues.

Is it feasible with rsync in an efficient way?

Thanks



--
Sent from: http://samba.2283325.n4.nabble.com/Samba-rsync-f2500462.html

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: Rsync to just upload new/updated files without comparing source/destination

Samba - rsync mailing list
If you remove the source files as rsync copies them
(--remove-source-files) then rsync will have nothing to compare.  It
will just go through the tree looking for files that aren't on the
target which would probably be any file it finds.  Just don't ever use
--delete in that scenario.  Adding --omit-dir-times will save you a lot
of output if you have any of --itemize-changes --verbose --progress.

When in doubt, do a --dry-run

On 11/19/2017 12:48 PM, Amit via rsync wrote:

> Hi,
>
> I have huge data on my destination system (>5 TB, > 400,000 files, nested
> folderss) and i am trying to publish new files from our shared location to
> destination system. I am looking forward to use rsync but want it to just
> push new/updated files uploaded on shared server to destination. I don't
> intent to keep all data to remain in sync on source server and want to
> delete the source file as soon as it's uploaded to destination.
>
> Essentially i want to keep same folder structure on source and destination
> (with ability for users to add new folders) but don't want to keep files on
> source. As per my understanding rsync compares files on source and
> destination and then performs the sync. I am worried that this comparison
> can take large amount of time due to data size.
>
> Is there a way by which i can just upload new/updated files and new folders
> to respective location in destination using rsync without doing any
> comparisons? I read about --ignore-existing flag but not sure if
> checksum/comparison between source and destination is still performed and
> will we run into performance issues.
>
> Is it feasible with rsync in an efficient way?
>
> Thanks
>
>
>
> --
> Sent from: http://samba.2283325.n4.nabble.com/Samba-rsync-f2500462.html
>
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
        Kevin Korb Phone:    (407) 252-6853
        Systems Administrator Internet:
        FutureQuest, Inc. [hidden email]  (work)
        Orlando, Florida [hidden email] (personal)
        Web page: http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

signature.asc (231 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Rsync to just upload new/updated files without comparing source/destination

Samba - rsync mailing list
Thanks for info. I was under the assumption that rsync will execute
followings steps:
1. Compare first to identify different files (new/updated) --> This is where
i am worried due to data size
2. Upload new/updated files
3 Remove source file

As per your comment, rsync will compare after uploading to check if files
are identical ?

Additionally i don't expect to delete directories as i want to maintain same
directory structure on source. Any insights on if following query will work
effeciently for adding new files to destination
rsync --remove-source-files -av -r <Source Root Folder> ssh
[hidden email]:<Destination Root Folder>

Assuming 5 TB of data available at destination and source has about 50 files
(New or Updated i.e. files with same name and in same folder), source file
will be removed as soon as it's uploaded.

Thanks



--
Sent from: http://samba.2283325.n4.nabble.com/Samba-rsync-f2500462.html

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: Rsync to just upload new/updated files without comparing source/destination

Samba - rsync mailing list
If the source only has 50 files then rsync only needs to check for those
50 files on the target.  If one exists rsync will compare the timestamp
to see if it needs copying.  Yes, rsync verifies a hash of the file
after transfer.

On 11/19/2017 02:41 PM, Amit via rsync wrote:

> Thanks for info. I was under the assumption that rsync will execute
> followings steps:
> 1. Compare first to identify different files (new/updated) --> This is where
> i am worried due to data size
> 2. Upload new/updated files
> 3 Remove source file
>
> As per your comment, rsync will compare after uploading to check if files
> are identical ?
>
> Additionally i don't expect to delete directories as i want to maintain same
> directory structure on source. Any insights on if following query will work
> effeciently for adding new files to destination
> rsync --remove-source-files -av -r <Source Root Folder> ssh
> [hidden email]:<Destination Root Folder>
>
> Assuming 5 TB of data available at destination and source has about 50 files
> (New or Updated i.e. files with same name and in same folder), source file
> will be removed as soon as it's uploaded.
>
> Thanks
>
>
>
> --
> Sent from: http://samba.2283325.n4.nabble.com/Samba-rsync-f2500462.html
>
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
        Kevin Korb Phone:    (407) 252-6853
        Systems Administrator Internet:
        FutureQuest, Inc. [hidden email]  (work)
        Orlando, Florida [hidden email] (personal)
        Web page: http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

signature.asc (231 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Rsync to just upload new/updated files without comparing source/destination

Samba - rsync mailing list
Thanks for your inputs. Surprisingly i tried executing the following command
but end up deleting ssh.exe in Cygwin folder when source folder path is
incorrect or not reachable.

rsync --remove-source-files -v -av -r /cygdrive/c/AmitTest/ ssh
username@destinationserver:/cygdrive/e/AMIT/AmitTest/

I am wondering if remove source is doing some tricks there but unable to
find root cause why ssh.exe will get deleted with this command. It has
happened twice since yesteday and today i noticed this happened when i
executed this command. Any idea why this can be happening?

Unfortunately i don't have way to copy ssh to Cygwin\bin folder as i keep
getting 'Access Denied' error from windows explorer and cmd (tried taking
ownership of Cygwin folder as well).


Thanks,
Amit



--
Sent from: http://samba.2283325.n4.nabble.com/Samba-rsync-f2500462.html

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: Rsync to just upload new/updated files without comparing source/destination

Samba - rsync mailing list
The "ssh" in that command is a second source parameter to be copied (and
then deleted).  using ssh as the transport has been the default for at
least 15 years and the option to tell it to use ssh was either -e ssh or
--rsh=ssh

BTW, -r is included in -a.  Also, -v isn't of much use without -i.

I would suggest that if you are going to use cygwin's rsync you should
do so from within cygwin's shell.  OTOH, I would also suggest that if
you are on windows you should use something native to windows like unison.

On 11/20/2017 03:28 PM, Amit via rsync wrote:

> Thanks for your inputs. Surprisingly i tried executing the following command
> but end up deleting ssh.exe in Cygwin folder when source folder path is
> incorrect or not reachable.
>
> rsync --remove-source-files -v -av -r /cygdrive/c/AmitTest/ ssh
> username@destinationserver:/cygdrive/e/AMIT/AmitTest/
>
> I am wondering if remove source is doing some tricks there but unable to
> find root cause why ssh.exe will get deleted with this command. It has
> happened twice since yesteday and today i noticed this happened when i
> executed this command. Any idea why this can be happening?
>
> Unfortunately i don't have way to copy ssh to Cygwin\bin folder as i keep
> getting 'Access Denied' error from windows explorer and cmd (tried taking
> ownership of Cygwin folder as well).
>
>
> Thanks,
> Amit
>
>
>
> --
> Sent from: http://samba.2283325.n4.nabble.com/Samba-rsync-f2500462.html
>
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
        Kevin Korb Phone:    (407) 252-6853
        Systems Administrator Internet:
        FutureQuest, Inc. [hidden email]  (work)
        Orlando, Florida [hidden email] (personal)
        Web page: http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

signature.asc (231 bytes) Download Attachment