Samba › Samba - rsync

Rsync like Time Machine

Classic

List

Threaded

6 messages Options

Clint Olsen

Rsync like Time Machine

I've been very interested in these discussions and uses of rsync as a "clone" of Time Machine. A couple of things have been keeping me from a fully automated solution. I'd like to eliminate the need for Samba/NFS mounts of any kind, because they have proven to be unreliable for me and under some operating environments (Cygwin) it breaks --link-dest. In most of the articles I've read, a target "date" directory is created with some sort of "latest" symlink for the --link-dest parameter. I can accomplish those tasks via remote ssh commands, but I was hoping there was a better way. For example, is there any circumstance where you can coax rsync into creating a target directory that's not there already?

% rsync <source> user@nas::module/<somedir-exists>/<newdir-with-date>

So, newdir-with-date doesn't exist (yet). I would like to have rsync create it for me. Is that even possible?

Thanks,

-Clint

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Greg Deback (rsync)

Re: Rsync like Time Machine

Hi,

As for the destination directory and the backup directory (--backup-dir), rsync will create the missing subdirectory (one level below the existing dir only), so yes for /<somedir-exists>/<newdir-with-date>, no for /<somedir-exists>/<newdir-with-year>/<newdir-with-month> on january 1st... But if you want this dir to be a symlink, you can't.

Greg

On Fri, Jul 27, 2012 at 7:16 PM, Clint Olsen <[hidden email]> wrote:

I've been very interested in these discussions and uses of rsync as a "clone" of Time Machine. A couple of things have been keeping me from a fully automated solution. I'd like to eliminate the need for Samba/NFS mounts of any kind, because they have proven to be unreliable for me and under some operating environments (Cygwin) it breaks --link-dest. In most of the articles I've read, a target "date" directory is created with some sort of "latest" symlink for the --link-dest parameter. I can accomplish those tasks via remote ssh commands, but I was hoping there was a better way. For example, is there any circumstance where you can coax rsync into creating a target directory that's not there already?

% rsync <source> user@nas::module/<somedir-exists>/<newdir-with-date>

So, newdir-with-date doesn't exist (yet). I would like to have rsync create it for me. Is that even possible?

Thanks,

-Clint

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Clint Olsen

Re: Rsync like Time Machine

Ok, that is helpful. As you can guess based on my question, it would be nice if all the automation can be done on the client side rather than having some specialized scripting on the receiving side to manage directories and symlinks etc.

Thanks,

-Clint

On Fri, Jul 27, 2012 at 12:55 PM, Greg Deback (rsync) <[hidden email]> wrote:

Hi,

As for the destination directory and the backup directory (--backup-dir), rsync will create the missing subdirectory (one level below the existing dir only), so yes for /<somedir-exists>/<newdir-with-date>, no for /<somedir-exists>/<newdir-with-year>/<newdir-with-month> on january 1st... But if you want this dir to be a symlink, you can't.

Greg

On Fri, Jul 27, 2012 at 7:16 PM, Clint Olsen <[hidden email]> wrote:

I've been very interested in these discussions and uses of rsync as a "clone" of Time Machine. A couple of things have been keeping me from a fully automated solution. I'd like to eliminate the need for Samba/NFS mounts of any kind, because they have proven to be unreliable for me and under some operating environments (Cygwin) it breaks --link-dest. In most of the articles I've read, a target "date" directory is created with some sort of "latest" symlink for the --link-dest parameter. I can accomplish those tasks via remote ssh commands, but I was hoping there was a better way. For example, is there any circumstance where you can coax rsync into creating a target directory that's not there already?

% rsync <source> user@nas::module/<somedir-exists>/<newdir-with-date>

So, newdir-with-date doesn't exist (yet). I would like to have rsync create it for me. Is that even possible?

Thanks,

-Clint

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Greg Deback (rsync)

Re: Rsync like Time Machine

If you really want to have a destination tree that looks like :

current -> 2012-07-22

2012-07-22/

2012-07-21/

with the current symlink pointing to the latest backup, you can manage to do it in two passes :

1. Create an empty directory '2012-07-22/' and the 'current' symlink pointing to it (relative path), and rsync it to the final destination in "replace" mode (ie. update, no deletion) with symlink on (-l). This will replace the existing 'current' symlink in the destination folder.

2. Rsync (-Ha or -Haz) your src folder to 'current/' or to '2012-07-22/' (leave the trailing slash)

Greg

On Fri, Jul 27, 2012 at 10:06 PM, Clint Olsen <[hidden email]> wrote:

Ok, that is helpful. As you can guess based on my question, it would be nice if all the automation can be done on the client side rather than having some specialized scripting on the receiving side to manage directories and symlinks etc.

Thanks,

-Clint

On Fri, Jul 27, 2012 at 12:55 PM, Greg Deback (rsync) <[hidden email]> wrote:

Hi,

As for the destination directory and the backup directory (--backup-dir), rsync will create the missing subdirectory (one level below the existing dir only), so yes for /<somedir-exists>/<newdir-with-date>, no for /<somedir-exists>/<newdir-with-year>/<newdir-with-month> on january 1st... But if you want this dir to be a symlink, you can't.

Greg

On Fri, Jul 27, 2012 at 7:16 PM, Clint Olsen <[hidden email]> wrote:

I've been very interested in these discussions and uses of rsync as a "clone" of Time Machine. A couple of things have been keeping me from a fully automated solution. I'd like to eliminate the need for Samba/NFS mounts of any kind, because they have proven to be unreliable for me and under some operating environments (Cygwin) it breaks --link-dest. In most of the articles I've read, a target "date" directory is created with some sort of "latest" symlink for the --link-dest parameter. I can accomplish those tasks via remote ssh commands, but I was hoping there was a better way. For example, is there any circumstance where you can coax rsync into creating a target directory that's not there already?

% rsync <source> user@nas::module/<somedir-exists>/<newdir-with-date>

So, newdir-with-date doesn't exist (yet). I would like to have rsync create it for me. Is that even possible?

Thanks,

-Clint

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

JD Paul

rsync --backup is non-atomic -- does it need to be?

In reply to this post by Clint Olsen

It appears that 'rsync --backup' is non-atomic.

The code in question is in backup.c, in make_simple_backup():

64 if (do_rename(fname, fnamebak) == 0) {
65 if (verbose > 1) {
66 rprintf(FINFO, "backed up %s to %s\n",
67 fname, fnamebak);
68 }
69 break;
70 }

This has a race condition because 'fname' will be gone until the
rename() from the temp file happens. It's a small window, but I've
seen it get tripped.

Is there some deep reason to NOT do

do_unlink(fnamebak);
do_link(fname, fnamebak);

instead of the do_rename(), so that fname never disappears, and is
updated atomically?

(This is, of course, not actual code; it requires error checking,
verbosity, and a judgement call on what to do about a backup file that
is not unlink()-able. I figured I'd ask about the underlying logic
before attempting a patch.)

Thanks --

JD Paul
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Clint Olsen

Re: Rsync like Time Machine

In reply to this post by Greg Deback (rsync)

This is a cool idea. I'll give it a try and see if I can make it work for me. One thing missing is the switches for "replace mode" you mentioned. I'll comb through the manpage of course...

Thanks,

-Clint

On Fri, Jul 27, 2012 at 1:16 PM, Greg Deback (rsync) <[hidden email]> wrote:

If you really want to have a destination tree that looks like :
current -> 2012-07-22
2012-07-22/

2012-07-21/

with the current symlink pointing to the latest backup, you can manage to do it in two passes :

1. Create an empty directory '2012-07-22/' and the 'current' symlink pointing to it (relative path), and rsync it to the final destination in "replace" mode (ie. update, no deletion) with symlink on (-l). This will replace the existing 'current' symlink in the destination folder.

2. Rsync (-Ha or -Haz) your src folder to 'current/' or to '2012-07-22/' (leave the trailing slash)
Greg

On Fri, Jul 27, 2012 at 10:06 PM, Clint Olsen <[hidden email]> wrote:

Ok, that is helpful. As you can guess based on my question, it would be nice if all the automation can be done on the client side rather than having some specialized scripting on the receiving side to manage directories and symlinks etc.

Thanks,

-Clint

On Fri, Jul 27, 2012 at 12:55 PM, Greg Deback (rsync) <[hidden email]> wrote:

Hi,

As for the destination directory and the backup directory (--backup-dir), rsync will create the missing subdirectory (one level below the existing dir only), so yes for /<somedir-exists>/<newdir-with-date>, no for /<somedir-exists>/<newdir-with-year>/<newdir-with-month> on january 1st... But if you want this dir to be a symlink, you can't.

Greg

On Fri, Jul 27, 2012 at 7:16 PM, Clint Olsen <[hidden email]> wrote:

I've been very interested in these discussions and uses of rsync as a "clone" of Time Machine. A couple of things have been keeping me from a fully automated solution. I'd like to eliminate the need for Samba/NFS mounts of any kind, because they have proven to be unreliable for me and under some operating environments (Cygwin) it breaks --link-dest. In most of the articles I've read, a target "date" directory is created with some sort of "latest" symlink for the --link-dest parameter. I can accomplish those tasks via remote ssh commands, but I was hoping there was a better way. For example, is there any circumstance where you can coax rsync into creating a target directory that's not there already?

% rsync <source> user@nas::module/<somedir-exists>/<newdir-with-date>

So, newdir-with-date doesn't exist (yet). I would like to have rsync create it for me. Is that even possible?

Thanks,

-Clint

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html