Re: rsync Digest, Vol 162, Issue 18

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: rsync Digest, Vol 162, Issue 18

Robert Bell-7
Folks,

Am 20.06.2016 um 22:01 schrieb Larry Irwin (gmail):
  > The scripts I use analyze the rsync log after it completes and then
sftp's a summary to the root of the just completed rsync.
  > If no summary is found or the summary is that it failed, the folder
rotation for that set is skipped and that folder is re-used on the
subsequent rsync.
  > The key here is that the folder rotation script runs separately from
the rsync script(s).

That is what we found to be important some years ago - do most of the
management outside of the scripts that do rsync.

In particular, our scripts prepare a target backup directory with a name
of the form:
        slash.20151122.seq.1379.current
   - the name of the area being backed-up is obvious, as is the date.
We use Tower of Hanoi management (see the update at
https://en.wikipedia.org/wiki/Backup_rotation_scheme#Tower_of_Hanoi), so
embed a sequence number in the backup directory name, and the suffix
.current to indicate a current backup target.

we can keep on trying the rsync backups until we get success, and the
scripts then remove the .current suffix.  This works because the rsync
includes the --delete option, used since the backup target can be a
recycled hard-linked directory: also recommended.

  > For each entity I want to rsync, I create a named folder to identify
it and the rsync'd data is held in sub-folders:
  > daily.[1-7] and monthly.[1-3]
  > When I rsync, I rsync into daily.0 using daily.1 as the link-dest.
  > Then the rotation script checks daily.0/rsync.summary - and if it
worked, it removes daily.7 and renames the daily folders.
  > On the first of the month, the rotation script removes monthly.3,
renames the other 2 and makes a complete hard-link copy of daily.1 to
monthly.1
  > It's been running now for about 4 years and, in my environment, the
10 copies take about 4 times the space of a single copy.
  > (we do complete copies of linux servers - starting from /)
  > If there's a good spot to post the scripts, I'd be glad to put them up.

I'd recommend Tower of Hanoi, as referenced above, for managing sets of
backups - far simpler than dealing with days and dates, etc.  For
example, the above mentions special actions on the first of the month -
what if that is missed because of some failure - more special case
scripting? Tower of Hanoi is self-healing....



Regards

Rob.

Dr Robert C. Bell
HPC National Partnerships | Scientific Computing
CSIRO Information Management and Technology
T +61 3 9545 2368 (Docklands and Clayton)
T +61 3 9669 8102 (Docklands only)
Mob +61 428 108 333
[hidden email]<mailto:[hidden email]> | www.csiro.au
https://wiki.csiro.au/display/ASC/Scientific+Computing+Homepage

Also: Bureau of Meteorology Scientific Computing Services

Street: CSIRO SC Lvl 11, 700 Collins Street, Docklands Vic 3008, Australia
Postal: CSIRO SC Lvl 11, GPO Box 1289, Melbourne Vic 3001, Australia
Street: CSIRO IMT, Rivett Room RNW 2.03, Clayton Vic 3168, Australia
Postal: CSIRO IMT, Private Bag 10, Clayton South Vic 3169, Australia
Parcels: CSIRO IMT, Gate 3 Normanby Road, Clayton Vic 3168, Australia


PLEASE NOTE
The information contained in this email may be confidential or
privileged.  Any unauthorised use or disclosure is prohibited.  If
you have received this email in error, please delete it immediately
and notify the sender by return email. Thank you.  To the extent
permitted by law, CSIRO does not represent, warrant and/or
guarantee that the integrity of this communication has been
maintained or that the communication is free of errors, virus,
interception or interference.

Please consider the environment before printing this email.

On 22/06/2016 10:00 pm, [hidden email] wrote:

> Send rsync mailing list submissions to
> [hidden email]
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://lists.samba.org/mailman/listinfo/rsync
> or, via email, send a message with subject or body 'help' to
> [hidden email]
>
> You can reach the person managing the list at
> [hidden email]
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of rsync digest..."
>
>
>
> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
> ---------------------------------------
>
>
>
> Today's Topics:
>
>    1. Re: rsync script for snapshot backups (Dennis Steinkamp)
>    2. Re: rsync script for snapshot backups (Petros Angelatos)
>
>
>
> _______________________________________
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
> rsync mailing list
> [hidden email]
> https://lists.samba.org/mailman/listinfo/rsync
>

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync Digest, Vol 162, Issue 18

Larry Irwin-2
I agree that the Tower of Hanoi distribution is a good method.
It ensures a better depth. And with 10 copies, it would really, really
be good depth.
With accounting systems though, it's good to have a backup of the data
just -before- they close the fiscal period...
(the call we get is -- OMG we forgot to post that $250K before we closed
-- admin is going to kill us -- can we go back?)
We've only had a couple of connectivity issues with 2 or 3 clients over
4 years that caused it to miss one of the monthly rotations.
(the auto-retry off business hours is what has increased the resiliency
of the process at sites with less than perfect internet service)

--
Larry Irwin
Email: [hidden email]

On 06/22/2016 09:20 PM, Bell, Robert (CSIRO IM&T, Docklands) wrote:

> Folks,
>
> Am 20.06.2016 um 22:01 schrieb Larry Irwin (gmail):
>  > The scripts I use analyze the rsync log after it completes and then
> sftp's a summary to the root of the just completed rsync.
>  > If no summary is found or the summary is that it failed, the folder
> rotation for that set is skipped and that folder is re-used on the
> subsequent rsync.
>  > The key here is that the folder rotation script runs separately
> from the rsync script(s).
>
> That is what we found to be important some years ago - do most of the
> management outside of the scripts that do rsync.
>
> In particular, our scripts prepare a target backup directory with a
> name of the form:
>     slash.20151122.seq.1379.current
>   - the name of the area being backed-up is obvious, as is the date.
> We use Tower of Hanoi management (see the update at
> https://en.wikipedia.org/wiki/Backup_rotation_scheme#Tower_of_Hanoi),
> so embed a sequence number in the backup directory name, and the
> suffix .current to indicate a current backup target.
>
> we can keep on trying the rsync backups until we get success, and the
> scripts then remove the .current suffix.  This works because the rsync
> includes the --delete option, used since the backup target can be a
> recycled hard-linked directory: also recommended.
>
>  > For each entity I want to rsync, I create a named folder to
> identify it and the rsync'd data is held in sub-folders:
>  > daily.[1-7] and monthly.[1-3]
>  > When I rsync, I rsync into daily.0 using daily.1 as the link-dest.
>  > Then the rotation script checks daily.0/rsync.summary - and if it
> worked, it removes daily.7 and renames the daily folders.
>  > On the first of the month, the rotation script removes monthly.3,
> renames the other 2 and makes a complete hard-link copy of daily.1 to
> monthly.1
>  > It's been running now for about 4 years and, in my environment, the
> 10 copies take about 4 times the space of a single copy.
>  > (we do complete copies of linux servers - starting from /)
>  > If there's a good spot to post the scripts, I'd be glad to put them
> up.
>
> I'd recommend Tower of Hanoi, as referenced above, for managing sets
> of backups - far simpler than dealing with days and dates, etc.  For
> example, the above mentions special actions on the first of the month
> - what if that is missed because of some failure - more special case
> scripting? Tower of Hanoi is self-healing....
>
>
>
> Regards
>
> Rob.
>
> Dr Robert C. Bell
> HPC National Partnerships | Scientific Computing
> CSIRO Information Management and Technology
> T +61 3 9545 2368 (Docklands and Clayton)
> T +61 3 9669 8102 (Docklands only)
> Mob +61 428 108 333
> [hidden email]<mailto:[hidden email]> | www.csiro.au
> https://wiki.csiro.au/display/ASC/Scientific+Computing+Homepage
>
> Also: Bureau of Meteorology Scientific Computing Services
>
> Street: CSIRO SC Lvl 11, 700 Collins Street, Docklands Vic 3008,
> Australia
> Postal: CSIRO SC Lvl 11, GPO Box 1289, Melbourne Vic 3001, Australia
> Street: CSIRO IMT, Rivett Room RNW 2.03, Clayton Vic 3168, Australia
> Postal: CSIRO IMT, Private Bag 10, Clayton South Vic 3169, Australia
> Parcels: CSIRO IMT, Gate 3 Normanby Road, Clayton Vic 3168, Australia
>
>
> PLEASE NOTE
> The information contained in this email may be confidential or
> privileged.  Any unauthorised use or disclosure is prohibited.  If
> you have received this email in error, please delete it immediately
> and notify the sender by return email. Thank you.  To the extent
> permitted by law, CSIRO does not represent, warrant and/or
> guarantee that the integrity of this communication has been
> maintained or that the communication is free of errors, virus,
> interception or interference.
>
> Please consider the environment before printing this email.
>
> On 22/06/2016 10:00 pm, [hidden email] wrote:
>> Send rsync mailing list submissions to
>>     [hidden email]
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>     https://lists.samba.org/mailman/listinfo/rsync
>> or, via email, send a message with subject or body 'help' to
>>     [hidden email]
>>
>> You can reach the person managing the list at
>>     [hidden email]
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of rsync digest..."
>>
>>
>>
>> To unsubscribe or change options:
>> https://lists.samba.org/mailman/listinfo/rsync
>> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>>
>> ---------------------------------------
>>
>>
>>
>> Today's Topics:
>>
>>    1. Re: rsync script for snapshot backups (Dennis Steinkamp)
>>    2. Re: rsync script for snapshot backups (Petros Angelatos)
>>
>>
>>
>> _______________________________________
>> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>>
>> rsync mailing list
>> [hidden email]
>> https://lists.samba.org/mailman/listinfo/rsync
>>
>


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html