--link-dest not working on remote server (running daemon)

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

--link-dest not working on remote server (running daemon)

Sam Holton
Hello,

I have read through the list of previous issues regarding this issue but haven't been able to resolve mine. I apologize in advance for the long text and am probably doing some simple typo. I have two servers in my setup:

Server 1
Doing rsync with --link-dest daily and working as expected. I'm getting the hard links in the new daily directories. 

Server 2
Running rsync daemon mode with following config

[offsite]
   path = /media/external/backup/
   comment = Offsite backup
   read only = no
   hosts allow = 192.168.2.0/24
   auth users = backup
   secrets file = /etc/rsyncd.scrt
   uid = 0
   gid = 0


My Goal
Server 1 has been running for several months now so it has several months of daily backups. I was able to do an initial sync to server 2 using -H option to keep my hardlink structure. That worked fine and my original plan was to just run the same rsync with -H after the daily backup was complete to keep both in sync. But that turned out to be very slow building the incremental file list as I'm guessing it scanned all files for each daily backup (even though they were hard lnked).

So my next plan was to just sync that latest daily backup from server 1 to server 2 and use the --link-dest option on server 2 to link it to the previous day.

The Problem
This is the command I'm troubleshooting right now

rsync -a -v -n -i --delete --link-dest=/backup-2016-02-01-0100 --password-file=/media/external/scripts/offsite_rsync.pass /media/external/backup/backup-2016-02-02-0100 backup@192.168.2.102::offsite

It seems to be sending all files as new files (i.e. not picking up the link-dest option). I've tried using no slash at the beginning of link-dest, tried using ./ tried full path. etc.

Here is a snippet of the output from server 1 which is running the command:

<f+++++++++ backup-2016-02-02-0100/media/external/owncloud/data/sam/files/Photos/2007/20070120 DC Air and Space Museum/IMGP0906.JPG


On server 2 there is no backup-2016-02-02-0100 directory. However, the link-dest option I'm using has the file on server 2:

-rw------- 147 www-data www-data 4454193 Jan 20  2007 /media/external/backup/backup-2016-02-01-0100/media/external/owncloud/data/sam/files/Photos/2007/20070120 DC Air and Space Museum/IMGP0906.JPG

Output of same file from server 1

-rw------- 151 www-data www-data 4454193 Jan 20  2007 /media/external/backup/backup-2016-02-02-0100/media/external/owncloud/data/sam/files/Photos/2007/20070120 DC Air and Space Museum/IMGP0906.JPG


Troubleshooting
If I manually create the copy of server 2 first (cp -al backup-2016-02-01-0100 backup-2016-02-02-0100) and then run rsync without the --link-dest option I get the expected results. Only the files that changed or were removed/added are transferred.

Also I tried running the actual transfer without -n and it is indeed transferring all the old files.

Any help is appreciated.

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: --link-dest not working on remote server (running daemon)

Kevin Korb
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 02/08/2016 03:33 PM, Sam Holton wrote:

> Hello,
>
> I have read through the list of previous issues regarding this
> issue but haven't been able to resolve mine. I apologize in advance
> for the long text and am probably doing some simple typo. I have
> two servers in my setup:
>
> *Server 1* Doing rsync with --link-dest daily and working as
> expected. I'm getting the hard links in the new daily directories.
>
>
> *Server 2* Running rsync daemon mode with following config
>
> [offsite] path = /media/external/backup/ comment = Offsite backup
> read only = no hosts allow = 192.168.2.0/24
> <http://192.168.2.0/24> auth users = backup secrets file =
> /etc/rsyncd.scrt uid = 0 gid = 0

Why are you using rsyncd instead of rsync over ssh?  Is --link-dest
even supposed to be supported via rsyncd?  Even if it is, you seem to
have your link-dest and your target parameters misaligned.

>
>
> *My Goal* Server 1 has been running for several months now so it
> has several months of daily backups. I was able to do an initial
> sync to server 2 using -H option to keep my hardlink structure.
> That worked fine and my original plan was to just run the same
> rsync with -H after the daily backup was complete to keep both in
> sync. But that turned out to be very slow building the incremental
> file list as I'm guessing it scanned all files for each daily
> backup (even though they were hard lnked).

It has to scan them to know that.

> So my next plan was to just sync that latest daily backup from
> server 1 to server 2 and use the --link-dest option on server 2 to
> link it to the previous day.
>
> *The Problem* This is the command I'm troubleshooting right now
>
> rsync -a -v -n -i --delete --link-dest=/backup-2016-02-01-0100
> --password-file=/media/external/scripts/offsite_rsync.pass
> /media/external/backup/backup-2016-02-02-0100
> backup@192.168.2.102::offsite
>
> It seems to be sending all files as new files (i.e. not picking up
> the link-dest option). I've tried using no slash at the beginning
> of link-dest, tried using ./ tried full path. etc.
>
> Here is a snippet of the output from server 1 which is running the
> command:
>
> <f+++++++++
> backup-2016-02-02-0100/media/external/owncloud/data/sam/files/Photos/2
007/20070120
>
>
DC Air and Space Museum/IMGP0906.JPG
>
>
> On server 2 there is no backup-2016-02-02-0100 directory. However,
> the link-dest option I'm using has the file on server 2:
>
> -rw------- 147 www-data www-data 4454193 Jan 20  2007
> /media/external/backup/backup-2016-02-01-0100/media/external/owncloud/
data/sam/files/Photos/2007/20070120
>
>
DC Air and Space Museum/IMGP0906.JPG
>
> Output of same file from server 1
>
> -rw------- 151 www-data www-data 4454193 Jan 20  2007
> /media/external/backup/backup-2016-02-02-0100/media/external/owncloud/
data/sam/files/Photos/2007/20070120
>
>
DC Air and Space Museum/IMGP0906.JPG

>
>
> *Troubleshooting* If I manually create the copy of server 2 first
> (cp -al backup-2016-02-01-0100 backup-2016-02-02-0100) and then run
> rsync without the --link-dest option I get the expected results.
> Only the files that changed or were removed/added are transferred.
>
> Also I tried running the actual transfer without -n and it is
> indeed transferring all the old files.
>
> Any help is appreciated.
>
>

- --
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
        Kevin Korb Phone:    (407) 252-6853
        Systems Administrator Internet:
        FutureQuest, Inc. [hidden email]  (work)
        Orlando, Florida [hidden email] (personal)
        Web page: http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAla4/lUACgkQVKC1jlbQAQcLsACfeC7yjw+ooxoKbYlupzcRYOb2
RiQAoN83Cy9t/7UMExoz1kG49556O6fS
=GPzH
-----END PGP SIGNATURE-----

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: --link-dest not working on remote server (running daemon)

Sam Holton
In reply to this post by Sam Holton
According to this thread it at least used to work https://lists.samba.org/archive/rsync/2006-April/015328.html

I don't want to use SSH because I'm running this on low power ARM boards which are already connected over VPN so no need for a second layer of transport encryption. I'm not sure what you mean by "link-dest and your target parameters misaligned". Both servers have the same directory structure so it may be a bit confusing. I'm trying to link to the previous day's backup on the remote server.

I forgot to mention that both servers are running 3.0.9  protocol version 30

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: --link-dest not working on remote server (running daemon)

Kevin Korb
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

by misaligned I meant that your source is a directory, your link-dest
is a directory (with the same name even), and your target is the root
of the share.  All 3 params should be directories and the link-dest
param should be a different date than the other two.

On 02/08/2016 04:24 PM, Sam Holton wrote:

> According to this thread it at least used to work
> https://lists.samba.org/archive/rsync/2006-April/015328.html
>
> I don't want to use SSH because I'm running this on low power ARM
> boards which are already connected over VPN so no need for a second
> layer of transport encryption. I'm not sure what you mean by
> "link-dest and your target parameters misaligned". Both servers
> have the same directory structure so it may be a bit confusing. I'm
> trying to link to the previous day's backup on the remote server.
>
> I forgot to mention that both servers are running 3.0.9  protocol
> version 30
>
>

- --
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
        Kevin Korb Phone:    (407) 252-6853
        Systems Administrator Internet:
        FutureQuest, Inc. [hidden email]  (work)
        Orlando, Florida [hidden email] (personal)
        Web page: http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAla5CtAACgkQVKC1jlbQAQcU2gCg7qicp4ZjcjCgSwEpqkjJhdWm
BvcAn0Uwi2A/LNDa0XS1PwtCnRPS2r5m
=npDn
-----END PGP SIGNATURE-----

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: --link-dest not working on remote server (running daemon)

Sam Holton
Thanks for the reply. The link-dest is different. It is Feb 1 while the source is Feb 2. 

I tried setting path = /media/external/ for the daemon and using 

rsync -a -v -i --delete --link-dest=backup-2016-02-01-0100 --password-file=/media/external/scripts/offsite_rsync.pass /media/external/backup/backup-2016-02-02-0100 backup@192.168.2.102::offsite/backup

So all three (link-dest, source, target) are directories but get the same result. The directory backup-2016-02-02-0100 is created on the target server and all files start to be transferred to it. Must be an issue with the way I'm setting link-dest


On Mon, Feb 8, 2016 at 4:42 PM Kevin Korb <[hidden email]> wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

by misaligned I meant that your source is a directory, your link-dest
is a directory (with the same name even), and your target is the root
of the share.  All 3 params should be directories and the link-dest
param should be a different date than the other two.

On 02/08/2016 04:24 PM, Sam Holton wrote:
> According to this thread it at least used to work
> https://lists.samba.org/archive/rsync/2006-April/015328.html
>
> I don't want to use SSH because I'm running this on low power ARM
> boards which are already connected over VPN so no need for a second
> layer of transport encryption. I'm not sure what you mean by
> "link-dest and your target parameters misaligned". Both servers
> have the same directory structure so it may be a bit confusing. I'm
> trying to link to the previous day's backup on the remote server.
>
> I forgot to mention that both servers are running 3.0.9  protocol
> version 30
>
>

- --
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
        Kevin Korb                      Phone:    (407) 252-6853
        Systems Administrator           Internet:
        FutureQuest, Inc.               [hidden email]  (work)
        Orlando, Florida                [hidden email] (personal)
        Web page:                       http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAla5CtAACgkQVKC1jlbQAQcU2gCg7qicp4ZjcjCgSwEpqkjJhdWm
BvcAn0Uwi2A/LNDa0XS1PwtCnRPS2r5m
=npDn
-----END PGP SIGNATURE-----

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: --link-dest not working on remote server (running daemon)

Kevin Korb
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

try: --link-dest=../backup-2016-02-01-0100

On 02/08/2016 04:51 PM, Sam Holton wrote:

> Thanks for the reply. The link-dest is different. It is Feb 1 while
> the source is Feb 2.
>
> I tried setting path = /media/external/ for the daemon and using
>
> rsync -a -v -i --delete --link-dest=backup-2016-02-01-0100
> --password-file=/media/external/scripts/offsite_rsync.pass
> /media/external/backup/backup-2016-02-02-0100
> backup@192.168.2.102::offsite/backup
>
> So all three (link-dest, source, target) are directories but get
> the same result. The directory backup-2016-02-02-0100 is created on
> the target server and all files start to be transferred to it. Must
> be an issue with the way I'm setting link-dest
>
>
> On Mon, Feb 8, 2016 at 4:42 PM Kevin Korb <[hidden email]
> <mailto:[hidden email]>> wrote:
>
> by misaligned I meant that your source is a directory, your
> link-dest is a directory (with the same name even), and your target
> is the root of the share.  All 3 params should be directories and
> the link-dest param should be a different date than the other two.
>
> On 02/08/2016 04:24 PM, Sam Holton wrote:
>> According to this thread it at least used to work
>> https://lists.samba.org/archive/rsync/2006-April/015328.html
>
>> I don't want to use SSH because I'm running this on low power
>> ARM boards which are already connected over VPN so no need for a
>> second layer of transport encryption. I'm not sure what you mean
>> by "link-dest and your target parameters misaligned". Both
>> servers have the same directory structure so it may be a bit
>> confusing. I'm trying to link to the previous day's backup on the
>> remote server.
>
>> I forgot to mention that both servers are running 3.0.9
>> protocol version 30
>
>
>
>
> -- Please use reply-all for most replies to avoid omitting the
> mailing list. To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync Before posting,
> read: http://www.catb.org/~esr/faqs/smart-questions.html
>

- --
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
        Kevin Korb Phone:    (407) 252-6853
        Systems Administrator Internet:
        FutureQuest, Inc. [hidden email]  (work)
        Orlando, Florida [hidden email] (personal)
        Web page: http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAla5EDEACgkQVKC1jlbQAQcjRwCeLXnOIk3F4ncxmYAvL0AbTKWA
PfkAoO/zF5ckV7D1eUMka/iXO+6nYMhZ
=H40g
-----END PGP SIGNATURE-----

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: --link-dest not working on remote server (running daemon)

Sam Holton
With the following server config:

log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock

[offsite]
   path = /media/external/
   comment = Offsite backup
   read only = no
   hosts allow = 192.168.2.0/24
   auth users = backup
   secrets file = /etc/rsyncd.scrt
   uid = 0
   gid = 0


I tried the following for --link-dest and they all tried to transfer all files

--link-dest=../backup-2016-02-01-0100
--link-dest=backup-2016-02-01-0100
--link-dest=/backup-2016-02-01-0100
--link-dest=/backup/backup-2016-02-01-0100
--link-dest=./backup/backup-2016-02-01-0100


On Mon, Feb 8, 2016 at 5:01 PM Kevin Korb <[hidden email]> wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

try: --link-dest=../backup-2016-02-01-0100

On 02/08/2016 04:51 PM, Sam Holton wrote:
> Thanks for the reply. The link-dest is different. It is Feb 1 while
> the source is Feb 2.
>
> I tried setting path = /media/external/ for the daemon and using
>
> rsync -a -v -i --delete --link-dest=backup-2016-02-01-0100
> --password-file=/media/external/scripts/offsite_rsync.pass
> /media/external/backup/backup-2016-02-02-0100
> backup@192.168.2.102::offsite/backup
>
> So all three (link-dest, source, target) are directories but get
> the same result. The directory backup-2016-02-02-0100 is created on
> the target server and all files start to be transferred to it. Must
> be an issue with the way I'm setting link-dest
>
>
> On Mon, Feb 8, 2016 at 4:42 PM Kevin Korb <[hidden email]
> <mailto:[hidden email]>> wrote:
>
> by misaligned I meant that your source is a directory, your
> link-dest is a directory (with the same name even), and your target
> is the root of the share.  All 3 params should be directories and
> the link-dest param should be a different date than the other two.
>
> On 02/08/2016 04:24 PM, Sam Holton wrote:
>> According to this thread it at least used to work
>> https://lists.samba.org/archive/rsync/2006-April/015328.html
>
>> I don't want to use SSH because I'm running this on low power
>> ARM boards which are already connected over VPN so no need for a
>> second layer of transport encryption. I'm not sure what you mean
>> by "link-dest and your target parameters misaligned". Both
>> servers have the same directory structure so it may be a bit
>> confusing. I'm trying to link to the previous day's backup on the
>> remote server.
>
>> I forgot to mention that both servers are running 3.0.9
>> protocol version 30
>
>
>
>
> -- Please use reply-all for most replies to avoid omitting the
> mailing list. To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync Before posting,
> read: http://www.catb.org/~esr/faqs/smart-questions.html
>

- --
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
        Kevin Korb                      Phone:    (407) 252-6853
        Systems Administrator           Internet:
        FutureQuest, Inc.               [hidden email]  (work)
        Orlando, Florida                [hidden email] (personal)
        Web page:                       http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAla5EDEACgkQVKC1jlbQAQcjRwCeLXnOIk3F4ncxmYAvL0AbTKWA
PfkAoO/zF5ckV7D1eUMka/iXO+6nYMhZ
=H40g
-----END PGP SIGNATURE-----

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: --link-dest not working on remote server (running daemon)

Christian Huldt
If you specify your target as offsite/backup I think you should specify
your link-dest as
--link-dest=offsite/backup/backup-2016-02-01-0100

...perhaps...


On 02/08/2016 11:10 PM, Sam Holton wrote:
> With the following server config: > > log file = /var/log/rsyncd.log > pid file = /var/run/rsyncd.pid >
lock file = /var/run/rsync.lock > > [offsite] >    path =
/media/external/ >    comment = Offsite backup >    read only = no >  
hosts allow = 192.168.2.0/24 <http://192.168.2.0/24> >    auth users =
backup >    secrets file = /etc/rsyncd.scrt >    uid = 0 >    gid = 0 >
> > I tried the following for --link-dest and they all tried to transfer
all files > > --link-dest=../backup-2016-02-01-0100 >
--link-dest=backup-2016-02-01-0100 > --link-dest=/backup-2016-02-01-0100
> --link-dest=/backup/backup-2016-02-01-0100 >
--link-dest=./backup/backup-2016-02-01-0100 > > > On Mon, Feb 8, 2016 at
5:01 PM Kevin Korb <[hidden email] <mailto:[hidden email]>> wrote: >

> try: --link-dest=../backup-2016-02-01-0100
>
> On 02/08/2016 04:51 PM, Sam Holton wrote:
> > Thanks for the reply. The link-dest is different. It is Feb 1 while
> > the source is Feb 2.
>
> > I tried setting path = /media/external/ for the daemon and using
>
> > rsync -a -v -i --delete --link-dest=backup-2016-02-01-0100
> > --password-file=/media/external/scripts/offsite_rsync.pass
> > /media/external/backup/backup-2016-02-02-0100
> > backup@192.168.2.102::offsite/backup
>
> > So all three (link-dest, source, target) are directories but get
> > the same result. The directory backup-2016-02-02-0100 is created on
> > the target server and all files start to be transferred to it. Must
> > be an issue with the way I'm setting link-dest
>
>
> > On Mon, Feb 8, 2016 at 4:42 PM Kevin Korb <[hidden email]
> <mailto:[hidden email]>
> > <mailto:[hidden email] <mailto:[hidden email]>>> wrote:
>
> > by misaligned I meant that your source is a directory, your
> > link-dest is a directory (with the same name even), and your target
> > is the root of the share.  All 3 params should be directories and
> > the link-dest param should be a different date than the other two.
>
> > On 02/08/2016 04:24 PM, Sam Holton wrote:
> >> According to this thread it at least used to work
> >> https://lists.samba.org/archive/rsync/2006-April/015328.html
>
> >> I don't want to use SSH because I'm running this on low power
> >> ARM boards which are already connected over VPN so no need for a
> >> second layer of transport encryption. I'm not sure what you mean
> >> by "link-dest and your target parameters misaligned". Both
> >> servers have the same directory structure so it may be a bit
> >> confusing. I'm trying to link to the previous day's backup on the
> >> remote server.
>
> >> I forgot to mention that both servers are running 3.0.9
> >> protocol version 30
>
>
>
>
> > -- Please use reply-all for most replies to avoid omitting the
> > mailing list. To unsubscribe or change options:
> > https://lists.samba.org/mailman/listinfo/rsync Before posting,
> > read: http://www.catb.org/~esr/faqs/smart-questions.html
>
>
> > >


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

signature.asc (501 bytes) Download Attachment