Samba › Samba - rsync

Problem syncing to Netapp (rsync: failed to set times on...)

Classic

List

Threaded

8 messages Options

BillDorrian

Problem syncing to Netapp (rsync: failed to set times on...)

Hey folks.

I have a machine that I use as an intermediary to rsync between a NAS in another building (the users there have poor bandwidth and need local storage) and the our Netapp located in our Datacenter. I get lots of the "rsync: failed to set times on..." errors, and files which have already been transferred just try to sync again anyway.

These are the mount options that I use for both sides:

mount -t cifs -o credentials=/root/.synccreds //nasdevice/folder /nas
mount -t cifs -o credentials=/root/.synccreds //netapp/folder /netapp

The ".synccreds" file has the credentials of an Active Directory Domain Admin account, which has "Full Control" on both the NAS and the Netapp.

Here is the command that I run to do the rsync:

rsync -rvt --delete --progress /nas/ /netapp/

Running rsync with "-i" shows that the files are transferring because of timestamp differences.

I tried the "-c" option in place of the "-t", but the server doing the sync just hung there for literally two days without anything transferring and no output. I realize that the -c option is slower, but yikes!

Any thoughts? Suggestions?

Thanks,
Bill D.

Bill Dorrian
Network Administrator
Desk: 904-273-7625
Cell: 904-859-9471

" THE PLAYERS begins May 10 on Golf's Greatest Stadium. For more information visit PGATOUR.COM/theplayers."

"The information contained in this transmission and any attachments may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message."

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Joachim Otahal (privat)

Re: Problem syncing to Netapp (rsync: failed to set times on...)

I know from a lot of NAS boxes that they tend to use their internal time to stamp files instead of the time given by a copy job.
The easiest way to test is to deliberately set the time off by a few hours on the box you monted the stuff on, the NAS and netapp (or the Server accessing the netapp) and create a file from your mount point, and check whether the time is right.

cifs isn't the fastest way in unix environments, and samba uses _quite_ an amount of CPU power if you are above 50 MB/s.
If in any way possible you should do it more directly, having an in-between box in the network causes (if -c is used) that all files are read from both boxes over the network just for the checksum, hence the bad performance.

These Netapps, are they "pure" storage and the server using them is either windows or linux? If yes: Put rsync directly on those servers. This also applies to most NAS boxes I know, they offer rsync directly, most of the time as server.

Joachim

[hidden email] schrieb:

Hey folks.

I have a machine that I use as an intermediary to rsync between a NAS in another building (the users there have poor bandwidth and need local storage) and the our Netapp located in our Datacenter. I get lots of the "rsync: failed to set times on..." errors, and files which have already been transferred just try to sync again anyway.

These are the mount options that I use for both sides:

mount -t cifs -o credentials=/root/.synccreds //nasdevice/folder /nas
mount -t cifs -o credentials=/root/.synccreds //netapp/folder /netapp

The ".synccreds" file has the credentials of an Active Directory Domain Admin account, which has "Full Control" on both the NAS and the Netapp.

Here is the command that I run to do the rsync:

rsync -rvt --delete --progress /nas/ /netapp/

Running rsync with "-i" shows that the files are transferring because of timestamp differences.

I tried the "-c" option in place of the "-t", but the server doing the sync just hung there for literally two days without anything transferring and no output. I realize that the -c option is slower, but yikes!

Any thoughts? Suggestions?

Thanks,
Bill D.

Bill Dorrian
Network Administrator
Desk: 904-273-7625
Cell: 904-859-9471
" THE PLAYERS begins May 10 on Golf's Greatest Stadium. For more information visit PGATOUR.COM/theplayers."

"The information contained in this transmission and any attachments may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message."

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Joachim Otahal (privat)

Re: Problem syncing to Netapp (rsync: failed to set times on...)

In reply to this post by BillDorrian

PS: Another (ugly) workaround: Use two linux boxes, place then both on
each side of the slow line. One side having the NAS mounted + running
rsync server, the other having the netapp mounted.
Then sync between those two linux boxes. Even if you have to use -c or
--ignore-times the "full read just for checksum" would only happen in
the LAN of each networks while the slow line has only the burden of the
checksums and actual transfers.

Joachim

[hidden email] schrieb:

> Hey folks.
>
> I have a machine that I use as an intermediary to rsync between a NAS
> in another building (the users there have poor bandwidth and need
> local storage) and the our Netapp located in our Datacenter. I get
> lots of the "rsync: failed to set times on..." errors, and files which
> have already been transferred just try to sync again anyway.
>
> These are the mount options that I use for both sides:
>
> mount -t cifs -o credentials=/root/.synccreds //nasdevice/folder /nas
> mount -t cifs -o credentials=/root/.synccreds //netapp/folder /netapp
>
>
> The ".synccreds" file has the credentials of an Active Directory
> Domain Admin account, which has "Full Control" on both the NAS and the
> Netapp.
>
>
> Here is the command that I run to do the rsync:
>
> rsync -rvt --delete --progress /nas/ /netapp/
>
> Running rsync with "-i" shows that the files are transferring because
> of timestamp differences.
>
> I tried the "-c" option in place of the "-t", but the server doing the
> sync just hung there for literally two days without anything
> transferring and no output. I realize that the -c option is slower,
> but yikes!
>
> Any thoughts? Suggestions?
>
> Thanks,
> Bill D.
>
>
> Bill Dorrian
> Network Administrator
> Desk: 904-273-7625
> Cell: 904-859-9471

Greg Deback (rsync)

Re: Problem syncing to Netapp (rsync: failed to set times on...)

In reply to this post by Joachim Otahal (privat)

On Mon, Apr 9, 2012 at 10:32 PM, Joachim Otahal (privat) <[hidden email]> wrote:

I know from a lot of NAS boxes that they tend to use their internal time to stamp files instead of the time given by a copy job.
The easiest way to test is to deliberately set the time off by a few hours on the box you monted the stuff on, the NAS and netapp (or the Server accessing the netapp) and create a file from your mount point, and check whether the time is right.

cifs isn't the fastest way in unix environments, and samba uses _quite_ an amount of CPU power if you are above 50 MB/s.
If in any way possible you should do it more directly, having an in-between box in the network causes (if -c is used) that all files are read from both boxes over the network just for the checksum, hence the bad performance.

These Netapps, are they "pure" storage and the server using them is either windows or linux? If yes: Put rsync directly on those servers. This also applies to most NAS boxes I know, they offer rsync directly, most of the time as server.

Joachim

[hidden email] schrieb:
Hey folks.

I have a machine that I use as an intermediary to rsync between a NAS in another building (the users there have poor bandwidth and need local storage) and the our Netapp located in our Datacenter. I get lots of the "rsync: failed to set times on..." errors, and files which have already been transferred just try to sync again anyway.

These are the mount options that I use for both sides:

mount -t cifs -o credentials=/root/.synccreds //nasdevice/folder /nas
mount -t cifs -o credentials=/root/.synccreds //netapp/folder /netapp

The ".synccreds" file has the credentials of an Active Directory Domain Admin account, which has "Full Control" on both the NAS and the Netapp.

Here is the command that I run to do the rsync:

rsync -rvt --delete --progress /nas/ /netapp/

Running rsync with "-i" shows that the files are transferring because of timestamp differences.

I tried the "-c" option in place of the "-t", but the server doing the sync just hung there for literally two days without anything transferring and no output. I realize that the -c option is slower, but yikes!

Any thoughts? Suggestions?

Thanks,
Bill D.

Bill Dorrian
Network Administrator
Desk: 904-273-7625
Cell: 904-859-9471
" THE PLAYERS begins May 10 on Golf's Greatest Stadium. For more information visit PGATOUR.COM/theplayers."

"The information contained in this transmission and any attachments may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message."

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

BillDorrian

Re: Problem syncing to Netapp (rsync: failed to set times on...)

Thanks for the suggestions, guys.

I'll try testing times on the NAS device, as well as the --size-only options. These are photos - I wonder what the odds are of a modified file having the same size as the original?

Bill Dorrian
Network Administrator
Desk: 904-273-7625
Cell: 904-859-9471

From:	"Greg Deback (rsync)" <[hidden email]>
To:	"Joachim Otahal (privat)" <[hidden email]>,
Cc:	[hidden email], [hidden email]
Date:	04/09/2012 04:48 PM
Subject:	Re: Problem syncing to Netapp (rsync: failed to set times on...)
Sent by:	[hidden email]

PPS. Yet another (a bit ugly) workaround, regarding how often the job is done : try the options --modify-window=N (N in seconds), to allow timestamp differences, or/and --size-only (comparing on filesize only).
Greg

On Mon, Apr 9, 2012 at 10:32 PM, Joachim Otahal (privat) <Jou@...> wrote:
I know from a lot of NAS boxes that they tend to use their internal time to stamp files instead of the time given by a copy job.
The easiest way to test is to deliberately set the time off by a few hours on the box you monted the stuff on, the NAS and netapp (or the Server accessing the netapp) and create a file from your mount point, and check whether the time is right.

cifs isn't the fastest way in unix environments, and samba uses _quite_ an amount of CPU power if you are above 50 MB/s.
If in any way possible you should do it more directly, having an in-between box in the network causes (if -c is used) that all files are read from both boxes over the network just for the checksum, hence the bad performance.

These Netapps, are they "pure" storage and the server using them is either windows or linux? If yes: Put rsync directly on those servers. This also applies to most NAS boxes I know, they offer rsync directly, most of the time as server.

Joachim

BillDorrian@...schrieb:
Hey folks.

I have a machine that I use as an intermediary to rsync between a NAS in another building (the users there have poor bandwidth and need local storage) and the our Netapp located in our Datacenter. I get lots of the "rsync: failed to set times on..." errors, and files which have already been transferred just try to sync again anyway.

These are the mount options that I use for both sides:

mount -t cifs -o credentials=/root/.synccreds //nasdevice/folder /nas
mount -t cifs -o credentials=/root/.synccreds //netapp/folder /netapp

The ".synccreds" file has the credentials of an Active Directory Domain Admin account, which has "Full Control" on both the NAS and the Netapp.

Here is the command that I run to do the rsync:

rsync -rvt --delete --progress /nas/ /netapp/

Running rsync with "-i" shows that the files are transferring because of timestamp differences.

I tried the "-c" option in place of the "-t", but the server doing the sync just hung there for literally two days without anything transferring and no output. I realize that the -c option is slower, but yikes!

Any thoughts? Suggestions?

Thanks,
Bill D.

Bill Dorrian
Network Administrator
Desk: 904-273-7625
Cell: 904-859-9471

" THE PLAYERS begins May 10 on Golf's Greatest Stadium. For more information visit PGATOUR.COM/theplayers."

Kyle Lanclos

Re: Problem syncing to Netapp (rsync: failed to set times on...)

Bill Dorrian wrote:
> These are photos - I wonder what the odds are of a modified file
> having the same size as the original?

If someone modifies the EXIF metadata (say, to correct a 'picture taken on'
timestamp for a camera that wasn't properly synchronized), the file size
would likely remain the same.

--Kyle
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Stier, Matthew

RE: Problem syncing to Netapp (rsync: failed to set times on...)

But the timestamp would not.

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Kyle Lanclos
Sent: Monday, April 09, 2012 5:06 PM
To: [hidden email]
Cc: [hidden email]
Subject: Re: Problem syncing to Netapp (rsync: failed to set times on...)

Bill Dorrian wrote:
> These are photos - I wonder what the odds are of a modified file
> having the same size as the original?

If someone modifies the EXIF metadata (say, to correct a 'picture taken on'
timestamp for a camera that wasn't properly synchronized), the file size
would likely remain the same.

--Kyle
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Joachim Otahal (privat)

Re: Problem syncing to Netapp (rsync: failed to set times on...)

Stier, Matthew schrieb:
> But the timestamp would not.

Be careful with that, I had cases where picture editors kept the
timestamps even if they did change the content. Only atime was changed,
mtime stayed. The affected users had the option selected in the program
(they thought of it as a speed up option), once it was changed it was
the way it should. So I run -c weekly and check how many are
transferred. Sometimes surprises come up.

Joachim

> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Kyle Lanclos
> Sent: Monday, April 09, 2012 5:06 PM
> To: [hidden email]
> Cc: [hidden email]
> Subject: Re: Problem syncing to Netapp (rsync: failed to set times on...)
>
> Bill Dorrian wrote:
>> These are photos - I wonder what the odds are of a modified file
>> having the same size as the original?
> If someone modifies the EXIF metadata (say, to correct a 'picture taken on'
> timestamp for a camera that wasn't properly synchronized), the file size
> would likely remain the same.
>
> --Kyle