rsync with 8bit file names?

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

rsync with 8bit file names?

@lbutlr
When trying to sync my TV folder to a mirror drive, episodes with non-ASCII characters in them cannot be processed by rsync. Anything I can do about this?

For example, I get this:

building file list ...
file has vanished: "/Volumes/TV/Raising Hope/Raising Hope - S04E01 - De?\#201ja Vu Man.mkv"

This is rsync under OS X:

$ rsync --version
rsync  version 2.6.9  protocol version 29
Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others.
<http://rsync.samba.org/>
Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles,
              inplace, IPv6, 64-bit system inums, 64-bit internal inums

I'd check macports, but since it's not been updated for 10.10 yet I'll need to wait a while.

--
Passion is the pill you can swallow forever Taking them one by one One
by One --Agents of Good Roots "Come On"

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync with 8bit file names?

@lbutlr

> On 11 Sep 2014, at 06:36 , LuKreme <[hidden email]> wrote:
>
> $ rsync --version
> rsync  version 2.6.9  protocol version 29

(I did install rsync 3.1.1, same results)

--
U is for UNA who slipped down a drain
V is for VICTOR squashed by a train

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync with 8bit file names?

Holger Hoffstaette
On Thu, 11 Sep 2014 07:58:15 -0600, LuKreme wrote:

>> On 11 Sep 2014, at 06:36 , LuKreme <[hidden email]> wrote:
>>
>> $ rsync --version
>> rsync  version 2.6.9  protocol version 29
>
> (I did install rsync 3.1.1, same results)

I have no problems rsync-ing files with German characters, but this is
on Linux. I've noticed that it is built with iconv support, so maybe
that's missing on your end(s)?

-h

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync with 8bit file names?

Thomas Floeren
On 11 Sep 2014, at 17:05, Holger Hoffstätte <[hidden email]> wrote:

> On Thu, 11 Sep 2014 07:58:15 -0600, LuKreme wrote:
>
>>> On 11 Sep 2014, at 06:36 , LuKreme <[hidden email]> wrote:
>>>
>>> $ rsync --version
>>> rsync  version 2.6.9  protocol version 29
>>
>> (I did install rsync 3.1.1, same results)
>
> I have no problems rsync-ing files with German characters, but this is
> on Linux. I've noticed that it is built with iconv support, so maybe
> that's missing on your end(s)?
>
No problems here under OS X 10.9.5. iconv and libiconv.2.dylib are present in /usr/bin and /usr/lib (it comes with OS X).

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync with 8bit file names?

@lbutlr
On 11 Sep 2014, at 11:49 , Thomas Floeren <[hidden email]> wrote:
> No problems here under OS X 10.9.5. iconv and libiconv.2.dylib are present in /usr/bin and /usr/lib (it comes with OS X).

Maybe it is the Drobo that is mucking up filenames then? The share is mounted used afp, so I figured that was not the issue:

Drobo5N._afpovertcp._tcp.local/TV /Volumes/TV


--
By the authority vested in me by Kaiser William II, I pronounce you man
and wife. Proceed with the execution.

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync with 8bit file names?

Henri Shustak
In reply to this post by @lbutlr
Hello,

Guessing the source drive is formatted HFS+ or HFS+J. What is the file system on the destination drive? I know that rsync is capable of correctly coping over file and folder names on Mac OS X which contain non-ASCII characters. Hence my question relating to the destination file system?

The following command should provide you with an answer :
diskutil info /Volumes/TV | grep "File System Personality" | awk -F "File System Personality:  " '{print $2}'

Maybe someone has a better way to work out the FileSystem volume which they can share? Typically there are many ways to skin a cat and that some approaches are more elegant than others.

--------------------------------------------------------------------
This email is protected by LBackup, an open source backup solution
http://www.lbackup.org

On 12/09/2014, at 12:36 AM, LuKreme <[hidden email]> wrote:

> When trying to sync my TV folder to a mirror drive, episodes with non-ASCII characters in them cannot be processed by rsync. Anything I can do about this?
>
> For example, I get this:
>
> building file list ...
> file has vanished: "/Volumes/TV/Raising Hope/Raising Hope - S04E01 - De?\#201ja Vu Man.mkv"
>
> This is rsync under OS X:
>
> $ rsync --version
> rsync  version 2.6.9  protocol version 29
> Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others.
> <http://rsync.samba.org/>
> Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles,
>              inplace, IPv6, 64-bit system inums, 64-bit internal inums
>
> I'd check macports, but since it's not been updated for 10.10 yet I'll need to wait a while.
>
> --
> Passion is the pill you can swallow forever Taking them one by one One
> by One --Agents of Good Roots "Come On"
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync with 8bit file names?

Henri Shustak
Hello,

Is the source Drobo a NAS or DAS unit?

Also, I am guessing that the /Volumes/TV is the source for the rsync. Is it possible that the file has actually moved or been renamed?

If the Drobo (hosted) volume is the source for the rsync command, then perhaps checking the file system format of that device is also worth while?

Hope this helps. Perhaps someone else on the list has some ideas?

I would suggest you consider compiling a recent version of rsync for your system. This should be quite straight forward on 10.10 (beta release). However, I must admit I have yet to try compiling rsync on 10.10 : http://www.lbackup.org/developer/rsync_hfs

Finally, what is the actual command you are running. This information may shed some light on why the command is failing.

Hope this helps.


---------------------------------------------------------------------
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org

On 12/09/2014, at 12:40 PM, LuKreme <[hidden email]> wrote:

> On 11 Sep 2014, at 16:22 , Henri Shustak <[hidden email]> wrote:
>> Guessing the source drive is formatted HFS+ or HFS+J.
>
> The source drive is a Drobo. The destination is Journaled HFS+.
>
> --
> 'I cannot! He has been kindness itself to me!' 'And you can be Death
> itself to him.'
>

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync with 8bit file names?

@lbutlr

> On 11 Sep 2014, at 19:31 , Henri Shustak <[hidden email]> wrote:
>
> Hello,
>
> Is the source Drobo a NAS or DAS unit?

Drobo 5N NAS.

> Also, I am guessing that the /Volumes/TV is the source for the rsync. Is it possible that the file has actually moved or been renamed?

No, the file is there.

> If the Drobo (hosted) volume is the source for the rsync command, then perhaps checking the file system format of that device is also worth while?

I am running rising on my Mac which has the Drobo’s TV share mounted at /Volumes/TV

> I would suggest you consider compiling a recent version of rsync for your system. This should be quite straight forward on 10.10 (beta release). However, I must admit I have yet to try compiling rsync on 10.10 : http://www.lbackup.org/developer/rsync_hfs

I installed rsync 3.1.1 and had the same result.

> Finally, what is the actual command you are running. This information may shed some light on why the command is failing.

rysnc -aP --exclude=“.*” /Volumes/TV /Volumes/VAULT

(where VAULT is a desktop HFS+J volume)
 

--
'Never say die, master. That's our motto, eh?' I CAN'T SAY IT'S EVER
REALLY BEEN MINE. --Hogfather

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync with 8bit file names?

Thomas Floeren
I just copied a file with Umlauts from the Mac to a smb mount and back, and from the Mac to an afp mount and back. No problems so far.

I’m using Homebrew’s rsync build (see below). Did you try the Homebrew build?

➜  ~  rsync --version
rsync  version 3.1.1  protocol version 31
Copyright (C) 1996-2014 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
   64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
   socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
   append, ACLs, xattrs, iconv, symtimes, no prealloc, file-flags

--
Thomas
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync with 8bit file names?

Henri Shustak
In reply to this post by @lbutlr

> When trying to sync my TV folder to a mirror drive, episodes with non-ASCII characters in them cannot be processed by rsync. Anything I can do about this?

(Q1) What do you mean by a mirror drive? Is this a RAID1 external enclosure or some sort of softRAID? Or is it just a copy of the TV folder onto another device?

(Q2) Is your Drobo's firmware all up-to date?

(Q3) Have you tried via SMB rather than AFP?


--------------------------------------------------------------------
This email is protected by LBackup, an open source backup solution
http://www.lbackup.org
@kreme.com> wrote:

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync with 8bit file names?

Henri Shustak
In reply to this post by @lbutlr
Hello again,

Three further questions :

(Q2-1) Have you tried this same setup when booted from another OS (e.g. 10.9)?

(Q2-2) Have you tried this using some other hardware?

(Q2-3) Have you tried coping this specific file over using the Finder (say to the Desktop) and then using rsync to copy it over to that destination drive?

--------------------------------------------------------------------
This email is protected by LBackup, an open source backup solution
http://www.lbackup.org

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync with 8bit file names?

Henri Shustak
In reply to this post by Henri Shustak
>>> When trying to sync my TV folder to a mirror drive, episodes with non-ASCII characters in them cannot be processed by rsync. Anything I can do about this?
>>
>> (Q1) What do you mean by a mirror drive? Is this a RAID1 external enclosure or some sort of softRAID? Or is it just a copy of the TV folder onto another device?
>
> A copy of the TV folder.

Okay, thanks for confirmation on this point. Have you confirmed that the file system is good on the destination directory. IE : Running Disk Utility to check the file system?


>> (Q2) Is your Drobo's firmware all up-to date?
> Yes
Great.

>> (Q3) Have you tried via SMB rather than AFP?
> No, but my understanding is that afp in 10.9 and 10.10 *si* smb.
I do not understand your reply? What is "*si*"?

>> (Q2-1) Have you tried this same setup when booted from another OS (e.g. 10.9)?
> No, I moved my machine to the DP with DP7 at the same time that I added this new external store to mirror my TV folder to.
I would strongly suggest trying this out on a system running 10.9 (not beta software) and seeing if this resolves the problem.

>> (Q2-2) Have you tried this using some other hardware?
> Limited options. I suppose i could move it to a Mac mini that is still running 10.9 to test, but then again, it’s about 15 files I can manually deal with out of 7TB which seems like less trouble that moving hardware around.
Sure, but you have asked for assistance from the rsync mailing list. It would be good to have answers to the following :
 Is there a problem on 10.10?
 Is there a problem with some of the hardware you are using?

>> (Q2-3) Have you tried coping this specific file over using the Finder (say to the Desktop) and then using rsync to copy it over to that destination drive?
> I copied the file in the finder to the destination, I did not try copying it and the using rsync locally, but I can do that.

I would give that a try and see if it makes any difference. If that resolves the problem then I suspect it is either something to do with the Drobo, 10.10 or even the network?

Hope this helps.

---------------------------------------------------------------------
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync with 8bit file names?

Thomas Floeren
Hi LuKreme,

in the meantime I moved two of my Macs to 10.10 (DP8; 14A361c) and I repeated the test (copying a umlaut file to and from an afp mount; see my mail from 16 Sep) with the same result, i.e. no problems. Didn’t test it with DP7 though.

Did you try to install rsync 3.1.1 via Homebrew? If it works with the HB install you know at least that something is fishy with your current rsync (or that dependencies are missing).

--
Thomas




> On 19 Sep 2014, at 2:39, Henri Shustak <[hidden email]> wrote:
>
>>>> When trying to sync my TV folder to a mirror drive, episodes with non-ASCII characters in them cannot be processed by rsync. Anything I can do about this?
>>>
>>> (Q1) What do you mean by a mirror drive? Is this a RAID1 external enclosure or some sort of softRAID? Or is it just a copy of the TV folder onto another device?
>>
>> A copy of the TV folder.
>
> Okay, thanks for confirmation on this point. Have you confirmed that the file system is good on the destination directory. IE : Running Disk Utility to check the file system?
>
>
>>> (Q2) Is your Drobo's firmware all up-to date?
>> Yes
> Great.
>
>>> (Q3) Have you tried via SMB rather than AFP?
>> No, but my understanding is that afp in 10.9 and 10.10 *si* smb.
> I do not understand your reply? What is "*si*"?
>
>>> (Q2-1) Have you tried this same setup when booted from another OS (e.g. 10.9)?
>> No, I moved my machine to the DP with DP7 at the same time that I added this new external store to mirror my TV folder to.
> I would strongly suggest trying this out on a system running 10.9 (not beta software) and seeing if this resolves the problem.
>
>>> (Q2-2) Have you tried this using some other hardware?
>> Limited options. I suppose i could move it to a Mac mini that is still running 10.9 to test, but then again, it’s about 15 files I can manually deal with out of 7TB which seems like less trouble that moving hardware around.
> Sure, but you have asked for assistance from the rsync mailing list. It would be good to have answers to the following :
> Is there a problem on 10.10?
> Is there a problem with some of the hardware you are using?
>
>>> (Q2-3) Have you tried coping this specific file over using the Finder (say to the Desktop) and then using rsync to copy it over to that destination drive?
>> I copied the file in the finder to the destination, I did not try copying it and the using rsync locally, but I can do that.
>
> I would give that a try and see if it makes any difference. If that resolves the problem then I suspect it is either something to do with the Drobo, 10.10 or even the network?
>
> Hope this helps.
>
> ---------------------------------------------------------------------
> This email is protected by LBackup, an open source backup solution.
> http://www.lbackup.org
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: rsync with 8bit file names?

@lbutlr

> On 21 Sep 2014, at 13:55 , Thomas Floeren <[hidden email]> wrote:
>
> in the meantime I moved two of my Macs to 10.10 (DP8; 14A361c) and I repeated the test (copying a umlaut file to and from an afp mount; see my mail from 16 Sep) with the same result, i.e. no problems. Didn’t test it with DP7 though.

rsync is working properly, the problem as it turns out is isolated to the Drobo. I was mislead, because I THOUGHT the finder copies were working. As I said in my last email:

> But, I was wrong about the Finder copy. The 8bit file names APPEARED to copy, and gave no error, and made the completion noise, but did not, in point of fact, copy. So this is a Drobo issue and not a rsync issue.

Thanks for your time.

--
You and me Sunday driving Not arriving

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html