backup to NTFS USB disk

[joop gerritse]

Hello, *

I am setting up a backup on a Linux system with Windows XP workstations. The backup goes to three alternating usb drives, each of which is NTFS formatted. The disks should be virtually identical but they do not seem to be.

First, my mount command is this (I edited a bit for brevity)

mount -t ntfs-3g -o locale=nl_NL.iso-8859-1,silent /dev/disk/by-id/usb-DiskA \ /mnt/tmp ||

mount -t ntfs-3g -o locale=nl_NL.iso-9959-1,silent /dev/disk/by-id/usb-DiskB \ /mnt/tmp ||

mount -t ntfs-3g -o locale=nl_NL.iso-9959-1,silent /dev/disk/by-id/usb-DiskC \ /mnt/tmp

if [ $? -eq 0 ]

then

<backup starts>

So, it first attempts to mount diskA; if that fails, it tries diskB, and so on.

DiskA has been running for a week without any problems or messages. Now I try to mount diskB, and I get:

ntfs-3g: Failed to access volume '/dev/disk/by-id/usb-DiskA': No such file or directory

Please type '/sbin/mount.ntfs-3g --help' for more information.

WARNING: Couldn't set locale to 'nl_NL.iso-9959-1' thus some file names may not

be correct or visible. Please see the potential solution at

http://ntfs-3g.org/support.html#locale

LiMu Backup Disk B

The first message is, of course, correct, because diskA is now no longer present. The second mount, of diskB, succeeds. Now pops up a warning, and I am not sure if it belongs to the absent diskA or to the present diskB. But apparently the latter is the case, because I get hundreds of messages like this:

rsync: recv_generator: failed to stat "/mnt/tmp/backup/museum/adlib/adlib documenten voor/catalogus objecten in eigen bezit/artikelen, algemeen/8467 wehrpa\#303\#237.doc": Invalid or incomplete multibyte or wide character (84)

rsync: recv_generator: failed to stat "/mnt/tmp/backup/museum/documenten/bestuur/archief, niet geschoond/publiciteit/idee\#343\#253n website.doc": Invalid or incomplete multibyte or wide character (84)

(I tried the link mentioned in the warning, bit it led me to www.tuxera.com, and does not seem relevant)

Now I have two questions:

1) why does DiskA (NTFS) accept this mount, while DiskB (NTFS) doesn't?

2) what is wrong with a file name like "ideeën website.doc", which is generated on an XT system, is written to an NTFS file system by SAMBA, and then refused on copying to another NTFS file system?

--

joop gerritse

Mühlenstraße 11

D-47546 Kalkar-Wissel

Germany

[phone]

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[joop gerritse]

On Thursday 12 April 2012 11:20:54 joop g wrote:

(I forgot to include my rsync commands)

> mount -t ntfs-3g -o locale=nl_NL.iso-8859-1,silent

> /dev/disk/by-id/usb-DiskA \ /mnt/tmp ||

> mount -t ntfs-3g -o locale=nl_NL.iso-9959-1,silent

> /dev/disk/by-id/usb-DiskB \ /mnt/tmp ||

> mount -t ntfs-3g -o locale=nl_NL.iso-9959-1,silent

> /dev/disk/by-id/usb-DiskC \ /mnt/tmp

> if [ $? -eq 0 ]

> then

> <backup starts>

/usr/bin/rsync -a /windows/ /mnt/tmp/backup/windows

/usr/bin/rsync -a /home/ /mnt/tmp/backup/home

/usr/local/bin/rsync -rltDh --no-group --no-owner --delete --modify-window=1 -stats --exclude='mail/vmware-data/' /museum/ /mnt/tmp/backup/museum

as you see, the last command reflects quite a few attempts to solve the problem, so far without success...

--

joop gerritse

Mühlenstraße 11

D-47546 Kalkar-Wissel

Germany

[phone]

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Joe]

Since no one has replied yet, I have one idea that *might* point to part
of your problem.  I've never had to deal with locale issues, so I have no
idea about that.

You have directories and file names with blanks in them.  In general, this
causes a lot of trouble for a lot of programs.  I'm not sure how rsync
handles them.

If no one chimes in with a more informed suggestion, you might try to find
some portion of the file tree you need to backup where there are no blanks
in file or directory names and run that as a test to see if the blanks are
getting you.

man rsync says:

       If  you  need  to  transfer  a filename that contains whitespace,
you can either specify the --protect-args (-s) option, or
       you’ll need to escape the whitespace in a way that the remote shell
will understand.  For instance:

              rsync -av host:'file\ name\ with\ spaces' /dest

This seems to only apply to actual arguments to the rsync command.

There's also a -0 option:

      -0, --from0
              This tells rsync that the rules/filenames it reads from a
file are terminated by a null (’\0’) character, not a  NL,
              CR, or CR+LF.  This affects --exclude-from, --include-from,
--files-from, and any merged files specified in a --fil‐
              ter rule.  It does not affect --cvs-exclude (since all names
read from a .cvsignore file are split on whitespace).

But that also just seems to apply to arguments from a file.

HTH

Joe

> Hello, *
>
> I am setting up a backup on a Linux system with Windows XP workstations.
> The
> backup goes to three alternating usb drives, each of which is NTFS
> formatted.
> The disks should be virtually identical but they do not seem to be.
>
> First, my mount command is this (I edited a bit for brevity)
>
> mount -t ntfs-3g -o locale=nl_NL.iso-8859-1,silent
> /dev/disk/by-id/usb-DiskA \
> /mnt/tmp ||
> mount -t ntfs-3g -o locale=nl_NL.iso-9959-1,silent
> /dev/disk/by-id/usb-DiskB \
> /mnt/tmp ||
> mount -t ntfs-3g -o locale=nl_NL.iso-9959-1,silent
> /dev/disk/by-id/usb-DiskC \
> /mnt/tmp
> if [ $? -eq 0 ]
> then
> <backup starts>
>
> So, it first attempts to mount diskA; if that fails, it tries diskB, and
> so on.
>
> DiskA has been running for a week without any problems or messages. Now I
> try
> to mount diskB, and I get:
>
> ntfs-3g: Failed to access volume '/dev/disk/by-id/usb-DiskA': No such file
> or
> directory
> Please type '/sbin/mount.ntfs-3g --help' for more information.
> WARNING: Couldn't set locale to 'nl_NL.iso-9959-1' thus some file names
> may not
>          be correct or visible. Please see the potential solution at
>          http://ntfs-3g.org/support.html#locale
> LiMu Backup Disk B
>
> The first message is, of course, correct, because diskA is now no longer
> present. The second mount, of diskB, succeeds. Now pops up a warning, and
> I am
> not sure if it belongs to the absent diskA or to the present diskB. But
> apparently the latter is the case, because I get hundreds of messages like
> this:
>
> rsync: recv_generator: failed to stat "/mnt/tmp/backup/museum/adlib/adlib
> documenten voor/catalogus objecten in eigen bezit/artikelen, algemeen/8467
> wehrpa\#303\#237.doc": Invalid or incomplete multibyte or wide character
> (84)
> rsync: recv_generator: failed to stat
> "/mnt/tmp/backup/museum/documenten/bestuur/archief, niet
> geschoond/publiciteit/idee\#343\#253n website.doc": Invalid or incomplete
> multibyte or wide character (84)
>
>
> (I tried the link mentioned in the warning, bit it led me to
> www.tuxera.com,
> and does not seem relevant)
>
> Now I have two questions:
> 1) why does DiskA (NTFS) accept this mount, while DiskB (NTFS) doesn't?
> 2) what is wrong with a file name like "ideeën website.doc", which is
> generated
> on an XT system, is written to an NTFS file system by SAMBA, and then
> refused
> on copying to another NTFS file system?
>
> --
> joop gerritse
> Mühlenstraße 11
> D-47546 Kalkar-Wissel
> Germany
> [phone]
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[joop gerritse]

On Friday 13 April 2012 21:22:42 [hidden email] wrote:

> Since no one has replied yet, I have one idea that *might* point to part

> of your problem. I've never had to deal with locale issues, so I have no

> idea about that.

>

> You have directories and file names with blanks in them. In general, this

> causes a lot of trouble for a lot of programs. I'm not sure how rsync

> handles them.

It is very kind of you to at least give it a try, but I am quite sure that spaces are not the problem. In fact, I have been experimenting further, and the situation is even more complicated.

First of all, I have to apologize to the list for my first question, regarding

> > WARNING: Couldn't set locale to 'nl_NL.iso-9959-1' thus some file names

> > may not

> >

> > be correct or visible. Please see the potential solution at

> > http://ntfs-3g.org/support.html#locale

I could try, of course, but it was clearly off-list.

Before I am to present my more complex problem, I might give some explanation. I am managing the computer systems for a small regional museum, and we do a lot of digitalization (pictures, movies), and of course regular backups, for which I use rsync. And we have three backup disks which are regularly changed. We are mostly running XP workstations, with a SAMBA server, but the backup disks are NTFS.

Currently I am replacing the old 1 TB backups (which are nearly full) with 2 TB disks. And then I saw a problem re-occeur which I thought I had solved before. Not only that, but I also saw it "vanish" this time, rather than being solved, as I thought before.

Take the example I gave already.

> > rsync: recv_generator: failed to stat "/mnt/tmp/backup/museum/adlib/adlib

> > documenten voor/catalogus objecten in eigen bezit/artikelen,

> > algemeen/8467 wehrpa\#303\#237.doc": Invalid or incomplete multibyte or

> > wide character (84)

> > rsync: recv_generator: failed to stat

> > "/mnt/tmp/backup/museum/documenten/bestuur/archief, niet

> > geschoond/publiciteit/idee\#343\#253n website.doc": Invalid or incomplete

> > multibyte or wide character (84)

Behind the backslashes you see a numerical code, which is clearly indicating a locality problem; in fact the file names are "8467 wehrpaß.doc" (with a German "sz" in it), and "ideeën website.doc", with a Dutch "e-trema" in it. Neither of these is a problem in Linux, Samba, Windows XP, or NTFS.

So I am pretty sure it _is_ a locality problem, and I had it before, when I installed the 1TB disks, and I _thought_ I had solved it.

But now I also discovered that the problem vanishes by itself, when you re-run rsync. If you have this "ideeën" file, it is rejected, and it is _not_ saved, but if you run rsync again, the message does not appear, and the file _is_ saved.

This must have led me astray the previous time in thinking that I had solved the problem, and it may also be responsible for some of the options I chose and which apparently were succesful.

So my question is now: how come a file is rejected the first time around, and simply accepted the second time, _without_ any change?

Well, as a system manager I can now be happy, for the problem goes away, but I am still interested in the mechanism behind it. My C knowledge is a bit rusty, so I am not really planning to go through the source code myself :-)

--

joop gerritse

Mühlenstraße 11

D-47546 Kalkar-Wissel

Germany

[phone]

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Greg Deback (rsync)]

Hello all,

I don't know if this would really help, but I had some problems with latin characters too while syncing between a Windows NT fileserver (NTFS) and a linux box (ext3), and I have solved it using --iconv option, as follow :
rsync -hxPHaz -e 'ssh' --stats --iconv=CP1252 --delete --backup --backup-dir=$BKP_DIR $SRC $DEST
(-h: human readable, -x: don't cross filesystems, -P: --partial --progress, -H: hard links, -a: archive, ie. own/group/mod/time/..., -z: compression)

Greg

On Sun, Apr 15, 2012 at 10:05 AM, joop g <[hidden email]> wrote:

On Friday 13 April 2012 21:22:42 [hidden email] wrote:

> Since no one has replied yet, I have one idea that *might* point to part

> of your problem. I've never had to deal with locale issues, so I have no

> idea about that.

>

> You have directories and file names with blanks in them. In general, this

> causes a lot of trouble for a lot of programs. I'm not sure how rsync

> handles them.

It is very kind of you to at least give it a try, but I am quite sure that spaces are not the problem. In fact, I have been experimenting further, and the situation is even more complicated.

First of all, I have to apologize to the list for my first question, regarding

> > WARNING: Couldn't set locale to 'nl_NL.iso-9959-1' thus some file names

> > may not

> >

> > be correct or visible. Please see the potential solution at

> > http://ntfs-3g.org/support.html#locale

I could try, of course, but it was clearly off-list.

Before I am to present my more complex problem, I might give some explanation. I am managing the computer systems for a small regional museum, and we do a lot of digitalization (pictures, movies), and of course regular backups, for which I use rsync. And we have three backup disks which are regularly changed. We are mostly running XP workstations, with a SAMBA server, but the backup disks are NTFS.

Currently I am replacing the old 1 TB backups (which are nearly full) with 2 TB disks. And then I saw a problem re-occeur which I thought I had solved before. Not only that, but I also saw it "vanish" this time, rather than being solved, as I thought before.

Take the example I gave already.

> > rsync: recv_generator: failed to stat "/mnt/tmp/backup/museum/adlib/adlib

> > documenten voor/catalogus objecten in eigen bezit/artikelen,

> > algemeen/8467 wehrpa\#303\#237.doc": Invalid or incomplete multibyte or

> > wide character (84)

> > rsync: recv_generator: failed to stat

> > "/mnt/tmp/backup/museum/documenten/bestuur/archief, niet

> > geschoond/publiciteit/idee\#343\#253n website.doc": Invalid or incomplete

> > multibyte or wide character (84)

Behind the backslashes you see a numerical code, which is clearly indicating a locality problem; in fact the file names are "8467 wehrpaß.doc" (with a German "sz" in it), and "ideeën website.doc", with a Dutch "e-trema" in it. Neither of these is a problem in Linux, Samba, Windows XP, or NTFS.

So I am pretty sure it _is_ a locality problem, and I had it before, when I installed the 1TB disks, and I _thought_ I had solved it.

But now I also discovered that the problem vanishes by itself, when you re-run rsync. If you have this "ideeën" file, it is rejected, and it is _not_ saved, but if you run rsync again, the message does not appear, and the file _is_ saved.

This must have led me astray the previous time in thinking that I had solved the problem, and it may also be responsible for some of the options I chose and which apparently were succesful.

So my question is now: how come a file is rejected the first time around, and simply accepted the second time, _without_ any change?

Well, as a system manager I can now be happy, for the problem goes away, but I am still interested in the mechanism behind it. My C knowledge is a bit rusty, so I am not really planning to go through the source code myself :-)

--

joop gerritse

Mühlenstraße 11

D-47546 Kalkar-Wissel

Germany

<a href="tel:%2B49-2824-971487" value="[phone]" target="_blank">[phone]

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html