Summary: Silence 'vanished files' messages
AssignedTo: [hidden email] ReportedBy: [hidden email] QAContact: [hidden email]
Please add an option to disable the 'vanished files' error messages and return
code. This way, running rsync from crontab/meep wouldn't send mails for this
rather unproblematic case when you already know about it.
------- Comment #2 from [hidden email] 2006-04-03 14:59 MST -------
I think filtering error messages should be the job of the cron script invoking
rsync, not of rsync itself; otherwise we'll find ourselves adding options to
disable each of the myriad errors rsync can produce. Do "rsync <...> 2>&1 |
grep -v 'file has vanished'" .
------- Comment #3 from [hidden email] 2006-06-07 09:20 MST -------
(In reply to comment #2)
> I think filtering error messages should be the job of the cron script invoking
> rsync, not of rsync itself; otherwise we'll find ourselves adding options to
> disable each of the myriad errors rsync can produce. Do "rsync <...> 2>&1 |
> grep -v 'file has vanished'" .
the problem here is that this isn't really an error, but basically a
------- Comment #4 from [hidden email] 2006-06-07 17:41 MST -------
(In reply to comment #3)
> when I'm calling rsync from meep (http://freshmeat.net/projects/meep/) I can't
> filter the error messages or change the error code before meep gets them.
Yes you can! Write an rsync wrapper script like this:
------- Comment #6 from [hidden email] 2006-06-09 19:17 MST -------
I still think that incorporating a patch into the official rsync for something
done so easily outside of rsync would be foolish. Of course you are welcome to
patch your own copy of rsync if you prefer that to a shell script.
You have a point about security updates. I wonder if anyone has a system that
will automatically apply local customizations to software updates.
OK, so I made a mistake and my script mangled the exit code. Please try the
attached improved version of the script. As for "not easy to fix": maybe it's
not easy for you to come up with a fix, but if my new script works, there will
be a fix that is easy to _use_ and that's what counts.
What |Removed |Added
CC| |[hidden email]
------- Comment #10 from [hidden email] 2006-10-13 03:54 MST -------
I have a theory that vanised files are caused
by clock sync of ntpdate or hwclock.
Please could you check if the vanished files apears only
when a cron job of ntpdate is done ?
Patching rsync to hide vanished files is just a work around
and this is very dangerous if you use "ignore-errors" with "delete"
options. Rsync will start to delete files who still present on
the source because of reported as "vanished" !
------- Comment #11 from [hidden email] 2006-10-14 19:01 MST -------
Andreas, you have a point about the special nature of the "file has vanished"
message. The message seems to be an artifact of rsync's implementation that
isn't meaningful at the level of rsync's purpose. I now believe the sender
should treat a vanished file V as if it never existed in the first place. That
(1) If V was a command-line argument or --files-from entry, the sender should
issue the same error it would have issued if V did not exist during file-list
(2) If the generator has itemized the transfer, the sender needs to print an
informational message stating that the transfer didn't actually happen so that
programs reading the output know to forget about it. This message is neither
an error nor a warning.
(3) If --delete is on and V exists in the destination, the generator needs to
consider deleting it. To this end, the sender uses a protocol extension to
tell the receiver that V vanished. The receiver passes the news to the
generator, which removes V from its file list. If the generator has already
deleted in V's containing directory, it deletes V immediately modulo protect
filters. If the receiving side is not modern enough to support this process,
the sender prints a warning to that effect and sets IOERR_VANISHED.
To a sender containing my proposed changes, a vanishing is no longer cause for
a warning or an exit code unless the receiving side is old and #3 applies. And
I anticipate that #3 will rarely happen because most source files that vanish
are short-lived temporary files that haven't existed for long enough to be
copied to the destination by a previous rsync run.
Even with the changes I proposed in comment 11, there are two cases in which
rsync's handling of a vanishing could be considered incorrect and thus merit a
warning and code 24. They are both demonstrated in the attached script.
4. A source file causes rsync to delete a non-regular destination file (which
could even be a nonempty directory with --force) and then vanishes. The end
result is that rsync deleted something from the destination without replacing
it, which shouldn't happen without --delete.
5. With --prune-empty-dirs, a source file preventing the pruning of its parent
directory and then vanishes. The end result is that rsync creates an empty
directory, which isn't supposed to happen with --prune-empty-dirs.
Both of these seem rare and a pain to fix, and there are other other concurrent
modifications that cause code 23, such as replacing a regular file with a
directory, which can't be read(2). I think making rsync completely correct and
warning-free in the face of concurrent modifications to the source is a losing
battle. I propose that we focus on reducing the inconvenience in the common
case by suppressing the warning and code 24 for "safe" vanishings, namely new
source files in unprunable directories, and perhaps handling #3. The need to
issue a code 23 when a mandatory file vanishes (#1) is a separate issue.
What |Removed |Added
CC| |[hidden email]
------- Comment #16 from [hidden email] 2009-05-18 07:59 CST -------
This is a 'me too' for '--ignore-vanished-files' flag. Lots of legitimate cases
where files could vanish, so having the option to treat this as normal (i.e.
prevent rsync from producing this warning and ensure a zero exit value) should
be simple enough, and would prevent lots of people (me included) from needing
to use wrapper scripts, hacks and/or just give it to receiving the annoying
cron messages it causes :)
Doing nightly backups from cron is an *extremely* common use-case of rsync, and
telling each and every user of rsync that they should either A) write their own
shell script to filter and discard error messages, or B) play a constant game
of whack-a-mole chasing down new --exclude options to add, is just silly. It
pushes work downstream that could be done upstream much more efficiently. Why
make each user write the same code over and over when the developer could write
it just once, and get it right?
Option A is bad because everyone gets to introduce differently-subtle bugs in
their script (wrong error code? accidentally ignoring too many error messages?
who knows. It's fragile.)
Option B is bad because I don't *want* to have a bunch of --exclude options for
my backup: disk and bandwidth are cheap, so I want my backup disk to be an
exact mirror. That way, when the disk holding / dies, I can just drop the
backup in and boot it, being confident that everything is exactly the same.