Should I worry about the "vanished files" warning?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Should I worry about the "vanished files" warning?

Samba - rsync mailing list
Hi

I would like to ask, when the "vanished files" warning is a sign that something bad is happening somewhere. I know that the `rsync-no-vanished` script can silence the warning, but I am wondering whether this is a sane thing to do.

My guess based on https://bugzilla.samba.org/show_bug.cgi?id=3653#c26 and https://bugzilla.samba.org/show_bug.cgi?id=10356 is that it's fine, but I'd like to verify :)

In my case, I have a batch process that creates ~500 files, some take 30 min, some take 8h to be created. The files are created in a tmp dir and are moved to a final dir when they are finished. Every 5 mins, I rsync the files from the final directory to a remote machine. I guess that during the sync process some files get re-written, when the batch process moves the fresh ones in the final dir.

My cron entry:
flock -xn /lock_file -c                                       \
"/usr/bin/rsync -e \"ssh -C\" --bwlimit=512k                  \
--recursive --delay-updates --quiet --update                  \
--include=\"*/\" --include=\"*.xml\" --include=\"*.csv\"      \
--exclude=\"*\"                                               \
/LOCAL_SOURCE_DIR/ REMOTE_IP:/DEST_DIR/"

Both source and destination are on
$ rsync --version
rsync  version 3.1.0  protocol version 31

Regards
Vangelis

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Reply | Threaded
Open this post in threaded view
|

Re: Should I worry about the "vanished files" warning?

Samba - rsync mailing list
All it means is that rsync saw a file that needed transferring but the
file was gone when rsync actually tried to open it.  So look at the
filenames and decide if it is a problem or not.

On 09/07/2017 10:27 AM, Vangelis Katsikaros via rsync wrote:

> Hi
>
> I would like to ask, when the "vanished files" warning is a sign that something bad is happening somewhere. I know that the `rsync-no-vanished` script can silence the warning, but I am wondering whether this is a sane thing to do.
>
> My guess based on https://bugzilla.samba.org/show_bug.cgi?id=3653#c26 and https://bugzilla.samba.org/show_bug.cgi?id=10356 is that it's fine, but I'd like to verify :)
>
> In my case, I have a batch process that creates ~500 files, some take 30 min, some take 8h to be created. The files are created in a tmp dir and are moved to a final dir when they are finished. Every 5 mins, I rsync the files from the final directory to a remote machine. I guess that during the sync process some files get re-written, when the batch process moves the fresh ones in the final dir.
>
> My cron entry:
> flock -xn /lock_file -c                                       \
> "/usr/bin/rsync -e \"ssh -C\" --bwlimit=512k                  \
> --recursive --delay-updates --quiet --update                  \
> --include=\"*/\" --include=\"*.xml\" --include=\"*.csv\"      \
> --exclude=\"*\"                                               \
> /LOCAL_SOURCE_DIR/ REMOTE_IP:/DEST_DIR/"
>
> Both source and destination are on
> $ rsync --version
> rsync  version 3.1.0  protocol version 31
>
> Regards
> Vangelis
>
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
        Kevin Korb Phone:    (407) 252-6853
        Systems Administrator Internet:
        FutureQuest, Inc. [hidden email]  (work)
        Orlando, Florida [hidden email] (personal)
        Web page: http://www.sanitarium.net/
        PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

signature.asc (231 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Should I worry about the "vanished files" warning?

Samba - rsync mailing list
Hi Kevin

Thanks for the tip. In my case it seems it's fine!

Regards
Vangelis

On 09/07/2017 08:51 PM, Kevin Korb via rsync wrote:

> All it means is that rsync saw a file that needed transferring but the
> file was gone when rsync actually tried to open it.  So look at the
> filenames and decide if it is a problem or not.
>
> On 09/07/2017 10:27 AM, Vangelis Katsikaros via rsync wrote:
>> Hi
>>
>> I would like to ask, when the "vanished files" warning is a sign that something bad is happening somewhere. I know that the `rsync-no-vanished` script can silence the warning, but I am wondering whether this is a sane thing to do.
>>
>> My guess based on https://bugzilla.samba.org/show_bug.cgi?id=3653#c26 and https://bugzilla.samba.org/show_bug.cgi?id=10356 is that it's fine, but I'd like to verify :)
>>
>> In my case, I have a batch process that creates ~500 files, some take 30 min, some take 8h to be created. The files are created in a tmp dir and are moved to a final dir when they are finished. Every 5 mins, I rsync the files from the final directory to a remote machine. I guess that during the sync process some files get re-written, when the batch process moves the fresh ones in the final dir.
>>
>> My cron entry:
>> flock -xn /lock_file -c                                       \
>> "/usr/bin/rsync -e \"ssh -C\" --bwlimit=512k                  \
>> --recursive --delay-updates --quiet --update                  \
>> --include=\"*/\" --include=\"*.xml\" --include=\"*.csv\"      \
>> --exclude=\"*\"                                               \
>> /LOCAL_SOURCE_DIR/ REMOTE_IP:/DEST_DIR/"
>>
>> Both source and destination are on
>> $ rsync --version
>> rsync  version 3.1.0  protocol version 31
>>
>> Regards
>> Vangelis
>>
>
>
>


--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html