[RFC PATCH] smbd: resilient file handle support

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

[RFC PATCH] smbd: resilient file handle support

Uri Simchoni-4
Hi,

Attached pls find a patch for supporting resilient file handles.
Comments are welcome. It doesn't include tests yet, but I would
appreciate technical feedback and any other feedback.

By and large, this patch stands on the shoulders of the durable handles
implementation. Same restrictions as with durable handles apply (durable
handles must be enabled, kernel oplocks, posix locks and kernel share
modes must be disabled).

I didn't plan this. The driver for doing this was that we've just found
out that Windows backup engine doesn't work with Samba and SMB dialect >
2.0.2 (see https://bugzilla.samba.org/show_bug.cgi?id=10159).

I don't know if I'm stepping on any toes here - I know resilient were
low priority because persistent handles are the "real deal", but it
appears that Windows backup requests resilient handles, at least when
persistent handles are not supported.

On to the technical stuff.

The patch implements resilient handles with the omission of lock
sequence verification ([MS-SMB2] 3.3.5.14), a mechanism used for
preventing the server from doing the same lock twice if it failed to ack
the first lock request because of disconnect.

 From my reading of [MS-SMB2], it looks like resilient handles are very
similar to durable handles. They serve the same purpose, namely to allow
seamless (transparent to the application) immediate reconnect by a
client if the connection breaks, while keeping locks/oplocks/share modes
intact. The main difference between durable and resilient is in how they
are being set up, namely that durable handles get created by create
context, and failure to make the handle durable does not fail the open,
whereas resilient handles get created by an IOCTL, and failure is a
failure of the IOCTL, which perhaps provides better feedback to the
application.

Data Model
===========
[MS-SMB2] describes "durability" and "resiliency" of a file handle as
two distinct properties, but it immediately becomes apparent that they
are mutually exclusive - enabling Open.IsResilient automatically
disables Open.IsDurable (3.3.5.15.9). Therefore it would seem beneficial
to treat a resilient file handle as a special type of durable handle,
and observe the difference. From my reading, the functional differences
amount to the following:

a. When an oplock or lease breaks on a disconnected durable handle, the
handle must be deleted, whereas a resilient handle is not deleted. From
functional point of view it means a reconnect after oplock break fails
on a durable handle but not on a resilient handle.

b. If the dialect is SMB2.1, lock sequence verification is done only on
resilient handles.

c. When a disconnect happens, resilient handles are always being
preserved, whereas durable handles must be preserved only if specific
types of oplock or lease are granted.

With respect to a), it seems that current Samba code does not delete
durable handles on oplock break.

With respect to b), the patch does not implement lock sequence checking
at all.

With respect to c), it seems like Samba implements this by only marking
a handle as durable if it has a handle lease, but upon disconnect, it
unconditionally preserves durable handles.

It follows that no significant change is required to implement resilient
handles - the only required thing is to handle the IOCTL and "durablize"
the handle, with the timeout of the resiliency request. I could add a
"resilient" variable that would tell the difference but at this stage
this would be a write-only variable.

Conformance with [MS-SMB2]
====================
Following are places in which Open.IsResilient is mentioned and how it's
being covered.

3.3.4.6 - not delete a disconnected resilient handle on oplock break -
we never delete it.

3.3.4.7 - not delete a disconnected resilient handle on lease break - we
never delete it.

3.3.5.6 - not delete a resilient handle on logoff - covered by marking
the handle as durable.

3.3.5.9 - on CREATE, set Open.DurableFileId to a unique value - a handle
can only be resilient at create time if it's a reconnect, and in that
case the Open.DurableFileId has already been set.

3.3.5.9.7 - Handling of reconnect - same as  durable.

3.3.5.9.12 - Handling of reconnectV2 - same as durable.

3.3.5.14 - Lock sequence verification - not implemented (that's a SHOULD)

3.3.5.14.1 - Updating the lock sequence array on Unlock - not
implemented (not needed since we don't verify sequence)

3.3.5.14.2 - Updating the lock sequence array on Lock - not implemented
(not needed since we don't verify sequence)

3.3.5.15.9 - Handling the resiliency request - that's the meat of this
patch, code can be carefully compared to the spec. The code essentially
"durablize" the handle, with the requested timeout.

3.3.6.4 - Resilient Open Scavenger Timer - The spec of what to do when a
handle expires is identical to durable handles. The difference is in how
timeouts are set. Our implementation of durable open scavenger is
compatible, timing-wise, with the spec.

3.3.7.1 - loss of connection, preserving a handle for reconnect -
resilient handles must be preserved. Durable handles must be preserved
only if Open.OplockLevel is equal to SMB2_OPLOCK_LEVEL_BATCH and
Open.OplockState is equal to Held. However, it seems like our code
always preserves durable handles, so the resilient==durable design holds
here too.

3.3.7.1 - triggering the scavenger time on loss of connection - same as
durable.

Thanks,
Uri.


resilient.patch.txt (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [RFC PATCH] smbd: resilient file handle support

Michael Adam-3
On 2016-02-11 at 23:42 +0200, Uri Simchoni wrote:
> Hi,
>
> Attached pls find a patch for supporting resilient file
> handles. Comments are welcome. It doesn't include tests yet,
> but I would appreciate technical feedback and any other
> feedback.

Cool!

A few comments up front before digging deeper into the
actual patches.

Note the hacked up implementation (by metze) in the
master-multi-channel-obnox branch:

https://git.samba.org/?p=obnox/samba/samba-obnox.git;a=commitdiff;h=8e68d390dc640ceb3aa9594c9d18d5d3d01c24f2

samba.org/?p=obnox/samba/samba-obnox.git;a=shortlog;h=refs/heads/master-multi-channel-obnox


> By and large, this patch stands on the shoulders of the durable
> handles implementation. Same restrictions as with durable
> handles apply (durable handles must be enabled, kernel oplocks,
> posix locks and kernel share modes must be disabled).
>
> I didn't plan this. The driver for doing this was that we've
> just found out that Windows backup engine doesn't work with
> Samba and SMB dialect > 2.0.2 (see
> https://bugzilla.samba.org/show_bug.cgi?id=10159).
>
> I don't know if I'm stepping on any toes here - I know
> resilient were low priority because persistent handles are the
> "real deal", but it appears that Windows backup requests
> resilient handles, at least when persistent handles are not
> supported.
Sure, we eventually want all of it!

> On to the technical stuff.
>
> The patch implements resilient handles with the omission of
> lock sequence verification ([MS-SMB2] 3.3.5.14), a mechanism
> used for preventing the server from doing the same lock twice
> if it failed to ack the first lock request because of
> disconnect.

Patches for lock sequence checking have recently been
proposed to this list by Günther, based on previous work
by Metze and me. Including a few tests. So we have that.

> From my reading of [MS-SMB2], it looks like resilient handles
> are very similar to durable handles. They serve the same
> purpose, namely to allow seamless (transparent to the
> application) immediate reconnect by a client if the connection
> breaks, while keeping locks/oplocks/share modes intact.
>
> The main difference between durable and resilient is in how
> they are being set up, namely that durable handles get created
> by create context, and failure to make the handle durable does
> not fail the open, whereas resilient handles get created by an
> IOCTL, and failure is a failure of the IOCTL, which perhaps
> provides better feedback to the application.
One main difference is that while durable handles are purely
best effort, i.e. when a second client tries to open a file
with a disconnected durable handle, that handle is closed
and the original opener can not reconnect it. For resilient
handles, there are iirc (need to look up the exact details),
certain weak guarantees where new openers are blocked for
a short disconnected time. The same is true for persistent
handles, but with stronger guarantees and you can also get
them without oplocks, on a scale-out share.

I think implementing these guarantees is the major challenge
in implementing resilient and durable handles.

The other aspect as you mentiond: On the requesting side,
durable handles (and persistent handles) are not exposed
to the application. The redirector takes care of them.
Resilient handles instead are explicitly requested by the
client application. I think that this was later considered
a bad choice since not all applications can benefit from that.
Maybe the increased level of control can indeed be a plus
for aware applications.

More comments later (not today...) when I've managed to
look thoroughly at the patches.

Cheers - Michael



> Data Model
> ===========
> [MS-SMB2] describes "durability" and "resiliency" of a file handle as two
> distinct properties, but it immediately becomes apparent that they are
> mutually exclusive - enabling Open.IsResilient automatically disables
> Open.IsDurable (3.3.5.15.9). Therefore it would seem beneficial to treat
> a resilient file handle as a special type of durable handle, and observe
> the difference. From my reading, the functional differences amount to the
> following:
>
> a. When an oplock or lease breaks on a disconnected durable handle, the
> handle must be deleted, whereas a resilient handle is not deleted. From
> functional point of view it means a reconnect after oplock break fails on
> a durable handle but not on a resilient handle.
>
> b. If the dialect is SMB2.1, lock sequence verification is done only on
> resilient handles.
>
> c. When a disconnect happens, resilient handles are always being
> preserved, whereas durable handles must be preserved only if specific
> types of oplock or lease are granted.
>
> With respect to a), it seems that current Samba code does not delete
> durable handles on oplock break.
>
> With respect to b), the patch does not implement lock sequence checking
> at all.
>
> With respect to c), it seems like Samba implements this by only marking a
> handle as durable if it has a handle lease, but upon disconnect, it
> unconditionally preserves durable handles.
>
> It follows that no significant change is required to implement resilient
> handles - the only required thing is to handle the IOCTL and "durablize"
> the handle, with the timeout of the resiliency request. I could add a
> "resilient" variable that would tell the difference but at this stage
> this would be a write-only variable.
>
> Conformance with [MS-SMB2]
> ====================
> Following are places in which Open.IsResilient is mentioned and how it's
> being covered.
>
> 3.3.4.6 - not delete a disconnected resilient handle on oplock break - we
> never delete it.
>
> 3.3.4.7 - not delete a disconnected resilient handle on lease break - we
> never delete it.
>
> 3.3.5.6 - not delete a resilient handle on logoff - covered by marking
> the handle as durable.
>
> 3.3.5.9 - on CREATE, set Open.DurableFileId to a unique value - a handle
> can only be resilient at create time if it's a reconnect, and in that
> case the Open.DurableFileId has already been set.
>
> 3.3.5.9.7 - Handling of reconnect - same as  durable.
>
> 3.3.5.9.12 - Handling of reconnectV2 - same as durable.
>
> 3.3.5.14 - Lock sequence verification - not implemented (that's a SHOULD)
>
> 3.3.5.14.1 - Updating the lock sequence array on Unlock - not implemented
> (not needed since we don't verify sequence)
>
> 3.3.5.14.2 - Updating the lock sequence array on Lock - not implemented
> (not needed since we don't verify sequence)
>
> 3.3.5.15.9 - Handling the resiliency request - that's the meat of this
> patch, code can be carefully compared to the spec. The code essentially
> "durablize" the handle, with the requested timeout.
>
> 3.3.6.4 - Resilient Open Scavenger Timer - The spec of what to do when a
> handle expires is identical to durable handles. The difference is in how
> timeouts are set. Our implementation of durable open scavenger is
> compatible, timing-wise, with the spec.
>
> 3.3.7.1 - loss of connection, preserving a handle for reconnect -
> resilient handles must be preserved. Durable handles must be preserved
> only if Open.OplockLevel is equal to SMB2_OPLOCK_LEVEL_BATCH and
> Open.OplockState is equal to Held. However, it seems like our code always
> preserves durable handles, so the resilient==durable design holds here
> too.
>
> 3.3.7.1 - triggering the scavenger time on loss of connection - same as
> durable.
>
> Thanks,
> Uri.
>

> From 1b4255b67034d7b9945814bf65364773118d44b3 Mon Sep 17 00:00:00 2001
> From: Uri Simchoni <[hidden email]>
> Date: Thu, 11 Feb 2016 13:56:04 +0200
> Subject: [PATCH] smbd: add support for resilent file handles
>
> Add support for resilient file handles, with the same
> restrictions as those for durable handles, and the following
> omission:
>
> 1. Verifying lock sequence [MS-SMB2] 3.3.5.14 - this is listed
>    as SHOULD
>
> This patch uses the durable file handle infrastructure, and
> internally makes no distinction between durable and resilient
> file handles - the difference is in how they are being set up.
>
> This is so because in MS-SMB2, a handle can be durable or
> resilient, but not both (see 3.3.5.15.9 - when resiliency
> request is received, IsResilient is set to TRUE and IsDurable
> is set to FALSE). The functional differences once the
> handle is set up are:
>
> 1. When an oplock or lease breaks on a disconnected, durable
>    handle, the handle  must be deleted, whereas  a resilient
>    disconnected handle is not deleted (3.3.4.6 and
>    3.3.4.7) - in Samba a durable handle is not deleted
>    by an oplock break.
>
> 2. If the dialect is SMB2.1, lock sequence verification is done
>    only on resilient handles - at this stage lock sequence
>    verification is not implemented.
>
> 3. Upon disconnect, durable handles are preserved only if certain
>    types of oplocks/leases were granted, and resilient handles are
>    always preserved. Here Samba only marks a handle as durable
>    if it has a handle lease, but upon disconnect it
>    unconditionally preserves durable handles.
>
> It follows that after Samba sets up a handle as durable, it also meets
> the resilient handle spec.
> ---
>  source3/smbd/smb2_ioctl_network_fs.c | 69 ++++++++++++++++++++++++++++++++++++
>  1 file changed, 69 insertions(+)
>
> diff --git a/source3/smbd/smb2_ioctl_network_fs.c b/source3/smbd/smb2_ioctl_network_fs.c
> index d8590de..0af6c83 100644
> --- a/source3/smbd/smb2_ioctl_network_fs.c
> +++ b/source3/smbd/smb2_ioctl_network_fs.c
> @@ -620,6 +620,67 @@ static NTSTATUS fsctl_srv_req_resume_key(TALLOC_CTX *mem_ctx,
>   return NT_STATUS_OK;
>  }
>  
> +static NTSTATUS fsctl_request_resiliency(struct smbXsrv_connection *conn,
> + struct files_struct *fsp,
> + DATA_BLOB *in_input)
> +{
> + NTSTATUS status;
> + uint32_t timeout = 0;
> + struct smbXsrv_open *op = fsp->op;
> +
> + /* Should we check whether fsp is NULL? */
> +
> + timeout = IVAL(in_input->data, 0x00);
> + /*
> + * Maximum value should be configurable -
> + * MaxResiliencyTimeout, with a default in
> + * Windows of 300 sec according to [MS-SMB2] 3.3.3
> + */
> + if (timeout > 300 * 1000) {
> + return NT_STATUS_INVALID_PARAMETER;
> + }
> +
> + if (op->global->backend_cookie.length == 0) {
> + status = SMB_VFS_DURABLE_COOKIE(fsp, op,
> + &op->global->backend_cookie);
> + if (!NT_STATUS_IS_OK(status)) {
> + return status;
> + }
> + }
> +
> + op->global->durable = true;
> +
> + if (timeout == 0) {
> + /* Default - 120 sec as per Server 2012 and later
> + * according to [MS-SMB2] 3.3.5.15.9
> + */
> + timeout = 120 * 1000;
> + }
> +
> + /* [MS-SMB2] defines different state variables for durable
> + * and resilient, but at the same time durable and resilient
> + * are mutually-exclusive, and the scavenging algorithm is
> + * identical.
> + * Therefore we keep the timeout in durable_timeout_msec.
> + */
> + op->global->durable_timeout_msec = timeout;
> +
> + /* no need to handle durable owner - it's recorded
> + * on every open even if the handle is not durable
> + * or resilient.
> + */
> +
> + status = smbXsrv_open_update(op);
> + DBG_DEBUG("smb2_create_send: smbXsrv_open_update "
> +  "returned %s\n",
> +  nt_errstr(status));
> + if (!NT_STATUS_IS_OK(status)) {
> + return status;
> + }
> +
> + return NT_STATUS_OK;
> +}
> +
>  static void smb2_ioctl_network_fs_copychunk_done(struct tevent_req *subreq);
>  
>  struct tevent_req *smb2_ioctl_network_fs(uint32_t ctl_code,
> @@ -698,6 +759,14 @@ struct tevent_req *smb2_ioctl_network_fs(uint32_t ctl_code,
>   }
>   return tevent_req_post(req, ev);
>   break;
> + case FSCTL_LMR_REQ_RESILIENCY:
> + status = fsctl_request_resiliency(state->smbreq->xconn,
> +  state->fsp, &state->in_input);
> + if (!tevent_req_nterror(req, status)) {
> + tevent_req_done(req);
> + }
> + return tevent_req_post(req, ev);
> + break;
>   default: {
>   uint8_t *out_data = NULL;
>   uint32_t out_data_len = 0;
> --
> 2.5.0
>


signature.asc (206 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [RFC PATCH] smbd: resilient file handle support

Uri Simchoni-4


On 02/12/2016 12:38 AM, Michael Adam wrote:

> On 2016-02-11 at 23:42 +0200, Uri Simchoni wrote:
>> Hi,
>>
>> Attached pls find a patch for supporting resilient file
>> handles. Comments are welcome. It doesn't include tests yet,
>> but I would appreciate technical feedback and any other
>> feedback.
> Cool!
>
> A few comments up front before digging deeper into the
> actual patches.
>
> Note the hacked up implementation (by metze) in the
> master-multi-channel-obnox branch:
>
> https://git.samba.org/?p=obnox/samba/samba-obnox.git;a=commitdiff;h=8e68d390dc640ceb3aa9594c9d18d5d3d01c24f2
>
> samba.org/?p=obnox/samba/samba-obnox.git;a=shortlog;h=refs/heads/master-multi-channel-obnox
OK that looks like a "fake resilient" implementation.

> <snip>
>
> One main difference is that while durable handles are purely
> best effort, i.e. when a second client tries to open a file
> with a disconnected durable handle, that handle is closed
> and the original opener can not reconnect it. For resilient
> handles, there are iirc (need to look up the exact details),
> certain weak guarantees where new openers are blocked for
> a short disconnected time. The same is true for persistent
> handles, but with stronger guarantees and you can also get
> them without oplocks, on a scale-out share.
>
> I think implementing these guarantees is the major challenge
> in implementing resilient and durable handles.
>
> <snip>

> Cheers - Michael
>
>
Thanks so much for summarizing what resilient handles are in technical
terms - I could not find such a concise description anywhere so I tried
to dig it from [MS-SMB2].

It looks like I got the differences-from-durable right (in my a-b-c
list), but may be way off-base claiming that Samba's durable handles are
already "resilient in disguise" (in that the original opener of a Samba
durable handle can reconnect to it if someone else has opened the file
while being disconnected).

I'll dig deeper, but probably you and others in the team can easily tell
the right answer.

Thanks,
Uri.


Reply | Threaded
Open this post in threaded view
|

Re: [RFC PATCH] smbd: resilient file handle support

Stefan Metzmacher-2
Hi Uri,

> I'll dig deeper, but probably you and others in the team can easily tell
> the right answer.

Once we have multi-channel support ready, we can implement durable handles
and resilient handles in a better way and together with using kernel
oblocks.
We just let smbd running after the (last) connection gets disconnected
and wait for the client to reconnect. The client will use the same
client guid
and will end up in its old smbd process.

As we already have a 'fake oplocks', which is very dangerous,
we can also have a 'fake resilient handles' option in order to
make the backup application happy. This could be set per client
via an 'include = /etc/samba/smb.conf.extra.%I' line in smb.conf.

But I would concentrate on getting multi-channel done and have
a real implementation for resilient handles.

metze


signature.asc (853 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [RFC PATCH] smbd: resilient file handle support

Ralph Böhme-3
On Fri, Feb 12, 2016 at 02:31:28PM +0100, Stefan Metzmacher wrote:

> Hi Uri,
>
> > I'll dig deeper, but probably you and others in the team can easily tell
> > the right answer.
>
> Once we have multi-channel support ready, we can implement durable handles
> and resilient handles in a better way and together with using kernel
> oblocks.
> We just let smbd running after the (last) connection gets disconnected
> and wait for the client to reconnect. The client will use the same
> client guid
> and will end up in its old smbd process.
>
> As we already have a 'fake oplocks', which is very dangerous,
> we can also have a 'fake resilient handles' option in order to
> make the backup application happy. This could be set per client
> via an 'include = /etc/samba/smb.conf.extra.%I' line in smb.conf.
>
> But I would concentrate on getting multi-channel done and have
> a real implementation for resilient handles.

agree, this makes the most sense to me too.

-Ralph

--
SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen
phone: +49-551-370000-0, fax: +49-551-370000-9
AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen
http://www.sernet.de,mailto:kontakt@...

Reply | Threaded
Open this post in threaded view
|

Re: [RFC PATCH] smbd: resilient file handle support

Michael Adam-3
In reply to this post by Uri Simchoni-4
On 2016-02-12 at 15:10 +0200, Uri Simchoni wrote:

>
>
> On 02/12/2016 12:38 AM, Michael Adam wrote:
> >On 2016-02-11 at 23:42 +0200, Uri Simchoni wrote:
> >>Hi,
> >>
> >>Attached pls find a patch for supporting resilient file
> >>handles. Comments are welcome. It doesn't include tests yet,
> >>but I would appreciate technical feedback and any other
> >>feedback.
> >Cool!
> >
> >A few comments up front before digging deeper into the
> >actual patches.
> >
> >Note the hacked up implementation (by metze) in the
> >master-multi-channel-obnox branch:
> >
> >https://git.samba.org/?p=obnox/samba/samba-obnox.git;a=commitdiff;h=8e68d390dc640ceb3aa9594c9d18d5d3d01c24f2
> >
> >samba.org/?p=obnox/samba/samba-obnox.git;a=shortlog;h=refs/heads/master-multi-channel-obnox
> OK that looks like a "fake resilient" implementation.
>
> ><snip>
> >
> >One main difference is that while durable handles are purely
> >best effort, i.e. when a second client tries to open a file
> >with a disconnected durable handle, that handle is closed
> >and the original opener can not reconnect it. For resilient
> >handles, there are iirc (need to look up the exact details),
> >certain weak guarantees where new openers are blocked for
> >a short disconnected time. The same is true for persistent
> >handles, but with stronger guarantees and you can also get
> >them without oplocks, on a scale-out share.
> >
> >I think implementing these guarantees is the major challenge
> >in implementing resilient and durable handles.
> >
> ><snip>
>
> >Cheers - Michael
> >
> >
> Thanks so much for summarizing what resilient handles are in technical terms
> - I could not find such a concise description anywhere so I tried to dig it
> from [MS-SMB2].
>
> It looks like I got the differences-from-durable right (in my a-b-c list),
> but may be way off-base claiming that Samba's durable handles are already
> "resilient in disguise" (in that the original opener of a Samba durable
> handle can reconnect to it if someone else has opened the file while being
> disconnected).
Right, if there is a second attempt to open a file, while a durable
handle on this file is disconnected, the durable is closed /
destroyed and the new opener is afterwards the only opener of the
file. This like this in the doc (MS-SMB2) and in samba code.
(Will point out the corresponding code sections later.)

As you described, it is different with resilient handles.

Michael

signature.asc (206 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [RFC PATCH] smbd: resilient file handle support

Uri Simchoni-4
On 02/12/2016 05:25 PM, Michael Adam wrote:

> On 2016-02-12 at 15:10 +0200, Uri Simchoni wrote:
>>
>>
>> On 02/12/2016 12:38 AM, Michael Adam wrote:
>>> On 2016-02-11 at 23:42 +0200, Uri Simchoni wrote:
>>>> Hi,
>>>>
>>>> Attached pls find a patch for supporting resilient file
>>>> handles. Comments are welcome. It doesn't include tests yet,
>>>> but I would appreciate technical feedback and any other
>>>> feedback.
>>> Cool!
>>>
>>> A few comments up front before digging deeper into the
>>> actual patches.
>>>
>>> Note the hacked up implementation (by metze) in the
>>> master-multi-channel-obnox branch:
>>>
>>> https://git.samba.org/?p=obnox/samba/samba-obnox.git;a=commitdiff;h=8e68d390dc640ceb3aa9594c9d18d5d3d01c24f2
>>>
>>> samba.org/?p=obnox/samba/samba-obnox.git;a=shortlog;h=refs/heads/master-multi-channel-obnox
>> OK that looks like a "fake resilient" implementation.
>>
>>> <snip>
>>>
>>> One main difference is that while durable handles are purely
>>> best effort, i.e. when a second client tries to open a file
>>> with a disconnected durable handle, that handle is closed
>>> and the original opener can not reconnect it. For resilient
>>> handles, there are iirc (need to look up the exact details),
>>> certain weak guarantees where new openers are blocked for
>>> a short disconnected time. The same is true for persistent
>>> handles, but with stronger guarantees and you can also get
>>> them without oplocks, on a scale-out share.
>>>
>>> I think implementing these guarantees is the major challenge
>>> in implementing resilient and durable handles.
>>>
>>> <snip>
>>
>>> Cheers - Michael
>>>
>>>
>> Thanks so much for summarizing what resilient handles are in technical terms
>> - I could not find such a concise description anywhere so I tried to dig it
>> from [MS-SMB2].
>>
>> It looks like I got the differences-from-durable right (in my a-b-c list),
>> but may be way off-base claiming that Samba's durable handles are already
>> "resilient in disguise" (in that the original opener of a Samba durable
>> handle can reconnect to it if someone else has opened the file while being
>> disconnected).
>
> Right, if there is a second attempt to open a file, while a durable
> handle on this file is disconnected, the durable is closed /
> destroyed and the new opener is afterwards the only opener of the
> file. This like this in the doc (MS-SMB2) and in samba code.
> (Will point out the corresponding code sections later.)
>
> As you described, it is different with resilient handles.
>
> Michael
>

OK, I'll sit back and wait then, as Metze had suggested. I can implement
"fake resilient" on a forked tree and get rid of it later, because I
have a way of guaranteeing that shares in which I enable "fake
resilient" are only being used for backup (and not HyperV).

What lead me to think that Samba durable handles are "resilient in
disguise" is that I didn't spot any deletion of the smbXsrv_open_global0
associated with the open upon oplock break.

Thanks,
Uri.

Reply | Threaded
Open this post in threaded view
|

[RFC PATCH] smbd: resilient file handle support

Bryan Burgin
In reply to this post by Uri Simchoni-4
Responding to: https://lists.samba.org/archive/samba-technical/2016-February/112112.html

And https://bugzilla.samba.org/show_bug.cgi?id=10159





Focusing on "Regarding "I didn't plan this. The driver for doing this was that we've just found out that Windows backup engine doesn't work with Samba and SMB dialect > 2.0.2 (see https://bugzilla.samba.org/show_bug.cgi?id=10159)", this was resolved several years ago when it was reported.  I'm with the Microsoft protocol support team and work with your Samba colleagues regularly on technical issues.  Please take note of KB 2920193: https://support.microsoft.com/en-us/kb/2920193: "Virtual hard disk cannot be created on an SMB server without resiliency support from a Windows computer".  For Windows 8 and Server 2012, you need to download and install the patch.  For Windows 8.1 and Server 2012 R2, it was fixed automatically as long as the system installed the update rollup 2919355: https://support.microsoft.com/en-us/kb/2919355.  This was fixed in Windows 10 before it was released, so there is no action necessary regarding Windows 10."
Reply | Threaded
Open this post in threaded view
|

Re: [RFC PATCH] smbd: resilient file handle support

Uri Simchoni-4
On 02/16/2016 08:27 PM, Bryan Burgin wrote:

> Responding to: https://lists.samba.org/archive/samba-technical/2016-February/112112.html
>
> And https://bugzilla.samba.org/show_bug.cgi?id=10159
>
>
>
>
>
> Focusing on "Regarding "I didn't plan this. The driver for doing this was that we've just found out that Windows backup engine doesn't work with Samba and SMB dialect > 2.0.2 (see https://bugzilla.samba.org/show_bug.cgi?id=10159)", this was resolved several years ago when it was reported.  I'm with the Microsoft protocol support team and work with your Samba colleagues regularly on technical issues.  Please take note of KB 2920193: https://support.microsoft.com/en-us/kb/2920193: "Virtual hard disk cannot be created on an SMB server without resiliency support from a Windows computer".  For Windows 8 and Server 2012, you need to download and install the patch.  For Windows 8.1 and Server 2012 R2, it was fixed automatically as long as the system installed the update rollup 2919355: https://support.microsoft.com/en-us/kb/2919355.  This was fixed in Windows 10 before it was released, so there is no action necessary regarding Windows 10."
>
Bryan,

I will check. I have reports that we fail Windows 10 backup too unless
downgrading to SMB2.0.2, but I haven't studied it closely as with Win8.

Thanks,
Uri.


Reply | Threaded
Open this post in threaded view
|

RE: [RFC PATCH] smbd: resilient file handle support

Bryan Burgin
Uri,

An update, following up on your comment "I [Uri] will check. I [Uri] have reports that we fail Windows 10 backup too unless downgrading to SMB2.0.2, but I haven't studied it closely as with Win8":

Regarding the Bugzilla report at https://bugzilla.samba.org/show_bug.cgi?id=10159 and the fix we created for Windows 8 via the on-demand (you have to pull down the fix) QFE patch https://support.microsoft.com/en-us/kb/2920193 and Windows 8.1 via the "important" Windows Update roll-up patch https://support.microsoft.com/en-us/kb/2919355 is also included in Windows 10.

The end-user scenarios that were reported back in 2013 was that the feature Windows Server Backup did not work when the target SMB2 server was a dialect 2.1 and above and, as was discovered as the root-cause, the server did not support FSCTL_LMR_REQUEST_RESILIENCY.  Samba servers at the time returned STATUS_NOT_SUPPORTED.  An API that Windows Server Backup relied on, CreateVirtualDisk(), did not consider FSCTL_LMR_REQUEST_RESILIENCY as an optional feature when the dialect was 2.1 or greater.  If the request for resiliency was not supported, the API failed and so did the feature.  The fix was to continue to request if FSCTL_LMR_REQUEST_RESILIENCY is supported (when the negotiated dialect is 2.1 or greater), but NOT to consider a response of STATUS_NOT_SUPPORTED as a hard failure.  This also affected other callers of CreateVirtualDisk(), which included using the Hyper-V Manager to create a new hard disk on a remote share.

I tested both Windows Server Backup on a Windows Server 2016 TP4 box and creating a VHDX from a Windows 10 box using the Hyper-V Manager against a server I modified to always return STATUS_NOT_SUPPORTED to FSCTL_LMR_REQUEST_RESILIENCY requests.  I am satisfied that the issue that was reported in 2013 and fixed soon thereafter continues to be fixed in Windows 10 and Server 2016.

If you have reports of issues, we would be very interested if additional information becomes available.  But, if there is a failure, it is likely a different root-cause that would need to be investigated.

Bryan

-----Original Message-----
From: Uri Simchoni [mailto:[hidden email]]
Sent: Tuesday, February 16, 2016 11:05 AM
To: Bryan Burgin <[hidden email]>; [hidden email]
Subject: Re: [RFC PATCH] smbd: resilient file handle support

On 02/16/2016 08:27 PM, Bryan Burgin wrote:

> Responding to:
> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2flists
> .samba.org%2farchive%2fsamba-technical%2f2016-February%2f112112.html&d
> ata=01%7c01%7cbburgin%40microsoft.com%7cab0f74b5dde44f54ca8c08d337041a
> a1%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=Wv8enCrIp9emtlr73s6T0X
> JQZ%2fWmmbRGimwOfJl5qlE%3d
>
> And
> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fbugzi
> lla.samba.org%2fshow_bug.cgi%3fid%3d10159&data=01%7c01%7cbburgin%40mic
> rosoft.com%7cab0f74b5dde44f54ca8c08d337041aa1%7c72f988bf86f141af91ab2d
> 7cd011db47%7c1&sdata=7yCAshPJHIz6uXjp5Fndo8juTs2Cx6tB0K2fb5kUwKs%3d
>
>
>
>
>
> Focusing on "Regarding "I didn't plan this. The driver for doing this was that we've just found out that Windows backup engine doesn't work with Samba and SMB dialect > 2.0.2 (see https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fbugzilla.samba.org%2fshow_bug.cgi%3fid%3d10159&data=01%7c01%7cbburgin%40microsoft.com%7cab0f74b5dde44f54ca8c08d337041aa1%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=7yCAshPJHIz6uXjp5Fndo8juTs2Cx6tB0K2fb5kUwKs%3d)", this was resolved several years ago when it was reported.  I'm with the Microsoft protocol support team and work with your Samba colleagues regularly on technical issues.  Please take note of KB 2920193: https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fsupport.microsoft.com%2fen-us%2fkb%2f2920193%3a&data=01%7c01%7cbburgin%40microsoft.com%7cab0f74b5dde44f54ca8c08d337041aa1%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=sZu50spM7WQE7tfVOqHEGl9Rqv7l8NYB%2bHpuLDHO4D8%3d "Virtual hard disk cannot be created on an SMB server without resiliency support from a Windows computer".  For Windows 8 and Server 2012, you need to download and install the patch.  For Windows 8.1 and Server 2012 R2, it was fixed automatically as long as the system installed the update rollup 2919355: https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fsupport.microsoft.com%2fen-us%2fkb%2f2919355.&data=01%7c01%7cbburgin%40microsoft.com%7cab0f74b5dde44f54ca8c08d337041aa1%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=zeZoFTxxOf3cN9QjVh8MOz5UzmvLrEQexHNlBITdevE%3d  This was fixed in Windows 10 before it was released, so there is no action necessary regarding Windows 10."
>
Bryan,

I will check. I have reports that we fail Windows 10 backup too unless downgrading to SMB2.0.2, but I haven't studied it closely as with Win8.

Thanks,
Uri.


Reply | Threaded
Open this post in threaded view
|

Re: [RFC PATCH] smbd: resilient file handle support

Uri Simchoni-4
Bryan,

Sorry for not updating - I haven't had time to figure it out yet, but
there's one thing I can already confirm, namely that the Win10 failure
against our box has nothing to do with resilient file handles - the
failure occurs before creating the .vhdx. I'll update with findings when
I have them.

Thanks,
Uri.

On 02/19/2016 08:53 PM, Bryan Burgin wrote:

> Uri,
>
> An update, following up on your comment "I [Uri] will check. I [Uri] have reports that we fail Windows 10 backup too unless downgrading to SMB2.0.2, but I haven't studied it closely as with Win8":
>
> Regarding the Bugzilla report at https://bugzilla.samba.org/show_bug.cgi?id=10159 and the fix we created for Windows 8 via the on-demand (you have to pull down the fix) QFE patch https://support.microsoft.com/en-us/kb/2920193 and Windows 8.1 via the "important" Windows Update roll-up patch https://support.microsoft.com/en-us/kb/2919355 is also included in Windows 10.
>
> The end-user scenarios that were reported back in 2013 was that the feature Windows Server Backup did not work when the target SMB2 server was a dialect 2.1 and above and, as was discovered as the root-cause, the server did not support FSCTL_LMR_REQUEST_RESILIENCY.  Samba servers at the time returned STATUS_NOT_SUPPORTED.  An API that Windows Server Backup relied on, CreateVirtualDisk(), did not consider FSCTL_LMR_REQUEST_RESILIENCY as an optional feature when the dialect was 2.1 or greater.  If the request for resiliency was not supported, the API failed and so did the feature.  The fix was to continue to request if FSCTL_LMR_REQUEST_RESILIENCY is supported (when the negotiated dialect is 2.1 or greater), but NOT to consider a response of STATUS_NOT_SUPPORTED as a hard failure.  This also affected other callers of CreateVirtualDisk(), which included using the Hyper-V Manager to create a new hard disk on a remote share.
>
> I tested both Windows Server Backup on a Windows Server 2016 TP4 box and creating a VHDX from a Windows 10 box using the Hyper-V Manager against a server I modified to always return STATUS_NOT_SUPPORTED to FSCTL_LMR_REQUEST_RESILIENCY requests.  I am satisfied that the issue that was reported in 2013 and fixed soon thereafter continues to be fixed in Windows 10 and Server 2016.
>
> If you have reports of issues, we would be very interested if additional information becomes available.  But, if there is a failure, it is likely a different root-cause that would need to be investigated.
>
> Bryan
>


Reply | Threaded
Open this post in threaded view
|

RE: [RFC PATCH] smbd: resilient file handle support

Bryan Burgin
Hi Uri,

Then there is some other cause to this.  I'll be happy to work on this with you or any Samba colleague that is willing to work on this.  When you are ready, please send mail to me ("bburgin (at) microsoft (dot) com") AND to Dochelp ("dochelp (at) microsoft (dot) com").  I would want exact repro instructions, including the Samba setup necessary,and network traces.

Bryan

-----Original Message-----
From: Uri Simchoni [mailto:[hidden email]]
Sent: Saturday, February 20, 2016 6:30 AM
To: Bryan Burgin <[hidden email]>; [hidden email]
Subject: Re: [RFC PATCH] smbd: resilient file handle support

Bryan,

Sorry for not updating - I haven't had time to figure it out yet, but there's one thing I can already confirm, namely that the Win10 failure against our box has nothing to do with resilient file handles - the failure occurs before creating the .vhdx. I'll update with findings when I have them.

Thanks,
Uri.

On 02/19/2016 08:53 PM, Bryan Burgin wrote:

> Uri,
>
> An update, following up on your comment "I [Uri] will check. I [Uri] have reports that we fail Windows 10 backup too unless downgrading to SMB2.0.2, but I haven't studied it closely as with Win8":
>
> Regarding the Bugzilla report at https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fbugzilla.samba.org%2fshow_bug.cgi%3fid%3d10159&data=01%7c01%7cbburgin%40microsoft.com%7cfacf215583da49b1555808d33a02437e%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=JnfGXRbqXN3BZtsbVAq0C%2buRMruZbHvbXgqCnMnuu4A%3d and the fix we created for Windows 8 via the on-demand (you have to pull down the fix) QFE patch https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fsupport.microsoft.com%2fen-us%2fkb%2f2920193&data=01%7c01%7cbburgin%40microsoft.com%7cfacf215583da49b1555808d33a02437e%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=hwXtyPbAXZDHSdnlA3dtJ5opMt%2bYiDxC%2f7OpCZoeykk%3d and Windows 8.1 via the "important" Windows Update roll-up patch https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fsupport.microsoft.com%2fen-us%2fkb%2f2919355&data=01%7c01%7cbburgin%40microsoft.com%7cfacf215583da49b1555808d33a02437e%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=JDXNArgPx%2bRmCB6ayXL7Oy3Dat0LVK9AZryu1UTR7f8%3d is also included in Windows 10.
>
> The end-user scenarios that were reported back in 2013 was that the feature Windows Server Backup did not work when the target SMB2 server was a dialect 2.1 and above and, as was discovered as the root-cause, the server did not support FSCTL_LMR_REQUEST_RESILIENCY.  Samba servers at the time returned STATUS_NOT_SUPPORTED.  An API that Windows Server Backup relied on, CreateVirtualDisk(), did not consider FSCTL_LMR_REQUEST_RESILIENCY as an optional feature when the dialect was 2.1 or greater.  If the request for resiliency was not supported, the API failed and so did the feature.  The fix was to continue to request if FSCTL_LMR_REQUEST_RESILIENCY is supported (when the negotiated dialect is 2.1 or greater), but NOT to consider a response of STATUS_NOT_SUPPORTED as a hard failure.  This also affected other callers of CreateVirtualDisk(), which included using the Hyper-V Manager to create a new hard disk on a remote share.
>
> I tested both Windows Server Backup on a Windows Server 2016 TP4 box and creating a VHDX from a Windows 10 box using the Hyper-V Manager against a server I modified to always return STATUS_NOT_SUPPORTED to FSCTL_LMR_REQUEST_RESILIENCY requests.  I am satisfied that the issue that was reported in 2013 and fixed soon thereafter continues to be fixed in Windows 10 and Server 2016.
>
> If you have reports of issues, we would be very interested if additional information becomes available.  But, if there is a failure, it is likely a different root-cause that would need to be investigated.
>
> Bryan
>


Reply | Threaded
Open this post in threaded view
|

Re: [RFC PATCH] smbd: resilient file handle support

Stefan Metzmacher-2
Hi,

> Then there is some other cause to this.  I'll be happy to work on this with you or any Samba colleague that is willing to
> work on this.  When you are ready, please send mail to me ("bburgin (at) microsoft (dot) com") AND to
> Dochelp ("dochelp (at) microsoft (dot) com").

AND cc: [hidden email] :-)

Thanks for the help Bryan!

metze


signature.asc (853 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [RFC PATCH] smbd: resilient file handle support

Uri Simchoni-4
On 02/22/2016 06:56 PM, Stefan Metzmacher wrote:

> Hi,
>
>> Then there is some other cause to this.  I'll be happy to work on this with you or any Samba colleague that is willing to
>> work on this.  When you are ready, please send mail to me ("bburgin (at) microsoft (dot) com") AND to
>> Dochelp ("dochelp (at) microsoft (dot) com").
>
> AND cc: [hidden email] :-)
>
> Thanks for the help Bryan!
>
> metze
>
So this time it's the SVHDX_OPEN_DEVICE_CONTEXT that we're not
supporting and Win10 backup (and probably Win8.1) are requesting. Should
I bring this in an orderly way like Bryan had suggested or is this
already a known issue?

(...and sorry for not being entirely accurate - the .vhdx does get
created but then opened with this create context that we do not support)

Thanks,
Uri

Reply | Threaded
Open this post in threaded view
|

RE: [RFC PATCH] smbd: resilient file handle support

Bryan Burgin
[MS-RSVD] now has two open device context structures: V1 and V2.  See the preview markup at http://download.microsoft.com/download/C/6/C/C6C3C6F1-E84A-44EF-82A9-49BD3AAD8F58/Windows/[MS-RSVD-Diff].pdf and 2.2.4.12 SVHDX_OPEN_DEVICE_CONTEXT and 2.2.4.32 SVHDX_OPEN_DEVICE_CONTEXT_V2.  You should be able to ignore either if you don't recognize this?  Are you ignoring or sending back an error?  There's a bit of discovery that I had with Steve French and ignoring contexts you don't understand is appropriate.

I would suggest opening a case with us so we (I) can work with you directly.  Please send mail to the Microsoft Dochelp alias as well as your technical mail list); we probably hijacked this thread too much already and should start a new thread anyway.  Also, please supply a network trace.

Bryan

-----Original Message-----
From: Uri Simchoni [mailto:[hidden email]]
Sent: Tuesday, February 23, 2016 1:30 AM
To: Stefan Metzmacher <[hidden email]>; Bryan Burgin <[hidden email]>; [hidden email]
Subject: Re: [RFC PATCH] smbd: resilient file handle support

On 02/22/2016 06:56 PM, Stefan Metzmacher wrote:

> Hi,
>
>> Then there is some other cause to this.  I'll be happy to work on
>> this with you or any Samba colleague that is willing to work on this.  
>> When you are ready, please send mail to me ("bburgin (at) microsoft (dot) com") AND to Dochelp ("dochelp (at) microsoft (dot) com").
>
> AND cc: [hidden email] :-)
>
> Thanks for the help Bryan!
>
> metze
>
So this time it's the SVHDX_OPEN_DEVICE_CONTEXT that we're not supporting and Win10 backup (and probably Win8.1) are requesting. Should I bring this in an orderly way like Bryan had suggested or is this already a known issue?

(...and sorry for not being entirely accurate - the .vhdx does get created but then opened with this create context that we do not support)

Thanks,
Uri