Re: [Samba] The connection would be broken when read rate is too big

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [Samba] The connection would be broken when read rate is too big

Jeremy Allison
On Wed, Mar 01, 2017 at 08:31:32AM +0000, Chenyehua via samba wrote:

> Hi samba-team,
> I met a problem recently:
>
> Problem Description:
> The connection would be broken when read rate is too big.
>
> Environment:
> Samba 4.3.11
> Ubuntu server 14.04
>
> Configuration:
> usershare allow guests = yes
>    max protocol = SMB3
>    large readwrite = yes
>    use sendfile = yes
>    aio read size = 1024
>    oplocks = yes
>    deadtime = 10
>    aio write behind = true
>    clustering = yes
>    store dos attributes = yes
>    vfs objects = acl_xattr
>    socket options = TCP_NODELAY SO_RCVBUF=131072 SO_SNDBUF=131072
>    ctdbd socket = /var/run/ctdb/ctdbd.socket
>    log level = 2
>    security = user
>
> I add some logs, found when the send queue is more than the max, it won’t allocate memory to the state->req in smbd_smb2_request_next_incoming function.
> Then, the smbd_smb2_io_handler function mark the fd to not readable because state->req == NULL, and never mark it to readable again.
> Then the Request Expiration Timer will break the connection.
>
> I don’t know why the fd is marked to readable when the send queue is more than the max.
> Or why mark the fd readable but not allocate memory to the state->req.
>
> My solution:
> Allocate memory to the state->req before return when the send queue is more than the max in smbd_smb2_request_next_incoming function.
>
> Could you give us some help about the problem.

I'm looking at this now. Your report looks correct.
Can you log a bug at bugzilla.samba.org so we can
track this ?

I'm redirecting the reply to [hidden email]
as it more properly belongs there than at [hidden email].

Cheers,

        Jeremy.

Reply | Threaded
Open this post in threaded view
|

Re: [Samba] The connection would be broken when read rate is too big

Jeremy Allison
On Wed, Mar 01, 2017 at 04:29:49PM -0800, Jeremy Allison wrote:

> On Wed, Mar 01, 2017 at 08:31:32AM +0000, Chenyehua via samba wrote:
> > Hi samba-team,
> > I met a problem recently:
> >
> > Problem Description:
> > The connection would be broken when read rate is too big.
> >
> > Environment:
> > Samba 4.3.11
> > Ubuntu server 14.04
> >
> > Configuration:
> > usershare allow guests = yes
> >    max protocol = SMB3
> >    large readwrite = yes
> >    use sendfile = yes
> >    aio read size = 1024
> >    oplocks = yes
> >    deadtime = 10
> >    aio write behind = true
> >    clustering = yes
> >    store dos attributes = yes
> >    vfs objects = acl_xattr
> >    socket options = TCP_NODELAY SO_RCVBUF=131072 SO_SNDBUF=131072
> >    ctdbd socket = /var/run/ctdb/ctdbd.socket
> >    log level = 2
> >    security = user
> >
> > I add some logs, found when the send queue is more than the max, it won’t allocate memory to the state->req in smbd_smb2_request_next_incoming function.
> > Then, the smbd_smb2_io_handler function mark the fd to not readable because state->req == NULL, and never mark it to readable again.
> > Then the Request Expiration Timer will break the connection.
> >
> > I don’t know why the fd is marked to readable when the send queue is more than the max.
> > Or why mark the fd readable but not allocate memory to the state->req.
> >
> > My solution:
> > Allocate memory to the state->req before return when the send queue is more than the max in smbd_smb2_request_next_incoming function.
> >
> > Could you give us some help about the problem.
>
> I'm looking at this now. Your report looks correct.
> Can you log a bug at bugzilla.samba.org so we can
> track this ?
>
> I'm redirecting the reply to [hidden email]
> as it more properly belongs there than at [hidden email].
Can you try the following (raw and untested) patch ?

It should also apply to 4.3.11.

If you can confirm it fixes your problem then I'll
make sure it gets pushed upstream.

Thanks !

Jeremy.
Reply | Threaded
Open this post in threaded view
|

Re: [Samba] The connection would be broken when read rate is too big

Ralph Böhme-2
On Wed, Mar 01, 2017 at 05:12:23PM -0800, Jeremy Allison wrote:

> On Wed, Mar 01, 2017 at 04:29:49PM -0800, Jeremy Allison wrote:
> > On Wed, Mar 01, 2017 at 08:31:32AM +0000, Chenyehua via samba wrote:
> > > I don’t know why the fd is marked to readable when the send queue is more than the max.
> > > Or why mark the fd readable but not allocate memory to the state->req.
> > >
> > > My solution: Allocate memory to the state->req before return when the send
> > > queue is more than the max in smbd_smb2_request_next_incoming function.
> > >
> > > Could you give us some help about the problem.
> >
> > I'm looking at this now. Your report looks correct.
> > Can you log a bug at bugzilla.samba.org so we can
> > track this ?
>
> Can you try the following (raw and untested) patch ?

Fwiw, I looked into this as well and came to the same conclusion.

Cheerio!
-slow

Reply | Threaded
Open this post in threaded view
|

Re: [Samba] The connection would be broken when read rate is too big

Jeremy Allison
On Thu, Mar 02, 2017 at 11:03:50AM +0100, Ralph Böhme wrote:

> On Wed, Mar 01, 2017 at 05:12:23PM -0800, Jeremy Allison wrote:
> > On Wed, Mar 01, 2017 at 04:29:49PM -0800, Jeremy Allison wrote:
> > > On Wed, Mar 01, 2017 at 08:31:32AM +0000, Chenyehua via samba wrote:
> > > > I don’t know why the fd is marked to readable when the send queue is more than the max.
> > > > Or why mark the fd readable but not allocate memory to the state->req.
> > > >
> > > > My solution: Allocate memory to the state->req before return when the send
> > > > queue is more than the max in smbd_smb2_request_next_incoming function.
> > > >
> > > > Could you give us some help about the problem.
> > >
> > > I'm looking at this now. Your report looks correct.
> > > Can you log a bug at bugzilla.samba.org so we can
> > > track this ?
> >
> > Can you try the following (raw and untested) patch ?
>
> Fwiw, I looked into this as well and came to the same conclusion.
Here's the official patch to the bug I logged:

https://bugzilla.samba.org/show_bug.cgi?id=12608

I managed to reproduce locally by
setting aio read size = 1 and then mangling smbd_smb2_request_next_incoming()
to block any reads if there were any outgoing entries
in the queue - but it wasn't reliably reproducible or easily
tested. If you can think of a good way to add a reliable
regression test I'd be happy to review, but I think the
patch is good on its own.

Please review and push if happy.

Cheers,

        Jeremy.

0001-s3-smbd-Restart-reading-the-incoming-SMB2-fd-when-th.patch (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Samba] The connection would be broken when read rate is too big

Ralph Böhme-2
On Thu, Mar 02, 2017 at 01:23:50PM -0800, Jeremy Allison wrote:

> On Thu, Mar 02, 2017 at 11:03:50AM +0100, Ralph Böhme wrote:
> > On Wed, Mar 01, 2017 at 05:12:23PM -0800, Jeremy Allison wrote:
> > > On Wed, Mar 01, 2017 at 04:29:49PM -0800, Jeremy Allison wrote:
> > > > On Wed, Mar 01, 2017 at 08:31:32AM +0000, Chenyehua via samba wrote:
> > > > > I don’t know why the fd is marked to readable when the send queue is more than the max.
> > > > > Or why mark the fd readable but not allocate memory to the state->req.
> > > > >
> > > > > My solution: Allocate memory to the state->req before return when the send
> > > > > queue is more than the max in smbd_smb2_request_next_incoming function.
> > > > >
> > > > > Could you give us some help about the problem.
> > > >
> > > > I'm looking at this now. Your report looks correct.
> > > > Can you log a bug at bugzilla.samba.org so we can
> > > > track this ?
> > >
> > > Can you try the following (raw and untested) patch ?
> >
> > Fwiw, I looked into this as well and came to the same conclusion.
>
> Here's the official patch to the bug I logged:
>
> https://bugzilla.samba.org/show_bug.cgi?id=12608
>
> I managed to reproduce locally by
> setting aio read size = 1 and then mangling smbd_smb2_request_next_incoming()
> to block any reads if there were any outgoing entries
> in the queue - but it wasn't reliably reproducible or easily
> tested.
Something like the attached should work to reproduce it. Untested, but you get
the idea.

> If you can think of a good way to add a reliable
> regression test I'd be happy to review, but I think the
> patch is good on its own.
>
> Please review and push if happy.

Pushed!

Cheerio!
-slow

reproducer-bug12608.patch (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Samba] The connection would be broken when read rate is too big

Stefan Metzmacher-2
Am 02.03.2017 um 22:39 schrieb Ralph Böhme:

> On Thu, Mar 02, 2017 at 01:23:50PM -0800, Jeremy Allison wrote:
>> On Thu, Mar 02, 2017 at 11:03:50AM +0100, Ralph Böhme wrote:
>>> On Wed, Mar 01, 2017 at 05:12:23PM -0800, Jeremy Allison wrote:
>>>> On Wed, Mar 01, 2017 at 04:29:49PM -0800, Jeremy Allison wrote:
>>>>> On Wed, Mar 01, 2017 at 08:31:32AM +0000, Chenyehua via samba wrote:
>>>>>> I don’t know why the fd is marked to readable when the send queue is more than the max.
>>>>>> Or why mark the fd readable but not allocate memory to the state->req.
>>>>>>
>>>>>> My solution: Allocate memory to the state->req before return when the send
>>>>>> queue is more than the max in smbd_smb2_request_next_incoming function.
>>>>>>
>>>>>> Could you give us some help about the problem.
>>>>>
>>>>> I'm looking at this now. Your report looks correct.
>>>>> Can you log a bug at bugzilla.samba.org so we can
>>>>> track this ?
>>>>
>>>> Can you try the following (raw and untested) patch ?
>>>
>>> Fwiw, I looked into this as well and came to the same conclusion.
>>
>> Here's the official patch to the bug I logged:
>>
>> https://bugzilla.samba.org/show_bug.cgi?id=12608
>>
>> I managed to reproduce locally by
>> setting aio read size = 1 and then mangling smbd_smb2_request_next_incoming()
>> to block any reads if there were any outgoing entries
>> in the queue - but it wasn't reliably reproducible or easily
>> tested.
>
> Something like the attached should work to reproduce it. Untested, but you get
> the idea.
You would also have to pass '1' instead of e->count to writev()
otherwise you'll just remove some bytes in the middle
and resend some bytes at the end.

metze


signature.asc (853 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Samba] The connection would be broken when read rate is too big

Stefan Metzmacher-2
In reply to this post by Jeremy Allison
Hi Jeremy,

> Here's the official patch to the bug I logged:
>
> https://bugzilla.samba.org/show_bug.cgi?id=12608
>
> I managed to reproduce locally by
> setting aio read size = 1 and then mangling smbd_smb2_request_next_incoming()
> to block any reads if there were any outgoing entries
> in the queue - but it wasn't reliably reproducible or easily
> tested. If you can think of a good way to add a reliable
> regression test I'd be happy to review, but I think the
> patch is good on its own.
>
> Please review and push if happy.
Looks good! Thanks!

metze


signature.asc (853 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Samba] The connection would be broken when read rate is too big

Ralph Böhme-2
In reply to this post by Stefan Metzmacher-2
On Fri, Mar 03, 2017 at 08:34:15AM +0100, Stefan Metzmacher wrote:

> Am 02.03.2017 um 22:39 schrieb Ralph Böhme:
> > On Thu, Mar 02, 2017 at 01:23:50PM -0800, Jeremy Allison wrote:
> >> On Thu, Mar 02, 2017 at 11:03:50AM +0100, Ralph Böhme wrote:
> >>> On Wed, Mar 01, 2017 at 05:12:23PM -0800, Jeremy Allison wrote:
> >>>> On Wed, Mar 01, 2017 at 04:29:49PM -0800, Jeremy Allison wrote:
> >>>>> On Wed, Mar 01, 2017 at 08:31:32AM +0000, Chenyehua via samba wrote:
> >>>>>> I don’t know why the fd is marked to readable when the send queue is more than the max.
> >>>>>> Or why mark the fd readable but not allocate memory to the state->req.
> >>>>>>
> >>>>>> My solution: Allocate memory to the state->req before return when the send
> >>>>>> queue is more than the max in smbd_smb2_request_next_incoming function.
> >>>>>>
> >>>>>> Could you give us some help about the problem.
> >>>>>
> >>>>> I'm looking at this now. Your report looks correct.
> >>>>> Can you log a bug at bugzilla.samba.org so we can
> >>>>> track this ?
> >>>>
> >>>> Can you try the following (raw and untested) patch ?
> >>>
> >>> Fwiw, I looked into this as well and came to the same conclusion.
> >>
> >> Here's the official patch to the bug I logged:
> >>
> >> https://bugzilla.samba.org/show_bug.cgi?id=12608
> >>
> >> I managed to reproduce locally by
> >> setting aio read size = 1 and then mangling smbd_smb2_request_next_incoming()
> >> to block any reads if there were any outgoing entries
> >> in the queue - but it wasn't reliably reproducible or easily
> >> tested.
> >
> > Something like the attached should work to reproduce it. Untested, but you get
> > the idea.
>
> You would also have to pass '1' instead of e->count to writev()
> otherwise you'll just remove some bytes in the middle
> and resend some bytes at the end.

indeed. :)

Cheerio!
-slow

Reply | Threaded
Open this post in threaded view
|

答复: [Samba] The connection would be broken when read rate is too big

Chenyehua
In reply to this post by Ralph Böhme-2
Thanks for the patch, Ralph
We applied your patch and began to test now!
The results may be reported in the next few days..

Cheers

-----邮件原件-----
发件人: Ralph Böhme [mailto:[hidden email]]
发送时间: 2017年3月3日 5:39
收件人: Jeremy Allison
抄送: chenyehua 11692 (RD); [hidden email]
主题: Re: [Samba] The connection would be broken when read rate is too big

On Thu, Mar 02, 2017 at 01:23:50PM -0800, Jeremy Allison wrote:

> On Thu, Mar 02, 2017 at 11:03:50AM +0100, Ralph Böhme wrote:
> > On Wed, Mar 01, 2017 at 05:12:23PM -0800, Jeremy Allison wrote:
> > > On Wed, Mar 01, 2017 at 04:29:49PM -0800, Jeremy Allison wrote:
> > > > On Wed, Mar 01, 2017 at 08:31:32AM +0000, Chenyehua via samba wrote:
> > > > > I don’t know why the fd is marked to readable when the send queue is more than the max.
> > > > > Or why mark the fd readable but not allocate memory to the state->req.
> > > > >
> > > > > My solution: Allocate memory to the state->req before return
> > > > > when the send queue is more than the max in smbd_smb2_request_next_incoming function.
> > > > >
> > > > > Could you give us some help about the problem.
> > > >
> > > > I'm looking at this now. Your report looks correct.
> > > > Can you log a bug at bugzilla.samba.org so we can track this ?
> > >
> > > Can you try the following (raw and untested) patch ?
> >
> > Fwiw, I looked into this as well and came to the same conclusion.
>
> Here's the official patch to the bug I logged:
>
> https://bugzilla.samba.org/show_bug.cgi?id=12608
>
> I managed to reproduce locally by
> setting aio read size = 1 and then mangling
> smbd_smb2_request_next_incoming() to block any reads if there were any
> outgoing entries in the queue - but it wasn't reliably reproducible or
> easily tested.

Something like the attached should work to reproduce it. Untested, but you get the idea.

> If you can think of a good way to add a reliable regression test I'd
> be happy to review, but I think the patch is good on its own.
>
> Please review and push if happy.

Pushed!

Cheerio!
-slow
-------------------------------------------------------------------------------------------------------------------------------------
本邮件及其附件含有杭州华三通信技术有限公司的保密信息,仅限于发送给上面地址中列出
的个人或群组。禁止任何其他人以任何形式使用(包括但不限于全部或部分地泄露、复制、
或散发)本邮件中的信息。如果您错收了本邮件,请您立即电话或邮件通知发件人并删除本
邮件!
This e-mail and its attachments contain confidential information from H3C, which is
intended only for the person or entity whose address is listed above. Any use of the
information contained herein in any way (including, but not limited to, total or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender
by phone or email immediately and delete it!
Reply | Threaded
Open this post in threaded view
|

Re: 答复: [Samba] The connection would be broken when read rate is too big

Ralph Böhme-2
On Fri, Mar 03, 2017 at 09:15:24AM +0000, Chenyehua wrote:
> Thanks for the patch, Ralph
> We applied your patch and began to test now!
> The results may be reported in the next few days..

don't forget to pass iovcnt=1 to writev as mentioned by metze.

Cheerio!
-slow

Reply | Threaded
Open this post in threaded view
|

答复: 答复: [Samba] The connection would be broken when read rate is too big

Chenyehua
Ok, thanks :-)

-----邮件原件-----
发件人: Ralph Böhme [mailto:[hidden email]]
发送时间: 2017年3月3日 17:27
收件人: chenyehua 11692 (RD)
抄送: Jeremy Allison; [hidden email]
主题: Re: 答复: [Samba] The connection would be broken when read rate is too big

On Fri, Mar 03, 2017 at 09:15:24AM +0000, Chenyehua wrote:
> Thanks for the patch, Ralph
> We applied your patch and began to test now!
> The results may be reported in the next few days..

don't forget to pass iovcnt=1 to writev as mentioned by metze.

Cheerio!
-slow
-------------------------------------------------------------------------------------------------------------------------------------
本邮件及其附件含有杭州华三通信技术有限公司的保密信息,仅限于发送给上面地址中列出
的个人或群组。禁止任何其他人以任何形式使用(包括但不限于全部或部分地泄露、复制、
或散发)本邮件中的信息。如果您错收了本邮件,请您立即电话或邮件通知发件人并删除本
邮件!
This e-mail and its attachments contain confidential information from H3C, which is
intended only for the person or entity whose address is listed above. Any use of the
information contained herein in any way (including, but not limited to, total or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender
by phone or email immediately and delete it!
Reply | Threaded
Open this post in threaded view
|

Re: 答复: [Samba] The connection would be broken when read rate is too big

Jeremy Allison
In reply to this post by Jeremy Allison
On Fri, Mar 03, 2017 at 08:31:25AM +0000, Chenyehua wrote:
> Thanks very much for the response, Jeremy and thanks for your kindness to help me to log the bug on the bugzilla ^_^
> Well, I applied your patch(the attached name is "|1") yesterday. The reading tests went well and the abnormal disconnection did not happen during those tests
> Besides, I have a problem:
> Do your know the detailed reason why the send queue finally grows larger than the max?

My guess is because you have those silly socket
options in your smb.conf. Remove them. Modern
kernels have self-tuning TCP.