Linux vs. Windows SMB buffering

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Linux vs. Windows SMB buffering

Samba - General mailing list
Hi all,

In our office we use a program that makes many small writes to a file. We
have been trying to use this program with our NAS, which hosts a Samba
share on the network. However, the performance of the program is
staggeringly slow on Window 7, 8.1, and 10.

To make the problem easily repeatable, I created a small C++ program to
just write 1 million lines to a file:
https://github.com/mevatron/nas-write-test

Next, I performed a packet capture of the two OS conversations with
Wireshark. I found the Linux SMB implementation buffer many hundreds of
lines into each write; whereas, Windows decides to immediately flush it's
buffer to disk on each line write, which obviously causes tremendous
protocol overhead and thus abysmal performance.

I have tried adjusting Lanmanworkstation parameters:
https://technet.microsoft.com/en-us/library/cc978438.aspx

The ones that looked promising are shown below:

MaxCollectionCount
"Specifies the amount of data that must be present in the character-mode
buffer of a named pipe to trigger a write operation. If the amount of data
in the buffer meets or exceeds this value, it is written immediately.
Otherwise, it is retained in the buffer until either more data is added or
the value of the CollectionTime entry expires.
Increasing the value of this entry can improve the performance of
named-pipe applications, but it does not affect applications, such as SQL
Server applications, that do their own buffering."

CollectionTime
"Specifies the maximum time that application data waiting to be written can
remain in the character-mode buffer of a named pipe .
The redirector writes data from the character-mode buffer when the amount
of buffered data reaches the size threshold established by the value of the
MaxCollectionCount entry. However, if the application has written at least
one byte of data to the buffer but not enough data to reach the size
threshold, the contents of the buffer are written when the time specified
by this value expires.
Increasing the value of this entry can improve the performance of named
pipe applications. However, it does not affect applications, such as SQL
Server applications, which do their own buffering."

Unfortunately, I am unable to get these parameters to have any effect.

I have also determined through testing that this problem persists even when
bypassing Samba on the NAS and just writing directly between to Windows
boxes. So, I don't think it's an issue with Samba, but with the Windows SMB
client.

I was hoping someone with more knowledge of SMB has seen this before or has
deeper insights into what I can do to solve it.

I have a more visual write up of all of this on ServerFault here:
https://serverfault.com/questions/808633/small-writes-to-smb-network-share-are-slow-on-windows-fast-over-cifs-linux-moun

I have a bounty of 100 points if you would like to collect :D

Thanks for your time!
Will
--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|

Re: Linux vs. Windows SMB buffering

Samba - General mailing list
On Fri, Oct 06, 2017 at 06:09:32PM +0000, Will Lucas via samba wrote:

> Hi all,
>
> In our office we use a program that makes many small writes to a file. We
> have been trying to use this program with our NAS, which hosts a Samba
> share on the network. However, the performance of the program is
> staggeringly slow on Window 7, 8.1, and 10.
>
> To make the problem easily repeatable, I created a small C++ program to
> just write 1 million lines to a file:
> https://github.com/mevatron/nas-write-test
>
> Next, I performed a packet capture of the two OS conversations with
> Wireshark. I found the Linux SMB implementation buffer many hundreds of
> lines into each write; whereas, Windows decides to immediately flush it's
> buffer to disk on each line write, which obviously causes tremendous
> protocol overhead and thus abysmal performance.
>
> I was hoping someone with more knowledge of SMB has seen this before or has
> deeper insights into what I can do to solve it.
>
> I have a more visual write up of all of this on ServerFault here:
> https://serverfault.com/questions/808633/small-writes-to-smb-network-share-are-slow-on-windows-fast-over-cifs-linux-moun

Does the wireshark trace show the Windows client asking for and
getting a RWH lease under SMB2 ? If so, then there's no reason
it can't be caching the entire file locally. Seems strange
behaviour from the Windows redirector here.

--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|

Re: Linux vs. Windows SMB buffering

Samba - General mailing list
In reply to this post by Samba - General mailing list
On Fri, Oct 06, 2017 at 06:09:32PM +0000, Will Lucas via samba wrote:

> Hi all,
>
> In our office we use a program that makes many small writes to a file. We
> have been trying to use this program with our NAS, which hosts a Samba
> share on the network. However, the performance of the program is
> staggeringly slow on Window 7, 8.1, and 10.
>
> To make the problem easily repeatable, I created a small C++ program to
> just write 1 million lines to a file:
> https://github.com/mevatron/nas-write-test
>
> Next, I performed a packet capture of the two OS conversations with
> Wireshark. I found the Linux SMB implementation buffer many hundreds of
> lines into each write; whereas, Windows decides to immediately flush it's
> buffer to disk on each line write, which obviously causes tremendous
> protocol overhead and thus abysmal performance.

Or alternatively, just use the Linux SMB client and don't port
the software to Windows :-).

--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|

Re: Linux vs. Windows SMB buffering

Samba - General mailing list
In reply to this post by Samba - General mailing list
On Fri, Oct 06, 2017 at 06:09:32PM +0000, Will Lucas via samba wrote:
> Hi all,
>
> In our office we use a program that makes many small writes to a file. We

Also, try running this program under Wine on Linux to a Linux-mounted SMB share...

--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|

Linux vs. Windows SMB buffering

Samba - General mailing list
In reply to this post by Samba - General mailing list
>
>
> Does the wireshark trace show the Windows client asking for and
> getting a RWH lease under SMB2 ? If so, then there's no reason
> it can't be caching the entire file locally. Seems strange
> behaviour from the Windows redirector here.
>

I have uploaded the Linux packet capture here (14MB):
https://drive.google.com/open?id=0B6UHr3GQEkQwWXJ1NjVwMkJXOEU

Also, here is the corresponding Windows packet capture as well (375MB):
https://drive.google.com/open?id=0B6UHr3GQEkQwbFhOY0lCcEFjZkU

> Or alternatively, just use the Linux SMB client and don't port the
software to Windows :-).

You have no idea how much I wish that was possible :D I'm trying to slowly
convert my colleagues to Linux :)

Thanks for the quick reply!
Will
--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|

Re: Linux vs. Windows SMB buffering

Samba - General mailing list
In reply to this post by Samba - General mailing list
On Fri, 2017-10-06 at 18:09 +0000, Will Lucas via samba wrote:
> Hi all,
>

...

> Next, I performed a packet capture of the two OS conversations with
> Wireshark. I found the Linux SMB implementation buffer many hundreds
> of
> lines into each write; whereas, Windows decides to immediately flush
> it's
> buffer to disk on each line write, which obviously causes tremendous
> protocol overhead and thus abysmal performance.
>
> I have tried adjusting Lanmanworkstation parameters:
> https://technet.microsoft.com/en-us/library/cc978438.aspx
>
> The ones that looked promising are shown below:
>
> MaxCollectionCount
> "Specifies the amount of data that must be present in the character-
> mode
> buffer of a named pipe to trigger a write operation. If the amount of
> data
> in the buffer meets or exceeds this value, it is written immediately.
> Otherwise, it is retained in the buffer until either more data is
> added or
> the value of the CollectionTime entry expires.
> Increasing the value of this entry can improve the performance of
> named-pipe applications, but it does not affect applications, such as
> SQL
> Server applications, that do their own buffering."

...

I'm no expert here but how does say SQL Server indicate that it does
its own buffering?  I think you have managed to go as far as you can
with your approach and have hit a fundamental Windowism.  You clearly
know what you are doing and have spent quite a while on hitting a
brickwall, that needs (Windows) source code to get to the bottom of.

Might I suggest that you re-evaluate how your program works? Getting
"Seems strange behaviour from the Windows redirector here." from Mr A
does imply that it may not be easy to fix.

A first thought is, can you buffer reads at the client end before
sending them to the network?

Cheers
Jon
--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|

Re: Linux vs. Windows SMB buffering

Samba - General mailing list
In reply to this post by Samba - General mailing list
On Fri, Oct 06, 2017 at 07:26:29PM +0000, Will Lucas wrote:

>
>     Does the wireshark trace show the Windows client asking for and
>     getting a RWH lease under SMB2 ? If so, then there's no reason
>     it can't be caching the entire file locally. Seems strange
>     behaviour from the Windows redirector here.
>
>
> I have uploaded the Linux packet capture here (14MB): https://drive.google.com/
> open?id=0B6UHr3GQEkQwWXJ1NjVwMkJXOEU
>
> Also, here is the corresponding Windows packet capture as well (375MB): https:/
> /drive.google.com/open?id=0B6UHr3GQEkQwbFhOY0lCcEFjZkU
> to Windows :-).

The Linux client is asking for SMB1 and using 1MB write sizes.

The Windows client is using SMB2 and *NOT ASKING FOR LEASES*.

This is why the performance is terrible. Because the file
as no lease, the Windows redirector must pass every single
WriteFile() system call onto the wire, no matter how small
the size.

If you can get the Windows SMB2 client to request a lease,
the performance problem will be solved.

What is your server ? Is it Samba ? If so what version ?

--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|

Re: Linux vs. Windows SMB buffering

Samba - General mailing list
> The Linux client is asking for SMB1 and using 1MB write sizes.
>
> The Windows client is using SMB2 and *NOT ASKING FOR LEASES*.
>
> This is why the performance is terrible. Because the file
> as no lease, the Windows redirector must pass every single
> WriteFile() system call onto the wire, no matter how small
> the size.
>
> If you can get the Windows SMB2 client to request a lease,
> the performance problem will be solved.

Jeremy,

Thanks for the response that's definitely more information than I've been
able to find over the past year!

> What is your server ? Is it Samba ? If so what version ?

My NAS is a Synology DS215j and it is running Samba. smbd -V gives me the
following:
Version 4.4.13
Synology Build 15152, Jul 13 2017 04:48:37

So, this morning I ran a test by enabling leases, which were not enabled on
the server before. So, now my smb.conf looks like this:
[global]
printcap name=cups
winbind enum groups=yes
include=/var/tmp/nginx/smb.netbios.aliases.conf
min protocol=NT1
socket options=TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=65536 SO_SNDBUF=65536
security=user
local master=no
realm=*
use sendfile=yes
min receivefile size=16384
passdb backend=smbpasswd
write cache size=2097152
smb2 leases=yes
printing=cups
max protocol=SMB3
winbind enum users=yes
getwd cache=true
load printers=yes
workgroup=WORKGROUP

I noticed now that the packet capture shows Negotiate Protocol Response
conversations between the server and the Windows client. It also seems the
client is now using a lease.  Unfortunately, the Windows client is still
immediately writing one line at a time :(

I have uploaded the new packet capture snippet here (2MB):
https://drive.google.com/open?id=0B6UHr3GQEkQwcXdEWVRFSXRMNnc

Hopefully, I'm getting closer? :) Does the smb.conf looks reasonable to you?

Thanks again for all your help, it's greatly appreciated!
Will

On Fri, Oct 6, 2017 at 7:09 PM Jeremy Allison <[hidden email]> wrote:

> On Fri, Oct 06, 2017 at 07:26:29PM +0000, Will Lucas wrote:
> >
> >     Does the wireshark trace show the Windows client asking for and
> >     getting a RWH lease under SMB2 ? If so, then there's no reason
> >     it can't be caching the entire file locally. Seems strange
> >     behaviour from the Windows redirector here.
> >
> >
> > I have uploaded the Linux packet capture here (14MB):
> https://drive.google.com/
> > open?id=0B6UHr3GQEkQwWXJ1NjVwMkJXOEU
> >
> > Also, here is the corresponding Windows packet capture as well
> (375MB): https:/
> > /drive.google.com/open?id=0B6UHr3GQEkQwbFhOY0lCcEFjZkU
> > to Windows :-).
>
> The Linux client is asking for SMB1 and using 1MB write sizes.
>
> The Windows client is using SMB2 and *NOT ASKING FOR LEASES*.
>
> This is why the performance is terrible. Because the file
> as no lease, the Windows redirector must pass every single
> WriteFile() system call onto the wire, no matter how small
> the size.
>
> If you can get the Windows SMB2 client to request a lease,
> the performance problem will be solved.
>
> What is your server ? Is it Samba ? If so what version ?
>
--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|

Re: Linux vs. Windows SMB buffering

Samba - General mailing list
On Mon, Oct 09, 2017 at 08:29:43PM +0000, Will Lucas wrote:

> > The Linux client is asking for SMB1 and using 1MB write sizes.
> > 
> > The Windows client is using SMB2 and *NOT ASKING FOR LEASES*.
> > 
> > This is why the performance is terrible. Because the file
> > as no lease, the Windows redirector must pass every single
> > WriteFile() system call onto the wire, no matter how small
> > the size.
> > 
> > If you can get the Windows SMB2 client to request a lease,
> > the performance problem will be solved.
>
> Jeremy,
>
> Thanks for the response that's definitely more information than I've been able
> to find over the past year!
>
> > What is your server ? Is it Samba ? If so what version ?
>
> My NAS is a Synology DS215j and it is running Samba. smbd -V gives me the
> following:
> Version 4.4.13
> Synology Build 15152, Jul 13 2017 04:48:37
>
> So, this morning I ran a test by enabling leases, which were not enabled on the
> server before. So, now my smb.conf looks like this:
> [global]
> printcap name=cups
> winbind enum groups=yes
> include=/var/tmp/nginx/smb.netbios.aliases.conf
> min protocol=NT1
> socket options=TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=65536 SO_SNDBUF=65536
> security=user
> local master=no
> realm=*
> use sendfile=yes
> min receivefile size=16384
> passdb backend=smbpasswd
> write cache size=2097152
> smb2 leases=yes
> printing=cups
> max protocol=SMB3
> winbind enum users=yes
> getwd cache=true
> load printers=yes
> workgroup=WORKGROUP
>
> I noticed now that the packet capture shows Negotiate Protocol Response
> conversations between the server and the Windows client. It also seems the
> client is now using a lease.  Unfortunately, the Windows client is still
> immediately writing one line at a time :(
>
> I have uploaded the new packet capture snippet here (2MB): https://
> drive.google.com/open?id=0B6UHr3GQEkQwcXdEWVRFSXRMNnc
>
> Hopefully, I'm getting closer? :) Does the smb.conf looks reasonable to you?

So it's now getting a RWH lease, but still doing the raw
syscall reads across the wire.

Does it do the same to a Windows server ? Looks like the
client application is deliberately turning off any
redirector caching by using FILE_FLAG_WRITE_THROUGH on
the CreateFile() Win32 call. If you can do the equivalent
of strace on Windows (I don't know how) you might be
able to confirm that and ask the app developer to stop
doing that.

https://msdn.microsoft.com/en-us/library/windows/desktop/aa363858(v=vs.85).aspx


FILE_FLAG_WRITE_THROUGH
0x80000000

       

Write operations will not go through any intermediate cache, they will go directly to disk.

For additional information, see the Caching Behavior section of this topic.


--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|

Re: Linux vs. Windows SMB buffering

Samba - General mailing list
> So it's now getting a RWH lease, but still doing the raw
> syscall reads across the wire.
>
> Does it do the same to a Windows server ? Looks like the
> client application is deliberately turning off any
> redirector caching by using FILE_FLAG_WRITE_THROUGH on
> the CreateFile() Win32 call. If you can do the equivalent
> of strace on Windows (I don't know how) you might be
> able to confirm that and ask the app developer to stop
> doing that.
>
>
https://msdn.microsoft.com/en-us/library/windows/desktop/aa363858(v=vs.85).aspx
>
>
> FILE_FLAG_WRITE_THROUGH
> 0x80000000
>
>
>
> Write operations will not go through any intermediate cache, they will go
directly to disk.
>
> For additional information, see the Caching Behavior section of this
topic.

Jeremy,

It turns out the program was definitely flushing the stream after every
write (I verified this was the case with ProcMon), but I couldn't figure
out why (as I wasn't specifying the flush explicitly). I also tried NFS on
Windows and it behaved exactly the same way. I was completely stumped until
someone pointed out that endl flushes the stream implicitly.

Now SMB is buffering properly! Your help was definitely instrumental in me
finding the ultimate cause, so thank you very much!

Will
--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba