Large Read/Write Patch

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Large Read/Write Patch

polivenok
Hi Guys,
 
We have noticed that on Windows 7 we expirence low trasfer rate for read/write operations and using wireshark we found that the reason was in buffer size. jcifs used 4K buffer even when we set snd/rcv buffer sizes and windows client used 64K. The reason of low 4K buffer was that it is default value for MaxBufferSize on windows samba server.
 
For normal SMB_COM_READ and SMB_COM_WRITE requests, the maximum buffer size is always equal to MaxBufferSize. But if CAP_LARGE_READX  or CAP_LARGE_WRITEX  capability is enabled on the SMB Negotiate Server Response,he maximum buffer size  used  is  61440 (60K)  for large read( SMB_COM_ READ_ANDX )  and 65535 (64K) for large write (SMB_COM_WRITE_ANDX) , regardless of MaxBufferSize.
 
More info can be found here:
    http://blogs.msdn.com/b/openspecification/archive/2009/04/10/smb-maximum-transmit-buffer-size-and-performance-tuning.aspx
 
With old LargeReadWrite.patch we had an issue with checking large/read write capability on linux samba server 3.5.4 so code for this check was modified a bit.
 
Below you can find some result for smb server with enabled large read/write:
jcifs-1.3.16                    ~2.5 M/s
jcifs-1.3.16 with patch    ~9M/s
Native client read/write   ~9.5M/s
Modified patch for 1.3.16 is attached.
 
Best Regards,
Dmytro Polivenok

LargeReadWrite.patch (8K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Large Read/Write Patch

Michael B Allen
On Mon, Jul 18, 2011 at 5:37 AM, polivenok <[hidden email]> wrote:

> Hi Guys,
>
> We have noticed that on Windows 7 we expirence low trasfer rate for
> read/write operations and using wireshark we found that the reason was in
> buffer size. jcifs used 4K buffer even when we set snd/rcv buffer sizes and
> windows client used 64K. The reason of low 4K buffer was that it is default
> value for MaxBufferSize on windows samba server.
>
> For normal SMB_COM_READ and SMB_COM_WRITE requests, the maximum buffer size
> is always equal to MaxBufferSize. But if CAP_LARGE_READX  or
> CAP_LARGE_WRITEX  capability is enabled on the SMB Negotiate Server
> Response,he maximum buffer size  used  is  61440 (60K)  for large read(
> SMB_COM_ READ_ANDX )  and 65535 (64K) for large write (SMB_COM_WRITE_ANDX) ,
> regardless of MaxBufferSize.
>
> More info can be found here:
>
> http://blogs.msdn.com/b/openspecification/archive/2009/04/10/smb-maximum-transmit-buffer-size-and-performance-tuning.aspx
>
> With old LargeReadWrite.patch we had an issue with checking large/read write
> capability on linux samba server 3.5.4 so code for this check was modified a
> bit.
>
> Below you can find some result for smb server with enabled large read/write:
> jcifs-1.3.16                    ~2.5 M/s
> jcifs-1.3.16 with patch    ~9M/s
> Native client read/write   ~9.5M/s
> Modified patch for 1.3.16 is attached.

Hi Dmytro,

I have placed your patch into the patches directory for others to use
and I have also added it to the TODO list for further consideration.
Unfortunately I do not have a lot of "Free" time these days (and I
just did a significant release) so I do not expect to get around to
this for a while. But when I do get around to adding this
functionality your patch should be very helpful. Thanks.

For those who may want to apply it, note that I have not really looked
at this patch closely other than to see that it does not do anything
obviously wrong. And note that the effects of buffer sizes on
performance can very greatly. Meaning it is possible that using a
larger buffer size results in throughput actually decreasing. This
might seem strange and although it is unlikely, I have seen it first
hand.

Mike

--
Michael B Allen
Java Active Directory Integration
http://www.ioplex.com/
Reply | Threaded
Open this post in threaded view
|

Re: Large Read/Write Patch

Robin Jansohn
I definitely support including this patch into the main package. Here are my results for jcifs-1.3.17 writing a 1GB file from linux to a windows share.

jcifs-1.3.17: 7.140865MB/s
jcifs-1.3.17-patch: 17.414967MB/s
smbclient: 41.6125MB/s

It's still not completely satisfactory when comparing to smbclient but it's at least a 2.5x increase in speed.
Reply | Threaded
Open this post in threaded view
|

Re: Large Read/Write Patch

Robin Jansohn
Unfortunately, when reading a 1GB file from the windows share to linux, the transfer speed does not improve with the patch and JCIFS in general is way, way slower than smbclient.

jcifs-1.3.17: 7.816794MB/s
jcifs-1.3.17-patch: 7.6646705MB/s
smbclient: 102.7856641MB/s
Reply | Threaded
Open this post in threaded view
|

Re: Large Read/Write Patch

Michael B Allen
On Wed, Dec 10, 2014 at 3:04 AM, Robin Jansohn <[hidden email]> wrote:
> Unfortunately, when reading a 1GB file from the windows share to linux, the
> transfer speed does not improve with the patch and JCIFS in general is way,
> way slower than smbclient.
>
> jcifs-1.3.17: 7.816794MB/s
> jcifs-1.3.17-patch: 7.6646705MB/s
> smbclient: 102.7856641MB/s

Hi Robin,

I just tried it and I got 22 MB/s with JCIFS vs 52 MB/s with
smbclient. I used examples/Get.java but changed the read buffer to
0xffff and commented out the println('#'). Also you need to run JCIFS
and smbclient a few times for the numbers to level off.

You might want to study a log and make sure there's nothing strange
going on like a name service timeout or something of that nature.
Also, if you mess around with properties that could easily break
something so try to stick with the default property values. Smbclient
is always going to be faster but it should be more like 2x and not 10x
like what you're seeing.

Mike

--
Michael B Allen
Java Active Directory Integration
http://www.ioplex.com/
Reply | Threaded
Open this post in threaded view
|

Re: Large Read/Write Patch

Robin Jansohn
Hi Michael,

I had a dependency problem which I fixed now. I can confirm that reading is faster too when using the patch. My (updated) results:

jcifs-1.3.17: 7.816794MB/s
jcifs-1.3.17-patch: 14.222222MB/s
smbclient: 102.7856641MB/s

I did not find anything suspicious in the logs so I guess this is the most I can get for now.

Robin
Reply | Threaded
Open this post in threaded view
|

Re: Large Read/Write Patch

Michael B Allen
On Fri, Dec 12, 2014 at 8:27 AM, Robin Jansohn <[hidden email]> wrote:

> Hi Michael,
>
> I had a dependency problem which I fixed now. I can confirm that reading is
> faster too when using the patch. My (updated) results:
>
> jcifs-1.3.17: 7.816794MB/s
> jcifs-1.3.17-patch: 14.222222MB/s
> smbclient: 102.7856641MB/s
>
> I did not find anything suspicious in the logs so I guess this is the most I
> can get for now.

A dependency problem? Are you running your test within a larger
application? If you run your test from within a webapp or something
like that, you might very well get different results. At least if you
want to compare it to smbclient, you would have to test it in a
similar manner like example/SmbGet. Otherwise, the comparison is not
terribly meaningful.

Mike

--
Michael B Allen
Java Active Directory Integration
http://www.ioplex.com/
Reply | Threaded
Open this post in threaded view
|

Re: Large Read/Write Patch

Robin Jansohn
No, I'm using a Maven repository and although I explicitly excluded the package it somehow did not work as expected. The test itself is run in a simple Main class, no webapp or similar large application.
Reply | Threaded
Open this post in threaded view
|

Re: Large Read/Write Patch

moder
In reply to this post by polivenok
 Hello,

I also confirm that read/write operations with the patch applied are significantly faster.

However, I think there is a problem with the patch:

According to the official SMB documentation:
http://download.microsoft.com/download/9/5/E/95EF66AF-9026-4BB0-A41D-A4F81802D92C/[MS-SMB].pdf

... sections 2.2.4.5, 2.2.4.5.2:
"Capabilities (4 bytes): A 32-bit field providing a set of server capability indicators. This bit field is
used to indicate to the client which features are supported by the server. Any value not listed in the
following table is unused. The server MUST set the unused bits to zero. The client MUST ignore
these bits.".

The CAP_LARGE_WRITEX flag that is part of the Capabilities (4 bytes) field is responsible for:

"The server supports large write operations. This
capability affects the maximum size, in bytes, of the
server buffer for receiving an SMB_COM_WRITE_ANDX
client request. When this capability is set by the server
(and set by the client in the
SMB_COM_SESSION_SETUP_ANDX request), then the
maximum server buffer size of bytes it writes can
exceed the MaxBufferSize field. Therefore, a client
can send a single SMB_COM_WRITE_ANDX request up
to this size.<34>
When signing is active on a connection, then clients
MUST limit write lengths to the MaxBufferSize value
negotiated by the server, irrespective of the value of
the CAP_LARGE_WRITEX flag."

... so it is true that we can safely ignore the MaxBufferSize and send larger chunks of data UNLESS signing is active on a connection. Therefore, I think checks for signing should be added to the patch.

Hope that helps!


Best regards,
M.D




 >-------- Оригинално писмо --------
 >От:  Robin Jansohn
 >Относно: Re: [jcifs] Large Read/Write Patch
 >До: [hidden email]
 >Изпратено на: Понеделник, 2014, Декември 15 09:08:49 EET
 >
 >
 >No, I'm using a Maven repository and although I explicitly excluded the
 >package it somehow did not work as expected. The test itself is run in a
 >simple Main class, no webapp or similar large application.
 >
 >
 >
 >--
 >View this message in context: http://samba.2283325.n4.nabble.com/Large-Read-Write-Patch-tp3674893p4677877.html
 >Sent from the Samba - jcifs mailing list archive at Nabble.com.
 >
Reply | Threaded
Open this post in threaded view
|

Re: Large Read/Write Patch

moder
In reply to this post by polivenok
 Hi,

I'm uploading a new patch. It now contains checks if signatures are desired. If that's the case then the old buffer size is used.

Best regards,
M.D




 >-------- Оригинално писмо --------
 >От:   M. D.  
 >Относно: Re: [jcifs] Large Read/Write Patch
 >До: Robin Jansohn
 >Изпратено на: Сряда, 2014, Декември 17 10:58:08 EET
 >
 >
 > Hello,
 >
 >I also confirm that read/write operations with the patch applied are significantly faster.
 >
 >However, I think there is a problem with the patch:
 >
 >According to the official SMB documentation:
 >http://download.microsoft.com/download/9/5/E/95EF66AF-9026-4BB0-A41D-A4F81802D92C/[MS-SMB].pdf
 >
 >... sections 2.2.4.5, 2.2.4.5.2:
 >&quot;Capabilities (4 bytes): A 32-bit field providing a set of server capability indicators. This bit field is
 >used to indicate to the client which features are supported by the server. Any value not listed in the
 >following table is unused. The server MUST set the unused bits to zero. The client MUST ignore
 >these bits.&quot;.
 >
 >The CAP_LARGE_WRITEX flag that is part of the Capabilities (4 bytes) field is responsible for:
 >
 >&quot;The server supports large write operations. This
 >capability affects the maximum size, in bytes, of the
 >server buffer for receiving an SMB_COM_WRITE_ANDX
 >client request. When this capability is set by the server
 >(and set by the client in the
 >SMB_COM_SESSION_SETUP_ANDX request), then the
 >maximum server buffer size of bytes it writes can
 >exceed the MaxBufferSize field. Therefore, a client
 >can send a single SMB_COM_WRITE_ANDX request up
 >to this size.
 >When signing is active on a connection, then clients
 >MUST limit write lengths to the MaxBufferSize value
 >negotiated by the server, irrespective of the value of
 >the CAP_LARGE_WRITEX flag.&quot;
 >
 >... so it is true that we can safely ignore the MaxBufferSize and send larger chunks of data UNLESS signing is active on a connection. Therefore, I think checks for signing should be added to the patch.
 >
 >Hope that helps!
 >
 >
 >Best regards,
 >M.D
 >
 >
 >
 >
 > >-------- Оригинално писмо --------
 > >От:  Robin Jansohn
 > >Относно: Re: [jcifs] Large Read/Write Patch
 > >До: [hidden email]
 > >Изпратено на: Понеделник, 2014, Декември 15 09:08:49 EET
 > >
 > >
 > >No, I'm using a Maven repository and although I explicitly excluded the
 > >package it somehow did not work as expected. The test itself is run in a
 > >simple Main class, no webapp or similar large application.
 > >
 > >
 > >
 > >--
 > >View this message in context: http://samba.2283325.n4.nabble.com/Large-Read-Write-Patch-tp3674893p4677877.html
 > >Sent from the Samba - jcifs mailing list archive at Nabble.com.
 > >
 >

LargeReadWrite.patch (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Large Read/Write Patch

Michael B Allen
On Wed, Dec 17, 2014 at 8:30 AM, M. D. <[hidden email]> wrote:
>  Hi,
>
> I'm uploading a new patch. It now contains checks if signatures are desired. If that's the case then the old buffer size is used.

Hi M. D.,

I put your patch in the patches directory.

Note to anyone using patches in the patches directory, they are not
scrutinized for correctness an the presence of a patch in the patches
directory is not an endorsement of the patch. Use at your own risk.

Thanks for the contrib,

Mike