Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf

classic Classic list List threaded Threaded
37 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Third Try: Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf

Samba - General mailing list
Sorry, Jeremy,

looks like we need to reiterate through those changes, as I am
constantly wondering whether just the one single line above, or all lines:

Am 14.07.2017 um 19:41 schrieb Jeremy Allison via samba:
> On Fri, Jul 14, 2017 at 04:19:09PM +0200, awl1 wrote:
>> [global]
>> server string = %h
>> max open files = 100000
>> deadtime = 15
>> dead time = 15
>> hide unreadable = yes
> Remove the above. Causes security descriptor lookup for every directory entry.
Remove just the one line "hide unreadable", correect?

>> load printers = no
>> log file = /var/log/samba.%m
>> max log size = 50
>> strict locking = no
>> lock directory = /var/samba
>> encrypt passwords = yes
>> case sensitive = true
>> default case = lower
>> preserve case = yes
>> short preserve case = yes
>> passdb backend = tdbsam
>> socket options = TCP_NODELAY IPTOS_LOWDELAY SO_KEEPALIVE
> Remove the above - voodoo bullsh*t not needed on modern kernels.
Remove just the one line "socket options", correct?

>> aio read size = 1
>> aio write size = 1
>> write cache size = 2097152
> REMOVE THE ABOVE WITH PREJUDICE !!!!!
As you already stated, removed just the one line "write cache size".

>> read raw = yes
>> write raw = yes
> Remove the above - only needed for SMB1.
Do you indeed mean only one line here, i.e. keep "read raw"!, but remove
"write raw"? Or rather remove both (which I assume)...

>> min receivefile size = 0
>> use sendfile = yes
>> large readwrite = yes
> Remove the above. SMB1 only.
Just the "large readwrite"? Again, I tend to think you might want me to
remove all three, correct?

>> max xmit = 32768
> Remove the above. May have devastating effects.
Done.

>> getwd cache = true
>> map untrusted to domain = yes
>> os level = 1
>> local master = yes
>> unix extensions = yes
>> domain master = no
>> preferred master = no
>> dns proxy = no
>> dos charset = CP850
>> unix charset = utf8
>> client ldap sasl wrapping = seal
>> allow trusted domains = yes
>> idmap uid = 20000-60000000
>> idmap gid = 20000-60000000
>> winbind separator = +
>> winbind nested groups = yes
>> winbind enum users = yes
>> winbind enum groups = yes
>> create mask = 0644
>> winbind use default domain = yes
>> map acl inherit = yes
>> nt acl support = yes
>> #map system = yes
>> bind interfaces only = yes
>> interfaces = lo,bond*
>> guest account = nobody
>> map to guest = Bad User
>> guest only = yes
>> follow symlinks = no
>> block size = 262144
>> dfree cache time = 5
>> large readwrite = yes
> Remove the above. SMB1 only.
Another occurrence (duplicate) of "large readwrite" - thanks to
Thecus... :-(
Removed.

>
>> getwd cache = yes
>> oplocks = yes
>> kernel oplocks = yes
> Remove the above. If you're not sharing via
> multiple protocols then this will make things
> worse.
All three lines, or both lines dealing with "oplocks", or indeed only
the single line "kernel oplocks"?

>> veto files = /folder.db/.AppleDouble/.AppleDB/.bin/.AppleDesktop/Network Trash Folder/:2eDS_Store/.DS_Store/TheFindByContentFolder/TheVolumeSettingsFolder/Temporary Items/.AppleDBcnid.lock/.VolumeIcon.icns/.Temporary Items/.Parent/.HSicon/._*/:*/
>> veto oplock files = /J0*.WMF/*_.GIF/J0*.JPG/*_.WMF/
> Remove the two lines above. Why are they needed ?
Please don't ask me - this is Thecus' voodoo for supporting Macs? (I
have a Windows/Linux/Solaris-Illumos only environment.)

Thanks a million one more time
Andreas


--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Third Try: Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf

Samba - General mailing list
On Fri, Jul 14, 2017 at 08:24:36PM +0200, awl1 wrote:

> Sorry, Jeremy,
>
> looks like we need to reiterate through those changes, as I am
> constantly wondering whether just the one single line above, or all
> lines:
>
> Am 14.07.2017 um 19:41 schrieb Jeremy Allison via samba:
> >On Fri, Jul 14, 2017 at 04:19:09PM +0200, awl1 wrote:
> >>[global]
> >>server string = %h
> >>max open files = 100000
> >>deadtime = 15
> >>dead time = 15
> >>hide unreadable = yes
> >Remove the above. Causes security descriptor lookup for every directory entry.
> Remove just the one line "hide unreadable", correect?

Yes.

> >>load printers = no
> >>log file = /var/log/samba.%m
> >>max log size = 50
> >>strict locking = no
> >>lock directory = /var/samba
> >>encrypt passwords = yes
> >>case sensitive = true
> >>default case = lower
> >>preserve case = yes
> >>short preserve case = yes
> >>passdb backend = tdbsam
> >>socket options = TCP_NODELAY IPTOS_LOWDELAY SO_KEEPALIVE
> >Remove the above - voodoo bullsh*t not needed on modern kernels.
> Remove just the one line "socket options", correct?

Yes.

> >>aio read size = 1
> >>aio write size = 1
> >>write cache size = 2097152
> >REMOVE THE ABOVE WITH PREJUDICE !!!!!
> As you already stated, removed just the one line "write cache size".

Yes.

> >>read raw = yes
> >>write raw = yes
> >Remove the above - only needed for SMB1.
> Do you indeed mean only one line here, i.e. keep "read raw"!, but
> remove "write raw"? Or rather remove both (which I assume)...

Both read raw and write raw.

> >>min receivefile size = 0
> >>use sendfile = yes
> >>large readwrite = yes
> >Remove the above. SMB1 only.
> Just the "large readwrite"?

Yes.

> Again, I tend to think you might want me
> to remove all three, correct?

No.

> >>max xmit = 32768
> >Remove the above. May have devastating effects.
> Done.
> >>follow symlinks = no
> >>block size = 262144
> >>dfree cache time = 5
> >>large readwrite = yes
> >Remove the above. SMB1 only.
> Another occurrence (duplicate) of "large readwrite" - thanks to
> Thecus... :-(
> Removed.
>
> >
> >>getwd cache = yes
> >>oplocks = yes
> >>kernel oplocks = yes
> >Remove the above. If you're not sharing via
> >multiple protocols then this will make things
> >worse.
> All three lines, or both lines dealing with "oplocks", or indeed
> only the single line "kernel oplocks"?

Oh yes - change the 'oplocks/kernel oplocks' lines to read:

        kernel oplocks = no
        kernel change notify = no
        smb2 leases = yes


> >>veto files = /folder.db/.AppleDouble/.AppleDB/.bin/.AppleDesktop/Network Trash Folder/:2eDS_Store/.DS_Store/TheFindByContentFolder/TheVolumeSettingsFolder/Temporary Items/.AppleDBcnid.lock/.VolumeIcon.icns/.Temporary Items/.Parent/.HSicon/._*/:*/
> >>veto oplock files = /J0*.WMF/*_.GIF/J0*.JPG/*_.WMF/
> >Remove the two lines above. Why are they needed ?
> Please don't ask me - this is Thecus' voodoo for supporting Macs? (I
> have a Windows/Linux/Solaris-Illumos only environment.)

Yes, they are mac voodoo.

--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Third Try: Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf

Samba - General mailing list
In reply to this post by Samba - General mailing list
On Fri, 14 Jul 2017 20:24:36 +0200
awl1 via samba <[hidden email]> wrote:

> Sorry, Jeremy,
>
> looks like we need to reiterate through those changes, as I am
> constantly wondering whether just the one single line above, or all
> lines:
>
> Am 14.07.2017 um 19:41 schrieb Jeremy Allison via samba:
> > On Fri, Jul 14, 2017 at 04:19:09PM +0200, awl1 wrote:
> >> [global]
> >> server string = %h
> >> max open files = 100000
> >> deadtime = 15
> >> dead time = 15
> >> hide unreadable = yes
> > Remove the above. Causes security descriptor lookup for every
> > directory entry.
> Remove just the one line "hide unreadable", correect?
> >> load printers = no
> >> log file = /var/log/samba.%m
> >> max log size = 50
> >> strict locking = no
> >> lock directory = /var/samba
> >> encrypt passwords = yes
> >> case sensitive = true
> >> default case = lower
> >> preserve case = yes
> >> short preserve case = yes
> >> passdb backend = tdbsam
> >> socket options = TCP_NODELAY IPTOS_LOWDELAY SO_KEEPALIVE
> > Remove the above - voodoo bullsh*t not needed on modern kernels.
> Remove just the one line "socket options", correct?
>
> >> aio read size = 1
> >> aio write size = 1
> >> write cache size = 2097152
> > REMOVE THE ABOVE WITH PREJUDICE !!!!!
> As you already stated, removed just the one line "write cache size".
>
> >> read raw = yes
> >> write raw = yes
> > Remove the above - only needed for SMB1.
> Do you indeed mean only one line here, i.e. keep "read raw"!, but
> remove "write raw"? Or rather remove both (which I assume)...
>
> >> min receivefile size = 0
> >> use sendfile = yes
> >> large readwrite = yes
> > Remove the above. SMB1 only.
> Just the "large readwrite"? Again, I tend to think you might want me
> to remove all three, correct?
>
> >> max xmit = 32768
> > Remove the above. May have devastating effects.
> Done.
>
> >> getwd cache = true
> >> map untrusted to domain = yes
> >> os level = 1
> >> local master = yes
> >> unix extensions = yes
> >> domain master = no
> >> preferred master = no
> >> dns proxy = no
> >> dos charset = CP850
> >> unix charset = utf8
> >> client ldap sasl wrapping = seal
> >> allow trusted domains = yes
> >> idmap uid = 20000-60000000
> >> idmap gid = 20000-60000000
> >> winbind separator = +
> >> winbind nested groups = yes
> >> winbind enum users = yes
> >> winbind enum groups = yes
> >> create mask = 0644
> >> winbind use default domain = yes
> >> map acl inherit = yes
> >> nt acl support = yes
> >> #map system = yes
> >> bind interfaces only = yes
> >> interfaces = lo,bond*
> >> guest account = nobody
> >> map to guest = Bad User
> >> guest only = yes
> >> follow symlinks = no
> >> block size = 262144
> >> dfree cache time = 5
> >> large readwrite = yes
> > Remove the above. SMB1 only.
> Another occurrence (duplicate) of "large readwrite" - thanks to
> Thecus... :-(
> Removed.
>
> >
> >> getwd cache = yes
> >> oplocks = yes
> >> kernel oplocks = yes
> > Remove the above. If you're not sharing via
> > multiple protocols then this will make things
> > worse.
> All three lines, or both lines dealing with "oplocks", or indeed only
> the single line "kernel oplocks"?
>
> >> veto files
> >> = /folder.db/.AppleDouble/.AppleDB/.bin/.AppleDesktop/Network
> >> Trash
> >> Folder/:2eDS_Store/.DS_Store/TheFindByContentFolder/TheVolumeSettingsFolder/Temporary
> >> Items/.AppleDBcnid.lock/.VolumeIcon.icns/.Temporary
> >> Items/.Parent/.HSicon/._*/:*/ veto oplock files
> >> = /J0*.WMF/*_.GIF/J0*.JPG/*_.WMF/
> > Remove the two lines above. Why are they needed ?
> Please don't ask me - this is Thecus' voodoo for supporting Macs? (I
> have a Windows/Linux/Solaris-Illumos only environment.)
>
> Thanks a million one more time
> Andreas
>
>

Do want to tell him Jeremy, or shall I ?

What you might ask, the answer is, You could remove about 95% of your
smb.conf and still have a working standalone server ;-)

Can I suggest you go and read this:

https://wiki.samba.org/index.php/Setting_up_Samba_as_a_Standalone_Server

Rowland

--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Third Try: Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf

Samba - General mailing list
In reply to this post by Samba - General mailing list



Am 14.07.2017 um 19:58 schrieb Jeremy Allison:
> Yes, but I'm not Thecus support, I'm only helping you :-).
Indeed, and much appreciated! ;-)

> Yes, but I'm guessing that some of the interactions
> here with these smb.conf params have never been tested
> with the move to SMB2.
Exactly. Thecus never provided anything more recent than 3.5.16.

OK, so here is my new global section, and the results from Wireshark:

[global]
server string = %h
max open files = 100000
deadtime = 15
dead time = 15
load printers = no
log file = /opt/samba/var/log/samba.%m
max log size = 50
strict locking = no
lock directory = /var/samba
encrypt passwords = yes
case sensitive = true
default case = lower
preserve case = yes
short preserve case = yes
passdb backend = tdbsam
aio read size = 1
aio write size = 1
write cache size = 2097152
min receivefile size = 0
use sendfile = yes
getwd cache = true
map untrusted to domain = yes
os level = 1
local master = yes
unix extensions = yes
domain master = no
preferred master = no
dns proxy = no
dos charset = CP850
unix charset = utf8
client ldap sasl wrapping = seal
allow trusted domains = yes
idmap uid = 20000-60000000
idmap gid = 20000-60000000
winbind separator = +
winbind nested groups = yes
winbind enum users = yes
winbind enum groups = yes
create mask = 0644
winbind use default domain = yes
map acl inherit = yes
nt acl support = yes
#map system = yes
bind interfaces only = yes
interfaces = lo,bond*
guest account = nobody
map to guest = Bad User
guest only = yes
follow symlinks = no
block size = 262144
dfree cache time = 5
getwd cache = yes
kernel oplocks = no
kernel change notify = no
smb2 leases = yes
workgroup = WORKGROUP
password server = *
security = user
auth methods = guest sam_ignoredomain
realm =
idmap backend = rid:WORKGROUP=20000-60000000
wins server = 192.168.1.1
client ntlmv2 auth = no
server signing = disabled
smb encrypt = disabled
delete veto files = yes


Most recent wireshark capture - result of the above changes to smb.conf:

===========================================================================
SMB2 Service Response Time Statistics - new:
Index  Procedure  Calls  Min SRT (s)  Max SRT (s)  Avg SRT (s)  Sum SRT (s)
---------------------------------------------------------------------------
SMB2
Close            6   1702     0.000571     0.222479     0.005067 8.623646
Create           5   2066     0.001876     0.022982     0.007718 15.946119
Find            14   1600     0.001372     0.265910     0.051195 81.911843
GetInfo         16     17     0.000455     0.010792     0.001690 0.028723
SetInfo         17   2055     0.000732     0.006802     0.001132 2.326217
Write            9   1026     0.000566     0.302602     0.001109 1.137562
---------------------------------------------------------------------------

Great! A huge difference already (this already speeds things up by 230
seconds!) - but still summing up to 109 seconds (as opposed to 25 from
Samba 3.6/SMB1)! :-)

So there still seems some "Find" issue left...!?

Looking forward on how to proceed - now probably in Wireshark, so please
give some more details on how to look for SMB2_QUERY_DIRECTORY in there...

Thanks & best regards
Andreas


--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Third Try: Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf

Samba - General mailing list
In reply to this post by Samba - General mailing list
On Fri, Jul 14, 2017 at 07:43:13PM +0100, Rowland Penny via samba wrote:
>
> Do want to tell him Jeremy, or shall I ?

You just did, thanks ! :-).

> What you might ask, the answer is, You could remove about 95% of your
> smb.conf and still have a working standalone server ;-)
>
> Can I suggest you go and read this:
>
> https://wiki.samba.org/index.php/Setting_up_Samba_as_a_Standalone_Server
>
> Rowland
>
> --
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba

--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Third Try: Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf

Samba - General mailing list
In reply to this post by Samba - General mailing list
Thanks Rowland,

I will immediately also do another run with such a minimal smb.conf as
well (imprerssive!), and report back the result this gives in
Wireshark... ;-)

Note that I simply used the previous 3.5 smb.conf from Thecus in order
not to change/break a running system...

Best
Andreas


Am 14.07.2017 um 20:43 schrieb Rowland Penny via samba:
> Do want to tell him Jeremy, or shall I ?
>
> What you might ask, the answer is, You could remove about 95% of your
> smb.conf and still have a working standalone server;-)
>
> Can I suggest you go and read this:
>
> https://wiki.samba.org/index.php/Setting_up_Samba_as_a_Standalone_Server



--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Third Try: Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf

Samba - General mailing list
Am 14.07.2017 um 20:55 schrieb awl1 via samba:
> I will immediately also do another run with such a minimal smb.conf as
> well (imprerssive!), and report back the result this gives in
> Wireshark... ;-)
[global]
log file = /opt/samba/var/log/samba.%m
max log size = 50
log level = 1
case sensitive = true
default case = lower
preserve case = yes
short preserve case = yes
guest account = nobody
map to guest = Bad User
workgroup = WORKGROUP
netbios name = N4200PRO

I get the following Wireshark statistics for the "Write" 1000 files
scenario with the above minimal config:

===========================================================================
SMB2 Service Response Time Statistics - new3:
Index  Procedure  Calls  Min SRT (s)  Max SRT (s)  Avg SRT (s)  Sum SRT (s)
---------------------------------------------------------------------------
SMB2
Close                    6   1925     0.000625     0.185844 0.005160
9.932725
Create                   5   2161     0.001894     0.024120 0.007871
17.008398
Find                    14   1646     0.001454     0.267518 0.050406
82.968297
GetInfo                 16     16     0.001134     0.011041 0.002474
0.039577
Ioctl                   11      1     0.000528     0.000528 0.000528
0.000528
SetInfo                 17   2151     0.000765     0.006874 0.001174
2.526085
Tree Connect             3      1     0.001610     0.001610 0.001610
0.001610
Tree Disconnect          4      1     0.005401     0.005401 0.005401
0.005401
Write                    9   1074     0.000570     0.092214 0.000931
0.999563
---------------------------------------------------------------------------

which sums up to roundabout 122 secs and is surprisingly similar to the
109 seconds from the edited/"optimised" lengthy config (about 10%
difference from the tuning is not too bad!)...

But I hope we still can get closer to the 25 seconds I used to see with
the old 3.5...

Many thanks so far & best regards
Andreas


--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Third Try: Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf

Samba - General mailing list
Am 14.07.2017 um 21:09 schrieb awl1:
> which sums up to roundabout 122 secs and is surprisingly similar to
> the 109 seconds from the edited/"optimised" lengthy config (about 10%
> difference from the tuning is not too bad!)...
Ouch, sorry, bad mistake: shouldn't have done the calculation in my
head, but use Excel/a calculator machine...

Sums up to only 112 seconds of course and then is truly impressive - I
admit I wouldn't notice the remaining 3 secs. difference to the "tuned"
version...

But we're still on our way down to hopefully the 25 or 30... ;-)

Sorry,
Andreas

--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Third Try: Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf

Samba - General mailing list
In reply to this post by Samba - General mailing list
On Fri, Jul 14, 2017 at 08:51:48PM +0200, awl1 wrote:
>
> OK, so here is my new global section, and the results from Wireshark:
>
> [global]
> write cache size = 2097152

Please remove the 'write cache size'.
> follow symlinks = no

Remove the 'follow symlinks' line.

--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Third Try: Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf

Samba - General mailing list
Both done, but does not result in any significant changes:

===========================================================================
SMB2 Service Response Time Statistics - new4:
Index  Procedure  Calls  Min SRT (s)  Max SRT (s)  Avg SRT (s) Sum SRT (s)
---------------------------------------------------------------------------
SMB2
Close                    6   1634     0.000638     0.225166 0.005001    
8.171705
Create                   5   2053     0.001845     0.025199 0.007823    
16.060968
Find                    14   1588     0.001458     0.275006 0.052488    
83.351477
GetInfo                 16     16     0.001148     0.010863 0.002318    
0.037086
Ioctl                   11      1     0.000542     0.000542 0.000542    
0.000542
SetInfo                 17   2043     0.000755     0.006908 0.001170    
2.390243
Tree Connect             3      1     0.001602     0.001602 0.001602    
0.001602
Tree Disconnect          4      1     0.056703     0.056703 0.056703    
0.056703
Write                    9   1020     0.000711     0.349776 0.001745    
1.780156
---------------------------------------------------------------------------

Sum is again roundabout 111 seconds.

Best,
Andreas


Am 14.07.2017 um 21:49 schrieb Jeremy Allison via samba:
> On Fri, Jul 14, 2017 at 08:51:48PM +0200, awl1 wrote:
>> OK, so here is my new global section, and the results from Wireshark:
>>
>> [global]
>> write cache size = 2097152
> Please remove the 'write cache size'.
>> follow symlinks = no
> Remove the 'follow symlinks' line.
>


--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Friendly Reminder: Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf

Samba - General mailing list
In reply to this post by Samba - General mailing list
Hello again, Jeremy,

OK, trying to move a little more forward with regards to my knowledge of
Wireshark and packet inspection/filtering ;-):

(1)

Using the most recent smb.conf with Samba 4.6.5 / SMB2, for 1020 files
written, we have:

Find                    14   1588     0.001458     0.275006 0.052488    
83.351477

There are 2140 packets with smb2.cmd == 14 (Find) of which are 1070 find
requests ((smb2.cmd == 14) && (smb2.flags.response == 0)) and 1070 find
responses ((smb2.cmd == 14) && (smb2.flags.response == 1))
(NOTE: why is there a difference to the number of 1588 Find calls as
shown in the service response time statistics???)

518 of these 1070 Find Requests do have the following type:
Find Request File: RFP_files SMB2_FIND_ID_BOTH_DIRECTORY_INFO Pattern:
*;Find Request File: RFP_files SMB2_FIND_ID_BOTH_DIRECTORY_INFO Pattern: *
Info Level: SMB2_FIND_ID_BOTH_DIRECTORY_INFO (37)

and 552 are of type (note the value for "pattern" of course does change
with each call):

Find Request File: RFP_files SMB2_FIND_NAME_INFO Pattern: RFP2372.htm
Info Level: SMB2_FIND_NAME_INFO (12)

(2)
With the original smb.conf in the original Samba 3.5.16 write scenario,
for 1024 files written, we have:

Transaction2 Sub-Commands
FIND_FIRST2                          1   2042     0.001599 0.017518    
0.003191     6.515678

There are 4084 packets with smb.trans2.cmd == 0x0001 (FIND_FIRST2) of
which are 2042 find requests ((smb.trans2.cmd == 0x0001) &&
(smb.flags.response == 0)) and 2042 find responses ((smb2.cmd == 14) &&
(smb.flags.response == 1))
(NOTE: Here, we see no difference to the 2042 Find calls as shown in the
service response time statistics...!?)

2039 of the 2042 FIND_FIRST2 requests are of type (value for "pattern"
changes with each call)
Trans2 Request, FIND_FIRST2, Pattern: \napp\Header_RFP_files\RFP.bmp
Level of Interest: Find File Names Info (259)

only 3 of the 2042 FIND_FIRST2 requests are of type
Trans2 Request, FIND_FIRST2, Pattern: \*
Level of Interest: Find File Both Directory Info (260)


(3) Base line:

It looks like the SMB2 logic does execute far too many
FIND_ID_BOTH_DIRECTORY_INFO calls with "*" search pattern - it should
rather execute more FIND_NAME_INFO calls (probably at least one per the
1020 files written).
Do you agree with this assessment? Does this account for the difference
in performance?

Many thanks one more time & best regards
Andreas


Am 14.07.2017 um 19:49 schrieb Jeremy Allison via samba:

> On Fri, Jul 14, 2017 at 07:44:38PM +0200, awl1 wrote:
>> Hello Jeremy,
>>
>> many thanks for getting back to me! :-)
>>
>> Am 14.07.2017 um 19:33 schrieb Jeremy Allison:
>>> It would be quicker for you to help I'm afraid. As you have nicely
>>> identified the SMB2_QUERY_DIRECTORY as cause of the regression,
>>> can you look into the wireshark traces and tell me what info level
>>> the SMB1 client is asking for and what info level the SMB2 client
>>> is asking for ? If the SMB2 client is also asking for security
>>> descriptors, this may be part of it.
>> I will try to do my best in helping you, but I will need more
>> information, as I am not yet clear what exactly you want me to look
>> for in Wireshark:
>>
>> How did you derive SMB2_QUERY_DIRECTORY from the information I
>> listed? For me, the main "Write" bottleneck pointed to SMB2 "Find",
>> how do I get to SMB2_QUERY_DIRECTORY from there?
>>
>> Index               Procedure  Calls  Min SRT (s)  Max SRT (s)  Avg
>> SRT (s)  Sum SRT (s)
>> Find                       14   1607     0.001383     0.746684
>> 0.193413   310.814294
>>
>> So in order to check I should open both traces and compare exactly
>> what information? It would be great if you can describe what I have
>> to do in the Wireshark application in as much detail as possible...
> First try fixing the smb.conf in the way I reviewed. Then
> let's look.
>


--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Friendly Reminder: Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf

Samba - General mailing list
... and something else is even more strange here:

Even though the parameter "pattern" to SMB2_FIND_NAME_INFO seems to
iterate through definitely existing file names, each and every one of
the 552 "SMB2_FIND_NAME_INFO" requests has the following response:

Find Response, Error: STATUS_NO_SUCH_FILE

which on the other hand, when looking at the packet details, maps to
"ErrorData: 00" which might mean that this is not a Samba error "no such
file", but maybe rather a Wireshark error in translating Error Data to
plain text!?):

SMB2 (Server Message Block Protocol version 2)
     SMB2 Header
     Find Response (0x0e)
         [Info Level: SMB2_FIND_NAME_INFO (12)]
         StructureSize: 0x0009
         Reserved: 0x0000
         Byte Count: 0
         Error Data: 00

Best,
Andreas


Am 14.07.2017 um 22:47 schrieb awl1:

> (1)
>
> Using the most recent smb.conf with Samba 4.6.5 / SMB2, for 1020 files
> written, we have:
>
> Find                    14   1588     0.001458     0.275006
> 0.052488    83.351477
>
> There are 2140 packets with smb2.cmd == 14 (Find) of which are 1070
> find requests ((smb2.cmd == 14) && (smb2.flags.response == 0)) and
> 1070 find responses ((smb2.cmd == 14) && (smb2.flags.response == 1))
> (NOTE: why is there a difference to the number of 1588 Find calls as
> shown in the service response time statistics???)
>
> 518 of these 1070 Find Requests do have the following type:
> Find Request File: RFP_files SMB2_FIND_ID_BOTH_DIRECTORY_INFO Pattern:
> *;Find Request File: RFP_files SMB2_FIND_ID_BOTH_DIRECTORY_INFO
> Pattern: *
> Info Level: SMB2_FIND_ID_BOTH_DIRECTORY_INFO (37)
>
> and 552 are of type (note the value for "pattern" of course does
> change with each call):
>
> Find Request File: RFP_files SMB2_FIND_NAME_INFO Pattern: RFP2372.htm
> Info Level: SMB2_FIND_NAME_INFO (12)



--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Friendly Reminder: Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf

Samba - General mailing list
On Fri, Jul 14, 2017 at 11:02:40PM +0200, awl1 wrote:

> ... and something else is even more strange here:
>
> Even though the parameter "pattern" to SMB2_FIND_NAME_INFO seems to
> iterate through definitely existing file names, each and every one
> of the 552 "SMB2_FIND_NAME_INFO" requests has the following
> response:
>
> Find Response, Error: STATUS_NO_SUCH_FILE
>
> which on the other hand, when looking at the packet details, maps to
> "ErrorData: 00" which might mean that this is not a Samba error "no
> such file", but maybe rather a Wireshark error in translating Error
> Data to plain text!?):

Hmmm. This looks to me it might be the root of your problems.

Look into it some more (I'm a bit busy with other stuff
right now).

--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Friendly Reminder: Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf

Samba - General mailing list
Hello Jeremy,

Am 14.07.2017 um 23:33 schrieb Jeremy Allison via samba:
> Look into it some more (I'm a bit busy with other stuff right now).
no worries, the issue really is not that time critical - I would only
like to see it resolved "sooner or later" in order to be able to update
my NAS to a recent Samba version (e.g. to be safe against SambaCry)
without the performance setback for the huge number of files scenario -
which right now still makes me return to Samba 3.5 when I need the "huge
number of small files" to be written with proper speed...


I have tried to determine some more information in the meantime.

While I think I might have stumbled over some issue that might cause at
least part of the performance regression, I am not completely sure, as I
am missing any information about the design concepts: How should the
process of writing a file to the Samba share take place properly - i.e.
which sequence of commands/message types (Find with Find subtypes,
Create, Write, Close, ...) would be expected vs. which sequence is
indeed seen.

 From what I am able to determine, the command sequence in SMB 1.5 for
the Write scenario with good performance looks like the following:

93    6.460072    192.168.1.121    192.168.1.221    SMB    208 Trans2
Request, FIND_FIRST2, Pattern: \napp\Header_Post_files\arches.png
94    6.466082    192.168.1.221    192.168.1.121    SMB    93 Trans2
Response, FIND_FIRST2, Error: STATUS_NO_SUCH_FILE
109    6.474144    192.168.1.121    192.168.1.221    SMB    212 NT
Create AndX Request, FID: 0xb6ba, Path: \napp\Header_Post_files\arches.png
110    6.482842    192.168.1.221    192.168.1.121    SMB    193 NT
Create AndX Response, FID: 0xb6ba
111    6.482978    192.168.1.121    192.168.1.221    SMB    130 Trans2
Request, QUERY_FILE_INFO, FID: 0xb6ba, Query File Internal Info
112    6.483318    192.168.1.221    192.168.1.121    SMB    126 Trans2
Response, FID: 0xb6ba, QUERY_FILE_INFO
115    6.484004    192.168.1.121    192.168.1.221    SMB    130 Trans2
Request, QUERY_FILE_INFO, FID: 0xb6ba, Query File Basic Info
116    6.484290    192.168.1.221    192.168.1.121    SMB    158 Trans2
Response, FID: 0xb6ba, QUERY_FILE_INFO
117    6.484473    192.168.1.121    192.168.1.221    SMB    142 Trans2
Request, SET_FILE_INFO, FID: 0xb6ba
118    6.486985    192.168.1.221    192.168.1.121    SMB    116 Trans2
Response, FID: 0xb6ba, SET_FILE_INFO
125    6.487335    192.168.1.121    192.168.1.221    SMB    69 Write
AndX Request, FID: 0xb6ba, 8707 bytes at offset 0
129    6.489836    192.168.1.221    192.168.1.121    SMB    105 Write
AndX Response, FID: 0xb6ba, 8707 bytes
129    6.489836    192.168.1.221    192.168.1.121    SMB    105 Write
AndX Response, FID: 0xb6ba, 8707 bytes
130    6.489999    192.168.1.121    192.168.1.221    SMB    174 Trans2
Request, SET_FILE_INFO, FID: 0xb6ba
131    6.491616    192.168.1.221    192.168.1.121    SMB    116 Trans2
Response, FID: 0xb6ba, SET_FILE_INFO
and about ten seconds later
13356    17.184267    192.168.1.121    192.168.1.221    SMB    99 Close
Request, FID: 0xb6ba
13359    17.188793    192.168.1.221    192.168.1.121    SMB    93 Close
Response, FID: 0xb6ba

So it seems to be "normal" that Samba executes a Find First Request
returning STATUS_NO_SUCH_FILE every time before actually
creating/writing to the file even in the "good" case.

Now when looking at the same file in the SMB2 3.1.1 Write scenario, we have

37    0.113403    192.168.1.121    192.168.1.221    SMB2    260 Find
Request File: napp\Header_Post_files SMB2_FIND_ID_BOTH_DIRECTORY_INFO
Pattern: *;Find Request File: napp\Header_Post_files
SMB2_FIND_ID_BOTH_DIRECTORY_INFO Pattern: *
38    0.115086    192.168.1.221    192.168.1.121    SMB2    434 Find
Response;Find Response, Error: STATUS_NO_MORE_FILES
49    0.125778    192.168.1.121    192.168.1.221    SMB2    486 Create
Request File: nappPS RFP_files\Header_Post_files\arches.png
50    0.139180    192.168.1.221    192.168.1.121    SMB2    374 Create
Response File: nappPS RFP_files\Header_Post_files\arches.png
51    0.139685    192.168.1.121    192.168.1.221    SMB2    275 GetInfo
Request FS_INFO/FileFsVolumeInformation File: nappPS
RFP_files\Header_Post_files\arches.png;GetInfo Request
FS_INFO/FileFsAttributeInformation File: nappPS
RFP_files\Header_Post_files\arches.png
52    0.141013    192.168.1.221    192.168.1.121    SMB2    258 GetInfo
Response;GetInfo Response
53    0.141292    192.168.1.121    192.168.1.221    SMB2    162 SetInfo
Request FILE_INFO/SMB2_FILE_ENDOFFILE_INFO File: nappPS
RFP_files\Header_Post_files\arches.png
54    0.142555    192.168.1.221    192.168.1.121    SMB2    124 SetInfo
Response
61    0.142938    192.168.1.121    192.168.1.221    SMB2    117 Write
Request Len:8707 Off:0 File: nappPS RFP_files\Header_Post_files\arches.png
65    0.144311    192.168.1.221    192.168.1.121    SMB2    138 Write
Response
66    0.144500    192.168.1.121    192.168.1.221    SMB2    194 SetInfo
Request FILE_INFO/SMB2_FILE_BASIC_INFO File: nappPS
RFP_files\Header_Post_files\arches.png
67    0.145521    192.168.1.221    192.168.1.121    SMB2    124 SetInfo
Response
and about ten seconds later
5653    10.808959    192.168.1.121    192.168.1.221    SMB2 146    Close
Request File: nappPS RFP_files\Header_Post_files\arches.png
5676    10.833568    192.168.1.221    192.168.1.121    SMB2 182    Close
Response

In this case, the Find operation does not look for the particular file
name that is about to be written, but seems to try and list the whole
current directory's content with a pattern of "*". Note that, looping
through the 2000 files to be written, the length of the Find Response
above increases (significantly!) with every file successfully copied:
When copying file number 1000, the Find Response sends back a list of
all 999 files that have been successfully copied to this directory
before, and this list of 999 file names is not needed for any meaningful
purpose, as the goal seems to be to check whether file number 1000
already exists in this list of 999 files (which it of course never
does!) or not.

The last call to SMB2_FIND_ID_BOTH_DIRECTORY_INFO that is contained in
the traces has a response length of about 64kB (containing filenames
that have already been written to the target directory but are not
needed/helpful in any way) and interestingly does not return
"STATUS_NO_MORE_FILES", but "STATUS_INFO_LENGTH_MISMATCH", maybe because
the buffer size for the result of the pattern lookup is only 64kB!?

Base line: I assume that this sequence must be optimized in terms of
always only calling the find operation subtype, like in SMB1 above, that
must not use a wildcard "*", but looks for the the exact name of the
file that is about to be written.

So far, so good - but there also is - for unknown reasons - also a more
lengthy scenario for some (552 of the 1020) files. I have not been able
to determine why for these files, there also is an additional call to
SMB2_FIND_NAME_INFO with the expected return code STATUS_NO_SUCH_FILE as
we have seen in the SMB 1.5 scenario above. Anyways, before this call,
we here also still have the unnecessary call to
SMB2_FIND_ID_BOTH_DIRECTORY_INFO with Pattern "*" as above.

The following sample is for file "napp1408.htm" some 10 seconds later -
which is the first file to show this new pattern. Before that time,
there is no single call to SMB2_FIND_NAME_INFO (why does this new
behavior only start at/after 10 seconds?):

5703    10.858710    192.168.1.121    192.168.1.221    SMB2 260    Find
Request File: nappPS RFP_files SMB2_FIND_ID_BOTH_DIRECTORY_INFO Pattern:
*;Find Request File: nappPS RFP_files SMB2_FIND_ID_BOTH_DIRECTORY_INFO
Pattern: *
5725    10.917195    192.168.1.221    192.168.1.121    SMB2 1034    Find
Response;Find Response, Error: STATUS_NO_MORE_FILES [!!! This response
grows in size with every iteration!!!]
5727    10.917420    192.168.1.121    192.168.1.221    SMB2 178    Find
Request File: nappPS RFP_files SMB2_FIND_NAME_INFO Pattern: napp1408.htm
[!!! this seems to be more or less equivalent than what is done in SMB
1.5 !!!]
5730    10.921782    192.168.1.221    192.168.1.121    SMB2 131    Find
Response, Error: STATUS_NO_SUCH_FILE
5735    10.923888    192.168.1.121    192.168.1.221    SMB2 422    
Create Request File: nappPS RFP_files\napp1408.htm
5738    10.934808    192.168.1.221    192.168.1.121    SMB2 374    
Create Response File: nappPS RFP_files\napp1408.htm
5739    10.935151    192.168.1.121    192.168.1.221    SMB2 162    
SetInfo Request FILE_INFO/SMB2_FILE_ENDOFFILE_INFO File: nappPS
RFP_files\napp1408.htm
5742    10.937324    192.168.1.221    192.168.1.121    SMB2 124    
SetInfo Response
5744    10.937512    192.168.1.121    192.168.1.221    SMB2 1475    
Write Request Len:2765 Off:0 File: nappPS RFP_files\napp1408.htm
5750    10.940218    192.168.1.221    192.168.1.121    SMB2 138    Write
Response
5751    10.940343    192.168.1.121    192.168.1.221    SMB2 194    
SetInfo Request FILE_INFO/SMB2_FILE_BASIC_INFO File: nappPS
RFP_files\napp1408.htm
5754    10.942392    192.168.1.221    192.168.1.121    SMB2 124    
SetInfo Response
10873    22.132423    192.168.1.121    192.168.1.221    SMB2 146    
Close Request File: nappPS RFP_files\napp1408.htm
10876    22.137120    192.168.1.221    192.168.1.121    SMB2 182    
Close Response


So what do you think? Looking forward to your assessment... ;-)

Thanks again & best regards
Andreas


--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Friendly Reminder: Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf

Samba - General mailing list
Hmm, in addition I just started wondering about the following:

As it obviously is my Windows client that creates and sends the SMB2
requests, maybe it is rather Microsoft to blame for choosing to sending
the "wrong"/suboptimal/poorly performing client-side SMB2 find request
(namely SMB2_FIND_ID_BOTH_DIRECTORY_INFO with pattern "*" instead of
SMB2_FIND_NAME_INFO with the particular file name about to be written?
Can Samba influence the Windows client earlier on in the message
exchange to send find requests differently?

And in case that is true that this is a shortfall of Microsoft's SMB2
client as opposed to their SMB1 client, then how could that be fixed?
Are there any secret registry entries to influence this, or would I need
to raise an issue with Microsoft and document this?

Really interesting... ;-):-(

I will now move forward and try to execute the same client scenario from
a Linux environment (will boot the exact same client-side PC hardware
into Ubuntu 16.04), do another tshark recording from there and report
back (probably tomorrow or on Monday) what this might tell us...

Best regards,
Andreas


Am 15.07.2017 um 19:13 schrieb awl1 via samba:

> In this case, the Find operation does not look for the particular file
> name that is about to be written, but seems to try and list the whole
> current directory's content with a pattern of "*". Note that, looping
> through the 2000 files to be written, the length of the Find Response
> above increases (significantly!) with every file successfully copied:
> When copying file number 1000, the Find Response sends back a list of
> all 999 files that have been successfully copied to this directory
> before, and this list of 999 file names is not needed for any
> meaningful purpose, as the goal seems to be to check whether file
> number 1000 already exists in this list of 999 files (which it of
> course never does!) or not.
>
> The last call to SMB2_FIND_ID_BOTH_DIRECTORY_INFO that is contained in
> the traces has a response length of about 64kB (containing filenames
> that have already been written to the target directory but are not
> needed/helpful in any way) and interestingly does not return
> "STATUS_NO_MORE_FILES", but "STATUS_INFO_LENGTH_MISMATCH", maybe
> because the buffer size for the result of the pattern lookup is only
> 64kB!?



--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Friendly Reminder: Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf

Samba - General mailing list
Hello Jeremy,

ouch - indeed, when using a Linux client (mount.cifs --vers=3.0),
client-side behaviour in terms of find requests is completely different:

Exact same setup, Linux Ubuntu 16.04 from exact same PC hardware that
was previously running Win10, copying the same 1031 files onto the NAS
results in the following:

===========================================================================
SMB2 Service Response Time Statistics - new_linux:
Index  Procedure  Calls  Min SRT (s)  Max SRT (s)  Avg SRT (s) Sum SRT (s)
---------------------------------------------------------------------------
SMB2
Close            6   4180     0.000496     0.004940 0.001097     4.586340
Create           5   5272     0.001412     0.023465 0.004757    25.081164
Find            14      4     0.000524     0.008630 0.004097     0.016389
GetInfo         16   1084     0.000525     0.009067 0.000664     0.719405
SetInfo         17   4122     0.000436     0.006486 0.001091     4.496511
Write            9   1031     0.000813     0.064120 0.001197     1.234281
---------------------------------------------------------------------------

which means a Grand Total Sum of SRT of roundabout 36 seconds (as
opposed to roundabout 112 seconds in the latest attempt from Windows!!!).

Note that the Linux mount.cifs vers=3.0 client does only four (!!!) Find
requests for the whole scenario... So why does Windows do 2140 find
requests (1036 times "SMB2_FIND_ID_BOTH_DIRECTORY_INFO, Pattern: *" plus
1104 times SMB2_FIND_NAME_INFO Pattern: <file name>).

While even the Linux SMB2 client is still slower than the Windows SMB1
client, I tend to think that the remaining difference from 25 seconds
with SMB 1.5 in Win10 to 36 seconds with SMB2 3.0 in Linux (44%) might
be tolerable - what do you think? ;-)


I have then also compared the Win10 Total Commander client to a Win10
Explorer client and a Win10 command line prompt executing "xcopy /s",
and the results for Win10 Explorer are again interesting:

For the default Windows explorer:

===========================================================================
SMB2 Service Response Time Statistics - explorer:
Index  Procedure  Calls  Min SRT (s)  Max SRT (s)  Avg SRT (s) Sum SRT (s)
---------------------------------------------------------------------------
SMB2
Close                 6   2966     0.000364     0.214817 0.002725
8.082934
Create                5   3118     0.000853     0.021998 0.005782
18.026942
Find                 14   1607     0.001326     0.255691 0.052544
84.438903
GetInfo              16   1053     0.000285     0.009104 0.000563
0.593040
Ioctl                11      6     0.000308     0.004001 0.002039
0.012235
Read                  8      1     0.000594     0.000594 0.000594
0.000594
SetInfo              17   2063     0.000757     0.006111 0.001136
2.344231
Tree Connect          3      1     0.001563     0.001563 0.001563
0.001563
Write                 9   1034     0.000303     0.305642 0.001412
1.460132
---------------------------------------------------------------------------

Grand Total Sum of SRT here: roundabout 115 seconds - pretty much the
same as from Total Commander, while the Win10 Command Prompt "xcopy /s"
is by far worst:

===========================================================================
SMB2 Service Response Time Statistics - xcopy:
Index  Procedure  Calls  Min SRT (s)  Max SRT (s)  Avg SRT (s) Sum SRT (s)
---------------------------------------------------------------------------
SMB2
Close            6   8047     0.000433     0.201613 0.001869    15.040585
Create           5   8268     0.001659     0.251251     0.015277 126.313894
Find            14   3741     0.002241     0.251251     0.047638 178.213167
GetInfo         16     31     0.000461     0.002741 0.001239     0.038413
SetInfo         17   4127     0.000755     0.016472 0.003262    13.463437
Write            9   1038     0.000713     0.083296 0.001479     1.535621
---------------------------------------------------------------------------

Grand Total Sum of SRT for xcopy: roundabout 335 seconds - note that
xcopy does no less than 3741 find requests in order to copy 1038
files... Unbelieveable... :-(:-(:-(

Looking forward to your comments, especially on how to proceed as most
probably it is indeed Microsoft to blame here: Does the Samba team have
contacts inside Microsoft to point them to their SMB2 performance
regressions!? ;-)

Thanks a million one more time & best regards
Andreas


Am 15.07.2017 um 20:06 schrieb awl1 via samba:

> Hmm, in addition I just started wondering about the following:
>
> As it obviously is my Windows client that creates and sends the SMB2
> requests, maybe it is rather Microsoft to blame for choosing to
> sending the "wrong"/suboptimal/poorly performing client-side SMB2 find
> request (namely SMB2_FIND_ID_BOTH_DIRECTORY_INFO with pattern "*"
> instead of SMB2_FIND_NAME_INFO with the particular file name about to
> be written? Can Samba influence the Windows client earlier on in the
> message exchange to send find requests differently?
>
> And in case that is true that this is a shortfall of Microsoft's SMB2
> client as opposed to their SMB1 client, then how could that be fixed?
> Are there any secret registry entries to influence this, or would I
> need to raise an issue with Microsoft and document this?
>
> Really interesting... ;-):-(
>
> I will now move forward and try to execute the same client scenario
> from a Linux environment (will boot the exact same client-side PC
> hardware into Ubuntu 16.04), do another tshark recording from there
> and report back (probably tomorrow or on Monday) what this might tell
> us...
>
> Best regards,
> Andreas
>
>
> Am 15.07.2017 um 19:13 schrieb awl1 via samba:
>> In this case, the Find operation does not look for the particular
>> file name that is about to be written, but seems to try and list the
>> whole current directory's content with a pattern of "*". Note that,
>> looping through the 2000 files to be written, the length of the Find
>> Response above increases (significantly!) with every file
>> successfully copied: When copying file number 1000, the Find Response
>> sends back a list of all 999 files that have been successfully copied
>> to this directory before, and this list of 999 file names is not
>> needed for any meaningful purpose, as the goal seems to be to check
>> whether file number 1000 already exists in this list of 999 files
>> (which it of course never does!) or not.
>>
>> The last call to SMB2_FIND_ID_BOTH_DIRECTORY_INFO that is contained
>> in the traces has a response length of about 64kB (containing
>> filenames that have already been written to the target directory but
>> are not needed/helpful in any way) and interestingly does not return
>> "STATUS_NO_MORE_FILES", but "STATUS_INFO_LENGTH_MISMATCH", maybe
>> because the buffer size for the result of the pattern lookup is only
>> 64kB!?



--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Friendly Reminder: Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf

Samba - General mailing list
In reply to this post by Samba - General mailing list
Hello Jeremy,

Am 14.07.2017 um 23:33 schrieb Jeremy Allison:
> Look into it some more (I'm a bit busy with other stuff right now).
when will you be available again to look into my latest findings (see
previous posts) and provide your feedback?

If you can confirm that it is Microsoft to blame, what would you propose
to raise this issue with them? Wouldn't this ideally be done by somebody
from the Samba team rather than me?

Many thanks & best regards
Andreas


--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
12
Loading...