Problem with AIO random read

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Problem with AIO random read

mikeliu
Hello list,

I setup Iometer to test AIO for 100% random read.
If "Transfer Request Size" is more than or equal to 256 kilobytes,in the beginning the transmission is good.
But 3~5 seconds later,the throughput will drop to zero.

Server OS:
Ubuntu Server 14.04.1 LTS

Samba:
Version 4.1.6-Ubuntu

Dialect:
SMB 2.0

AIO settings :
aio read size = 1
aio write size = 1
vfs objects = aio_pthread

Client OS:
Win 8.1 professional 64Bit

Iometer Setting:
Iometer 2006.07.27
Worker : 8
Max Disk Size (Sectors) : 16777216
Outstanding I/Os : 128
Starting Disk Sector : 0
100% Random read

Any insight into why this is so and any other tuning suggestion is
appreciated.

Thanks.

Regards,
Mike
Reply | Threaded
Open this post in threaded view
|

Re: Problem with AIO random read

Volker Lendecke
On Wed, Dec 03, 2014 at 03:45:24AM -0800, mikeliu wrote:

> Hello list,
>
> I setup Iometer to test AIO for 100% random read.
> If "Transfer Request Size" is more than or equal to 256 kilobytes,in the
> beginning the transmission is good.
> But 3~5 seconds later,the throughput will drop to zero.
>
> Server OS:
> Ubuntu Server 14.04.1 LTS
>
> Samba:
> Version 4.1.6-Ubuntu
>
> Dialect:
> SMB 2.0
>
> AIO settings :
> aio read size = 1
> aio write size = 1
> vfs objects = aio_pthread

Please don't use this module. pthread aio is built in for
4.1. aio_pthread is only used for creating files
asynchronously, read/write is done without any special
module.

Removing that module won't help you probably though. We need
to really take a look at what's happening, this should just
work fine.

When the system is stalled, how many threads does the smbd
have active? /proc/pid/tasks will tell you. Also, can you
attach to smbd with gdb and show a backtrace?

Maybe it does not happen anymore under strace, but could you
strace -f that smbd while its running into the issue?

Ah, btw, you don't happen to have sendfile/recvfile active?
I think those don't play well with aio.

With best regards,

Volker Lendecke

--
SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen
phone: +49-551-370000-0, fax: +49-551-370000-9
AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen
http://www.sernet.de, mailto:[hidden email]
--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|

Re: Problem with AIO random read

mikeliu
Hello Volker,

Thanks for kindly feedback :)

> When the system is stalled, how many threads does the smbd
>have active? /proc/pid/tasks will tell you.

At the beginning ...
pidstat -t|grep smbd
11:41:16 AM     0      5388         -    0.00    0.00    0.00    0.00     1  smbd
11:41:16 AM     0         -      5388    0.00    0.00    0.00    0.00     1  |__smbd
11:41:16 AM     0      5390         -    0.00    0.00    0.00    0.00     1  smbd
11:41:16 AM     0         -      5390    0.00    0.00    0.00    0.00     1  |__smbd
11:41:16 AM 65534      5423         -    0.03    0.26    0.00    0.28     1  smbd
11:41:16 AM 65534         -      5423    0.02    0.20    0.00    0.22     1  |__smbd
11:41:16 AM 65534         -      5593    0.00    0.00    0.00    0.00     1  |__smbd
11:41:16 AM 65534         -      5594    0.00    0.00    0.00    0.01     1  |__smbd
11:41:16 AM 65534         -      5595    0.00    0.00    0.00    0.00     1  |__smbd
11:41:16 AM 65534         -      5599    0.00    0.00    0.00    0.00     1  |__smbd

When the throughput drop to zero ...
pidstat -t|grep smbd
02:37:45 PM     0      5388         -    0.00    0.00    0.00    0.00     1  smbd
02:37:45 PM     0         -      5388    0.00    0.00    0.00    0.00     1  |__smbd
02:37:45 PM     0      5390         -    0.00    0.00    0.00    0.00     0  smbd
02:37:45 PM     0         -      5390    0.00    0.00    0.00    0.00     0  |__smbd
02:37:45 PM     0      5423         -    0.03    0.34    0.00    0.38     0  smbd
02:37:45 PM     0         -      5423    0.03    0.26    0.00    0.29     0  |__smbd


>Maybe it does not happen anymore under strace, but could you
>strace -f that smbd while its running into the issue?

When I executed strace on smbd,the error did not occur.(Timing ?)
For log.smbd,the highest debug level I could reproduce this error is log level 5.


> Ah, btw, you don't happen to have sendfile/recvfile active?
> I think those don't play well with aio.
# testparm -sv 2>/dev/null|grep -E 'receivefile|sendfile'
        min receivefile size = 0
        use sendfile = No
 
This symptom did not occur in Samba-4.0.13,but it occur after upgrading to Samba-4.0.14.
So I use Samba-4.0.13 and patch this one "bug #10298: Reduce smb2_server processing overhead".
This symptom can reproduce again.

Regards,
Mike
Reply | Threaded
Open this post in threaded view
|

Re: Problem with AIO random read

Volker Lendecke
On Wed, Dec 03, 2014 at 11:11:13PM -0800, mikeliu wrote:
> This symptom did not occur in Samba-4.0.13,but it occur after upgrading to
> Samba-4.0.14.
> So I use Samba-4.0.13 and patch this one "bug #10298: Reduce smb2_server
> processing overhead".
> This symptom can reproduce again.

Just for my understanding: The patches from bug 10298 break
aio for you? Without those patches it works, with them it
stalls?

Thanks,

Volker

--
SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen
phone: +49-551-370000-0, fax: +49-551-370000-9
AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen
http://www.sernet.de, mailto:[hidden email]
--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|

Re: Problem with AIO random read

mikeliu
> Without those patches it works, with them it stalls?
Yes.

Regards,
Mike
Reply | Threaded
Open this post in threaded view
|

Re: Problem with AIO random read

Stefan Metzmacher-2
Am 05.12.2014 um 02:30 schrieb mikeliu:
>> Without those patches it works, with them it stalls?
> Yes.

Can you please file a bug under https://bugzilla.samba.org/ ?

Thanks!
metze
--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|

Re: Problem with AIO random read

mikeliu
Hello metze,

Link:
https://bugzilla.samba.org/show_bug.cgi?id=11084

Thanks.

Regards,
Mike