Re: [Samba] Samba4 high cpu load

[steve-2]

On 05/04/12 00:55, Günter Kukkukk wrote:

> On Wednesday 04 April 2012 15:33:46 steve wrote:
>> OpenSUSE 12.1
>> Version 4.0.0alpha19-GIT-7290a62
>>
>> Upon starting, s4 burns the CPU for around 5 minutes:
>>
>>     PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>
>>    3672 root      20   0 72780  20m 2388 R 95.4  1.1   0:36.84 samba
>>
>> After which all is well. Maybe this is just openSUSE as on Ubuntu it's
>> less than 5 minutes (but still there).
>>
>> Any ideas?
>> Cheers,
>> Steve
>
> when i had upgraded to opensuse 12.1 some months ago, i also noticed a
> 3 to 5 minutes hang with high CPU usage when samba4 was starting.
> S4 was started interactively with some debug info as
>     ./sbin/samba -i -M single -d3
> and before the hang a strange message
>     "WARNING! no socket to connect to"
> was displayed.
>
> Running s4 inside gdb showed the hang during startup of the internal ldap service.
> The CPU was "hard working" inside "the GNU Multiple Precision Arithmetic Library"
>    /usr/lib/libgmp.so.10
>
> Further investigation showed that pkcs11 was using the gnome-keyring module
>     /usr/lib/pkcs11/gnome-keyring-pkcs11.so
> This module was also displaying the strange string "WARNING! no socket to connect to"
> (see also /etc/pkcs11/modules/* )
> Btw - i'm running KDE here.
>
> I de-installed gnome-keyring and most pkcs11 related stuff - and the s4 hang
> was gone!  :-)
>
> It already took me a lot of time those days - so i did no further investigations ...
> Possibly it's enough to only de-install gnome-keyring.
>
> It should be noted, that samba4 is still inside its startup sequence when that hang occurs,
> not all modules have been initialized, so s4 is not able to operate properly at all during
> that (3 - 5 minutes) hang state!
>
> Cheers, Günter
>

Hi
Sorry to open this one up again but the problem with startup times on
openSUSe remain.

The startup is now over 5 minutes. I have removed gnome keyring and the
pkcs11 stuff.

The delay is the same interactive or not. can anyone point me in the
right direction to trace this?
Cheers,
Steve

[Andrew Bartlett]

On Mon, 2012-04-30 at 16:04 +0200, steve wrote:

> On 05/04/12 00:55, Günter Kukkukk wrote:
> > On Wednesday 04 April 2012 15:33:46 steve wrote:
> >> OpenSUSE 12.1
> >> Version 4.0.0alpha19-GIT-7290a62
> >>
> >> Upon starting, s4 burns the CPU for around 5 minutes:
> >>
> >>     PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> >>
> >>    3672 root      20   0 72780  20m 2388 R 95.4  1.1   0:36.84 samba
> >>
> >> After which all is well. Maybe this is just openSUSE as on Ubuntu it's
> >> less than 5 minutes (but still there).
> >>
> >> Any ideas?
> >> Cheers,
> >> Steve
> >
> > when i had upgraded to opensuse 12.1 some months ago, i also noticed a
> > 3 to 5 minutes hang with high CPU usage when samba4 was starting.
> > S4 was started interactively with some debug info as
> >     ./sbin/samba -i -M single -d3
> > and before the hang a strange message
> >     "WARNING! no socket to connect to"
> > was displayed.
> >
> > Running s4 inside gdb showed the hang during startup of the internal ldap service.
> > The CPU was "hard working" inside "the GNU Multiple Precision Arithmetic Library"
> >    /usr/lib/libgmp.so.10
> >
> > Further investigation showed that pkcs11 was using the gnome-keyring module
> >     /usr/lib/pkcs11/gnome-keyring-pkcs11.so
> > This module was also displaying the strange string "WARNING! no socket to connect to"
> > (see also /etc/pkcs11/modules/* )
> > Btw - i'm running KDE here.
> >
> > I de-installed gnome-keyring and most pkcs11 related stuff - and the s4 hang
> > was gone!  :-)
> >
> > It already took me a lot of time those days - so i did no further investigations ...
> > Possibly it's enough to only de-install gnome-keyring.
> >
> > It should be noted, that samba4 is still inside its startup sequence when that hang occurs,
> > not all modules have been initialized, so s4 is not able to operate properly at all during
> > that (3 - 5 minutes) hang state!
> >
> > Cheers, Günter
> >
> Hi
> Sorry to open this one up again but the problem with startup times on
> openSUSe remain.
>
> The startup is now over 5 minutes. I have removed gnome keyring and the
> pkcs11 stuff.
>
> The delay is the same interactive or not. can anyone point me in the
> right direction to trace this?

When I last looked at it with Günter, we determined that this was in the
gnutls library, where we are required to generate a DH key pair for
LDAPS support.  The operation is declared as 'expected to take a long
time' in the manpage.

What we need to figure out is:  is the way we call and use gnutls
correct, is this a bug in gnutls, or is there a different best practice
we should follow.

Andrew Bartlett

--
Andrew Bartlett                                http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org

[Günter Kukkukk-2]

On Monday 30 April 2012 16:04:37 steve wrote:

> On 05/04/12 00:55, Günter Kukkukk wrote:
> > On Wednesday 04 April 2012 15:33:46 steve wrote:
> >> OpenSUSE 12.1
> >> Version 4.0.0alpha19-GIT-7290a62
> >>
> >> Upon starting, s4 burns the CPU for around 5 minutes:
> >>     PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> >>    
> >>    3672 root      20   0 72780  20m 2388 R 95.4  1.1   0:36.84 samba
> >>
> >> After which all is well. Maybe this is just openSUSE as on Ubuntu it's
> >> less than 5 minutes (but still there).
> >>
> >> Any ideas?
> >> Cheers,
> >> Steve
> >
> > when i had upgraded to opensuse 12.1 some months ago, i also noticed a
> > 3 to 5 minutes hang with high CPU usage when samba4 was starting.
> > S4 was started interactively with some debug info as
> >
> >     ./sbin/samba -i -M single -d3
> >
> > and before the hang a strange message
> >
> >     "WARNING! no socket to connect to"
> >
> > was displayed.
> >
> > Running s4 inside gdb showed the hang during startup of the internal ldap
> > service. The CPU was "hard working" inside "the GNU Multiple Precision
> > Arithmetic Library"
> >
> >    /usr/lib/libgmp.so.10
> >
> > Further investigation showed that pkcs11 was using the gnome-keyring
> > module
> >
> >     /usr/lib/pkcs11/gnome-keyring-pkcs11.so
> >
> > This module was also displaying the strange string "WARNING! no socket to
> > connect to" (see also /etc/pkcs11/modules/* )
> > Btw - i'm running KDE here.
> >
> > I de-installed gnome-keyring and most pkcs11 related stuff - and the s4
> > hang was gone!  :-)
> >
> > It already took me a lot of time those days - so i did no further
> > investigations ... Possibly it's enough to only de-install
> > gnome-keyring.
> >
> > It should be noted, that samba4 is still inside its startup sequence when
> > that hang occurs, not all modules have been initialized, so s4 is not
> > able to operate properly at all during that (3 - 5 minutes) hang state!
> >
> > Cheers, Günter
>
> Hi
> Sorry to open this one up again but the problem with startup times on
> openSUSe remain.
>
> The startup is now over 5 minutes. I have removed gnome keyring and the
> pkcs11 stuff.
>
> The delay is the same interactive or not. can anyone point me in the
> right direction to trace this?
> Cheers,
> Steve

I have started again to track that down.
Will write a test applet to catch that as simple as possible, to discuss
it with the gnutls devs.

As a workaround you can use
    tls enabled = no
in the [global] section of smb.conf

I'll keep you informed about my findings.

Cheers, Günter

[steve-2]

On 04/05/12 03:23, Günter Kukkukk wrote:
> On Monday 30 April 2012 16:04:37 steve wrote:
>> On 05/04/12 00:55, Günter Kukkukk wrote:
>>> On Wednesday 04 April 2012 15:33:46 steve wrote:
>>>> OpenSUSE 12.1
>>>> Version 4.0.0alpha19-GIT-7290a62

>
> I have started again to track that down.
> Will write a test applet to catch that as simple as possible, to discuss
> it with the gnutls devs.
>
> As a workaround you can use
>      tls enabled = no
> in the [global] section of smb.conf
>
> I'll keep you informed about my findings.
>
> Cheers, Günter

Thanks Günter
The workaround works fine. Please let me know if there is anything I can
test. I've switched to Ubuntu for the moment but have left this S4
install on openSUSE in case I can test anything.
Cheers,
Steve

[Günter Kukkukk-2]

On Friday 04 May 2012 08:53:25 steve wrote:

> On 04/05/12 03:23, Günter Kukkukk wrote:
> > On Monday 30 April 2012 16:04:37 steve wrote:
> >> On 05/04/12 00:55, Günter Kukkukk wrote:
> >>> On Wednesday 04 April 2012 15:33:46 steve wrote:
> >>>> OpenSUSE 12.1
> >>>> Version 4.0.0alpha19-GIT-7290a62
> >
> > I have started again to track that down.
> > Will write a test applet to catch that as simple as possible, to discuss
> > it with the gnutls devs.
> >
> > As a workaround you can use
> >
> >      tls enabled = no
> >
> > in the [global] section of smb.conf
> >
> > I'll keep you informed about my findings.
> >
> > Cheers, Günter
>
> Thanks Günter
> The workaround works fine. Please let me know if there is anything I can
> test. I've switched to Ubuntu for the moment but have left this S4
> install on openSUSE in case I can test anything.
> Cheers,
> Steve

did some further investigations - intermediate results:
The "samba4 hang with high cpu usage" happens during "gnutls_dh_params_generate2"
which calculates the Diffie-Hellman key.

One can check/simulate the same behaviour with:
    certtool --generate-dh-params --bits 1024
or to get a file
    certtool --generate-dh-params --bits 1024 --outfile dh1024.pem

The time it takes to calculate this key depends at least on the used
gnutls version! Using certtool -v
opensuse 11.4   (GnuTLS) 2.8.6       fast
opensuse 12.1   (GnuTLS) 3.0.3       slow
ubuntu 12.04    (GnuTLS) 2.12.14     fast

I'll do further investigations the next days.
To use TLS with samba4 with those slow versions, one can generate
this DH key with certool, as noted above.
One must then add that param file to smb.conf in the [global] section:

   tls dh params file = /path/to/dh1024.pem

I'm atm not quite sure whether this dh param file creation should
be directed to cron to generate a new one - say every week ... (?)

Cheers, Günter

Some further readings:
http://lists.gnu.org/archive/html/help-gnutls/2011-12/msg00008.html
http://lists.gnu.org/archive/html/help-gnutls/2011-12/msg00012.html
Also this bug is fixed in the 3.0.3 version:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=475168
The opensuse 12.1 version only reads 32 bytes (256 bit) from /dev/urandom
One can check this with:
strace -e trace=open,read -s12 certtool --generate-dh-params --bits 1024