samba 4.7.3 DLZ performance regression

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

samba 4.7.3 DLZ performance regression

Samba - General mailing list
Hello,

I'm seeing a huge performance regression with Samba 4.7.3 DLZ and BIND
9.8.2rc1-RedHat-9.8.2-0.62.rc1.el6_9.4.  If I remove DLZ external DNS
requests are a few ms.  If it is enabled then internal and external DNS
requests are a 1-3 seconds or timeout.  Downgrading BIND had no effect.

I reverted just the DLZ modules back to 4.7.0 and performance is back to
normal:

# cp /usr/src/samba-4.7.0/bin/modules/bind9/dlz_bind9* /usr/local/samba/lib/bind9
# echo 'export LDB_MODULES_PATH=/usr/src/samba-4.7.0/bin/modules/ldb' >> /etc/sysconfig/named
# service named restart

Thanks,
Arthur

This e-mail and any attachments may contain CONFIDENTIAL information, including PROTECTED HEALTH INFORMATION. If you are not the intended recipient, any use or disclosure of this information is STRICTLY PROHIBITED; you are requested to delete this e-mail and any attachments, notify the sender immediately, and notify the Mediture Privacy Officer at [hidden email].


--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|

Re: samba 4.7.3 DLZ performance regression

Samba - General mailing list
Hi,

We've just had another person notice this issue and we're currently
looking into it. If we need someone to test the patch fix, or if we need
more information, we can let you know.

I've filed a bug as well:

https://bugzilla.samba.org/show_bug.cgi?id=13191


Cheers,

Garming


On 14/12/17 09:40, Arthur Ramsey via samba wrote:

> Hello,
>
> I'm seeing a huge performance regression with Samba 4.7.3 DLZ and BIND
> 9.8.2rc1-RedHat-9.8.2-0.62.rc1.el6_9.4.  If I remove DLZ external DNS
> requests are a few ms.  If it is enabled then internal and external
> DNS requests are a 1-3 seconds or timeout.  Downgrading BIND had no
> effect.
>
> I reverted just the DLZ modules back to 4.7.0 and performance is back
> to normal:
>
> # cp /usr/src/samba-4.7.0/bin/modules/bind9/dlz_bind9*
> /usr/local/samba/lib/bind9
> # echo 'export LDB_MODULES_PATH=/usr/src/samba-4.7.0/bin/modules/ldb'
> >> /etc/sysconfig/named
> # service named restart
>
> Thanks,
> Arthur
>
> This e-mail and any attachments may contain CONFIDENTIAL information,
> including PROTECTED HEALTH INFORMATION. If you are not the intended
> recipient, any use or disclosure of this information is STRICTLY
> PROHIBITED; you are requested to delete this e-mail and any
> attachments, notify the sender immediately, and notify the Mediture
> Privacy Officer at [hidden email].
>
>


--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|

Re: samba 4.7.3 DLZ performance regression

Samba - General mailing list
On Fri, 2017-12-15 at 10:16 +1300, Garming Sam via samba wrote:
> Hi,
>
> We've just had another person notice this issue and we're currently
> looking into it. If we need someone to test the patch fix, or if we need
> more information, we can let you know.
>
> I've filed a bug as well:
>
> https://bugzilla.samba.org/show_bug.cgi?id=13191

Attached is a possible set of patches.

As indicated, a search for a not-existing name, be it one that will
match with a wildcard or one that wont, will still be slow.

Andrew Bartlett

--
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team         https://samba.org
Samba Development and Support, Catalyst IT  
https://catalyst.net.nz/services/samba




--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba

dns-wildcard-performance.patch.txt (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: samba 4.7.3 DLZ performance regression

Samba - General mailing list
Thanks Andrew, much better than vanilla 4.7.3 but it is still performs
much worse than vanilla 4.7.0 especially for DLZ zones.

$ dig @127.0.0.1 google.com
$ perl -e "foreach(1..500) { system('dig', '@127.0.0.1', 'google.com'); }" | grep time > 4.7.0_external.log
$ perl -e "foreach(1..500) { system('dig', '@127.0.0.1', 'mediture.dom'); }" | grep time > 4.7.0_internal.log

$ cat 4.7.0_internal.log.gz | perl dns_stats.pl
Max process time (ms): 7
Standard deviation of process time (ms): 0.456807778713746
Mean process time (ms): 2.088

$ cat 4.7.0_external.log.gz | perl dns_stats.pl
Max process time (ms): 3
Standard deviation of process time (ms): 0.332814559570528
Mean process time (ms): 1.884

Then after installing Samba4, restarting samba4 and restarting BIND:

$ dig @127.0.0.1 google.com
$ perl -e "foreach(1..500) { system('dig', '@127.0.0.1', 'google.com'); }" | grep time > 4.7.3_external.log
$ perl -e "foreach(1..500) { system('dig', '@127.0.0.1', 'mediture.dom'); }" | grep time > 4.7.3_internal.log

$ cat 4.7.3_external.log.gz | perl dns_stats.pl
Max process time (ms): 985
Standard deviation of process time (ms): 169.45778909383
Mean process time (ms): 66.36

$ cat 4.7.3_internal.log.gz | perl dns_stats.pl
Max process time (ms): 975
Standard deviation of process time (ms): 238.891289268035
Mean process time (ms): 404.935871743487

On 12/14/2017 05:44 PM, Andrew Bartlett wrote:
>   Attached is a possible set of patches.
>
> As indicated, a search for a not-existing name, be it one that will
> match with a wildcard or one that wont, will still be slow.
>
> Andrew Bartlett
>


This e-mail and any attachments may contain CONFIDENTIAL information, including PROTECTED HEALTH INFORMATION. If you are not the intended recipient, any use or disclosure of this information is STRICTLY PROHIBITED; you are requested to delete this e-mail and any attachments, notify the sender immediately, and notify the Mediture Privacy Officer at [hidden email].
--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|

Re: samba 4.7.3 DLZ performance regression

Samba - General mailing list
On Thu, 2017-12-14 at 18:52 -0600, Arthur Ramsey wrote:
> Thanks Andrew, much better than vanilla 4.7.3 but it is still performs
> much worse than vanilla 4.7.0 especially for DLZ zones.

I understand.  The key issue is that to support the wildcards we
changed the way we use the database, the SCOPE_ONELEVEL index (compared
with SCOPE_BASE used exclusively in the past) over 7000 possible values
is the choke point on the database I'm testing with.

This is much improved in master, but sadly the primary fix turns out to
be the GUID index change (which is too much to backport), rather than
(as I had hoped) the supporting changes.

Are the stats you included on the patched 4.7.3, or the original
release?

Thanks,

Andrew Bartlett
--
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team         https://samba.org
Samba Development and Support, Catalyst IT  
https://catalyst.net.nz/services/samba





--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|

Re: samba 4.7.3 DLZ performance regression

Samba - General mailing list
Patched.  I can repeat test with vanilla if you like but it was much
worse.  The mean was probably ~1750ms, the stdev ~1000ms and the max >
3000ms.

On 12/14/2017 7:52 PM, Andrew Bartlett wrote:
> Are the stats you included on the patched 4.7.3, or the original
> release?

This e-mail and any attachments may contain CONFIDENTIAL information, including PROTECTED HEALTH INFORMATION. If you are not the intended recipient, any use or disclosure of this information is STRICTLY PROHIBITED; you are requested to delete this e-mail and any attachments, notify the sender immediately, and notify the Mediture Privacy Officer at [hidden email].


--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|

Re: samba 4.7.3 DLZ performance regression

Samba - General mailing list
In reply to this post by Samba - General mailing list
Can you provide a reference to the "GUID index change" commit so I can
more easily track when it gets included in a stable release?  Currently
targeted for 4.8.0?

On 12/14/2017 7:52 PM, Andrew Bartlett wrote:
> This is much improved in master, but sadly the primary fix turns out to
> be the GUID index change (which is too much to backport), rather than
> (as I had hoped) the supporting changes.


This e-mail and any attachments may contain CONFIDENTIAL information, including PROTECTED HEALTH INFORMATION. If you are not the intended recipient, any use or disclosure of this information is STRICTLY PROHIBITED; you are requested to delete this e-mail and any attachments, notify the sender immediately, and notify the Mediture Privacy Officer at [hidden email].


--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba
Reply | Threaded
Open this post in threaded view
|

Re: samba 4.7.3 DLZ performance regression

Samba - General mailing list
On Thu, 2017-12-14 at 20:17 -0600, Arthur Ramsey via samba wrote:
> Can you provide a reference to the "GUID index change" commit so I can
> more easily track when it gets included in a stable release?  Currently
> targeted for 4.8.0?

Yes, the GUID index change, which combined with a change to the way we
handle SCOPE_ONELEVEL indexed queries is the best fix for this issue.
These changes are all in master.

However we were able to devise a backportable fix to the index code and
we have proposed that for 4.7.  It changes the index code in a
different way, one that is much more limited and so more appropriate
for the backport.  With this the critical 'not found' response time is
returned to at around 20ms.

I'm currently working to get that approved for release in 4.7.4,
however the patches I proposed have been objected to.  As such, I can't
tell you when this will be resolved or Samba 4.7.4 will be released,
but I hope it is soon.

Feel free to CC yourself to the bug:
https://bugzilla.samba.org/show_bug.cgi?id=13191

I'll continue to make the case for getting this fixed as soon as
possible.

Thanks,

Andrew Bartlett
--
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba


--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba