Quantcast

[ROADMAP] Catalyst's focus on Samba

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[ROADMAP] Catalyst's focus on Samba

Samba - samba-technical mailing list
G'Day,

Over a week ago, I promised to write a broad overview of the things
that Catalyst is looking at in Samba over the next few months, so as to
avoid surprises, encourage collaboration and to encourage all our users
about what they might see for 4.7 if everything goes to plan.  

Logging
-------

As you have seen we have made a massive effort to get proper
authentication and authorization logging in Samba.   We hope to extend
this to the KDC, but even with the current status covering NTLM, it is
a massive step forward.

Performance
-----------

We are building an AD performance measurement tool.  The idea is that
this tool will replicate real traffic (like a number of the smbtorture
commands in the past) and allow us to measure if Samba's performance
has changed, and how it compares with (eg) Windows.

Once we build that, and alongside it, we plan to address performance
hot spots as we see them.  We have made massive strides in Samba
performance so far, and we plan to continue to address those in the AD
DC that we see.  I hope to see further improvements in our search
performance (see latest ldb index patches for another 2%, on top of 10%
from the libndr work by douglas).  We love flame graphs for this work.

Scale
-----

Tied up in performance work is a desire to have samba scale more, to
serve more user and hold more groups/group members.  The direction of
this work is less certain, but having broken the back of the 'too many
links melts Samba' issue, I'm sure we will be asked to do more here.  

Specific possibilities include LMDB and a GUID-based index scheme.

We also expect to deploy the above tool at representations of large
networks, and that should help us understand better how well Samba
performs when scaled up.

Multi-process LDAP
------------------

We have made our Samba AD netlogon server multi-process, and we have
been asked to make the LDAP server multi-process as well.  We hope to
do that with a prefork system, or address the fork() and exit() costs
enough for the standard model to be practical.

Replication Correctness
-----------------------

The Catalyst Samba team has already been asked to implement the
REPL_GET_TGT flag, which will make our replication code safer and more
correct.  This will build on the REPL_GET_ANC code that landed for 4.6.

RODC
----

A number of RODC patches were posted recently (for msDS-RevealedUsers),
and we hope to get the RODC into a much more tested and deployable
condition soon.  

General Samba improvements
--------------------------

Beyond just these things, it has been great working with so many others
on general improvements to Samba.  It was great to land the Python
patches for more python3, and the ability to disable python recently.
I love reviewing the great work others are doing making Samba better,
as well as digging into it myself!

I'm excited to be part of such a busy team, both at Catalyst and in the
broader Samba community.  It is also really exiting to see where Samba
AD is being deployed.  In Open Source and Free Software we don't hear
about most of our most interesting deployments, but it is really quite
fun to see 'Effectively managing a Samba-based Active Directory domain'
end up in the desired experience line of a job ad:

https://au.indeed.com/cmp/Expressway-Spares/jobs/Desktop-Network-Support-Analyst-acc961a80fd9d5b3

I think this means we really did make it!

Thanks,

Andrew Bartlett
--
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [ROADMAP] Catalyst's focus on Samba

Samba - samba-technical mailing list
Hi Andrew,

> Scale
> -----
>
> Tied up in performance work is a desire to have samba scale more, to
> serve more user and hold more groups/group members.  The direction of
> this work is less certain, but having broken the back of the 'too many
> links melts Samba' issue, I'm sure we will be asked to do more here.  
>
> Specific possibilities include LMDB and a GUID-based index scheme.

If you play with lmdb, please try to understand how OpenLDAP does the
indexing. I think we should also make use of the virtual databases
of lmdb, so that we can keep the index records away from the real objects.
We could also have the different partitions within a single lmdb file.

metze


signature.asc (853 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [ROADMAP] Catalyst's focus on Samba

Samba - samba-technical mailing list
Hi Andrew & Metze,

Am Dienstag, 21. März 2017, 11:49:21 schrieb Stefan Metzmacher via samba-
technical:

> Hi Andrew,
>
> > Scale
> > -----
> >
> > Tied up in performance work is a desire to have samba scale more, to
> > serve more user and hold more groups/group members.  The direction of
> > this work is less certain, but having broken the back of the 'too many
> > links melts Samba' issue, I'm sure we will be asked to do more here.
> >
> > Specific possibilities include LMDB and a GUID-based index scheme.
>
> If you play with lmdb, please try to understand how OpenLDAP does the
> indexing. I think we should also make use of the virtual databases
> of lmdb, so that we can keep the index records away from the real objects.
> We could also have the different partitions within a single lmdb file.
>
> metze
Please also note that increasing the number of named databases in LMDB comes
at a price for startup, see mdb_env_set_maxdbs in the API:

http://www.lmdb.tech/doc/group__mdb.html#gaa2fc2f1f37cb1115e733b62cab2fcdbc

OpenLDAP back-mdb currently limits this to 128 at compile time. You could
probably change that limit at runtime too though, when opening the
environment.

The other limit too look out for are the 511 byte default key size limit. See
dn2id of back-mdb for a way around that. I've isolated and re-implemented that
algorithm here: https://github.com/reqa/ldap-lmdb-dntree

Best regards,
Arvid



--
Arvid Requate
Open Source Software Engineer

CeBIT 2017
20. bis 24. März in Hannover
Treffen Sie uns im Open Source Park (Halle 3/D36)

Univention GmbH
be open.
Mary-Somerville-Str.1
28359 Bremen
Tel. : +49 421 22232-52
Fax : +49 421 22232-99

Geschäftsführer: Peter H. Ganten
HRB 20755 Amtsgericht Bremen
Steuer-Nr.: 71-597-02876

signature.asc (817 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[LMDB] Re: [ROADMAP] Catalyst's focus on Samba

Samba - samba-technical mailing list
In reply to this post by Samba - samba-technical mailing list
On Tue, 2017-03-21 at 11:49 +0100, Stefan Metzmacher wrote:

> Hi Andrew,
>
> > Scale
> > -----
> >
> > Tied up in performance work is a desire to have samba scale more,
> > to
> > serve more user and hold more groups/group members.  The direction
> > of
> > this work is less certain, but having broken the back of the 'too
> > many
> > links melts Samba' issue, I'm sure we will be asked to do more
> > here.  
> >
> > Specific possibilities include LMDB and a GUID-based index scheme.
>
> If you play with lmdb, please try to understand how OpenLDAP does the
> indexing. I think we should also make use of the virtual databases
> of lmdb, so that we can keep the index records away from the real
> objects.
> We could also have the different partitions within a single lmdb
> file.
Thanks for the suggestions.  That certainly could make it tidier.

Andrew Bartlett

signature.asc (879 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [ROADMAP] Catalyst's focus on Samba

Samba - samba-technical mailing list
In reply to this post by Samba - samba-technical mailing list
On 12/03/17 21:51, Andrew Bartlett via samba-technical wrote:

> Scale
> -----
>
> Tied up in performance work is a desire to have samba scale more, to
> serve more user and hold more groups/group members.  The direction of
> this work is less certain, but having broken the back of the 'too many
> links melts Samba' issue, I'm sure we will be asked to do more here.  
>
> Specific possibilities include LMDB and a GUID-based index scheme.
>
I spent some time looking at abstracting ldb_tdb to interface with a generic key-value store. This was so that LMDB could be plugged in as minimally as possible. There is a quick lmdb backend hacked together partly from Jakub's patches and it's still a mess of patches (and I hadn't quite resolved the build system issues with where to build the library), but the core functionality was there. It could provision and run Samba tests normally and used all the existing indexing schemes.

The interfaces currently use TDB_DATA which should be converted to DATA_BLOB instead.

With some synthetic tests, it improved write performance noticeably despite there being no large structural changes. However under some of the test loads (virtual list view tests), the overall performance was noticeably slower which needs to be investigated. However, if the goal is to bypass the 32-bit limit, then the current arrangement should suffice.

Of the key-value interface operations to implement, I believe the case required for re-indexing was still incomplete.

There's also a bug I found in regards to read-lock counts being broken in ldb_tdb, but I haven't established the consequences of such a bug.


http://git.catalyst.net.nz/gitweb?p=samba.git;a=shortlog;h=refs/heads/lmdb-wip2


Cheers,

Garming


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [LMDB] Re: [ROADMAP] Catalyst's focus on Samba

Samba - samba-technical mailing list
In reply to this post by Samba - samba-technical mailing list
On Wednesday, 22 March 2017 02:02:41 CEST Andrew Bartlett via samba-technical
wrote:

> On Tue, 2017-03-21 at 11:49 +0100, Stefan Metzmacher wrote:
> > Hi Andrew,
> >
> > > Scale
> > > -----
> > >
> > > Tied up in performance work is a desire to have samba scale more,
> > > to
> > > serve more user and hold more groups/group members.  The direction
> > > of
> > > this work is less certain, but having broken the back of the 'too
> > > many
> > > links melts Samba' issue, I'm sure we will be asked to do more
> > > here.  
> > >
> > > Specific possibilities include LMDB and a GUID-based index scheme.
> >
> > If you play with lmdb, please try to understand how OpenLDAP does the
> > indexing. I think we should also make use of the virtual databases
> > of lmdb, so that we can keep the index records away from the real
> > objects.
> > We could also have the different partitions within a single lmdb
> > file.
>
> Thanks for the suggestions.  That certainly could make it tidier.

Jakub Hrozek has a branch with lmdb support and has also written a lot of ldb
tests. The tests use cmocka. We already use cmocka in Samba.

It would be great to get those tests upstream first. Let me know if you need
help with cmocka.


        Andreas


--
Andreas Schneider                   GPG-ID: CC014E3D
Samba Team                             [hidden email]
www.samba.org

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [LMDB] Re: [ROADMAP] Catalyst's focus on Samba

Samba - samba-technical mailing list
On Tue, 2017-03-28 at 08:23 +0200, Andreas Schneider via samba-
technical wrote:

> On Wednesday, 22 March 2017 02:02:41 CEST Andrew Bartlett via samba-
> technical 
> wrote:
> > On Tue, 2017-03-21 at 11:49 +0100, Stefan Metzmacher wrote:
> > > Hi Andrew,
> > >
> > > > Scale
> > > > -----
> > > >
> > > > Tied up in performance work is a desire to have samba scale
> > > > more,
> > > > to
> > > > serve more user and hold more groups/group members.  The
> > > > direction
> > > > of
> > > > this work is less certain, but having broken the back of the
> > > > 'too
> > > > many
> > > > links melts Samba' issue, I'm sure we will be asked to do more
> > > > here.  
> > > >
> > > > Specific possibilities include LMDB and a GUID-based index
> > > > scheme.
> > >
> > > If you play with lmdb, please try to understand how OpenLDAP does
> > > the
> > > indexing. I think we should also make use of the virtual
> > > databases
> > > of lmdb, so that we can keep the index records away from the real
> > > objects.
> > > We could also have the different partitions within a single lmdb
> > > file.
> >
> > Thanks for the suggestions.  That certainly could make it tidier.
>
> Jakub Hrozek has a branch with lmdb support and has also written a
> lot of ldb 
> tests. The tests use cmocka. We already use cmocka in Samba.
>
> It would be great to get those tests upstream first. Let me know if
> you need 
> help with cmocka.

I will keep that in mind!  Has Jakub proposed the tests for inclusion?

As you know, I'm always keen to review patches :-)

While on our roadmap, there isn't any reason to expect the LMDB is
impending, we have plenty of other fish to fry first, so there should
be plenty of time to get those ducks in a row.  

LDB certainly does need more unit tests.

Thanks,

Andrew Bartlett
--
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [LMDB] Re: [ROADMAP] Catalyst's focus on Samba

Samba - samba-technical mailing list
On Tuesday, 28 March 2017 09:15:31 CEST Andrew Bartlett via samba-technical
wrote:

> On Tue, 2017-03-28 at 08:23 +0200, Andreas Schneider via samba-
>
> technical wrote:
> > On Wednesday, 22 March 2017 02:02:41 CEST Andrew Bartlett via samba-
> > technical
> >
> > wrote:
> > > On Tue, 2017-03-21 at 11:49 +0100, Stefan Metzmacher wrote:
> > > > Hi Andrew,
> > > >
> > > > > Scale
> > > > > -----
> > > > >
> > > > > Tied up in performance work is a desire to have samba scale
> > > > > more,
> > > > > to
> > > > > serve more user and hold more groups/group members.  The
> > > > > direction
> > > > > of
> > > > > this work is less certain, but having broken the back of the
> > > > > 'too
> > > > > many
> > > > > links melts Samba' issue, I'm sure we will be asked to do more
> > > > > here.  
> > > > >
> > > > > Specific possibilities include LMDB and a GUID-based index
> > > > > scheme.
> > > >
> > > > If you play with lmdb, please try to understand how OpenLDAP does
> > > > the
> > > > indexing. I think we should also make use of the virtual
> > > > databases
> > > > of lmdb, so that we can keep the index records away from the real
> > > > objects.
> > > > We could also have the different partitions within a single lmdb
> > > > file.
> > >
> > > Thanks for the suggestions.  That certainly could make it tidier.
> >
> > Jakub Hrozek has a branch with lmdb support and has also written a
> > lot of ldb
> > tests. The tests use cmocka. We already use cmocka in Samba.
> >
> > It would be great to get those tests upstream first. Let me know if
> > you need
> > help with cmocka.
>
> I will keep that in mind!  Has Jakub proposed the tests for inclusion?
>
> As you know, I'm always keen to review patches :-)
>
> While on our roadmap, there isn't any reason to expect the LMDB is
> impending, we have plenty of other fish to fry first, so there should
> be plenty of time to get those ducks in a row.  
>
> LDB certainly does need more unit tests.
Here is a patchset from Jakub his branch

https://github.com/jhrozek/samba-ldb-mdb/commits/lmdb

which only adds some unit tests.

cd lib/ldb
./configure
make -j
make test


Cheers,


        Andreas


--
Andreas Schneider                   GPG-ID: CC014E3D
Samba Team                             [hidden email]
www.samba.org

ldb_unittests.patch (57K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [LMDB] Re: [ROADMAP] Catalyst's focus on Samba

Samba - samba-technical mailing list
On Tuesday, 28 March 2017 09:55:00 CEST Andreas Schneider via samba-technical
wrote:

> On Tuesday, 28 March 2017 09:15:31 CEST Andrew Bartlett via samba-technical
>
> wrote:
> > On Tue, 2017-03-28 at 08:23 +0200, Andreas Schneider via samba-
> >
> > technical wrote:
> > > On Wednesday, 22 March 2017 02:02:41 CEST Andrew Bartlett via samba-
> > > technical
> > >
> > > wrote:
> > > > On Tue, 2017-03-21 at 11:49 +0100, Stefan Metzmacher wrote:
> > > > > Hi Andrew,
> > > > >
> > > > > > Scale
> > > > > > -----
> > > > > >
> > > > > > Tied up in performance work is a desire to have samba scale
> > > > > > more,
> > > > > > to
> > > > > > serve more user and hold more groups/group members.  The
> > > > > > direction
> > > > > > of
> > > > > > this work is less certain, but having broken the back of the
> > > > > > 'too
> > > > > > many
> > > > > > links melts Samba' issue, I'm sure we will be asked to do more
> > > > > > here.
> > > > > >
> > > > > > Specific possibilities include LMDB and a GUID-based index
> > > > > > scheme.
> > > > >
> > > > > If you play with lmdb, please try to understand how OpenLDAP does
> > > > > the
> > > > > indexing. I think we should also make use of the virtual
> > > > > databases
> > > > > of lmdb, so that we can keep the index records away from the real
> > > > > objects.
> > > > > We could also have the different partitions within a single lmdb
> > > > > file.
> > > >
> > > > Thanks for the suggestions.  That certainly could make it tidier.
> > >
> > > Jakub Hrozek has a branch with lmdb support and has also written a
> > > lot of ldb
> > > tests. The tests use cmocka. We already use cmocka in Samba.
> > >
> > > It would be great to get those tests upstream first. Let me know if
> > > you need
> > > help with cmocka.
> >
> > I will keep that in mind!  Has Jakub proposed the tests for inclusion?
> >
> > As you know, I'm always keen to review patches :-)
> >
> > While on our roadmap, there isn't any reason to expect the LMDB is
> > impending, we have plenty of other fish to fry first, so there should
> > be plenty of time to get those ducks in a row.
> >
> > LDB certainly does need more unit tests.
>
> Here is a patchset from Jakub his branch
>
> https://github.com/jhrozek/samba-ldb-mdb/commits/lmdb
>
> which only adds some unit tests.
>
> cd lib/ldb
> ./configure
> make -j
> make test
I've enabled subunit output.

--
Andreas Schneider                   GPG-ID: CC014E3D
Samba Team                             [hidden email]
www.samba.org

ldb-tests.patch (58K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

ldb cmocka tests

Samba - samba-technical mailing list
On Wed, 2017-03-29 at 15:22 +0200, Andreas Schneider via samba-
technical wrote:

> > Here is a patchset from Jakub his branch
> >
> > https://github.com/jhrozek/samba-ldb-mdb/commits/lmdb
> >
> > which only adds some unit tests.
> >
> > cd lib/ldb
> > ./configure
> > make -j
> > make test
>
> I've enabled subunit output.
I've tried to integrate this (I'm chasing ldb bugs, so this seemed the
ideal opportunity), but it fails on a host without cmocka while trying
to unconditionally use the test binary.  It also overwrote the result
of the shell-script test suite.

Please squash the attached in, then work out how cmocka is expected to
be handled in autobuild for the samba-libs build, as samba-libs (where
bundling is disabled) it fails with:

Checking for system cmocka >=
1.0                                                               : not
found 
ERROR: System library cmocka of version 1.0 not found, and bundling
disabled

Thanks!

Andrew Bartlett

--
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba

0001-fixup-add-cmocka.patch (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ldb cmocka tests

Samba - samba-technical mailing list
On Thursday, 30 March 2017 09:19:05 CEST Andrew Bartlett via samba-technical
wrote:

> On Wed, 2017-03-29 at 15:22 +0200, Andreas Schneider via samba-
>
> technical wrote:
> > > Here is a patchset from Jakub his branch
> > >
> > > https://github.com/jhrozek/samba-ldb-mdb/commits/lmdb
> > >
> > > which only adds some unit tests.
> > >
> > > cd lib/ldb
> > > ./configure
> > > make -j
> > > make test
> >
> > I've enabled subunit output.
>
> I've tried to integrate this (I'm chasing ldb bugs, so this seemed the
> ideal opportunity), but it fails on a host without cmocka while trying
> to unconditionally use the test binary.  It also overwrote the result
> of the shell-script test suite.
>
> Please squash the attached in, then work out how cmocka is expected to
> be handled in autobuild for the samba-libs build, as samba-libs (where
> bundling is disabled) it fails with:
>
> Checking for system cmocka >=
> 1.0                                                               : not
> found
> ERROR: System library cmocka of version 1.0 not found, and bundling
> disabled
I've fixed it, but this pkg-config checking with waf is so horrible. Why do we
need to to call CHECK_LIB in addition if pkg-config already provided
everything.

However, patchset attached ...



        Andreas

--
Andreas Schneider                   GPG-ID: CC014E3D
Samba Team                             [hidden email]
www.samba.org

ldb-tests-v2.patch (59K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ldb cmocka tests

Samba - samba-technical mailing list
On Thursday, 30 March 2017 10:03:51 CEST Andreas Schneider via samba-technical
wrote:

> On Thursday, 30 March 2017 09:19:05 CEST Andrew Bartlett via samba-technical
> wrote:
> > On Wed, 2017-03-29 at 15:22 +0200, Andreas Schneider via samba-
> >
> > technical wrote:
> > > > Here is a patchset from Jakub his branch
> > > >
> > > > https://github.com/jhrozek/samba-ldb-mdb/commits/lmdb
> > > >
> > > > which only adds some unit tests.
> > > >
> > > > cd lib/ldb
> > > > ./configure
> > > > make -j
> > > > make test
> > >
> > > I've enabled subunit output.
> >
> > I've tried to integrate this (I'm chasing ldb bugs, so this seemed the
> > ideal opportunity), but it fails on a host without cmocka while trying
> > to unconditionally use the test binary.  It also overwrote the result
> > of the shell-script test suite.
> >
> > Please squash the attached in, then work out how cmocka is expected to
> > be handled in autobuild for the samba-libs build, as samba-libs (where
> > bundling is disabled) it fails with:
> >
> > Checking for system cmocka >=
> > 1.0                                                               : not
> > found
> > ERROR: System library cmocka of version 1.0 not found, and bundling
> > disabled
>
> I've fixed it, but this pkg-config checking with waf is so horrible. Why do
> we need to to call CHECK_LIB in addition if pkg-config already provided
> everything.
>
> However, patchset attached ...
This one is with ldb_rename tests!


        Andreas

--
Andreas Schneider                   GPG-ID: CC014E3D
Samba Team                             [hidden email]
www.samba.org

ldb-tests-v3.patch (66K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [LMDB] Re: [ROADMAP] Catalyst's focus on Samba

Samba - samba-technical mailing list
In reply to this post by Samba - samba-technical mailing list
On Tuesday, 28 March 2017 08:23:19 CEST Andreas Schneider via samba-technical
wrote:

> On Wednesday, 22 March 2017 02:02:41 CEST Andrew Bartlett via
> samba-technical
> wrote:
> > On Tue, 2017-03-21 at 11:49 +0100, Stefan Metzmacher wrote:
> > > Hi Andrew,
> > >
> > > > Scale
> > > > -----
> > > >
> > > > Tied up in performance work is a desire to have samba scale more,
> > > > to
> > > > serve more user and hold more groups/group members.  The direction
> > > > of
> > > > this work is less certain, but having broken the back of the 'too
> > > > many
> > > > links melts Samba' issue, I'm sure we will be asked to do more
> > > > here.
> > > >
> > > > Specific possibilities include LMDB and a GUID-based index scheme.
> > >
> > > If you play with lmdb, please try to understand how OpenLDAP does the
> > > indexing. I think we should also make use of the virtual databases
> > > of lmdb, so that we can keep the index records away from the real
> > > objects.
> > > We could also have the different partitions within a single lmdb
> > > file.
> >
> > Thanks for the suggestions.  That certainly could make it tidier.
>
> Jakub Hrozek has a branch with lmdb support and has also written a lot of
> ldb tests. The tests use cmocka. We already use cmocka in Samba.

There is a TODO for the branch which will probably be helpful if you start
working on it ...

https://github.com/jhrozek/samba-ldb-mdb/issues

--
Andreas Schneider                   GPG-ID: CC014E3D
Samba Team                             [hidden email]
www.samba.org

Loading...