Selftest environment interdependency graph

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Selftest environment interdependency graph

Samba - samba-technical mailing list
Hi all,

I've been having a look at parallelisation of the testing process. To
that end, I have, through the Perl-parsing power of regex(!), created a
graph of dependencies between envs, including the time used running
suites for each env in one run on the catalyst cloud. This is attached
as svg for your viewing pleasure.



The tooling used to create it is also attached - let me know if you have
any issues with it (I started writing it with gawk then switched to ruby
for the second half, so I wouldn't be surprised if you do!).


Thanks,
- Jamie

samba_env_dependencies.svg (23K) Download Attachment
envtimes.rb (978 bytes) Download Attachment
depgraph.sh (616 bytes) Download Attachment
deptree.gawk (516 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Selftest environment interdependency graph

Samba - samba-technical mailing list
On Monday, 8 January 2018 07:16:17 CET Jamie McClymont via samba-technical
wrote:
> Hi all,
>
> I've been having a look at parallelisation of the testing process. To
> that end, I have, through the Perl-parsing power of regex(!), created a
> graph of dependencies between envs, including the time used running
> suites for each env in one run on the catalyst cloud. This is attached
> as svg for your viewing pleasure.

Hi Jamie,

looking at the graph we see that some things should be changed. Metze and I
looked into moving ad_member to ad_dc:

https://git.samba.org/?p=asn/
samba.git;a=commitdiff;h=7f8dc6ff0662b5e3f6bfeab30139f1de37fc2edf

However we got sidetracked by other stuff. So it doesn't work yet. There are
several tests failing for unknown reasons.

More WIP is here:
https://git.samba.org/?p=asn/samba.git;a=shortlog;h=refs/heads/master-selftest



        Andreas


--
Andreas Schneider                   GPG-ID: CC014E3D
Samba Team                             [hidden email]
www.samba.org



Reply | Threaded
Open this post in threaded view
|

Re: Selftest environment interdependency graph

Samba - samba-technical mailing list
On Mon, 2018-01-08 at 10:06 +0100, Andreas Schneider via samba-
technical wrote:

> On Monday, 8 January 2018 07:16:17 CET Jamie McClymont via samba-technical
> wrote:
> > Hi all,
> >
> > I've been having a look at parallelisation of the testing process. To
> > that end, I have, through the Perl-parsing power of regex(!), created a
> > graph of dependencies between envs, including the time used running
> > suites for each env in one run on the catalyst cloud. This is attached
> > as svg for your viewing pleasure.
>
> Hi Jamie,
>
> looking at the graph we see that some things should be changed. Metze and I
> looked into moving ad_member to ad_dc:
>
> https://git.samba.org/?p=asn/
> samba.git;a=commitdiff;h=7f8dc6ff0662b5e3f6bfeab30139f1de37fc2edf
>
> However we got sidetracked by other stuff. So it doesn't work yet. There are
> several tests failing for unknown reasons.
>
> More WIP is here:
> https://git.samba.org/?p=asn/samba.git;a=shortlog;h=refs/heads/master-selftest

Yeah, there is much still to untangle here.  The NT4 stuff should be
really simple to split out, but every time I try I get very odd
failures, like the ones Jamie already chased down, and now this one:

UNEXPECTED(failure): samba4.winbind.wbclient.wbcListUsers(nt4_dc:local)
REASON: Exception: Exception:
../nsswitch/libwbclient/tests/wbclient.c:361: wbcLookupUserSids(&sid,
1, &num_sids, &sids) was WBC_ERR_DOMAIN_NOT_FOUND, expected
WBC_ERR_SUCCESS: wbcLookupUserSids of S-1-5-21-2923062406-1030520007-
827082798-1158 failed

Andrew Bartlett

--
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba


Reply | Threaded
Open this post in threaded view
|

Re: Selftest environment interdependency graph

Samba - samba-technical mailing list
On Monday, 8 January 2018 10:12:16 CET Andrew Bartlett wrote:

> On Mon, 2018-01-08 at 10:06 +0100, Andreas Schneider via samba-
>
> technical wrote:
> > On Monday, 8 January 2018 07:16:17 CET Jamie McClymont via samba-technical
> >
> > wrote:
> > > Hi all,
> > >
> > > I've been having a look at parallelisation of the testing process. To
> > > that end, I have, through the Perl-parsing power of regex(!), created a
> > > graph of dependencies between envs, including the time used running
> > > suites for each env in one run on the catalyst cloud. This is attached
> > > as svg for your viewing pleasure.
> >
> > Hi Jamie,
> >
> > looking at the graph we see that some things should be changed. Metze and
> > I
> > looked into moving ad_member to ad_dc:
> >
> > https://git.samba.org/?p=asn/
> > samba.git;a=commitdiff;h=7f8dc6ff0662b5e3f6bfeab30139f1de37fc2edf
> >
> > However we got sidetracked by other stuff. So it doesn't work yet. There
> > are several tests failing for unknown reasons.
> >
> > More WIP is here:
> > https://git.samba.org/?p=asn/samba.git;a=shortlog;h=refs/heads/master-self
> > test
> Yeah, there is much still to untangle here.  The NT4 stuff should be
> really simple to split out, but every time I try I get very odd
> failures, like the ones Jamie already chased down, and now this one:
>
> UNEXPECTED(failure): samba4.winbind.wbclient.wbcListUsers(nt4_dc:local)
> REASON: Exception: Exception:
> ../nsswitch/libwbclient/tests/wbclient.c:361: wbcLookupUserSids(&sid,
> 1, &num_sids, &sids) was WBC_ERR_DOMAIN_NOT_FOUND, expected
> WBC_ERR_SUCCESS: wbcLookupUserSids of S-1-5-21-2923062406-1030520007-
> 827082798-1158 failed

I think this is the test we also hit last time and then got sidetracked :-(

A problem is that a lot of tests taint environments and do not clean up.
Leaving test users around or use the Administrator account and change that
than creating a temporary user :-(


However, thanks for the work on this :-)

--
Andreas Schneider                   GPG-ID: CC014E3D
Samba Team                             [hidden email]
www.samba.org



Reply | Threaded
Open this post in threaded view
|

Re: Selftest environment interdependency graph

Samba - samba-technical mailing list
On Mon, Jan 08, 2018 at 10:19:41AM +0100, Andreas Schneider via samba-technical wrote:

> On Monday, 8 January 2018 10:12:16 CET Andrew Bartlett wrote:
> > On Mon, 2018-01-08 at 10:06 +0100, Andreas Schneider via samba-
> >
> > technical wrote:
> > > On Monday, 8 January 2018 07:16:17 CET Jamie McClymont via samba-technical
> > >
> > > wrote:
> > > > Hi all,
> > > >
> > > > I've been having a look at parallelisation of the testing process. To
> > > > that end, I have, through the Perl-parsing power of regex(!), created a
> > > > graph of dependencies between envs, including the time used running
> > > > suites for each env in one run on the catalyst cloud. This is attached
> > > > as svg for your viewing pleasure.
> > >
> > > Hi Jamie,
> > >
> > > looking at the graph we see that some things should be changed. Metze and
> > > I
> > > looked into moving ad_member to ad_dc:
> > >
> > > https://git.samba.org/?p=asn/
> > > samba.git;a=commitdiff;h=7f8dc6ff0662b5e3f6bfeab30139f1de37fc2edf
> > >
> > > However we got sidetracked by other stuff. So it doesn't work yet. There
> > > are several tests failing for unknown reasons.
> > >
> > > More WIP is here:
> > > https://git.samba.org/?p=asn/samba.git;a=shortlog;h=refs/heads/master-self
> > > test
> > Yeah, there is much still to untangle here.  The NT4 stuff should be
> > really simple to split out, but every time I try I get very odd
> > failures, like the ones Jamie already chased down, and now this one:
> >
> > UNEXPECTED(failure): samba4.winbind.wbclient.wbcListUsers(nt4_dc:local)
> > REASON: Exception: Exception:
> > ../nsswitch/libwbclient/tests/wbclient.c:361: wbcLookupUserSids(&sid,
> > 1, &num_sids, &sids) was WBC_ERR_DOMAIN_NOT_FOUND, expected
> > WBC_ERR_SUCCESS: wbcLookupUserSids of S-1-5-21-2923062406-1030520007-
> > 827082798-1158 failed
>
> I think this is the test we also hit last time and then got sidetracked :-(
>
> A problem is that a lot of tests taint environments and do not clean up.
> Leaving test users around or use the Administrator account and change that
> than creating a temporary user :-(
I just ran into a similar issue in the nt4_member environment: setup uses net
join, not net rpc join, so the the join when attempted with ADS fails which
triggers creating of a record for the DCs *IP* in the negative conncection
cache.

This caused interesting, timing dependent failure modes later in
winbindd. There, some codepatch check the negative connection cache with the
server name (init_dc_connection), others use the the IP
(check_domain_online_handler).

Attached patch fixed the issue for me by using net rpc join in the selftest
setup. Please review & push if happy.

-slow

--
Ralph Boehme, Samba Team       https://samba.org/
Samba Developer, SerNet GmbH   https://sernet.de/en/samba/

nt4member-join.patch (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Selftest environment interdependency graph

Samba - samba-technical mailing list
On Mon, 2018-01-08 at 10:30 +0100, Ralph Böhme wrote:

>
>
> I just ran into a similar issue in the nt4_member environment: setup uses net
> join, not net rpc join, so the the join when attempted with ADS fails which
> triggers creating of a record for the DCs *IP* in the negative conncection
> cache.
>
> This caused interesting, timing dependent failure modes later in
> winbindd. There, some codepatch check the negative connection cache with the
> server name (init_dc_connection), others use the the IP
> (check_domain_online_handler).
>
> Attached patch fixed the issue for me by using net rpc join in the selftest
> setup. Please review & push if happy.

Thanks!

Reviewed-by: Andrew Bartlett <[hidden email]>

pushed!

Andrew Bartlett

--
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba


Reply | Threaded
Open this post in threaded view
|

Re: Selftest environment interdependency graph

Samba - samba-technical mailing list
On Mon, Jan 08, 2018 at 10:52:08PM +1300, Andrew Bartlett wrote:
> Reviewed-by: Andrew Bartlett <[hidden email]>
>
> pushed!

thanks!

Fwiw, while debugging this problem I noticed an issue in
selftest/target/Samba3.pm:wait_for_start() in the nt4_member environment:

  $ret = system(Samba::bindir_path($self, "net") ." $envvars->{CONFIGURATION} groupmap add rid=513 unixgroup=domusers type=domain");

fails with an error "Can't lookup UNIX group domusers" so the subsequent code is
skipped. I couldn't find the reason, so if anyone can give me a clue that would
be much appreciated.

-slow

--
Ralph Boehme, Samba Team       https://samba.org/
Samba Developer, SerNet GmbH   https://sernet.de/en/samba/

Reply | Threaded
Open this post in threaded view
|

Re: Selftest environment interdependency graph

Samba - samba-technical mailing list
On ma, 08 tammi 2018, Ralph Böhme via samba-technical wrote:

> On Mon, Jan 08, 2018 at 10:52:08PM +1300, Andrew Bartlett wrote:
> > Reviewed-by: Andrew Bartlett <[hidden email]>
> >
> > pushed!
>
> thanks!
>
> Fwiw, while debugging this problem I noticed an issue in
> selftest/target/Samba3.pm:wait_for_start() in the nt4_member environment:
>
>   $ret = system(Samba::bindir_path($self, "net") ." $envvars->{CONFIGURATION} groupmap add rid=513 unixgroup=domusers type=domain");
>
> fails with an error "Can't lookup UNIX group domusers" so the subsequent code is
> skipped. I couldn't find the reason, so if anyone can give me a clue that would
> be much appreciated.
'net groupmap' calls aren't wrapped with NSS_WRAPPER_PASSWD=/NSS_WRAPPER_GROUP=
unlike winbindd startup just before them. As result, our wrapped passwd
and group files aren't in use and group domusers is not found.

--
/ Alexander Bokovoy

Reply | Threaded
Open this post in threaded view
|

Re: Selftest environment interdependency graph

Samba - samba-technical mailing list
On Mon, Jan 08, 2018 at 02:30:47PM +0200, Alexander Bokovoy wrote:

> On ma, 08 tammi 2018, Ralph Böhme via samba-technical wrote:
> > On Mon, Jan 08, 2018 at 10:52:08PM +1300, Andrew Bartlett wrote:
> > > Reviewed-by: Andrew Bartlett <[hidden email]>
> > >
> > > pushed!
> >
> > thanks!
> >
> > Fwiw, while debugging this problem I noticed an issue in
> > selftest/target/Samba3.pm:wait_for_start() in the nt4_member environment:
> >
> >   $ret = system(Samba::bindir_path($self, "net") ." $envvars->{CONFIGURATION} groupmap add rid=513 unixgroup=domusers type=domain");
> >
> > fails with an error "Can't lookup UNIX group domusers" so the subsequent code is
> > skipped. I couldn't find the reason, so if anyone can give me a clue that would
> > be much appreciated.
> 'net groupmap' calls aren't wrapped with NSS_WRAPPER_PASSWD=/NSS_WRAPPER_GROUP=
> unlike winbindd startup just before them. As result, our wrapped passwd
> and group files aren't in use and group domusers is not found.

Man, you're the man! :) Thanks!

I'll prepare a patch.

-slow

--
Ralph Boehme, Samba Team       https://samba.org/
Samba Developer, SerNet GmbH   https://sernet.de/en/samba/

Reply | Threaded
Open this post in threaded view
|

ENV based inter-dependencies (was: Re: Selftest environment interdependency graph)

Samba - samba-technical mailing list
On Mon, 2018-01-08 at 14:06 +0100, Ralph Böhme wrote:

> On Mon, Jan 08, 2018 at 02:30:47PM +0200, Alexander Bokovoy wrote:
> > On ma, 08 tammi 2018, Ralph Böhme via samba-technical wrote:
> > > On Mon, Jan 08, 2018 at 10:52:08PM +1300, Andrew Bartlett wrote:
> > > > Reviewed-by: Andrew Bartlett <[hidden email]>
> > > >
> > > > pushed!
> > >
> > > thanks!
> > >
> > > Fwiw, while debugging this problem I noticed an issue in
> > > selftest/target/Samba3.pm:wait_for_start() in the nt4_member environment:
> > >
> > >   $ret = system(Samba::bindir_path($self, "net") ." $envvars->{CONFIGURATION} groupmap add rid=513 unixgroup=domusers type=domain");
> > >
> > > fails with an error "Can't lookup UNIX group domusers" so the subsequent code is
> > > skipped. I couldn't find the reason, so if anyone can give me a clue that would
> > > be much appreciated.
> >
> > 'net groupmap' calls aren't wrapped with NSS_WRAPPER_PASSWD=/NSS_WRAPPER_GROUP=
> > unlike winbindd startup just before them. As result, our wrapped passwd
> > and group files aren't in use and group domusers is not found.
>
> Man, you're the man! :) Thanks!
>
> I'll prepare a patch.

Also in the 'this is going to case bugs' department:

In Samba4.pm we have this gem, setting global environment variables
(rather than only in a forked child or system() call):

                # Add hosts file for name lookups
                $ENV{NSS_WRAPPER_HOSTS} = $testenv_vars->{NSS_WRAPPER_HOSTS};
                if (defined($testenv_vars->{RESOLV_WRAPPER_CONF})) {
                        $ENV{RESOLV_WRAPPER_CONF} = $testenv_vars->{RESOLV_WRAPPER_CONF};
                } else {
                        $ENV{RESOLV_WRAPPER_HOSTS} = $testenv_vars->{RESOLV_WRAPPER_HOSTS};
                }

Who knows, maybe we can finally split the monster apart!

Andrew Bartlett
--
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba


Reply | Threaded
Open this post in threaded view
|

Re: ENV based inter-dependencies (was: Re: Selftest environment interdependency graph)

Samba - samba-technical mailing list
On Tue, 2018-01-09 at 07:11 +1300, Andrew Bartlett via samba-technical
wrote:

> On Mon, 2018-01-08 at 14:06 +0100, Ralph Böhme wrote:
> > On Mon, Jan 08, 2018 at 02:30:47PM +0200, Alexander Bokovoy wrote:
> > > On ma, 08 tammi 2018, Ralph Böhme via samba-technical wrote:
> > > > On Mon, Jan 08, 2018 at 10:52:08PM +1300, Andrew Bartlett wrote:
> > > > > Reviewed-by: Andrew Bartlett <[hidden email]>
> > > > >
> > > > > pushed!
> > > >
> > > > thanks!
> > > >
> > > > Fwiw, while debugging this problem I noticed an issue in
> > > > selftest/target/Samba3.pm:wait_for_start() in the nt4_member environment:
> > > >
> > > >   $ret = system(Samba::bindir_path($self, "net") ." $envvars->{CONFIGURATION} groupmap add rid=513 unixgroup=domusers type=domain");
> > > >
> > > > fails with an error "Can't lookup UNIX group domusers" so the subsequent code is
> > > > skipped. I couldn't find the reason, so if anyone can give me a clue that would
> > > > be much appreciated.
> > >
> > > 'net groupmap' calls aren't wrapped with NSS_WRAPPER_PASSWD=/NSS_WRAPPER_GROUP=
> > > unlike winbindd startup just before them. As result, our wrapped passwd
> > > and group files aren't in use and group domusers is not found.
> >
> > Man, you're the man! :) Thanks!
> >
> > I'll prepare a patch.
>
> Also in the 'this is going to case bugs' department:
>
> In Samba4.pm we have this gem, setting global environment variables
> (rather than only in a forked child or system() call):
>
>                 # Add hosts file for name lookups
>                 $ENV{NSS_WRAPPER_HOSTS} = $testenv_vars->{NSS_WRAPPER_HOSTS};
>                 if (defined($testenv_vars->{RESOLV_WRAPPER_CONF})) {
>                         $ENV{RESOLV_WRAPPER_CONF} = $testenv_vars->{RESOLV_WRAPPER_CONF};
>                 } else {
>                         $ENV{RESOLV_WRAPPER_HOSTS} = $testenv_vars->{RESOLV_WRAPPER_HOSTS};
>                 }
>
> Who knows, maybe we can finally split the monster apart!

There is also this line in Samba3.pm:provision()

$ENV{KRB5CCNAME} = $ret{KRB5_CCACHE};

From:
8a029b5d2aa2604c7d79e70d1a372bd029fabd75
selftest: Do not use a central credential cache
   
The concept of that patch was fine, but the stamping into the global
environment must never happen in the env setup, otherwise they will
cross-pollute.  It must be set with the others for each test call (in
the forked child).

Andrew Bartlett

--
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba