[WIP][PATCH] Run make test envs in parallel in autobuild

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[WIP][PATCH] Run make test envs in parallel in autobuild

Andrew Bartlett
This patch was inspired by the fact that even our basic python tests do
not run in the travis-ci triggered by github pull requests.

I also wanted to see what time we could save by running some tests from
the separated environments in a different autobuild task (which maps to
distinct travis-ci tasks).

On the Catalyst cloud, this decreases overall time on a 4 CPU server by
80mins, to 180min from 260min.  It may be making the flapping tests
flap more also however, and these need to be understood.

Tasks that run in under 45mins can run in travis-ci, which is a good
thing.

Anyway, for now it is food for thought and a possible way to reduce the
ever-expanding build and test times, and ensure more of our tests run
for github users.

Thanks,

Andrew Bartlett
--
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba

0001-selftest-Allow-autobuild-to-run-nt4_-ad_dc-tests-in-.patch (8K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [WIP][PATCH] Run make test envs in parallel in autobuild

Samba - samba-technical mailing list
On Tue, 2017-02-28 at 21:58 +1300, Andrew Bartlett wrote:

> This patch was inspired by the fact that even our basic python tests do
> not run in the travis-ci triggered by github pull requests.
>
> I also wanted to see what time we could save by running some tests from
> the separated environments in a different autobuild task (which maps to
> distinct travis-ci tasks).
>
> On the Catalyst cloud, this decreases overall time on a 4 CPU server by
> 80mins, to 180min from 260min.  It may be making the flapping tests
> flap more also however, and these need to be understood.
>
> Tasks that run in under 45mins can run in travis-ci, which is a good
> thing.
>
> Anyway, for now it is food for thought and a possible way to reduce the
> ever-expanding build and test times, and ensure more of our tests run
> for github users.

I'm giving this another go, after comments at the team meeting.

Hopefully with less flapping tests overall, this becomes practical.

Thanks!

Andrew Bartlett
--
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba


Reply | Threaded
Open this post in threaded view
|

Re: [WIP][PATCH] Run make test envs in parallel in autobuild

Samba - samba-technical mailing list
On Friday, 5 May 2017 22:41:45 CEST Andrew Bartlett via samba-technical wrote:

> On Tue, 2017-02-28 at 21:58 +1300, Andrew Bartlett wrote:
> > This patch was inspired by the fact that even our basic python tests do
> > not run in the travis-ci triggered by github pull requests.
> >
> > I also wanted to see what time we could save by running some tests from
> > the separated environments in a different autobuild task (which maps to
> > distinct travis-ci tasks).
> >
> > On the Catalyst cloud, this decreases overall time on a 4 CPU server by
> > 80mins, to 180min from 260min.  It may be making the flapping tests
> > flap more also however, and these need to be understood.
> >
> > Tasks that run in under 45mins can run in travis-ci, which is a good
> > thing.
> >
> > Anyway, for now it is food for thought and a possible way to reduce the
> > ever-expanding build and test times, and ensure more of our tests run
> > for github users.
>
> I'm giving this another go, after comments at the team meeting.
>
> Hopefully with less flapping tests overall, this becomes practical.

Thanks for looking into this Andrew!

Reply | Threaded
Open this post in threaded view
|

Re: [WIP][PATCH] Run make test envs in parallel in autobuild

Samba - samba-technical mailing list
On Tue, 2017-05-09 at 22:15 +0200, Andreas Schneider via samba-
technical wrote:

> On Friday, 5 May 2017 22:41:45 CEST Andrew Bartlett via samba-technical wrote:
> > On Tue, 2017-02-28 at 21:58 +1300, Andrew Bartlett wrote:
> > > This patch was inspired by the fact that even our basic python tests do
> > > not run in the travis-ci triggered by github pull requests.
> > >
> > > I also wanted to see what time we could save by running some tests from
> > > the separated environments in a different autobuild task (which maps to
> > > distinct travis-ci tasks).
> > >
> > > On the Catalyst cloud, this decreases overall time on a 4 CPU server by
> > > 80mins, to 180min from 260min.  It may be making the flapping tests
> > > flap more also however, and these need to be understood.
> > >
> > > Tasks that run in under 45mins can run in travis-ci, which is a good
> > > thing.
> > >
> > > Anyway, for now it is food for thought and a possible way to reduce the
> > > ever-expanding build and test times, and ensure more of our tests run
> > > for github users.
> >
> > I'm giving this another go, after comments at the team meeting.
> >
> > Hopefully with less flapping tests overall, this becomes practical.
>
> Thanks for looking into this Andrew!
Just a status update on this.  It seems with my figures above I may
have made autobuild faster simply by removing all the nt4_dc tests,
rather than making things more parallel.  Oops.

Before I push anything I'll get more stats and I'll be counting the
number of tests run to ensure we haven't lost any.

The attached patch series makes us parallel while, I hope, keeping the
nt4_dc and nt4_member tests.

Also attached is a patch I have under test to reverse a regression I
noticed where in a preparitory patch for this, started under-testing
the samba-systemkrb5 autobuild target, oops!

Thanks,

Andrew Bartlett

--
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba

attachment0 (2K) Download Attachment
parallel-test.patch.txt (12K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

[PATCH] Restore samba-systemkrb5 tests

Samba - samba-technical mailing list
The attached patch addresses a regression that caused only 7 tests to
run in the samba-systemkrb5 autobuild target.

We now match the previous number of tests, plus one that didn't match
the previous regular expression.

I do apologise for the regression.

Please review/push!

Thanks,

Andrew Bartlett
--
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba

attachment0 (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Restore samba-systemkrb5 tests

Samba - samba-technical mailing list
Am 22.07.2017 um 03:25 schrieb Andrew Bartlett via samba-technical:
> The attached patch addresses a regression that caused only 7 tests to
> run in the samba-systemkrb5 autobuild target.
>
> We now match the previous number of tests, plus one that didn't match
> the previous regular expression.
>
> I do apologise for the regression.
>
> Please review/push!

Pushed to autobuild.

metze



signature.asc (853 bytes) Download Attachment