Disable the flapping tests for now?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Disable the flapping tests for now?

Samba - samba-technical mailing list
This is a bit late, but anyway:

Something has changed, and the rfc2307 and whoami tests are flapping
50% - 75% of the time.  

If history is any guide this is causing some considerable stress.

I wonder if we could disable the flapping tests and give a couple of
days breathing room for those who reasonably expected patches to land
to do so, then release 4.8.0rc1?

(Garming has been trying to land David's GPO patches for example, and I
keep seeing Jeremy in the autobuild queue but without a patch landing).

I normally don't like declaring something flapping until we know why
that is the case, as doing so removes the pressure to actually fix the
underlying issue.  

My rationale here is that unlike the case at 4.5.0rc1 with flapping DRS
tests, these flapping tests seem to be quite specific and effectively
removing them wouldn't reduce the overall testing significantly.

What do folks think?

Andrew Bartlett

--
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba


Reply | Threaded
Open this post in threaded view
|

Re: Disable the flapping tests for now?

Samba - samba-technical mailing list
On Thursday, 11 January 2018 19:35:38 CET Andrew Bartlett via samba-technical
wrote:

> This is a bit late, but anyway:
>
> Something has changed, and the rfc2307 and whoami tests are flapping
> 50% - 75% of the time.
>
> If history is any guide this is causing some considerable stress.
>
> I wonder if we could disable the flapping tests and give a couple of
> days breathing room for those who reasonably expected patches to land
> to do so, then release 4.8.0rc1?
>
> (Garming has been trying to land David's GPO patches for example, and I
> keep seeing Jeremy in the autobuild queue but without a patch landing).
>
> I normally don't like declaring something flapping until we know why
> that is the case, as doing so removes the pressure to actually fix the
> underlying issue.
>
> My rationale here is that unlike the case at 4.5.0rc1 with flapping DRS
> tests, these flapping tests seem to be quite specific and effectively
> removing them wouldn't reduce the overall testing significantly.
>
> What do folks think?
I've had a short look with Ralph yesterday. Currently a patch for the change
secret test is on the way to autobuild. The whoami test fails creating an smb
connection to nt4_member using the machine account. It is possible that an
earlier test breaks the secret.

Possible culprit, see attached (in autobuild right now).

For the idmap_rfc2703 test, it looks like that something prevents output.
There are a lot of "echo" in it to give information but you don't find them in
the autobuild logs. It would be great if someone could look into why we don't
get any output.

I will look into the whoami test.

This is not a NAK but we need to improve debug output of that tests, would be
great to do that first.


        Andreas


--
Andreas Schneider                   GPG-ID: CC014E3D
Samba Team                             [hidden email]
www.samba.org

0001-s3-test-Always-validate-the-join-after-changing-the-.patch (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Disable the flapping tests for now?

Samba - samba-technical mailing list
On Fri, 2018-01-12 at 09:34 +0100, Andreas Schneider wrote:

>
>
> I've had a short look with Ralph yesterday. Currently a patch for the change
> secret test is on the way to autobuild. The whoami test fails creating an smb
> connection to nt4_member using the machine account. It is possible that an
> earlier test breaks the secret.
>
> Possible culprit, see attached (in autobuild right now).
>
> For the idmap_rfc2703 test, it looks like that something prevents output.
> There are a lot of "echo" in it to give information but you don't find them in
> the autobuild logs. It would be great if someone could look into why we don't
> get any output.
>
> I will look into the whoami test.
>
> This is not a NAK but we need to improve debug output of that tests, would be
> great to do that first.
>

Thank you so much Andreas for the analysis.  

I'm very happy to run multiple builds on the Catalyst Cloud for any
branches folks desire.  

We can also, if desired, preserve the machines for a time for analysis.
Naturally everything costs, but so multiple builds 'just in case' and
days spent re-pushing autobuilds.   :-)

As you know I want to try and do some CI via gitlab, and this will only
be viable if we have a reliable build, so this remains a key priority.

Thanks,

Andrew Bartlett

--
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba


Reply | Threaded
Open this post in threaded view
|

Re: Disable the flapping tests for now?

Samba - samba-technical mailing list
In reply to this post by Samba - samba-technical mailing list
On Fri, Jan 12, 2018 at 07:35:38AM +1300, Andrew Bartlett via samba-technical wrote:

> This is a bit late, but anyway:
>
> Something has changed, and the rfc2307 and whoami tests are flapping
> 50% - 75% of the time.  
>
> If history is any guide this is causing some considerable stress.
>
> I wonder if we could disable the flapping tests and give a couple of
> days breathing room for those who reasonably expected patches to land
> to do so, then release 4.8.0rc1?
>
> (Garming has been trying to land David's GPO patches for example, and I
> keep seeing Jeremy in the autobuild queue but without a patch landing).
>
> I normally don't like declaring something flapping until we know why
> that is the case, as doing so removes the pressure to actually fix the
> underlying issue.  
>
> My rationale here is that unlike the case at 4.5.0rc1 with flapping DRS
> tests, these flapping tests seem to be quite specific and effectively
> removing them wouldn't reduce the overall testing significantly.
>
> What do folks think?

yes, yes, yes.

See my other mail, included in metze's and my autobuild.

-slow

--
Ralph Boehme, Samba Team       https://samba.org/
Samba Developer, SerNet GmbH   https://sernet.de/en/samba/