sn-devel flapping tests summary since 2017-02-10

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

sn-devel flapping tests summary since 2017-02-10

Samba - samba-technical mailing list

This is the latest in an occasional series consisting of me running a
script[1] that counts intermittent failing tests, as indicated by all the
emails sent to the samba-cvs[2] list by the test robot that have
reached me.

In this case the date range has been selected to miss the masktest
failures that were discussed here and fixed in mid-February.

The tests that flapped more than once are:

$ ./parse-email/parse-autobuild-email --file-regex sn-devel-144 --since
2017-02-10 --filter-line=': ([\w.]+)'
found 54 lines matching '^UNEXPECTED' in 87 files matching 'sn-devel-144'
  19 samba4.drs.replica_sync.python
   6
samba.tests.dcerpc.dnsserver.samba.tests.dcerpc.dnsserver.DnsserverTests.test_accept_valid_commands
   3 samba4.drs.repl_move.python
   3
samba.tests.dcerpc.dnsserver.samba.tests.dcerpc.dnsserver.DnsserverTests.test_add_similar_value
   3 samba4.blackbox.masktest.masktest
   2 samba3.raw.notify.mask
   2 samba3.blackbox.smbclient_s3.crypt
   2 samba3.blackbox.smbclient_s3.sign
   2
samba.tests.blackbox.samba_dnsupdate.samba.tests.blackbox.samba_dnsupdate.SambaDnsUpdateTests.test_samba_dnsupate_set_ip

or in more detail:

$ ./parse-email/parse-autobuild-email --file-regex sn-devel-144 --since
2017-02-10
found 54 lines matching '^UNEXPECTED' in 87 files matching 'sn-devel-144'
   7 UNEXPECTED(failure):
samba4.drs.replica_sync.python(promoted_dc).replica_sync.DrsReplicaSyncTestCase.test_ReplRenames(promoted_dc:local)
   4 UNEXPECTED(failure):
samba.tests.dcerpc.dnsserver.samba.tests.dcerpc.dnsserver.DnsserverTests.test_accept_valid_commands(ad_dc)
   4 UNEXPECTED(error):
samba4.drs.replica_sync.python(vampire_dc).replica_sync.DrsReplicaSyncTestCase.test_ReplConflictsRemoteWin(vampire_dc:local)
   3 UNEXPECTED(error):
samba4.drs.repl_move.python(promoted_dc).repl_move.DrsMoveBetweenTreeOfObjectTestCase.test_ReplicateAddInConflictOU2(promoted_dc)
   3 UNEXPECTED(failure): samba4.blackbox.masktest.masktest(ad_dc_ntvfs)
   3 UNEXPECTED(error):
samba4.drs.replica_sync.python(vampire_dc).replica_sync.DrsReplicaSyncTestCase.test_ReplConflictsRemoteWin_with_child(vampire_dc:local)
   2 UNEXPECTED(error):
samba4.drs.replica_sync.python(promoted_dc).replica_sync.DrsReplicaSyncTestCase.test_ReplConflictsRemoteWin_with_child(promoted_dc:local)
   2 UNEXPECTED(failure):
samba.tests.dcerpc.dnsserver.samba.tests.dcerpc.dnsserver.DnsserverTests.test_accept_valid_commands(ad_dc_ntvfs)
   2 UNEXPECTED(failure): samba3.blackbox.smbclient_s3.sign (nt4_member)
member creds.follow symlinks = no(nt4_member)
   2 UNEXPECTED(failure):
samba.tests.blackbox.samba_dnsupdate.samba.tests.blackbox.samba_dnsupdate.SambaDnsUpdateTests.test_samba_dnsupate_set_ip(chgdcpass:local)
   2 UNEXPECTED(failure):
samba.tests.dcerpc.dnsserver.samba.tests.dcerpc.dnsserver.DnsserverTests.test_add_similar_value(ad_dc_ntvfs)
   2 UNEXPECTED(failure): samba3.raw.notify.mask(nt4_dc)
   2 UNEXPECTED(failure): samba3.blackbox.smbclient_s3.crypt
(nt4_dc).follow symlinks = no(nt4_dc)


Finally, here is a histogram to show how we've been going over time
(ignoring the abandonware known as "sn-devel-104"):

$ ./parse-email/parse-autobuild-email  --file-regex '(sn-devel-144)'
--histogram
2015-11   5 #####
2015-12  18 ##################
2016-01  36 ####################################
2016-02  35 ###################################
2016-03  47 ###############################################
2016-04  49 #################################################
2016-05  55 #######################################################
2016-06  58 ##########################################################
2016-07  53 #####################################################
2016-08  50 ##################################################
2016-09  24 ########################
2016-10  22 ######################
2016-11  23 #######################
2016-12  22 ######################
2017-01  44 ############################################
2017-02  29 #############################
2017-03  22 ######################
2017-04  18 ##################


We are getting better! More tests and fewer flaps. Luckily April has
been such a short month.

Douglas



[1]
http://git.catalyst.net.nz/gitweb?p=samba-cloud-autobuild.git;a=blob;f=parse-email/parse-autobuild-email
[2] https://lists.samba.org/archive/samba-cvs/

Reply | Threaded
Open this post in threaded view
|

Re: sn-devel flapping tests summary since 2017-02-10

Samba - samba-technical mailing list
On Thu, 2017-04-20 at 14:46 +1200, Douglas Bagnall via samba-technical
wrote:

> This is the latest in an occasional series consisting of me running a
> script[1] that counts intermittent failing tests, as indicated by all
> the
> emails sent to the samba-cvs[2] list by the test robot that have
> reached me.
>
> In this case the date range has been selected to miss the masktest
> failures that were discussed here and fixed in mid-February.
>
> The tests that flapped more than once are:
>
> $ ./parse-email/parse-autobuild-email --file-regex sn-devel-144 --
> since
> 2017-02-10 --filter-line=': ([\w.]+)'
> found 54 lines matching '^UNEXPECTED' in 87 files matching 'sn-devel-
> 144'
>   19 samba4.drs.replica_sync.python
Thanks to a suggestion from Garming, I think I have a patch for this.
It is in autobuild now.  It certainly shouldn't hurt, and it could well
help to keep replication a little more controlled while we test this,
particularly when 3 DCs are up, but only two are being controlled.

Thanks,

Andrew Bartlett
--
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team         https://samba.org
Samba Development and Support, Catalyst IT  
https://catalyst.net.nz/services/samba




0001-selftest-Do-not-enable-inbound-replication-during-re.patch (18K) Download Attachment