G'Day All.
I ran some tests overnight as promised. The first thing to say is that we (sadly) need to drop Douglas' visualisation patches. There are some python errors in the error cases which show up only at the end of a full run (because the DB has junk in it) that are not handled. Then I think we need to run tests on less than this full branch. I'll try: - master plus the flapping additions - metze's branch minus Douglas' patches - asn's branch with the flapping additions (but not whoami) We historically have always got into a muddle when we combine everybody's patch into one push, it feels like it would save time but actually takes longer: This is because it assumes that all the patches work, and for example I've put in good, tested code that failed, but should have just failed its own autobuild, not held up yours. For master, I think some builds with just the flapping tests marked would be good, then put that in. Then do the rest by topic, owned by the author. In the medium term, Jamie (one of my new developers at Catalyst) is working to untangle our testsuite inter-dependences. The aim here is to find sets of tests that: - are reliable - do no depend on each other - consume < 4GB of RAM - take less than 1 hour (And then to split these into parallel test environments) At Catalyst, running cloud builds for test is quite normal, often before posting and generally before pushing. But I've noticed that even for me that the closer I get to the release deadline, the less likely I am to wait for a full 5 hour build for the absolute final patch. I'm more likely to do what I did with the talloc patch: trust earlier tests on different code and the newly written tests and aim at autobuild. What I would like to get to is a norm where when posting patches for review, we post them to (say) gitlab by habit, and by the time they are reviewed a clear 'passed/failed' flag is shown so we don't waste time on patches that won't pass. In the meantime I'll run our 5-hour testsuite a few more times in hope of getting the data on what can safely land for 4.8. Thanks, Andrew Bartlett -- Andrew Bartlett http://samba.org/~abartlet/ Authentication Developer, Samba Team http://samba.org Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba |
On Sat, 2018-01-13 at 07:49 +1300, Andrew Bartlett via samba-technical
wrote: > G'Day All. > > I ran some tests overnight as promised. > > The first thing to say is that we (sadly) need to drop Douglas' > visualisation patches. There are some python errors in the error cases > which show up only at the end of a full run (because the DB has junk in > it) that are not handled. Once I started looking at branches, I see a workaround for that has been written. Thanks! > Then I think we need to run tests on less than this full branch. > > I'll try: > - master plus the flapping additions > - metze's branch minus Douglas' patches > - asn's branch with the flapping additions (but not whoami) I am building (x4) in the Catalyst Cloud: 39 sec ago master-and-flapping shortlog | log | tree 11 min ago no-winbind-for-4.8 shortlog | log | tree 13 min ago no-catalyst-for-4.8 shortlog | log | tree 18 min ago catalyst-for-4.8 shortlog | log | tree 35 min ago asn-whoami shortlog | log | tree https://git.samba.org/?p=abartlet/samba.git/.git;a=summary Plus metze's new autobuild tree. I'll post some results later today. Andrew Bartlett > We historically have always got into a muddle when we combine > everybody's patch into one push, it feels like it would save time but > actually takes longer: This is because it assumes that all the patches > work, and for example I've put in good, tested code that failed, but > should have just failed its own autobuild, not held up yours. > > For master, I think some builds with just the flapping tests marked > would be good, then put that in. Then do the rest by topic, owned by > the author. > > In the medium term, Jamie (one of my new developers at Catalyst) is > working to untangle our testsuite inter-dependences. The aim here is > to find sets of tests that: > - are reliable > - do no depend on each other > - consume < 4GB of RAM > - take less than 1 hour > > (And then to split these into parallel test environments) > > At Catalyst, running cloud builds for test is quite normal, often > before posting and generally before pushing. But I've noticed that > even for me that the closer I get to the release deadline, the less > likely I am to wait for a full 5 hour build for the absolute final > patch. I'm more likely to do what I did with the talloc patch: trust > earlier tests on different code and the newly written tests and aim at > autobuild. > > What I would like to get to is a norm where when posting patches for > review, we post them to (say) gitlab by habit, and by the time they are > reviewed a clear 'passed/failed' flag is shown so we don't waste time > on patches that won't pass. > > In the meantime I'll run our 5-hour testsuite a few more times in hope > of getting the data on what can safely land for 4.8. > > Thanks, > > Andrew Bartlett Andrew Bartlett http://samba.org/~abartlet/ Authentication Developer, Samba Team http://samba.org Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba |
In reply to this post by Samba - samba-technical mailing list
Am 12.01.2018 um 19:49 schrieb Andrew Bartlett:
> G'Day All. > > I ran some tests overnight as promised. > > The first thing to say is that we (sadly) need to drop Douglas' > visualisation patches. There are some python errors in the error cases > which show up only at the end of a full run (because the DB has junk in > it) that are not handled. > > Then I think we need to run tests on less than this full branch. > > I'll try: > - master plus the flapping additions > - metze's branch minus Douglas' patches > - asn's branch with the flapping additions (but not whoami) I think these can wait. > We historically have always got into a muddle when we combine > everybody's patch into one push, it feels like it would save time but > actually takes longer: This is because it assumes that all the patches > work, and for example I've put in good, tested code that failed, but > should have just failed its own autobuild, not held up yours. > > For master, I think some builds with just the flapping tests marked > would be good, then put that in. Then do the rest by topic, owned by > the author. > > In the medium term, Jamie (one of my new developers at Catalyst) is > working to untangle our testsuite inter-dependences. The aim here is > to find sets of tests that: > - are reliable > - do no depend on each other > - consume < 4GB of RAM > - take less than 1 hour > > (And then to split these into parallel test environments) > > At Catalyst, running cloud builds for test is quite normal, often > before posting and generally before pushing. But I've noticed that > even for me that the closer I get to the release deadline, the less > likely I am to wait for a full 5 hour build for the absolute final > patch. I'm more likely to do what I did with the talloc patch: trust > earlier tests on different code and the newly written tests and aim at > autobuild. > > What I would like to get to is a norm where when posting patches for > review, we post them to (say) gitlab by habit, and by the time they are > reviewed a clear 'passed/failed' flag is shown so we don't waste time > on patches that won't pass. > In the meantime I'll run our 5-hour testsuite a few more times in hope > of getting the data on what can safely land for 4.8. Please you my latest autobuild branch. It just failed with some really rare flapping tests, e.g. samba.nbt.dgram. We also saw some pam_winbindd failures, while setting up the ad_member env. I'll try a few more times with the whole branch, then I'll start pushing just the first chunks. metze |
On Fri, 2018-01-12 at 20:32 +0100, Stefan Metzmacher wrote:
> Am 12.01.2018 um 19:49 schrieb Andrew Bartlett: > > G'Day All. > > > > I ran some tests overnight as promised. > > > > The first thing to say is that we (sadly) need to drop Douglas' > > visualisation patches. There are some python errors in the error cases > > which show up only at the end of a full run (because the DB has junk in > > it) that are not handled. > > > > Then I think we need to run tests on less than this full branch. > > > > I'll try: > > - master plus the flapping additions > > - metze's branch minus Douglas' patches > > I fixed Douglas' patches and your talloc patches. > > > - asn's branch with the flapping additions (but not whoami) > > I think these can wait. > > > We historically have always got into a muddle when we combine > > everybody's patch into one push, it feels like it would save time but > > actually takes longer: This is because it assumes that all the patches > > work, and for example I've put in good, tested code that failed, but > > should have just failed its own autobuild, not held up yours. > > > > For master, I think some builds with just the flapping tests marked > > would be good, then put that in. Then do the rest by topic, owned by > > the author. > > > > In the medium term, Jamie (one of my new developers at Catalyst) is > > working to untangle our testsuite inter-dependences. The aim here is > > to find sets of tests that: > > - are reliable > > - do no depend on each other > > - consume < 4GB of RAM > > - take less than 1 hour > > > > (And then to split these into parallel test environments) > > > > At Catalyst, running cloud builds for test is quite normal, often > > before posting and generally before pushing. But I've noticed that > > even for me that the closer I get to the release deadline, the less > > likely I am to wait for a full 5 hour build for the absolute final > > patch. I'm more likely to do what I did with the talloc patch: trust > > earlier tests on different code and the newly written tests and aim at > > autobuild. > > > > What I would like to get to is a norm where when posting patches for > > review, we post them to (say) gitlab by habit, and by the time they are > > reviewed a clear 'passed/failed' flag is shown so we don't waste time > > on patches that won't pass. > > It would be nice to have that. > > > In the meantime I'll run our 5-hour testsuite a few more times in hope > > of getting the data on what can safely land for 4.8. > > Please you my latest autobuild branch. 3 of your autobuild and 2 of 'no-catalyst-for-4.8' of these just failed with: [1533(9721)/2234 at 2h13m38s] samba4.nbt.dgram(ad_dc_ntvfs) netlogon reply from 127.0.0.33:138 netlogon reply from 127.0.0.33:138 netlogon reply from 127.0.0.33:138 netlogon reply from 127.0.0.33:138 UNEXPECTED(failure): samba4.nbt.dgram.netlogon2(ad_dc_ntvfs) REASON: Exception: Exception: ../source4/torture/nbt/dgram.c:396: response->data.samlogon.data.nt5_ex.command was 21 (0x15), expected 19 (0x13): Got incorrect netlogon response command or [1533(9721)/2234 at 2h12m36s] samba4.nbt.dgram(ad_dc_ntvfs) smbtorture 4.9.0pre1-DEVELOPERBUILD Using seed 1515794453 UNEXPECTED(failure): samba4.nbt.dgram.netlogon(ad_dc_ntvfs) REASON: Exception: Exception: ../source4/torture/nbt/dgram.c:148: Expression `response != ((void *)0)' failed: Failed to receive a netlogon reply packet > It just failed with some really rare flapping tests, e.g. > samba.nbt.dgram. We also saw some pam_winbindd failures, > while setting up the ad_member env. Two of the tests I did of Andreas's patch set (with the flapping patches on top), and one on the catalyst-for-4-8 branch failed with: [64(1161)/2230 at 49m50s] samba.tests.pam_winbind(local)(ad_member) ERROR: Testsuite[samba.tests.pam_winbind(local)(ad_member)] REASON: unable to set up environment ad_member - exiting I think we should revert the change to make ad_member use ad_dc. I'll test master with such a revert (and the flappy tests changes). > I'll try a few more times with the whole branch, then I'll > start pushing just the first chunks. I'll put the autobuild results all in this link so you don't have to guess from my summary, you can check the full details: https://seafile.catalyst.net.nz/d/92adffd354044ffaa3c4/ I remain at your disposal to work on builds for this (in between spending the sunny weekend with the family). Andrew Bartlett -- Andrew Bartlett http://samba.org/~abartlet/ Authentication Developer, Samba Team http://samba.org Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba |
Am 12.01.2018 um 23:25 schrieb Andrew Bartlett via samba-technical:
> On Fri, 2018-01-12 at 20:32 +0100, Stefan Metzmacher wrote: >> Am 12.01.2018 um 19:49 schrieb Andrew Bartlett: >>> G'Day All. >>> >>> I ran some tests overnight as promised. >>> >>> The first thing to say is that we (sadly) need to drop Douglas' >>> visualisation patches. There are some python errors in the error cases >>> which show up only at the end of a full run (because the DB has junk in >>> it) that are not handled. >>> >>> Then I think we need to run tests on less than this full branch. >>> >>> I'll try: >>> - master plus the flapping additions >>> - metze's branch minus Douglas' patches >> >> I fixed Douglas' patches and your talloc patches. >> >>> - asn's branch with the flapping additions (but not whoami) >> >> I think these can wait. >> >>> We historically have always got into a muddle when we combine >>> everybody's patch into one push, it feels like it would save time but >>> actually takes longer: This is because it assumes that all the patches >>> work, and for example I've put in good, tested code that failed, but >>> should have just failed its own autobuild, not held up yours. >>> >>> For master, I think some builds with just the flapping tests marked >>> would be good, then put that in. Then do the rest by topic, owned by >>> the author. >>> >>> In the medium term, Jamie (one of my new developers at Catalyst) is >>> working to untangle our testsuite inter-dependences. The aim here is >>> to find sets of tests that: >>> - are reliable >>> - do no depend on each other >>> - consume < 4GB of RAM >>> - take less than 1 hour >>> >>> (And then to split these into parallel test environments) >>> >>> At Catalyst, running cloud builds for test is quite normal, often >>> before posting and generally before pushing. But I've noticed that >>> even for me that the closer I get to the release deadline, the less >>> likely I am to wait for a full 5 hour build for the absolute final >>> patch. I'm more likely to do what I did with the talloc patch: trust >>> earlier tests on different code and the newly written tests and aim at >>> autobuild. >>> >>> What I would like to get to is a norm where when posting patches for >>> review, we post them to (say) gitlab by habit, and by the time they are >>> reviewed a clear 'passed/failed' flag is shown so we don't waste time >>> on patches that won't pass. >> >> It would be nice to have that. >> >>> In the meantime I'll run our 5-hour testsuite a few more times in hope >>> of getting the data on what can safely land for 4.8. >> >> Please you my latest autobuild branch. > > 3 of your autobuild and 2 of 'no-catalyst-for-4.8' of these just failed > with: > > [1533(9721)/2234 at 2h13m38s] samba4.nbt.dgram(ad_dc_ntvfs) > netlogon reply from 127.0.0.33:138 > netlogon reply from 127.0.0.33:138 > netlogon reply from 127.0.0.33:138 > netlogon reply from 127.0.0.33:138 > UNEXPECTED(failure): samba4.nbt.dgram.netlogon2(ad_dc_ntvfs) > REASON: Exception: Exception: ../source4/torture/nbt/dgram.c:396: > response->data.samlogon.data.nt5_ex.command was 21 (0x15), expected 19 > (0x13): Got incorrect netlogon response command > > or > > [1533(9721)/2234 at 2h12m36s] samba4.nbt.dgram(ad_dc_ntvfs) > smbtorture 4.9.0pre1-DEVELOPERBUILD > Using seed 1515794453 > UNEXPECTED(failure): samba4.nbt.dgram.netlogon(ad_dc_ntvfs) > REASON: Exception: Exception: ../source4/torture/nbt/dgram.c:148: > Expression `response != ((void *)0)' failed: Failed to receive a > netlogon reply packet >> It just failed with some really rare flapping tests, e.g. >> samba.nbt.dgram. We also saw some pam_winbindd failures, >> while setting up the ad_member env. > > Two of the tests I did of Andreas's patch set (with the flapping > patches on top), and one on the catalyst-for-4-8 branch failed with: > > [64(1161)/2230 at 49m50s] samba.tests.pam_winbind(local)(ad_member) > ERROR: Testsuite[samba.tests.pam_winbind(local)(ad_member)] > REASON: unable to set up environment ad_member - exiting > > I think we should revert the change to make ad_member use ad_dc. I'll > test master with such a revert (and the flappy tests changes). >> I'll try a few more times with the whole branch, then I'll >> start pushing just the first chunks. > > I'll put the autobuild results all in this link so you don't have to > guess from my summary, you can check the full details: > > https://seafile.catalyst.net.nz/d/92adffd354044ffaa3c4/ > > I remain at your disposal to work on builds for this (in between > spending the sunny weekend with the family). metze |
On Fri, 2018-01-12 at 23:33 +0100, Stefan Metzmacher wrote:
> Am 12.01.2018 um 23:25 schrieb Andrew Bartlett via samba-technical: > > > > 3 of your autobuild and 2 of 'no-catalyst-for-4.8' of these just failed > > with: > > > > [1533(9721)/2234 at 2h13m38s] samba4.nbt.dgram(ad_dc_ntvfs) > > netlogon reply from 127.0.0.33:138 > > netlogon reply from 127.0.0.33:138 > > netlogon reply from 127.0.0.33:138 > > netlogon reply from 127.0.0.33:138 > > UNEXPECTED(failure): samba4.nbt.dgram.netlogon2(ad_dc_ntvfs) > > REASON: Exception: Exception: ../source4/torture/nbt/dgram.c:396: > > response->data.samlogon.data.nt5_ex.command was 21 (0x15), expected 19 > > (0x13): Got incorrect netlogon response command > > > > or > > > > [1533(9721)/2234 at 2h12m36s] samba4.nbt.dgram(ad_dc_ntvfs) > > smbtorture 4.9.0pre1-DEVELOPERBUILD > > Using seed 1515794453 > > UNEXPECTED(failure): samba4.nbt.dgram.netlogon(ad_dc_ntvfs) > > REASON: Exception: Exception: ../source4/torture/nbt/dgram.c:148: > > Expression `response != ((void *)0)' failed: Failed to receive a > > netlogon reply packet > > We also got something like this 2 times. I just saw it 3 more times on no-winbind-for-4.8. Those I have also uploaded. > > > > Two of the tests I did of Andreas's patch set (with the flapping > > patches on top), and one on the catalyst-for-4-8 branch failed with: > > > > [64(1161)/2230 at 49m50s] samba.tests.pam_winbind(local)(ad_member) > > ERROR: Testsuite[samba.tests.pam_winbind(local)(ad_member)] > > REASON: unable to set up environment ad_member - exiting > > > > I think we should revert the change to make ad_member use ad_dc. I'll > > test master with such a revert (and the flappy tests changes). > > We also got this a few times. > > > I'll put the autobuild results all in this link so you don't have to > > guess from my summary, you can check the full details: > > > > https://seafile.catalyst.net.nz/d/92adffd354044ffaa3c4/ I've uploaded the latest results there. Sorry for the mbox, but I figure you can unpack it. > > I remain at your disposal to work on builds for this (in between > > spending the sunny weekend with the family). > > Ok, I'm now trying just the talloc/tevent/tdb + flapping changes. Sounds like a plan. Thanks! Andrew Bartlett -- Andrew Bartlett http://samba.org/~abartlet/ Authentication Developer, Samba Team http://samba.org Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba |
Free forum by Nabble | Edit this page |