[PATCH] Add more tests for dbcheck

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

[PATCH] Add more tests for dbcheck

Samba - samba-technical mailing list
This patch adds some more tests to our dbcheck test suite to try and
cover some important cases and the recent 'dbcheck: Allow removal of
one-way links to missing objects' patch is tested.

It also tries to make some of the errors less scary and into warnings,
if they can happen in normal operation.

On this basis I will build a testsuite for Andrej Gessel's patch:

[PATCH] Fix: Failed to remove backlink of memberOf when deleting
dsdb_module_search_dn: did not find base dn

Please review!

Thanks,

Andrew Bartlett
--
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team         https://samba.org
Samba Development and Support, Catalyst IT  
https://catalyst.net.nz/services/samba




dbcheck-fix-and-test.patch (58K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Add more tests for dbcheck

Samba - samba-technical mailing list
Am 30.10.2017 um 04:05 schrieb Andrew Bartlett via samba-technical:

> This patch adds some more tests to our dbcheck test suite to try and
> cover some important cases and the recent 'dbcheck: Allow removal of
> one-way links to missing objects' patch is tested.
>
> It also tries to make some of the errors less scary and into warnings,
> if they can happen in normal operation.
>
> On this basis I will build a testsuite for Andrej Gessel's patch:
>
> [PATCH] Fix: Failed to remove backlink of memberOf when deleting
> dsdb_module_search_dn: did not find base dn
>
> Please review!
I'll try to have a look at the end of the week.

I'll try to combine it with my dbcheck patches to fix
the consequences of https://bugzilla.samba.org/show_bug.cgi?id=13095

See
https://git.samba.org/?p=metze/samba/wip.git;a=shortlog;h=refs/heads/v4-6-duplicate-links
and
https://git.samba.org/?p=metze/samba/wip.git;a=shortlog;h=refs/heads/v4-7-duplicate-links

metze



signature.asc (853 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Add more tests for dbcheck

Samba - samba-technical mailing list
On Mon, 2017-10-30 at 13:53 +0100, Stefan Metzmacher via samba-
technical wrote:

> Am 30.10.2017 um 04:05 schrieb Andrew Bartlett via samba-technical:
> > This patch adds some more tests to our dbcheck test suite to try and
> > cover some important cases and the recent 'dbcheck: Allow removal of
> > one-way links to missing objects' patch is tested.
> >
> > It also tries to make some of the errors less scary and into warnings,
> > if they can happen in normal operation.
> >
> > On this basis I will build a testsuite for Andrej Gessel's patch:
> >
> > [PATCH] Fix: Failed to remove backlink of memberOf when deleting
> > dsdb_module_search_dn: did not find base dn
> >
> > Please review!
>
> I'll try to have a look at the end of the week.
>
> I'll try to combine it with my dbcheck patches to fix
> the consequences of https://bugzilla.samba.org/show_bug.cgi?id=13095
>
> See
> https://git.samba.org/?p=metze/samba/wip.git;a=shortlog;h=refs/heads/v4-6-duplicate-links
> and
> https://git.samba.org/?p=metze/samba/wip.git;a=shortlog;h=refs/heads/v4-7-duplicate-links

Thanks for looking into this!  I've looked over the overall diff and it
certainly looks reasonable and shouldn't conflict too badly (except on
the expected output stuff, needing re-generation).

Andrew Bartlett
--
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba


Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] Add more tests for dbcheck

Samba - samba-technical mailing list
In reply to this post by Samba - samba-technical mailing list
On Mon, 2017-10-30 at 13:53 +0100, Stefan Metzmacher wrote:

> Am 30.10.2017 um 04:05 schrieb Andrew Bartlett via samba-technical:
> > This patch adds some more tests to our dbcheck test suite to try
> > and
> > cover some important cases and the recent 'dbcheck: Allow removal
> > of
> > one-way links to missing objects' patch is tested.
> >
> > It also tries to make some of the errors less scary and into
> > warnings,
> > if they can happen in normal operation.
> >
> > On this basis I will build a testsuite for Andrej Gessel's patch:
> >
> > [PATCH] Fix: Failed to remove backlink of memberOf when deleting
> > dsdb_module_search_dn: did not find base dn
> >
> > Please review!
>
> I'll try to have a look at the end of the week.
>
> I'll try to combine it with my dbcheck patches to fix
> the consequences of https://bugzilla.samba.org/show_bug.cgi?id=13095
>
> See
> https://git.samba.org/?p=metze/samba/wip.git;a=shortlog;h=refs/heads/
> v4-6-duplicate-links
> and
> https://git.samba.org/?p=metze/samba/wip.git;a=shortlog;h=refs/heads/
> v4-7-duplicate-links
I've tried to forward-port that to master, which was a mission, but I
think I've done it.  More importantly I've improved the tests, and
found it handles one issue I couldn't get the old dbcheck to find,
which is a backlink to a deleted object.

The tests have been changed to use administrator and enterprise admins
to keep stable GUIDs, and now reliably pass.

There is still more work to do to squash this in to a for-master patch
series, and I may have totally mucked up the rebase, so please look
carefully.

I wouldn't normally get into your WIP tree, but it seemed more helpful
than continuing parallel development.

Thanks for all your work here, and I hope this is useful!

http://git.catalyst.net.nz/gitweb?p=samba.git;a=shortlog;h=refs/heads/abartlet-dbcheck-links-2017-11

Andrew Bartlett
--
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team         https://samba.org
Samba Development and Support, Catalyst IT  
https://catalyst.net.nz/services/samba




signature.asc (879 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

[PATCH] run-time and dbcheck time wrong backlink handling (was: Re: [PATCH] Add more tests for dbcheck)

Samba - samba-technical mailing list
On Tue, 2017-10-31 at 13:41 +1300, Andrew Bartlett via samba-technical
wrote:

> On Mon, 2017-10-30 at 13:53 +0100, Stefan Metzmacher wrote:
> >
> > I'll try to have a look at the end of the week.
> >
> > I'll try to combine it with my dbcheck patches to fix
> > the consequences of https://bugzilla.samba.org/show_bug.cgi?id=13095
> >
> > See
> > https://git.samba.org/?p=metze/samba/wip.git;a=shortlog;h=refs/heads/
> > v4-6-duplicate-links
> > and
> > https://git.samba.org/?p=metze/samba/wip.git;a=shortlog;h=refs/heads/
> > v4-7-duplicate-links
>
> I've tried to forward-port that to master, which was a mission, but I
> think I've done it.  More importantly I've improved the tests, and
> found it handles one issue I couldn't get the old dbcheck to find,
> which is a backlink to a deleted object.
>
> The tests have been changed to use administrator and enterprise admins
> to keep stable GUIDs, and now reliably pass.
>
> There is still more work to do to squash this in to a for-master patch
> series, and I may have totally mucked up the rebase, so please look
> carefully.
>
> I wouldn't normally get into your WIP tree, but it seemed more helpful
> than continuing parallel development.
>
> Thanks for all your work here, and I hope this is useful!
>
> http://git.catalyst.net.nz/gitweb?p=samba.git;a=shortlog;h=refs/heads/abartlet-dbcheck-links-2017-11
G'Day,

I've updated that branch, and now have patches covering runtime as well
as dbcheck-time behaviour.  In particular, I can now cope with deleting
objects that had backlinks that we can assert are faulty (object
tomebstoned or missing).

I was hoping to get these fixes in to 4.7.1 but I think I'm too late
for that, but if you can help get these in to master that would be
great.  I've tried un-tangling it again from your code but that just
looks like being more difficult all-round.

If you could take a look at what I've done and finish tidying up your
patches that would be great.  If you could fix the existing failure in
dbcheck-links.dbcheck_dangling_multi_valued_clean that would be even
better!

The output is below, hidden normally because it is in knownfail.

Thanks!

Andrew Bartlett

It was:

failure: dangling_multi_valued_dbcheck [
Checking 230 objects
ERROR: Can't fix missing multi-valued backlinks on CN=dangling-multi3,CN=Users,DC=release-4-5-0-pre1,DC=samba,DC=corp
ERROR: Can't fix missing multi-valued backlinks on CN=dangling-multi3,CN=Users,DC=release-4-5-0-pre1,DC=samba,DC=corp
ERROR: Can't fix missing multi-valued backlinks on CN=dangling-multi3,CN=Users,DC=release-4-5-0-pre1,DC=samba,DC=corp
ERROR: Can't fix missing multi-valued backlinks on CN=dangling-multi3,CN=Users,DC=release-4-5-0-pre1,DC=samba,DC=corp
ERROR: Can't fix missing multi-valued backlinks on CN=dangling-multi2,CN=Users,DC=release-4-5-0-pre1,DC=samba,DC=corp
ERROR: Can't fix missing multi-valued backlinks on CN=dangling-multi2,CN=Users,DC=release-4-5-0-pre1,DC=samba,DC=corp
ERROR: Can't fix missing multi-valued backlinks on CN=dangling-multi2,CN=Users,DC=release-4-5-0-pre1,DC=samba,DC=corp
ERROR: Can't fix missing multi-valued backlinks on CN=dangling-multi2,CN=Users,DC=release-4-5-0-pre1,DC=samba,DC=corp
Please use --fix to fix these errors
Checked 230 objects (8 errors)
]

And is now more verbose:

failure: dbcheck_dangling_multi_valued_clean [
Checking 234 objects
ERROR: Link (forward) mismatch for 'msDS-RevealedUsers' (4) on 'CN=DOUGLASB-DESKTO,OU=Domain Controllers,DC=release-4-5-0-pre1,DC=samba,DC=corp' to 'msDS-RevealedDSAs' (0) on 'CN=dangling-multi2,CN=Users,DC=release-4-5-0-pre1,DC=samba,DC=corp'
ERROR: Can't fix missing multi-valued backlinks on CN=dangling-multi2,CN=Users,DC=release-4-5-0-pre1,DC=samba,DC=corp
ERROR: Link (forward) mismatch for 'msDS-RevealedUsers' (4) on 'CN=DOUGLASB-DESKTO,OU=Domain Controllers,DC=release-4-5-0-pre1,DC=samba,DC=corp' to 'msDS-RevealedDSAs' (0) on 'CN=dangling-multi2,CN=Users,DC=release-4-5-0-pre1,DC=samba,DC=corp'
ERROR: Can't fix missing multi-valued backlinks on CN=dangling-multi2,CN=Users,DC=release-4-5-0-pre1,DC=samba,DC=corp
ERROR: Link (forward) mismatch for 'msDS-RevealedUsers' (4) on 'CN=DOUGLASB-DESKTO,OU=Domain Controllers,DC=release-4-5-0-pre1,DC=samba,DC=corp' to 'msDS-RevealedDSAs' (0) on 'CN=dangling-multi2,CN=Users,DC=release-4-5-0-pre1,DC=samba,DC=corp'
ERROR: Can't fix missing multi-valued backlinks on CN=dangling-multi2,CN=Users,DC=release-4-5-0-pre1,DC=samba,DC=corp
ERROR: Link (forward) mismatch for 'msDS-RevealedUsers' (4) on 'CN=DOUGLASB-DESKTO,OU=Domain Controllers,DC=release-4-5-0-pre1,DC=samba,DC=corp' to 'msDS-RevealedDSAs' (0) on 'CN=dangling-multi2,CN=Users,DC=release-4-5-0-pre1,DC=samba,DC=corp'
ERROR: Can't fix missing multi-valued backlinks on CN=dangling-multi2,CN=Users,DC=release-4-5-0-pre1,DC=samba,DC=corp
ERROR: Link (forward) mismatch for 'msDS-RevealedUsers' (4) on 'CN=DOUGLASB-DESKTO,OU=Domain Controllers,DC=release-4-5-0-pre1,DC=samba,DC=corp' to 'msDS-RevealedDSAs' (2) on 'CN=dangling-multi3,CN=Users,DC=release-4-5-0-pre1,DC=samba,DC=corp'
ERROR: Can't fix missing multi-valued backlinks on CN=dangling-multi3,CN=Users,DC=release-4-5-0-pre1,DC=samba,DC=corp
ERROR: Link (forward) mismatch for 'msDS-RevealedUsers' (4) on 'CN=DOUGLASB-DESKTO,OU=Domain Controllers,DC=release-4-5-0-pre1,DC=samba,DC=corp' to 'msDS-RevealedDSAs' (2) on 'CN=dangling-multi3,CN=Users,DC=release-4-5-0-pre1,DC=samba,DC=corp'
ERROR: Can't fix missing multi-valued backlinks on CN=dangling-multi3,CN=Users,DC=release-4-5-0-pre1,DC=samba,DC=corp
ERROR: Link (forward) mismatch for 'msDS-RevealedUsers' (4) on 'CN=DOUGLASB-DESKTO,OU=Domain Controllers,DC=release-4-5-0-pre1,DC=samba,DC=corp' to 'msDS-RevealedDSAs' (2) on 'CN=dangling-multi3,CN=Users,DC=release-4-5-0-pre1,DC=samba,DC=corp'
ERROR: Can't fix missing multi-valued backlinks on CN=dangling-multi3,CN=Users,DC=release-4-5-0-pre1,DC=samba,DC=corp
ERROR: Link (forward) mismatch for 'msDS-RevealedUsers' (4) on 'CN=DOUGLASB-DESKTO,OU=Domain Controllers,DC=release-4-5-0-pre1,DC=samba,DC=corp' to 'msDS-RevealedDSAs' (2) on 'CN=dangling-multi3,CN=Users,DC=release-4-5-0-pre1,DC=samba,DC=corp'
ERROR: Can't fix missing multi-valued backlinks on CN=dangling-multi3,CN=Users,DC=release-4-5-0-pre1,DC=samba,DC=corp
WARNING: Link (back) mismatch for 'msDS-RevealedDSAs' (2) on 'CN=dangling-multi3,CN=Users,DC=release-4-5-0-pre1,DC=samba,DC=corp' to 'msDS-RevealedUsers' (4) on 'CN=DOUGLASB-DESKTO,OU=Domain Controllers,DC=release-4-5-0-pre1,DC=samba,DC=corp'
WARNING: Link (back) mismatch for 'msDS-RevealedDSAs' (2) on 'CN=dangling-multi3,CN=Users,DC=release-4-5-0-pre1,DC=samba,DC=corp' to 'msDS-RevealedUsers' (4) on 'CN=DOUGLASB-DESKTO,OU=Domain Controllers,DC=release-4-5-0-pre1,DC=samba,DC=corp'
Please use --fix to fix these errors
Checked 234 objects (8 errors)
--
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team         https://samba.org
Samba Development and Support, Catalyst IT  
https://catalyst.net.nz/services/samba




signature.asc (879 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] run-time and dbcheck time wrong backlink handling

Samba - samba-technical mailing list
Hi Andrew,

>>> I'll try to combine it with my dbcheck patches to fix
>>> the consequences of https://bugzilla.samba.org/show_bug.cgi?id=13095
>>>
>>> See
>>> https://git.samba.org/?p=metze/samba/wip.git;a=shortlog;h=refs/heads/
>>> v4-6-duplicate-links
>>> and
>>> https://git.samba.org/?p=metze/samba/wip.git;a=shortlog;h=refs/heads/
>>> v4-7-duplicate-links
>>
>> I've tried to forward-port that to master, which was a mission, but I
>> think I've done it.  More importantly I've improved the tests, and
>> found it handles one issue I couldn't get the old dbcheck to find,
>> which is a backlink to a deleted object.
>>
>> The tests have been changed to use administrator and enterprise admins
>> to keep stable GUIDs, and now reliably pass.
>>
>> There is still more work to do to squash this in to a for-master patch
>> series, and I may have totally mucked up the rebase, so please look
>> carefully.
>>
>> I wouldn't normally get into your WIP tree, but it seemed more helpful
>> than continuing parallel development.
>>
>> Thanks for all your work here, and I hope this is useful!
>>
>> http://git.catalyst.net.nz/gitweb?p=samba.git;a=shortlog;h=refs/heads/abartlet-dbcheck-links-2017-11
>
> G'Day,
>
> I've updated that branch, and now have patches covering runtime as well
> as dbcheck-time behaviour.  In particular, I can now cope with deleting
> objects that had backlinks that we can assert are faulty (object
> tomebstoned or missing).
>
> I was hoping to get these fixes in to 4.7.1 but I think I'm too late
> for that, but if you can help get these in to master that would be
> great.  I've tried un-tangling it again from your code but that just
> looks like being more difficult all-round.
>
> If you could take a look at what I've done and finish tidying up your
> patches that would be great.  If you could fix the existing failure in
> dbcheck-links.dbcheck_dangling_multi_valued_clean that would be even
> better!
>
> The output is below, hidden normally because it is in knownfail.
Thanks! I cleaned up my patches and squashed a few things.

See
https://git.samba.org/?p=metze/samba/wip.git;a=shortlog;h=refs/heads/master4-duplicate-links

I left the last 4 unsquashed for further review from you.

We also need some more BUG: lines here and there.

https://git.samba.org/?p=metze/samba/wip.git;a=shortlog;h=refs/heads/v4-7-duplicate-links
contains backports

I'm currently running private autobuilds with both branches.

metze


signature.asc (853 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] run-time and dbcheck time wrong backlink handling

Samba - samba-technical mailing list
On Thu, 2017-11-23 at 15:44 +0100, Stefan Metzmacher wrote:

> Hi Andrew,
>
> > > > I'll try to combine it with my dbcheck patches to fix
> > > > the consequences of https://bugzilla.samba.org/show_bug.cgi?id=13095
> > > >
> > > > See
> > > > https://git.samba.org/?p=metze/samba/wip.git;a=shortlog;h=refs/heads/
> > > > v4-6-duplicate-links
> > > > and
> > > > https://git.samba.org/?p=metze/samba/wip.git;a=shortlog;h=refs/heads/
> > > > v4-7-duplicate-links
> > >
> > > I've tried to forward-port that to master, which was a mission, but I
> > > think I've done it.  More importantly I've improved the tests, and
> > > found it handles one issue I couldn't get the old dbcheck to find,
> > > which is a backlink to a deleted object.
> > >
> > > The tests have been changed to use administrator and enterprise admins
> > > to keep stable GUIDs, and now reliably pass.
> > >
> > > There is still more work to do to squash this in to a for-master patch
> > > series, and I may have totally mucked up the rebase, so please look
> > > carefully.
> > >
> > > I wouldn't normally get into your WIP tree, but it seemed more helpful
> > > than continuing parallel development.
> > >
> > > Thanks for all your work here, and I hope this is useful!
> > >
> > > http://git.catalyst.net.nz/gitweb?p=samba.git;a=shortlog;h=refs/heads/abartlet-dbcheck-links-2017-11
> >
> > G'Day,
> >
> > I've updated that branch, and now have patches covering runtime as well
> > as dbcheck-time behaviour.  In particular, I can now cope with deleting
> > objects that had backlinks that we can assert are faulty (object
> > tomebstoned or missing).
> >
> > I was hoping to get these fixes in to 4.7.1 but I think I'm too late
> > for that, but if you can help get these in to master that would be
> > great.  I've tried un-tangling it again from your code but that just
> > looks like being more difficult all-round.
> >
> > If you could take a look at what I've done and finish tidying up your
> > patches that would be great.  If you could fix the existing failure in
> > dbcheck-links.dbcheck_dangling_multi_valued_clean that would be even
> > better!
> >
> > The output is below, hidden normally because it is in knownfail.
>
> Thanks! I cleaned up my patches and squashed a few things.
>
> See
> https://git.samba.org/?p=metze/samba/wip.git;a=shortlog;h=refs/heads/master4-duplicate-links
>
> I left the last 4 unsquashed for further review from you.
>
> We also need some more BUG: lines here and there.
>
> https://git.samba.org/?p=metze/samba/wip.git;a=shortlog;h=refs/heads/v4-7-duplicate-links
> contains backports
>
> I'm currently running private autobuilds with both branches.
Thanks.  I've squashed it, added the sign-off from Andrej and reviewed
it.  It is in an autobuild here as well.

I've also looked over the diff compared with the branch I had
previously, and the changes make sense.

I think this is ready to go, so can you or I push it?

http://git.catalyst.net.nz/gitweb?p=samba.git;a=shortlog;h=refs/heads/metze-master4-duplicate-links

Thanks,

Andrew Bartlett
--
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team         https://samba.org
Samba Development and Support, Catalyst IT  
https://catalyst.net.nz/services/samba




dbcheck-fix-and-test.patch.txt (175K) Download Attachment
signature.asc (879 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] run-time and dbcheck time wrong backlink handling

Samba - samba-technical mailing list
On Fri, 2017-11-24 at 11:48 +1300, Andrew Bartlett via samba-technical
wrote:

>
> Thanks.  I've squashed it, added the sign-off from Andrej and reviewed
> it.  It is in an autobuild here as well.
>
> I've also looked over the diff compared with the branch I had
> previously, and the changes make sense.
>
> I think this is ready to go, so can you or I push it?
>
> http://git.catalyst.net.nz/gitweb?p=samba.git;a=shortlog;h=refs/heads/metze-master4-duplicate-links

Thanks for pushing it, I see it just landed!

Andrew Bartlett
--
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba