NT_STATUS_FILE_IS_A_DIRECTORY in nosymlinks test

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

NT_STATUS_FILE_IS_A_DIRECTORY in nosymlinks test

Samba - samba-technical mailing list
G'Day Andreas,

I've been trying to make our test environments run in parallel, but
when I enable that on the Catalyst Cloud, using this branch:

https://git.samba.org/?p=abartlet/samba.git/.git;a=shortlog;h=refs/heads/parallel-test

I (about 50% of the time) get failures in
test_nosymlinks() in test_smbclient_s3.sh
UNEXPECTED(failure): samba3.blackbox.smbclient_s3.SMB3.plain
(ad_member) member creds.follow symlinks = no(ad_member)

(sorry for the garbled output, something very odd is going on in the
output stream with the escaping)

REASON: Exception: Exception: CLI_FORCE_INTERACTIVE=yes /home/ubuntu/autobuild/b20147/samba/bin/smbclient -ULOCALADMEMBER/ubuntu%loCalMemberPass //LOCALADMEMBER/nosymlinks -I 127.0.0.29 --configfile=/home/ubuntu/autobuild/b20147/samba/bin/ab/client/client.conf -mSMB3
NT_STATUS_ACCESS_DENIED opening remote file     est\source
CLI_FORCE_INTERACTIVE=yes /home/ubuntu/autobuild/b20147/samba/bin/smbclient -ULOCALADMEMBER/ubuntu%loCalMemberPass //LOCALADMEMBER/nosymlinks -I 127.0.0.29 --configfile=/home/ubuntu/autobuild/b20147/samba/bin/ab/client/client.conf -mSMB3
CLI_FORCE_INTERACTIVE=yes /home/ubuntu/autobuild/b20147/samba/bin/smbclient -ULOCALADMEMBER/ubuntu%loCalMemberPass //LOCALADMEMBER/nosymlinks -I 127.0.0.29 --configfile=/home/ubuntu/autobuild/b20147/samba/bin/ab/client/client.conf -mSMB3
NT_STATUS_FILE_IS_A_DIRECTORY opening remote file       est ooar       estfile
Try "help" do get a list of possible commands.
smb: \> cd test ooar
smb:    est ooar\> ls
  .                                   D        0  Fri Jun 30 03:27:08 2017
  ..                                  D        0  Fri Jun 30 03:27:08 2017
  testfile                            D        0  Fri Jun 30 03:27:08 2017

                10287416 blocks of size 1024. 3260044 blocks available
smb:    est ooar\> get testfile -
NT_STATUS_FILE_IS_A_DIRECTORY opening remote file       est ooar       estfile
smb:    est ooar\> quit
failed - NT_STATUS_XXXX doing cd fooar; get testfile on 
osymlinks

It would be great to make some progress on this, because in the
successful runs, I see autobuild times drop by 60mins (288 -> 218 on an
identical VM)!

I had assumed this was due to re-use of the dev/inode, but I tried the
attached and it didn't help.  (This happens when files are created or
deleted on the client side, not the server side).

Another idea would be to wipe the xattr.tdb file between each test.

My final idea is to have as much of the deltree and mkdir etc be done
over SMB, so they will get an entry in the xattr.tdb, but I've not had
time to do this yet.

However, as you have looked into this before, perhaps you have other
suggestions.  If you could help out, the potential gains are
impressive.

Thanks,

Andrew Bartlett

--
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team         https://samba.org
Samba Development and Support, Catalyst IT  
https://catalyst.net.nz/services/samba




0001-selftest-Define-xattr_tdb-file-per-sub-directory.patch (8K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: NT_STATUS_FILE_IS_A_DIRECTORY in nosymlinks test

Samba - samba-technical mailing list
On Friday, 30 June 2017 07:03:17 CEST Andrew Bartlett via samba-technical
wrote:
> G'Day Andreas,

Hi Andrew,
 

> I've been trying to make our test environments run in parallel, but
> when I enable that on the Catalyst Cloud, using this branch:
>
> https://git.samba.org/?p=abartlet/samba.git/.git;a=shortlog;h=refs/heads/par
> allel-test
>
> I (about 50% of the time) get failures in
> test_nosymlinks() in test_smbclient_s3.sh
> UNEXPECTED(failure): samba3.blackbox.smbclient_s3.SMB3.plain
> (ad_member) member creds.follow symlinks = no(ad_member)
>
> (sorry for the garbled output, something very odd is going on in the
> output stream with the escaping)

I've seen the same and tried to find the issue but couldn't nail it down.

> REASON: Exception: Exception: CLI_FORCE_INTERACTIVE=yes
> /home/ubuntu/autobuild/b20147/samba/bin/smbclient
> -ULOCALADMEMBER/ubuntu%loCalMemberPass //LOCALADMEMBER/nosymlinks -I
> 127.0.0.29
> --configfile=/home/ubuntu/autobuild/b20147/samba/bin/ab/client/client.conf
> -mSMB3 NT_STATUS_ACCESS_DENIED opening remote file     est\source
> CLI_FORCE_INTERACTIVE=yes /home/ubuntu/autobuild/b20147/samba/bin/smbclient
> -ULOCALADMEMBER/ubuntu%loCalMemberPass //LOCALADMEMBER/nosymlinks -I
> 127.0.0.29
> --configfile=/home/ubuntu/autobuild/b20147/samba/bin/ab/client/client.conf
> -mSMB3 CLI_FORCE_INTERACTIVE=yes
> /home/ubuntu/autobuild/b20147/samba/bin/smbclient
> -ULOCALADMEMBER/ubuntu%loCalMemberPass //LOCALADMEMBER/nosymlinks -I
> 127.0.0.29
> --configfile=/home/ubuntu/autobuild/b20147/samba/bin/ab/client/client.conf
> -mSMB3 NT_STATUS_FILE_IS_A_DIRECTORY opening remote
> file       est ooar       estfile Try "help" do get a list of possible
> commands.
> smb: \> cd test ooar
> smb:    est ooar\> ls
>   .                                   D        0  Fri Jun 30 03:27:08 2017
>   ..                                  D        0  Fri Jun 30 03:27:08 2017
>   testfile                            D        0  Fri Jun 30 03:27:08 2017
>
>                 10287416 blocks of size 1024. 3260044 blocks available
> smb:    est ooar\> get testfile -
> NT_STATUS_FILE_IS_A_DIRECTORY opening remote
> file       est ooar       estfile smb:    est ooar\> quit
> failed - NT_STATUS_XXXX doing cd fooar; get testfile on
> osymlinks
>
> It would be great to make some progress on this, because in the
> successful runs, I see autobuild times drop by 60mins (288 -> 218 on an
> identical VM)!
>
> I had assumed this was due to re-use of the dev/inode, but I tried the
> attached and it didn't help.  (This happens when files are created or
> deleted on the client side, not the server side).

I've sent a mail to Jeremy who wrote the code and test for the CVE regression,
but he asked for a reproducer. I can only reproduce it if I run the test 20
times in a loop then normally it happens once.

> Another idea would be to wipe the xattr.tdb file between each test.
>
> My final idea is to have as much of the deltree and mkdir etc be done
> over SMB, so they will get an entry in the xattr.tdb, but I've not had
> time to do this yet.

Hmm. Jeremy any ideas?




        Andreas


P.S. Metze and I started to improve selftest. However I couldn't get the
following patch working yet:

https://git.samba.org/?p=asn/samba.git;a=shortlog;h=refs/heads/master-selftest-ok

Some tests are failing but we couldn't nail it down to why and where.


--
Andreas Schneider                   GPG-ID: CC014E3D
Samba Team                             [hidden email]
www.samba.org

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: NT_STATUS_FILE_IS_A_DIRECTORY in nosymlinks test

Samba - samba-technical mailing list
On Mon, 2017-07-03 at 08:20 +0200, Andreas Schneider wrote:

> On Friday, 30 June 2017 07:03:17 CEST Andrew Bartlett via samba-technical
> wrote:
> > G'Day Andreas,
>
> Hi Andrew,
>  
> > I've been trying to make our test environments run in parallel, but
> > when I enable that on the Catalyst Cloud, using this branch:
> >
> > https://git.samba.org/?p=abartlet/samba.git/.git;a=shortlog;h=refs/heads/par
> > allel-test
> >
> > I (about 50% of the time) get failures in
> > test_nosymlinks() in test_smbclient_s3.sh
> > UNEXPECTED(failure): samba3.blackbox.smbclient_s3.SMB3.plain
> > (ad_member) member creds.follow symlinks = no(ad_member)
> >
> > (sorry for the garbled output, something very odd is going on in the
> > output stream with the escaping)
>
> I've seen the same and tried to find the issue but couldn't nail it down.
>
> > REASON: Exception: Exception: CLI_FORCE_INTERACTIVE=yes
> > /home/ubuntu/autobuild/b20147/samba/bin/smbclient
> > -ULOCALADMEMBER/ubuntu%loCalMemberPass //LOCALADMEMBER/nosymlinks -I
> > 127.0.0.29
> > --configfile=/home/ubuntu/autobuild/b20147/samba/bin/ab/client/client.conf
> > -mSMB3 NT_STATUS_ACCESS_DENIED opening remote file     est\source
> > CLI_FORCE_INTERACTIVE=yes /home/ubuntu/autobuild/b20147/samba/bin/smbclient
> > -ULOCALADMEMBER/ubuntu%loCalMemberPass //LOCALADMEMBER/nosymlinks -I
> > 127.0.0.29
> > --configfile=/home/ubuntu/autobuild/b20147/samba/bin/ab/client/client.conf
> > -mSMB3 CLI_FORCE_INTERACTIVE=yes
> > /home/ubuntu/autobuild/b20147/samba/bin/smbclient
> > -ULOCALADMEMBER/ubuntu%loCalMemberPass //LOCALADMEMBER/nosymlinks -I
> > 127.0.0.29
> > --configfile=/home/ubuntu/autobuild/b20147/samba/bin/ab/client/client.conf
> > -mSMB3 NT_STATUS_FILE_IS_A_DIRECTORY opening remote
> > file       est ooar       estfile Try "help" do get a list of possible
> > commands.
> > smb: \> cd test ooar
> > smb:    est ooar\> ls
> >   .                                   D        0  Fri Jun 30 03:27:08 2017
> >   ..                                  D        0  Fri Jun 30 03:27:08 2017
> >   testfile                            D        0  Fri Jun 30 03:27:08 2017
> >
> >                 10287416 blocks of size 1024. 3260044 blocks available
> > smb:    est ooar\> get testfile -
> > NT_STATUS_FILE_IS_A_DIRECTORY opening remote
> > file       est ooar       estfile smb:    est ooar\> quit
> > failed - NT_STATUS_XXXX doing cd fooar; get testfile on
> > osymlinks
> >
> > It would be great to make some progress on this, because in the
> > successful runs, I see autobuild times drop by 60mins (288 -> 218 on an
> > identical VM)!
> >
> > I had assumed this was due to re-use of the dev/inode, but I tried the
> > attached and it didn't help.  (This happens when files are created or
> > deleted on the client side, not the server side).
>
> I've sent a mail to Jeremy who wrote the code and test for the CVE regression,
> but he asked for a reproducer. I can only reproduce it if I run the test 20
> times in a loop then normally it happens once.
>
> > Another idea would be to wipe the xattr.tdb file between each test.
> >
> > My final idea is to have as much of the deltree and mkdir etc be done
> > over SMB, so they will get an entry in the xattr.tdb, but I've not had
> > time to do this yet.
>
> Hmm. Jeremy any ideas?

I tried to do the deltree and mkdir, but it just became a time sink of
failed authentication and other random pain.

https://git.samba.org/?p=abartlet/samba.git/.git;a=shortlog;h=refs/heads/parallel-test

See

https://git.samba.org/?p=abartlet/samba.git/.git;a=commitdiff;h=c1988e2bebb04b3a8f17d08b2761cabecd235459
https://git.samba.org/?p=abartlet/samba.git/.git;a=commitdiff;h=7a67ac8b71271aeec4664ff341522569d7922abb

The key is to never do local rm, mkdir our file creation operations.
ln is probably OK, but anything else risks dev/inode re-use.


> P.S. Metze and I started to improve selftest. However I couldn't get the
> following patch working yet:
>
> https://git.samba.org/?p=asn/samba.git;a=shortlog;h=refs/heads/master-selftest-ok
>
> Some tests are failing but we couldn't nail it down to why and where.

OK.  I'll take a peek when I get a chance.

Thanks,

Andrew Bartlett

--
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: NT_STATUS_FILE_IS_A_DIRECTORY in nosymlinks test

Samba - samba-technical mailing list
On Monday, 3 July 2017 10:51:42 CEST Andrew Bartlett wrote:

> > > Another idea would be to wipe the xattr.tdb file between each test.
> > >
> > > My final idea is to have as much of the deltree and mkdir etc be done
> > > over SMB, so they will get an entry in the xattr.tdb, but I've not had
> > > time to do this yet.
> >
> > Hmm. Jeremy any ideas?
>
> I tried to do the deltree and mkdir, but it just became a time sink of
> failed authentication and other random pain.
>
> https://git.samba.org/?p=abartlet/samba.git/.git;a=shortlog;h=refs/heads/par
> allel-test
>
> See
>
> https://git.samba.org/?p=abartlet/samba.git/.git;a=commitdiff;h=c1988e2bebb0
> 4b3a8f17d08b2761cabecd235459
> https://git.samba.org/?p=abartlet/samba.git/.git;a=commitdiff;h=7a67ac8b712
> 71aeec4664ff341522569d7922abb

Thanks for the patches, looks like we want a deltree/remove_tree function in
smbclient :)

>
> The key is to never do local rm, mkdir our file creation operations.
> ln is probably OK, but anything else risks dev/inode re-use.

Jeremy, can you add a remove_tree function and then fix the test based on
Andrews patches?


> > P.S. Metze and I started to improve selftest. However I couldn't get the
> > following patch working yet:
> >
> > https://git.samba.org/?p=asn/samba.git;a=shortlog;h=refs/heads/master-self
> > test-ok
> >
> > Some tests are failing but we couldn't nail it down to why and where.
>
> OK.  I'll take a peek when I get a chance.

Thanks, currently the winbind.structs test fails with an error that it can't
connect over DCERPC if you run all ad_dc tests. If you run the test alone it
works.

Looks like a test taints the environment.



        Andreas

--
Andreas Schneider                   GPG-ID: CC014E3D
Samba Team                             [hidden email]
www.samba.org

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: NT_STATUS_FILE_IS_A_DIRECTORY in nosymlinks test

Samba - samba-technical mailing list
On Mon, Jul 03, 2017 at 11:20:41AM +0200, Andreas Schneider wrote:
> On Monday, 3 July 2017 10:51:42 CEST Andrew Bartlett wrote:
> Jeremy, can you add a remove_tree function and then fix the test based on
> Andrews patches?

Don't have time to do this immediately. However, smbclient del
command follows the 'resurse' flag. I suggest testing that to
see if it will do what is required for the remove_tree case.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: NT_STATUS_FILE_IS_A_DIRECTORY in nosymlinks test

Samba - samba-technical mailing list
In reply to this post by Samba - samba-technical mailing list
On Mon, Jul 03, 2017 at 08:20:22AM +0200, Andreas Schneider wrote:

> On Friday, 30 June 2017 07:03:17 CEST Andrew Bartlett via samba-technical
> wrote:
> > G'Day Andreas,
>
> Hi Andrew,
>  
> > I've been trying to make our test environments run in parallel, but
> > when I enable that on the Catalyst Cloud, using this branch:
> >
> > https://git.samba.org/?p=abartlet/samba.git/.git;a=shortlog;h=refs/heads/par
> > allel-test
> >
> > I (about 50% of the time) get failures in
> > test_nosymlinks() in test_smbclient_s3.sh
> > UNEXPECTED(failure): samba3.blackbox.smbclient_s3.SMB3.plain
> > (ad_member) member creds.follow symlinks = no(ad_member)
> >
> > (sorry for the garbled output, something very odd is going on in the
> > output stream with the escaping)
>
> I've seen the same and tried to find the issue but couldn't nail it down.
>
> > REASON: Exception: Exception: CLI_FORCE_INTERACTIVE=yes
> > /home/ubuntu/autobuild/b20147/samba/bin/smbclient
> > -ULOCALADMEMBER/ubuntu%loCalMemberPass //LOCALADMEMBER/nosymlinks -I
> > 127.0.0.29
> > --configfile=/home/ubuntu/autobuild/b20147/samba/bin/ab/client/client.conf
> > -mSMB3 NT_STATUS_ACCESS_DENIED opening remote file     est\source
> > CLI_FORCE_INTERACTIVE=yes /home/ubuntu/autobuild/b20147/samba/bin/smbclient
> > -ULOCALADMEMBER/ubuntu%loCalMemberPass //LOCALADMEMBER/nosymlinks -I
> > 127.0.0.29
> > --configfile=/home/ubuntu/autobuild/b20147/samba/bin/ab/client/client.conf
> > -mSMB3 CLI_FORCE_INTERACTIVE=yes
> > /home/ubuntu/autobuild/b20147/samba/bin/smbclient
> > -ULOCALADMEMBER/ubuntu%loCalMemberPass //LOCALADMEMBER/nosymlinks -I
> > 127.0.0.29
> > --configfile=/home/ubuntu/autobuild/b20147/samba/bin/ab/client/client.conf
> > -mSMB3 NT_STATUS_FILE_IS_A_DIRECTORY opening remote
> > file       est ooar       estfile Try "help" do get a list of possible
> > commands.
> > smb: \> cd test ooar
> > smb:    est ooar\> ls
> >   .                                   D        0  Fri Jun 30 03:27:08 2017
> >   ..                                  D        0  Fri Jun 30 03:27:08 2017
> >   testfile                            D        0  Fri Jun 30 03:27:08 2017
> >
> >                 10287416 blocks of size 1024. 3260044 blocks available
> > smb:    est ooar\> get testfile -
> > NT_STATUS_FILE_IS_A_DIRECTORY opening remote
> > file       est ooar       estfile smb:    est ooar\> quit
> > failed - NT_STATUS_XXXX doing cd fooar; get testfile on
> > osymlinks
> >
> > It would be great to make some progress on this, because in the
> > successful runs, I see autobuild times drop by 60mins (288 -> 218 on an
> > identical VM)!
> >
> > I had assumed this was due to re-use of the dev/inode, but I tried the
> > attached and it didn't help.  (This happens when files are created or
> > deleted on the client side, not the server side).
>
> I've sent a mail to Jeremy who wrote the code and test for the CVE regression,
> but he asked for a reproducer. I can only reproduce it if I run the test 20
> times in a loop then normally it happens once.
>
> > Another idea would be to wipe the xattr.tdb file between each test.
> >
> > My final idea is to have as much of the deltree and mkdir etc be done
> > over SMB, so they will get an entry in the xattr.tdb, but I've not had
> > time to do this yet.
>
> Hmm. Jeremy any ideas?

FYI. I now have a working smbclient "deltree <wcard>" command.

I'll add a test for it and then propose for inclusion.

Should be easy for you to add this into the nosymlinks tests once it's
in and we'll see if it fixes the problem !

Cheers,

        Jeremy.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: NT_STATUS_FILE_IS_A_DIRECTORY in nosymlinks test

Samba - samba-technical mailing list
On Wed, 2017-07-05 at 17:30 -0700, Jeremy Allison wrote:
>
> FYI. I now have a working smbclient "deltree <wcard>" command.
>
> I'll add a test for it and then propose for inclusion.
>
> Should be easy for you to add this into the nosymlinks tests once
> it's
> in and we'll see if it fixes the problem !

Could you add it in to nosymlinks and localsymlinks, as I understand it
is your test?

You don't need to prove the test is or isn't flapping, just re-writing
the nosymlinks test and the localsymlinks test to never make local file
operations (other than ln) while still proving the security assertions
and passing normally should do it.

Thanks,

Andrew Bartlett

--
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team         https://samba.org
Samba Development and Support, Catalyst IT  
https://catalyst.net.nz/services/samba





Loading...