autobuild over 4 hours

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

autobuild over 4 hours

Samba - samba-technical mailing list
Just a quick note to other confused developers:

Autobuild on sn-devel has just ticked close to/over 4 hours, so is
starting to fail with timelimit exceeded again.  

The immediate fix is to bump up the time limit to 5 hours, but the
better fix is my parallel build patch (saves 60 mins!).  

Unfortunately that is blocked on the nosymlinks flapping test (but at
least we know what is required for that).

Thanks,

Andrew Bartlett
--
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: autobuild over 4 hours

Samba - samba-technical mailing list
On Thu, 06 Jul 2017 08:02:01 +1200, Andrew Bartlett via samba-technical
<[hidden email]> wrote:

> Just a quick note to other confused developers:
>
> Autobuild on sn-devel has just ticked close to/over 4 hours, so is
> starting to fail with timelimit exceeded again.  
>
> The immediate fix is to bump up the time limit to 5 hours, but the
> better fix is my parallel build patch (saves 60 mins!).  
>
> Unfortunately that is blocked on the nosymlinks flapping test (but at
> least we know what is required for that).

How do we increase the time limit?  At least in the interim...

peace & happiness,
martin

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: autobuild over 4 hours

Samba - samba-technical mailing list
On Thu, 2017-07-06 at 13:35 +1000, Martin Schwenke via samba-technical
wrote:

> On Thu, 06 Jul 2017 08:02:01 +1200, Andrew Bartlett via samba-
> technical
> <[hidden email]> wrote:
>
> > Just a quick note to other confused developers:
> >
> > Autobuild on sn-devel has just ticked close to/over 4 hours, so is
> > starting to fail with timelimit exceeded again.  
> >
> > The immediate fix is to bump up the time limit to 5 hours, but the
> > better fix is my parallel build patch (saves 60 mins!).  
> >
> > Unfortunately that is blocked on the nosymlinks flapping test (but
> > at
> > least we know what is required for that).
>
> How do we increase the time limit?  At least in the interim... 
We certainly could.  I decided not to give up so easily, and I think
I've found a few mins with the attached patches (currently under test).

We avoid quite some time if we don't fork and don't call kinit over and
over in some of the famous DRS tests.

Either way is really fine by me, as cloud VMs are slower I've been
setting a 5 hour max anyway for some time.

Thanks,

Andrew Bartlett


--
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team         https://samba.org
Samba Development and Support, Catalyst IT  
https://catalyst.net.nz/services/samba




less-kinit-in-drs.patch.txt (52K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: autobuild over 4 hours

Samba - samba-technical mailing list
On Thu, Jul 06, 2017 at 05:12:29PM +1200, Andrew Bartlett via samba-technical wrote:

> On Thu, 2017-07-06 at 13:35 +1000, Martin Schwenke via samba-technical
> wrote:
> > On Thu, 06 Jul 2017 08:02:01 +1200, Andrew Bartlett via samba-
> > technical
> > <[hidden email]> wrote:
> >
> > > Just a quick note to other confused developers:
> > >
> > > Autobuild on sn-devel has just ticked close to/over 4 hours, so is
> > > starting to fail with timelimit exceeded again.  
> > >
> > > The immediate fix is to bump up the time limit to 5 hours, but the
> > > better fix is my parallel build patch (saves 60 mins!).  
> > >
> > > Unfortunately that is blocked on the nosymlinks flapping test (but
> > > at
> > > least we know what is required for that).
> >
> > How do we increase the time limit?  At least in the interim... 
>
> We certainly could.  I decided not to give up so easily, and I think
> I've found a few mins with the attached patches (currently under test).
>
> We avoid quite some time if we don't fork and don't call kinit over and
> over in some of the famous DRS tests.
>
> Either way is really fine by me, as cloud VMs are slower I've been
> setting a 5 hour max anyway for some time.

The C part - rb+ me with one query:

+static PyObject *py_ccache_name(PyObject *self, PyObject *unused)
+{
+       struct ccache_container *ccc;
+       char *name = NULL;
+       PyObject *py_name;
+       int ret;
+       ccc = pytalloc_get_type(self, struct ccache_container);

        ^^^^^
        Can pytalloc_get_type fail and return NULL ? If so
        we need a != NULL check after this.

+       ret = krb5_cc_get_full_name(ccc->smb_krb5_context->krb5_context,
+                                   ccc->ccache, &name);
+       if (ret == 0) {
+               py_name = PyString_FromStringOrNULL(name);
+               SAFE_FREE(name);
+       } else {
+               PyErr_SetString(PyExc_RuntimeError,
+                               "Failed to get ccache name");
+               return NULL;
+       }
+       return py_name;
+}

Sorry, would love to rb+ the python but my python-fu isn't
up to it yet.

Cheers,

        Jeremy.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: autobuild over 4 hours

Samba - samba-technical mailing list
On Thu, 2017-07-06 at 15:06 -0700, Jeremy Allison wrote:

> On Thu, Jul 06, 2017 at 05:12:29PM +1200, Andrew Bartlett via samba-
> technical wrote:
> > On Thu, 2017-07-06 at 13:35 +1000, Martin Schwenke via samba-
> > technical
> > wrote:
> > > On Thu, 06 Jul 2017 08:02:01 +1200, Andrew Bartlett via samba-
> > > technical
> > > <[hidden email]> wrote:
> > >
> > > > Just a quick note to other confused developers:
> > > >
> > > > Autobuild on sn-devel has just ticked close to/over 4 hours, so
> > > > is
> > > > starting to fail with timelimit exceeded again.  
> > > >
> > > > The immediate fix is to bump up the time limit to 5 hours, but
> > > > the
> > > > better fix is my parallel build patch (saves 60 mins!).  
> > > >
> > > > Unfortunately that is blocked on the nosymlinks flapping test
> > > > (but
> > > > at
> > > > least we know what is required for that).
> > >
> > > How do we increase the time limit?  At least in the interim... 
> >
> > We certainly could.  I decided not to give up so easily, and I
> > think
> > I've found a few mins with the attached patches (currently under
> > test).
> >
> > We avoid quite some time if we don't fork and don't call kinit over
> > and
> > over in some of the famous DRS tests.
> >
> > Either way is really fine by me, as cloud VMs are slower I've been
> > setting a 5 hour max anyway for some time.
>
> The C part - rb+ me with one query:
>
> +static PyObject *py_ccache_name(PyObject *self, PyObject *unused)
> +{
> +       struct ccache_container *ccc;
> +       char *name = NULL;
> +       PyObject *py_name;
> +       int ret;
> +       ccc = pytalloc_get_type(self, struct ccache_container);
>
> ^^^^^
> Can pytalloc_get_type fail and return NULL ? If so
> we need a != NULL check after this.

In general yes, in this case no, because self is always bound to the
python type it came from, so as long as the python object was created
by PyCredentialCacheContainer_from_ccache_container() (which it is) we
are pretty safe.

For the rest, I have a branch under test at:

https://git.samba.org/?p=abartlet/samba.git/.git;a=shortlog;h=refs/heads/faster-drs-krb5-ccache

Thanks!

Andrew Bartlett
--
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team         https://samba.org
Samba Development and Support, Catalyst IT  
https://catalyst.net.nz/services/samba





Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[PATCH] Use the same process and krb5 ccache when invoking samba-tool in drs tests

Samba - samba-technical mailing list
This patch removes a number of calls to fork() inside python, and
instead of passing -Uusername%password on the internal command line, we
pass around the name of the krb5 credentials cache we already have.

I've not been able to quantify the speed benefits - there seems to be
more noise than signal in the timing here, hence my other patch
reducing idle time, but when we had these tests under perf/FlameGraph
the string2key routines really dominated the time what was spent on the
CPU.

Earlier in the thread, Jeremy looked over the C parts, but if someone
could look over the rest that would be great!

Please review!

Thanks,

Andrew Bartlett

--
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba

faster-drs-krb5-ccache.patch.txt (51K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PATCH] Use the same process and krb5 ccache when invoking samba-tool in drs tests

Samba - samba-technical mailing list
On Sat, 2017-07-22 at 08:19 +1200, Andrew Bartlett via samba-technical
wrote:
> This patch removes a number of calls to fork() inside python, and
> instead of passing -Uusername%password on the internal command line, we
> pass around the name of the krb5 credentials cache we already have.
>
> I've not been able to quantify the speed benefits - there seems to be
> more noise than signal in the timing here, hence my other patch
> reducing idle time, but when we had these tests under perf/FlameGraph
> the string2key routines really dominated the time what was spent on the
> CPU.

Actually, if I use my tevent_immediate patch as the baseline, it saves
about 80 seconds per repl_move test on my laptop.

Aside from making the timing more reliable, it is however independent
of that patch.

> Earlier in the thread, Jeremy looked over the C parts, but if someone
> could look over the rest that would be great!
>
> Please review!
>
> Thanks,
>
> Andrew Bartlett
>
--
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba


Loading...