[PATCHES] Generate shorter name for extra python files

classic Classic list List threaded Threaded
40 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: [PATCHES] Generate shorter name for extra python files

Samba - samba-technical mailing list
On Fri, 2017-07-21 at 15:46 +0200, Stefan Metzmacher via samba-
technical wrote:

> Hi Lukas,
>
> > > as this will take a bit to discuss, I'll prepare new talloc, tevent
> > > and ldb releases without this.
> > >
> > > We need an ldb release for 4.7.0rc3 which is planed for next Tuesday.
> > >
> >
> > I thought I answer all questions and Abrew wrote that:
> > "(in that respect, I've not done a full review)"
> >
> > I am not sure how to move this forward. Or where is a problem.
> > But I would really appreciate solution which works for upstream
> > and all downstream debian, fedora ...
>
> It's just time, some people are/will be on vacation, so it takes a bit
> of time to have a solid solution.
>
> We should not rush this into the release on Thursday. There will
> be at least 1 or 2 rc's before 4.7.0.
>
> But the most important thing is that we should not add a solution
> to ldb-1.2.1, which we than have to revert for ldb-1.2.2.

I agree, we need to get this right.  Skipping it for rc3 should be
fine.

This area is complex, and I've been (rather unsuccessfully ;-) on leave
for the past couple of weeks, and it need some serious thought, and I
think the input of the original author to understand the original
rationale.

Then we need to publish the ABI files and binaries, then add a new
function, re-publish and check it actually works, the new function is
tagged with the new version, the old functions untagged in the ABI, the
.so and then the thing debian uses to track those.

That will give me the confidence we have got it right.

Thanks,

Andrew Bartlett


--
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba


Reply | Threaded
Open this post in threaded view
|

Re: [PATCHES] Generate shorter name for extra python files

Samba - samba-technical mailing list
On (22/07/17 13:43), Andrew Bartlett via samba-technical wrote:

>On Fri, 2017-07-21 at 15:46 +0200, Stefan Metzmacher via samba-
>technical wrote:
>> Hi Lukas,
>>
>> > > as this will take a bit to discuss, I'll prepare new talloc, tevent
>> > > and ldb releases without this.
>> > >
>> > > We need an ldb release for 4.7.0rc3 which is planed for next Tuesday.
>> > >
>> >
>> > I thought I answer all questions and Abrew wrote that:
>> > "(in that respect, I've not done a full review)"
>> >
>> > I am not sure how to move this forward. Or where is a problem.
>> > But I would really appreciate solution which works for upstream
>> > and all downstream debian, fedora ...
>>
>> It's just time, some people are/will be on vacation, so it takes a bit
>> of time to have a solid solution.
>>
>> We should not rush this into the release on Thursday. There will
>> be at least 1 or 2 rc's before 4.7.0.
>>
>> But the most important thing is that we should not add a solution
>> to ldb-1.2.1, which we than have to revert for ldb-1.2.2.
>
>I agree, we need to get this right.  Skipping it for rc3 should be
>fine.
>
>This area is complex, and I've been (rather unsuccessfully ;-) on leave
>for the past couple of weeks, and it need some serious thought, and I
>think the input of the original author to understand the original
>rationale.
>
Do you mean purpose what was purpose of ABI/*.sigs instead of writing
version script directly? Or something related to names for extra
python util libraries.

>Then we need to publish the ABI files and binaries, then add a new
>function, re-publish and check it actually works, the new function is
>tagged with the new version, the old functions untagged in the ABI, the
>.so and then the thing debian uses to track those.
>

Do you mean something like removing pytalloc_BaseObject_check
from pytalloc-util-2.1.6.sigs, pytalloc-util-2.1.7.sigs
pytalloc-util-2.1.8.sigs ?

It would be like moving pytalloc-util-2.1.6.sigs from 2.1.6
to 2.1.9. Which sounds a little bit simpler to me for testing purposes.

LS

Reply | Threaded
Open this post in threaded view
|

Re: [PATCHES] Generate shorter name for extra python files

Samba - samba-technical mailing list
In reply to this post by Samba - samba-technical mailing list
On (22/07/17 13:43), Andrew Bartlett via samba-technical wrote:

>On Fri, 2017-07-21 at 15:46 +0200, Stefan Metzmacher via samba-
>technical wrote:
>> Hi Lukas,
>>
>> > > as this will take a bit to discuss, I'll prepare new talloc, tevent
>> > > and ldb releases without this.
>> > >
>> > > We need an ldb release for 4.7.0rc3 which is planed for next Tuesday.
>> > >
>> >
>> > I thought I answer all questions and Abrew wrote that:
>> > "(in that respect, I've not done a full review)"
>> >
>> > I am not sure how to move this forward. Or where is a problem.
>> > But I would really appreciate solution which works for upstream
>> > and all downstream debian, fedora ...
>>
>> It's just time, some people are/will be on vacation, so it takes a bit
>> of time to have a solid solution.
>>
>> We should not rush this into the release on Thursday. There will
>> be at least 1 or 2 rc's before 4.7.0.
>>
>> But the most important thing is that we should not add a solution
>> to ldb-1.2.1, which we than have to revert for ldb-1.2.2.
>
>I agree, we need to get this right.  Skipping it for rc3 should be
>fine.
>

And we skipped rc4 as well. But I hope it is still possible
to get patches to final 4.7.0 :-)

>This area is complex, and I've been (rather unsuccessfully ;-) on leave
>for the past couple of weeks, and it need some serious thought, and I
>think the input of the original author to understand the original
>rationale.
>
>Then we need to publish the ABI files and binaries, then add a new
>function, re-publish and check it actually works, the new function is
>tagged with the new version, the old functions untagged in the ABI, the
>.so and then the thing debian uses to track those.
>
>That will give me the confidence we have got it right.
>

LS

Reply | Threaded
Open this post in threaded view
|

Re: [PATCHES] Generate shorter name for extra python files

Samba - samba-technical mailing list
On Tue, 2017-08-15 at 14:22 +0200, Lukas Slebodnik wrote:

> On (22/07/17 13:43), Andrew Bartlett via samba-technical wrote:
> > On Fri, 2017-07-21 at 15:46 +0200, Stefan Metzmacher via samba-
> > technical wrote:
> > > Hi Lukas,
> > >
> > > > > as this will take a bit to discuss, I'll prepare new talloc,
> > > > > tevent
> > > > > and ldb releases without this.
> > > > >
> > > > > We need an ldb release for 4.7.0rc3 which is planed for next
> > > > > Tuesday.
> > > > >
> > > >
> > > > I thought I answer all questions and Abrew wrote that:
> > > > "(in that respect, I've not done a full review)"
> > > >
> > > > I am not sure how to move this forward. Or where is a problem.
> > > > But I would really appreciate solution which works for upstream
> > > > and all downstream debian, fedora ...
> > >
> > > It's just time, some people are/will be on vacation, so it takes
> > > a bit
> > > of time to have a solid solution.
> > >
> > > We should not rush this into the release on Thursday. There will
> > > be at least 1 or 2 rc's before 4.7.0.
> > >
> > > But the most important thing is that we should not add a solution
> > > to ldb-1.2.1, which we than have to revert for ldb-1.2.2.
> >
> > I agree, we need to get this right.  Skipping it for rc3 should be
> > fine.
> >
>
> And we skipped rc4 as well. But I hope it is still possible
> to get patches to final 4.7.0 :-)

Perhaps.  Sadly we are all busy, but this is still our aim.  Is there a
particular need for this in 4.7?

> > This area is complex, and I've been (rather unsuccessfully ;-) on
> > leave
> > for the past couple of weeks, and it need some serious thought, and
> > I
> > think the input of the original author to understand the original
> > rationale. 

This part is critical.  

Petr,

Can you comment on what the design intent was here?

> > Then we need to publish the ABI files and binaries, then add a new
> > function, re-publish and check it actually works, the new function
> > is
> > tagged with the new version, the old functions untagged in the ABI,
> > the
> > .so and then the thing debian uses to track those. 
> >
> > That will give me the confidence we have got it right.
> >

This is still critical.  We need to ensure this works for a new
incremental release, as that is where the last similar scheme failed.

Andrew Bartlett

--
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team         https://samba.org
Samba Development and Support, Catalyst IT  
https://catalyst.net.nz/services/samba





Reply | Threaded
Open this post in threaded view
|

Re: [PATCHES] Generate shorter name for extra python files

Samba - samba-technical mailing list
On ke, 16 elo 2017, Andrew Bartlett via samba-technical wrote:

> On Tue, 2017-08-15 at 14:22 +0200, Lukas Slebodnik wrote:
> > On (22/07/17 13:43), Andrew Bartlett via samba-technical wrote:
> > > On Fri, 2017-07-21 at 15:46 +0200, Stefan Metzmacher via samba-
> > > technical wrote:
> > > > Hi Lukas,
> > > >
> > > > > > as this will take a bit to discuss, I'll prepare new talloc,
> > > > > > tevent
> > > > > > and ldb releases without this.
> > > > > >
> > > > > > We need an ldb release for 4.7.0rc3 which is planed for next
> > > > > > Tuesday.
> > > > > >
> > > > >
> > > > > I thought I answer all questions and Abrew wrote that:
> > > > > "(in that respect, I've not done a full review)"
> > > > >
> > > > > I am not sure how to move this forward. Or where is a problem.
> > > > > But I would really appreciate solution which works for upstream
> > > > > and all downstream debian, fedora ...
> > > >
> > > > It's just time, some people are/will be on vacation, so it takes
> > > > a bit
> > > > of time to have a solid solution.
> > > >
> > > > We should not rush this into the release on Thursday. There will
> > > > be at least 1 or 2 rc's before 4.7.0.
> > > >
> > > > But the most important thing is that we should not add a solution
> > > > to ldb-1.2.1, which we than have to revert for ldb-1.2.2.
> > >
> > > I agree, we need to get this right.  Skipping it for rc3 should be
> > > fine.
> > >
> >
> > And we skipped rc4 as well. But I hope it is still possible
> > to get patches to final 4.7.0 :-)
>
> Perhaps.  Sadly we are all busy, but this is still our aim.  Is there a
> particular need for this in 4.7?
It sets Python module names for a release. Changing them later would be a bad
practice. Even though they aren't like sonames, they still better to be
fixed in advance.

I could give my RB+ but I thought there is a benefit from having it all
vetted by a wider community.


--
/ Alexander Bokovoy

Reply | Threaded
Open this post in threaded view
|

Re: [PATCHES] Generate shorter name for extra python files

Samba - samba-technical mailing list
In reply to this post by Samba - samba-technical mailing list
On (16/08/17 10:02), Andrew Bartlett wrote:

>On Tue, 2017-08-15 at 14:22 +0200, Lukas Slebodnik wrote:
>> On (22/07/17 13:43), Andrew Bartlett via samba-technical wrote:
>> > On Fri, 2017-07-21 at 15:46 +0200, Stefan Metzmacher via samba-
>> > technical wrote:
>> > > Hi Lukas,
>> > >
>> > > > > as this will take a bit to discuss, I'll prepare new talloc,
>> > > > > tevent
>> > > > > and ldb releases without this.
>> > > > >
>> > > > > We need an ldb release for 4.7.0rc3 which is planed for next
>> > > > > Tuesday.
>> > > > >
>> > > >
>> > > > I thought I answer all questions and Abrew wrote that:
>> > > > "(in that respect, I've not done a full review)"
>> > > >
>> > > > I am not sure how to move this forward. Or where is a problem.
>> > > > But I would really appreciate solution which works for upstream
>> > > > and all downstream debian, fedora ...
>> > >
>> > > It's just time, some people are/will be on vacation, so it takes
>> > > a bit
>> > > of time to have a solid solution.
>> > >
>> > > We should not rush this into the release on Thursday. There will
>> > > be at least 1 or 2 rc's before 4.7.0.
>> > >
>> > > But the most important thing is that we should not add a solution
>> > > to ldb-1.2.1, which we than have to revert for ldb-1.2.2.
>> >
>> > I agree, we need to get this right.  Skipping it for rc3 should be
>> > fine.
>> >
>>
>> And we skipped rc4 as well. But I hope it is still possible
>> to get patches to final 4.7.0 :-)
>
>Perhaps.  Sadly we are all busy, but this is still our aim.  Is there a
>particular need for this in 4.7?
>

ATM current version of libraries (with architecture in names for python3)
are not widely used among distributions (only fedora 27 which is in devel phase)
But official samba 4.7.0 might be adopted quite fast due to samba-ad-dc with
MIT krb5. This is a reason why I would prefer to solve this till official
4.7.0. And I would prefer solution which is acceptable for most of
distributions. Downstream only patches are bad.

LS

Reply | Threaded
Open this post in threaded view
|

Re: [PATCHES] Generate shorter name for extra python files

Samba - samba-technical mailing list
On Wednesday, 16 August 2017 12:51:59 CEST Lukas Slebodnik via samba-technical
wrote:

> On (16/08/17 10:02), Andrew Bartlett wrote:
> >On Tue, 2017-08-15 at 14:22 +0200, Lukas Slebodnik wrote:
> >> On (22/07/17 13:43), Andrew Bartlett via samba-technical wrote:
> >> > On Fri, 2017-07-21 at 15:46 +0200, Stefan Metzmacher via samba-
> >> >
> >> > technical wrote:
> >> > > Hi Lukas,
> >> > >
> >> > > > > as this will take a bit to discuss, I'll prepare new talloc,
> >> > > > > tevent
> >> > > > > and ldb releases without this.
> >> > > > >
> >> > > > > We need an ldb release for 4.7.0rc3 which is planed for next
> >> > > > > Tuesday.
> >> > > >
> >> > > > I thought I answer all questions and Abrew wrote that:
> >> > > > "(in that respect, I've not done a full review)"
> >> > > >
> >> > > > I am not sure how to move this forward. Or where is a problem.
> >> > > > But I would really appreciate solution which works for upstream
> >> > > > and all downstream debian, fedora ...
> >> > >
> >> > > It's just time, some people are/will be on vacation, so it takes
> >> > > a bit
> >> > > of time to have a solid solution.
> >> > >
> >> > > We should not rush this into the release on Thursday. There will
> >> > > be at least 1 or 2 rc's before 4.7.0.
> >> > >
> >> > > But the most important thing is that we should not add a solution
> >> > > to ldb-1.2.1, which we than have to revert for ldb-1.2.2.
> >> >
> >> > I agree, we need to get this right.  Skipping it for rc3 should be
> >> > fine.
> >>
> >> And we skipped rc4 as well. But I hope it is still possible
> >> to get patches to final 4.7.0 :-)
> >
> >Perhaps.  Sadly we are all busy, but this is still our aim.  Is there a
> >particular need for this in 4.7?
>
> ATM current version of libraries (with architecture in names for python3)
> are not widely used among distributions (only fedora 27 which is in devel
> phase) But official samba 4.7.0 might be adopted quite fast due to
> samba-ad-dc with MIT krb5. This is a reason why I would prefer to solve
> this till official 4.7.0. And I would prefer solution which is acceptable
> for most of distributions. Downstream only patches are bad.

I agree we NEED to fix this for 4.7. It will be a pain changing it later!


I would argue that this is a blocker for the 4.7 release.


        Andreas

--
Andreas Schneider                   GPG-ID: CC014E3D
Samba Team                             [hidden email]
www.samba.org

Reply | Threaded
Open this post in threaded view
|

Re: [PATCHES] Generate shorter name for extra python files

Samba - samba-technical mailing list
On Wed, 2017-08-16 at 13:54 +0200, Andreas Schneider wrote:

> On Wednesday, 16 August 2017 12:51:59 CEST Lukas Slebodnik via samba-technical
> wrote:
> > On (16/08/17 10:02), Andrew Bartlett wrote:
> > > On Tue, 2017-08-15 at 14:22 +0200, Lukas Slebodnik wrote:
> > > > On (22/07/17 13:43), Andrew Bartlett via samba-technical wrote:
> > > > > On Fri, 2017-07-21 at 15:46 +0200, Stefan Metzmacher via samba-
> > > > >
> > > > > technical wrote:
> > > > > > Hi Lukas,
> > > > > >
> > > > > > > > as this will take a bit to discuss, I'll prepare new talloc,
> > > > > > > > tevent
> > > > > > > > and ldb releases without this.
> > > > > > > >
> > > > > > > > We need an ldb release for 4.7.0rc3 which is planed for next
> > > > > > > > Tuesday.
> > > > > > >
> > > > > > > I thought I answer all questions and Abrew wrote that:
> > > > > > > "(in that respect, I've not done a full review)"
> > > > > > >
> > > > > > > I am not sure how to move this forward. Or where is a problem.
> > > > > > > But I would really appreciate solution which works for upstream
> > > > > > > and all downstream debian, fedora ...
> > > > > >
> > > > > > It's just time, some people are/will be on vacation, so it takes
> > > > > > a bit
> > > > > > of time to have a solid solution.
> > > > > >
> > > > > > We should not rush this into the release on Thursday. There will
> > > > > > be at least 1 or 2 rc's before 4.7.0.
> > > > > >
> > > > > > But the most important thing is that we should not add a solution
> > > > > > to ldb-1.2.1, which we than have to revert for ldb-1.2.2.
> > > > >
> > > > > I agree, we need to get this right.  Skipping it for rc3 should be
> > > > > fine.
> > > >
> > > > And we skipped rc4 as well. But I hope it is still possible
> > > > to get patches to final 4.7.0 :-)
> > >
> > > Perhaps.  Sadly we are all busy, but this is still our aim.  Is there a
> > > particular need for this in 4.7?
> >
> > ATM current version of libraries (with architecture in names for python3)
> > are not widely used among distributions (only fedora 27 which is in devel
> > phase) But official samba 4.7.0 might be adopted quite fast due to
> > samba-ad-dc with MIT krb5. This is a reason why I would prefer to solve
> > this till official 4.7.0. And I would prefer solution which is acceptable
> > for most of distributions. Downstream only patches are bad.
>
> I agree we NEED to fix this for 4.7. It will be a pain changing it later!
>
>
> I would argue that this is a blocker for the 4.7 release.

Can you help out on this?  The work has been stalled waiting for:
 - A comment from Petr on the original design intent
 - confirmation of which symbols are generated after another new
version beyond the introduction of this patch.  (Because that is what
broke, and made me revert, a similar change in Debian).

Thanks,

Andrew Bartlett
--
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba


Reply | Threaded
Open this post in threaded view
|

Re: [PATCHES] Generate shorter name for extra python files

Samba - samba-technical mailing list
On 08/17/2017 09:33 PM, Andrew Bartlett via samba-technical wrote:

> On Wed, 2017-08-16 at 13:54 +0200, Andreas Schneider wrote:
>> On Wednesday, 16 August 2017 12:51:59 CEST Lukas Slebodnik via samba-technical
>> wrote:
>>> On (16/08/17 10:02), Andrew Bartlett wrote:
>>>> On Tue, 2017-08-15 at 14:22 +0200, Lukas Slebodnik wrote:
>>>>> On (22/07/17 13:43), Andrew Bartlett via samba-technical wrote:
>>>>>> On Fri, 2017-07-21 at 15:46 +0200, Stefan Metzmacher via samba-
>>>>>>
>>>>>> technical wrote:
>>>>>>> Hi Lukas,
>>>>>>>
>>>>>>>>> as this will take a bit to discuss, I'll prepare new talloc,
>>>>>>>>> tevent
>>>>>>>>> and ldb releases without this.
>>>>>>>>>
>>>>>>>>> We need an ldb release for 4.7.0rc3 which is planed for next
>>>>>>>>> Tuesday.
>>>>>>>>
>>>>>>>> I thought I answer all questions and Abrew wrote that:
>>>>>>>> "(in that respect, I've not done a full review)"
>>>>>>>>
>>>>>>>> I am not sure how to move this forward. Or where is a problem.
>>>>>>>> But I would really appreciate solution which works for upstream
>>>>>>>> and all downstream debian, fedora ...
>>>>>>>
>>>>>>> It's just time, some people are/will be on vacation, so it takes
>>>>>>> a bit
>>>>>>> of time to have a solid solution.
>>>>>>>
>>>>>>> We should not rush this into the release on Thursday. There will
>>>>>>> be at least 1 or 2 rc's before 4.7.0.
>>>>>>>
>>>>>>> But the most important thing is that we should not add a solution
>>>>>>> to ldb-1.2.1, which we than have to revert for ldb-1.2.2.
>>>>>>
>>>>>> I agree, we need to get this right.  Skipping it for rc3 should be
>>>>>> fine.
>>>>>
>>>>> And we skipped rc4 as well. But I hope it is still possible
>>>>> to get patches to final 4.7.0 :-)
>>>>
>>>> Perhaps.  Sadly we are all busy, but this is still our aim.  Is there a
>>>> particular need for this in 4.7?
>>>
>>> ATM current version of libraries (with architecture in names for python3)
>>> are not widely used among distributions (only fedora 27 which is in devel
>>> phase) But official samba 4.7.0 might be adopted quite fast due to
>>> samba-ad-dc with MIT krb5. This is a reason why I would prefer to solve
>>> this till official 4.7.0. And I would prefer solution which is acceptable
>>> for most of distributions. Downstream only patches are bad.
>>
>> I agree we NEED to fix this for 4.7. It will be a pain changing it later!
>>
>>
>> I would argue that this is a blocker for the 4.7 release.
>
> Can you help out on this?  The work has been stalled waiting for:
>   - A comment from Petr on the original design intent

I'm sorry for missing this thread; my vacation-mode filtering is too eager.

As for the original design, "extra python" allows building two versions
of Python by a single `make` invocation. Building for both 2 and 3 at
once was a requirement to get the Python3 patches in.
To make this work, I've reused the mechanism Python uses to allow binary
files for multiple versions to co-exist. The "ABI tag" contains just the
information needed to distinguish between ABI-incompatible versions.

Note that "/usr/lib64/pkgconfig/pyldb-util36.pc" does not include all
relevant information: the "m" in "cpython-36m-x86_64-linux-gnu" encodes
build-time configuration; "dm" would mean a debug build with
incompatible ABI.
See [PEP 3149] if you want details.

Using the ABI tag as-is essentially puts the burden of distinguishing
ABI incompatibilities on Python developers/distributors. It's also quite
easy to ask Python for it [0], so, hopefully, buildsystems of packages
that depend on the utils can be made to support this.

That said, I'm not familiar with pkg-config or multilib packaging
outside of Python, or across distros. This is the Python way to do it;
please adjust if it makes sense.


Another note on the original design intent: generally, the "extra
python" needs to be version 3, and the "normal" Python could be either 2
*or* a different version of 3. It's never been tested with py3 as the
non-extra Python, and I'm sure it doesn't actually work now, but at some
point it'll be necessary to make the "non-extra" Python be Python 3,
preferably with the relevant name mangling. Please don't assume that
"not extra-python" implies Python 2.


[PEP 3149]: https://www.python.org/dev/peps/pep-3149/
[0] import sysconfig; sysconfig.get_config_var('SOABI')


--
Petr Viktorin

Reply | Threaded
Open this post in threaded view
|

Re: [PATCHES] Generate shorter name for extra python files

Samba - samba-technical mailing list
On Tue, 2017-08-22 at 11:25 +0200, Petr Viktorin via samba-technical
wrote:

>
> I'm sorry for missing this thread; my vacation-mode filtering is too eager.
>
> As for the original design, "extra python" allows building two versions
> of Python by a single `make` invocation. Building for both 2 and 3 at
> once was a requirement to get the Python3 patches in.
> To make this work, I've reused the mechanism Python uses to allow binary
> files for multiple versions to co-exist. The "ABI tag" contains just the
> information needed to distinguish between ABI-incompatible versions.
>
> Note that "/usr/lib64/pkgconfig/pyldb-util36.pc" does not include all
> relevant information: the "m" in "cpython-36m-x86_64-linux-gnu" encodes
> build-time configuration; "dm" would mean a debug build with
> incompatible ABI.
> See [PEP 3149] if you want details.
>
> Using the ABI tag as-is essentially puts the burden of distinguishing
> ABI incompatibilities on Python developers/distributors. It's also quite
> easy to ask Python for it [0], so, hopefully, buildsystems of packages
> that depend on the utils can be made to support this.
>
> That said, I'm not familiar with pkg-config or multilib packaging
> outside of Python, or across distros. This is the Python way to do it;
> please adjust if it makes sense.
>
>
> Another note on the original design intent: generally, the "extra
> python" needs to be version 3, and the "normal" Python could be either 2
> *or* a different version of 3. It's never been tested with py3 as the
> non-extra Python, and I'm sure it doesn't actually work now, but at some
> point it'll be necessary to make the "non-extra" Python be Python 3,
> preferably with the relevant name mangling. Please don't assume that
> "not extra-python" implies Python 2.
>
>
> [PEP 3149]: https://www.python.org/dev/peps/pep-3149/
> [0] import sysconfig; sysconfig.get_config_var('SOABI')

Thanks for the details.  If you could work with Lumir and Andreas to
propose something that follows the above and meets Lumir's requirements
that would be great.  We can't be the first package to have produced a
python C binding, surely there is some guidance already?

Finally, awkward as it is, if we don't even have a common design intent
I can't support blocking 4.7 on this.  A rush really isn't good for
this kind of thing.

Thanks,

Andrew Bartlett

--
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba


Reply | Threaded
Open this post in threaded view
|

Re: [PATCHES] Generate shorter name for extra python files

Samba - samba-technical mailing list
On 08/27/2017 09:02 AM, Andrew Bartlett wrote:

> On Tue, 2017-08-22 at 11:25 +0200, Petr Viktorin via samba-technical
> wrote:
>>
>> I'm sorry for missing this thread; my vacation-mode filtering is too eager.
>>
>> As for the original design, "extra python" allows building two versions
>> of Python by a single `make` invocation. Building for both 2 and 3 at
>> once was a requirement to get the Python3 patches in.
>> To make this work, I've reused the mechanism Python uses to allow binary
>> files for multiple versions to co-exist. The "ABI tag" contains just the
>> information needed to distinguish between ABI-incompatible versions.
>>
>> Note that "/usr/lib64/pkgconfig/pyldb-util36.pc" does not include all
>> relevant information: the "m" in "cpython-36m-x86_64-linux-gnu" encodes
>> build-time configuration; "dm" would mean a debug build with
>> incompatible ABI.
>> See [PEP 3149] if you want details.
>>
>> Using the ABI tag as-is essentially puts the burden of distinguishing
>> ABI incompatibilities on Python developers/distributors. It's also quite
>> easy to ask Python for it [0], so, hopefully, buildsystems of packages
>> that depend on the utils can be made to support this.
>>
>> That said, I'm not familiar with pkg-config or multilib packaging
>> outside of Python, or across distros. This is the Python way to do it;
>> please adjust if it makes sense.
>>
>>
>> Another note on the original design intent: generally, the "extra
>> python" needs to be version 3, and the "normal" Python could be either 2
>> *or* a different version of 3. It's never been tested with py3 as the
>> non-extra Python, and I'm sure it doesn't actually work now, but at some
>> point it'll be necessary to make the "non-extra" Python be Python 3,
>> preferably with the relevant name mangling. Please don't assume that
>> "not extra-python" implies Python 2.
>>
>>
>> [PEP 3149]: https://www.python.org/dev/peps/pep-3149/
>> [0] import sysconfig; sysconfig.get_config_var('SOABI')
>
> Thanks for the details.  If you could work with Lumir and Andreas to
> propose something that follows the above and meets Lumir's requirements
> that would be great.  We can't be the first package to have produced a
> python C binding, surely there is some guidance already?

Samba is not the first package with a Python binding, but those "_util"
libraries are quite unique.

General guidance for providing a C API for Python extension modules is
provided in Python documentation [0]: roughly, put the function pointers
(and a version) in a struct, wrap it in a Capsule object, and provide
that as a Python-level object.
To use this, there's a PyCapsule_Import helper that imports a module,
fetches the attribute, and unwraps it to get the struct.

This solves problems with naming/discovery/linking of C-level utilities:
everything is found via the extension module.
But in Samba's case, it would require changes to all code that currently
uses the _util libraries.


[0]
https://docs.python.org/3/extending/extending.html#providing-a-c-api-for-an-extension-module


--
Petr Viktorin

Reply | Threaded
Open this post in threaded view
|

Re: [PATCHES] Generate shorter name for extra python files

Samba - samba-technical mailing list
On (29/08/17 10:35), Petr Viktorin via samba-technical wrote:

>On 08/27/2017 09:02 AM, Andrew Bartlett wrote:
>> On Tue, 2017-08-22 at 11:25 +0200, Petr Viktorin via samba-technical
>> wrote:
>> >
>> > I'm sorry for missing this thread; my vacation-mode filtering is too eager.
>> >
>> > As for the original design, "extra python" allows building two versions
>> > of Python by a single `make` invocation. Building for both 2 and 3 at
>> > once was a requirement to get the Python3 patches in.
>> > To make this work, I've reused the mechanism Python uses to allow binary
>> > files for multiple versions to co-exist. The "ABI tag" contains just the
>> > information needed to distinguish between ABI-incompatible versions.
>> >
>> > Note that "/usr/lib64/pkgconfig/pyldb-util36.pc" does not include all
>> > relevant information: the "m" in "cpython-36m-x86_64-linux-gnu" encodes
>> > build-time configuration; "dm" would mean a debug build with
>> > incompatible ABI.
>> > See [PEP 3149] if you want details.
>> >
>> > Using the ABI tag as-is essentially puts the burden of distinguishing
>> > ABI incompatibilities on Python developers/distributors. It's also quite
>> > easy to ask Python for it [0], so, hopefully, buildsystems of packages
>> > that depend on the utils can be made to support this.
>> >
>> > That said, I'm not familiar with pkg-config or multilib packaging
>> > outside of Python, or across distros. This is the Python way to do it;
>> > please adjust if it makes sense.
>> >
>> >
>> > Another note on the original design intent: generally, the "extra
>> > python" needs to be version 3, and the "normal" Python could be either 2
>> > *or* a different version of 3. It's never been tested with py3 as the
>> > non-extra Python, and I'm sure it doesn't actually work now, but at some
>> > point it'll be necessary to make the "non-extra" Python be Python 3,
>> > preferably with the relevant name mangling. Please don't assume that
>> > "not extra-python" implies Python 2.
>> >
>> >
>> > [PEP 3149]: https://www.python.org/dev/peps/pep-3149/
>> > [0] import sysconfig; sysconfig.get_config_var('SOABI')
>>
>> Thanks for the details.  If you could work with Lumir and Andreas to
>> propose something that follows the above and meets Lumir's requirements
>> that would be great.  We can't be the first package to have produced a
>> python C binding, surely there is some guidance already?
>
>Samba is not the first package with a Python binding, but those "_util"
>libraries are quite unique.
>
>General guidance for providing a C API for Python extension modules is
>provided in Python documentation [0]: roughly, put the function pointers (and
>a version) in a struct, wrap it in a Capsule object, and provide that as a
>Python-level object.
>To use this, there's a PyCapsule_Import helper that imports a module, fetches
>the attribute, and unwraps it to get the struct.
>
>This solves problems with naming/discovery/linking of C-level utilities:
>everything is found via the extension module.
>But in Samba's case, it would require changes to all code that currently uses
>the _util libraries.
>
>
>[0] https://docs.python.org/3/extending/extending.html#providing-a-c-api-for-an-extension-module
>
I think LDVERSION (+ fallback to VERSION) should work well in this case
and does not contain architecture or platform in the string,
 which is fine for dynamic library.

  sh$ python3
  Python 3.6.2 (default, Sep  1 2017, 12:03:48)
  [GCC 7.1.1 20170802 (Red Hat 7.1.1-7)] on linux
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import sysconfig; sysconfig.get_config_var('SOABI')
  'cpython-36m-x86_64-linux-gnu'
  >>> from distutils.sysconfig import get_config_var
  >>> get_config_var('LDVERSION') or get_config_var('VERSION')
  '3.6m'
 
  sh$ python2
  Python 2.7.13 (default, Aug 16 2017, 12:56:26)
  [GCC 7.1.1 20170802 (Red Hat 7.1.1-7)] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import sysconfig; sysconfig.get_config_var('SOABI')
  >>> from distutils.sysconfig import get_config_var
  >>> get_config_var('LDVERSION') or get_config_var('VERSION')
  '2.7'


Sorry that it took me long time to update patches.

LS

short_names_v3.patches (26K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCHES] Generate shorter name for extra python files

Samba - samba-technical mailing list
On 09/13/2017 04:06 PM, Lukas Slebodnik wrote:

> On (29/08/17 10:35), Petr Viktorin via samba-technical wrote:
>> On 08/27/2017 09:02 AM, Andrew Bartlett wrote:
>>> On Tue, 2017-08-22 at 11:25 +0200, Petr Viktorin via samba-technical
>>> wrote:
>>>>
>>>> I'm sorry for missing this thread; my vacation-mode filtering is too eager.
>>>>
>>>> As for the original design, "extra python" allows building two versions
>>>> of Python by a single `make` invocation. Building for both 2 and 3 at
>>>> once was a requirement to get the Python3 patches in.
>>>> To make this work, I've reused the mechanism Python uses to allow binary
>>>> files for multiple versions to co-exist. The "ABI tag" contains just the
>>>> information needed to distinguish between ABI-incompatible versions.
>>>>
>>>> Note that "/usr/lib64/pkgconfig/pyldb-util36.pc" does not include all
>>>> relevant information: the "m" in "cpython-36m-x86_64-linux-gnu" encodes
>>>> build-time configuration; "dm" would mean a debug build with
>>>> incompatible ABI.
>>>> See [PEP 3149] if you want details.
>>>>
>>>> Using the ABI tag as-is essentially puts the burden of distinguishing
>>>> ABI incompatibilities on Python developers/distributors. It's also quite
>>>> easy to ask Python for it [0], so, hopefully, buildsystems of packages
>>>> that depend on the utils can be made to support this.
>>>>
>>>> That said, I'm not familiar with pkg-config or multilib packaging
>>>> outside of Python, or across distros. This is the Python way to do it;
>>>> please adjust if it makes sense.
>>>>
>>>>
>>>> Another note on the original design intent: generally, the "extra
>>>> python" needs to be version 3, and the "normal" Python could be either 2
>>>> *or* a different version of 3. It's never been tested with py3 as the
>>>> non-extra Python, and I'm sure it doesn't actually work now, but at some
>>>> point it'll be necessary to make the "non-extra" Python be Python 3,
>>>> preferably with the relevant name mangling. Please don't assume that
>>>> "not extra-python" implies Python 2.
>>>>
>>>>
>>>> [PEP 3149]: https://www.python.org/dev/peps/pep-3149/
>>>> [0] import sysconfig; sysconfig.get_config_var('SOABI')
>>>
>>> Thanks for the details.  If you could work with Lumir and Andreas to
>>> propose something that follows the above and meets Lumir's requirements
>>> that would be great.  We can't be the first package to have produced a
>>> python C binding, surely there is some guidance already?
>>
>> Samba is not the first package with a Python binding, but those "_util"
>> libraries are quite unique.
>>
>> General guidance for providing a C API for Python extension modules is
>> provided in Python documentation [0]: roughly, put the function pointers (and
>> a version) in a struct, wrap it in a Capsule object, and provide that as a
>> Python-level object.
>> To use this, there's a PyCapsule_Import helper that imports a module, fetches
>> the attribute, and unwraps it to get the struct.
>>
>> This solves problems with naming/discovery/linking of C-level utilities:
>> everything is found via the extension module.
>> But in Samba's case, it would require changes to all code that currently uses
>> the _util libraries.
>>
>>
>> [0] https://docs.python.org/3/extending/extending.html#providing-a-c-api-for-an-extension-module
>>
>
> I think LDVERSION (+ fallback to VERSION) should work well in this case
> and does not contain architecture or platform in the string,
>   which is fine for dynamic library.
>
>    sh$ python3
>    Python 3.6.2 (default, Sep  1 2017, 12:03:48)
>    [GCC 7.1.1 20170802 (Red Hat 7.1.1-7)] on linux
>    Type "help", "copyright", "credits" or "license" for more information.
>    >>> import sysconfig; sysconfig.get_config_var('SOABI')
>    'cpython-36m-x86_64-linux-gnu'
>    >>> from distutils.sysconfig import get_config_var
>    >>> get_config_var('LDVERSION') or get_config_var('VERSION')
>    '3.6m'
>  
>    sh$ python2
>    Python 2.7.13 (default, Aug 16 2017, 12:56:26)
>    [GCC 7.1.1 20170802 (Red Hat 7.1.1-7)] on linux2
>    Type "help", "copyright", "credits" or "license" for more information.
>    >>> import sysconfig; sysconfig.get_config_var('SOABI')
>    >>> from distutils.sysconfig import get_config_var
>    >>> get_config_var('LDVERSION') or get_config_var('VERSION')
>    '2.7'
>
>
> Sorry that it took me long time to update patches.
This looks reasonable to me, two nitpicks:

The resulting filename is "libpyldb-util36m.so.1", with no separator
before the tag. Shouldn't there be one?
If just "-" is used as a separator, the tag can't be a number otherwise
the buildsystem apparently considers it a version. I suggest adding
either "-py" in front, or using "-cp" (short for "CPython") to match
[PEP 425].

And, env['PYTHON_SO_ABI_FLAG'] is no longer necessary.


[PEP 425]: https://www.python.org/dev/peps/pep-0425/#abi-tag

--
Petr Viktorin

0001-Separate-the-Python-ABI-flag-from-the-rest-of-the-so.patch (1K) Download Attachment
0002-Remove-PYTHON_SO_ABI_FLAG.patch (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCHES] Generate shorter name for extra python files

Samba - samba-technical mailing list
On (19/09/17 14:30), Petr Viktorin via samba-technical wrote:

>On 09/13/2017 04:06 PM, Lukas Slebodnik wrote:
>> On (29/08/17 10:35), Petr Viktorin via samba-technical wrote:
>> > On 08/27/2017 09:02 AM, Andrew Bartlett wrote:
>> > > On Tue, 2017-08-22 at 11:25 +0200, Petr Viktorin via samba-technical
>> > > wrote:
>> > > >
>> > > > I'm sorry for missing this thread; my vacation-mode filtering is too eager.
>> > > >
>> > > > As for the original design, "extra python" allows building two versions
>> > > > of Python by a single `make` invocation. Building for both 2 and 3 at
>> > > > once was a requirement to get the Python3 patches in.
>> > > > To make this work, I've reused the mechanism Python uses to allow binary
>> > > > files for multiple versions to co-exist. The "ABI tag" contains just the
>> > > > information needed to distinguish between ABI-incompatible versions.
>> > > >
>> > > > Note that "/usr/lib64/pkgconfig/pyldb-util36.pc" does not include all
>> > > > relevant information: the "m" in "cpython-36m-x86_64-linux-gnu" encodes
>> > > > build-time configuration; "dm" would mean a debug build with
>> > > > incompatible ABI.
>> > > > See [PEP 3149] if you want details.
>> > > >
>> > > > Using the ABI tag as-is essentially puts the burden of distinguishing
>> > > > ABI incompatibilities on Python developers/distributors. It's also quite
>> > > > easy to ask Python for it [0], so, hopefully, buildsystems of packages
>> > > > that depend on the utils can be made to support this.
>> > > >
>> > > > That said, I'm not familiar with pkg-config or multilib packaging
>> > > > outside of Python, or across distros. This is the Python way to do it;
>> > > > please adjust if it makes sense.
>> > > >
>> > > >
>> > > > Another note on the original design intent: generally, the "extra
>> > > > python" needs to be version 3, and the "normal" Python could be either 2
>> > > > *or* a different version of 3. It's never been tested with py3 as the
>> > > > non-extra Python, and I'm sure it doesn't actually work now, but at some
>> > > > point it'll be necessary to make the "non-extra" Python be Python 3,
>> > > > preferably with the relevant name mangling. Please don't assume that
>> > > > "not extra-python" implies Python 2.
>> > > >
>> > > >
>> > > > [PEP 3149]: https://www.python.org/dev/peps/pep-3149/
>> > > > [0] import sysconfig; sysconfig.get_config_var('SOABI')
>> > >
>> > > Thanks for the details.  If you could work with Lumir and Andreas to
>> > > propose something that follows the above and meets Lumir's requirements
>> > > that would be great.  We can't be the first package to have produced a
>> > > python C binding, surely there is some guidance already?
>> >
>> > Samba is not the first package with a Python binding, but those "_util"
>> > libraries are quite unique.
>> >
>> > General guidance for providing a C API for Python extension modules is
>> > provided in Python documentation [0]: roughly, put the function pointers (and
>> > a version) in a struct, wrap it in a Capsule object, and provide that as a
>> > Python-level object.
>> > To use this, there's a PyCapsule_Import helper that imports a module, fetches
>> > the attribute, and unwraps it to get the struct.
>> >
>> > This solves problems with naming/discovery/linking of C-level utilities:
>> > everything is found via the extension module.
>> > But in Samba's case, it would require changes to all code that currently uses
>> > the _util libraries.
>> >
>> >
>> > [0] https://docs.python.org/3/extending/extending.html#providing-a-c-api-for-an-extension-module
>> >
>>
>> I think LDVERSION (+ fallback to VERSION) should work well in this case
>> and does not contain architecture or platform in the string,
>>   which is fine for dynamic library.
>>
>>    sh$ python3
>>    Python 3.6.2 (default, Sep  1 2017, 12:03:48)
>>    [GCC 7.1.1 20170802 (Red Hat 7.1.1-7)] on linux
>>    Type "help", "copyright", "credits" or "license" for more information.
>>    >>> import sysconfig; sysconfig.get_config_var('SOABI')
>>    'cpython-36m-x86_64-linux-gnu'
>>    >>> from distutils.sysconfig import get_config_var
>>    >>> get_config_var('LDVERSION') or get_config_var('VERSION')
>>    '3.6m'
>>    sh$ python2
>>    Python 2.7.13 (default, Aug 16 2017, 12:56:26)
>>    [GCC 7.1.1 20170802 (Red Hat 7.1.1-7)] on linux2
>>    Type "help", "copyright", "credits" or "license" for more information.
>>    >>> import sysconfig; sysconfig.get_config_var('SOABI')
>>    >>> from distutils.sysconfig import get_config_var
>>    >>> get_config_var('LDVERSION') or get_config_var('VERSION')
>>    '2.7'
>>
>>
>> Sorry that it took me long time to update patches.
>
>This looks reasonable to me, two nitpicks:
>
>The resulting filename is "libpyldb-util36m.so.1", with no separator before
>the tag. Shouldn't there be one?
>If just "-" is used as a separator, the tag can't be a number otherwise the
>buildsystem apparently considers it a version. I suggest adding either "-py"
>in front, or using "-cp" (short for "CPython") to match [PEP 425].
>
Sounds reasonable.

>And, env['PYTHON_SO_ABI_FLAG'] is no longer necessary.
>
>
>[PEP 425]: https://www.python.org/dev/peps/pep-0425/#abi-tag
>

Thank you for comments and review.

>From 15c791b2ceeabab1fc938d6aab85c8c5e2313776 Mon Sep 17 00:00:00 2001
>From: Petr Viktorin <[hidden email]>
>Date: Tue, 19 Sep 2017 14:00:32 +0200
>Subject: [PATCH 1/2] Separate the Python ABI flag from the rest of the so name
>
>Signed-off-by: Petr Viktorin <[hidden email]>
>---
I hope you don't mind that I squashed this patch to the 1st one in patchset.

LS

short_names_v4.patches (28K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCHES] Generate shorter name for extra python files

Samba - samba-technical mailing list
Thanks for sending in the new patch set.

My concern is that by having the same python ABI file in samba, it will
break with:

#if PY_MAJOR_VERSION < 3

static void py_cobject_talloc_free(void *ptr)
{
        talloc_free(ptr);
}

_PUBLIC_ PyObject *pytalloc_CObject_FromTallocPtr(void *ptr)
{
        if (ptr == NULL) {
                Py_RETURN_NONE;
        }
        return PyCObject_FromVoidPtr(ptr, py_cobject_talloc_free);
}

#endif

from pytalloc-util.c

If you build with python 2 that will not be included in the ABI, and if
you build with python3 it will.

That is why we need distinct ABI files for python 2 and python 3.

Andrew Bartlett

--
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team         https://samba.org
Samba Development and Support, Catalyst IT  
https://catalyst.net.nz/services/samba





Reply | Threaded
Open this post in threaded view
|

Re: [PATCHES] Generate shorter name for extra python files

Samba - samba-technical mailing list
On (21/09/17 09:27), Andrew Bartlett wrote:

>Thanks for sending in the new patch set.
>
>My concern is that by having the same python ABI file in samba, it will
>break with:
>
>#if PY_MAJOR_VERSION < 3
>
>static void py_cobject_talloc_free(void *ptr)
>{
>        talloc_free(ptr);
>}
>
>_PUBLIC_ PyObject *pytalloc_CObject_FromTallocPtr(void *ptr)
>{
>        if (ptr == NULL) {
>                Py_RETURN_NONE;
>        }
>        return PyCObject_FromVoidPtr(ptr, py_cobject_talloc_free);
>}
>
>#endif
>
>from pytalloc-util.c
>
>If you build with python 2 that will not be included in the ABI, and if
>you build with python3 it will.
>
>That is why we need distinct ABI files for python 2 and python 3.
>
I do not have a problem with distinct ABI files for python 2 and python 3.
Initial version of patches didn't contain it. Files were merged
per requests in previous mails. Or did I misunderstand something?

LS

Reply | Threaded
Open this post in threaded view
|

Re: [PATCHES] Generate shorter name for extra python files

Samba - samba-technical mailing list
On (21/09/17 22:38), Lukas Slebodnik via samba-technical wrote:

>On (21/09/17 09:27), Andrew Bartlett wrote:
>>Thanks for sending in the new patch set.
>>
>>My concern is that by having the same python ABI file in samba, it will
>>break with:
>>
>>#if PY_MAJOR_VERSION < 3
>>
>>static void py_cobject_talloc_free(void *ptr)
>>{
>>        talloc_free(ptr);
>>}
>>
>>_PUBLIC_ PyObject *pytalloc_CObject_FromTallocPtr(void *ptr)
>>{
>>        if (ptr == NULL) {
>>                Py_RETURN_NONE;
>>        }
>>        return PyCObject_FromVoidPtr(ptr, py_cobject_talloc_free);
>>}
>>
>>#endif
>>
>>from pytalloc-util.c
>>
>>If you build with python 2 that will not be included in the ABI, and if
>>you build with python3 it will.
>>
>>That is why we need distinct ABI files for python 2 and python 3.
>>
>I do not have a problem with distinct ABI files for python 2 and python 3.
>Initial version of patches didn't contain it. Files were merged
>per requests in previous mails. Or did I misunderstand something?
>
Actually,
I cannot see any problem with current patches and
pytalloc_CObject_FromTallocPtr

sh# objdump -T /usr/lib64/libpytalloc-util.so.2.1.10 | grep CObject
0000000000000000      DF *UND*  0000000000000000              PyCObject_FromVoidPtr
0000000000001820 g    DF .text  0000000000000024  PYTALLOC_UTIL_2.0.6 pytalloc_CObject_FromTallocPtr
sh# objdump -T /usr/lib64/libpytalloc-util-cp36m.so.2.1.10 | grep CObject
sh# echo $?
1

Andrew,
Could you provide me steps to reproduce your case?

LS

Reply | Threaded
Open this post in threaded view
|

Re: [PATCHES] Generate shorter name for extra python files

Samba - samba-technical mailing list
On Fri, 2017-09-22 at 00:07 +0200, Lukas Slebodnik wrote:

> On (21/09/17 22:38), Lukas Slebodnik via samba-technical wrote:
> > On (21/09/17 09:27), Andrew Bartlett wrote:
> > > Thanks for sending in the new patch set.
> > >
> > > My concern is that by having the same python ABI file in samba, it will
> > > break with:
> > >
> > > #if PY_MAJOR_VERSION < 3
> > >
> > > static void py_cobject_talloc_free(void *ptr)
> > > {
> > >         talloc_free(ptr);
> > > }
> > >
> > > _PUBLIC_ PyObject *pytalloc_CObject_FromTallocPtr(void *ptr)
> > > {
> > >         if (ptr == NULL) {
> > >                 Py_RETURN_NONE;
> > >         }
> > >         return PyCObject_FromVoidPtr(ptr, py_cobject_talloc_free);
> > > }
> > >
> > > #endif
> > >
> > > from pytalloc-util.c
> > >
> > > If you build with python 2 that will not be included in the ABI, and if
> > > you build with python3 it will.
> > >
> > > That is why we need distinct ABI files for python 2 and python 3.
> > >
> >
> > I do not have a problem with distinct ABI files for python 2 and python 3.
> > Initial version of patches didn't contain it. Files were merged
> > per requests in previous mails. Or did I misunderstand something?
> >
>
> Actually,
> I cannot see any problem with current patches and
> pytalloc_CObject_FromTallocPtr
>
> sh# objdump -T /usr/lib64/libpytalloc-util.so.2.1.10 | grep CObject
> 0000000000000000      DF *UND*  0000000000000000              PyCObject_FromVoidPtr
> 0000000000001820 g    DF .text  0000000000000024  PYTALLOC_UTIL_2.0.6
> pytalloc_CObject_FromTallocPtr
> sh# objdump -T /usr/lib64/libpytalloc-util-cp36m.so.2.1.10 | grep CObject
> sh# echo $?
> 1
>
> Andrew,
> Could you provide me steps to reproduce your case?

I'm talking about the files in Samba's ABI/ subdirectories.  

I misunderstood, I thought there was only going to be one ABI
file.  However while there are now python3 ABI files I see two
problems:

 - The name (the cp3m bit needs to become .py3 again)
lib/talloc/ABI/pytalloc-util-cp35m-2.1.10.sigs

 - The vscript:
cat bin/default/lib/talloc/pytalloc-util-cp35m.vscript | grep CObject
                pytalloc_CObject_FromTallocPtr;

This contains references to CObject_FromTallocPtr

This suggests to me that there is a cross-leakage between the python2
and python3 ABI stuff.

Andrew Bartlett

--
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba


Reply | Threaded
Open this post in threaded view
|

Re: [PATCHES] Generate shorter name for extra python files

Samba - samba-technical mailing list
On (23/09/17 08:16), Andrew Bartlett wrote:

>On Fri, 2017-09-22 at 00:07 +0200, Lukas Slebodnik wrote:
>> On (21/09/17 22:38), Lukas Slebodnik via samba-technical wrote:
>> > On (21/09/17 09:27), Andrew Bartlett wrote:
>> > > Thanks for sending in the new patch set.
>> > >
>> > > My concern is that by having the same python ABI file in samba, it will
>> > > break with:
>> > >
>> > > #if PY_MAJOR_VERSION < 3
>> > >
>> > > static void py_cobject_talloc_free(void *ptr)
>> > > {
>> > >         talloc_free(ptr);
>> > > }
>> > >
>> > > _PUBLIC_ PyObject *pytalloc_CObject_FromTallocPtr(void *ptr)
>> > > {
>> > >         if (ptr == NULL) {
>> > >                 Py_RETURN_NONE;
>> > >         }
>> > >         return PyCObject_FromVoidPtr(ptr, py_cobject_talloc_free);
>> > > }
>> > >
>> > > #endif
>> > >
>> > > from pytalloc-util.c
>> > >
>> > > If you build with python 2 that will not be included in the ABI, and if
>> > > you build with python3 it will.
>> > >
>> > > That is why we need distinct ABI files for python 2 and python 3.
>> > >
>> >
>> > I do not have a problem with distinct ABI files for python 2 and python 3.
>> > Initial version of patches didn't contain it. Files were merged
>> > per requests in previous mails. Or did I misunderstand something?
>> >
>>
>> Actually,
>> I cannot see any problem with current patches and
>> pytalloc_CObject_FromTallocPtr
>>
>> sh# objdump -T /usr/lib64/libpytalloc-util.so.2.1.10 | grep CObject
>> 0000000000000000      DF *UND*  0000000000000000              PyCObject_FromVoidPtr
>> 0000000000001820 g    DF .text  0000000000000024  PYTALLOC_UTIL_2.0.6
>> pytalloc_CObject_FromTallocPtr
>> sh# objdump -T /usr/lib64/libpytalloc-util-cp36m.so.2.1.10 | grep CObject
>> sh# echo $?
>> 1
>>
>> Andrew,
>> Could you provide me steps to reproduce your case?
>
>I'm talking about the files in Samba's ABI/ subdirectories.  
>
>I misunderstood, I thought there was only going to be one ABI
>file.  However while there are now python3 ABI files I see two
>problems:
>
> - The name (the cp3m bit needs to become .py3 again)
>lib/talloc/ABI/pytalloc-util-cp35m-2.1.10.sigs
>
> - The vscript:
>cat bin/default/lib/talloc/pytalloc-util-cp35m.vscript | grep CObject
> pytalloc_CObject_FromTallocPtr;
>
>This contains references to CObject_FromTallocPtr
>
But as i showed you in previous example it is not a problem that
symbol is in version script because it is not part of util library
for python3. Version script is used as a whitelist to allow symbols from
objects *.o appear in library. But if symbol is missing in object files
it will not be part of library even though it is in version script.


https://www.akkadia.org/drepper/dsohowto.pdf
Section 2.2 says:
  The concept of export maps is to tell the linker explicitly
  which symbols to export from the generated object.

And it is much simpler to use version script(export maps)
then __attribute__ ((visibility ("hidden")))


But if you prefer to have separate version script(ABI/*) for python2
and python3 I do not have a problem with that. I can just see a small
benefit in de-duplication.

LS

Reply | Threaded
Open this post in threaded view
|

Re: [PATCHES] Generate shorter name for extra python files

Samba - samba-technical mailing list
On Fri, 2017-09-22 at 22:49 +0200, Lukas Slebodnik wrote:

>
> But as i showed you in previous example it is not a problem that
> symbol is in version script because it is not part of util library
> for python3. Version script is used as a whitelist to allow symbols from
> objects *.o appear in library. But if symbol is missing in object files
> it will not be part of library even though it is in version script.
>
>
> https://www.akkadia.org/drepper/dsohowto.pdf
> Section 2.2 says:
>   The concept of export maps is to tell the linker explicitly
>   which symbols to export from the generated object.
>
> And it is much simpler to use version script(export maps)
> then __attribute__ ((visibility ("hidden")))
>
>
> But if you prefer to have separate version script(ABI/*) for python2
> and python3 I do not have a problem with that. I can just see a small
> benefit in de-duplication.

The ABI files are not just for the version scripts.  They are also used
to determine if a symbol is removed or the signature changed, and
triggers logic to reject such a build.

If there is just one ABI file for both, then if python2 requires a
different ABI to python3, then either on the other will fail a build.

However, we don't want ABI files for each possible python3 version (35m
etc), just for python3 overall, hence the .py3 thing.

I hope this helps clarify things.

Andrew Bartlett

--
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba


12