Quantcast

Cross-compiling with 4.1 ?

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Cross-compiling with 4.1 ?

Jeremy Allison
Hi all,

In our haste to remove the autoconf build (for
which I'm as guilty as anyone else) we may
have lost something important - that's the
ability to cross-compile 4.1.0 and above.

Has anyone done this with master/4.1.0 ?

Does it work ? I know there are several
OEMs who absolutely depend on this to
create working file server builds.

Jeremy.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cross-compiling with 4.1 ?

Thiago Fernandes Crepaldi
Hello Jeremy,

since 4.0.3, I've had to change my build environment for using chroot +
qemu user emulation to compile it for arm. I had too many troubles trying
to use cross compilers on 4.0.x, but I havent tried cross compile 4.1.x
yet, though.


On Thu, Aug 22, 2013 at 6:41 PM, Jeremy Allison <[hidden email]> wrote:

> Hi all,
>
> In our haste to remove the autoconf build (for
> which I'm as guilty as anyone else) we may
> have lost something important - that's the
> ability to cross-compile 4.1.0 and above.
>
> Has anyone done this with master/4.1.0 ?
>
> Does it work ? I know there are several
> OEMs who absolutely depend on this to
> create working file server builds.
>
> Jeremy.
>



--
Thiago Fernandes Crepaldi (aka Crepaldi)
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cross-compiling with 4.1 ?

David Disseldorp-2
In reply to this post by Jeremy Allison
Hi Jeremy,

On Thu, 22 Aug 2013 14:41:28 -0700
Jeremy Allison <[hidden email]> wrote:

> Hi all,
>
> In our haste to remove the autoconf build (for
> which I'm as guilty as anyone else) we may
> have lost something important - that's the
> ability to cross-compile 4.1.0 and above.
>
> Has anyone done this with master/4.1.0 ?
>
> Does it work ? I know there are several
> OEMs who absolutely depend on this to
> create working file server builds.

Tridge added wafsamba cross compilation support in 2010 via the
--cross-execute and --cross-answers options. Both are documented at:
        http://wiki.samba.org/index.php/Waf#cross-compiling

I've not tested Waf cross compilation personally, but as the saying
goes... ;-)

Cheers, David
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cross-compiling with 4.1 ?

abartlet
In reply to this post by Thiago Fernandes Crepaldi
On Thu, 2013-08-22 at 19:08 -0300, Thiago Fernandes Crepaldi wrote:
> Hello Jeremy,
>
> since 4.0.3, I've had to change my build environment for using chroot +
> qemu user emulation to compile it for arm. I had too many troubles trying
> to use cross compilers on 4.0.x, but I havent tried cross compile 4.1.x
> yet, though.

The code hasn't changed, essentially, for 4.1 compared with 4.0 in this
respect.

The primary issue, as far as I'm aware, is that the python version on
both build and target absolutely have to match, because currently the
python configure tests are run directly in the python interpreter, not
via the --cross-answers stuff (which everything else uses, or at least
everything should use).

There may well be other issues:  I don't know of anybody doing a deep
investigation into this since the initial support was added.

Andrew Bartlett

--
Andrew Bartlett
http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org
Samba Developer, Catalyst IT                   http://catalyst.net.nz


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cross-compiling with 4.1 ?

Jeremy Allison
On Fri, Aug 23, 2013 at 03:22:13PM +1200, Andrew Bartlett wrote:

> On Thu, 2013-08-22 at 19:08 -0300, Thiago Fernandes Crepaldi wrote:
> > Hello Jeremy,
> >
> > since 4.0.3, I've had to change my build environment for using chroot +
> > qemu user emulation to compile it for arm. I had too many troubles trying
> > to use cross compilers on 4.0.x, but I havent tried cross compile 4.1.x
> > yet, though.
>
> The code hasn't changed, essentially, for 4.1 compared with 4.0 in this
> respect.

Yeah, but 4.0 still has the autoconf code, and 4.1 doesn't :-).
That's the big change.

> The primary issue, as far as I'm aware, is that the python version on
> both build and target absolutely have to match, because currently the
> python configure tests are run directly in the python interpreter, not
> via the --cross-answers stuff (which everything else uses, or at least
> everything should use).
>
> There may well be other issues:  I don't know of anybody doing a deep
> investigation into this since the initial support was added.

I'm getting complaints it's not working, so I'll dig a little
deeper.

Jeremy.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cross-compiling with 4.1 ?

abartlet
On Thu, 2013-08-22 at 20:26 -0700, Jeremy Allison wrote:

> On Fri, Aug 23, 2013 at 03:22:13PM +1200, Andrew Bartlett wrote:
> > On Thu, 2013-08-22 at 19:08 -0300, Thiago Fernandes Crepaldi wrote:
> > > Hello Jeremy,
> > >
> > > since 4.0.3, I've had to change my build environment for using chroot +
> > > qemu user emulation to compile it for arm. I had too many troubles trying
> > > to use cross compilers on 4.0.x, but I havent tried cross compile 4.1.x
> > > yet, though.
> >
> > The code hasn't changed, essentially, for 4.1 compared with 4.0 in this
> > respect.
>
> Yeah, but 4.0 still has the autoconf code, and 4.1 doesn't :-).
> That's the big change.

Sure, I wasn't trying to minimize the issue - the reverse, I was trying
to avoid the false hope that the build system fairies might have fixed
it for 4.1 :-)

> > The primary issue, as far as I'm aware, is that the python version on
> > both build and target absolutely have to match, because currently the
> > python configure tests are run directly in the python interpreter, not
> > via the --cross-answers stuff (which everything else uses, or at least
> > everything should use).
> >
> > There may well be other issues:  I don't know of anybody doing a deep
> > investigation into this since the initial support was added.
>
> I'm getting complaints it's not working, so I'll dig a little
> deeper.

Thanks,

Andrew Bartlett

--
Andrew Bartlett
http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org
Samba Developer, Catalyst IT                   http://catalyst.net.nz


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cross-compiling with 4.1 ?

Jeremy Allison
On Fri, Aug 23, 2013 at 03:30:45PM +1200, Andrew Bartlett wrote:

> On Thu, 2013-08-22 at 20:26 -0700, Jeremy Allison wrote:
> > On Fri, Aug 23, 2013 at 03:22:13PM +1200, Andrew Bartlett wrote:
> > > On Thu, 2013-08-22 at 19:08 -0300, Thiago Fernandes Crepaldi wrote:
> > > > Hello Jeremy,
> > > >
> > > > since 4.0.3, I've had to change my build environment for using chroot +
> > > > qemu user emulation to compile it for arm. I had too many troubles trying
> > > > to use cross compilers on 4.0.x, but I havent tried cross compile 4.1.x
> > > > yet, though.
> > >
> > > The code hasn't changed, essentially, for 4.1 compared with 4.0 in this
> > > respect.
> >
> > Yeah, but 4.0 still has the autoconf code, and 4.1 doesn't :-).
> > That's the big change.
>
> Sure, I wasn't trying to minimize the issue - the reverse, I was trying
> to avoid the false hope that the build system fairies might have fixed
> it for 4.1 :-)

Hahahaha ! I *love* that phrase :-). "the build system fairies..."

I'll have to use that in a talk sometime :-).
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cross-compiling with 4.1 ?

Christophe Vu-Brugier
In reply to this post by Jeremy Allison
Dear Jeremy Allison,

On Thu, 22 Aug 2013 14:41:28 -0700, Jeremy Allison wrote :
> In our haste to remove the autoconf build (for
> which I'm as guilty as anyone else) we may
> have lost something important - that's the
> ability to cross-compile 4.1.0 and above.
>
> Has anyone done this with master/4.1.0 ?
>
> Does it work ?

Two months ago, I tried to cross compile samba 4.0.7 and it did not
work. I looked a little bit why Samba 4 did not cross-compile but did
not manage to make it build fully. Here is what I saw:

 * some programs are built for the target whereas they are expected to
   be built for the host. This is the case for some heimdal binaries.

 * in buildtools/wafadmin/Tools/python.py, I think that the
   PYTHONXCPREFIX variable should be used if the user defines it. This
   is necessary in order to build python packages for the target.

There are probably other errors occuring when cross-compiling Samba 4.
The errors I mention are just the first that I encountered.


> I know there are several
> OEMs who absolutely depend on this to
> create working file server builds.

My employer (LaCie) is such an OEM. We are sticking with Samba 3.6
because of the cross-compilation issues of Samba 4.0.

I am a bit busy at the moment, but I will try to cross-compile the
latest version of Samba, look more deeply at the errors and come up with
patches. I think the first step is to open a bug report.

Best regards,

--
Christophe Vu-Brugier


LaCie will welcome you at IBC Amsterdam (13-17 Sept) on booth 7.G17 (Hall7). Come and visit us.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cross-compiling with 4.1 ?

abartlet
On Mon, 2013-09-02 at 11:19 +0200, Christophe Vu-Brugier wrote:

> Dear Jeremy Allison,
>
> On Thu, 22 Aug 2013 14:41:28 -0700, Jeremy Allison wrote :
> > In our haste to remove the autoconf build (for
> > which I'm as guilty as anyone else) we may
> > have lost something important - that's the
> > ability to cross-compile 4.1.0 and above.
> >
> > Has anyone done this with master/4.1.0 ?
> >
> > Does it work ?
>
> Two months ago, I tried to cross compile samba 4.0.7 and it did not
> work. I looked a little bit why Samba 4 did not cross-compile but did
> not manage to make it build fully. Here is what I saw:
>
>  * some programs are built for the target whereas they are expected to
>    be built for the host. This is the case for some heimdal binaries.

This doesn't surprise me, and is the biggest challenge in
cross-compilation.

>  * in buildtools/wafadmin/Tools/python.py, I think that the
>    PYTHONXCPREFIX variable should be used if the user defines it. This
>    is necessary in order to build python packages for the target.
>
> There are probably other errors occuring when cross-compiling Samba 4.
> The errors I mention are just the first that I encountered.
>
>
> > I know there are several
> > OEMs who absolutely depend on this to
> > create working file server builds.
>
> My employer (LaCie) is such an OEM. We are sticking with Samba 3.6
> because of the cross-compilation issues of Samba 4.0.

Given the autoconf build is still supported in 4.0, you should at least
be able to move to that.  Of course, we do need to fix this for you to
move to 4.1.

> I am a bit busy at the moment, but I will try to cross-compile the
> latest version of Samba, look more deeply at the errors and come up with
> patches. I think the first step is to open a bug report.

Thanks.

Andrew Bartlett

--
Andrew Bartlett
http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org
Samba Developer, Catalyst IT                   http://catalyst.net.nz


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cross-compiling with 4.1 ?

abartlet
In reply to this post by Christophe Vu-Brugier
On Mon, 2013-09-02 at 11:19 +0200, Christophe Vu-Brugier wrote:

> Dear Jeremy Allison,
>
> On Thu, 22 Aug 2013 14:41:28 -0700, Jeremy Allison wrote :
> > In our haste to remove the autoconf build (for
> > which I'm as guilty as anyone else) we may
> > have lost something important - that's the
> > ability to cross-compile 4.1.0 and above.
> >
> > Has anyone done this with master/4.1.0 ?
> >
> > Does it work ?
>
> Two months ago, I tried to cross compile samba 4.0.7 and it did not
> work. I looked a little bit why Samba 4 did not cross-compile but did
> not manage to make it build fully. Here is what I saw:
>
>  * some programs are built for the target whereas they are expected to
>    be built for the host. This is the case for some heimdal binaries.

Exactly which binaries are wrong, and are there examples of others that
are right that we could use as an example to fix this?

>  * in buildtools/wafadmin/Tools/python.py, I think that the
>    PYTHONXCPREFIX variable should be used if the user defines it. This
>    is necessary in order to build python packages for the target.
>
> There are probably other errors occuring when cross-compiling Samba 4.
> The errors I mention are just the first that I encountered.
>
>
> > I know there are several
> > OEMs who absolutely depend on this to
> > create working file server builds.
>
> My employer (LaCie) is such an OEM. We are sticking with Samba 3.6
> because of the cross-compilation issues of Samba 4.0.
>
> I am a bit busy at the moment, but I will try to cross-compile the
> latest version of Samba, look more deeply at the errors and come up with
> patches. I think the first step is to open a bug report.

Did you get anywhere with patches?

Thanks,

Andrew Bartlett

--
Andrew Bartlett
http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org
Samba Developer, Catalyst IT                   http://catalyst.net.nz


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cross-compiling with 4.1 ?

Christophe Vu-Brugier
Dear Andrew Bartlett,

On Wed, 09 Oct 2013 16:10:50 +1300, Andrew Bartlett wrote :

> On Mon, 2013-09-02 at 11:19 +0200, Christophe Vu-Brugier wrote:
>
> [snip]
>
> > Two months ago, I tried to cross compile samba 4.0.7 and it did not
> > work. I looked a little bit why Samba 4 did not cross-compile but did
> > not manage to make it build fully. Here is what I saw:
> >
> >  * some programs are built for the target whereas they are expected to
> >    be built for the host. This is the case for some heimdal binaries.
>
> Exactly which binaries are wrong, and are there examples of others that
> are right that we could use as an example to fix this?

Some binaries are built with use_hostcc=True, for instance in
source4/heimdal_build. These binaries are "asn1_compile" and
"compile_et". Unfortunately, setting use_hostcc to True does nothing
because the logic to use the host compiler is missing from Samba's
buildtools.

 

> >  * in buildtools/wafadmin/Tools/python.py, I think that the
> >    PYTHONXCPREFIX variable should be used if the user defines it. This
> >    is necessary in order to build python packages for the target.
> >
> > There are probably other errors occuring when cross-compiling Samba 4.
> > The errors I mention are just the first that I encountered.
> >
> >
> > > I know there are several
> > > OEMs who absolutely depend on this to
> > > create working file server builds.
> >
> > My employer (LaCie) is such an OEM. We are sticking with Samba 3.6
> > because of the cross-compilation issues of Samba 4.0.
> >
> > I am a bit busy at the moment, but I will try to cross-compile the
> > latest version of Samba, look more deeply at the errors and come up with
> > patches. I think the first step is to open a bug report.
>
> Did you get anywhere with patches?

Not really, I have some ugly patches that allow me to pass the
configure step, but the build is still failing. However, I have
gathered more information about what prevents Samba 4.1 from being
cross compiled.

1/ Python. Samba is looking for the system python libraries and
   headers. We need to add a way to specify an alternative python
   path to a cross-compiled version of python.
   For that, we can get inspiration from distutils-cross

   https://bitbucket.org/lambacck/distutilscross/src

2/ C includes. The build passes -I/usr/include -I/usr/local/include to
   gcc. However, passing host system headers is wrong when
   cross-compiling. Moreover, I think there is no need to do that
   because if you are building for the host, the compiler looks for
   headers in /usr/include and /usr/local/include (see `cpp -v`).

3/ Host tools that are built for the target which I mentioned above.

Best regards,

--
Christophe Vu-Brugier
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cross-compiling with 4.1 ?

Jeremy Allison
In reply to this post by abartlet
On Tue, Sep 03, 2013 at 09:09:30AM +1200, Andrew Bartlett wrote:
>
> Given the autoconf build is still supported in 4.0, you should at least
> be able to move to that.  Of course, we do need to fix this for you to
> move to 4.1.

The lack of cross compiling for 4.1.x is becoming a serious
impediment for OEMs for Samba 4.1.x.

I've flagged bug:

https://bugzilla.samba.org/show_bug.cgi?id=10128

as a critical regression.

Do we have anyone on the Team who is a waf expert
and willing to step up and fix this ?

If not the only alternative will be to restore
autoconf functionality for building smbd and the
other file server components in the next release
of 4.1.x.

If we can't fix this it is clear that we prematurely
replaced autoconf without having a replacement for
all of the parts that Samba users depend on, and
we must revisit that decision.

Jeremy.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cross-compiling with 4.1 ?

Kai Blin-4
On 15/10/13 21:12, Jeremy Allison wrote:

> On Tue, Sep 03, 2013 at 09:09:30AM +1200, Andrew Bartlett wrote:
>>
>> Given the autoconf build is still supported in 4.0, you should at least
>> be able to move to that.  Of course, we do need to fix this for you to
>> move to 4.1.
>
> The lack of cross compiling for 4.1.x is becoming a serious
> impediment for OEMs for Samba 4.1.x.
>
> I've flagged bug:
>
> https://bugzilla.samba.org/show_bug.cgi?id=10128
>
> as a critical regression.
>
> Do we have anyone on the Team who is a waf expert
> and willing to step up and fix this ?

I'm comfortable enough with waf, I guess, but I don't know much about
cross-compiling. However, if anybody can point me at how to set up a
cross-compile environment for autoconf in 4.0, I'd be willing to give it
a shot.

> If not the only alternative will be to restore
> autoconf functionality for building smbd and the
> other file server components in the next release
> of 4.1.x.

And you're saying that we have a team member who could fix
cross-compilation problems on the autoconf build? Maybe that person
would be a good candidate to work with me to get cross-compiles into waf.

Cheers,
Kai

--
Kai Blin
Worldforge developer http://www.worldforge.org/
Wine developer http://wiki.winehq.org/KaiBlin
Samba team member http://www.samba.org/samba/team/
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cross-compiling with 4.1 ?

Sander Stoks
In reply to this post by Jeremy Allison

On Oct 16, 2013, at 8:00 PM, Kai Blin wrote:

Do we have anyone on the Team who is a waf expert
and willing to step up and fix this ?

I'm comfortable enough with waf, I guess, but I don't know much about cross-compiling. However, if anybody can point me at how to set up a cross-compile environment for autoconf in 4.0, I'd be willing to give it a shot.

I have a setup using ptxdist, maybe there's some way we can work together?  My setup is part of a rather large build (the entire distro we're making for our embedded system is built with it) so probably overkill in this case, but if there are specific things I can do to test things out, I'd love to help.

Kind regards,
Sander Stoks
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cross-compiling with 4.1 ?

Kai Blin-4
On 19/10/13 20:49, Sander Stoks wrote:

Hi Sander,

> I have a setup using ptxdist, maybe there's some way we can work
> together?  My setup is part of a rather large build (the entire
> distro we're making for our embedded system is built with it) so
> probably overkill in this case, but if there are specific things I
> can do to test things out, I'd love to help.

The really annoying part is getting a cross-compile environment set up,
and that's where I'm currently failing. Might be a bit until I get to a
point where I can reliably build a hello world C file for one of my
embedded platforms.

Cheers,
Kai

--
Kai Blin
Worldforge developer http://www.worldforge.org/
Wine developer http://wiki.winehq.org/KaiBlin
Samba team member http://www.samba.org/samba/team/
Loading...