[Scratchbox2] How to use "python setup.py build" under scratchbox

[Lauri T. Aarnio]

Many thanks for the explanation. I did read about the tools root,
but disregarded that as unimportant.

long time ago it was possible to use "devel" without a separate tools
root, I think. But obviously that haven't been tested lately, and on
the other hand, it is a feature which isn't really useful anyway, as
we have learned.

I'll try to explain the history of these more complex mapping modes,

Can you be a bit more specific about what this tools root should
include?
Should this file system include the same libraries and header files
as the target rootfs?

The short answer is that it should contain all executables, libraries
and other files needed by the tools that you select to use as native
binaries. That is typically a subset of programs that you have in the
target filesystem.
But don't forget that some tools need some programs indirectly, i.e.
if a tool is written in perl, then you need to have the complete perl
system also in "tools". Header files should not be needed, but those
won't cause any harm either.

Of course, it isn't that simple in practice - if it was, then you
could live with the "simple" mode as well. "simple" is enough for many
things - maybe 80% of existing software packages can be built with it.
But then there are things that don't work at all, or just produce
incorrect result with that kind of simple approach.

The first thing to keep in mind is the reason why we are trying to use
native tools: If everything was running as emulated binaries (using
Qemu), the system would be awfully slow.

But the second thing is that you can't just replace all tools with
native versions in real life (as the "simple" mode is doing), because
some of those tools really depend on the cpu architecture.

Some specific cases that have presented problems in this sense are
related building extensions for the scripting languages (perl &
python, which are in fact so special cases that the executables need
to exist both in the "target root" and "tools root", and both versions
are used!)

Selecting the set of tools that can be installed to the "tools root"
is a somewhat iterative process... if unsure, having only target's
version in target root is a safe, although slow choice. Sb2 runs it
with Qemu as usual.
One way to go is to start by having just the most common things in the
tools root, and continue by adding programs whenever a time-consuming
program has been identified (tools root can be managed with "sb2 -m
tools -R")

Yet another thing to notice is that there may be more or less hard
bindings between the versions of the tools and some files (config
files, etc) in the source packages that you are trying to compile.
"Autotools" are even too famous in this sense :-(

So, when we had noticed that the "simple" mode is not comprehensive
enough for us, another mapping mode was created - the "maemo" mode,
which was later renamed as "devel". There are several reasons why it
uses tools from a separate directory, but the biggest ones are related
to management of the complete development environment: We must be able
to control what tools and what versions of the tools are available.
Also, this strategy also allowed us to use the SDK on a host that is
not based on debian.

In the early days we took the tools directly from debian, because
Maemo is based on debian. At that time (>3 years ago) we didn't have
all tools available in Maemo's repositories (since they were
separately available for SB1, there had been little need for them in
Maemo itself), and so we installed tools for sb2 directly from debian
repositories.

At least theoretically "devel" can still be used with a set of tools
which does not fully match your target's OS. But I don't recommend that:

Maemo is based on Debian, but it isn't Debian. Being based on
something does not mean that the sources would 100% compatible when
OSS tools requirements are considered. Some source packages have had
really strict requirements considering versions of some tools, in fact
stricter than what is announced in the packaging/control files... we
found out (the hard way, of course) that sometimes debian had things
that were too old, sometimes it had too new versions!

I could probably spend several hours or days telling stories about
unbelievable incompabilities that exists somewhere between open-source
software packages and the tools that they require. Many, if not all of
those stories are also true for the current SB1. But lets skip that,
and go directly to the next step:

At some point we had to admit that the only relatively safe way is to
use "target" and "tools" filesystems that are both based on the
completely same sources. This leaves the risk that some tools are
aware of the CPU architecture, but it eliminates other risks.

Also, a nice consequence was that this allowed to do both
simplifications and extensions to the mapping mode which is used for
development. Such changes also contained some new risks, of course. I
didn't want to break existing systems; That is why the "accel" mode
was born - it is a successor, but not a direct replacement for
"devel" (one example about differences is that "accel" assumes that
contents of /usr/share can be taken from either place, target or tools
root, whereas "devel" tries to solve things with a set of hard-to-
maintain mapping rules)

So, a built-in requirement of "accel" is that the set of tools is
built from the exactly same sources that you are using in the target.
Which is the only configuration that can be recommended anyway, as we
have learned. Alternatively, "devel" can be used, but it should not
offer any advantages over "accel". (In fact, because I don't know any
cases where "accel" would be worse than "devel" if the same-codebase-
for-both requirement is met, I'm planning to drop the aging "devel"
mode from the next major version of SB2)

BTW, sb2 will be using the dynamic linker/loader (ld.so = ld-
linux.so.*) from the tools root in "accel" and "devel" modes. There
are some things that work better if you apply the patches that are
shipped with SB2 and rebuild glibc; see external_patches/README for
details.

Lauri

[Lauri T. Aarnio]

Dear all,
I have now generated a tools root but I think I have done something
wrong because now is the sb2-init command not working correctly.
I generated a qemux86 image with the angstrom online build system
(Narcissus) this image should be basically the same as the image I
have for my beagleboard but then for the i686 architecture.
t /home/hansan/Beagelboard/i86_rootfs/ beagle_angstrom2 /usr/local/
angstrom/arm/bin/arm-angstrom-linux-gnueabi-gcc
Using /usr/local/angstrom/arm/bin/arm-angstrom-linux-gnueabi-gcc to
Reading mode-specific settings..
Finished writing sb2.gcc.config
gcc configured.
sb2-init: Target architecture is 'arm'
sb2-init: Host architecture is 'i[3456]86'
Reading mode-specific settings..
Finished writing sb2.config
cannot open shared object file: No such file or directory
cannot open shared object file: No such file or directory
cannot open shared object file: No such file or directory
Running /home/hansan/sb2/bin/sb2-build-libtool failed
You can run this manually later, otherwise your
sb2 environment is correctly setup and ready to use
Probably there is something wrong with my tools root.

Does your tools root work properly if you chroot to it?

I am also not able to call sb2 /home/hansan/sb2/bin/sb2-build-
libtool successfully.
bin/sb2-build-libtool

You can skip building of that, once you have a working tools root
(i.e. run sb2-init with -n)

I think that the problem is that my tools root fs expect to be a
real root fs and does not expect to reside in a directory. Probably
that breaks some paths to libraries...
Some idea how to fix this, or some tips on how to get a decent tools
root.

Check the glibc patches that are available in external_patches subdir
of sb2's sources, those may help. For example, sometimes RPATH options
in executables have been causing trouble, but the patches will solve
that and other things..

Lauri

[Han Hartgers]

On Thu, Mar 17, 2011 at 1:15 AM, Lauri T. Aarnio

Dear all,

I have now generated a tools root but I think I have done something wrong
because now is the sb2-init command not working correctly.
I generated a qemux86 image with the angstrom online build system
(Narcissus) this image should be basically the same as the image I have for
my beagleboard but then for the i686 architecture.

Removed some text

Probably there is something wrong with my tools root.
Does your tools root work properly if you chroot to it?

Yes my tools root works properly when in chroot in it. All normal programs
and utilities (tar grep less vi man) are working. I can update and upgrade
packages with the opkg package manager and I can compile a "hello world"
program.
But this tools root is a bit limited and based on busybox. I can imagine
that I will run into some new limitations.

I am also not able to call sb2 /home/hansan/sb2/bin/sb2-build-libtool

successfully.
/home/hansan/sb2/bin/sb2-build-libtool

You can skip building of that, once you have a working tools root (i.e. run
sb2-init with -n)

I think that the problem is that my tools root fs expect to be a real root
fs and does not expect to reside in a directory. Probably that breaks some
paths to libraries...
Some idea how to fix this, or some tips on how to get a decent tools root.

Check the glibc patches that are available in external_patches subdir of
sb2's sources, those may help. For example, sometimes RPATH options in
executables have been causing trouble, but the patches will solve that and
other things..
Lauri

I really have to rebuild my tools root to include all patches. This will
take me some time. I will come back to you when I was able to do so.

Greetings,

Han