bild

Daniel Stender

Freier Devops-Engineer

Copyright © 2015–2018
Powered by Hugo
Impressum

Fixing ppc64el chroots for Sbuild

Feb 5, 2016
DebianDebian diarySbuildQemu

Currently there are some problems with using ppc64el chroots for Sbuild. I’ve came across this trying to perform some local test builds for libvigraimpex/1.10.0+git20160120.803d5d4-11. Actually, ppc64el2 could be emulated with Qemu which makes it possible to easily set up a foreign arch base system chroot for the use with the builder Sbuild. But at present there are some bugs in some members of the toolchain which is used for that, which need some workaround to get it running.

Base system chroots for Sbuild resp. Schroot3 – the chroot tool which is used by Sbuild – are set up with the tool sbuild-createchroot. And with that chroot creation tool foreign arch chroots could be easily created by employing qemu-debootstrap4. This tools wraps around debootstrap5, which is the default bootstrapper of sbuild-createchroot. qemu-debootstrap conveniently automatically performs a multi-stage operation with debootstrap for setting up any wanted Debian supported foreign arch chroot6:

$ sudo sbuild-createchroot --arch=ppc64el --debootstrap=qemu-debootstrap --make-sbuild-tarball=/var/lib/sbuild/unstable-ppc64el-sbuild.tar.gz
  unstable `mktemp -d` http://httpredir.debian.org/debian

However, current Debootstrap (1.0.78) has problems with running the second stage out of the freshly created chroot:

I: SUITE: unstable
I: TARGET: /tmp/tmp.HP6CzLphi7
I: MIRROR: http://httpredir.debian.org/debian
I: Running qemu-debootstrap --arch=ppc64el --variant=buildd --verbose --include=fakeroot,build-essential,debfoster --components=main
   --resolve-deps unstable /tmp/tmp.HP6CzLphi7 http://httpredir.debian.org/debian
I: Running command: debootstrap --arch ppc64el --foreign --variant=buildd --verbose --include=fakeroot,build-essential,debfoster --components=main
   --resolve-deps unstable /tmp/tmp.HP6CzLphi7 http://httpredir.debian.org/debian
I: Retrieving Release 
I: Retrieving Release.gpg 
I: Checking Release signature
I: Valid Release signature (key id 126C0D24BD8A2942CC7DF8AC7638D0442B90D010)
I: Retrieving Packages 
I: Validating Packages 
I: Resolving dependencies of required packages...
I: Resolving dependencies of base packages...
I: Found additional required dependencies: adduser dmsetup init-system-helpers insserv libapparmor1 libaudit-common libaudit1 libbz2-1.0 libcap2 libcap2-bin
   libcryptsetup4 libdb5.3 libdebconfclient0 libdevmapper1.02.1 libgcrypt20 libgpg-error0 libkmod2 libncursesw5 libsemanage-common libsemanage1 libsystemd0
   libudev1 libustr-1.0-1 systemd systemd-sysv
I: Found additional base dependencies: binutils bzip2 cpp cpp-5 debian-archive-keyring dpkg-dev g++ g++-5 gcc gcc-5 gnupg gpgv libapt-pkg5.0 libasan2 libatomic1
   libc-dev-bin libc6-dev libcc1-0 libdpkg-perl libfakeroot libgc1c2 libgcc-5-dev libgdbm3 libgmp10 libgomp1 libisl15 libitm1 liblz4-1 libmpc3 libmpfr4
   libperl5.22 libreadline6 libstdc++-5-dev libstdc++6 libubsan0 libusb-0.1-4 linux-libc-dev make patch perl perl-modules-5.22 readline-common xz-utils 
I: Checking component main on http://httpredir.debian.org/debian...
I: Retrieving libacl1 2.2.52-2
I: Validating libacl1 2.2.52-2
I: Retrieving adduser 3.113+nmu3
I: Validating adduser 3.113+nmu3
{...}
I: Extracting liblzma5...
I: Extracting zlib1g...
I: Running command: chroot /tmp/tmp.HP6CzLphi7 /debootstrap/debootstrap --second-stage
I: Keyring file not available at /usr/share/keyrings/debian-archive-keyring.gpg; switching to https mirror https://mirrors.kernel.org/debian
E: Error running qemu-debootstrap at /usr/sbin/sbuild-createchroot line 233.

The log in /tmp/tmp.HP6CzLphi7/debootstrap/debootstrap.log doesn’t recorded anything on this failure7. This problem has been already reported to the bug tracker8, twice9, and concerns any emulated architectures. Downgrading helps with this, I’ve got myself 1.0.75 from the Debian snapshot, build and installed it10.

So, that’s fixed!

But currrently there is a another problem in the way for things running smoothly. There is a problem with Qemu’s ppc64le emulation11 in the foreign chroot, which has been set up by Debootstrap, so that the second stage processing wouldn’t proceed, either:

$ sudo debootstrap --foreign --arch ppc64el unstable ppc64el_unstable http://httpredir.debian.org/debian
{...}
$ sudo cp /usr/bin/qemu-ppc64le-static ppc64el_unstable/usr/bin/
$ sudo DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true LC_ALL=C LANGUAGE=C LANG=C chroot ppc64el_unstable/
  /debootstrap/debootstrap --second-stage
I: Keyring file not available at /usr/share/keyrings/debian-archive-keyring.gpg; switching to https mirror
   https://mirrors.kernel.org/debian

The log shows that there’s a serious problem in the emulation layer:

$ less ppc64el_unstable/debootstrap/debootstrap.log
gpgv: Signature made Thu Feb  4 15:59:40 2016 CET using RSA key ID 
gpgv: Good signature from "Debian Archive Automatic Signing Key (7.0/wheezy) <>"
gpgv: Signature made Thu Feb  4 15:59:40 2016 CET using RSA key ID 2B90D010
gpgv: Good signature from "Debian Archive Automatic Signing Key (8/jessie) <>"
Invalid instruction
NIP 0000000010007190   LR 0000000010007168 CTR 00000040009f2530 XER 0000000000000000 CPU#0
MSR 8000000002806001 HID0 0000000000000000  HF 0000000002806001 idx 0
TB 00014970 64295804327982
GPR00 0000000010007168 0000004000800400 000000001008ab00 00000000100842d8
GPR04 0000004000800428 0000000000000000 0000000000000000 0000000000000000
GPR08 000000000defd33c 0000000056b3a2e0 000000000103969a 0000000000000078
GPR12 00000040009f2530 0000004000b7e750 0000000010084284 000000001008426c
GPR16 00000000100841d8 0000000010083fc8 0000000010083ba0 0000000010084310
GPR20 0000000010084290 0000000010084200 0000000010083ec0 00000000100842c8
GPR24 0000000000000100 0000000010083cd8 0000000000000002 0000000010017ee0
GPR28 0000000000000001 0000000010083f60 0000000000000003 0000000010083fd0
CR   [ E  L  -  -  -  G  G  E  ]             RES ffffffffffffffff
FPR00 41cdcd6500000000 0000000000000000 0000000000000000 0000000000000000
FPR04 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPR08 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPR12 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPR16 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPR20 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPR24 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPR28 0000000000000000 0000000000000000 0000000000000000 0000000000000000
FPSCR 0000000000002000
qemu: uncaught target signal 4 (Illegal instruction) - core dumped
Illegal instruction

I’ve reported this problem today12, and very quick got great replies including an explanation where this failure comes from (great support!). I’m very happy that there’s an easy workaround available in just exporting resp. setting the QEMU_CPU environment variable the right way:

$ sudo QEMU_CPU=POWER8 DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true LC_ALL=C LANGUAGE=C LANG=C chroot ppc64el_unstable/
  /debootstrap/debootstrap --second-stage
I: Keyring file not available at /usr/share/keyrings/debian-archive-keyring.gpg; switching to https mirror https://mirrors.kernel.org/debian
I: Installing core packages...
I: Installing core packages...
I: Unpacking required packages...
I: Unpacking libacl1:ppc64el...
I: Unpacking adduser...
{...}
I: Base system installed successfully.
$ sudo chroot ppc64el_unstable/ /bin/uname -m
ppc64le
$ sudo QEMU_CPU=POWER8 chroot ppc64el_unstable/ apt-get update
Get:1 http://mirror.unitedcolo.de/debian unstable InRelease [278 kB]
Get:2 http://mirror.de.leaseweb.net/debian unstable/main Translation-en [5289 kB]
Fetched 5567 kB in 9s (590 kB/s)
Reading package lists... Done
$

So far, so good.

However, isn’t that easy to integrate this fix into the workflow with sbuild-createchroot because to my experience just exporting QEMU_CPU doesn’t work well for the somewhat higher leveled Sbuild tools13. The best way seems to be to take a root folder which has been created with bare Debootstrap like described above, and to integrate it into the Sbuild/Schroot setup manually. It takes a little work, but runs cleanly afterwards.

First of all, the root folder which has been created should be copied to the side of the other chroots which are already available for Sbuild in /var/lib/sbuild (being either tarballs or directories, file/folder names doesn’t play a role). After that, a proper configuration file is needed in /etc/schroot/chroot.d, you can call it unstable-ppc64el-sbuild or whatever:

[unstable-ppc64el-sbuild]
type=directory
description=Debian unstable/ppc64el
directory=/var/lib/sbuild/ppc64el_unstable
groups=root,sbuild
root-groups=root,sbuild
profile=sbuild

Then the chroot is available for Schroot along with the others, which already have been created on another occasion:

~$ sudo schroot -l
chroot:unstable-amd64-sbuild
chroot:unstable-arm64-sbuild
chroot:unstable-i386-sbuild
chroot:unstable-ppc64el-sbuild
source:unstable-amd64-sbuild
source:unstable-arm64-sbuild
source:unstable-i386-sbuild

The “injection” of the needed environment variable QEMU_CPU could be done with a shell script running from the host, like it’s used for employing Ccache in Sbuild chroots. For that, first it’s needed to make the folder on the host where the script is going to be stored available for the chroot(s). If you want e.g. /usr/local/bin, you need to add this to /etc/schroot/sbuild/fstab:

$ echo "/usr/local/bin /usr/local/bin none rw,bind 0 0" | sudo tee -a /etc/schroot/sbuild/fstab

The needed simple script then just goes like this:

#!/bin/sh
export QEMU_CPU=POWER8
exec "$@"

To use that script as command prefix for everything running inside the chroot it’s needed to give that also to the configuration in /etc/schroot/chroot.d. Finally, Overlayfs could be set as union-type for the chroot to boost the performance a bit (the emulator remains being a bootleneck, though):

[unstable-ppc64el-sbuild]
type=directory
description=Debian unstable/ppc64el
directory=/var/lib/sbuild/unstable-ppc64el
groups=root,sbuild
root-groups=root,sbuild
profile=sbuild
command-prefix=/usr/local/bin/export-qemu-cpu
union-type=overlay

And there you go with the running ppc64el chroot for Sbuild:

$ sudo sbuild-update -udcar unstable-ppc64el-sbuild
unstable-ppc64el-sbuild: Performing update.
Get:1 http://httpredir.debian.org/debian unstable InRelease [278 kB]
Fetched 278 kB in 1s (157 kB/s)    
Reading package lists... Done
unstable-ppc64el-sbuild: Performing dist-upgrade.
Reading package lists... Done
Building dependency tree... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
unstable-ppc64el-sbuild: Performing clean.
unstable-ppc64el-sbuild: Performing autoclean.
Reading package lists... Done
Building dependency tree... Done
unstable-ppc64el-sbuild: Performing autoremove.
Reading package lists... Done
Building dependency tree... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Now this chroot could be used by doing sbuild --dist=unstable --arch=ppc64el.

Happy ppc64el-ing!

  1. [return]
  2. That’s the Debian port. The official designator for the architecture is “ppc64le”: https://en.wikipedia.org/wiki/Ppc64 [return]
  3. [return]
  4. Package: [return]
  5. [return]
  6. [return]
  7. The complaint about the missing keyring file has nothing to do with it. [return]
  8. [return]
  9. [return]
  10. [return]
  11. The concerned emulation binary is qemu-ppc64le-static, package: qemu-user-static. [return]
  12. [return]
  13. Please be careful if you want to experiment with this, there is the danger to face not properly closing chroots and other very ugly things. [return]