A complete Debian package build in detail

Feb 11, 2016
DebianDebian packagingGNU Make

[article not yet completed; 2016-10-27: minor language improvements, a couple of additions, corrected some mistakes, make hyperlinks more user friendly]

Working through the build process of my adopted package , the C++ computer vision library with attached Python bindings, I’ve though if this would be written down that could be interesting for people who want to get (deeper) into Debian packaging. The outline of source files in the upstream tarball isn’t that complicated and could be grasped also by people who are not that acquainted with this particular package¹. The debian/rules preserves “old school” containing explicit individual build targets², like one has to know what they do also when using the modern Debhelper sequencer. And it’s well crafted using a broad spectrum of GNU Make features which are important to have mastered when dealing with advanced packaging challenges³.

Building a Debian package is automatic compilation of sources (if isn’t Architecture: any), processing of documentation, and treatment of files according to the Debian Policy, so that a pre-build C++ library like this could be installed by a single command (we’re taking this for granted nowadays but as a matter of fact the system of contribution which has been developed is quite marvellous). But instead of talking much about that let’s get down into the “machine room” where things are actually happening.

The package build which is going to be discussed is , build on amd64. If you want to study this article, it would be best to have the corresponding debian/rules file lying on the side, otherwise it would be hard to follow.

In order not to exceed the limits also of an extensive blog posting like this, I’m not getting into the Makefiles which are created by CMake during the process⁴, but going to remain on the working level of the debian/rules. Thus, CMake is only sparsely discussed here (maybe I’m going to write a succeeding blog post on that exceeded background of the build).

Also, I’m not going to presents snippets from the extended, more detailled build log which is put out if the environment variable DH_VERBOSE for the Debhelper modules is set to “1”, because that would be too exhaustive. Please examine that log level on your own if you like (you would have to build the package again to get that log).

dpkg-buildpackage

In the sbuild log from the build server in the buildd network there is a lot of things going on to provide the proper build environment like downloading and unpacking the needed packages, but the core package build process begins when the builder dpkg-buildpackage (package: dpkg-dev) gets started:

dpkg-buildpackage: source package libvigraimpex
dpkg-buildpackage: source version 1.10.0+git20160120.803d5d4-1+b1
dpkg-buildpackage: source distribution sid
 dpkg-source --before-build libvigraimpex-1.10.0+git20160120.803d5d4
dpkg-buildpackage: host architecture amd64

The individual steps of which the build process consists are described in the manpage of dpkg-buildpackage(1).

First, a couple of standard build environment variables like are being checked if and how they are set (probably in debian/rules itself).

Then, the init hook runs (which could be used to inject commands), and dpkg-source --before-build is executed, which runs the corresponding hook of the source package, see dpkg-source(1). But with our package, there is nothing to be done here.

dpkg-buildpackage then checks if the build-dependencies can be satisfied, meaning if the needed packages are present in the Debian setup which is used to build the packages. Building in a chroot with Sbuild when correct build deps are given in debian/control, this test always passes when nothing happened while getting them, which have to be fixed separately. However, this test could be passed for whatever reason by giving -d to dpkg-buildpackage⁵.

debian/rules clean

Then, another hook (preclean) is called, and after that the clean procedure gets triggered by calling the debian/rules clean target:

 fakeroot debian/rules clean
dh_testdir
dh_clean
rm -rf obj*/
rm -rf doc/

Cleaning now is for removing all build artifacts and to restore the unvaried source package folder if another build process has taken place before.

Some targets in debian/rules need to get called by root for the ownership of generated files in the created packages, and fakeroot is commonly used to emulate UID/GID 0⁶. A hint: on the command line, you could also do dpkg-buildpackage -Tclean to clean up, or use the tool debclean for doing that.

The clean target in our package goes like this (it’s defined below, at the very end of debian/rules):

.PHONY: clean
clean:
        dh_testdir
        dh_clean
        rm -rf obj*/
        rm -rf doc/

.PHONY is a of GNU Make which saves target names which aren’t actually products (“phony” targets) from producing errors if files of the same names actually exist. Like if you have a project which includes a directory test and where the Makefile has a target of the same name: if you call make test you are getting an error, if .PHONY isn’t set for this target. Although clean (and the other phony targets in debian/rules) doesn’t exists in the root directory of the source package, it’s elegant style to always protect the targets with a corresponding .PHONY.

The Debhelper module dh_testdir is for testing if the directory is correct among other things, and it’s put at the beginning of each new build target, see dh_testdir(1) for information on this.

dh_clean removes things down a list of common artifacts plus the files which are given in debian/clean. Although this module is capable of removing complete folders recursively like rm -r does (when a folder is given with a wildcard in debian/clean), this is archieved here by adding two wanted folder removals to the clean target in debian/rules.

As a side note, if the Debhelper sequencer is used (see dh(1) manpage) evoking the clean target also triggers dh_auto_clean to be run, which seeks for the appropriate command for the used build system like make clean or python setup.py clean. But that’s not needed here because the Makefiles which are generated by Cmake aren’t used for cleaning.

It’s generally desired for good source packages that there are no build artifacts after the build has completed, which aren’t covered by the clean configuration to get automatically removed. This leads to an error on that the present files doesn’t match the content of the upstream tarball which prevent a consecutive build process to complete, which isn’t wanted. However, in a clean chroot since no previous build has taken place, the pre-build cleaning process doesn’t find anything to remove, if the clean target configuration isn’t faulty. Although chroot based building always provides fresh build environments for every build and dumps them afterwards, the maintainer has to check that the clean target works flawless. The clean target must undo any effects that the build and binary targets have had.

debian/rules build

The next thing which happens in the build is that dpkg-buildpackage runs another hook (the source hook) and builds the source package by running dpkg-source -b⁷, if a binary-only build hasn’t been triggered (by one of the options -b, -B or -A, see below).

It then runs the build hook and calls debian/rules . The build targets are to be found in debian/rules like this:

.PHONY: build build-arch build-indep

build: build-arch build-indep

build-indep:
        # Nothing to do in build-indep, "thanks" to bug #374029.

build-arch:
    dh_testdir
        echo DEBUGMEMSIZE ; if which free ; then free ; fi

If build targets are written out like here and the Debhelper sequencer isn’t used (which covers them all by its universal target %:), build-arch and build-indep are required individually (the latter remains empty here, though)⁸, while build (also required) performs the complete configuration and compilation of the package by recursively calling the other ones as prerequisites⁹.

If no special build flavour is chosen for dpkg-buildpakage (build-arch: -B, build-indep: -A), it calls build by default (or by giving -b without building a source package before). Please note that in the build log from the buildd server the target build-arch is explicitly selected: on the build server for amd64 of course the arch dependent build target is wanted.

And, .PHONY targets could be accumulated like it’s written here by a previous maintainer. That’s just a matter of style, others use them individually above each target.

build-arch

After the build or the build-arch target has been triggered, first again dh_testdir is run to check over if things are like expected.

After that Debhelper module the CLI tool free is run to print out the memory size of the build system:

 debian/rules build-arch
dh_testdir
echo DEBUGMEMSIZE ; if which free ; then free ; fi
DEBUGMEMSIZE
/usr/bin/free
              total        used        free      shared  buff/cache   available
Mem:                                  
Swap:                

The call of free is protected with a shell test which checks whether the tool is available on the build system before running it to avoid an error if it isn’t. free is part of the package procps (containing tools for quering the proc/ filesystem), which usually is installed. But the check here for it being actually available with which is doing no harm (which returns boolean “True” if that tool returns anything, so that the if-condition is fulfilled).

By the way, echo could have to prevent that the line is printed out when run by GNU Make.

obj/CMakeCache.txt

Now, cmake is executed to create the needed build file:

    $(MAKE) -f debian/rules \
                obj/CMakeCache.txt \

obj/CMakeCache.txt:
        mkdir -p $(dir $(@))
        cd $(dir $(@)) && cmake .. $(cmake_options) \
                -DPYTHON_EXECUTABLE=/usr/bin/python \
                -DWITH_VIGRANUMPY=0

The target obj/CMakeCache.txt in debian/rules (to be found below the build-arch paragraph) is called recursively with make -f/--file.

In Makefiles, the is commonly used instead of running make directly to be more flexible if GNU Make alternatives are going to be used.

Another detail, GNU Make lines can be broken with a backslash to make things more readable.

This target creates the directory obj in the build path with mkdir, where the C++ library is going to be build into. The build-in (“automatic”) of Make (which can be shortened to $@) always represents the name of the actual target, thus obj/CMakeCache.txt.

And the extracts the leading directory path out of that, thus obj/. By the way, the complete expression $(dir $(@)) resp. $(dir $@) can be shortened to: .

By using variables it’s only needed to change the build target if something other than obj is wanted as build directory, that’s flexible and rugged style which is always recommended. To support any build directories -p/--parent is given for mkdir, which isn’t really needed for obj/, because there is only one additional folder level involved.

Then it got changed into $(dir $(@)) with cd and cmake is then invoked on the parent directory (which is the root directory of the unpacked source package) to create the needed build setup.

These operations are combined by the conditional execution operator &&, by that the consecutive command is only executed if the preceding command succeeded (by a returning exit status of zero). It’s needed like that because every line in the Makefile gets running in a separate subshell, so the cd command and cmake .. couldn’t be put into separate lines.

Cmake then runs with expanded $(cmake_options) and the two extra arguments to create the build setup into obj¹⁰. In the build log, the lines discussed expand like this (some line breaks added here):

mkdir -p obj/
cd obj/ && cmake .. -DCMAKE_INSTALL_PREFIX=/usr
-DCMAKE_VERBOSE_MAKEFILE=ON
-DWITH_OPENEXR=ON
-DCMAKE_C_FLAGS_RELEASE="-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -pipe -Wdate-time -D_FORTIFY_SOURCE=2"
-DCMAKE_CXX_FLAGS_RELEASE="-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -pipe -Wdate-time -D_FORTIFY_SOURCE=2"
-DCMAKE_SHARED_LINKER_FLAGS_RELEASE="-Wl,-z,relro -Wl,--as-needed" \
        -DPYTHON_EXECUTABLE=/usr/bin/python \
        -DWITH_VIGRANUMPY=0

The Python interpreter is needed to build the documentation later in binary-indep, therefore the path to the PYTHON_EXECUTEABLE is given to Cmake here, although in this directory the Python package is not going to be build (that is said by setting WITH_VIGRANUMPY to zero).

The variable $(cmake_options) (this name is arbitrary) was defined above in debian/rules:

cmake_options = \
    -DCMAKE_INSTALL_PREFIX=/usr \
    -DCMAKE_VERBOSE_MAKEFILE=ON \
    -DWITH_OPENEXR=ON \
    -DCMAKE_C_FLAGS_RELEASE="$(CFLAGS)" \
    -DCMAKE_CXX_FLAGS_RELEASE="$(CXXFLAGS)" \
    -DCMAKE_SHARED_LINKER_FLAGS_RELEASE="$(LDFLAGS)"

buildflags

The variables for the settings for the C compiler ($(CFLAGS)), the C++ compiler ($(CXXFLAGS)), and the linker ($(LDFLAGS)), which are set here to pass the wanted settings into the build setup have been defined further above, right at the head of debian/rules:

CFLAGS := $(shell env DEB_CFLAGS_MAINT_APPEND=-pipe dpkg-buildflags --get CFLAGS; dpkg-buildflags --get CPPFLAGS)
CXXFLAGS := $(shell env DEB_CXXFLAGS_MAINT_APPEND=-pipe dpkg-buildflags --get CXXFLAGS; dpkg-buildflags --get CPPFLAGS)
LDFLAGS := $(shell env DEB_LDFLAGS_MAINT_APPEND=-Wl,--as-needed dpkg-buildflags --get LDFLAGS)

Let’s examine these GNU Make constructions from the inside to the outside:

Current standard buildflags on Debian systems can be always put out by dpkg-buildflags. When the Debhelper sequencer isn’t used (compat level nine automatically passes them by dh_auto_configure) they need to get pulled into Makefile variables manually, like it happens here. Individual buildflag sets can be queried by dpkg-buildflags --get separately for the compiler, the linker, for Objective C etc.

Standard are the inclusion of GDB debugging symbols (-g, they get stripped out then later into the debugging package, see below) and the compilation with optimization level -O2.

A warning on the use of __DATE__ and __TIME__ and __TIMESTAMP__ preprocessor macros is given for the preprocessor (-Wdate-time) to support the reproducible builds project¹¹. In the future this is going to be changed into an error.

To set build flags for was a release goal for Debian Jessie, so -fstack-protector-strong and -Wformat -Werror=format-security are set for the compilers, -D_FORTIFY_SOURCE=2 is set for the preprocessor, and -Wl,-z,relro for the linker.

Unfortunately, the preprocessor flags ($(CPPFLAGS)) can not be given to Cmake like the other ones¹². But they are just added here to the compiler flags, that’s an easy workaround.

The GNU Make is for the expansion of single commands.

The environment variables of the type DEB__MAINT_APPEND could be used to set additional flags (see dpkg-buildflags(1)). env here runs dpkg-buildflags over them to add -pipe (use pipes instead of temporary files ) to the compiler flags.

And the option -Wl,--as-needed (to link only what’s actually needed) is added to the linker flags that way.

The Makefile variables could be called as wanted, but there is no reasonable cause to choose different names for them. Anyway, := is the normal (immediately evaluated) variable assignment operator in the Make language.

cmake then runs and creates the build setup. Exhaustive information is put out while this happens, please see the build log for details.

obj.python%/CMakeCache.txt

After that process has been completed, the same operation is performed to create the build setup for the Python bindings package into a different build folder.

The full Makefile line as a matter of fact doesn’t only recursively calls the target obj/CMakeCache.txt like already discussed above, but at the same time also another target obj.python%/CMakeCache.txt which actually is separated in debian/rules by another line break with a backslash:

    $(MAKE) -f debian/rules \
                obj/CMakeCache.txt \
                $(patsubst %,obj.%/CMakeCache.txt,$(shell pyversions -r))

obj.python%/CMakeCache.txt:
        mkdir -p $(dir $(@))
        cd $(dir $(@)) && \
            CXXFLAGS=-fno-strict-aliasing cmake .. $(cmake_options) $(call cmake_python_options,$(*))

More stuff for GNU Make gourmets:

This target includes a percent character which is called “stem”. That is a placeholder for any number of any character, and the usage of that makes this target flexible to call.

In build-arch above, the target which is to be called together with resp. after obj/CMakeCache.txt is generated by the function $(patsubst ,,). With that, % in the string obj.%/CMakeCache.txt gets replace by the shell output of pyversions -r.

pyversions -r prints out a list of all supported Python 2 versions at that time, and in Debian unstable that currently is python2.7 (the default supported Python versions are always available through the python-all packages). The actual target which is called then is obj.python2.7/CMakeCache.txt, and obj.python%/CMakeCache.txt matches that.

The build log then continues like this:

mkdir -p obj.python2.7/

The reason for doing things that way is to be able to build for several supported Python versions in the same build, if they are supported (as a side note, py3versions -r currently puts out: python3.4 python3.5). That’s also the reason for that the Python package of Vigra is build into a separate folder here.

After this folder has been created, cmake gets running again to create the build setup for the Python bindings package (again, some line breaks added here for convenience):

CXXFLAGS=-fno-strict-aliasing cmake .. -DCMAKE_INSTALL_PREFIX=/usr
-DCMAKE_VERBOSE_MAKEFILE=ON
-DWITH_OPENEXR=ON
-DCMAKE_C_FLAGS_RELEASE="-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -pipe -Wdate-time -D_FORTIFY_SOURCE=2"
-DCMAKE_CXX_FLAGS_RELEASE="-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -pipe -Wdate-time -D_FORTIFY_SOURCE=2"
-DCMAKE_SHARED_LINKER_FLAGS_RELEASE="-Wl,-z,relro -Wl,--as-needed"
-DPYTHON_EXECUTABLE=/usr/bin/python2.7
-DPYTHON_INCLUDE_DIR=/usr/include/python2.7/
-DPYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython2.7.so
-DBoost_PYTHON_LIBRARY_RELEASE=/usr/lib/x86_64-linux-gnu/libboost_python-py27.so

$(cmake_options) again are given for this run of cmake (they are needed to build the C++ extensions for the Python bindings), but by $(call cmake_python_options,$(*)) also the $(cmake_python_options) are also added, they are also defined above in debian/rules (the name cmake_python_options is also arbitrary):

cmake_python_options = \
    -DPYTHON_EXECUTABLE=/usr/bin/python$(1) \
    -DPYTHON_INCLUDE_DIR=/usr/include/python$(1)/ \
    -DPYTHON_LIBRARY=$(wildcard /usr/lib/libpython$(1).so /usr/lib/$(DEB_HOST_MULTIARCH)/libpython$(1).so) \
    -DBoost_PYTHON_LIBRARY_RELEASE=$(wildcard /usr/lib/libboost_python-py$(subst .,,$(1)).so /usr/lib/$(DEB_HOST_MULTIARCH)/libboost_python-py$(subst .,,$(1)).so)

By using the Make the variable $(1) in $(cmake_python_options) (which is the first parameter of $(call)) gets expanded to $*. And that designates the stem (%) of the current target (being “2.7”). Thus, e.g. /usr/bin/python$(1) for DPYTHON_EXECUTABLE expands into /usr/bin/python2.7, when $(cmake_python_options) is evaluated for the cmake run (for that reason the lazy assignment operator = is used, here)¹³. The same happens for the other uses of $(1), like it could be seen in the build log output.

Another Make function which is used here is , which returns pattern only if filenames do exist in the filesystem. Thus for example /usr/lib/x86_64-linux-gnu/libpython2.7.so for PYTHON_LIBRARY gets returned ($(DEB_HOST_MULTIARCH) on amd64 expands to x86_64-linux-gnu).

And for the search for the location of the Boost extension also $(subst) is again used for getting 27 out of $(1) being 2.7.

On the side, it’s rather unnecessary to seek for other locations as the ones which are getting returned by the two uses of $(wildcard) here, though.

One more thing, configure targets could have been installed for these operations which create the build setups with cmake.

custom checks

The function $(wildcard) is also used for the integration of libhdf5 into the build:

ifneq ($(wildcard /usr/lib/$(DEB_HOST_MULTIARCH)/hdf5/serial/libhdf5.so),)
  export CMAKE_INCLUDE_PATH=/usr/include/hdf5/serial
  export CMAKE_LIBRARY_PATH=/usr/lib/$(DEB_HOST_MULTIARCH)/hdf5/serial
endif

If the build dependency against libhdf5-dev in debian/control can’t be fulfilled the build breaks before dpkg-buildpackage starts, and therefore if it got this this point the library is going to be available anyway, if there isn’t any other problem with the build environment which needs to be fixed separately. But extra checking for whether the library (and not the package) is really available in the build setup before Cmake is told to include it does no harm.

The Make operator ifneq here verifies that the return of $(wildcard) isn’t equal to “empty value”, and then exports the needed import paths into two special variables for Cmake. If the import succeeded could be rechecked in the log of the cmake run (Found HDF5: /usr/lib/x86_64-linux-gnu/hdf5/serial/libhdf5.so).

The syntax of that is ifneq (arg1, arg2) or ifneq 'arg1' 'arg2' (there are some other slightly different variations possible).

In obj.python%/CMakeCache.txt, it follows a custom extra shell based check if deps for the Python package are available for the build:

if ! grep VIGRANUMPY_DEPENDENCIES $(@) > /dev/null; then \
        echo "Error: Numpy for Python $* not found" >&2; \
        rm -f $(@); \
        exit 1; \
fi

If grep VIGRANUMPY_DEPENDENCIES obj.python2.7/CMakeCache.txt ($(@) resp. $@ always represents the actual target) returns nothing (in plain English: “if grep doesn’t puts something out on /dev/null”), then an error message is put out to stderr (>&2), the target gets removed (rm is customary extra protected with -f: ignore nonexistent files), and the make process is exited with an error level of 1, and that means the build would break completely¹⁴. These are no GNU Make, but pure shell scripting matters.

This target for the configuration of the build of the Python package then completes with a shell based manipulation of the created configuration files:

 # We don't want to link with libpythonX.Y.
 # We don't want SONAMES (except vigranumpycore.so)
 find $(dir $(@))/vigranumpy/ -name 'link.txt' -exec sed -i \
        -e 's/ -lpython[0-9.]\+ / /g' \
        -e '/-Wl,-soname,vigranumpycore/b' \
        -e 's/ -Wl,-soname,[^ ]\+ / /' \
        {} +

find seeks all configurations files in vigranumpy/ with the name link.txt (that returns 16 hits) in the build path and performs a couple of changes on them using sed -i/--in-place¹⁵. Like said above, I’m not going deeper into CMake stuff here, but if you want to examine a fresh build outline you could perform like $ debian/rules obj.python2.7/CMakeCache.txt (do quilt push -a first) in the source package root and browse around in the new folder. These files contain the actual command lines commands which are run for the compilation of the extensions.

In these files, (in this environment) needless build flags like -lpython2.7 (the regular expression[0-9.]\+ means “one or more occurrences of either 0-9 or .”) are removed (by replacing them with “empty value” with a search-and-replace s/foo//g). In the same way, flags like -soname,sampling.so get removed (the regex [^ ] means “anything other than empty space”, hence: “any letter or digit”).

The checks for VIGRANUMPY_DEPENDENCIES and the makeover operations are echoed in the buildlog. This target completes with them.

building

In build-arch, Make proceeds to the next line:

        $(MAKE) -f debian/rules \
                doc/changelog \
                obj/build-stamp \
                $(patsubst %,obj.%/build-stamp,$(shell pyversions -r))

That are three more targets which are called recursively and at the same time, and which make executes down consecutively. First, doc/changelog:

doc/changelog: docsrc/credits_changelog.dxx
        sh debian/convert-changelog.sh < $(<) > $(@)

The upstream tarball doesn’t ship a plain text changelog, but a HTML encoded one in the file docsrc/credits_changelog.dxx. The target uses the custom script debian/convert-changelog.sh for the conversion of that into a plain text doc/changelog. That upstream changelog, which is wanted but preferred being in , is going to copied into all single packages later.

We haven’t yet covered the meaning of $(<) (it could be also written as: $<), which is the GNU Make variable for the first (resp. the only one) prerequisite of the current target.

The next target, obj/build-stamp then actually performs the build:

obj/build-stamp: obj/CMakeCache.txt
        $(MAKE) -C $(dir $(@))
        chmod a-x vigranumpy/test/*.py # nosetest would skip executable *.py
        touch $(@)

obj/CMakeCache.txt is given as prerequisite because that target needs to have being executed (creating the build setup) before the build can happen.

CMake generated several inter-depending Makefiles in the build path, and the main one gets running by initiating make -C .

Get yourself a coffee, the library needs a while to build.

The following line is a little bit strange and I would guess that it got inserted by a later hand: it’s not good style to alter the files of the upstream tarball by operations like chmod, that should be always done when they have been copied into build or install paths. On the top of that, that’s not the build target for testing here.

This target completes using the which employs touch to mark the otherwise as being completed..

The following target for the building of the Python modules then goes like this:

obj.python%/build-stamp: obj.python%/CMakeCache.txt obj/build-stamp
        cd obj && find . -name '*.o' -o -name '*.so' -o -name '*.so.*' | xargs cp -v --preserve=links --parents -t ../$(dir $(@))
        $(MAKE) -C $(dir $(@))
        touch $(@)

The compiled objects of the C++ library are needed to build the Python bindings and in a normal, non-Debian build setup the Python package wouldn’t be build separately, so they would be already present in place. Because they are not here in this special setup which provides probably multiple Python builds in separate directories, they need to get copied before the Python modules could be build if you don’t want them being build multiple times. That’s what happening here by running find to search for in obj/ and to copy them. Please check out the build log for details on what’s actually gets copied by here.

After the needed objects are made available by copying, the build of the Python modules gets started. That empty target then also gets stamped by touch.

tests

The build-arch target then proceeds with running the tests on the stuff which has been build just before:

ifeq ($(filter nocheck,$(DEB_BUILD_OPTIONS)),)
  ifneq (,$(filter $(DEB_HOST_ARCH_CPU),powerpc))
        : # run the tests, but ignore test results
        -$(MAKE) -k -f debian/rules \
                obj/test-stamp \
                $(patsubst %,obj.%/test-stamp,$(shell pyversions -r))
  else
        $(MAKE) -k -f debian/rules \
                obj/test-stamp \
                $(patsubst %,obj.%/test-stamp,$(shell pyversions -r))
  endif
endif

This is another conditional execution here which checks with ifeq if probably nocheck has been set in $(DEB_BUILD_OPTIONS), which should prevent the tests of getting run. That switch could be used to save time if you need a quick build for whatever reason¹⁶, and and wouldn’t be needed. The Debhelper module dh_auto_test notices nocheck exits automatically, but here that’s needed to set up.

The string analysis is used for that, which returns everything in text which matches pattern. The construction ifeq ($(filter nocheck,$(DEB_BUILD_OPTIONS)),) therefore gets boolean true and proceeds if $(filter) doesn’t puts out anything (which means nocheck isn’t to be found in $(DEB_BUILD_OPTIONS)), which is equal to “empty value”.

But there’s another, a nested condition here which checks for if the build architecture (which is available in $(DEB_HOST_ARCH_CPU)) is powerpc, which also is quite strange. The filter function here isn’t only reverted non like the other occasions in this rules files, and unnecessary complex, because ifeq ($(DEB_HOST_ARCH_CPU),powerpc) would have done that. And, there’s no reason to query $(DEB_HOST_ARCH_CPU) over just $(DEB_HOST_ARCH). Furthermore, it’s rather $(DEB_BUILD_ARCH_CPU) (building on) than HOST_ARCH (building for) what is wanted here (failsaving the tests on that arch). And, the : in the comment line isn’t correct. These lines are definitely not from the guy who wrote the majority of this debian/rules. This whole condition is going to be removed because it’s currently not needed.

Anyway, the failsafe of the testsuite is achieved by putting - in front of $(MAKE), which prevents the build to break if an exit with an error level beyond zero (thanks for Dmitry Smirnov to pointing me to this feature).

To put -k/--keep-going for make makes the whole test suite run through completely also when a single test fails (the build breaks after completion, then).

Get yourself just another coffee, a big mug. That mother really gets tested through.

The targets being called to get the testsuites into action are going like this:

obj/test-stamp: obj/build-stamp
        sed -i 's/exit\ 1/exit\ 0/' obj/test/impex/run_test_impex.sh # temporarily failsafe
        $(MAKE) check -C $(dir $(@))
        touch $(@)

obj.python%/test-stamp:: obj.python%/build-stamp
        $(MAKE) vigranumpytest -C $(dir $(@))
        touch $(@)

In the generated main Makefiles, the targets check and vigranumpytest are for running the test suites completely (by the way, you can examine the individual build targets by typing $ make in the build directories, that puts them all out. The cmake output is very fine grained: there are 217 targets).

The failsafe of the single test impex/run_test_impex.sh has been installed by me for this package revision, because there are currently some minor problems with processing of EXR images, which are needed to get deeper into¹⁷.

binary-arch

Now the C++ library and the Python modules have been build and tested out, and we’re a substantial way down the 136 pages of the printed out buildlog (Iceweasel print preview, 80% scale).

After build-arch has been completed, dpkg-buildpackage automatically calls the binary hook followed by fakeroot debian/rules binary (or binary-arch for the architecture dependent build procedure on the buildd server). This target could be individually set into action by starting $ dpkg-buildpackage -G.

The binary targets contains all that is necessary for the user to build the binary packages. binary itself is again a target without commands which only depends on binary-arch and binary-indep:

.PHONY: binary binary-arch binary-indep

binary: binary-arch binary-indep

binary-arch: build-arch
        dh_testdir
        dh_testroot
            $(MAKE) -f debian/rules \
                        obj/install-stamp \
                        $(patsubst %,obj.%/install-stamp,$(shell pyversions -r))

binary-arch now runs all things which are needed to create the architecture dependent binary packages of the libvigraimpex source package¹⁸. build-arch is given a prerequisite for that target because it just can’t be packed what not have been build before.

Again, the Debhelper module dh_testdir initiates the target, but here dh_testroot is also wanted here to check over if the target got run as root (resp. using fakeroot, see above), because files in the binary packages get UID/GID 0 (remember that you have to use sudo for apt and dpkg).

The install target for both, the shared library and the Python package is one and the same, because there are no different things to do for both:

%/install-stamp: %/build-stamp
        $(MAKE) -C $(dir $(@)) install DESTDIR=$(CURDIR)/debian/tmp/
        touch $(@)

For the call of this target on the stuff in the Python build folder, the Make function is used in binary-arch. With that, the stem % in obj.%/install-stamp gets replaced by that with the output of $(shell pyversions -r). For python2.7 that means the called target is obj.python2.7/install-stamp, and that calls the target %/install-stamp, which has %/build-stamp as prerequisite, which expands to obj.python2.7/build-stamp.

Like explained above, this Make programming style is open for the case multiple Python versions would be supported at the same time while build depending on python-all-dev (or, in upcoming package versions, python3-all-dev).

The target install in the cmake generated main Makefile here appears after -C , but that has no particular effect.

First the C++ library gets installed into debian/tmp by giving $(DESTDIR), then the Python modules are installed the same way. $(CURDIR) always refers to the to the source folder root, to use that makes the path more robust. The debian/tmp is commonly used in packaging for collecting stuff¹⁹, and one advantage of that approach is that it gets covered by dh_clean automatically.

dh_install -a

The next thing which is worked down by binary-arch is to run the Debhelper module dh_install. Like all the other modules in this target, -a is given to act only on the architecture dependend packages (this is a shared Debhelper option which is described in the manpage debhelper(1)).

dh_install queries .install files in debian/, and in this source package there is debian/libvigraimpex5v5.install, debian/libvigraimpex-dev.install and debian/python-vigra.install to be found. These registers contain lines on which files are going to be installed into the binaries packages, and at this stage in the build process that means that they are copied from debian/tmp into individual debian/ folders. For example, libvigraimpex5v5.install just contains /usr/lib/lib*.so.*, and this copies the shared library debian/tmp/usr/lib/libvigraimpex.so.5.1.10.0 intodebian/libvigraimpex5v5/usr/lib` (I’am cheating a bit here, there was a package/SONAME discrepancy with this build).

These modules usually don’t have any particular output in the build log, but only in the more verbose build log which is generated if the build got run with the environment variable DH_VERBOSE set to 1. Like written above, if you would like to examine what’s happening “under the hood” with these modules, you would have to generate such a log on your own. Or, you can also run the modules individually like $ DH_VERBOSE=1 fakeroot dh_install -a in the source root to inspect what they’re doing (ideally, after a completed non-chroot build).

dh_installdocs -a

The next Debhelper module which is employed is dh_installdocs -a. The documentations don’t have been build yet, only the C++ library and the Python modules, but this module copies things like debian/copyright and debian/README.Debian into the right place (/usr/share/doc/) in the arch dependent package directories²⁰. The register debian/libvigraimpex5v5.docs is given, which also gets README.md in the source root copied into that binary package (Markdown is nearly plain text, but to convert it into pure plain text would be preferred).

Files for Debhelper modules like install, docs and README.Debian mostly could take two forms, they could carry a package dependent filename like .install, or appear “independently” without a leading binary package designation in the filename. The latter is the case for example with README.Debian in this source package. Files like that get copied always into the first package which is given in debian/control, and that is libvigraimpex-dev here. Because this README only contains information on how to handle the examples which are shipped with the development package, that’s exactly what is wanted. If you want to overcome a “sloppy” packaging style, you could designate the stuff in debian/ also when only a single binary package is build, though.

dh_installchangelogs -a

Obvious by its name, dh_installchangelogs copies changelogs, but also debian/NEWS. That is the Debian changelog debian/changelog, and this module also seeks for if a changelog which is shipped by upstream could be found somewhere in the source tree. For that, it checks standard files like doc/changelog, doc/CHANGELOG, CHANGELOG etc. (you can take a look inside /usr/bin/dh_installchangelogs to research the mechanism, Debhelper modules are Perl scripts), and could be helped by giving the path to the upstream changelog on the command line, or by providing debian/changelog in the command line.

doc/changelog (which has been generated by the build target doc/changelog: docsrc/credits_changelog.dxx above) is given here for the command line in debian/rules, but that most likely could be omitted because the module will find it.

debian/.changelog is a standard location for the upstream changelogs (that’s not a register), that could be alternatively used here instead of generating into doc/changelog, but then it’s needed to get individually cleaned (it’s of course a difference if you extra ship an upstream changelog in debian, or if you generate it in the build process. The latter is a temporary file, the other isn’t).

dh_installexamples -a

Then, dh_installexamples is used to install … the examples. For Vigra, examples are shipped by upstream in src/examples, and they are copied into the development package, which makes sense. debian/libvigraimpex-dev.examples is set up for that, and the files which are given here end up in debian/libvigraimpex-dev/usr/share/doc/libvigraimpex-dev/examples²¹.

A previous package maintainer has written a Makefile to generate executables from the sources in the examples folder, which is copied out from debian/. Like written on above, information on how to handle the examples given in README.Debian. There are some file patterns missing in the examples register, though.

In the binary-arch target it follows then a find operation to remove executable flags from everything which is in that folder: find -path './debian/*/usr/share/doc/*/examples/*' -exec chmod a-x '{}' +. That’s the right way to perform something like this: always copied files should be changed, and not the instances in the source tree, like chmod a-x vigranumpy/test/*.py does above. Upstream remains whatsoever, and the Debian stuff is written on the top of it (like patches). But, to employ find here to search through all package directories isn’t really needed, and src/examples currently is clean from such unwanted flags.

By the way, files being executable while they shouldn’t are often found if upstream develops on Windows⁰.

dh_installman -a

Manpages are being installed by dh_installman. There is only one manpage involved, there is the helper vigra-config (Python script to query installation information) in the development package. Upstream doesn’t ship a manpage, so somebody previously maintaining this package wrote one for that script (debian/vigra.config.1).

The copying information is given in debian/libvigraimpex-dev.manpages, but that register could be omitted and the manpage could be given on the command line because libvigraimpex-dev is the first package listed in debian/control, but that are variations of packaging style which actually doesn’t play are role (that maybe could be used for maintainer profiling).

dh_lintian

This is a Debhelper module which installs override files for Lintian, if they are any. Currently, there is a override for the shared module package libvigraimpex5v5 in debian/rules, which got renamed like this for the libstdc++6 transition, so that there is going to be a Lintian complaint on package-name-doesnt-match-sonames²².

By the way, if you hard code Debhelper modules instead of using the sequencer²³, it’s not advisable to remove modules which are temporarily out of purpose like that (that lintian override will become obsolete with the next SONAME bump), but to keep them anyway. That saves you from overlooking it when they need to get back in.

To have it always ready to work properly, it should be also dh_lintian -a given for that module, and dh_lintian -i again in the build-indep target for the architecture independent packages.

To discuss this issue again by this occasion, you have more control over the build process if you write you debian/rules that way, and there are also other advantages. On the downside, you have to follow the development of Debian very closely if new things are becoming available, and mostly new modules are very quick becoming expected to get employed in your package (like for example dh-strip-nondeterminism for reproducible builds)²⁴. Thus, it’s of course more convenient to use the sequencer (which always keeps an up-to-date chain), and most of the sponsors would advise to do that, instead of hand writing a Debhelper chain. Even if you exactly know what you’re doing, and you’re doing everything right.

dh_python2 -a

This is an additional module for Debhelper which is used for Python packaging (by the way, present addons could be always listed by $ dh -l). The package dh-python which contains that module²⁵ is given as build-dependency in debian/control. It’s very important to run that Debhelper addon if you’re packaging Python modules, and dh_python2 performs a couple of changes here, indeed. For example, the file names of the extensions get renamed to include a multiarch triplet (like analysis.so into analysis.x86_64-linux-gnu.so on amd64).

dh_link -a

dh_link does the job of creating symbolic links into the binary package directories, which then are shipped in the binary packages. For the architecture dependent packages, there are no .links files in debian/, but the module links libvigraimpex.so in the development package libvigraimpex-dev (the development linkage file)²⁶ and libvigraimpex.so.5 in the library package libvigraimpex5 (needed for the dynamic linker)²⁷ against the shared object libvigraimpex.so.5.1.10.0 in /usr/lib, automatically. That links have been already created during the build into debian/tmp/usr/lib, but these aren’t copied into the binary package directories, but recreated by dh_link in the proper places.

dh_compress -a

dh_compress compresses several things like the changelogs and the shipped manpage, but also the cxx sources in the examples. The examples are mend to be copied out into some private path and to be used there, like it’s explained in README.Debian. The little walkthrough which is given there also points to the fact that they need to get unzipped before they could be compiled with the included special Makefile. That’s o.k., no need to exclude them from being compressed with -X.

dh_fixperms -a

dh_fixperms runs over the binary directories and checks for if permissions would need to be fixed. It automatically detects and corrects things like if they are images with spurious execute flags sets and such things.

Flaws which dh_fixperms can’t detect (you can again examine the detect mechanism by reading the Perl script /usr/bin/dh_fixperms) are getting complained about by Lintian after the build has been completed. In such cases it’s needed to install manual corrections, and mostly they have to be quite particular like e.g. a chmod 644 is needed for the single file nltk/test/dependeny.doctest in the Python Natural Language Toolkit package²⁸.

It could be discussed if such operations would belong to the modules which installs them (like the manual checkover for the examples directly after dh_installexamples, above), or to dh_fixperms (like when using the Debhelper sequencer, if you would install an override target for dh_installexamples or an override for dh_fixperms which includes that, if needed), but that’s again stuff merely for the packaging style profiler.

Anyway, it’s always a good idea to report flaws like this to upstream if he could correct it. Some do, some just don’t. I have mixed experiences relating to this issue over all my packages.

dh_strip -a

dh_strip is a great Debhelper module which has been decisively further developed lately. It strips out the debugging symbols of of the library, which have been compiled (see on the standard buildflags above), into separate packages which can be installed, too, to have them available for Gdb. Not so much time ago, extra debug packages had to be definied in debian/control but they are created automatically now automatically for every package which includes C/C++ code, which is a big advantage.

dh_makeshlibs -V ‘libvigraimpex5v5 (>= 1.10.0)’

This Debhelper module generates the shlibs control file for the shared library in debian//DEBIAN/shlibs, or calls dpkg-gensymbols to processes the symbols control file in debian/.symbols for the binary package. Each shared libary must provide either shlibs²⁹ or a symbols control file³⁰ by the Debian Policy. libvigraimpex so far ships a shlibs control file.

dh_makeshlibs runs over all packages given in debian/control but can’t find anything for debian/libvigraimpex-doc and debian/python-vigra-doc (there is the error message on No such file or directory in the buildlog). That’s because of the missing -a, but there wouldn’t be anything to process in these install directories, anyway.

[note: sorry it’s getting very briefly from here on due to the lack of time. I want to have this finished now, but I’ll add more information for the following paragraphs in a couple of days]

dh_shlibdeps -a

dh_shlibdeps calculates dependencies for shared libraries.³¹.

dh_numpy -p python-vigra

dh_numpy adds a versioned dependency for the Numpy library for the ${python:Depends} substvar of the Python package in debian/control.

dh_installdeb -a

dh_installdeb installs control files into the debian//DEBIAN directories.

dh_gencontrol -a

This Debhelper module is a wrapper around dpkg-gencontrol which generates debian//DEBIAN/control.

dh_m5sums -a

dh_md5sums generates debian//Md5sums files.

dh_builddeb -a

Finally, dh_builddeb calls dpkg-deb to build the arch dependend Debian packages.

binary-indep

This target doesn’t get called on the amd64 build server, so this buildlog ends here. What’s missing are the documentation packages libvigraimpex-doc and python-vigra-doc. They got build and packaged by the binary-indep target, which begins like that:

PHONY: binary-indep
binary-indep: build-indep
        dh_testdir
        dh_testroot
        $(MAKE) -f debian/rules \
                doc/vigra/build-stamp \
                doc/vigranumpy/build-stamp

The target which actually builds the documentation for the library is doc/vigra/build-stamp:

doc/vigra/build-stamp: obj/build-stamp
        $(MAKE) doc_cpp -C obj/
        touch $(@)

For the Doxygen documentation the library needs it to be build, therefore obj/build-stamp is a prerequisite of this target.

In the main Makefile which has been generated in obj/ by the target obj/CMakeCache.txt, doc_cpp is the target which builds the documentation for the library. That’s got called here by this target in debian/rules, and the docs then get build into doc/vigra.

After that has completed, the target doc/vigranumpy/build-stamp is called to build the documentation for the Python package:

doc/vigranumpy/build-stamp: obj.$(shell pyversions -d)/build-stamp doc/vigra/build-stamp
        ln -sf $(CURDIR)/vigranumpy/docsrc/c_api_replaces.txt obj.$(shell pyversions -d)/vigranumpy/docsrc/
        $(MAKE) doc_python -C obj.$(shell pyversions -d)/
        touch $(@)

From the viewpoint of debian/rules the Python docs are build the same way, but in the single default Python build dir (obj.$(shell pyversions -d)). The target in the Makefile of the build setup created by cmake for that is doc_python.

The Doxygen documentation needs to be have been build before, and for the case this target gets called individually, doc/vigra/buildstamp is given here as prerequisite.

There is a file c_api_replaces.txt in the source package tree, which needs to be copied resp. linked into the Python build directory before.

dh_installdocs -i here uses excludes to filter out certain files from the copy pattern doc/vigra and doc/vigranumpy (and all of them have hits in the current build).

Then some little makeovers take place on the libvigraimpex-doc package, and after that some on python-vigra-doc. Among them, mv -f would save rm -rf html.

The custom check if dh_sphinxdoc is available could be spared, too (by the way ifeq uses quotation marks here unlike above), I don’t think that there is the danger that it gets dropped out of the Sphinx package.

The excludes for dh_compress should be validated, either.