1. prevent issues before uploading to the archive

  2. enable auto-updating

  3. test archive-wide rebuilds of a new go version

prevent issues before uploading to the archive

Currently, we have a number of broken packages in the Debian archive. The vast majority of broken packages used to work at some point, so changes broke them.

In the current workflow, there is no good place to stage updates and observe their effect: the only option we have is to upload to the Debian archive, hope for the best, and wait for FTBFS bugs to eventually be reported (doesn’t always happen).

While individual developers could run ratt to rebuild the reverse dependencies of the package they are about to upload, this takes long enough to be prohibitively expensive.

With the CI setup described here, the effects of a change on the Debian archive can be gathered within a minute, which should be quick enough that we can base our workflow around this signal.

enable auto-updating

Once we have a setup in which we can say with high confidence whether a change introduces new breakages, suggesting a change to update to the latest upstream version becomes simple enough to be automated.

test archive-wide rebuilds of a new go version

Almost coincidentally, this infrastructure is also well-suited for testing the effects of changing the Go compiler. See Go 1.10 build/test failures for an example email thread.


Have a look at https://docs.gitlab.com/ce/ci/quick_start/README.html to learn about how GitLab’s CI infrastructure works if you’re not yet familiar with it.

GitLab CI runner

The gitlab-ci-multi-runner 1.11.4+dfsg-2 Debian package seems to be incompatible with salsa.debian.org, hence I built the latest upstream version, and run the gitlab-runner-linux-amd64 binary directly for the time being.

TODO: figure out the incompatibility, fix gitlab-ci-multi-runner

The configuration file reads:

concurrent = 1
check_interval = 0

  name = "chuchi"
  url = "https://salsa.debian.org"
  token = "<redacted>"
  executor = "docker"
    tls_verify = false
    image = "stapelberg/ci:latest"
    privileged = true
    network_mode = "host"
    disable_cache = false
    volumes = ["/srv/cache:/cache:rw", "/srv/gopath:/srv/gopath:rw", "/srv/chroot:/srv/chroot:ro", "/var/lib/schroot:/var/lib/schroot"]
    shm_size = 0

Running in privileged mode is required for the overlay we create in .gitlab-ci.yml.

Docker container

The “ci” Docker container image makes available the ci-build and ci-diff tools, as well as the Go compiler.

As an optimization, all known non-Go dependencies (for cgo, and for running tests) are pre-installed in the Docker image.

The Dockerfile reads:

FROM debian:sid

# As an optimization, we install the dependencies of packages which use
# cgo. This list should be updated periodically, and the Docker container should
# be rebuilt periodically.
RUN apt-get update && apt-get install -y libgeoip-dev libpcap-dev liblmdb-dev libsqlite3-dev librdkafka-dev libsystemd-dev libtspi-dev libcups2-dev libglib2.0-dev libusb-1.0-0-dev libyara-dev libcap-dev libssl-dev libnss3-dev libldap2-dev libpam0g-dev libsasl2-dev libaugeas-dev libxml2-dev libavahi-client-dev libcairo2-dev libgtk-3-dev libqrencode-dev lxc-dev libseccomp-dev libsecret-1-dev

# Test dependencies
RUN apt-get install -y git bzr fuse zip sqlite3 mongodb-server redis-server bsdiff

RUN apt-get install -y golang-1.10-go golang-1.9-go build-essential dh-golang

# Build dependencies
RUN apt-get install -y golang-golang-x-tools tmpl golang-statik golang-goprotobuf-dev golang-github-gogo-protobuf-dev

# Set up an unprivileged user for running builds
RUN adduser --disabled-password --gecos '' build

RUN ln -sf /usr/lib/go-1.10/bin/go /usr/bin/go

ADD ci-build /usr/bin/ci-build
ADD ci-diff /usr/bin/ci-diff

ADD exemptions.json /var/lib/ci-build/exemptions.json

TODO: add link to up-to-date Dockerfile


The .gitlab-ci.yml file creates an overlay (so that we can modify /srv/gopath and discard our changes later), builds the world, copies the changes of the commit under test into the GOPATH overlay, rebuilds the world, and prints new breakages.

image: stapelberg/ci2

    - before-applying-commit.json
    - after-applying-commit.json
    # Create an overlay to discard writes to /srv/gopath/src after the build:
    - "rm -rf /cache/overlay/{upper,work}"
    - "mkdir -p /cache/overlay/{upper,work}"
    - "mount -t overlay overlay -o lowerdir=/srv/gopath/src,upperdir=/cache/overlay/upper,workdir=/cache/overlay/work /srv/gopath/src"
    - "export GOPATH=/srv/gopath"
    - "export GOCACHE=/cache/go"

    # Build the world as-is:
    - "ci-build -exemptions=/var/lib/ci-build/exemptions.json > before-applying-commit.json"

    # Copy this package into the overlay:
    - "export IMPORT_PATH=$(sed -n 's/^xs-go-import-path: \\([^,]*\\).*$/\\1/ip' debian/control)"
    - "cp -ar . /srv/gopath/src/${IMPORT_PATH?}"

    # Rebuild the world:
    - "ci-build -exemptions=/var/lib/ci-build/exemptions.json > after-applying-commit.json"
    - "ci-diff before-applying-commit.json after-applying-commit.json"

Find the up-to-date version of this file at https://salsa.debian.org/go-team/ci/blob/master/config/gitlabciyml.go


Program pgt-gopath constructs a Go workspace src directory from the Debian unstable archive.

This eliminates the computationally intensive step of identifying reverse dependencies of Debian packages, and eliminates the overhead of installing .deb packages, which is orders of magnitude slower than pgt-gopath.

In our setup, pgt-gopath manages /srv/gopath.


ci-build walks the $GOPATH created by pgt-gopath, examines the Debian packaging to figure out import paths to build/test, and builds all packages in parallel.

ci-build takes about 30 seconds when known breakages are skipped and the Go cache is up-to-date.

When done, ci-build dumps the package build/test results to stdout.


ci-diff compares two ci-build outputs. New breakages will be printed, existing breakages will be skipped.


A solution that is orders of magnitude faster than existing solutions usually needs to make trade-offs.

In this case, the big difference is that the CI infrastructure does not use the Debian package build process at all: instead of calling dpkg-buildpackage (or higher-level wrappers such as sbuild), we are invoking the go tool directly.

This means packages must be buildable/testable after extracting their source (think apt source pkg). In other words: removing files using rm(1) in override_dh_auto_configure does not have any effect in the CI infrastructure.

Note that environment variables are evaluated, though: DH_GOLANG_GO_GENERATE and other Debian::Debhelper::Buildsystem::golang(3pm) options are honored.

Current status of the archive as seen by CI

We have about 50 failing packages out of about 1000 packages total.

Note that 50 failing packages does not mean 50 distinct issues: failure in one package affects all of its reverse dependencies.

Also note that of the remaining issues, a bunch of packages just fail to build from source currently. The CI infrastructure supports exemptions, but I have not gone through all issues and located the corresponding tracking bugs yet.

Required fixes

Here are a few example issues to give you a sense for the kind of changes required to fix the current breakages:

  1. rkt uses a custom Makefile

  2. A handful of packages requires code generation (of .proto files, for example). The required commands need to be expressed using //go:generate comments instead of Makefiles or custom invocations in debian/rules.

  3. A few packages delete parts of the upstream source before building via debian/rules. These need to be converted to use dh_clean(1) (which the CI infrastructure understands) or exclude the files via debian/copyright to begin with.