Goals/Motivation

  1. prevent issues before uploading to the archive

  2. enable auto-updating

  3. test archive-wide rebuilds of a new go version

prevent issues before uploading to the archive

Currently, we have a number of broken packages in the Debian archive. The vast majority of broken packages used to work at some point, so changes broke them.

In the current workflow, there is no good place to stage updates and observe their effect: the only option we have is to upload to the Debian archive, hope for the best, and wait for FTBFS bugs to eventually be reported (doesn’t always happen).

While individual developers could run ratt to rebuild the reverse dependencies of the package they are about to upload, this takes long enough to be prohibitively expensive.

With the CI setup described here, the effects of a change on the Debian archive can be gathered within a minute, which should be quick enough that we can base our workflow around this signal.

enable auto-updating

Once we have a setup in which we can say with high confidence whether a change introduces new breakages, suggesting a change to update to the latest upstream version becomes simple enough to be automated.

test archive-wide rebuilds of a new go version

Almost coincidentally, this infrastructure is also well-suited for testing the effects of changing the Go compiler. See Go 1.10 build/test failures for an example email thread.

Implementation

Have a look at https://docs.gitlab.com/ce/ci/quick_start/README.html to learn about how GitLab’s CI infrastructure works if you’re not yet familiar with it.

Docker container

The “ci” Docker container image makes available the ci-build and ci-diff tools, as well as the Go compiler.

As an optimization, all known non-Go dependencies (for cgo, and for running tests) are pre-installed in the Docker image.

The Dockerfile reads:

FROM golang:1.16.6-buster as builder

WORKDIR /app
COPY . .
RUN go build -v salsa.debian.org/go-team/infra/pkg-go-tools/cmd/ci-build
RUN go build -v salsa.debian.org/go-team/infra/pkg-go-tools/cmd/ci-diff
RUN go build -v salsa.debian.org/go-team/infra/pkg-go-tools/cmd/pgt-gopath

FROM debian:sid

LABEL maintainer="Aloïs Micard <creekorful@debian.org>"

# As an optimization, we install the dependencies of packages which use
# cgo. This list should be updated periodically, and the Docker container should
# be rebuilt periodically.
RUN apt-get update && apt-get install -y libgeoip-dev libpcap-dev liblmdb-dev libsqlite3-dev librdkafka-dev libsystemd-dev libtspi-dev libcups2-dev libglib2.0-dev libusb-1.0-0-dev libyara-dev libcap-dev libssl-dev libnss3-dev libldap2-dev libpam0g-dev libsasl2-dev libaugeas-dev libxml2-dev libavahi-client-dev libcairo2-dev libgtk-3-dev libqrencode-dev lxc-dev libseccomp-dev libsecret-1-dev

# Test dependencies
RUN apt-get install -y git bzr fuse zip sqlite3 redis-server bsdiff

RUN apt-get install -y golang-1.15-go build-essential dh-golang git-buildpackage quilt

# Build dependencies
RUN apt-get install -y golang-golang-x-tools tmpl golang-statik golang-goprotobuf-dev golang-github-gogo-protobuf-dev

# Set up an unprivileged user for running builds
RUN groupadd ci -g 1042 && \
    useradd -m ci -u 1042 -g 1042

RUN ln -sf /usr/lib/go-1.15/bin/go /usr/bin/go

COPY --from=builder /app/ci-build /usr/bin/ci-build
COPY --from=builder /app/ci-diff /usr/bin/ci-diff
COPY --from=builder /app/pgt-gopath /usr/bin/pgt-gopath

COPY docker/ci/exemptions.json /var/lib/ci-build/exemptions.json

Up-to-date Dockerfile

.gitlab-ci.yml

The .gitlab-ci.yml file creates an overlay (so that we can modify /srv/gopath and discard our changes later), builds the world, copies the changes of the commit under test into the GOPATH overlay, rebuilds the world, and prints new breakages.

test_the_archive:
  artifacts:
    paths:
    - before-applying-commit.json
    - after-applying-commit.json
  script:
    # Create an overlay to discard writes to /srv/gopath/src after the build:
    - "rm -rf /cache/overlay/{upper,work}"
    - "mkdir -p /cache/overlay/{upper,work}"
    - "mount -t overlay overlay -o lowerdir=/srv/gopath/src,upperdir=/cache/overlay/upper,workdir=/cache/overlay/work /srv/gopath/src"
    - "export GOPATH=/srv/gopath"
    - "export GOCACHE=/cache/go"
    # Build the world as-is:
    - "ci-build -unprivileged_uid=1042 -unprivileged_gid=1042 -exemptions=/var/lib/ci-build/exemptions.json > before-applying-commit.json"
    # Copy this package into the overlay:
    - "GBP_CONF_FILES=:debian/gbp.conf gbp buildpackage --git-no-pristine-tar --git-ignore-branch --git-ignore-new --git-export-dir=/tmp/export --git-no-overlay --git-tarball-dir=/nonexistent --git-cleaner=/bin/true --git-builder='dpkg-buildpackage -S -d --no-sign'"
    - "pgt-gopath -dsc /tmp/export/*.dsc"
    # Rebuild the world:
    - "ci-build -unprivileged_uid=1042 -unprivileged_gid=1042 -exemptions=/var/lib/ci-build/exemptions.json > after-applying-commit.json"
    - "ci-diff before-applying-commit.json after-applying-commit.json"

Find the up-to-date version of this file at https://salsa.debian.org/go-team/infra/pkg-go-tools/-/blob/master/config/gitlabciyml.go

pgt-gopath

Program pgt-gopath constructs a Go workspace src directory from the Debian unstable archive.

This eliminates the computationally intensive step of identifying reverse dependencies of Debian packages, and eliminates the overhead of installing .deb packages, which is orders of magnitude slower than pgt-gopath.

For more details, see https://salsa.debian.org/go-team/infra/pkg-go-tools/-/blob/master/cmd/pgt-gopath/gopath.go

In our setup, pgt-gopath manages /srv/gopath.

ci-build

ci-build walks the $GOPATH created by pgt-gopath, examines the Debian packaging to figure out import paths to build/test, and builds all packages in parallel.

ci-build takes about 30 seconds when known breakages are skipped and the Go cache is up-to-date.

When done, ci-build dumps the package build/test results to stdout.

For more details, see https://salsa.debian.org/go-team/infra/pkg-go-tools/-/blob/master/cmd/ci-build/build.go

ci-diff

ci-diff compares two ci-build outputs. New breakages will be printed, existing breakages will be skipped.

For more details, see https://salsa.debian.org/go-team/infra/pkg-go-tools/-/blob/master/cmd/ci-diff/diff.go

Implications/caveats

A solution that is orders of magnitude faster than existing solutions usually needs to make trade-offs.

In this case, the big difference is that the CI infrastructure does not use the Debian package build process at all: instead of calling dpkg-buildpackage (or higher-level wrappers such as sbuild), we are invoking the go tool directly.

This means packages must be buildable/testable after extracting their source (think apt source pkg). In other words: removing files using rm(1) in override_dh_auto_configure does not have any effect in the CI infrastructure.

Note that environment variables are evaluated, though: DH_GOLANG_GO_GENERATE and other Debian::Debhelper::Buildsystem::golang(3pm) options are honored.

Current status of the archive as seen by CI

We have about 50 failing packages out of about 1000 packages total.

Note that 50 failing packages does not mean 50 distinct issues: failure in one package affects all of its reverse dependencies.

Also note that of the remaining issues, a bunch of packages just fail to build from source currently. The CI infrastructure supports exemptions, but I have not gone through all issues and located the corresponding tracking bugs yet.

Required fixes

Here are a few example issues to give you a sense for the kind of changes required to fix the current breakages:

  1. rkt uses a custom Makefile

  2. A handful of packages requires code generation (of .proto files, for example). The required commands need to be expressed using //go:generate comments instead of Makefiles or custom invocations in debian/rules.

  3. A few packages delete parts of the upstream source before building via debian/rules. These need to be converted to use dh_clean(1) (which the CI infrastructure understands) or exclude the files via debian/copyright to begin with.

Setup a new Runner

The infra/provisioning repository contains Ansible files and instructions to setup a Runner for the Go team.

Install Ansible and the needed playbook(s) on your machine

(this must be run on your local machine)

user@dev:~$ sudo apt-get update && sudo apt-get install -y ansible
user@dev:~$ ansible-galaxy install geerlingguy.docker

Use Ansible to provision the host

(this must be run on your local machine)

user@dev:~$ git clone git@salsa.debian.org:go-team/infra/provisioning.git
user@dev:~$ cd provisioning
user@dev:~/provisioning$ echo -e '[ci-runner]\n127.0.0.1' > .inventory # replace 127.0.0.1 by your machine IP
user@dev:~/provisioning$ ansible-playbook -i .inventory ci-runner.yaml

Register the runner on Salsa

(this must be run on the runner)

Head over https://salsa.debian.org/groups/go-team/-/settings/ci_cd to get the registration URL / token.

user@ci-runner:~$ docker run --rm -it -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register \
  --non-interactive \
  --url "https://salsa.debian.org/" \
  --registration-token "REGISTRATION_TOKEN" \
  --executor "docker" \
  --docker-image registry.salsa.debian.org/go-team/infra/pkg-go-tools/ci:050820211618 \
  --description "Go team runner" \
  --tag-list "go-ci" \
  --run-untagged="false"

Tweak the Gitlab runner configuration file

(this must be run on the runner)

SSH on the runner, and edit the file located at /srv/gitlab-runner/config/config.toml. Adapt the configuration file using the values below:

concurrent = 1
check_interval = 0

[[runners]]
  name = "<redacted>"
  url = "https://salsa.debian.org"
  token = "<redacted>"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "registry.salsa.debian.org/go-team/infra/pkg-go-tools/ci:050820211618"
    privileged = true
    network_mode = "host"
    disable_cache = false
    volumes = ["/srv/cache:/cache:rw", "/srv/gopath:/srv/gopath:rw"]
    shm_size = 0

Notes:

  • Generally, the volumes, network_mode And privileged fields must be adapted, the rest should already be good.

  • Running in privileged mode is required for the overlay we create in .gitlab-ci.yml.

  • The image field is mandatory in the configuration file, but will be ignored because it is overrided in the project .gitlab-ci.yml

Start the Gitlab runner Systemd service

(this must be run on the runner)

user@ci-runner:~$ sudo systemctl start gitlab-runner

Then email the go-team to inform us about the new runner. It could be great to create shell account for other members of the team to team maintain the runner and eliminate any single point of failure.