Microcontroller Continuous Integration with Docker_

February 06, 2018 @12:56

The Background

Right out of the gate I'll admit that my implementation is a bit naive, but it is if nothing else an example of what can be accomplished with a little bit of work. In general my microcontroller development workflow has been tied to a particular system largely using the vendor supplied tools like MPLAB X or Atmel Studio. This is usually OK as I need to have physical access to either the prototype or production hardware for testing and verification. From a high level it generally looks like this:

Three Swans Inn Lighting Controller

Lately I have switched most of my Atmel development away from their IDE and just use a Makefile to build and flash chips. This actually lets me write and build the code from anywhere as this is all stored in a git repository, which is checked out on a system that I can access from anywhere. The build tool chains are a bit obtuse though and keeping all the moving parts around so I could at least ensure the code compiles has been a challenge.

The Goal

So, containers! I made a couple containers with the Atmel and Microchip tool chains in them, with a little glue I was able to connect the post-receive hook of my git repository to my Docker instance to produce on-commit builds of the appropriate project.

Part of the glue is based on the fact that I have all the firmware source in a single git repository for ease of maintenance so I try to determine what changed for building so I don't have to rebuild everything all at once. I also have several different microcontrollers that I target so there is a little logic in the hook to launch the right container for the right code.

The Hook

I snagged most of this from the post-receive hook that handles the deployment of this website. The biggest change was detecting which project within the repository needs to be built.

#!/bin/sh
# microcode post-receive hook
# (c) 2018 Matthew J. Ernisse <matt@going-flying.com>
# All Rights Reserved.
#

set -e

GIT_AUTHOR=""
GIT_BRANCH=""
GIT_DIR=$(git rev-parse --git-dir 2>/dev/null)
PROJECTS=""
REV=0

try_container()
{
    local mapping="\
        led-timer:atmelbuilder \
        led-strand:atmelbuilder \
        bar-lighting:atmelbuilder \
        led-gadget:microchipbuilder \
        "
    if [ -z "$1" ]; then
        echo "usage: try_container project"
        return 1
    fi

    for x in $mapping; do
        if [ "$1" = "${x%%:*}" ]; then
            start-build-container.py \
                "${x##*:}" "$1" "$REV" "$GIT_AUTHOR"
            return
        fi
    done
}

if [ -z "$GIT_DIR" ]; then
    echo >&2 "fatal: post-receive GIT_DIR not set"
    exit 1
fi


while read oldrev newrev refname; do
    GIT_BRANCH=$refname
    REV=$newrev
done

GIT_AUTHOR=$(git show --format='%ae' --no-patch $REV)

for fn in $(git diff-tree --no-commit-id --name-only -r $REV); do
    PROJECTS="$PROJECTS $(dirname $fn)"
done

if [ ! "$GIT_BRANCH" = "refs/heads/master" ]; then
    exit
fi

for project in $PROJECTS; do
    try_container "$project"
done

The Container Launcher

This is basically a stripped down version of the container module from my Flask youtubedown front end.

#!/usr/bin/env python3
'''
start-build-container.py (c) 2018 Matthew J. Ernisse <matt@going-flying.com>
All Rights Reserved.

Redistribution and use in source and binary forms,
with or without modification, are permitted provided
that the following conditions are met:

    * Redistributions of source code must retain the
      above copyright notice, this list of conditions
      and the following disclaimer.
    * Redistributions in binary form must reproduce
      the above copyright notice, this list of conditions
      and the following disclaimer in the documentation
      and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR
TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
'''
import docker
import os
import sys

# Configuration
DOCKER_CA="CA.crt"
DOCKER_CLIENT=(
    "cert.pem",
    "key.pem"
)
DOCKER_ENGINE="whale.example.com:2367"
DOCKER_REPOSITORY="whale.example.com"

def run_container(container_name, image, args):
    ''' Execute a container. '''
    global DOCKER_CA, DOCKER_CLIENT, DOCKER_ENGINE
    tls_config = docker.tls.TLSConfig(
        client_cert=DOCKER_CLIENT,
        verify=DOCKER_CA
    )

    client = docker.DockerClient(
        base_url=DOCKER_ENGINE,
        tls=tls_config
    )

    client.containers.run(
        image,
        args,
        auto_remove=True,
        detach=True,
        name=container_name,
        volumes={
            '/var/www/autobuild': {
                'bind': '/output',
                'mode': 'rw',
            }
        }
    )

def usage():
    print("usage: {} image_name project_name git_rev author".format(
        os.path.basename(sys.argv[0]),
        file=sys.stderr
    )


if __name__ == "__main__":
    if not len(sys.argv) == 5:
        print("Invalid number of arguments.", file=sys.stderr)
        usage()
        sys.exit(1)

    builder = sys.argv[1]
    project = sys.argv[2]
    git_rev = sys.arv[3]
    author = sys.argv[4]

    container_name = "{}-builder--{}--{}".format(
        project,
        git_rev,
        author
    )

    image = "{}/{}:latest".format(DOCKER_REPOSITORY, builder)

    try:
        run_container(container_name, image, project)
        print("*** Running {}...".format(image))
    except Exception as e:
        print("!!! Failed to start container: {}".format(str(e)))

Conclusion

A short while after I push a commit a container will run,build the project, and emit an archive containing the relevant .hex files for the microcontroller as well as a long of the process. I still have some work to do on the Microchip Makefile but for the most part this makes things a lot easier. I can develop from any workstation as long as I have the programming utilities for the chips and if I don't I can at least ensure that code builds every time I commit. The plumbing is pretty generic so I'm sure I'll find other places to use it, for example I was thinking I should try to find a way to build and push the Docker images to my private registry upon commit.

Subscribe via RSS. Send me a comment.