Introduction

Simple container-based CICD platform configured using YAML files and managed using UI and Fly CLI

Concepts

Tasks = Usually a script that performs some single unit of work and are basically simple functions that take inputs as parameters. Keeping them small and simple allows you to easily run them from your local machine. Tasks should exit 0 or fail
Resources = Are entities that are made available to tasks to be able to complete their unit of work. They can be pulled from (e.g. Git) or pushed to (e.g. some PaaS).
Jobs = A high-level view of a set of tasks that complete a build. Jobs are defined in a build plan, which is a sequence of steps to complete to fulfil the job

Architecture

Concourse is a container based architecture that has 3 core components.

ATC (Air Traffic Control) = This is core of Concourse and is made up of multiple parts. A Postgres database for storing pipeline data and build logs, an API server for programatically running requests (Fly CLI will use this), the Web UI and a scheduler for managing pipelines. It can be HA and behind a load-balancer; and all instances must point to the same DB. There is some documentation for clustering using a remote DB and this is probably a good case for building a self-managing DB cluster with something like Manatee because of having such important information
TSA (maybe Transport Security Agent) = handles SSH connections from workers to ATC, provides authentication. Handles register-worker and forward-worker requests over SSH to register workers to a local ATC or forward to an ATC on a remote network through a reverse tunnel
Worker = Handle the actual workload. They are stateless and register for work with via the TSA. They implement Garden for managing container orchestration of tasks which are part of a build. Garden claims to be platform-agnostic but describes only 2 backends: RunC on Linux; and Windows

Known issues

Error response from daemon: error creating aufs mount
resource script ‘/opt/resource/check []’ failed: exit status 1 stderr: failed to fetch digest: 401 Unauthorized - using an incorrect docker image address can cause this.

Installation

Standar Docker Engine

Basic buid was easy enough with predefined Compose file

Triton Autopilot version

Fly CLI

Download the binary from the downloads page and copy to /usr/local/bin

Setup

Auth

Fly

Default Login
First login with the pre-set account (concourse:changeme) on the “main” team.

$ fly -t flight-school login -c http://192.168.0.1:8080

Create account

Create Account

$ fly -t flight-school set-team -n my-team \
    --basic-auth-username flight-school \
    --basic-auth-password changeme

Create target and login

$ fly -t flight-school login -n my-team -c http://192.168.0.1:8080

-t is our “target”, which is a name we are giving to any given deployment and must be used on all subsequent requests to this environment. The URL must be an externally available address, even if you are running this all locally. 127.0.0.1 won’t work because has to also work in a separate network namespace, for execute to be able to receive your uploaded bits from the web node

UI

Did find an issue where Login on Safari was silently failing with a 401 unauthorised but works on Firefox and haven’t found the issue since

Basic Pipeline

The flight school example is a good step by step guide but note the following:

set up pipeline with fly -t flight-school set-pipeline -p flight-school -c ci/pipeline.yml --load-vars-from ci/params.yml
unpause pipeline with fly -t flight-school unpause-pipeline -p flight-school
No such file errors with no file mentioned. exec failed: no such file or directory could indicate a missing file on the container created to run the tests. In the example it is a missing /bin/bash because of using Busybox Linux
The path to the test script is one depth wrong ./flight-school/ci/test.sh should be .ci/test.sh because we’ve already cd’d into the flight-school directory
file: flight-school/build.yml in the pipeline job isn’t relative to the PWD but seems to be relative to the pipeline on the Concourse server. I saw this when removing flight-school from the path (because of previous point) and the pipeline failed with config path 'build.yml' does not specify where the file lives
Any changes to the pipeline configuration requires ran fly -t flight-school set-pipeline -p flight-school -c ci/pipeline.yml --load-vars-from ci/params.yml to be ran before pushing to Git
In the UI Build Jobs section (e.g. http://192.168.0.1:8080/teams/my-team/pipelines/flight-school/jobs/test-app/builds/3), the left most build is the most recent
Clicking on each flight-school in the pipeline shows the status and output of the query to Git resource
Clicking on test-app show the status and output of the test-app job
In the UI, resources linked between jobs when they are part of a dependent pipeline. For example:
```
- get: docker-image-resource
    passed:
    - build-docker-image
```
would be linked to the previous build-docker-image job. Whereas, when simply writing:
```
- get: docker-image-resource
```
docker-image-resource would be a terminal point
In the UI, the root of resources are highlighted with white text.

List recent builds

$ fly -t flight-school builds
id  pipeline/job            build  status   start                     end  duration
1   flight-school/test-app  1      started  2017-03-31@15:10:27+0100  n/a  1m52s+

Watch a whole build fly -t flight-school watch -b 1
Watch a single job ffly -t flight-school watch -j flight-school/test-app
dashed lines in the UI mean that there is no trigger defined between the two endpoints

Resources

Semver

In this example, branch and file are literally setup up to have a separate branch called “version” to keep a record of the current version

branch: version
file: version

On a put, there seems to be a relative reference to files for the pipeline. For example the lines:

- put: resource-version
    params:
      file: resource-version/number

doesn’t appear to keep a file called number in the associated repository for the resource (on any branch); and when checking volumes:

(gaz@gMac)-(11:38:53)-(tmp)
$fly -t my-target vs | grep number
ad292182-4f11-4f96-51b7-24b7e144f93d  indefinite  indefinite  1546b853a2f2  cache   number:0.0.369                                                                                          0.0 MiB  
10780936-11fb-41ce-6115-6bc1722355bd  indefinite  indefinite  430fa8ee329a  cache   number:0.0.316                                                                                          0.0 MiB  
e3d58d3f-cf1b-4a0c-5056-67585bf06d74  indefinite  indefinite  54e95a499ad7  cache   number:0.0.246                                                                                          0.0 MiB  
(gaz@gMac)-(11:39:09)-(tmp)
$

Resource Types

3rd party community resource types

Need to be declared to be used. For example the Generic HTTP API, needs a resource declaration like:

resource_types:
- name: http-api
  type: docker-image
  source:
    repository: aequitas/concourse-http-api-resource

Jobs

serial: a boolean that forces a given job to not run in parallel to others
serial_groups: string: an array of group names to ensure don’t run in parallel

Concourse