Concourse – A Modern CI/CD Tool

Concourse is a CI/CD tool created by a team at a company called Pivotal. Pivotal is a company known for building tools for software development. The Pivotal team wanted to solve a problem in the CI/CD world which is that many existing tools, like Jenkins, had become complicated with too many plugins. Even though having many plugins gives a flexibility and support for every kind of specific need, things can quickly get out of hand. Their goal was to create a tool that developers could use without spending too much time on setup. Therefore, Concourse was designed to be lightweight, predictable, and easy to scale. It focused on automation and repeatability. Developers could describe their workflows as code, which made things transparent and easy to share.

Why Concourse?

Concourse keeps things simple but powerful. Jenkins as the most commonly used and also one of the oldest CI/CD tools, has a lot of features and plugins. GitHub Actions is newer and easy to use, especially for developers who already use GitHub. It is great for simple workflows, but it’s not as powerful for complex setups. It also doesn’t give as much control as some teams might need. However Concourse takes a different approach. It avoids plugins completely, instead, everything is based on clear rules and configurations. This makes it easier to understand and maintain. Pipelines in Concourse are written as YAML files. These files describe the entire workflow, step by step. Teams can save these files in their version control systems, so everyone can see and edit them.

Another unique feature of Concourse is how it runs tasks. Every task runs in a separate container. This means the tasks don’t depend on the state of the machine. They are isolated, so they don’t affect each other. This makes builds more predictable and easier to debug. Concourse is also not perfect. It has a smaller community compared to Jenkins, so finding help can be harder. The YAML files can also feel a bit complicated at first. But once you understand the basics, it becomes very powerful.

Resources are a key concept in Concourse. They represent inputs and outputs for your pipelines, like a Git repository, a Docker image, or even a file in an S3 bucket. When a job or task needs a resource, Concourse checks if the resource has changed since the last run. This ensures that your pipelines always work with the latest version of your code, artifacts, or dependencies. They are also shared across jobs within the same pipeline. For example, if multiple jobs use the same Git repository, Concourse fetches it only once. This saves time and ensures consistency between jobs. So this feature makes Concourse to utilize pipelines efficiently and easy to manage, even in complex workflows.

Strengths and Weaknesses of Concourse

One of its main strengths is pipeline-as-code. You define all workflows in YAML files. This means the pipelines are version-controlled and easy to share. It also ensures that the workflows are consistent across environments.

It is stateless workers. Concourse runs every task in a container. This makes the builds isolated and predictable. There is no leftover state from previous builds, so everything is clean each time.

Concourse is very scalable. You can add more workers easily when the workload increases. This makes it a good choice for larger teams or projects.

It also comes with a built-in UI. The UI shows the pipelines visually. You can see the progress of jobs, tasks, and resources in real time. It’s easy to understand what is happening in your workflows.

Finally, Concourse has first-class support for containers. If your team uses Docker, Concourse is a perfect fit. You can run all tasks inside Docker images without much setup.

As I said above, Concourse is not perfect. It has some weaknesses that you should consider.

First, it has a steep learning curve. If you are new to CI/CD tools, it might take time to understand how Concourse works. The YAML configuration files can feel confusing at the beginning.

Second, Concourse has a smaller community compared to tools like Jenkins. This means there are fewer tutorials, plugins, and third-party resources available. Finding solutions to specific problems can take longer.

Another issue is limited integrations. While this keeps it simple, it also means you need to create custom resources for some tasks. This can be extra work for your team.

The YAML files can become verbose for large pipelines. If your workflows are complex, the pipeline definitions can get long and hard to manage. It requires careful planning to keep them organized.

How Concourse Works

Concourse is built around a few simple but powerful concepts: pipelines, jobs, tasks, resources, and steps. These are the building blocks of its workflows.

• Pipelines: A pipeline is a complete workflow. It defines how your code moves from development to deployment. Pipelines are written in YAML files.
• Jobs: Jobs are groups of tasks that need to run together. For example, a job might include running tests, building the application, and deploying it.
• Tasks: Tasks are the smallest units of work in Concourse. Each task is a single script or command, often running in a container.
• Resources: Resources are inputs and outputs for jobs. For example, a Git repository can be a resource. Concourse automatically fetches or updates these resources during the pipeline run.
• Steps: Steps are the actions within a job. They include commands like get (fetching a resource), put (uploading a resource), or task (running a script).

Concourse separates these elements clearly. This makes pipelines easier to understand and maintain. Tasks always run in containers, which ensures every step is clean and isolated. This design also makes it easier to debug problems.

Concourse Basics

Here I will show some basics but this is not a tutorial or this will not go into details. There is a web site which provides a comprehensive tutorial about Concourse. I strongly suggest you to visit it if you are interested in: https://concoursetutorial.com/

Let’s get our hands dirty and look how it works. To get started with Concourse, you need to understand its YAML syntax and the fly CLI tool. The pipeline is defined in a .yml file. For example:

jobs:
  - name: test-app
    plan:
      - get: code
      - task: run-tests
        config:
          platform: linux
          image_resource:
            type: docker-image
            source: {repository: busybox}
          run:
            path: sh
            args:
              - -c
              - echo "Running tests"
resources:
  - name: code
    type: git
    source:
      uri: https://github.com/example/repo.git

Running Concourse Locally via Docker

Running Concourse locally is quick and easy. All you need is Docker and Docker Compose installed on your machine. Concourse provides a ready-to-use docker-compose.yml file provided officially by Concourse, so you don’t need to configure anything yourself. You can also download via curl as follows:

curl -O https://concourse-ci.org/docker-compose.yml

You can modify the yaml file as you wish or just run with the default settings on it. Then you can simply run it via Docker as follows:

docker-compose up

And you also need another tool called fly in order to interact with Concourse. You use it to log in, upload pipelines, and monitor jobs. Without fly, you cannot configure or manage pipelines in Concourse.

After your Concourse instance is running:

1. Visit the local Concourse web interface (it is http://localhost:8080 unless configured differently)
2. Click on the “Download CLI” link for your operating system (e.g., Linux, macOS, or Windows).
3. Add the downloaded file to your system’s PATH so you can run fly commands from anywhere.

Using Fly is also easy

Login to Concourse:

fly -t local login -c http://localhost:8080 -u test -p test

Set a pipeline: Upload a pipeline YAML file:

fly -t local set-pipeline -p my-pipeline -c pipeline.yml

Trigger a job: Run a job manually:

fly -t local trigger-job -j my-pipeline/my-job

Monitor builds: View the status of running builds:

fly -t local builds

Conclusion

To sum up, Concourse is a modern CI/CD tool designed for simplicity, scalability, and repeatability. Its pipeline-as-code approach and stateless worker model make it reliable and predictable. While it has a steeper learning curve and fewer integrations than some competitors, its clean design and container-based workflows make it a powerful choice for teams focused on automation and efficiency. Setting it up locally with Docker is quick, and the fly CLI provides full control over your pipelines. With some practice, Concourse can become an essential part of your development process.

Sources

Documentation about Concourse can be found at https://concourse-ci.org/docs.html

Suleyman Cabir Ataman, PhD

Sharing on social media: