Resources are the heart and soul of Concourse. They represent all external inputs to and outputs of jobs in the pipeline and define methods of periodically checking the external input for changes. As such resources have an in method to defining how external data is fetched, an out method for publishing the data and a check method to periodically check the external data for changes.

Resources are listed under the resources: key in the pipeline configuration.

Resources have 2 mandatory fields:

• name: short and simple, will be referenced by build plans of jobs in the pipeline
• type: determines how the pipeline interacts with external resources. Out of the box, Concourse comes with a few "base" resource types to cover common CI use cases like dealing with Git repositories and S3 buckets. Beyond these core types, each pipeline can configure its own resource types by specifying the top level resource_types key see using resource types.

As part of this tutorial environment, our concourse workers have access to a git-sample git repository at http://git-server:8080/git-sample.git

To pull in the Git repository into the pipeline, we edit pipeline.yml and add a top-level section resources:

resources:
- name: resource-git-sample
  type: git
  source:
    uri: http://git-server:8080/git-sample.git
    branch: master

Our pipeline consists of jobs and each job has a single build plan. A build plan is a sequence of steps to execute.

The 3 basic types of step are:

• get: Get a resource defined in the resoures: key and make it available to subsequent steps via the given name. These get steps can trigger the job plan when its resource check determines it. The flag to control this is trigger and it is false by default.
• task: Execute a task
• put: Pushes to the given resource. All artifacts collected during the plan's execution will be available in the working directory.

These steps may fetch down or update Resources, or execute Tasks. Once the resource has been defined, we can use it in our job steps

We add this get step to our job plan, making our final pipeline become this:

resources:
- name: resource-git-sample
  type: git
  source:
    uri: http://git-server:8080/git-sample.git
    branch: master

jobs:
- name: job-hello-world
  public: true
  plan:
  - get: resource-git-sample # get external git resource
  - task: hello-world
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: {repository: busybox}
      run:
        path: ls
        args: ['-alR']

Apply the changes with the set-pipeline command:

fly -t tutorial sp -p hello-world -c pipeline.yml

Notice the clear overview of changes we made to the pipeline

apply configuration? [yN]:

Press y

Force a resource check (alias cr) for the git resource to validate the configuration:

fly -t tutorial cr -r hello-world/resource-git-sample

And test the pipeline by triggering the job and watching the output

fly -t tutorial tj -j hello-world/job-hello-world -w

Question: Did the resource-git-sample show up in the task directory listing?

If not, can you think of a reason why and the way to fix it?

We learned how to define resoures and use them as inputs in our job plans. Often we will want to also publish the output of our jobs as well.

In this step, we will create a slightly more complex pipeline which commits changes to a git resource and pushes these changes back to the external resource. Side note: External is not really the right term here as we run the git remote within our learning environment.

In the "Task Basics" scenario we looked at extracting complex run commands into standalone shell scripts. Similarly, for job steps of type task, we may inline the task configuration as we did previously or refer to an external file containing the task configuration using, in this case, the file key.

  ...
  plan:
  ...
  - task: bump-timestamp-file
    file: resource-git-sample/ci/task-bump-timestamp.yml
  ...

With pipelines we now need to store the task file and task script somewhere outside of Concourse. The most common resource type to store our task files and task scripts is the git resource type. Perhaps your task files could be fetched via the s3 resource type from an AWS S3 bucket; or the archive resource type to extract them from a remote archive file. Or perhaps the task files could be pre-baked into the image_resource base Docker image.

In our use case, our task file git-sample/ci/task-bump-timestamp.yml and task script git-sample/ci/bump-timestamp.sh are stored in our project git repository. The task script will fork the git repository to updated-git-sample folder and bump the date in the bumpme file, committing the change when done.

Create a new pipeline configuration via the CLI: touch pipeline_output_git.yml

Open the file: pipeline_output_git.yml

Add the following pipeline configuration which is similar to previous pipelines, but the task config is not in-line, referring to a file instead:

resources:
- name: resource-git-sample
  type: git
  source:
    uri: http://git-server:8080/git-sample.git
    branch: master

jobs:
- name: job-bump-date
  serial: true
  plan:
  - get: resource-git-sample
  - task: bump-timestamp-file
    file: resource-git-sample/ci/task-bump-timestamp.yml

Finally, to push the updated-git-sample generated by bump-timestamp.sh script, add a put step to the job plan, with repository being the only required parameter:

  - put: resource-git-sample
    params:
      repository: updated-git-sample

Refer to the concourse/git-resource documentation for an overview of all the parameters supported for publishing this resource.

Create the new pipeline as bump-date:

fly -t tutorial sp -p bump-date -c pipeline_output_git.yml

Unpause the bump-date pipeline:

fly -t tutorial up -p bump-date

And trigger the job with the -w flag to watch its progress:

fly -t tutorial tj -j bump-date/job-bump-date -w

To confirm the job worked as expected, pull and review the changes made by concourse in your local repository:

cd git-sample
git pull
git log --pretty=format:"%h%x09%ad%x09%s by %an"
git diff HEAD~1 HEAD
cd -

The name of resources and the name of task outputs, determine the name used to access them by other tasks.

The bump-timestamp-file task can access anything from resource-git-sample under the resource-git-sample/ path. Since the relative path of task-bump-timestamp.yml task file inside this repo is ci/task-bump-timestamp.yml, the task: bump-timestamp-file references it by joining the two: file: resource-git-sample/ci/task-bump-timestamp.yml

There is a benefit and a downside to abstracting tasks into YAML files outside of the pipeline.

One benefit is that the behavior of the task can be kept in sync with the primary input resource (for example, a software project with tasks for running tests, building binaries, etc).

One downside is that the pipeline.yml no longer explains exactly what commands will be invoked. Comprehension of pipeline behavior is potentially reduced.

But one benefit of extracting inline tasks into task files is that pipeline.yml files can get long and it can be hard to read and comprehend all the YAML. Instead, give tasks long names so that readers can understand what the purpose and expectation of the task is at a glance.

But one downside of extracting inline tasks into files is that fly set-pipeline is no longer the only step to updating a pipeline.

From now onwards, any change to your pipeline might require you to do one or both:

• Update concourse with fly set-pipeline on a change to the job build plan and/or input/output resources
• Update your primary source of task files and task scripts with git commit and git push

If a pipeline is not performing new behaviour then it might be you skipped one of the two steps above.

Due to the benefits vs downsides of the two approaches - inline task configuration vs YAML file task configuration - you will see both approaches used in the wider community of Concourse users.

So far we have mainly triggered jobs manually through the fly trigger-job command.

Other options are:

• Use the Concourse Web UI: click the + button on a job (we didn't do this as we're using the command line only in this tutorial)
• Sending POST request to Concourse API
• Trigger jobs using resources

The primary way that Concourse jobs will be triggered to run is through resources changing. A git repo has a new commit? Run a job to test it. A GitHub project cuts a new release? Run a job to pull down its attached files and do something with them.

As mentioned in Step 4 of this scenario, triggering is controlled by the steps in the build plan. A get step may use its resource check method to trigger depending jobs.

The time resource comes with Concourse out of the box and is meant mainly to satisfy "trigger this build at least once every 5 minutes" type of requirements.

Open the file: pipeline_output_git.yml

Add the time resource named 2m with an interval of 2 minutes and use it in a triggering get step:

resources:
- name: resource-git-sample
  type: git
  source:
    uri: http://git-server:8080/git-sample.git
    branch: master
- name: 2m
  type: time
  source:
    interval: 2m

jobs:
- name: job-bump-date
  serial: true
  plan:
  - get: resource-git-sample
  - get: 2m
    trigger: true              # trigger job every 2 minutes
  - task: bump-timestamp-file
    file: resource-git-sample/ci/task-bump-timestamp.yml
  - put: resource-git-sample
    params:
      repository: updated-git-sample

Update the bump-date pipeline configuration in Concourse:

fly -t tutorial sp -p bump-date -c pipeline_output_git.yml

You may watch the builds to see the bump-date pipeline run every 2 minutes

watch fly -t tutorial bs -p bump-date

Note: Press CTRL+C to stop the watch

Why does time resource configured with interval: 2m trigger "approximately" every 2 minutes?

resources are checked every minute, but there's a shorter (10sec) interval for determining when a build should run; time resource is to just ensure a build runs on some rough periodicity; we use it to e.g. continuously run integration/acceptance tests to weed out flakiness

• • alex*

The net result is that a timer of 2m will trigger every 2 to 3 minutes.

In this step we will make an actual pipeline - one job passing results to another job upon success.

In all previous sections our pipelines have only had a single job. For all their wonderfulness, they haven't yet felt like actual pipelines. Jobs passing results between jobs. This is where Concourse shines even brighter.

Open the file: pipeline_output_git.yml - which should be as follows:

resources:
- name: resource-git-sample
  type: git
  source:
    uri: http://git-server:8080/git-sample.git
    branch: master
- name: 2m
  type: time
  source:
    interval: 2m

jobs:
- name: job-bump-date
  serial: true
  plan:
  - get: resource-git-sample
  - get: 2m
    trigger: true
  - task: bump-timestamp-file
    file: resource-git-sample/ci/task-bump-timestamp.yml
  - put: resource-git-sample
    params:
      repository: updated-git-sample

Add a second job job-show-date which will run whenever the first job successfully completes:

- name: job-show-date
  plan:
  - get: resource-git-sample
    passed: [job-bump-date]
    trigger: true
  - task: show-date
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: {repository: busybox}
      inputs:
        - name: resource-git-sample
      run:
        path: cat
        args: [resource-git-sample/bumpme]

Update the bump-date pipeline configuration in Concourse:

fly -t tutorial sp -p bump-date -c pipeline_output_git.yml

And trigger the job with the -w flag to watch its progress:

fly -t tutorial tj -j bump-date/job-bump-date -w

The latest resource-git-sample commit fetched down in job-show-date will be the exact commit used in the last successful job-bump-date job. If you manually created and pushed a new git commit in the git-sample repository and manually ran the job-show-date job it would continue to use the previous commit it used, and ignore your new commit. This is the power of pipelines.

Cleaning up

As this pipeline is triggered every 2 mintues, you may delete the bump-date pipeline (and lose all its build history) with the destroy-pipeline (alias dp) command:

fly -t tutorial dp -p bump-date