gitlab CI pipeline — learnings, tips and tricks

In this article, we will cover few areas to make our CICD pipelines more efficient.

Understanding Gitlab CI CD rules

• Gitlab CICD rules are a little confusing at the beginning and we can easily make many mistakes.
• The below video helped me in understanding them to effectively build pipelines.
• Rule evaluation will stop when first rule is met that matches the condition. After this, the rest of the rules may not be evaluated. This is painful while working with multiple rules. But be aware.

Conditional variables

The ability to conditionally modify variable can be handy in many places. By default, variable could have a value and based on a condition, it could have a different value.

By writing this rule within workflow we get this conditional variable value for all jobs.

variables:
  DOCKER_TAG: $CI_COMMIT_REF_NAME
workflow:
  rules:
    - if: $CI_COMMIT_REF_NAME == "main"
      variables:
        DOCKER_TAG: "latest" # Override

Test in local system before pushing code

At times, we would like to run the gitlab build locally to avoid silly mistakes and spending lot of time waiting for the build flow.

If we have lot of external dependency like connecting to AWS etc., testing in local may be too hard. The following is a handy way to test in local, if we have simple straightforward gitlab CICD flow.

We can run the gitlab cicd flow, the flows that actually gets executed inside a runner in local. We can use gitlab-runner docker image and execute the flow.

From the base directory of our repository,

• mount the whole application into gitlab-runner
• attach our local system’s `docker.sock`. This helps to reuse docker images that we already have in host system.

docker run --rm \
  --name gitlab-runner \
  -v $PWD:$PWD \
  -v /var/run/docker.sock:/var/run/docker.sock \
  gitlab/gitlab-runner:latest

Pytest with poetry

Here is an example that showcases pytest execution in `gitlab-ci`

image: python:3.10-slim
services:
  - docker:stable-dind
stages: 
  - unit_test
unit_test:
  stage: unit_test
  before_script:
    - pip install poetry
    - poetry export --dev -f requirements.txt --output requirements.txt --without-hashes
    - pip install -r requirements.txt
  script:
    - pytest
  allow_failure: false

Multi line scripts

Many time gitlab-ci.yaml is hard to read because some of the commands are too long. But gitlab-ci yaml uses generic yaml parser. So we can break long lines into multiple lines without any hassle. No escape characters like \ or / or \n etc is needed. Following works without any issues

docker_publish:
    stage: docker
    script:
      - DOCKER_BUILDKIT=1 docker build 
            --build-arg argument_1=some_thing   
            --build-arg argument_2=some_thing_else  
            -t $DOCKER_REGISTRY/awesome_app:latest .

Debug variables

There are times we use a variable but gitlab pipeline get’s unexpected value. May be, the value is overriden in the hierrachy of the project group.

To quickly debug, create a temporary repo at the same level and have a simple .gitlab-ci.yaml with following content. This will print all env

print:
  script:
    - env
    - '& echo "$MY_VARIABLE"'

Reducing duplication

There are different strategies to reduce duplication in .gitlab-ci file.

• Using extends

.tests:
  script: rake test
  stage: test
  only:
    refs:
      - branches
rspec:
  extends: .tests
  script: rake rspec
  only:
    variables:
      - $RSPEC

• Using includes

include:
  remote: 'https://gitlab.com/awesome-project/raw/main/.before-script-template.yml'
or
include: '/templates/.after-script-template.yml'

If we have multiple projects which share similar continuous integration flow, extracting it out as separate common repo and including it improves the whole ecosystem.

References

• gitlab ci — docker in docker (dind) flow well explained — here
• gitlab rules & variables — Documentation
• Choose when to run jobs — Documentation
• gitlab — workflow — rules — variables — Documentation