Ready to build better conversations?

Simple to set up. Easy to use. Powerful integrations.

Get free access

CI/CD plays a crucial role in Software Development. At Aircall, most of our pipelines run on self-hosted Gitlab runners on AWS EKS (Elastic Kubernetes Service). We use Infrastructure as Code to deploy with Terraform module the official gitlab-runner Helm chart.

To ensure both reliability and security, it’s required to keep the Gitlab runners up to date as Gitlab.com is updated continuously.

Until now, updates to the latest version were performed manually but it becomes a repetitive and time-consuming task. The goal was to reduce toil on SRE with automation.

Objectives:

• Be notified when new GitLab Runner versions are released.

• Automate the upgrading process with as few manual actions from SRE as possible.

• Keep using our GitLab merge request approval for changes to be reviewed before deployment.

• Separate merge requests by the environment to have changes tested beforehand.

What Is Renovate?

Renovate is a dependency update automation tool. It automatically detects available updates by scanning static code on Git repository.

When new versions are detected, Renovate opens a merge request with the necessary changes.

It is highly customisable and provides release notes and commit histories to help make decisions on deployment.

How We Implemented It

On Gitlab, Renovate recommends including the renovate-runner project to have it run on CI pipeline job.

Adding a scheduled pipeline, this project scans a list of repositories at a specific time.

On the first scan, an onboarding merge request is opened on repositories using a predefined configuration.

We included default attributes such as commit prefix (chore) and label (renovate).

Handling Custom Cases

Initially, the variable we targeted for upgrade (gitlab_runner_chart_version) was not detected by the built-in package manager.

Coming from a custom module, Renovate was not able to detect this dependency. To handle this case, we created a template using the regex custom manager.

Group Merge Request

Grouping similar changes is a feature that is extensively used on Renovate to reduce noise. Here, we use it to separate changes between environments. It allows us to proceed in two steps :

1. Runners used on SRE repositories are upgraded.

2. If no errors, the same procedure is applied to all runners.

Send notification on Slack 

The team needs to be notified on Slack when merge requests are opened by Renovate to take action. For this notification, we add a label (slack) that is caught by GitLab job rules ($CI_MERGE_REQUEST_LABELS =~ /slack/). When triggered, the job posts a message to our Slack channel with a link to the repository.

Results

Issues/Difficulties

• Testing and debugging Renovate configuration can be laborious on GitLab, we had to test locally with a docker-compose running Renovate and Forgejo (fork of Gitea) to validate changes beforehand.

• Maintenance is low, as renovate-runner is updated continuously by Mend Renovate team, but breaking changes are expected and renovate.json configuration may need to be fixed.

• For ease of review, Renovate retrieves Release Notes on merge request descriptions. This feature does not work out of the box with GitHub. Due to API limitations with unauthorised requests on Github, a token must be set to fetch most changelogs.

Improvement

• The automerge feature would be a nice addition to merge specific patch or minor upgrade. However, this feature would require working around our mandatory approvals.

• We plan on storing a sharable config preset on dedicated repositories to extend them in projects. The most used presets will also be included in the default onboarding merge request.

Conclusion

It was the SRE team's first use of automatic dependency upgrade, and we found Renovate to be a good candidate to ease the toil on the GitLab runner upgrade.

We will have to see its full potential in the long run, but we are already testing it for other needs, such as Terraform providers and module upgrades.

Published on January 22, 2024.