5 MIN READ
The CI/CD pipeline is the place where the code, the credentials, the secrets, the production access, and the third party integrations all meet, and the same place is also where the security controls tend to be the thinnest. Engineers are in a hurry, the security review gets the lightest touch, the third party plugins are pulled in without much thought, the secrets land in environment variables that get logged, and the build artifacts get signed with keys that live in the same environment that just got breached.
The pipeline amounts to the supply chain, and the supply chain sits as the attack surface. The recent incidents in the space point to the shape of what is coming, not the shape of what has already happened.
- The Codecov breach, where the bash uploader script was modified to exfiltrate environment variables from every build that used the script.
- The SolarWinds breach, where the build pipeline was compromised and the signed updates were pushed to 18,000 customers.
- The 3CX breach, where the installer was trojanised and the trojanised installer was signed and distributed.
- The xz utils backdoor, where a single maintainer introduced a backdoor into a library that ended up in SSH implementations across the industry.
The pattern runs the same way in every case. The pipeline got compromised, the build artifact got signed, the signed artifact got distributed, and the customers who trusted the signing got compromised with it.
What the seven things to fix are
The honest version of the defence is to treat the CI/CD pipeline as the production system it actually is, with the same security controls, the same audit, the same review, the same rigour as the production system the pipeline is building. The pipeline is not a side effect of the development process. The pipeline serves as the production system for the build, and that is the place where the security controls need to be the most thorough.
1. Pin everything. Pin the action versions, the container base image digests, the dependency versions, the third party plugin versions. Pinning is the difference between a pipeline that uses a known good version and a pipeline that uses whatever the upstream maintainer pushed last night. The recent supply chain attacks have all exploited the unpinned dependency.
2. Use a secret manager, not environment variables. Environment variables are the secret the build needs, and they are also the secret the log captures, and the log is the secret an attacker exfiltrates. A secret manager serves as the place the build fetches the secret, and the build output never logs it, and a compromised runner does not get the secret with it.
3. Sign the artifacts with a hardware backed key. The signing key inside the CI/CD pipeline will get exfiltrated eventually, and an exfiltrated signing key is the reason a signed artifact can no longer be trusted. A hardware backed key (YubiKey, cloud HSM) cannot be exfiltrated the same way, and the artifact signed with it still means what it is supposed to mean.
4. Run the build in a fresh, ephemeral runner. A long lived runner accumulates state, and accumulated state is what an attacker compromises. A fresh, ephemeral runner starts clean for every build, and a runner that starts clean does not carry the state from a previous compromise into the next build.
5. Scan the build artifacts before they are signed and distributed. The vulnerability scanner, the secret scanner, the malware scanner. Scanning is the check that catches the bad artifact before it gets signed and shipped, and the check that catches the bad artifact early is the one that prevents the breach.
6. Restrict the network egress from the build runner. A runner that can reach the internet can exfiltrate secrets, and a runner that can reach the package mirrors can pull a compromised package. A runner with restricted network egress can only reach the internal artifact store, and that is a runner that cannot exfiltrate the secrets and cannot pull the compromised package in the first place.
7. Audit the pipeline changes the way you audit the production changes. An unreviewed pipeline change is a change an attacker pushes once they have compromised an engineer account. A pipeline change with the audit trail behind it is a change AppSec can review, and a change AppSec can review is the one where the compromise gets caught.
What to do this quarter
Pick one pipeline. Apply the seven items. See what breaks. The items are not free, the items are going to slow the build down a little, the items are going to require engineering orgs to do the work they have been quietly skipping. The items are the work, and the work sits as the difference between the org that catches the next breach and the one that features in the next disclosure.
The pipeline sits as the weakest link, and the work that closes the gap has been on the back of the napkin for years. The org that treats the pipeline as the production system it actually is, with the same controls and the same audit, will be the one still standing when the next incident lands.
The bottom line
Seven items, one shared theme. The CI/CD pipeline serves as the production system for the build, and production systems get the same controls. The org that treats the pipeline like the rest of its estate will be fine. The one that treats it as plumbing is going to learn the lesson from a breach.
Sources & Further Reading
All claims in this article are sourced from primary documentation, vendor advisories, and reputable security researchers.
Spotted an error? Email the editor. Corrections are issued with a visible correction note.
Editorial standards. Every article on humanrequired.org is reviewed by a human editor before publication. AI may assist with drafting or research; final editorial control is human. Read the full standards.
