Your CI/CD pipeline becomes the place where the code, the credentials, the secrets, the production access, and the third party integrations all meet, and the place is also where the security tends to be the thinnest. The CI/CD pipeline runs as the place where the developer is in a hurry, where the security review becomes the lightest, where the third party plugins are pulled in without much thought, where the secrets are stored in environment variables that get logged, where the build artifacts are signed with keys that are stored in the same environment that just got breached.
The CI/CD pipeline counts as the supply chain, and the supply chain stands as the attack surface. The recent incidents in the space have been the shape of the future, not the shape of the past.
- The Codecov breach, where the bash uploader script was modified to exfiltrate environment variables from every build that used the script.
- The SolarWinds breach, where the build pipeline was compromised and the signed updates were pushed to 18,000 customers.
- The 3CX breach, where the installer was trojanised and the trojanised installer was signed and distributed.
- The xz utils backdoor, where a single maintainer introduced a backdoor into a library that ended up in SSH implementations across the industry.
The pattern stands as the same in every case. The pipeline got compromised, the build artifact got signed, the signed artifact got distributed, and the customers who trusted the signing got compromised.
What the seven things to fix are
The honest version of the defence is to treat the CI/CD pipeline as the production system it actually is, with the same security controls, the same audit, the same review, the same rigour as the production system the pipeline is building. The CI/CD pipeline is not a side effect of the development process. The CI/CD pipeline amounts to the production system for the build, and the production system for the build runs as the place where the security controls need to be the most thorough.
1. Pin everything. Pin the action versions, the container base image digests, the dependency versions, the third party plugin versions. Pinning counts as the difference between the pipeline that uses a known good version and the pipeline that uses whatever the upstream maintainer pushed last night. The recent supply chain attacks have all exploited the unpinned dependency.
2. Use a secret manager, not environment variables. The environment variable runs as the secret the build needs, and the environment variable becomes the secret the log captures, and the log sits as the secret the attacker exfiltrates. The secret manager serves as the secret the build needs to fetch, and the secret manager does not log the secret to the build output, and the secret manager does not exfiltrate the secret when the build runner is compromised.
3. Sign the artifacts with a hardware backed key. The signing key in the CI/CD pipeline runs as the signing key that is going to get exfiltrated, and the exfiltrated signing key serves as the reason the signed artifact can no longer be trusted. The hardware backed key (YubiKey, cloud HSM) runs as the key that the attacker cannot exfiltrate, and the key the attacker cannot exfiltrate becomes the key that the signed artifact can still be trusted with.
4. Run the build in a fresh, ephemeral runner. The long lived runner sits as the runner that accumulates the state, and the accumulated state becomes the state the attacker compromises. The fresh, ephemeral runner becomes the runner that starts clean for every build, and the runner that starts clean for every build sits as the runner that does not carry the state from the previous compromise.
5. Scan the build artifacts before they are signed and distributed. The vulnerability scanner, the secret scanner, the malware scanner. The scanner amounts to the check that catches the bad artifact before the bad artifact is signed and distributed, and the check that catches the bad artifact sits as the check that prevents the breach.
6. Restrict the network egress from the build runner. The build runner that can reach the internet stands as the build runner that can exfiltrate the secrets, and the build runner that can reach the package mirrors counts as the build runner that can pull the compromised package. The build runner with the restricted network egress becomes the runner that can only reach the internal artifact store, and the runner that can only reach the internal artifact store stands as the runner that cannot exfiltrate the secrets and cannot pull the compromised package.
7. Audit the pipeline changes the way you audit the production changes. The pipeline change amounts to the change that the developer pushes without review, and the unreviewed pipeline change sits as the change that the attacker pushes when the attacker compromises the developer account. The pipeline change with the audit trail serves as the change that the security team can review, and the change the security team can review becomes the change that is going to be the change the security team catches the compromise on.
What to do this quarter
Pick one pipeline. Apply the seven items. See what breaks. The items are not free, the items are going to slow the build down a little, the items are going to require the engineering team to do the work the engineering team has been skipping. The items are the work, and the work counts as the work that the next breach is going to expose the absence of.
The pipeline amounts to the weakest link. The pipeline serves as the work. The pipeline amounts to the thing the engineering team has to do well, and the engineering team has to do the work this quarter.
The bottom line
The patterns the post covers have been showing up in production for long enough that the patterns have names, the failures, the mitigations, the gaps. The work the security team and the engineering team and the operations team are quietly doing today sits as the work that decides whether the practice the post names sits as a tool the team uses or a liability the team is paying for.
Sources & Further Reading
All claims in this article are sourced from primary documentation, vendor advisories, and reputable security researchers.
Spotted an error? Email the editor. Corrections are issued with a visible correction note.
Editorial standards. Every article on humanrequired.org is reviewed by a human editor before publication. AI may assist with drafting or research; final editorial control is human. Read the full standards.
