What Is SLSA? SLSA Explained In 5 Minutes
You’ve probably heard that software supply chain attacks are increasing rapidly and that the damage can be devastating. Both business and security...
6 min read
Gal Ofri
:
Sep 19, 2022 2:58:57 PM
Malicious actors are poisoning your artifacts in an attempt to infect your software supply chain so that you deploy those compromised (i.e., poisoned) artifacts to your production servers. In the worst scenarios, your production software then serves as a distribution mechanism for the poisoned artifact to additionally compromise your own customers and users.
Do you prefer a video? check out my talk at DevSecCon Lightning 2022, covering this very topic with some cool illustrations:
Let’s picture a modern CI/CD pipeline – one that is probably like yours. The pipeline starts with an SCM where your code waits for the build server to pick it up and produce a set of artifacts. The artifacts are packaged, then uploaded to an artifact registry, where they are stored for a while before being deployed.
When a release occurs, the artifacts are pulled from the registry and deployed to your production servers, serving your customers with your latest and greatest software. (What a beautiful picture!)
But what if someone gained access to your registry and replaced your artifact with a different, malicious artifact? This is called artifact poisoning.
(Note: You may know or refer to “artifacts” as “packages” or vice versa. For purposes of this blog, consider the two terms interchangeable.)
Artifact (aka, package) poisoning refers to the alteration or modification of software artifacts and packages by a malicious actor. This was the case with the infamous attack on Codecov, where attackers used leaked credentials to upload a malicious artifact to a GCS bucket (Google Cloud Storage), from which users then downloaded the malicious artifact directly.
By gaining access to your registry server, a malicious actor could replace your artifact with a malicious one. The replaced package could act exactly like the original one, except for an additional, harmful piece of code that could:
Steal access tokens that are stored in your code or environment variables which would allow entry to your cloud assets (e.g., AWS access key)
Redirect your customers' credentials or PII (e.g., credit card information) to the attacker
Enable lateral movement that can expand the blast radius
Divert your CPU and network resources to run the attacker’s workloads on your costly servers. (Did anyone say crypto mining?)
Even worse, distributing your infected software to other vendors who rely on it could be the perfect way for an attacker to initiate a destructive, industry-wide supply chain attack - as we saw in the case of SolarWinds.
If your development team has adopted DevOps for cloud-native application development, you are at risk. In order to release software faster and more continuously, a variety of new development tools and techniques are being rapidly adopted. Artifact registries like JFrog Artifactory and Harbor are now in widespread use by development teams. Attackers have taken notice of their widespread use and now seek to compromise these systems in your development pipelines.
Just sign your artifacts!
A Digital Signature enables one party to state that a piece of media is trusted by them using a private key. By sharing the associated public key, consumers can verify that the media is authentic (created by them) and has integrity (not altered or tampered with).
By signing artifacts before uploading them to the registry, you guarantee that the artifacts were not tampered with after they have been uploaded. Sharing your public key (with your CI/CD pipeline and customers who directly consume artifacts) allows everyone to verify the authenticity of the artifacts.
A number of open-source projects started emerging as software supply chain integrity became an increasingly serious issue. Some examples are Notary Project, Codenotary/CAS, and sigstore/cosign.
Obviously, many questions come to mind:
How do I sign artifacts?
Where do I store the signature?
How would I verify the signature?
To answer those questions, let’s take a look at cosign as an example so we can learn why we should be extra cautious with it, and how we should use it to address the problem.
Cosign is the CLI client for the sigstore toolchain, which is a set of tools and standards for signing, verifying and assuring software integrity. It provides key-pair generation; signing; and verification of artifacts.
Given a path to the artifact registry and a private key, cosign generates and uploads a signature to the registry. The path the looks like this: registry/image:sha256-4fb53f12d2ec18199f16d7c305a12c54cda68cc622484bfc3b7346a44d5024ac.sig
An artifact with the “v1.1” tag is stored in the registry.
The CI/CD invokes cosign sign
with a private-key and the path of the artifact, namely image:v1.1.
Cosign generates a signature and uploads it to the same image in the registry, but with a new tag that contains the image digest in it.
The user would invoke cosign verify
before downloading the image and confirm that the image is signed.
While simple and straightforward, an elusive security weakness is lurking in this flow. To understand it, we need to take a small step back and talk about how artifacts are typically used.
There are two primary ways to reference artifacts: digests and tags.
Every artifact has a digest, which is usually a hash value that corresponds to its content. Cosign uses the SHA-256 algorithm for hashing. A hash value (digest) looks like this: 4fb53f12d2ec18199f16d7c305a12c54cda68cc622484bfc3b7346a44d5024ac
Although a digest maps directly to the content of the artifact, it does not provide a contextual meaning for its user and is fixed to a specific version of the artifact.
So users use tags instead. Tags are labels that can be attached to artifacts. They serve as aliases for specific versions of the artifact. For example, with tags a user can always use “latest” to get the last version of the desired image.
One type of tag that deserves special mention is an immutable tag, which are used for different purposes. Only some registries support these types of tags. The “latest” tag, for example, does not make sense with immutable tags because it cannot be updated once the new version is introduced.
To wrap it up, here’s a brief comparison of digests and tags:
Digests |
Tags |
---|---|
Immutable |
Mutable |
Contextless & Long |
Human-Readable |
Auto-Calculated |
Generated by the user |
Correlate to artifact’s content |
Arbitrary label assigned by the user, often to describe the purpose/version of the artifact |
The differences between digests and tags may seem minor, but these small differences can greatly impact whether or not your software supply chain is compromised.
So, you have your package ready for uploading. You add the v1.1
tag, upload it to the registry, then invoke cosign on image:v1.1
to sign it. OOPS!
A malicious actor with access to your registry just injected a malicious artifact into your registry and made you sign it. They’d do so by uploading the malicious artifact and overwriting the tag to point to the new artifact - right after you uploaded and before you signed it.
An artifact with the v1.1 tag is stored in the registry.
A malicious actor with access to the registry uploads an artifact with malicious code to the same path, namely image:v1.1.
The CI/CD invokes cosign sign
with a private-key and the path of the artifact with the v1.1 tag, but the tag now points to the malicious artifact - with a different digest.
Cosign generates a signature and uploads it to the same image in the registry, but with a new tag that contains the digest of the malicious image in it.
The user would invoke cosign verify
before downloading the image and confirm that the image is signed. The problem is that the signed image is actually the malicious one - leading the user to run malicious code.
First, let’s clarify the root issue in play: we want to make sure that we sign the artifact that we created. Since tags are aliases and the registry is remote, signing based on a remote tag allows a malicious attacker to initiate a MITM (man-in-the-middle) attack and point the tag somewhere else, causing us to sign the wrong artifact.
To prevent the attack, we should make sure to pin down the version before we sign it.
Instead of using cosign with tags, use digests. You can then add whatever tag you want directly on the remote registry.
Since the digests map directly to our image, we can be confident that we sign the correct image.
If the attacker alters the tag on the remote side after we tag it, verifying the image will fail because the digest on the signature does not match the digest of the image.
This method requires you to use digests rather than tags, which may be less familiar.
Note that this does not prevent you from continuing using tags throughout the rest of your registry actions, which includes during verification. You just need to be careful when signing!
When you verify the image, make sure to follow the same principles: use the tag to verify the image, then extract the digest from the 'verify' output and use that digest when you deploy.
In this blog, we discussed artifact signing and why doing so is crucial to harden your CI/CD pipeline to protect against the injection of malicious artifacts (aka, artifact poisoning). Further, we learned that securely signing artifacts is not as straightforward as it might seem, even when using the most popular off-the-shelf signing tools.
Artifact poisoning is just a single type of attack on a single point in your software supply chain, but can be used to create industry-wide headlines as we saw in the Codecov incident when their artifacts were poisoned. To most effectively reduce your risk from software supply chain attacks, examine your attack surface holistically. For example:
Do you know which artifact registries your development team is using in which development pipelines?
Do you know which artifact registries are most and least critical to your business? (We recommend signing all artifacts, but knowing which are business critical can aid prioritization discussions.)
Are you enforcing strong access requirements to protect your artifact registries from unwelcome intrusions?
Do you regularly scan for secrets to reduce the risk that your artifact registries are compromised due to leaked credentials (as seen with Codecov)?
To help answer questions like these and prevent software supply chain attacks, book a demo with Legit Security today or learn more software supply chain security best practices.
Join the Legit Security Newsletter to stay up-to-date on the latest tips, tricks, and tech-industry news.
You’ve probably heard that software supply chain attacks are increasing rapidly and that the damage can be devastating. Both business and security...
The Legit Security Research Team discovered a new class of software supply chain vulnerabilities that leverages artifact poisoning and attacks the...