Why You Can Still Get Hacked Even After Signing Your Software Artifacts

Malicious actors are poisoning your artifacts in an attempt to infect your software supply chain so that you deploy those compromised (i.e., poisoned) artifacts to your production servers. In the worst scenarios, your production software then serves as a distribution mechanism for the poisoned artifact to additionally compromise your own customers and users.

Do you prefer a video? check out my talk at DevSecCon Lightning 2022, covering this very topic with some cool illustrations: 

Let’s picture a modern CI/CD pipeline – one that is probably like yours. The pipeline starts with an SCM where your code waits for the build server to pick it up and produce a set of artifacts. The artifacts are packaged, then uploaded to an artifact registry, where they are stored for a while before being deployed.

When a release occurs, the artifacts are pulled from the registry and deployed to your production servers, serving your customers with your latest and greatest software. (What a beautiful picture!)

But what if someone gained access to your registry and replaced your artifact with a different, malicious artifact? This is called artifact poisoning.

(Note: You may know or refer to “artifacts” as “packages” or vice versa. For purposes of this blog, consider the two terms interchangeable.)

What Is Artifact Poisoning?

Artifact (aka, package) poisoning refers to the alteration or modification of software artifacts and packages by a malicious actor. This was the case with the infamous attack on Codecov, where attackers used leaked credentials to upload a malicious artifact to a GCS bucket (Google Cloud Storage), from which users then downloaded the malicious artifact directly.

By gaining access to your registry server, a malicious actor could replace your artifact with a malicious one. The replaced package could act exactly like the original one, except for an additional, harmful piece of code that could:

• Steal access tokens that are stored in your code or environment variables which would allow entry to your cloud assets (e.g., AWS access key)

• Redirect your customers' credentials or PII (e.g., credit card information) to the attacker

• Enable lateral movement that can expand the blast radius

• Divert your CPU and network resources to run the attacker’s workloads on your costly servers. (Did anyone say crypto mining?)

Even worse, distributing your infected software to other vendors who rely on it could be the perfect way for an attacker to initiate a destructive, industry-wide supply chain attack - as we saw in the case of SolarWinds.

Am I At Risk of Artifact Poisoning?

If your development team has adopted DevOps for cloud-native application development, you are at risk. In order to release software faster and more continuously, a variety of new development tools and techniques are being rapidly adopted. Artifact registries like JFrog Artifactory and Harbor are now in widespread use by development teams. Attackers have taken notice of their widespread use and now seek to compromise these systems in your development pipelines.

How To Prevent Artifact Poisoning

Just sign your artifacts!

A Digital Signature enables one party to state that a piece of media is trusted by them using a private key. By sharing the associated public key, consumers can verify that the media is authentic (created by them) and has integrity (not altered or tampered with).

By signing artifacts before uploading them to the registry, you guarantee that the artifacts were not tampered with after they have been uploaded. Sharing your public key (with your CI/CD pipeline and customers who directly consume artifacts) allows everyone to verify the authenticity of the artifacts.

A number of open-source projects started emerging as software supply chain integrity became an increasingly serious issue. Some examples are Notary Project, Codenotary/CAS, and sigstore/cosign.

Obviously, many questions come to mind:

• How do I sign artifacts?

• Where do I store the signature?

• How would I verify the signature?

To answer those questions, let’s take a look at cosign as an example so we can learn why we should be extra cautious with it, and how we should use it to address the problem.

What Is Cosign?

Cosign is the CLI client for the sigstore toolchain, which is a set of tools and standards for signing, verifying and assuring software integrity. It provides key-pair generation; signing; and verification of artifacts.

Given a path to the artifact registry and a private key, cosign generates and uploads a signature to the registry. The path the looks like this: registry/image:sha256-4fb53f12d2ec18199f16d7c305a12c54cda68cc622484bfc3b7346a44d5024ac.sig

1. An artifact with the “v1.1” tag is stored in the registry.

2. The CI/CD invokes cosign sign with a private-key and the path of the artifact, namely image:v1.1.

3. Cosign generates a signature and uploads it to the same image in the registry, but with a new tag that contains the image digest in it.

4. The user would invoke cosign verify before downloading the image and confirm that the image is signed.

While simple and straightforward, an elusive security weakness is lurking in this flow. To understand it, we need to take a small step back and talk about how artifacts are typically used.

The differences between digests and tags may seem minor, but these small differences can greatly impact whether or not your software supply chain is compromised.

Using Cosign With Artifact Tagging Is A Security Risk!

So, you have your package ready for uploading. You add the v1.1 tag, upload it to the registry, then invoke cosign on image:v1.1 to sign it. OOPS!

A malicious actor with access to your registry just injected a malicious artifact into your registry and made you sign it. They’d do so by uploading the malicious artifact and overwriting the tag to point to the new artifact - right after you uploaded and before you signed it.

1. An artifact with the v1.1 tag is stored in the registry.

2. A malicious actor with access to the registry uploads an artifact with malicious code to the same path, namely image:v1.1.

3. The CI/CD invokes cosign sign with a private-key and the path of the artifact with the v1.1 tag, but the tag now points to the malicious artifact - with a different digest.

4. Cosign generates a signature and uploads it to the same image in the registry, but with a new tag that contains the digest of the malicious image in it.

5. The user would invoke cosign verify before downloading the image and confirm that the image is signed. The problem is that the signed image is actually the malicious one - leading the user to run malicious code.

Best Practices to Prevent Artifact Poisoning Attacks

First, let’s clarify the root issue in play: we want to make sure that we sign the artifact that we created. Since tags are aliases and the registry is remote, signing based on a remote tag allows a malicious attacker to initiate a MITM (man-in-the-middle) attack and point the tag somewhere else, causing us to sign the wrong artifact.

To prevent the attack, we should make sure to pin down the version before we sign it.

The Smart Way: Sign using Digests 

How It Works

Instead of using cosign with tags, use digests. You can then add whatever tag you want directly on the remote registry.

Why It works

Since the digests map directly to our image, we can be confident that we sign the correct image.

If the attacker alters the tag on the remote side after we tag it, verifying the image will fail because the digest on the signature does not match the digest of the image.

Caveats

This method requires you to use digests rather than tags, which may be less familiar.

Note that this does not prevent you from continuing using tags throughout the rest of your registry actions, which includes during verification. You just need to be careful when signing!

When you verify the image, make sure to follow the same principles: use the tag to verify the image, then extract the digest from the 'verify' output and use that digest when you deploy. 

Holistically Reduce Your Software Supply Chain Security Risks

In this blog, we discussed artifact signing and why doing so is crucial to harden your CI/CD pipeline to protect against the injection of malicious artifacts (aka, artifact poisoning). Further, we learned that securely signing artifacts is not as straightforward as it might seem, even when using the most popular off-the-shelf signing tools.

Artifact poisoning is just a single type of attack on a single point in your software supply chain, but can be used to create industry-wide headlines as we saw in the Codecov incident when their artifacts were poisoned. To most effectively reduce your risk from software supply chain attacks, examine your attack surface holistically. For example:

• Do you know which artifact registries your development team is using in which development pipelines?

• Do you know which artifact registries are most and least critical to your business? (We recommend signing all artifacts, but knowing which are business critical can aid prioritization discussions.)

• Are you enforcing strong access requirements to protect your artifact registries from unwelcome intrusions?

• Do you regularly scan for secrets to reduce the risk that your artifact registries are compromised due to leaked credentials (as seen with Codecov)?

To help answer questions like these and prevent software supply chain attacks, book a demo with Legit Security today or learn more software supply chain security best practices.