6 min read

Why You Can Still Get Hacked Even After Signing Your Software Artifacts

Featured Image

Malicious actors are poisoning your artifacts in an attempt to infect your software supply chain so that you deploy those compromised (i.e. poisoned) artifacts to your production servers. In the worst scenarios, your production software then serves as a distribution mechanism for the poisoned artifact to additionally compromise your own customers and users.

Let’s picture a modern CI/CD pipeline – one that is probably like yours. The pipeline starts with an SCM where your code waits for the build server to pick it up and produce a set of artifacts. The artifacts are packaged, then uploaded to an artifact registry, where they are stored for a while before being deployed.

When a release occurs, the artifacts are pulled from the registry and deployed to your production servers, serving your customers with your latest and greatest software. (What a beautiful picture!)

But what if someone gained access to your registry and replaced your artifact with a different, malicious artifact? This is called artifact poisoning.

(Note: You may know or refer to “artifacts” as “packages” or vice versa. For purposes of this blog, consider the two terms interchangeable.)

poison image

What Is Artifact Poisoning?

Artifact (aka, package) poisoning refers to the alteration or modification of software artifacts and packages by a malicious actor. This was the case with the infamous attack on Codecov, where attackers used leaked credentials to upload a malicious artifact to a GCS bucket (Google Cloud Storage), from which users then downloaded the malicious artifact directly.

By gaining access to your registry server, a malicious actor could replace your artifact with a malicious one. The replaced package could act exactly like the original one, except for an additional, harmful piece of code that could:

  • Steal access tokens that are stored in your code or environment variables which would allow entry to your cloud assets (e.g., AWS access key)

  • Redirect your customers' credentials or PII (e.g., credit card information) to the attacker

  • Enable lateral movement that can expand the blast radius

  • Divert your CPU and network resources to run the attacker’s workloads on your costly servers. (Did anyone say crypto mining?)

Even worse, distributing your infected software to other vendors who rely on it could be the perfect way for an attacker to initiate a destructive, industry-wide supply chain attack - as we saw in the case of SolarWinds.

 

HubSpot Video

 

Am I At Risk of Artifact Poisoning?

If your development team has adopted DevOps for cloud-native application development, you are at risk. In order to release software faster and more continuously, a variety of new development tools and techniques are being rapidly adopted. Artifact registries like JFrog Artifactory and Harbor are now in widespread use by development teams. Attackers have taken notice of their widespread use and now seek to compromise these systems in your development pipelines.

How To Prevent Artifact Poisoning

signing ups package analogy image

Just sign your artifacts!

A Digital Signature enables one party to state that a piece of media is trusted by them, using a private key. By sharing the associated public key, consumers can verify that the media is authentic (created by them) and has integrity (not altered or tampered with).

By signing artifacts before uploading them to the registry, you guarantee that the artifacts were not tampered with after they have been uploaded. Sharing your public key (with your CI/CD pipeline and customers who directly consume artifacts) allows everyone to verify the authenticity of the artifacts.

A number of open-source projects started emerging as software supply chain integrity becomes an increasingly serious issue. Some examples are Notary Project, Codenotary/CAS and sigstore/cosign.

Obviously, many questions come to mind:

  • How do I sign artifacts?

  • Where do I store the signature?

  • How would I verify the signature?

To answer those questions, let’s take a look at cosign as an example so we can learn why we should be extra cautious with it, and how we should use it to address the problem.

What Is Cosign?

Cosign is the CLI client for the sigstore toolchain, which is a set of tools and standards for signing, verifying and assuring software integrity. It provides key-pair generation; signing; and verification of artifacts.

Given a path to the artifact registry and a private key, cosign generates and uploads a signature to the registry. The path the looks like this: registry/image:sha256-4fb53f12d2ec18199f16d7c305a12c54cda68cc622484bfc3b7346a44d5024ac.sig


basic cosign artifact signing flow diagram v2

  1. An artifact with the “v1.1” tag is stored in the registry.

  2. The CI/CD invokes cosign sign with a private-key and the path of the artifact, namely image:v1.1.

  3. Cosign generates a signature and uploads it to the same image in the registry, but with a new tag that contains the image digest in it.

  4. The user would invoke cosign verify before downloading the image and confirm that the image is signed.

While simple and straightforward, an elusive security weakness is lurking in this flow. To understand it, we need to take a small step back and talk about how artifacts are typically used.

Referencing Artifacts: Digests and Tags

There are two primary ways to reference artifacts: digests and tags.

Every artifact has a digest, which is usually a hash value that corresponds to its content. Cosign uses the SHA-256 algorithm for hashing. A hash value (digest) looks like this: 4fb53f12d2ec18199f16d7c305a12c54cda68cc622484bfc3b7346a44d5024ac

Although a digest maps directly to the content of the artifact, it does not provide a contextual meaning for its user and is fixed to a specific version of the artifact.

So users use tags instead. Tags are labels that can be attached to artifacts. They serve as aliases for specific versions of the artifact. For example, with tags a user can always use “image:v1.1” to get the last version of the desired image.

One type of tag that deserves special mention is an immutable tag, which are used for different purposes. Only some registries support these types of tags. The “latest” tag, for example, does not make sense with immutable tags because it cannot be updated once the new version is introduced.

To wrap it up, here’s a brief comparison of digests and tags:

Digests

Tags

Immutable

Mutable

Contextless & Long

Human-Readable

Auto-Calculated

Generated by the user

Correlate to artifact’s content

Arbitrary label assigned by the user, often to describe the purpose/version of the artifact


The differences between digests and tags may seem minor, but these small differences can greatly impact whether or not your software supply chain is compromised.

Using Cosign With Artifact Tagging Is A Security Risk!

So, you have your package ready for uploading. You add the v1.1 tag, upload it to the registry, then invoke cosign on image:v1.1 to sign it. OOPS!

A malicious actor with access to your registry just injected a malicious artifact into your registry and made you sign it. They’d do so by uploading the malicious artifact and overwriting the tag to point to the new artifact - right after you uploaded and before you signed it.

compromised cosign artifact signing flow diagram where malicious actor swaps poisoned artifact for the good artifact and then makes you sign the bad artifact v2

  1. An artifact with the v1.1 tag is stored in the registry.

  2. A malicious actor with access to the registry uploads an artifact with malicious code to the same path, namely image:v1.1.

  3. The CI/CD invokes cosign sign with a private-key and the path of the artifact with the v1.1 tag, but the tag now points to the malicious artifact - with a different digest.

  4. Cosign generates a signature and uploads it to the same image in the registry, but with a new tag that contains the digest of the malicious image in it.

  5. The user would invoke cosign verify before downloading the image and confirm that the image is signed. The problem is that the signed image is actually the malicious one - leading the user to run malicious code.

Best Practices to Prevent Artifact Poisoning Attacks

First, let’s clarify the root issue in play: we want to make sure that we sign the artifact that we created. Since tags are aliases and the registry is remote, signing based on a remote tag allows a malicious attacker to initiate a MITM (man-in-the-middle) attack and point the tag somewhere else, causing us to sign the wrong artifact.

To prevent the attack, we have two options:

The Smart Way: Digests (Recommended)

How It Works

Instead of using cosign with tags, use digests. You can then add whatever tag you want directly on the remote registry.example of how to use digests to securely sign artifacts

Why It works

Since the digests map directly to our image, we can be certain that we sign the correct image.

If the attacker alters the tag on the remote side after we tag it, verifying the image will fail because the digest on the signature does not match the digest of the image.

Caveats

This method requires you to use digests rather than tags, which may be less familiar.

Note that this does not prevent you from continuing using tags throughout the rest of your registry actions, which includes during verification. You just need to be careful when signing!

The Hard Way: Tags (Alternate Method)

How It Works

We can first set up a local registry, upload the image to it, and use cosign to sign it locally. Then, we can upload both the image and the signature to the corresponding paths on the remote registry.example of securely using tags to sign an artifact

Why It works

Since we sign the image locally, access to the remote registry is not enough to allow a malicious actor to intervene by altering the tag before we sign it. If the attacker alters the tag on the remote registry after we upload it, verifying the image will fail because the digest on the signature does not match the digest of the image.

Caveats

  • Effort - Need to set up a local registry.

  • Overhead - Need to upload everything twice.

  • Cumbersome - Need to upload everything manually, repeating work that cosign could do for us.

Holistically Reduce Your Software Supply Chain Security Risks

In this blog, we discussed artifact signing and why doing so is crucial to harden your CI/CD pipeline to protect against the injection of malicious artifacts (aka, artifact poisoning). Further, we learned that securely signing artifacts is not as straightforward as it might seem, even when using the most popular off-the-shelf signing tools.

Artifact poisoning is just a single type of attack on a single point in your software supply chain, but can be used to create industry-wide headlines as we saw in the Codecov incident when their artifacts were poisoned. To most effectively reduce your risk from software supply chain attacks, examine your attack surface holistically. For example:

  • Do you know which artifact registries your development team is using in which development pipelines?

  • Do you know which artifact registries are most and least critical to your business? (We recommend signing all artifacts, but knowing which are business critical can aid prioritization discussions.)

  • Are you enforcing strong access requirements to protect your artifact registries from unwelcome intrusions?

  • Do you regularly scan for secrets to reduce the risk that your artifact registries are compromised due to leaked credentials (as seen with Codecov)?

To help answer questions like these and prevent software supply chain attacks, book a demo with Legit Security today or learn more software supply chain security best practices.

Related Blogs

Software Supply Chain Risks: What Every CISO Needs to Know

As software technology evolves, it’s being continuously integrated into nearly every aspect of business processes. And while this has given many...

Read More

Why You Can Still Get Hacked Even After Signing Your Software Artifacts

Malicious actors are poisoning your artifacts in an attempt to infect your software supply chain so that you deploy those compromised (i.e. poisoned)...

Read More

New Software Supply Chain Attack Installs Trojans on Adobe's Magento E-Commerce Platform

A popular vendor of Magento-Wordpress plug-ins/integrations with over 200,000 downloads, has been hacked. This recent attack is a reminder that...

Read More