Last July, we published an article exploring the dangers of vulnerable self-hosted runners and how they can lead to severe software supply chain attacks. A recent blog post by security researcher and bug bounty hunter Adnan Khan provides strong evidence for the threats we outlined and their destructive outcomes. GitHub itself was found vulnerable, as well as various notable organizations, such as PyTorch, Tensorflow, Microsoft DeepSpeed, and Chia Networks.
In this blog post, we’ll provide an overview of Khan's work and delve into the specifics of a supply chain attack that could have had a massive impact on numerous companies globally by targeting a fundamental component of the world's largest open-source software projects.
The GitHub Actions Runners Vulnerability
GitHub Actions, being the largest CI/CD service on the market and native within GitHub, offers two types of build runners: GitHub’s hosted runners (SaaS) and self-hosted runners - running on customer-provided environments. The vulnerability exploited by Khan involved the latter, where he identified a critical misconfiguration in GitHub’s actions/runner-images repository, leading to the ability to modify releases, add code directly to the main branch, and set up paths to supply chain compromise.
The Exploitation Process
Exploiting this vulnerability involved gaining access to internal GitHub infrastructure and secrets. The access potentially allowed the insertion of malicious code into all of GitHub’s runner base images, creating the opportunity to launch a supply chain attack against every GitHub customer using hosted runners. This process involved several steps:
- Identify public repositories using self-hosted runners in a non-ephemeral way (i.e., using the same environment across consecutive jobs, as opposed to running each job on a clean image), allowing persistence.
- Gain initial trust by contributing innocent content (e.g., fixing a typo), thereby overcoming GitHub’s ‘Require approval for first-time contributors’ mitigation.
- Introduce another Pull-Request, executing malicious code on the runner - like a remote access tool - allowing the attacker to gain persistence on the runner, steal secrets, and use them to increase his blast radius.
The Potential Impact
Khan’s post highlights the severe consequences of this vulnerability. He could have inserted arbitrary code into the main branch, potentially impacting the weekly deployment of runner images. He also gained access to internal macOS private cloud vCenter and Azure credentials, posing a serious threat to the integrity of GitHub's infrastructure.
Mitigation and Disclosure
This work emphasizes the importance of the mitigations we listed in our article above. Following self-hosted runners security best practices is essential for the security posture of an organization’s CI/CD environment. Khan reported the vulnerability through GitHub's HackerOne program and was awarded a $20,000 bug bounty. GitHub acknowledged the report and implemented initial mitigations.
Broader Implications
Khan and his colleague John Stawinski expanded their research to other organizations, highlighting systemic issues with self-hosted runners in CI/CD environments. They discovered vulnerabilities across various organizations, emphasizing the existential threat posed by these types of attacks. These organizations include PyTorch, Tensorflow, Microsoft DeepSpeed, and Chia Networks.
Conclusion
Adnan Khan's exploration of this supply chain attack sheds light on the vulnerabilities inherent in widely-used CI/CD services like GitHub Actions. His detailed account of exploiting the GitHub Actions Runners vulnerability serves as a cautionary tale for organizations relying on such services, underscoring the need for robust security measures and vigilant monitoring of CI/CD pipelines.
It’s important to remember that GitHub Actions isn’t the only CI/CD service that is susceptible to this kind of attack. Using self-hosted runners on GitLab CI, Azure DevOps Pipelines and more, requires security expertise and extra caution, as they are prone to runner-based vulnerabilities.
To learn more about the looming threat of CI/CD attacks and how to avoid them, contact Legit Security or request a demo of our platform.
