On April 1st, GitLab announced Critical Security Release CVE-2022-1162, disclosing a very bizarre vulnerability and illustrating some important lessons in securing a software supply chain. Although published on April’s Fool's Day, unfortunately, there was no prank involved. After all, a CVE with a CVSS score of 9.1 is no joke.
In this post, we’re going to analyze what happened and share the cautionary tale of misplaced trust across a software supply chain. In short, the consequences of allowing presumably trusted third-party access to your mainline code branch without sufficient code reviews and secret scanning capabilities can be potentially devastating. We’ll go into a brief forensic review, discuss the potential implications, and most importantly describe how these kinds of security risks can be prevented.
GitLab Disclosure
The GitLab vulnerability was in the OmniAuth-based user authentication mechanism, and was found by the GitLab security-research team. GitLab stated in their disclosure:
A hardcoded password was set for accounts registered using an OmniAuth provider (e.g. OAuth, LDAP, SAML) in GitLab CE/EE versions 14.7 prior to 14.7.7, 14.8 prior to 14.8.5, and 14.9 prior to 14.9.2 allowing attackers to potentially take over accounts.
That means GitLab servers of both Community and Enterprise editions running all recent versions had a backdoor. The implication was that an adversary could take over accounts and log in as any user using any OmniAuth providers by entering the hardcoded password.
Intro to OmniAuth
OmniAuth is a Standardized Multi-Provider Authentication library.
OmniAuth is a library that standardizes multi-provider authentication for web applications. It was created to be powerful, flexible, and do as little as possible. Any developer can create strategies for OmniAuth that can authenticate users via disparate systems.
By leveraging different strategies, application developers can easily plug-in authentication methods to enable secure login to their applications.
GitLab supports and advertises integrating different OmniAuth providers.
Mysterious origins - where did this backdoor come from?
How did it suddenly happen? This backdoor wasn’t out there for a long time - only newer versions were affected by it.
The original commit - https://gitlab.com/gitlab-org/gitlab/-/merge_requests/76318/diffs?commit_id=5339bc9a22f6392df7615fc3ec7a608dd95d876e looks like a legitimate commit: a ton of test files changes that enhance the password strength in testing. But looking further into it, one file stands out - lib/GitLab/auth/o_auth/user.rb. Only one small change on that pull request was deployed to production code, and it was a very critical one:
Yep! Instead of the user’s authenticated hash, GitLab used a hardcoded password. How did this code get merged into the main branch? The answer is, probably, the large number of appearances of the same parameter in that commit, namely GitLab::password.test_default.
34 files were modified in this commit:
- First, the test_default password was introduced.
- Then, 32 test files used test_default instead of the weak passwords they previously contained, to test “more complex passwords”.
- Finally, the hardcoded password test_default was used in lib/GitLab/auth/o_auth/user.rb to replace the auth_hash.password value, affecting the login to the real server.
Why was this commit required at all? The JiHu project
The commit title was “JH need more complex passwords”. So, what is JH? https://gitlab.cn/en/company/
GitLab licensed its technology to a new independent Chinese company, called JiHu. This independent company will help drive adoption of the GitLab complete DevOps platform in China and foster the GitLab community and open source contributions.
In China, there are some regulations concerning passwords, phone numbers storage, etc. To deal with those regulations, GitLab started an independent company named JiHu, which created a special edition tailored for China’s regulations, ensuring GitLab will be broadly usable in China.
Similar to CE/EE, JH is also based on the GitLab repository, but the JH project is not completely integrated back into GitLab. The JH repository was archived and moved to a different place 3 months ago. The repository movement occurred around the time of the “JH needs more complex passwords” commit (the last commit written to the archived project was pushed 4 days after the faulty commit).
The work was managed with the following methodology:
- Contributors opened merge requests in the JH project.
- The JH maintainers approved and merged the MRs in the JH project.
- The JH maintainers created MRs to the mainline GitLab project.
- Finally, mainline GitLab maintainers approved the MRs to adopt the changes from the JH project.
If you are interested in playing with the software, you can find the installation kit for Ubuntu here.
The commit’s author, @memorycancel, contributed many good commits to GitLab for the JH deployment, and is one of the biggest contributors to the GitLab JH project.
As this was the latest commit @memorycancel submitted to the JH and GitLab projects, following a long period of active contribution, we cannot be certain about the motives behind that commit. This kind of malicious activity can also happen as a result of a compromised account, among other reasons.
Resolution and Mitigation
- GitLab released versions 14.9.2, 14.8.5, and 14.7.7 for GitLab EE/CE, which fix the issue (and also serve as the monthly security release for March).
- The fix is simply reverting the original commit made on January 10th, 2022.
- Although per GitLab’s statement there are no known compromised accounts, they proactively issued password resets for a selected set of users.
- In addition, GitLab released a script aimed at identifying user accounts that are potentially impacted by the vulnerability.
Lessons for Future Prevention
How could this incident be prevented? Following software security best practices can help significantly, and recently frameworks have started to emerge to address these types of risks, such as SLSA and SSDF.
A few mitigation points that relate to this case:
- Secret scanning is a crucial part of a secured development process. The hardcoded secret that was pushed could have been prevented by using a secret detection status check on the MR.
- The code review process can be improved by increasing the number of required approvers. For example, looking at the SLSA framework, this is one of the critical Source Requirements: Two-person reviews.
- Reviewing multiple code changes within a single commit can be tough. When looking at a refactor commit like this one, the code review must be thorough. Incorporating a better way to distinguish source and test files will allow for a better visualization in future pull requests, thus making it easier for the reviewer to catch these types of subtle changes. One possible way to implement this is by using the CODEOWNERS file, indicating which owners should approve source code modifications and which owners should approve test code modifications.
- Regarding the possibility that the account was compromised - employing authentication security measures, such as MFA, can reduce the risk considerably.
This incident is yet another case showing us we can never predict the next hole that will become an ominous vulnerability in our supply chain. It is also a cautionary tale about extending trust to entities with access to your supply chain, and about the limitations of human security checks and their ability to consistently detect security issues.
However, new solution approaches are available to help that leverage (a) automated discovery and security analysis across the supply chain environment, (b) detect any removal of mandatory security controls including code reviews; and (c) automated secret scanning within the code to prevent downstream vulnerabilities.
If you are interested in learning how the Legit Security platform can protect against the risks described earlier and much more, please contact us. If you’d like to learn more about other attack patterns documented by popular frameworks check out The 3 Riskiest Software Supply Chain Attack Patterns Across Frameworks.
