Navigating the intricacies of software development and deployment often feels akin to threading a needle in a constantly shifting and changing environment, especially when trying to maintain the security and integrity of sensitive data. One of those intricacies includes the presence of 'secrets' embedded within software code. These can include username/passwords, tokens and API keys that grant access to operational processes, databases, and third-party services at the code level, and it's essential to keep them secure.
When improperly used, these secrets become critical vulnerabilities that can lead to unauthorized access of critical systems, compliance violations, and data breaches. And this is a growing problem. As software ecosystems become more complex, applications communicate with more and more external services, and the CI/CD pipeline continues to expand, the sheer volume and variety of secrets in code multiply.
This article will serve as a guide to secrets management where we’ll define secrets, offer best practices for security and maintenance, and provide a blueprint for building and maintaining a robust, secure CI/CD pipeline.
What Are ‘Secrets’?
Within software development and DevOps, "secrets" refers to passwords, tokens, API keys, PII and other sensitive data that have been embedded within software code and require stringent security control. By nature, secrets are restricted-access pieces of data that, if exposed, can lead to severe consequences. Despite their critical nature though, secrets often hide in plain sight across multiple areas of software source code as well as:
Environment settings: Configuration settings and environment variables may hold crucial data that should be kept under wraps.
APIs: These include connection strings or tokens that facilitate communication between different software services.
Production Infrastructure: Secrets can reside in server configurations, database strings, or any part of the deployment setup.
Tools: From compilers to continuous integration tools, many software utilities utilize secrets to function optimally.
Types of Secrets
Understanding the types of secrets that may be embedded in code is foundational to maintaining a secure CI/CD pipeline and help can departments improve their secrets awareness and visibility:
Username and Password Credentials: These are classic authentication methods and if a threat actor finds them, they may be able to use them access other critical systems.
API keys: These are unique identifiers used to authenticate a user, developer, or calling program to an API. Their exposure could lead to unauthorized access or misuse of the services they unlock.
SSH Keys: These are digital signatures used for securely accessing remote systems. If compromised, they can also lead to unauthorized system access and potential breaches.
Git Credentials: These are used to access code repositories across third-party databases. If exposed, unauthorized code changes or theft might occur, impacting the development process or leading to a critical security incident.
Not all secrets are system credentials or specific code strings. Sometimes, developers inadvertently leave personal identifiable information (PII) like healthcare data, credit card numbers, or bank account information within the code. In many cases, this is "test" data that’s accidentally left but can still leave an organization vulnerable to a breach and/or heavy fines for compliance violations if it’s exposed.
Properly managing and safeguarding secrets is extremely important as part of comprehensively maintaining a secure CI/CD pipeline, especially since these assets are becoming more and more at risk.
Why You Shouldn’t Sleep on Securing Secrets
The complexity and pace of modern secure software development leads to intricate processes for integration and deployment and, in order to manage the complexity, results in increasing introduction of secrets into the CI/CD pipeline. This is to facilitate interconnectivity and the access demands of multiple applications that require rapid access across different data sources, cloud services, and third-party servers. This typically results in “secret sprawl,” which refers to the risk organizations face as a result of the expanding use of secrets in code.
Some of the areas where secrets are passed through and stored include:
App/Microservice Code: Where applications could potentially expose secrets within the pre-production development environment.
Scripts: Automation or setup scripts may contain secrets crucial for efficient operation, but can also be exposed if not properly secured.
Automation Tools: CI tools that might require access to other systems in order to facilitate automated actions.
Code Repositories: Places where source code is stored and may also contain embedded secrets.
The fact that these code bases are at varying degrees of the software development stage make it even more difficult to manage, increasing the risk that they may get exposed. The fact that secrets can essentially be hidden anywhere makes visibility crucial for secrets management. WIthout the right visibility in place, either through tools or by ensuring that teams are aligned, “secrets islands” may emerge, further hindering the secrets management process.
Even if an organization utilizes vault solutions designed to secure secrets, their complexity can sometimes hinder rapid and efficient retrieval, especially in dynamic CI/CD environments. This can impact productivity which can further lead to a slowdown in overall software development, minimizing a crucial competitive advantage for many software-based organizations. This makes secrets management even trickier as department leaders need to balance security, productivity, and overall efficiency.
Secrets Management Best Practices for Every Step of the CI/CD Pipeline
To improve secrets management and avoid common pitfalls with developer environments, we have a few best practices that lead to operational efficiency as well as robust security.
Don’t Hard Code Your Secrets
This is one of the most effective secrets management practices. Educate your developers on the dangers of hardcoding secrets to prevent secrets from making their way into your code in the first place or to help ensure they can be removed easily. By reinforcing the value of properly managing secrets and utilizing secure mechanisms to access them, you’re taking a big step towards minimizing risk related to secrets in code.
Scan Scripts & Code Regularly
Continuous and ongoing monitoring is crucial given how often things change within a given environment. By regularly scanning scripts and source code for improper secrets usage or unintended secret exposures, you can quickly identify and remediate vulnerabilities before they can be exploited. Automated scanning solutions can be used here as an efficient and proactive measure for securing secrets.
Utilize One-Time Passwords Whenever Possible
As part of an overall secure CI/CD pipeline, one-time passwords (OTPs) offer a secure authentication mechanism and greatly reduce the risk associated with secrets like passwords or other authentication elements. OTPs are designed to work only a single time so if they are intercepted, they hold no value beyond their immediate use, rendering potential breaches ineffective.
Always Rotate Passwords after Use
Leveraging secret management tools to enforce periodic password rotations minimizes the window of opportunity for any malicious actors to exploit secrets in code, ensuring a constantly evolving security posture that minimizes secrets exposure.
Adopt a Policy of Least Privilege Approach
Embracing CI/CD security best practices means granting access to sensitive assets and elevated permissions only when absolutely necessary. A policy of least privilege ensures that systems and individuals can't exceed their necessary access bounds, minimizing the risk of a compromised system or account being used maliciously.
Maintain Rigorous Access Control Standards
As part of effective secrets management, establishing and maintaining stringent access control standards ensures that only vetted individuals or systems can access, modify, or manage secrets. Alongside the policy of least privileged approach, you can significantly reduce the risk of account takeover attacks that may come from accidentally exposed secrets.
Store Secrets in a Secure Location
The storage location of a secret is a large part of what determines its vulnerability. Despite the challenges involved with a vault solution, an organization should still invest in some kind of secure storage solution for secrets management, designed to protect against external threats.
Prioritize Secrets Encryption
Even in the most secure storage locations, secrets should never be stored unencrypted. Prioritizing secrets encryption adds an additional layer of protection so that even if a secrets storage or vault is exposed, it won’t necessarily mean that the content of the secrets is immediately available or accessible.
These best practices should be considered complementary approaches that provide a defense-in-depth security strategy across secrets management and beyond. These guidelines can be best implemented via a mix of tools, processes, and policies.
Secure Your CI/CD Pipeline with Secrets Management Best Practices
Secrets management is often overlooked or even ignored (inadvertently) but it’s a crucial element of a secure CI/CD pipeline, ensuring operational integrity and comprehensive security in software development and deployment. To have an effective and secure CI/CD pipeline, a secure secrets management is required.
Visibility is the starting point when it comes to secrets management. If you don’t have total visibility of your SDLC environment, assets, and VI/CD processes, you can’t fully manage and account for all your secrets. Investing in an asset discovery and secret scanning platform designed for security developer environments can help streamline your secure CI/CD pipeline, promote more efficient DevSecOps, and more comprehensively secure your secrets. The Legit Security platform was built with securing the software supply chain in mind and it's secret scanning capabilities can serve as one of the most important tools in a DevSecOps team. Book a demo to see our secret scanning and secure software delivery capabilities in action.
