Committing secrets to code repos or version control systems is considered one of the major security incidents, as a developer or code owners, we should be conscious about what we are pushing to the VCS. The reason for considering a security incident is, if we accidentally commit those secrets to public repos, it can be used by bad people to do illegal things.
By secrets what I mean is
- username and password (database, online services)
- API keys or access tokens
- cloud account credentials (AWS, GCloud, Azure, DigitalOcean etc.,)
- webhook credentials (slack, MS teams)
- private keys (SSH, digital certificates)
- JWT secrets
These secrets may be used inside the codebase inside a file or fetched from .env.
In recent days bad people spin up several machines to mine cryptocurrencies or use them to perform Denial of Service attacks.
It is important to implement a secret scanning step in SDLC to reduce risks minimally. There are a few things that can be followed to reduce committing these secrets to the codebase.
- ignoring files in VCS (.gitignore, svn:ignore)
- adding git hooks to dev workflow
- adding code scanning tools in CI/CD pipeline
Contents
.env approach
During development, we use various external services by using their API keys or tokens. It is good to avoid hard coding the tokens directly inside our code. we can make use of Environment Variables, by declaring them in .env file. Most of the popular languages and frameworks support Environment variables.
Now you may think if we are using them from the .env file wouldn’t this also make the exact situation we are trying to handle. We can add .env file to .gitignore so that it will not be pushed to the remote repo. So to make sure other developers don’t face any issues because they don’t have .env file, we can add .env.template file with dummy values and update our documentation to make them create .env file.
DB_USERNAME=real_value
DB_PASSWORD=real_valueDB_USERNAME=change_this_value
DB_PASSWORD=change_this_valueBy using this approach, if a new developer clones our code they will have to create a .env file and copy the contents .env.template file into .env file and update the values. This will make sure, no real credentials will be pushed to the remote repository.
Git Hooks
This approach helps us to detect and avoid scenarios where we may have used credentials inside codebase other than .env file. Here are the few open source solutions,
- talisman
- git-secrets
- GitGuardian
Talisman
Talisman is a tool that installs a hook to your repository to ensure that potential secrets or sensitive information do not leave the developer’s workstation. It validates the outgoing changeset for things that look suspicious – such as potential SSH keys, authorization tokens, private keys, etc.
Talisman supports Linux, Mac, and Windows operating systems. Talisman can be used at the global level or single repo level. It can be set up as a pre-commit or pre-push hook to git repos. The recommended way to install the talisman is as a pre-commit hook.
we can then use pre-commit or husky to make use of talisman pre-commit to check runs during every commit.
Talisman detects base64, hex, pem files, credit card numbers, and also custom patterns that can be added to the checking by adding them to the .talismanrc file in the root or inside repo. This tool also provides git history scanning and an HTML reporting facility.
to scan a repo, run talisman –scan, and to get HTML report run talisman –scanWithHtml inside the repo folder.
git-secrets
git-secrets is created and maintained by AWS labs, is also an open-source tool like the talisman.
It Prevents you from committing secrets and credentials into git repositories
It supports a pre-commit hook by default. We need to configure what kind of secrets it needs to look for in a repo. Running git secrets --register-aws it will add the AWS-specific checking. It also supports custom pattern checking by using git secrets --add. git secrets --list will list all patterns used for scanning the repo. To scan a repo run git secrets –scan.
GitGuardian
- can be configured with remote code hosting sites like GitHub, Gitlab, and Bitbucket. GitGuardian scans and reports repositories in real-time.
- can also be configured as git hooks in the developer machine or added to CI/CD pipelines.
- CI/CD pipeline support is available for all major players in the market like Azure pipelines, Bitbucket, GitHub Actions, Gitlab, Circle CI, Jenkins, Travis CI.
The above software packages prevent adding secrets to the git history. But we may have scenarios where we have to remove secrets from git history which were added in the past. To solve this type of scenario we can make use of tools like BFG repo-cleaner or git filter-repo.
CI/CD secrets
We need to be cautious when running CI/CD pipelines because we will run into scenarios where we need to use access tokens or private keys to access remote instances or services.
CI/CD platforms provide secret storage facilities where we add them and use them whenever needed in the pipeline. We also need to make sure these secrets are masked and don’t visible plainly in the pipeline logs.
Previous Incidents
Summary
- always be conscious about what you are pushing to remote repositories
- use git pre-commit hooks to ensure no secret is added to git history
