We know storing credentials or other sensitive values in a configuration file (e.g. Kubernetes yaml file) is bad, but how can we get values easily replaced without having to do a complicated string substitution or writing a custom Python script?
I often have personal environment variable files for projects that I use to store credentials and configurations in. Before working on a project I would the corresponding configuration file into the shell session. However, these files cannot be stored in git repositories or shared with coworkers or bots. Even worse, sometimes the repositories have files in them that need to be changed, which is dangerous, because it’s easy to accidentally commit these files.
Well as it turns out, there already is a good solution and it is called
envsubst. We can use
envsubst to substitute environment variable placeholders inside configuration files and we can even pipe it into other commands like Kubernetes’
envsubst < config.txt
Some systems have gettext with envsubst preinstalled. However, if it is missing, you can install it using a package manager. For macOS you can use homebrew:
Learn more about homebrew in my article about setting about your development machine in one script
brew install gettext
Let’s say, we have an existing configuration file, that want to give to someone or use with a bot. Ideally we don’t want to include credentials in that file.
# my configuration file server: https://gitlab.com/skofgar username: foo_user password: mymonkey
1. Create sample configuration file
We don’t want to check that information into a git repository nor do we want it laying around or send it to someone like this , but how can we improve this? Lets replace the information we don’t want in the file with environment variables:
server: $SERVER_URL username: $USER_NAME password: $USER_PASSWORD
2. Configure environment variables
Then define these environment variables either by defining them in the shell session with:
export SERVER_URL=https://gitlab.com/skofgar export USER_NAME=foo_user export USER_PASSWORD=mymonkey
or save them to a file (e.g.
.env) and then loading them into your current shell session by using
Make sure to useRead more here: https://ostechnix.com/difference-between-defining-bash-variables-with-and-without-export/
export, otherwise your variables are considered shell variables and might not be accessible to
To run an actual substitution, perform the following command:
envsubst < config.txt # Expected output: # server: https://gitlab.com/skofgar # username: foo_user # password: mymonkey
It is also possible to write your substitution to a new file:
envsubst < config.txt > confidential_config.txt
Piping substitution into Kubernetes and other tools
It is possible to pipe the output into other commands like
kubectl for Kubernetes (k8s).
# pipe into less envsubst < config.txt | less # pipe a deployment "deploy.yml" into kubectl apply envsubst < deploy.yml | kubectl apply -f -
This is a great way to improve your Continuous Integration and Continuous Deployment (CI/CD) pipelines or just simplify your own workflow. I use this for a project where we have to replace credentials that are temporarily needed, that we didn’t want to check into the git repository and that a CI pipeline also needs to access.
Let me know in the comments what you think of this solution and if you have come across an alternative approach.
Here’s a summary of all steps combined:
# print content of configuration file cat config.txt # expected output: # server: $SERVER_URL # username: $USER_NAME # password: $USER_PASSWORD # load environment variables source .env # replace environment variables in file content envsubst < config.txt # expected output # server: https://gitlab.com/skofgar # username: foo_user # password: mymonkey # replace environment variables and write to new file envsubst < config.txt > confidential_config.txt # pipe into less envsubst < config.txt | less # pipe a deployment "deploy.yml" into kubectl apply envsubst < deploy.yml | kubectl apply -f -
Some of the sources I used for this article are:
- envsubst for Kubernetes: https://serverfault.com/a/843883
- gettext documentation: https://www.gnu.org/software/gettext/
Also published on Medium.