Use cases for Python environment variables
Today’s post focuses on environment variables in Python. They are one of several possible mechanisms for setting various configuration parameters. We can:
- read environment variables (through os.environ or dotenv ) [the current post]
- have the script accept command-line arguments (use argparse )
- load configuration settings from a file, such as:
- a JSON file (use json )
- a YAML file (use pyyaml )
- a XML file (use lxml , ElementTree or minidom )
- an INI file (use configparser ) [check out this post]
- your DIY file format (for which you will be rolling your own parser)
What is the best solution?
The answer is… it depends.
There is no one-size-fits-all solution. It depends on what you’re trying to achieve and on how the current software architecture looks like. If you’re working on a command-line tool that must accommodate a plethora of options, chances are you’ll be using argparse . For other types of projects (such as a server or a client), a configuration file might be more practical. Yet in other situations you may also want to consider using environment variables.
We will be looking in more detail at three such use cases in this post, where we will see how environment variables can be a good choice.
But first, let’s get the next point out of the way:
But environment variables are evil, right?
Indeed, using environment variables for non-sensitive information that you could just as well transmit via command-line arguments or via a configuration file is not ideal. Why? Because being environment variables, they actually live outside of the code base. Sure, you can access them based on their key (their name) and attach some meaning to them, but this is neither the most Pythonic, nor the most effective way, to do things (if this can be avoided).
Nevertheless, there are also legit cases where environment variables are preferable:
- when setting execution mode (e.g. debug or development mode vs production mode)
- when they improve security practices
- when they are the only way to get some values into a “black box” (more on that later)
Before diving into the use cases, let us first briefly see how to access environment variables in Python.
Accessing environment variables in Python
Environment variables are read through os.environ . Although they can also be modified or cleared, such changes are only effective in the current Python session (and for subprocesses started with os.system() , popen() , fork() and execv() ). In other words, if you change an environment variable in a Python script, the change will not be reflected in the environment once that script exits.
In the most simple form, you can export an environment variable through the shell:
Then you can read its value through os.environ :
Note that, for non-existent keys, os.environ.get() returns None .
Also note that the values of all environment variables are strings. To address this, you may want to roll your own small environment parser. Mine looks like this:
I use get_env_setting() to retrieve a value from os.environ (if the key exists) and I try to convert it to different data types:
- first, as a bool (this is because if I set boolean environment variables in Python, I store their str() representation, meaning ‘True’ for True and ‘False’ for False );
- if this fails, the value is converted to an int ;
- if this fails as well, the value is converted to a float :
- if successful, parse_string() returns a float ;
- if not, it returns a str .
To set multiple environment variables, you could create a bash script and ensure you run it before starting the Python script that needs these environment variables. But there is something more effective than this: dotenv allows you to load environment variables from a .env file having the following format:
Notice the .env file understands UNIX expansion (e.g. $
dotenv loads the environment variables from .env into the environment:
Now the environment variables DOMAIN , ADMIN_EMAIL and ROOT_URL are accessible to the Python script and may be retrieved via os.environ.get() as shown above.
Use case: setting execution mode
Here is a classic use case for environment variables. Suppose you don’t want to add an explicit -d / —debug flag for your app. Then you could just export an environment variable to do the trick:
The app would behave differently depending on the value of MY_APP_DEBUG .
Taking this idea one step further, you could use an environment variable MY_APP_MODE to choose between development , staging and production modes.
Use case: securing access tokens
Many applications require access tokens: they can be API tokens, database passwords and so on. Storing such sensitive information inside the code base is just an accident waiting to happen, no matter how sure you are that you’re never going to commit that special extra line to version control.
Here’s where environment variables come in handy. You could add your secret tokens to the .env file and load it with dotenv as we’ve seen above. Of course, you’d need to make sure that your .gitignore or .hgignore contains the .env file.
In short, instead of:
prefer adding your SECRET_TOKEN to .env , adding .env to your version control’s ignore file, and finally:
Use case: injecting configuration into a black box
This final use case is something you’re not going to come across very often in internet discussions. It’s something I call a “black box”, meaning code that you have no control over: you didn’t write it, you cannot change it but you have to run it. Along these lines, remember how I wrote in a previous post about creating a Python script that runs user code from other Python scripts. That’s the kind of use case I am referring to.
OK, you may ask, but why. Why would you want to run code that you have no control over? Well, suppose you’re writing a testing framework that other people may use to write tests for… well, testing stuff. The tests are not relevant, only the part about having to run them is. There are two aspects at play here:
- the framework is a library that users import from in order to write their tests;
- the framework is also a framework, meaning a master runner script that runs the user scripts.
For the users’ sake, their only task should be to read and understand the framework’s well-documented API. They should not have to fiddle around with passing configuration options into their code. The configuration options for running their scripts through the framework may be sent through the command line and/or through configuration files.
What a user script typically does is to import abstractions from the framework and to use them for creating and executing tests. For example:
Let us suppose that if the user script is ran with verbosity off (default), it only shows the test result:
When the script is ran with the —verbose flag, it displays phase results as well:
Now remember that what the fancy_framework does among other things is to simply run the provided user_script.py . How should a fancy_framework.Test object know whether verbosity is on when its execute() method is called? Here is where environment variables step in to save the day:
- The fancy_framework exports an environment variable FANCY_FRAMEWORK_VERBOSITY according to the user’s choice (whether the —verbose flag was used).
- When a Test object is initialized, it reads the value of FANCY_FRAMEWORK_VERBOSITY from os.environ and stores it in an instance variable self._verbose .
- When the execute() method of the Test instance is called, details are printed to stdout only if self._verbose is true.
Here is a simplified version (using the get_env_setting() helper we’ve seen above):
Isn’t that neat? In this use case we’ve seen how environment variables can be used to inject configuration into a black-box system.
Check out this article for a discussion of passing configuration options in Python in such a way as to only use identifiers instead of strings for the configuration keys.
How to setup .env in Python
A .env is a simple file that you can create that can store and host environment variables. Environment variables offer a way for your application to store and access variables relating to the environment of your application.
This could be anything from API keys to login credentials and passwords or special flags that you pass to your application to indicate production or development builds. .env files should never be committed to your project and should remain local only. As long as your system is not compromised, your API keys will not be compromised either.
Why use a .env file?
.env are used to store sensitive environment variables locally.
This means you can store variables pertaining to the environment, such as production or development variables needed across your application. These types of variables include API keys and login credentials for database connections and other application sensitive information.
How to setup a .env file
1.To start using a .env simply create a file called .env in the root of your project.
2.Add the .env file to your .gitignore file. If you do not have a .gitignore you can download a default Python .gitignore.
.gitignore should be located in the root of your project (same place as .env).
The purpose of a .gitignore file is to prevent git from committing specific files that are listed in this file, hence the name .gitignore since it ignores any file or directory listed in this file.
3.Set your environment variables, API keys, and any sensitive information such as login credentials inside the .env file using the following format:
Exit fullscreen mode
Note the < and > delimiter just means you replace the contents of that with the actual API_KEY you desire to hide. An example of a .env file is listed below:
Exit fullscreen mode
4.Before accessing the environment variable in our application we need to install a package that lets us locate and load the .env file.
For this, we’ll use Python’s package installer to retrieve the package python-dotenv for us, which will allow us to load our .env and access our variables within our application.
Simply install python-dotenv by using the following command in the terminal:
Exit fullscreen mode
If this command does not work, you can try alternatively using python -m pip install python-dotenv to install the package.
5.Now that we have the correct package installed we can load the .env file into our application.
Normally we would have to hardcode the location of our .env file, but luckily there is a package to automatically locate the .env file included in python-dotenv , this function is called find_dotenv() which attempts to find our .env within our project.
Exit fullscreen mode
6.At this point, we have our .env file loaded in memory, but we have no way of accessing the variables just yet.
Python has a built-in method for this. We’ll be using the os package which is already included in Python. The os package offers us a function called getenv() which will allow us to get our environment variables.
Exit fullscreen mode
What if you are creating an open source application that needs a .env file?
If you intend on making your project public but don’t want to expose your API keys .env is still the solution.
A good practice is creating a .env.example file that you commit to your code base. This file should only be a template of the environment variables the user is required to input and should not actually contain any keys or sensitive data. So for our example, we would create a .env.example file with the following contents:
Exit fullscreen mode
This can be safely committed to our project without exposing any sensitive information.
In your applications README you can indicate that a user needs to correctly set this file up and they can use the following command to create their own .env file from this template:
Exit fullscreen mode
This command simply copies the example .env file and creates an actual .env file, here the user can modify the .env file and add their credentials, API keys, etc without exposing it to the world since the .gitignore is set to ignore the .env file.
.env Naming Conventions
1. .env should have variables capitalized. This allows for better code readability, this convention is common when it comes to naming constant variables. An example is:
Переменные окружения для Python проектов
При разработки web-приложения или бота мы часто имеем дело с какой-либо секретной информацией, различными токенами и паролями (API-ключами, секретами веб-форм). "Хардкодить" эту информацию, а тем более сохранять в публично доступной системе контроля версий это очень плохая идея.
Самый простой путь решения данной проблемы, это создание отдельного конфигурационного файла со всей чувствительной информацией и добавление его в .gitignore . Минус такого подхода в том, что в гит нужно держать ещё и шаблон конфигурационного файла и не забывать его периодически обновлять.
Более продвинутый подход, это использование переменных окружения. Переменные окружения это именованные переменные, содержащие текстовую информацию, которую могут использовать запускаемые программы. Например, чтобы запустить flask-приложение, вначале нужно указать в переменной окружения FLASK_APP имя нашего приложения:
С помощью переменных окружения можно получать различные параметры приложение и секретные ключи:
Чтобы не задавать каждый раз вручную переменные окружения при новом запуске терминала, можно воспользоваться пакетом python-dotenv. Он позволяет загружать переменные окружения из файла .env в корневом каталоге приложения.
Теперь можно создать файл .env со всеми переменными среды, которые необходимы вашему приложению. Важно, добавьте .env -файл в .gitignore , не храните его в системе контроля версий.
Этот .env-файл можно использовать для всех переменных конфигурации, но его нельзя использовать для переменных среды FLASK_APP и FLASK_DEBUG , так как они необходимы уже в процессе начальной загрузки приложения.
Переменные среды могут быть автоматически загружены при входе в папку с проектом, это особенно удобно при работе с несколькими проектами одновременно. Сделать это позволяет утилита direnv. Direnv — это менеджер переменных среды для терминала, поддерживает bash, zsh, tcsh и др. оболочки. Позволяет автоматически загружать и выгружать переменные среды в зависимости от вашего текущего каталога. Это позволяет иметь переменные среды, специфичные для каждого проекта. Перед каждым приглашением проверяется наличие файла .envrc в текущем и родительском каталогах. Если файл существует, он загружается в подшаблон bash, и все экспортированные переменные затем захватываются direnv, а затем становятся доступными для оболочки.
Далее необходимо внести изменения для настройки нашей оболочки, для bash необходимо в конец файла
Using Environment Variables in Python
Below is a basic tutorial about how to set up environment variables and call them in Python. I’ve decided to create this how-to after I didn’t find any straight forward explanation on the internet. Let me start by telling you why it is a good idea to use environment variables in your code.
- Improved Security: You shouldn’t be storing passwords, tokens, or sensitive information in your source code. It’s how credential leaks like Shibu Inu’s happen… I’ve also been part of awkward company wide demos where sensitive information is on display for the entire organization to see.
- CI/CD workflows are more efficient when credentials aren’t hardcoded. I am not going to explain CI/CD is in this post, that’s another rabbit hole for a different day. Bottom line, everyone hates doing redundant things and hard coded credentials are redundant. Using environment variables stops you from having to manually change those values over and over.
Now let’s begin…
First I am going to start with a list of prerequisites.
- At least Windows 10 operating system (macOS has different requirements) — intuitive code editor
- The Python extension for VS Code
- Python downloaded and added to PATH variable on your computer
Python Libraries & Functions
Step 1: Launch VS Code, create a project folder, and download dotenv library
Create a new folder on your desktop that you will be able to open once you launch VS Code. For the purpose of this demo, I am going to create a folder names Environment Variables. This folder is used to store a set of files specific to this project and is good practice for managing code.
Launch VS Code and open the folder.
After you need to install the dotenv library to your terminal. Don’t worry about the os library as it is native to python (meaning one of the preinstalled libraries). To install the library using VS Code go to Terminal>New Terminal and in the terminal window type the code below.
Step 2: Create a .env file, a python file, and a .gitignore file
Select the new file icon and then type .env this will create a file with a gear icon beside it to store your environment variables. Next, create your python file. I am going to name my file envVar.py. After, create a .gitignore file, this is necessary because you will need to reference your .env file in the .gitignore file so all your passwords aren’t pushed to a shared repository (if you are pushing your code to a repo).
Step 3: Adding environment variables to the .env file
I am going to add some Azure variables in here as an example because I work with Azure a lot. The same thing can be done if you need to store login credentials, like a username and password. So in the environment file I am going to use the following code snippet.
After you add the details hit ctrl+K S or File> Save All
Step 4: Reference the environment variables in your python script
Drum roll please… Now it’s time to call these environment variables and store them in your script. While in the envVar.py file, you will need to import the libraries mentioned above, then use the load_dotenv() method which allows the current envVar.py file to “acknowledge” the .env file.
Then, you will assign the needed environment variables to their own variable within the envVar.py script by using the os library, getenv() method, and reference the name of the respective environment variables in the .env file.
I’ve used a print statement so we can confirm that the script is indeed calling the environment variables from the .env file. You’ll need to save the envVar.py file by hitting hit ctrl+K S or File> Save All.
Call the python script, using a new terminal (you should be pointed to the path where your project folder and files are located). Run the following snippet ‘python envVar.py’ in the terminal to test your script.
Now you shouldn’t typically print your environment variables out like this but I am confirming that you can pull them in to a script from the .env file.
Step 5: Using .gitignore to leave out the.env file when pushing to a repository
Okay so if you are going to be pushing your code to a repository and the whole point of using environment variables is to NOT share secret values, then make sure you add your .env file to the .gitignore file. gitignore is used to intentionally ignore files when tracking changes using version control or code management tools.
All you have to do is type /.env in the .gitignore file to reference the .env file where your values are stored. Save everything by using ctrl+K S or File> Save All.
You just created, stored, and used environment variables.
Well how do other team members get these environment variables then?
If you are sharing sensitive information like passwords, keys, token, certificates, etc. you should be using a key vault, encrypted email, or service like secret server to share that information.
Okay for some reason my environment variables are pointing to ones I just used in a different script, what’s going on?