Extend Python VENV: Organize Dependencies Your Way
Tags: tutorial, python, cicd
Published: 2023-05-11 12:49:42

###  Introduction
Virtual environments are great way to organise development process by isolating project specific packages. In such way Python has a built-in tool `venv` for creating virtual environments. In this tutorial we are going to explore how to extend its functionality by implementing a feature that stores information about production and development related packages in separate requirements file. 

This tutorial will guide you how to implement such feature, explaining logic behind the solution and purpose of each function in the script. Prerequisites for this tutorial are basic knowledge of Python and experience with virtual environments. Examples from this tutorial requires Python 3.5+.

###  Tools

There are different ways to split dependencies in Python. [Poetry](https://python-poetry.org/docs/master/managing-dependencies/) provides versatile options to configure Python app including staging: production, development and testing. It also provides a lot of other useful options for Python developers. 

But, despite powerfull functionality there might be reasons to use simpler alternatives like `venv`. If you are working on a complex project, planning to publish your project on python repository like PyPi or it's historically based on Poetry then staying with Poetry makes sense. 

Other alternatives worth mentioning are virtualenv and pipenv. In data science and scientific computing also `conda` has substantial popularity. While these tools provide options to customize your virtual environment it's not always needed.


### Simpler alternatives

If you are working on a pet project, simple service or just don't want to complicate code with extra third party tools then using `venv` can be an ideal solution. It's built-in Python library and has limited set of commands which makes it easy to learn. In the simplest scenario there no need to learn `venv` at all. Just navige to the project directory and create virtual environment:

```bash
python -m venv .venv
```

What it will do is to call python module `venv` and create virtual environment in `.venv` directory under the current directory.

Other common names for virtual environment folders are `venv`, `env`, `.env`. Now this folder contains executables for python and it will store installed packages. Let's now install some packages. But before we can do that virtual environment must be activated

```bash
source .venv/bin/activate
```

When you no longer need virtual environment active deactivate it by simply typing `deactivate`.

In our hypothetical scenario let's assume that we are building a web app on Flask, but before that let's check that we are really in the virtual environment by calling `which python`. It will produce a path to currently used python executable. If path leads to the current directory and ends like `.venv/bin/python` then environment is activated and we are good to go.

Installing Flask

```bash
pip install Flask
```

Packages especially complex ones like Flask doesn't exist in isolation, it depends on other python packages that will be installed alongside. To ensure that this is the case call `pip freeze`. It shows a list of installed packages in our virtual environment with their corresponding versions.

A good practice is to maintain a text file `requirements.txt` with relevant dependencies so other developers can setup environment for work by installing all required packages by calling `pip install -r requirements.txt`. 

Creating `requirements.txt`

```
pip freeze > requirements.txt
```

Apart from packages that are required for running an app usually there is a need in tools that are used in development and testing. For this example we choose a package for testing `pytest`, for maintaining code style in accordance with PEP8 let's install `flake8` (for manual check) and `autopep8` that will format our code properly.

```bash
pip install pytest flake8 autopep8
```

Type `pip freeze` to see that now there are more packages. And finally save to file: `pip freeze > requirements.txt`

### Keeping dependencies separate
Here we came to the point where `venv` simplicity gives us a bit of inconvenience. Currently installed packages can be divided into 2 groups - one group of packages related to Flask are necessary for running the app. However it's redundant to install test packages into production environment. 

The solution is to maintain two separate files - one for app packages (`requirements.txt`) another for development packages (`requirements-dev.txt`). In this case testing packages are also stored in `-dev` file like for example in [Flasgger](https://github.com/flasgger/flasgger#how-to-run-tests). It's possible to split dependencies manually removing them from original `requirements.txt` and saving them into `requirements-dev.txt`. 

However if new packages will be installed this process should be repeated. At this point we have two options: either moving to more advances third-party virtual environment management tools or we can automate this process.

### Automation
Basically this process consists of two steps: save current packages list into file: `pip freeze > requirements.txt` and then some script sould sort packages keeping app related packages in `requirements.txt` while moving development packages to its development counterpart. 

This sequence can be executed manually from terminal however on UNIX based operation systems including MacOS there is more convenient solution: [make](https://www.gnu.org/software/make/manual/make.html) utility. Basic usage is `make command-name`. This utility when called search for file named `Makefile` and execute sequence of commands under command name section.

Let's create our Makefile and command

```bash
freeze:
    pip freeze > requirements.txt
    python -m split_dependencies.py
```

Calling `make freeze` will save dependencies into `requirements.txt` and call python script to split dependencies between separate files. To run it correctly first let's create the basis for a future script. Create file `split_dependencies.py` and put the following code into it:

```python
if __name__ == "__main__":
    pass
```

Now run `make freeze`. If everything has been done right terminal will produce the list of executed commands:

```
pip freeze > requirements.txt
python -m split_dependencies
```

At this point the following file structure is presented in working directory:

```
.
├── .venv
├── Makefile
├── requirements.txt
└── split_dependencies.py
```

### Script architecture
It's time to start developing the script. But before coding let's think about things that our script should be doing. It's going to consists of multiple functions and main executable function `run()` that will be responsible for executing other functions.

It's a good practice to follow single responsibility principle while coding function. So, every function will have only one task. For example list of packages should be loaded from file (`load_requirements`) but for cleaning data from newline character another function can be used. Then to obtain cleared list of packages these function will be called in a sequence one passing data to another forming data pipeline.  

A list of functions with their interfaces that will be used in the script separated by their responsibility:

**Load**
* `load_requirements(fname="requirements.txt")`
* `clean_list(items: list)` - remove newline character from each string loaded from `requirements.txt`

**Logic**
* `is_dev_requirement(item: str)` - check if current package is related to development category
* `is_prod_requirement(item: str)` - check if current package is related to production (or app running) category
* `extract(criteria: Callable, data: list)` - filter list of packages by criteria (`is_dev_requirement`, `is_prod_requirement`)

**Save**
* `prepare_data(data: list)` - prepare data for saving (joining list of filtered packages to string)
* `save_requirements(fname: str, data: str)` - save requirements to file (`requirements.txt` and `requirements-dev.txt`)

Since packages related to development and testing usually much less in quantity compared to app related packages it makes sense to store information about them for filtering purposes. Here is an example of a list with packages that might be used for development:

```python
DEV_REQUIREMENTS = ["autopep8", "black", 
                    "flake8", "pytest-asyncio", 
                    "pytest", "Faker"]
```

Also during the coding we are going to use python [typing](https://docs.python.org/3/library/typing.html) to make code more readable in terms what datatypes are used and your IDE can use this information for providing type hinting. 

Further type checkers like [mypy](https://github.com/python/mypy) can be used to enable static typing (Python by design is dynamic typing language). Since Python 3.5 typing is a built-in feature. However some additional functionality for typing can be achieved by using `typing` package (shipped with Python distribution).

### Coding

Using developed script architecture let's code the script. It's going to be managed by `run()` function where script logic will be implemented. 

**Step 1:** loading list of packages from `requirements.txt` and cleaning it from newline characters.

**Step 2:** forming two lists of packages - for production and development

**Step 3:** preparing data and saving it into two text files.

Here is a final version of a script. Feel free to experiment with it:

```python
from typing import Callable

DEV_REQUIREMENTS = ["autopep8", "black", 
                    "flake8", "pytest-asyncio", 
                    "pytest", "Faker"]


def run():
    dependencies = clean_list(load_requirements())
    dev_dependencies = extract(is_dev_requirement, 
                               dependencies)
    prod_dependencies = extract(is_prod_requirement, 
                                dependencies)
    save_requirements("requirements-dev.txt", 
                      prepare_data(dev_dependencies))
    save_requirements("requirements.txt", 
                      prepare_data(prod_dependencies))


def extract(criteria: Callable, data: list) -> list:
    return list(filter(lambda item: criteria(item), data))


def is_dev_requirement(item: str) -> bool:
    package_name, _ = item.split("==")
    return package_name in DEV_REQUIREMENTS


def is_prod_requirement(item: str) -> bool:
    return not is_dev_requirement(item)


def load_requirements(fname="requirements.txt") -> list:
    with open(fname, "r") as f:
        dependencies = f.readlines()
    return dependencies


def save_requirements(fname: str, data: str) -> None:
    with open(fname, "w") as f:
        f.write(data)


def prepare_data(data: list) -> str:
    return "\n".join(data) + "\n"


def clean_list(items: list) -> list:
    return list(map(lambda item: item.strip(), items))


if __name__ == "__main__":
    run()
```

### Commentary

* `load_requirements` - loads packages from file to list using `readlines` method
* `clean_list` removes newline character from each string through map which applies `item.strip()` to each element of `items` list
* `extract` - filters initial list using built-in function `filter`
* `is_dev_requirement` - takes a string with information about package (example: `Flask==2.2.3`), extract package name and its version and checks whether package in the list of development packages
* `is_prod_requirement` - check whether package is production related, calls `is_dev_requirement` and returns negated result
* `prepare_data` joins list of packages by newline character and adds one at the end since according to convention in UNIX based opearation systems
* `save_requirements` - writes data to file, called twice since packages are saved into two separate files. Writing is performed with `w` flag since this is text data
* `run` - implements script logic and ensures data exchange between functions

Now if someone wants to work on the project in order to set it up dependencies should be installed from both requirements files:

```bash
pip install -r requirements.txt
pip install -r requirements-dev.txt
```

### Conclusions

In this tutorial we looked at the way of extending `venv` functionality with a simple Python script that can be shipped with a project. Regarding the options on extensibility of this script there are plenty of things that can be done. For example splitting packages into three parts instead of two by introducing testing stage or applying sorting to make package list more convenient to navigate. The latter can be especially handy in large projects with lots of dependencies.

You can find the files and code used in this tutorial here: [https://github.com/dmikhr/split-dependencies-demo](https://github.com/dmikhr/split-dependencies-demo)