micropipenv: the one installation tool that covers Pipenv, Poetry and pip-tools
Tags: python
Published: 2020-11-03 21:06:21

In this article, we will take a look at a tool called "[micropipenv](https://github.com/thoth-station/micropipenv)". Its main goal is to serve as a common layer for installing Python dependencies as specified by [pip](https://pypi.org/project/pip), [pip-tools](https://pypi.org/project/pip-tools), [Pipenv](https://pypi.org/project/pipenv) or [Poetry](https://pypi.org/project/poetry).

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/4jq1tgxu0jewcjsjc116.png)

If you are a Python developer, you’ve probably experienced that dependency hell where you can easily end up in with Python’s packaging. A tool called `pip` has been around for quite some time, but its implementation is not that sufficient if you develop or maintain any larger Python-based application.

# pip-tools

A project called [pip-tools](http://pypi.org/project/pip-tools) tries to address the lack of dependency management in pip. It uses two text files - `requirements.in` and `requirements.txt`. The first file, `requirments.in` managed by a user, states direct dependencies of an application with version range specifications. The latter one, `requirements.txt` managed by `pip-tools`, acts as a lock file stating all the packages in specific versions necessary to run an application (an analogy to `npm-shrinkwrap.json` or `package-lock.json` from the npm ecosystem).

To use `pip-tools`, you need to explicitly maintain a Python virtual environment (if you need one). Starting Python 3.3., you can issue the following command to create one and activate it:

```
python3 -m venv venv/
source venv/bin/active
```

After you have created the virtual environment, commands `pip-compile` and `pip-sync` will become your friends.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/ghuqi26hct9ei00bttbd.png)
> pip-tools workflow as described in pip-tools package description available on PyPI

For dependencies needed to execute your test suite, `pip-tools` introduced a convention using `dev-requirements.in` and `dev-requirements.txt`. The semantics can be analogically deduced.

## A note to `setup.py` and `requirments.txt`

You have probably seen `requirements.txt` file used with `setup.py`. Note the difference with `pip-tools` that can be misleading for newcomers. If you want to publish a library, you don’t want to restrict all the versions by pinning transitive dependencies to specific versions as you don’t know how the resolved software stack will look like on an application side. The final resolution should always happen on the application level, not on the library level. You, as a library maintainer, just want to provide information about the compatibility of your library with dependencies used within the library.

# Pipenv

[Kenneth Reitz](https://twitter.com/kennethreitz), the author of one of the most popular Python library — [requests](https://pypi.org/project/requests/), introduced a project called [Pipenv](https://pipenv.pypa.io/en/latest/) in [January 2017](https://www.kennethreitz.org/essays/announcing-pipenv). The project gained popularity and attention from the community very quickly.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/jtqw85b2ugkv0047rxja.png)
> Pipenv: Python Dev Workflow for Humans

The main aim of the project was to simplify dependency management and make it more user-friendly. It helped to maintain a virtual environment and manage dependencies with one single command, using newly introduced files `Pipfile` (TOML) and `Pipfile.lock` (JSON).

# Pipenv in our team

We, at Red Hat, adopted Pipenv and made it one of the options for dependency management during deployment in the [OpenShift’s Source-To-Image](https://github.com/openshift/source-to-image) build process (see for example [Fedora-based Python container images](https://github.com/sclorg/s2i-python-container/tree/master/3.7) or [Thoth’s Python s2i container images](https://github.com/thoth-station/s2i-thoth)). It gained popularity also in our [AICoE](https://github.com/aicoe) and [Thoth team](https://github.com/thoth-station) where de-facto all the repositories with Python source code use Pipenv for dependency management.

# Poetry

Another option is to use [Poetry](http://pypi.org/project/poetry). It looks like Poetry attracted the Python community, especially during the silent phase of Pipenv. Poetry uses Pipenv: Python Dev Workflow for Humans
a different type of lock format than Pipenv and is not compatible with Pipenv or pip at all.

# micropipenv

Some of the requirements and ideas lead to introducing a new tool called [micropipenv](http://pypi.org/project/micropipenv). No, there is no intention to introduce another `pip`, `pip-tools`, `Poetry` or `Pipenv`.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/ienmpp1jyvupsivjyms1.png)
> https://xkcd.com/1987/


This lightweight implementation (one file, around 1200 LOC with comments) has one optional dependency and can install your requirements from files that are managed using Pipenv, Poetry, pip-tools, or simple `requirements.txt` file as commonly used together with `setup.py`. The only thing you need to do is to issue:

```
micropipenv install
```

The tool will automatically detect what type of requirements file or lock file you use for managing your dependencies and performs the desired installation.

Moreover, the tool offers a simple conversion feature where any lock file can be transformed to `requirements.txt` and/or `requirements.in`.

To produce a pip-tools style `requirements.in` and `requirements.txt` you can simply perform the following commands, assuming you have Pipenv or Poetry lock files present in the directory:

```
micropipenv requirements --no-dev > requirements.txt
micropipenv requirements --no-dev --only-direct > requirements.in
micropipenv requirements --no-default > dev-requirements.txt
micropipenv requirements --no-default --only-direct > dev-requirements.in
```

If you just want to install dependencies, you don’t need to install micropipenv at all. You can just simply download it and let it do its one time job:

```
curl https://raw.githubusercontent.com/thoth-station/micropipenv/master/micropipenv.py | python3 - install
```

See [documentation and sources](https://github.com/thoth-station/micropipenv) for more info.

# Thoth

One of the ongoing efforts in Red Hat’s Office of the CTO is a project called Thoth. The newly introduced tool in this article, micropipenv, was born in this project. Check one of our publicly available demos to see micropipenv in action (the [micropipenv demo starts at 9:00](https://www.youtube.com/watch?v=I-QC83BcLuo&t=9m)):

{% youtube I-QC83BcLuo %}

You can follow our [YouTube channel](https://www.youtube.com/channel/UClUIDuq_hQ6vlzmqM59B2Lw) for more updates.

---

# Why micropipenv?

The main reason behind micropipenv was to reduce the maintenance cost of Pipenv during its silent phase. However, it turned out to be a good idea to have such a minimalistic tool for installing dependencies, especially when it comes to containerized applications. The main advantage of micropipenv turned out to be its size.
When deploying applications in containerized environments, it’s a really good idea to maintain a lock file for the application. As the lock file states the whole dependency stack already resolved, there is no reason why there needs to be shipped Poetry or Pipenv in the container image. A tool that just installs the dependencies from any lock file supplied seems to be like a minimalistic way to go to reduce container image size and software present in it (and thus shipped with the application).

[A simple size comparison done a while back showed approximately 30.4MiB difference](https://github.com/fridex/s2i-example-micropipenv) when Pipenv was not installed into the containerized environment in comparison to a single file approach using micropipenv.

> micropipenv can read in `Pipfile.lock`, `requirements.txt` or `poetry.lock` stating already resolved software stack and install it using pip.

The design of the CLI made micropipenv a straightforward tool to make a compatibility layer between all popular Python dependency management tools available out there in the open-source world.

---

# Project Thoth

Project Thoth is an application that aims to help Python developers. If you wish to be updated on any improvements and any progress we make in project Thoth, feel free to subscribe to our [YouTube channel](https://www.youtube.com/channel/UClUIDuq_hQ6vlzmqM59B2Lw) where we post updates as well as recordings from scrum demos. We also have a [Twitter account](https://twitter.com/thothstation).