Making sense of python’s development tools in 2019
Tags: python
Published: 2019-03-08 21:23:19

---
title: Making sense of python’s development tools in 2019
published: true
description: Recommendations for setting up python projects and avoiding dependency hell
tags: python
---
Recently at work I’ve been involved in a learning group for developers new to python, and I’ve realised that the tools I’m used to using are pretty confusing for newcomers:
 
- Lots of them have very similar names but do different things, so it’s easy to confuse them in casual conversation (virtualenv, venv, pyenv, pipenv)
- There’s not one, obvious way to do everyday development tasks
- There’s a lot of active development around python packaging tools, so the best way to do something today probably won’t be the best way in one or two years time
- If you google stuff you have to trawl through a lot of out of date information

Because of this I decided I decided to demo some different tools to give an idea of what tools are available and why you might want to use them. [You can read all of my notes on github](https://github.com/MatMoore/python-build-tools-demo).

A few of the developers I’m working with are familiar with Ruby, and I realised my development workflow is basically identical in python and ruby, so I put together this rosetta stone for ruby and python tools:

| Ruby thing  | Python thing | Description |
| ------------- | ------------- | ---|
| Rubygems  | PyPi  | The centralised package respository | 
| `gem install bundler`  | `pip install pipenv`  | Install a package |
| Gemspec | setup.py | Specify metadata and dependencies for a package |
| Gemfile | Pipenv | Specify dependencies for an application |
| Gemfile.lock | Pipenv.lock | Pinned versions for the full dependency tree of an application  |
| `bundle install` | `pipenv install` | Install packages for a project | 
| rbenv | pyenv | Manage multiple versions of the interpreter |
| `bundle init` | `pipenv --python 3.7` | Start a new application project |
| `bundle gem` | ??? | Generate a package skeleton |
| rspec | pytest | Popular testing framework |
| rubocop | flake8 | Popular linting tool |
| `binding.pry` | `breakpoint()` (3.7+) | Built in debugger |

I also learned some new things myself while preparing this demo:

# 1. Virtualenv is not really needed anymore
In modern versions of python you can just use `python -m venv` to create a virtualenv. So you don’t need to install a tool through pip anymore (hurray).

Tools like `pipenv` use virtualenvs under the hood so you don't need to manage them yourself. Which is great, because as long as projects are isolated from each other, who cares where the packages are installed to? 

If [PEP 582](https://www.python.org/dev/peps/pep-0582/) is accepted, then instead of using virtualenvs there will be a standard folder in the project directory that packages can be installed to, a bit like `node_modules` in javascript.

# 2. Pipenv will probably stick around a while
For my demo I wanted to mention alternative tools even if I hadn’t used them personally. I was conscious that while pipenv [does seem to be the direction of travel](https://packaging.python.org/guides/tool-recommendations/), there is also a lot of doubt about it online.

I was curious to see whether this was influencing adoption, so I used the [PyPI downloads dataset](https://packaging.python.org/guides/analyzing-pypi-package-downloads/) on google bigquery to count the number of downloads for a handful of tools:

![Popularity of tools over the last few months](https://raw.githubusercontent.com/MatMoore/python-build-tools-demo/master/tool-usage.png) 

*Note: the data point for February isn’t a whole month, and this counts all downloads even if the package is downloaded as a dependency of another tool*

While there are definitely other contenders, pipenv still seems like a decent choice if you are starting a new python project in 2019.

# 3. Semantic versioning requirements will make your life a lot easier
One of the tasks I wanted to focus on for this demo was updating a project’s dependencies. This is important because not being able to keep up with package updates is a form of technical debt. It accumulates, and over time it becomes hard to update anything.

![Jenga fail](https://media.giphy.com/media/PlnQNcQ4RYOhG/giphy.gif)

This is particularly problematic when you depend on frameworks like Django or Flask. Using versions of software with known vulnerabilities in is the 2nd highest application security risk according to the [OWASP Top 10 2017](https://www.owasp.org/index.php/Top_10-2017_A9-Using_Components_with_Known_Vulnerabilities). 

To be able to manage dependencies over time, you should be able to separate information about what your application actually cares about (direct dependencies) from the full dependency graph at any point in time.

The former is all about *flexibility* - you want to define how your app can change as new versions of your dependencies get released. This is what the `Pipenv` file is for. The latter is all about *specificicity* - it makes sure that you run the same code on different machines. This is what the `Pipenv.lock` file is for. I’m using pipenv as an example, but this applies regardless of the tool you’re using.

Python introduced a syntax to describe version requirements way back in [PEP 440](https://www.python.org/dev/peps/pep-0440/#version-specifiers%3E). If a library follows [semantic versioning](https://semver.org), then you can write something like `requests~=2.2` to mean any version greater than or equal to `2.2`, but less than `3`. If you want to be stricter you can use `requests~=2.2.1` which means anything greater than or equal to `2.2.1` but less than `2.3` (i.e. you’ll only accept bug fixes).

If you follow this pattern, then routine updates to apply bug fixes and security patches can be completely automated. I am a huge fan of [dependabot](https://dependabot.com/) because it does all the boring work for you: it monitors for changes in your dependencies and then creates pull requests for you to review.