Python local-packages, à la npm node_modules
Tags: python, pip, npm
Published: 2019-11-26 00:24:54

## Updates
Follow-up post where I made this into simple program called [pipm](https://dev.to/k4ml/pipm-pip-install-to-local-packages-and-pip-run-4goi).
**************** ## ***********

Firstly, this thing doesn't exists. Unlike npm or php composer, packaging tools in python by default will install to the global packages location first.

In npm or composer, if you don't specify `npm install -g`, then by default the package will be installed to folder `node_modules` in the project root. This quite intuitive, local first, then global.

In python's pip, if you don't want to install to the global packages directory, you can restrict it `--user` flag. For example:-

```
python3 -mpip install --user requests
```

On linux, above command usually will install `requests` package to user's `$HOME/.local/lib/python3.x/site-packages` directory. You can see that this only "local" to the user, not project unlike npm `node_modules` above.

In python, the solution is to use virtualenv. To use virtualenv however, there are at least 2 schools of taught. The first, and also the majority is to put the virtualenv external to your project.

So you might create a directory in your `$HOME/.virtualenvs` to store all your virtual environment. Creating new virtualenv may look like this:-

```
python3 -mvenv $HOME/.virtualenvs/myproject
```

Then you will 'activate' the environment in order to make it active:-

```
source $HOME/.virtualenvs/myproject/bin/activate
```

From now on, when you run `python` it will actually invoke `$HOME/.virtualenvs/myproject/bin/python`. And any packages you install with pip, for example:-

```
python -mpip install requests
```

will install the requests library to `$HOME/.virtualenvs/myproject/lib/python3.x/site-packages/requests`.

But doing all this manually found to be tedious, so there's tools like virtualenvwrapper, or modern tools like pipenv or poetry that automatically create virtualenv for you.

However I don't really like this approach. Forgetting to 'activate' the virtualenv or worst, activating wrong virtualenv for your project is one of the most painful debugging experience.

So I am in the second camp, who create the virtualenv inside our project dir. For example, if I have a project in `$HOME/myproject`, I'll do:-

```
cd $HOME/myproject
python -mvenv venv
```

So above command will create the virtualenv in the project directory - $HOME/myproject/venv. From now on, whenever I need to invoke python, I'll run `./venv/bin/python` from my project root. To install packages, I'll run:-

```
./venv/bin/python -mpip install requests
```
That will install `requests` to `./venv/lib/python3.x/site-packages`. Explicit is better than implicit.

If using poetry, you can achieve similar effect by having in your project's `poetry.toml`:-

```
[virtualenvs]
in-project = true
```

Ok, back to the original topic. What if we don't want to use virtualenv? Fortunately, pip has a flag called `--target` where if specified it will install the package to the target's location. For example:-

```
python -mpip install -t local-packages requests
```

(Using name `local-packages` as a play to `site-packages`)

Above command will instead install requests to the directory `local-packages`, which you can create in your project root, similar to `node_modules`.

Recent version of pip will also created the package's executable script (if they have one) in `local-packages/bin` directory. For example if we install package httpie, which provide executable script named `http` in `local-packages/bin/http`.

Unfortunately, if we try to run the executable script it will fail:-

```
./local-packages/bin/http 
Traceback (most recent call last):
  File "./local-packages/bin/http", line 6, in <module>
    from httpie.__main__ import main
ModuleNotFoundError: No module named 'httpie'
```

This is not surprising as `./local-packages` dir (which has our httpie packages) is not on python `sys.path` which is a list of paths where python look for modules to import. We can check it through the interactive console:-

```
python
>>> import sys
>>> sys.path
['', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/home/kamal/.local/lib/python3.7/site-packages', '/usr/lib/python3.7/site-packages']
```

To add additional path, we can specify `PYTHONPATH` environment variable:-

```
PYTHONPATH=$PWD/local-packages ./local-packages/bin/http https://httpbin.org/get
HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 177
Content-Type: application/json
Date: Mon, 25 Nov 2019 22:36:19 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block

{
    "args": {},
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip, deflate",
        "Host": "httpbin.org",
        "User-Agent": "HTTPie/1.0.3"
    },
    "origin": "14.192.213.59, 14.192.213.59",
    "url": "https://httpbin.org/get"
}
```

So it works now. But there's one gotcha. If somehow we have `httpie` already installed in global or user site-packages, that one probably will get used instead of the one we have in our `local-packages`. This is bad and can cause another headache when debugging. And this is the reason we use virtualenv. But we don't want to use virtualenv, right?

Fortunately, python has another flag. The `-S` flag will disable processing site-packages. From the [docs][S-flag]:-

>Disable the import of the module site and the site-dependent manipulations of sys.path that it entails. Also disable these manipulations if site is explicitly imported later (call site.main() if you want them to be triggered).

But we run into another problem if we invoke python with the `-S` flag:-

```
PYTHONPATH=$PWD/local-packages python -S -mhttpie https://httpbin.org/get
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/kamal/python/testpip/local-packages/httpie/__main__.py", line 18, in <module>
    main()
  File "/home/kamal/python/testpip/local-packages/httpie/__main__.py", line 10, in main
    from .core import main
  File "/home/kamal/python/testpip/local-packages/httpie/core.py", line 23, in <module>
    from httpie.client import get_response
  File "/home/kamal/python/testpip/local-packages/httpie/client.py", line 8, in <module>
    from httpie import sessions
  File "/home/kamal/python/testpip/local-packages/httpie/sessions.py", line 11, in <module>
    from httpie.plugins import plugin_manager
  File "/home/kamal/python/testpip/local-packages/httpie/plugins/__init__.py", line 10, in <module>
    from httpie.plugins.manager import PluginManager
  File "/home/kamal/python/testpip/local-packages/httpie/plugins/manager.py", line 2, in <module>
    from pkg_resources import iter_entry_points
ModuleNotFoundError: No module named 'pkg_resources'
```

Let's try copying the pkg_resources module into our `local-packages` dir:-

```
cp -a /usr/lib/python3.7/site-packages/pkg_resources local-packages/
```

No luck. We will find out that we need these modules to be available:-

* pkg_resources
* six
* appdirs
* packaging
* pyparsing

And it turned out that all these modules actually part of setuptools, another beast in python packaging. So in short, in order to make our `local-packages` works, we need to install setuptools as well:-

```
python -mpip install -t local-packages setuptools
python -mpip install -t local-packages httpie
```

And now invoking python with `-S` flag should work:-

```
PYTHONPATH=$PWD/local-packages python -S -mhttpie https://httpbin.org/get
HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 177
Content-Type: application/json
Date: Tue, 26 Nov 2019 00:15:54 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block

{
    "args": {},
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip, deflate",
        "Host": "httpbin.org",
        "User-Agent": "HTTPie/1.0.3"
    },
    "origin": "14.192.213.59, 14.192.213.59",
    "url": "https://httpbin.org/get"
}
```

And that's it. We finally manage to invoke python, free from global site-packages and using our `local-packages` dir to place all dependencies for our project.

End notes: In practice, we're using [buildout] to achieve the same objectives.

[S-flag]:https://docs.python.org/3/using/cmdline.html#id3
[buildout]:https://github.com/buildout/buildout/