Python: DIY virtualenv
Tags: python, virtualenv
Published: 2019-01-03 23:10:56

`virtualenv` in Python is a very well known tools. Almost all tutorial will recommend you to use it. In the past, I'd also wrote about why you [should not use the system python][system-python].

Being essential it is, starting with python 3.4 it was bundled within python itself as `venv` module.

There's nothing magic about `virtualenv` actually. It just a copy (or symlink) to the python interpreter that you already have in your system, plus a couple of other files.

As we have learned when we kids, the best way to learn and understand stuff is by breaking it, or try to build it from scratch. So let's try to build venv/virtualenv without using the built-in module.

As mentioned before, virtualenv is just a collection of directories and files.

```
mkdir myenv
mkdir myenv/bin
```
Then we need to find out where is the system python interpreter.

```
which python3
/usr/local/bin/python3
```

Let's copy it into our "virtualenv":-

```
cp /usr/local/bin/python3 myenv/bin/
```

Now let's try to invoke our "new" python interpreter:-

```
myenv/bin/python3
```

We should see the usual prompt such as:-

```
Python 3.6.0 (default, Jan 24 2017, 16:44:16)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
```

Now our "virtualenv" got it own interpreter, we also must make sure that it has it own `site-packages` directory, where all the packages we're going to install will live. This is the main reason we use virtualenv, so that packages we're installing do not mess up with the system python or other project virtualenv. Run `myenv/bin/python3` and run this code:-

```
>>> sys.path
['', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python36.zip', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/lib-dynload', '/Users/kamal/Library/Python/3.6/lib/python/site-packages', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages']
>>> sys.prefix
'/Library/Frameworks/Python.framework/Versions/3.6'
```

Notice that our `myenv` directory is none in the output. The [docs][sys_prefix] on `sys.prefix` says this:-

> If a virtual environment is in effect, this value will be changed in site.py to point to the virtual environment. The value for the Python installation will still be available, via base_prefix.

And following the link on [virtual environment][venv] it says this:-

> A virtual environment is a directory tree which contains Python executable files and other files which indicate that it is a virtual environment.

But what indicate a virtual environment? I cheated a bit here and created a new virtualenv using `python -mvenv tmp_env` and in the newly created virtualenv, I noticed a file called `pyenv.cfg` which contain this:-

```
home = /usr/local/bin
include-system-site-packages = false
version = 3.6.0
```

So let's try adding this file in our virtualenv as `myenv/pyenv.cfg`. After adding this file, our interpreter will give the following output:-

```
>>> import sys
>>> sys.path
['', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python36.zip', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/lib-dynload']
>>> sys.prefix
'/Users/kamal/myenv'
```

Notice the `sys.prefix` value. Now we're onto something. It correctly pointed to our virtualenv directory. However our virtualenv still not in `sys.path`. I forgot something! We still haven't created the `lib` directory. So let's do it now:-

```
mkdir -p myenv/lib/python3.6/site-packages
```

And check our `sys.path`:-

```
>>> sys.path
['', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python36.zip', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/lib-dynload', '/Users/kamal/myenv/lib/python3.6/site-packages']
```

Now our virtualenv is added to `sys.path`! Time to install some packages:-

```
myenv/bin/python3 -mpip install requests
```

But we will get the following error:-

```
/Users/kamal/myenv/bin/python3: No module named pip
```

Obviously our `site-packages` directory still empty, so we don't have pip yet. Fortunately getting pip is not that hard.

```
wget https://bootstrap.pypa.io/get-pip.py
myenv/bin/python3 get-pip.py
Collecting pip
  Using cached https://files.pythonhosted.org/packages/c2/d7/90f34cb0d83a6c5631cf71dfe64cc1054598c843a92b400e55675cc2ac37/pip-18.1-py2.py3-none-any.whl
Collecting setuptools
  Using cached https://files.pythonhosted.org/packages/37/06/754589caf971b0d2d48f151c2586f62902d93dc908e2fd9b9b9f6aa3c9dd/setuptools-40.6.3-py2.py3-none-any.whl
Collecting wheel
  Using cached https://files.pythonhosted.org/packages/ff/47/1dfa4795e24fd6f93d5d58602dd716c3f101cfd5a77cd9acbe519b44a0a9/wheel-0.32.3-py2.py3-none-any.whl
Installing collected packages: pip, setuptools, wheel
Successfully installed pip-18.1 setuptools-40.6.3 wheel-0.32.3
```

Now we have `pip` installed, let's try again installing `requests`:-

```
myenv/bin/python3 -mpip install requests
```

This time, `requests` will be installed without problem. Let's check it get installed into the correct location:-

```
myenv/bin/python3
>>> import requests
>>> requests
<module 'requests' from '/Users/kamal/myenv/lib/python3.6/site-packages/requests/__init__.py'>
```

It's correct, we have a fully functioning virtualenv now!

Astute reader might noticed that all this far, we have been invoking our python as `myenv/bin/python3`. What if we want to invoke it simply as `python3`? In virtualenv, there's a concept of `activate` that you do after you created the new env. This basically just adding the newly created env directory to the `PATH` environment variable, which is a list of search path used by the OS shell to find where the particular program exists. The `activate` script basically look like this:-

```
OLD_PATH=$PATH
PATH=`pwd`/myenv/bin:$PATH
```

To `deactivate`, we replace `PATH` back to `OLD_PATH`.

Alternatively, we can also start a subshell with PATH containing the new PATH:-

```
PATH=`pwd`/myenv/bin:$PATH sh
```

And to deactivate, we simply `exit` from the subshell. Personally I don't recommend this "activate" thing. I prefer to invoke the virtualenv interpreter directly, using it full path.

[sys_prefix]:https://docs.python.org/3/library/sys.html#sys.prefix
[venv]:https://docs.python.org/3/library/venv.html#venv-def
[system-python]:https://dev.to/k4ml/system-python-is-not-for-you-e4g