# Python Distributions and Packaging

## Summary


The Python Software Foundation does regular releases of Python, which we call the standard distribution. This is a good starting point for beginners. However, there are alternatives distributions. Developers may opt for these alternatives to suit their specific requirements or simplifying the installation process. 

Third-party packages are usually distributed via a central repository called the *Python Package Index (PyPI)*. For developers who wish to release their software to the Python community, Python provides many tools to package their software in a standard way and release the same on PyPI. As of February 2018, PyPI has over 130,000 packages.

## Discussion

### What's the standard Python distribution and what does it include?

The Python Software Foundation (PSF) releases the standard distribution of Python periodically at Python.org. This includes Python 3.x and Python 2.x until the latter is retired. Installables are available for Windows and Mac OS X. It's also possible to install Python on Windows using a package manager such as Chocolatey. 

For Linux/UNIX, source code is available although most of them come pre-installed with Python that can also be updated with their respective package managers. Python is also available for other OS such as IBM i 7.1+, iOS, VMS, HP-UX, and more; but these lag behind the official release. 

The standard implementation of Python is in C and is named *CPython*. This includes implementation of the entire standard library. 


### Are there implementations of Python other than CPython?

Yes. Implementation refers to that of the interpreter itself that takes Python code and converts into bytecode. With *CPython*, the interpreter is written in C. This enables easy interfacing of C code with Python code. 

Likewise, we have *IronPython* for .NET and *Jython* for JVM. *PyPy* uses a JIT compiler for faster execution. *Stackless Python* augments CPython to support microthreads. *MicroPython* is a slim version of CPython that can run on microcontrollers. 

These alternative implementations lag CPython releases. For example, as of February 2018, IronPython is not available for Python 3 and its Python 2 release hasn't been updated since December 2016. Jython is available only for Python 2. PyPy is available for both 2 and 3 though lagging the latest version of CPython. 


### What are some alternatives to the standard Python distribution?

There are a number of alternative Python distributions. ActivePython is a commercial distribution that bundles many useful third-party packages along with the standard library. There's also a free Community Edition. Anaconda and Enthought are two distributions customized for scientific computing and analysis. While Anaconda is free, Enthought is commercial. Intel offers an enhancement of Anaconda. 

Python(x,y) is based on Qt and Spyder. This may be good place to start for MATLAB/IDL/C/C++/Fortran programmers who wish to switch to Python. Other alternatives include WinPython, Conceptive Python, PyIMSL, and more. Stackless Python is limited by the heap and not the stack for recursion, and it consumes less memory. 

The purpose of all of these distributions is to simplify the process of installation and package updates. Commercial distributions also offer support such porting Python to an exotic platform or maintaining Python 2.x for legacy codebases. 

What if you wish to use Python without bothering with any installation at all? PythonAnywhere allows you to code and execute on the cloud from a web browser. It's ideal for teachers and students.


### What's the purpose of PyPI?

PyPI serves as a central repository for sharing third-party packages. Developers can go to PyPI and search for packages that they might find useful in their applications. If source code is open (in sites outside PyPI), they can also improve the package. This is often a better approach than developing something from scratch. Note two variants of PyPI:

    + Production site: Main download repository, powered by a codebase called *Warehouse*.
    + Test site: Site where developers can test their packaging before releasing to production site.While it's possible to download a package manually (as source or binary), build and install the same, having packages on PyPI makes the process easier. Packages are easier to find. We can use a standard tool such as `pip` to manage packages and `pip` automatically knows where to look for packages.


### As a Python user, how can I install a package?

The tool *pip* can be used to install a package from PyPI. It replaces an earlier tool named *easy\_install*. If pip itself is missing on your system, use system tools to install the same; or download get-pip.py and execute `python get-pip.py`. While pip is sufficient to install pre-built binaries, to build from source you will also need *setuptools* and *wheel*, which can installed with the command `pip install setuptools wheel`. 

If you're working on multiple Python projects, each on a different version of Python, then it's recommended to set up *virtual environments* and install required packages within each environment. Run `pip install virtualenv` and then create/activate it using `virtualenv myproj; source myproj/bin/activate`. It's common to list all project requirements in a file such as requirements.txt and then install all of them by running `pip -r requirements.txt`. This file typically mentions each package name followed by a suitable version in a format specified in PEP 440. 


### I don't find pip suitable for my workflow. What are my options?

An easier approach to using `pip` and `virtualenv` separately, or updating requirements.txt independently, is to use the package `pipenv`, which manages and tracks dependencies using `Pipfile` and `Pipfile.lock` files. 

If you wish to create custom workflows, `pip-tools` is recommended. Other alternatives include `hatch` and `poetry`. 


### What's the definition of a package, packaging and distribution?

*Package* is a bundle of one or more Python modules that can be shared, downloaded and installed. A package is really nothing more than a folder that contains its module files (and optionally sub-packages) and a &lowbar;&lowbar;init&lowbar;&lowbar;.py. The process and tools used in making and uploading packages is called *packaging*. The term *distribution* is usually reserved for a Python distribution and must not be confused for package distribution. 


### What are Eggs and Wheels?

Both are built distribution formats for packages. Wheels have replaced Eggs as the preferred format. Installer pip supports only Wheels while Eggs can be installed with the older *easy\_install* tool. Wheels don't include compiled Python files. In a built distribution, files and metadata are included. The files are simply moved to correct locations while installing. There's no separate build step before installation. 

Python packaging specifies how to express dependencies and integration in the database of Python distributions. Wheels implements these but not Eggs. Wheels have a richer naming convention. Their binary format is standardized. For these reasons, Wheels are preferred. 

There are three types of Wheels:

    + **Universal Wheel**: Project is pure Python and natively supports both Python 2 and 3.
    + **Pure Python Wheel**: Project is pure Python and does not natively supports both Python 2 and 3.
    + **Platform Wheel**: Project contains compiled extensions.Since pip will prefer a wheel over source distribution, avoid making universal wheels if your project contains extensions. If you're not providing the source and your project is not pure Python, then multiple wheels will need to be provided for every platform that you wish to support.


### Can you guide me in packaging and distributing my own package?

The following are needed and can be installed using pip: 

    + **setuptools**: This extends the older *distutils*. The project specification file is setup.py, which can further be controlled with setup.cfg.
    + **wheel**: This enables creation of wheels.
    + **twine**: It's possible to upload your package to PyPI by executing setup.py (register and upload). However, Twine is preferred. This enables you to upload your package to PyPI securely. Control of where to upload can be specified in file .pypirc in your home folder.
    + **tox**: This enables local installation and testing of your package in different environments. Configuration is usually given in a file named tox.ini. A similar testing can be done on the cloud using a service such as Travis CI.The command `python setup.py clean --all sdist bdist_wheel` will perform the build: sdist for source distribution and bdist\_wheel for creating wheels. This will create two folders build/ and dist/. The former contains temporary files used during the build process. The latter contains the final source archive (\*.tar.gz) and/or binary wheels (\*.whl).

## Milestones

2000

Python adds *disutils* to the standard library as a cross-platform build and distribution system. A replacement to this called *disutils2* was worked on during 2009-2012 but this didn't make it into Python 3.3. 

2003

PyPI goes online. 

2004

Package installer *easy\_install* is released as part of *setuptools*. Problem with easy\_install is that it isn't easy to uninstall packages. 

2008

Package installer *pip* is released as an alternative to *easy\_installer*. This uses *Wheel* rather than *Egg* as the built distribution format. It's possible to use this to save package requirements to a file (`pip freeze > reqs.txt`) and recreate the same environment on another system (`pip install -r reqs.txt`). Wheel itself is published later in 2012-13. 

Feb  
2011

*The Python Packaging Authority (PyPA)* is created. It's job is to drive standardization, maintain tools and documentation related to Python packaging. It recognizes that the Python packaging and distribution infrastructure needs improvements. 

Jul  
2012

Anaconda 0.8.0. In February 2018, Anaconda 5.1.0 is released. 

Sep  
2012

WinPython v0.6 is released. In November 2017, WinPython 3.6.3 is released. 

Mar  
2014

Python 3.4 is released and it includes *pip* installer by default. 

Apr  
2018

Kenneth Reitz releases version 11.10.1 of **pipenv** at PyPI. Meanwhile, **Warehouse** replaces legacy code to power PyPI. This means that requests to `pypi.python.org` are redirected to `pypi.org` and they both share the same data store.