Using non-PyPI Package Indices
Tags: python, packaging
Published: 2019-03-08 02:49:28

---
title: Using non-PyPI Package Indices
published: true
description: A tutorial on using Python Package Indices other than PyPI.
tags: python, packaging
canonical_url: https://pydist.com/blog/using-non-pypi-package-indices
---

By default, Python tools like `pip` install packages from PyPI, the Python Package Index. However, most tools allow you to use any server that implements the same API as PyPI (usually but not always the [Simple Repository API](https://www.python.org/dev/peps/pep-0503/)). There are two main reasons you might want to use an alternative package index:
- An index that mirrors PyPI let you keep installing packages even if they are deleted from PyPI or PyPI goes offline.
- You can upload packages to a private index instead of PyPI if you do not want to make them public.

How to configure your tools depends on whether you are using an alternative index *in place of* or *in addition to* PyPI. Mirrors can be used either way, while a private index is usually used in addition to PyPI. Note that [PyDist](https://pydist.com) or a self-hosted [devpi](https://github.com/devpi/devpi) instance can act as both a mirror and a private index.

Unfortunately, you need to configure each of your tools individually. 

## Configuring `pip`

There are two configuration options for `pip` that allow it to use alternative indices:
- The *index URL*, which is where `pip` will initially check for packages. It defaults to `https://pypi.org/simple`.
- The *extra index URLs* which are where `pip` will check for packages if they are not found in the index url. If multiple are provided, they are checked in order.

If you are using a mirror, you can either set it as the index to install all packages through the mirror, or you can set it as an extra index to only use it as a backup. If you are using a private mirror to compliment PyPI, it is tempting to use it as an extra index. However, if a package by the same name as one of your private packages is published on PyPI, this will cause `pip` to install that package instead. Thus it is safest to set the private index's URL as the index and use PyPI as an extra index.

You can configure `pip` in a number of different ways; which you should choose depends on what is easiest to set in your infrastructure and how broadly you want the configuration to apply.
- Via the command line arguments `--index-url` and `--extra-index-url`
- Via the environment variables `PIP_INDEX_URL` and `PIP_EXTRA_INDEX_URL` (note that this can only be set once, so you can only add one extra index this way)
- Via the `index-url` and `extra-index-url` settings in a [`pip.conf` file](https://pip.pypa.io/en/stable/user_guide/#config-file) which can live:
  - In the root of your virtual environment (e.g. `./venv`)
  - In your user config's `pip` subdirectory (usually `~/.config/pip` on Linux)
  - In your system-wide config directory (`/etc` on Linux)

## Configuring `twine`

The standard tool to upload Python packages is `twine`. However, only some
 rivate python indices support uploading with `twine`&mdash;to my knowledge only [PyDist](https://pydist.com) and [Gemfury](https://gemfury.com) do among hosted solutions. Other indices will generally offer their own python packages for deployment.

There are two ways to configure `twine` to upload to a repository other than PyPI:
- Via the command line `--repository-url` argument
- Via a [`.pypirc` file](https://docs.python.org/3/distutils/packageindex.html#the-pypirc-file) in your home directory

The `.pypirc` file should look like:
```
[distutils]
index-servers =
    pydist

[pydist]
repository: <index-url>
username: <username>
password: <password>
```

If you include multiple `index-servers` in `.pypirc`, you can pass the name you gave the index server (`pydist` in the example above) to the `--repository` flag when uploading. The `.pypirc` file is convenient because you will not be prompted for a username/password.

## Configuring `pipenv`

Pipenv is probably the most popular dependency-locking tool for Python. Some guides suggest using the `PIP_INDEX_URL` and `PIP_EXTRA_INDEX_URL` environment variables to configure the `pipenv` like you would `pip`, but this is not handled correctly and the maintainer [told me not to use it](https://github.com/pypa/pipenv/issues/1285).

Instead, you can use the `[[source]]` section at the top of your `Pipfile`:
```
[[source]]
url = '<index-url>'
verify_ssl = true
name = 'pydist'
```
If you want to use multiple package indices, you can include multiple `[[source]]` sections&mdash;when `pipenv` finds packages it tries them in the order they are specified, or if you declare a package like `mypackage = { version="*", index="pydist" }` it will try the specified index first. However, `pipenv`'s handling of multiple indices is [currently buggy](https://github.com/pypa/pipenv/issues/2159).