Easily Load non-Python Data Files from a Python Package with importlib.resources
Tags: python
Published: 2020-08-27 00:49:15

Do you ever store files alongside your Python files, and want to read them from within a running Python script?

Using `import` works great on Python modules and packages, but `import` will not work on non-Python data files, such as text files (including JSON, HTML, csv, etc.) or binary files (such as images).

## `importlib.resources` vs `__file__`

You could hack something together with the `__file__` variable, which refers to the current Python module as a file in the filesystem. And many developers do this.

However, `importlib.resources` exists, since Python 3.7, and presents a more reliable and better-looking way to load data files.

## An example

To demonstrate, let's experiment with a plain text file alongside a Python [module], in a [package].

We'll start with a simple project with [Poetry](https://python-poetry.org/) (feel free 
 to use the tool of your choice instead, or [read my intro to installing and using Poetry](https://dev.to/bowmanjd/getting-started-with-python-poetry-3ica).

```console
poetry new --src hello
cd hello
poetry install
```

In `src/hello`, we can create two files, `greeting.txt` and `greet.py`.

`greeting.txt`:

```
Hello, {recipient}!
```

`greet.py`

```python
"""Tools for greeting others."""

import importlib.resources


def greet(recipient):
    """Greet a recipient."""
    template = importlib.resources.read_text("hello", "greeting.txt")
    return template.format(recipient=recipient)
```

Now, you can launch a python console with `poetry run python`, and try the following:

```pycon
>>> from hello import greet
>>> greet.greet("World")
'Hello, World!\n'
>>> greet.greet("Universe")
'Hello, Universe!\n'
>>>
```

Note the call to `importlib.resources.read_text()`. This is what reads the contents of the file specified. There are other goodies available, such as `importlib.resources.read_binary()`. See [the `importlib.resources` docs](https://docs.python.org/3/library/importlib.html#module-importlib.resources) for further explanation and examples.

## An alternative, and why it isn't as good

Of course, this also works:

```python
import pathlib


def greet2(recipient):
    """Greet a recipient, hackily."""
    template = pathlib.Path(__file__).parent.joinpath("greeting.txt").read_text()
    return template.format(recipient=recipient)
```

There is no shame in this implementation, but it just feels a bit more hacky, using a dunder (double underscore) variable, getting the parent directory, etc. Not a big deal though.

What is a big deal is that Python packages can be bundled together in different ways. One of these ways is in a zip file. The __file__ reference will not work properly in such a zip file, while the `importlib.resources` will work fine.

In addition, `importlib.resources` makes convenient use of the import path. Let Python find the package and file in question; no need to work out the paths relative to __file__ when you are loading data from files that are located in other packages. In other words, if you can `import hello` then you can `importlib.resources.read_text("hello"...)` no matter what script or module you are currently running.

## Continue to dunder

There is one clever hack I should still mention. The line in `greet()` could be written:

```python
    template = importlib.resources.read_text(__package__, "greeting.txt")
```

The difference is that instead of naming the package explicitly with "hello", we use the dunder variable `__package__` that returns the name of the existing package.

This, of course, only works if you are attempting to load resources from the same package that contains the call to `importlib.resources`. If you move the function to a different package, it will fail.

So, while not recommended (explicit is better than implicit), at least you are aware of this option.

## The backport: `importlib_resources`

Want to use `importlib.resources` with older versions of Python, such as 3.5 or 3.6?

Thankfully, [the `importlib_resources` package exists](https://importlib-resources.readthedocs.io/).

You may `poetry add importlib-resources` or `pip install importlib-resources` then use `importlib_resources` in place of any `importlib.resources` in this article's examples.

## Other reading

As you design your modules and packages, you may find it helpful to read [my brief intro to package/module structure in Python](https://dev.to/bowmanjd/python-module-vs-package-4m8e).

Happy developing!

[module]: https://docs.python.org/3/tutorial/modules.html#more-on-modules
[package]: https://docs.python.org/3/tutorial/modules.html#packages