Python entry points
Tags: python, programming, tutorial, learning
Published: 2023-10-19 07:52:45

Recently I became interested in using entry points to "invade" the namespace of a package. We'll define a base package which will allow different functionality through modules. We'll then expose the entry points and load any and all modules which adhere to our definitions.
We'll simulate a video/audio encoding tool. This should help make thing a bit more realistic. This will NOT be a tutorial on encoding/decoding, so if you're looking for something like that, this is the wrong article for you.

## The base package

We need three modules in our base package: the main module which will expose the functionality to the users, a default module which will demonstrate the API and a plugins module which will expose a function to load plugins.
We'll call our base package "ender" (encoder + decoder), because it's fun and I'm also a fan of the books by Orson Scott Card. We'll create a plugin module "ender_mp3" which will use entry points to provide its functionality.
The file structure for the base package is as follows:

```bash
ender/
├── ender
│   ├── __init__.py
│   ├── empty.py
│   ├── main.py
│   └── plugins.py
├── poetry.lock
├── pyproject.toml
├── README.md
```

Now let's define the main part of our core package. We'll create a class to manage all modules and run them.

```python
# in main.py
class Ender:
	def __init__(self):
		self.modules = []

	def work(self):
		for module in self.modules:
			module.work()

def main():
	e = Ender()
	e.work()
	return 0


if __name__ == '__main__':
	# behave like a nice CLI and return 0 when successful
	exit(main())
```

As you can see from the code above, the module is very simple. It initializes a list of modules and exposes a method `work`. When that method is called, all the registered modules will do their thing. We'll extend this later, but for now it's going to be enough to demonstrate how to register modules and use them.
You can run the main file, and you'll see nothing, because we haven't registered any modules.

```bash
poetry run python ender/main.py
```

Let's fix that and register a default empty encoder/decoder. First create the module where it will live:

```python
# ender/empty.py
class EmptyEncoderDecoder:
	def work(self):
		print(f"'{type(self).__name__}'", "Default empty encoder/decoder working...")

	def register(self, core):
		core.modules.append(self)


def register(core):
	m = EmptyEncoderDecoder()
	m.register(core)
```

We've defined the empty encoder/decoder and we've included a `register` method, which we'll use to add our module to the core. We've also exposed a `register` function which will be called by our plugin loading logic. Let's go implement that:

```python
# in ender/plugins.py
import importlib.metadata


def load_plugins():
    entry_points = importlib.metadata.entry_points()
    plugins = entry_points.select(group='ender.plugins')

    plugin_modules = []
    for plugin in plugins:
        plugin_modules.append(plugin.load())
    return plugin_modules

```

Let's break down what's going on in this file. We've declared a function `load_plugins` which can be imported by the core module and used to fetch all registered plugins. The line `plugins = entry_points.select(group='ender.plugins')` loads any module from any package which has an entry point named `'ender.plugins'` defined either in a `setup.py` file or in the `pyproject.toml`. After that the module is actually loaded and then passed down to the list which is returned from the `load_plugins` function. Let's take a look at the entry point registration in pyproject.toml (managed by `poetry`):

```toml
# in the pyproject.toml of the base ender package
[tool.poetry.plugins."ender.plugins"]
"default" = "ender.empty"
```

Note: For other plugin management tools this section will look differently.

You can see we've registered a `"default"` module within the `"ender.plugins"` entry point, which points to the actual module `'ender.empty'`, which just maps to `ender/empty.py`. Now when the `load_plugins` function is run, the module will be loaded. Let's wire that up in our `main.py`:

```python
# in ender/main.py
from ender.plugins import load_plugins

def main():
	e = Ender()
	# load and register plugin modules
	plugins = load_plugins()
	for plugin in plugins:
		plugin.register(e)
	
	e.work()
	return 0
```

If we run the code now, we'll get the output:

```bash
$ poetry run python ender/main.py
'EmptyEncoderDecoder' Default empty encoder/decoder working...
```

## Take it one step further

Now we have made the local default encoder/decoder work, let's go and make a separate package. Create a plugin package using poetry:

```bash
poetry new ender_mp3
```

The file structure we get is:

```
ender_mp3/
├── ender_mp3
│   ├── __init__.py
├── pyproject.toml
├── README.md
└── tests
    └── __init__.py
```

Let's create our module in the ender_mp3 directory and add an mp3 encoder/decoder:

```python
# in ender_mp3/ender_mp3/mp3_module.py
class Mp3EncoderDecoder:
    def register(self, core):
        core.modules.append(self)

    def work(self):
        print(f"'{type(self).__name__}'", "Encoding/Decoding MP3.")


def register(core):
    m = Mp3EncoderDecoder()
    m.register(core)

```

Then add an entry point to your `pyproject.toml`:

```toml
# in ender_mp3/pyproject.toml
[tool.poetry.plugins."ender.plugins"]
"mp3" = "ender_mp3.mp3_module"
```

The same as before, we've registered our plugin. Notice that we're not making use of the name we've given our plugin, here `'mp3'`, but it is indeed assigned to our `EntryPoint`. Depending on the use-case it might be needed or not.

Let's install it now and see it all come together. First go back to your `ender` directory and run:

```bash
poetry add ../ender_mp3
```

This is assuming you've put the two directories next to each other:

```
.
├── ender
│   ├── ender
│   │   ├── __init__.py
│   │   ├── empty.py
│   │   ├── main.py
│   │   └── plugins.py
│   ├── poetry.lock
│   ├── pyproject.toml
│   ├── README.md
└── ender_mp3
    ├── ender_mp3
    │   ├── __init__.py
    │   └── mp3_module.py
    ├── pyproject.toml
    ├── README.md
```

Now when you run your main package, you'll get more output:

```bash
$ poetry run python ender/main.py
'EmptyEncoderDecoder' Default empty encoder/decoder working...
'Mp3EncoderDecoder' Encoding/Decoding MP3.
```

## Beyond the basics

There are of course packages which have dealt with all the tricky parts of this workflow. We'll take a look at 'pluggy' which takes this concept to another level.

Let's rewrite our core package and then we'll go on to rewriting the plugin.

Install `pluggy`

```bash
poetry add pluggy
```

First create a spec file for specific hooks:

```python
# in ender/hookspec.py
import pluggy

hookspec = pluggy.HookspecMarker('ender.plugins')


@hookspec
def ender_plugins_load_module():
    pass

```

Now let's make our default implementation use the `pluggy` specs:

```python
# in ender/empty.py
from ender.hookimpl import hookimpl


class DefaultModule:
    def work(self):
        print(f"'{type(self).__name__}'", "Default empty module")

    def register(self, core):
        core.modules.append(self)


@hookimpl
def ender_plugins_load_module():
    return DefaultModule()

```

So we've replaced our register function with a hook implementation annotated function which just returns the module.

Now let's go ahead and modify our `main.py`:

```python
# in ender/main.py
import pluggy
from ender import hookspec, default_module


class Ender:
    def __init__(self):
        self.modules = []

    def work(self):
        for module in self.modules:
            module.work()


def main():
    ender = Ender()
    pm = get_plugin_manager()
    modules = pm.hook.ender_plugins_load_module()
    for module in modules:
        module.register(ender)

    ender.work()
    return 0


def get_plugin_manager():
    pm = pluggy.PluginManager("ender.plugins")
    pm.add_hookspecs(hookspec)
    pm.load_setuptools_entrypoints("ender.plugins")
    return pm


if __name__ == '__main__':
    exit(main())

```

We've now introduced a `get_plugin_manager` function, which will create a plugin manager object for the "ender.plugins" entry point. We'll tell it about our hook specification and then we'll load the entry point.

In our `main` we'll again initialize `Ender` and then we'll get the objects we receive from the `ender_plugins_load_module` hook, loop as before and register them with the `ender` instance.

Note that we haven't done a lot of work, just changed a couple of lines in some files. The `pyproject.toml` definition stays the same.

Let's go and prepare our plugin:

```python
from ender.hookimpl import hookimpl


class Mp3EncoderDecoder:
    def register(self, core):
        core.modules.append(self)

    def work(self):
        print(f"'{type(self).__name__}'", "Encoding MP3")


@hookimpl
def ender_plugins_load_module():
    return Mp3EncoderDecoder()
```

Now we're doing something a bit weird here. We're importing the `hookimpl` from the core package, although we haven't installed it. You'll probably get some errors because of this in your IDE, but just ignore them. This will work just fine.

Now replace the `register` function with the annotated implementation and you're done. Install the plugin:

```bash
# let's just remove the old package from our virtualenv
# from the ender/ project directory
poetry remove ender_mp3
poetry add ../ender_mp3
```

Now when you run your project you'll see the same message as before. Voila, we've moved our crude plugin implementation to this magnificent much more advanced version. The awesome part is that now we're able to define multiple hooks for our core package, use attributes when running the hooked functions and generally achieve much more, with not a lot of extra work.

## Let's go even further

Ok so now we're actually able to move some logic around, because pluggy is just that powerful. We can now easily split the encoders and the decoders and load them separately into different objects.

For that we'll rename our spec to `load_encoders` and add another one called `load_decoders`.

```python
# in ender/hookspec.py
import pluggy

hookspec = pluggy.HookspecMarker('ender.plugins')


@hookspec
def ender_plugins_load_encoders():
    pass


@hookspec
def ender_plugins_load_decoders():
    pass

```

Note that the hook specifications **MUST** be named after the marker. In our case the marker is `ender.plugins` which would translate to `ender_plugins`. This is important and the whole setup won't work if you don't follow this convention.

Let's quickly update our mp3 encoder/decoder and make sure it will work.

```python
# in ender_mp3/mp3_module.py
from ender.hookimpl import hookimpl


class Mp3Encoder:
    def work(self):
        print("mp3 encoder working...")


class Mp3Decoder:
    def work(self):
        print("mp3 decoder working...")


@hookimpl
def ender_plugins_load_encoders():
    return Mp3Encoder()


@hookimpl
def ender_plugins_load_decoders():
    return Mp3Decoder()

```

As you can see we've split our single module into two separate classes, each responsible for it's own task, e.g. encoding or decoding.

Let's update the default empty module as well:

```python
# in ender/empty.py
from ender.hookimpl import hookimpl


class DefaultEncoder:
    def work(self):
        print("Default encoder working")


class DefaultDecoder:
    def work(self):
        print("Default decoder working")


@hookimpl
def ender_plugins_load_encoders():
    return DefaultEncoder()


@hookimpl
def ender_plugins_load_decoders():
    return DefaultDecoder()

```

And finally let's update the main package:

```python
# in ender/main.py
import pluggy
from ender import hookspec, default_module


class Ender:
    def __init__(self, hook):
        self.hook = hook
        self.encoders = []
        self.decoders = []
        encoders = self.hook.ender_plugins_load_encoders()
        for encoder in encoders:
            self.encoders.append(encoder)

        decoders = self.hook.ender_plugins_load_decoders()
        for decoder in decoders:
            self.decoders.append(decoder)

    def work(self):
        for enc in self.encoders:
            enc.work()

        for dec in self.decoders:
            dec.work()


def main():
    pm = get_plugin_manager()
    ender = Ender(pm.hook)
    ender.work()
    return 0


def get_plugin_manager():
    pm = pluggy.PluginManager("ender.plugins")
    pm.add_hookspecs(hookspec)
    pm.load_setuptools_entrypoints("ender.plugins")
    return pm


if __name__ == '__main__':
    exit(main())

```

Let's start near the bottom. We've kept the `get_plugin_manager` function intact and we load it before we initialize the `Ender` class. We'll just add a hook argument and pass it to the instance.

In the `__init__` method we accept the hook and use it to load the `encoders` and `decoders` and save them to their respective property for later use.

We'll also update our work logic to use the encoders and decoders and we're done. Just reinstall the plugin and take a look:

```bash
poetry remove ender_mp3
poetry add ../ender_mp3
```

Run the application

```bash
$ poetry run python ender/main.py
mp3 encoder working...
Default encoder working
mp3 decoder working...
Default decoder working
```

You've probably noticed that the order of the modules is now switched. This is because the modules are loaded in LIFO order. If you want to restore the order you can just sort the encoders and decoders after loading them.

## Further thoughts

We've seen how to go from the simple idea of loading plugins through the awesome entry points feature, and we explored how to scale up to use something like `pluggy` to achieve more complex setups. With all that in place we'd be capable of performing a more complex setup when we load the encoders and decoders. We should expand our main application to allow the user to select if they want to encode or decode, add some basic implementations for certain audio and video types, overwrite them when loading plugins, or at least give the user the opportunity to do so. These are just some ideas. Feel free to experiment on this basis.