Dependency Injection with Import Hooks in Python 3
Tags: python, metaprogramming, dependencyinjection
Published: 2018-12-26 14:18:53

---
title: Dependency Injection with Import Hooks in Python 3
published: true
description: Learn how to leverage `sys.meta_path` to write an easy-to-use dependency injection framework.
tags: python, metaprogramming, dependency injection
---

`sys.meta_path` is one of the best-kept secrets in the Python standard library, despite being squarely located in a module every Python programmer is familiar with. Some might say the secrecy is well-warranted, though; it's not usually a good idea to go messing around with tools and functions that may subvert programmer expectations the way this oddly-named list potentially can. But isn't "do not" better than "know not?" So before a quick primer on dependency injection, let's examine some of the ins and outs of `sys.meta_path`.

Whenever you use an `import` statement, say, `import os`, that's functionally equivalent to the following code:

```python
os = __import__("os")
```

It's a mild oversimplification, since stuff like submodules would require a little bit more work (as we'll have to deal with later, actually), but the main point to be made here is that the `__import__` built-in does the heavy lifting under the hood for module imports in Python, and now we know that it exists, we can examine its control flow. First, it checks `sys.modules`, a dict containing already-loaded modules, just in case we've already imported the requested module before. Next, it checks `sys.path` - a list containing filesystem paths for Python to search through for potential modules; you probably know about this list already. However, if the `sys.meta_path` list is not empty, than *before* looking through `sys.path`, every meta importer hook in `sys.meta_path` will be queried before the filesystem is checked, giving us an easy way to intercept module imports and do all sorts of freakishly wonderful things, like dependency injection.

So what is dependency injection? It's roughly an applied version of the [dependency inversion principle](https://en.wikipedia.org/wiki/Dependency_inversion_principle), often for module-sized stuff. Say your application has several different front-ends, maybe targeting different platforms or UI toolkits. As long as you have a common interface, dependency injection would allow you to do something like:

```python
user_info = info_getter.get_user_info()
frontend = dependency_injector.get("frontend")
frontend.display_user_info(user_info)
```

"So what? I can do that already with a Facade or something, maybe a config switch," I can hear you say. For very simple use cases, that may actually be better - simple is better than complex, remember? However, with dependency injection, we can easily provide module-level reuse, as well as significantly cut down on boilerplate. What if I told you, by the end of this article, you could write something like this:

```python
import myapp.dependency.frontend as frontend
popup = frontend.Popup(popup_style=frontend.ERROR)
popup.display()
```

One import that works regardless of whether we're using GTK+, Qt, ncurses, or whatever your program needs to function. You might be thinking some very dark thoughts at this point. It's true, all sorts of dark magic could be performed depending on the details of our import hooks. But readability counts, so it's important that we not stray too far off the beaten path. So let's set some requirements for our dependency injection framework before writing our implementation:

* All dependency-injected import paths begin with `myapp.virtual`
* Provided dependencies are known beforehand and registered with a `provide` method

All the magic - and there's not all that much - relies almost entirely on creating a meta path finder and adding it to `sys.meta_path`. A meta path finder is a very simple object with a single public method, `find_spec`:

```python
import importlib.abc
import importlib.machinery

class DependencyInjectorFinder(importlib.abc.MetaPathFinder):
    def __init__(self, loader):
        # we'll write the loader in a minute, hang tight
        self._loader = loader
    def find_spec(self, fullname, path, target=None):
        """Attempt to locate the requested module
        fullname is the fully-qualified name of the module,
        path is set to __path__ for sub-modules/packages, or None otherwise.
        target can be a module object, but is unused in this example.
        """
        if self._loader.provides(fullname):
            return self._gen_spec(fullname)
    def _gen_spec(self, fullname):
        spec = importlib.machinery.ModuleSpec(fullname, self._loader)
        return spec
# we'll also add it to sys.meta_path later
```

If a meta path finder provides the requested module, then it should return an instance of the `importlib.machinery.ModuleSpec` class, which is a fairly simple affair with a small handful of attributes that lets Python's import machinery know what it needs to know to take the next steps in importing the module the user requested. For our purposes, we're interested in two attributes (the only required ones): `ModuleSpec.name`, which is the name of the requested module, and `ModuleSpec.loader`, which is the loader object that Python should use to actually load the module - you'll notice the `self._loader` lines above that reference a loader object, as well. A loader object is a very simple class with two required methods in modern Python (3.4 onwards): `create_module`, which takes a `ModuleSpec` as its sole argument and returns an object that Python will consider to be the new module, and `exec_module`, which takes the new module as its sole argument and executes it. So a no-op, barebones loader looks something like this:

```python
class Loader(importlib.abc.Loader):
    def create_module(self, spec):
        raise NotImplementedError
    def exec_module(self, module):
        raise NotImplementedError
```

In addition to those two methods, we should implement the `provide` method we talked about earlier to signal that a certain dependency is provided, in addition to the `provides` method that our finder referenced earlier, that indicates whether the requested module is part of our dependency injection framework. Here's the implementation:

```python
import importlib.abc
import sys
import types


class DependencyInjectorLoader(importlib.abc.Loader):
    _COMMON_PREFIX = "myapp.virtual."
    def __init__(self):
        self._services = {}
        # create a dummy module to return when Python attempts to import
        # myapp and myapp.virtual, the :-1 removes the last "." for
        # aesthetic reasons :) 
        self._dummy_module = types.ModuleType(self._COMMON_PREFIX[:-1])
        # set __path__ so Python believes our dummy module is a package
        # this is important, since otherwise Python will believe our
        # dummy module can have no submodules
        self._dummy_module.__path__ = []
    def provide(self, service_name, module):
        """Register a service as provided via the given module
        A service is any Python object in this context - an imported module,
        a class, etc."""
        self._services[service_name] = module
    def provides(self, fullname):
        if self._truncate_name(fullname) in self._services:
            return True
        else:
            # this checks if we should return the dummy module,
            # since this evaluates to True when importing myapp and
            # myapp.virtual
            return self._COMMON_PREFIX.startswith(fullname)
    def create_module(self, spec):
        """Create the given module from the supplied module spec
        Under the hood, this module returns a service or a dummy module,
        depending on whether Python is still importing one of the names listed
        in _COMMON_PREFIX.
        """
        service_name = self._truncate_name(spec.name)
        if service_name not in self._services:
            # return our dummy module since at this point we're loading
            # *something* along the lines of "myapp.virtual" that's not
            # a service
            return self._dummy_module
        module = self._services[service_name]
        return module
    def exec_module(self, module):
        """Execute the given module in its own namespace
        This method is required to be present by importlib.abc.Loader,
        but since we know our module object is already fully-formed,
        this method merely no-ops.
        """
        pass
    def _truncate_name(self, fullname):
        """Strip off _COMMON_PREFIX from the given module name
        Convenience method when checking if a service is provided.
        """
        return fullname[len(self._COMMON_PREFIX):]
```

There's seemingly a lot of code here at first glance (even though most of it is comments!), so let's walk through this class' methods. Up first are the `provide` and `provides` methods that we've talked about earlier; there's not a whole lot of magic present in either one. `create_module` returns a so-called dummy module if we're trying to import either `myapp` or `myapp.virtual`, and there's a good reason for that. Say we have the following line of code:

```python
import myapp.virtual.frontend
```

Under the hood, this generates three distinct searches in Python's import machinery: One for `myapp`, another for `myapp.virtual`, and lastly one for `myapp.virtual.frontend`. Since obviously `myapp` and `myapp.virtual` don't actually exist anywhere on the system, but Python will complain if they aren't loaded, we claim we provide both - note how `provides` will return `True` if queried for both `myapp` and `myapp.virtual` and return an empty dummy module to placate Python's import machinery. Only when we encounter `myapp.virtual.frontend` do we check against the list of provided dependency-injection services.

At this point, we've actually written all the plumbing we need to, so let's wrap up and see how we'd use these classes in practice:

```python
import sys


class DependencyInjector:
    """
    Convenience wrapper for DependencyInjectorLoader and DependencyInjectorFinder.
    """
    def __init__(self):
        self._loader = DependencyInjectorLoader()
        self._finder = DependencyInjectorFinder(self._loader)
    def install(self):
        sys.meta_path.append(self._finder)
    def provide(self, service_name, module):
        self._loader.provide(service_name, module)


class FrontendModule:
    class Popup:
        def __init__(self, message):
            self._message = message
        def display(self):
            print("Popup:", self._message)


if __name__ == "__main__":
    injector = DependencyInjector()
    # we could install() and then provide() if we wanted, order isn't
    # important for the below two calls
    injector.provide("frontend", FrontendModule())
    injector.install()
    # note that these last three lines could exist in any other source file,
    # as long as injector.install() was called somewhere first
    import myapp.virtual.frontend as frontend
    popup = frontend.Popup("Hello World!")
    popup.display()
```

At this point, it'd be worth it to take a step back and examine at a high level what is going on behind the scenes:

1. The call to `injector.install` appends our `DependencyInjectorFinder` onto the `sys.meta_path` list, so Python will use it for future imports.
2. The line `import myapp.virtual.frontend as frontend` triggers three module searches, as mentioned earlier - one for `myapp`, then `myapp.virtual`, then `myapp.virtual.frontend`:
  * Python combs through `sys.meta_path`, looking for meta path finders. If your system Python is anything like mine, our `DependencyInjectorFinder` will be the only one not defined by `_frozen_importlib`, which is a part of Python itself.
  * For each found meta path finder, Python queries the finder's `find_spec` method, seeing if the finder provides the given module.
  * Obviously `myapp.virtual.frontend` doesn't exist on the filesystem, so it falls to our meta path finder to handle it. In all three cases, we return a `ModuleSpec` instance with `name` set to the same name that Python asked if we could find, and `loader` set to our custom `DependencyInjectorLoader`.
  * Next, Python will call the `create_module` method on our loader for the `ModuleSpec` in question. For `myapp` and `myapp.virtual`, our loader recognizes that these are dummy modules and returns the same dummy module for both (tricking Python into believing the module was loaded), but returns the instance of `FrontendModule` we gave it with `injector.provide()` upon being asked to load `myapp.virtual.frontend`. Python allows any valid object to function as a module, so a plain old class is perfectly fine to return.
  * Python will finally call the `exec_module` method on our loader, passing it the object we returned from `create_module`. Python requires the `exec_module` method to be present but doesn't really care about its behavior; normally, the method would execute the module's code in a new namespace, but since we already have fully-formed modules ready to go - either our pre-made dummy module or our `FrontendModule` instance - we simply do nothing inside `exec_module`.
  * Wash, rinse, and repeat for each successive module search.
3. After all that, `frontend.Popup` is functionally the same thing as `FrontendModule.Popup`, and the rest, as they say, is history.

As I mentioned earlier in the article, it's important not to abuse language features like this - with great power comes great responsibility, after all - and this especially rings true when it comes to metaprogramming. Still, there's a time and place for everything, import hooks included; but please use them responsibly.

Finally, the complete code example is up as a [gist on GitHub](https://gist.github.com/DangerOnTheRanger/027fc0af3bd010e8265e552c5b3ab4f1) if you'd like something you can download and pore over. Happy hooking!