A complete guide to i18n in Python
Tags: python, tutorial, i18n
Published: 2019-09-25 18:02:19

This is a start to finish guide showing how to do internationalization (i18n) for a Python application. When I added i18n to [handroll](http://handroll.github.io/), I struggled to find clear advice for supporting other languages. This is one opinionated view explaining how I got there.

Table of Contents:

* [Marking strings](#marking)
* [Extracting the master list](#extracting)
* [Getting translations for other languages](#pot2po)
* [Packaging it together](#packaging)
* [Testing out the package](#testing)
* [Questions?](#questions)

## Overview

To internationalize code, you have to treat user text strings in a certain way. All text strings must be wrapped with a special function call. This special function marks the strings as something that needs translation.  Once all the strings are marked, i18n tools can scan your code to make a master list of everything. With the master list available, translators can produce a list of translated strings for each desired language. The translated strings are added back to the Python code and it's all bundled up into a nice translated final product. Then you may want to [test that out](https://www.mattlayman.com/blog/2013/test-your-packaging/).

That's a lot of stuff, but I'll explain each part of that process in detail.  For the purposes of this guide, all of my example code will refer to the `handroll` package. If this is your Python code, replace `handroll` with the name of your top level Python package.

<a id='marking'></a>
## Marking strings

The special function that I mentioned in the overview looks like `_('Hello World')`. This function comes from the `gettext` module.  Python uses GNU gettext to do translation so let's look at the handroll code that creates the `_` function in `handroll/i18n.py`.

```python
import gettext
import os

localedir = os.path.join(os.path.abspath(os.path.dirname(__file__)), 'locale')
translate = gettext.translation('handroll', localedir, fallback=True)
_ = translate.gettext
```

This little chunk of code makes a translation object using a `locale` directory as the source of translated strings. Your `locale` directory doesn't exist yet, but it is where your application will look for translations at run time. From the translation object, I made an `_` alias for `translate.gettext`. This underscore is not particularly special, however, it is the conventional name of this function that is used by the Python community.

I hope there is enough here now to understand what will happen at run time. The important idea is that `'Hello World'` acts like a key.  When the code executes, that "key" will be used to look in the locale data to find a matching translation. The return value of `_` will be the translated string or the original if the translation is missing.

### `_` and `format` strings

One weirdness that you will quickly encounter if you use `format` for your strings is where to close the `_` parentheses. Here is an example to make it clear.

```python
_('Close the underscore function first. A {number}').format(number=42)
```

<a id='extracting'></a>
## Extracting the master list

With all strings marked with `_()`, it is time to extract them into a master list. Gettext refers to this master list as the `Portable Object Template` (`POT`) file. To generate the POT file, I turned to a tool called [Babel](http://babel.pocoo.org/). Babel is designed to help developers handle i18n issues in a wide variety of ways. It's cool with a lot of features, but for this guide, I'll focus on its gettext support.

Babel extends a `setup.py` file with new commands. (Python packaging is out of scope for this guide so if `setup.py` is completely foreign to you, please check out [Distributing Python Modules](https://docs.python.org/2/distutils/introduction.html) for more information.)

To generate a POT file, run `python setup.py extract_messages`. You will need some settings similar to:

```ini
[extract_messages]
input_dirs = handroll
output_file = handroll/locale/handroll.pot
```

<a id='pot2po'></a>
## Getting translations for other languages

Now that you have generated a POT file, you are ready to get translated versions of your strings. Remember that the *T* in POT is for *template*.  The translated files are called PO files, and translators use the POT file as a starting point to generate their PO file. Doing translation by manually creating the PO file is possible, but it is tedious.  At this point, I turned to a service called [Transifex](https://www.transifex.com/).

Transifex is focused on making translation easy. It's free for open source projects and really easy to use. I set up a project for handroll, configured Transifex to "watch" the POT file for changes, and translators can easily translate all the strings in the project. Transifex has an API that enabled me to pull PO files back into the repository whenever I needed. The script is a little too long to put in this post, but you can [look at it on GitHub](https://github.com/handroll/handroll/blob/master/transifex.py).

One important thing that the script does is store PO files in a specific structure in the `locale` directory. Gettext expects an order like `<language>/LC_MESSAGES/handroll.po`.

<a id='packaging'></a>
## Packaging it together

After the translated PO files are in place, it is time to package your translation data. The setup process includes extra data in `MANIFEST.in`. Add a line similar to the one below to ensure all `locale` data is put into the package archive.

```txt
recursive-include handroll/locale *
```

So far I've lied a little bit for clarity in explanation. Gettext does not use PO files directly for looking up translations. The truth is that the PO file must be compiled into a binary Machine Object (MO) file (for faster speed). Again, we can turn to Babel for help. Babel has a `compile_catalog` command that can compile PO into MO. You can run it with `python setup.py compile_catalog`.

It will need settings like:

```ini
[compile_catalog]
domain = handroll
directory = handroll/locale
```

Because translations will not work without MO files, I extended `setup.py` so that the `sdist` command will always run `compile_catalog`.

```python
from setuptools.command.sdist import sdist


class Sdist(sdist):
    """Custom ``sdist`` command to ensure that mo files are always created."""

    def run(self):
        self.run_command('compile_catalog')
        # sdist is an old style class so super cannot be used.
        sdist.run(self)
```

If you do this, don't forget to add `cmdclass={'sdist': Sdist}` and `setup_requires=['Babel']` in your `setup` call.

At this point, running `python setup.py sdist` should create a tarball with translations for your project. You're almost done!

<a id='testing'></a>
## Testing out the package

Testing translations is not easy. If you go the extra mile, you can know that people from all different cultures can enjoy your software.  But translation testing is not easy because you need tests that run every string in your code. For handroll, that meant I had to get to 100% test coverage to get those really strange corner cases.

The reason to test is that incorrectly translated strings can destroy your code. If you have a format string like `_('Hello {name}!').format(name='Johnny')` and a translator makes a mistake like `'¡Hola {nombre}!'`, then that code would break for Spanish users.

```pycon
>>> '¡Hola {nombre}!'.format(name='Johnny')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'nombre'
```

To test out these strings, you need two things: MO files and the `LANGUAGE` environment variable. Let's assume you use the [nose](https://nose.readthedocs.org/en/latest/) test runner.  To test out your Spanish support, you can run:

```console
$ LANGUAGE=es nosetests
..................................
----------------------------------------------------------------------
Ran 34 tests in 1.440s

OK
```

In whatever continuous integration system you use, run your unit tests through each language you support to give yourself some confidence that translations have not broken the software. The [handroll tox.ini](https://github.com/handroll/handroll/blob/master/tox.ini) provides an example to compare against.  [Tox](https://tox.readthedocs.org/en/latest/) is awesome for this kind of stuff.

<a id='questions'></a>
## Questions?

In this guide, I did my best to document all that I did to internationalize my project. I hope that all the concrete details help you see how to translate your own project.

This article first appeared on [mattlayman.com](https://www.mattlayman.com/blog/2015/i18n/).