stopes - Modular Machine Translation Pipelines (2022)

Abstract: Neural machine translation, as other natural language deep learning applications, is hungry for data. As research evolves, the data pipelines supporting that research evolve too, oftentimes re-implementing the same core components. Despite the potential of modular codebases, researchers have but little time to put code structure and reusability first. Unfortunately, this makes it very hard to publish clean, reproducible code to benefit a wider audience. In this paper, we motivate and describe stopes , a framework that addresses these issues while empowering scalability and versatility for research use cases. This library was a key enabler of the No Language Left Behind project, establishing new state of the art performance for a multilingual machine translation model covering 200 languages. stopes and the pipelines described are released under the MIT license at https://github.com/facebookresearch/stopes .

## introduction machine translation (mt) aims at removing language barriers in our connected society. the current trend in the mt research field is moving towards using deep machine learning models training either many bi-lingual models, translating between a single pair of languoids, or multi-lingual models that handle many languoids at once. training data usually comes from open aligned data sources such as @xcite , @xcite or raw web corpora like commoncrawl (cc). recently, initiatives like @xcite and the no language left behind project @xcite strive to extend the scope of supported languoids by training on large scale datasets, reaching over 18 and 25 billion sentence pairs respectively. the end-to-end process of developing and iterating on a neural machine translation model in-* lead library maintainers. in research use cases, the main focus is on getting results fast. we have observed that the path of predilection is to build ad-hoc solutions to solve the problem directly at hand, often adapting older scripts or copying snippets of code that colleagues have found to work. this enables quick iteration on research ideas but it causes a lot of problems on the long term: we present a new framework, stopes , that was developed to solve some of the problems discussed in the scope of the no language left behind (nllb) @xcite machine translation project to process billions of sentences in over 200 languages. the goal of the framework is to ensure a good separation between the hardware setup and the core logic of the data processing by proposing a clean api for sharing commonly used processing steps, while enforcing consistent and shareable configurations of experiments. stopes can scale to a research project like nllb, but is built to be versatile and can be applied to other research use cases, https://facebookresearch.github. io/stopes/docs/quickstart provides an example of running this on a smaller dataset. in section 3, we introduce the design of this python library, with concrete examples in section 4. section 5 discusses applications within the nllb where stopes is used. ## related work while large scale data processing architectures already exist, they are often optimized for production use cases. spark (spark), ray (ray) or beam @xcite are a few leading examples. these frameworks have a steep learning curve and do not always map easily to research clusters' setup or researchers' work habits. they can also prove very challenging to use with nascent research ideas and tools, whose codebases are not yet stable or production-ready. bitextor @xcite provides a bitext mining pipeline built in python, but it lacks modularity and requires learning the complex apis of snakemake. with stopes , we are aiming for a "minimal api surface" without sacrificing features, providing a clean yet versatile api that can be used as if writing standard python scripts (see section refsec:example). this simple api has its drawbacks, but it makes it easier to pick up for researchers than complex graph planning systems like luigi @xcite and airflow (airflow), these industry standards are better suited for production pipelines that do not change often and are maintainted by production teams. spacy @xcite provides research oriented nlp pipelines, but is less flexible than stopes as our framework is more geared towards describing sometimes pipelines in pure python. ## framework the general architecture of stopes is geared towards pipelines that can be run as separate, sometimes interdependent, jobs on a cluster or in multiprocessing. the idea being that a pipeline can be divided in a set of separate steps that can be expressed as processing units. jobs can be sent to a job scheduler, like slurm @xcite , which is widespread on academic compute clusters or fblearner @xcite , which meta uses for distributed machine learning pipelines; or run locally on a single computer depending on the data scale. the idea behind the stopes framework is to make it easy to build reproducible pipelines. this is done though modules, a module is just a python class with a run function that executes something. a module can then be scheduled with the stopes ' launcher, this will decide where the code gets executed (locally or on a cluster) and then wait for the results to be ready. ## example code figure 1 shows a sample usage of the stopes library foot_1 to build a faiss index foot_2 . faiss @xcite is a tool that can be used to build large scale indexes and perform nearest neighbor searches on them; faiss has become a keystone to machine translation research as it allows for efficient alignment of multilingual text when using language-agnostics embeddings like @xcite or @xcite (e.g. @xcite or section 5.2). it takes tensors as input, but first the index has to be trained, usually on a sample of the data we want to store in the index. line 5: we initialize the launcher to be able to schedule modules for execution. the launcher is managed by a configuration, so we can easily change where the code is executed (slurm cluster, aws, locally) and other constraints of the execution. every call to 'launcher.schedule' will be managed by the designated 'launcher', sent to the cluster once, or in multiple jobs if necessary, or just retrieved from the cache if the config permits. line 7: we initialize an encoding module, which takes in raw text and embeds it. stopes provides code to embed text with laser2 and laser3 as well as with hugging-face sentence-transformers @xcite . to keep the code short, we only show the pipeline glue and not each module implementation. line 12: we create a sample from the embedded text files. here, update(config.sample, @xmath0. this pattern can also be seen in the other steps of this pipeline where each step is connected to the previous through intermediate output results. ## applications the stopes library was used to build the major data processing pipelines that are used to build the nllb large multilingual translation models as well as its distilled version @xcite . these pipelines were battle tested on petabytes of data and are open-sourced at https://github. com/facebookresearch/stopes . in this section we discuss some of the pipelines and show how they can reuse the same modules. source code can be found in the above github repository. ## conclusion the stopes framework provides a clean api to describe research pipelines for machine translation. we have shown that this is useful for developing large scale machine translation datasets and models for the no language left behind project @xcite . we believe that this framework and its reference implementations of common steps in nlp pipelines is versatile and can be used to help researchers in the field. the stopes framework documentation and sources can be found under the mit license at https://facebookresearch. github.io/stopes/ and has been tested to not require a complex cluster setup. we therefore hope that it will help other researchers focus on their research goals, and avoid time-consuming technical details not unique to their specific task.