Latest Development in the F o T ran Project – Scaling Up Language Coverage in Neural Machine Translation Using Distributed Training with Language-Specific Components (2022)

Abstract: We describe the enhancement of a multilingual NMT toolkit developed as part of the FoTran project. We devise our modular attention-bridge model, which connects language-specific components through a shared network layer. The system now supports distributed training over many nodes and GPUs in order to substantially scale up the number of languages that can be included in a modern neural translation architecture. The model enables the study of emerging language-agnostic representations and also provides a modular toolkit for efficient machine translation.

## introduction the fotran project aims at developing models for natural language understanding trained on implicit information given by large collections of human translations. @xcite it is funded by a european research council consolidation grant, running from 2018 to 2023 within the language technology research group at the university of helsinki under coordination of prof. jörg tiedemann. cross-lingual grounding, useful for resolving ambiguities through translation, is a guiding principle of the project. consequently, we developed a model for multilingual nmt specifically designed to obtain meaning representations injected with multilingual data (vázquez et al., 2020). former project results pointed towards the improvement of both the translation quality and the abstractions acquired by our model when including more languages (vázquez et al., 2019; raganato et al., 2019). due to the use of language-specific modules, the overall model architecture grows when languages and translation directions are added. doing this on a single device does not scale beyond the memory limits of that specific computing node. this is a limitation for testing the project hypothesis that training on increasing amounts of linguistically diverse data improves the abstractions found by the model -eventually leading to languageindependent meaning representations useful for machine translation and tasks that require semantic reasoning and inference. here, we propose strategies to address those issues: (1) distribute modules across several processing units, (2) efficiently train the network over many translation directions, and (3) reuse the trained modules without having to load the entire network. these together deliver a cost-effective multilingual nmt system that can further be used for extracting multilingual meaning representations. @xcite despite the high computational resources needed to scale up the number of translation directions when training the model, its modularity allows to reuse the trained components on relatively small processing units, making multilingual models more affordable and increasing their availability. ## methodology the implementation follows an encoder-decoder architecture, incorporating language specific encoders and decoders to enable multilingual training. they are connected via a shared innerattention layer that summarizes the encoder information in a fixed-size vector representation (vázquez et al., 2019), which in turn can be applied to downstream tasks (vázquez et al., 2020). we refer to all encoders, decoders and to the shared layer as modules. encoders and decoders are language-specific as they only see training data from specific translation directions. we distribute the model across multiple processing units by loading, in each device, encoders and decoders for a subset of the training translation directions. the inner-attention layer is shared across all processing units. all modules that are present in more than one device are initialized with the same weights, and gradients for these parameters are communicated across devices to ensure that they remain synchronous. in general, allocating language pairs with common source/target languages on the same device decreases both the total memory footprint of the model and the amount of communication needed to keep the modules synced. formally, we define the partition of the training language pairs l @xmath0, t ij ) over n units as @xmath1, l 12 , . . .), . . . , (l n 1 , l n 2 , . . .)}, where the first subscripts indicate to which device each pair is assigned, and the second is an incremental index over all pairs assigned to the same device. whenever x @xmath2, with @xmath3, t } representing either a source or a target language, we need to load a copy of the same module in devices i and k. this will also impact the training time as it requires communicating gradients across devices to keep modules synced. however, when dealing with a high number of translation directions (and a limited number of source and target languages) it becomes impossible to avoid this condition: gathering together language pairs based on the source (target) language could result in a scattered configuration based on target (source) languages. we address these problems using two strategies. first, we solve an allocation problem to minimize inter-device communication. since in most cases the problem has no feasible exact solutions, we approximate a solution using the hungarian algorithm over a cost matrix that makes it cheaper to assign the same language to a given gpu. second, we propose to schedule the gradient updates to minimize the waiting time when inter-device communication happens. at each training step the ith device starts performing a forward pass over a training batch for the language pair l i1 and accumulates gradients over all the language pairs l ij , where j runs from one to the number of language pairs assigned to the processing unit. afterwards, gradients of modules that are present in multiple processing units are averaged across devices. module weights are then updated according to the computed gradients. we ensure that all copies of all modules have nonzero gradients that can be communicated, preventing the training loop from hanging. we also save the modules individually to be loaded and used independently in an efficient way. this makes the system more portable and user-friendly for further fine-tuning, generating translations, and experimentation with multilingual sentence-representations. ## final remarks fotran aims at testing and analyzing representations obtained from massively multilingual nmt systems, and we devised a model architecture that is optimized for training large models (with a sufficiently large high-performance cluster). after training, it can also easily be used in non-resourceintensive settings due to its modular design. next, we intend to systematically explore the effect of increasing language diversity and how the abstraction capabilities of the inner representations are affected in different settings.