MAMMOTH : Massively Multilingual Modular Open Translation @ H elsinki (2024)

Abstract: NLP in the age of monolithic large language models is approaching its limits in terms of size and information that can be handled. The trend goes to modularization, a necessary step into the direction of designing smaller sub-networks and components with specialized functionality. In this paper, we present the MAMMOTH toolkit: a framework designed for training massively multilingual modular machine translation systems at scale, initially derived from OpenNMT-py and then adapted to ensure efficient training across computation clusters.We showcase its efficiency across clusters of A100 and V100 NVIDIA GPUs, and discuss our design philosophy and plans for future information.The toolkit is publicly available online at https://github.com/Helsinki-NLP/mammoth.

## introduction the field of nlp has recently witnessed a hastened transition towards ever-larger monolithic neural networks, exposed to gargantuan amounts of data so as to properly fit their humongous number of parameters. there is also a growing consensus that this is not a sustainable trend: this approach falters whenever data is scarce, and leads to costs-both financial and ecological-that cannot be disregarded. the problems of scalability are especially prominent in the field of multilingual nlp. scaling a multilingual model to a high number of languages is prone to suffer from interference, also known as the curse of multilinguality, leading to degradation in per-language performance, mainly due to the limited model capacity @xcite . increasing the overall model size, on the other hand, hits the ceiling in terms of trainability limited by hardware, data and training algorithms. modularity is one approach attempting to answer the challenges of scalability. what is modularity? modularity can be viewed in two complementary ways: as sparsity or as conditional computation. in the former, modularity enforces a principled sparsity of the network, so as to allow a model to be large at training time, but small during inference. in the latter, a modular approach entails routing the information flow to a module in order to select specific model parameters to be used in specific circumstances. a canonical example is the use of languagespecific encoders and decoders for machine translation: for the translation direction going from a source language l s to a target language l t , one would use an encoder trained on on the data from the l s source language, and a decoder trained to handle any and all inference to the l t target language. note, that the encoder will typically have seen data with l s as source language but other languages than l t as target languages, and mutatis mutandis for the decoder. this dynamic selection of modules entails a principled sparse activation: in this translation scenario, any encoder or decoder linked to some third language l z would not contribute to the computation, as if was set at zero. this design can therefore lead to more efficient inference, since we can avoid computations for a large proportion of the parameters. the modularity also aids in the interpretability of parameters, as it is easy to determine which tasks a parameter contributes to. it also fosters the design of reusable neural network components: modules can in principle be combined to allow for zero-shot adaptations to novel tasks and situations @xcite . ## related work while there exist many open-source frameworks for training nmt systems that have been used for experiments in modular nmt, such as fairseq @xcite , to the best of our knowledge none of them is specifically targeted at modularity. mam-moth is the first open-source toolkit to jointly address the issues of scalability, multilinguality and modularity. the most relevant point of comparison would be the adapterhub of @xcite : it extends the transformers library @xcite an nlp-centric model sharing platform and training library) so as to enable training of adapter modules for pre-trained state-of-the-art systems. we base our mammoth modular toolkit on the widely used opennmt-py @xcite , a customizable library for training nmt and nlg models with a focus on efficiency based on pytorch: the more thorough documentation and more systematic organization of opennmt-py proved a better starting point for mammoth than the fairseq or transformers libraries. another motivation behind mammoth is the lack of a standard for the different architectures of existing works on modular systems. modularity in multilingual nmt has been addressed in a wide range of custom implementations. one common approach is to train language-specific encoders and decoders. @xcite introduce the attention bridge, an intermediate cross-lingual shared layer in between the encoder and decoder. similarly, @xcite train language-specific encoders-decoders without sharing any parameters at all. @xcite also exploit this method and explore different sharing strategies for the decoder while keeping the decoder language or language-group-specific. @xcite experiment with massive multilingual mt and propose a plug-and-play approach with detachable modules per language. other methods investigate the use of language-specific transformer layers @xcite , by keeping some layers source or target language-specific in the encoder. mammoth supports all of the above, and provides a new unifying standard framework for training and testing modular nmt at scale. ## toolkit design we now turn to a description of our toolkit. this section first details the requirements for the toolkit section 3.1, before moving to the design philosophy we adopted in response to the practical needs outlined in section 3.2. finally, we list and describe all major components of our toolkit in section 3.3. ## performances we utilize the europarl dataset (koehn, 2005)a multilingual resource extracted from european parliament proceedings containing texts in 21 european languages -for model training to showcase the efficiency of our toolkit. we report performances on the europarl dataset across various parametersharing schemes and computing clusters. modeling. we use a sentencepiece model trained on opus tatoeba challenge data with 64k vocabulary size. 2 we adopt three sharing schemes: 1) a language-specific one that uses a balanced architecture with a 6-layer transformer encoder and a 6-layer decoder for each language, 2) a partially shared one that has eight layers of encoders (4 shared and language-specific layers) and four layers of decoders, and 3) a fully shared one with 9layer encoder and 4-layer decoder for all languages. for the transformer network, we enable position encoding and use a dimension of 512 and 8 attention heads. the feed-forward dimension within the transformer is 2048. setup. we utilize two types of gpus: nvidia v100 and a100. we run the toolkit with python 3.9.16 and pytorch 2.1.0. we use a maximum sequence length for source and target languages of 200, and a batch size of 4096 tokens. the detailed setup guide for the experiment is available in our documentation, 3 including data processing, configurations, and launching scripts. benchmarking results. we studied the performance of mammoth on csc's nvidia v100 and a100 clusters, where each node contains four nvidia v100/a100 gpus connected via nvlink and nodes are interconnected via infiniband. scaling benchmarks were undertaken such that the number of tasks increases proportional to the number of allocated gpus as we are interested in the ability to scale to larger problems with similar load per gpu. in practice, we train one task on each available gpu for this benchmark. for benchmarking purposes, this task is defined over synthetic data derived from the europarl dataset @xcite : we concatenate bi-texts for all available translation directions, and then randomly split it into 20 subcorpora. while individual data-points remain coherent, this shuffling process allows us to sidestep concerns about variation in linguistic factors, such as sentence length, that were found to be a concern in an earlier iteration of this benchmark. moreover, we studied task scaling with three different approaches. firstly, an approach where all source languages use language-specific encoders and decoders, denoted by "independent." secondly, an approach where all source languages use the same shared encoder but decoders are target language specific, denoted by "shared." finally, an approach denoted as "partially shared", where the encoder begins with a stack of language-specific layers, followed by a stack of shared encoder layers, finally passing information to language-specific decoders, similar to the setup discussed in figure 2b . figure 3 shows task scaling benchmarks from a single node up to 5 nodes using the synthetic dataset. we achieve nearly ideal scaling in all of the scenarios: drops in performance compared to our ideal reference are limited to 5% at most. to give more context to the benchmarks, we measured the memory footprint and utilizations independently on each gpu. the mean utilization percentage with the five nodes run with languagespecific encoders and decoders was 85%, and the device memory footprint was 7.7g out of 32g. furthermore, we also did performance profiling using nvidia nsight systems tool to better understand the communication, using the 2-node setup. all communication in the code is outsourced to the nvidia collective communications library (nccl) and at present, it appears that approximately 28 % of the gpu-kernel execution time is spent on nccl allreduce. further communication studies and optimizations with different architectures and setups remain subject to future work. nevertheless, as we are able to achieve linear scaling beyond a single node, it suggests that the communication overheads are alleviated at scale (relative to tokens processed per second). environmental costs. we run the benchmarking experiments on csc's carbon-neutral data center powered by hydropower. we measure our carbon footprint using the direct eq. co2 of 0kg/kwh and indirect of 0.024kg/kwh as the carbon efficiency. 4 each benchmarking experiment runs for around one hour. our total carbon footprint is 0.11 kg eq. co2. the estimated carbon footprint for training a model over 24 hours is 0.14 kg eq. co2. ## conclusions and next developments in this paper, we have introduced mammoth, a toolkit for training modular encoder-decoder neural networks at scale. the toolkit is publicly available under a cc-by license, and we warmly welcome the help of new developers and researchers wishing to extend it. further planned development of the mam-moth toolkit will specifically focus on the following elements: