Modeling Task-Aware MIMO Cardinality for Efficient Multilingual Neural Machine Translation (2021)

Abstract: Neural machine translation has achieved great success in bilingual settings, as well as in multilingual settings. With the increase of the number of languages, multilingual systems tend to underperform their bilingual counterparts. Model capacity has been found crucial for massively multilingual NMT to support language pairs with varying typological characteristics. Previous work increases the modeling capacity by deepening or widening the Transformer. However, modeling cardinality based on aggregating a set of transformations with the same topology has been proven more effective than going deeper or wider when increasing capacity. In this paper, we propose to efficiently increase the capacity for multilingual NMT by increasing the cardinality. Unlike previous work which feeds the same input to several transformations and merges their outputs into one, we present a Multi-Input-Multi-Output (MIMO) architecture that allows each transformation of the block to have its own input. We also present a task-aware attention mechanism to learn to selectively utilize individual transformations from a set of transformations for different translation directions. Our model surpasses previous work and establishes a new state-of-the-art on the large scale OPUS-100 corpus while being 1.31 times as fast.

## introduction multilingual translation between multiple language pairs with a single model @xcite @xcite has some advantages compared to bilingual systems @xcite @xcite , e.g., easy deployment, enabling transfer learning across languages and zero-shot translation. despite their advantages, multilingual systems tend to underperform their bilingual counterparts as the number of languages increases @xcite . this is due to the fact that multilingual nmt must distribute its modeling capacity over different translation directions. @xcite show that the model capacity is crucial for massively multilingual nmt to support language pairs with varying typological characteristics, and propose to increase the modeling capacity by deepening the transformer. however, compared to going deeper or wider, modeling cardinality based on aggregating a set of transformations with the same topology has been proven more effective when we increase the model capacity @xcite . in this paper, we efficiently increase the capacity of the multilingual nmt model by increasing the cardinality, i.e. stacking sub-layers that aggregate a set of transformations with the same topology. our main contributions are as follows: trans 1 trans 2 trans n … input output + trans 1 trans 2 trans n … input output 1 output n output 2 … trans 1 trans 2 trans n … trans 1 trans 2 trans n … trans 1 trans 2 trans n … output (a) (b) (c) (d) each "trans" is a sub-layer that runs in the order of: @xmath0, where the transforming unit can be either multi-head attention or ffn, as depicted in figure 2 . we aggregate the final output of layer normalization of each "trans" in the block into the input fed to the next block in different ways (i.e., (a)-(d)). zhang et al. (2020) overcome the capacity bottleneck of multilingual nmt via deepening nmt architectures. @xcite present a highly modularized network architecture for image classification. the network is constructed by repeating a building block that aggregates a set of transformations with the same topology. for a given input i, the block adopts n networks of the same topology trans to process i and merges their outputs into the final output o of the layer: this design strategy exposes a new dimension, namely "cardinality" (the size of the set of transformations), as an essential factor in addition to the dimensions of depth and width. @xcite empirically show that increasing cardinality is more effective than going deeper or wider when we increase the capacity to improve classification accuracy. yan et al. (2020) present a multi-unit transformer to efficiently improve the translation performance by increasing cardinality instead of depth. however, their work implements stacks of @xmath1, is developed for bilingual sentence-level transformation, and requires the additional design of a biasing module and sequential dependency that guide and encourage complementariness among different units. by contrast, our work aims at efficiently increasing the capacity for multilingual translation, proposes the mimo transformation (figure 1 (c )) between stacked blocks, and naturally uses the translation task in attention form to guide individual transformations of the set to learn different representations for different translation directions. ## related work multilingual nmt includes one-to-many @xcite , many-to-many @xcite and zero-shot @xcite scenarios. a simple solution is to insert a target language token at the beginning of the input sentence @xcite . multilingual nmt has to handle different languages in one joint representation space, neglecting their linguistic diversity, especially for massively multilingual nmt @xcite @xcite . most studies focus on how to mitigate this representation bottleneck @xcite @xcite @xcite @xcite @xcite @xcite . there are also studies on the trade-off between shared and language-specific parameters @xcite , on the training of multilingual nmt @xcite @xcite , and on analyzing translations from multilingual nmt @xcite or the trained model @xcite . transferring a pre-trained multilingual nmt model can help improve the performance of downstream language pairs @xcite , especially for low-resource scenarios @xcite . multilingual data also has been proven useful for unsupervised nmt @xcite . ## conclusion we propose to efficiently increase the capacity for multilingual nmt by increasing the cardinality. we present a mimo architecture that allows each transformation of the block to have its own input. we also present a task-aware attention mechanism to learn to selectively utilize individual transformations from a set of transformations for different translation directions. our model surpasses previous work and establishes a new state-of-the-art on the large scale opus-100 corpus while being 1.31 times as fast.