Dozens of Translation Directions or Millions of Shared Parameters? Comparing Two Types of Multilinguality in Modular Machine Translation (2023)

Abstract: There are several ways of implementing multilingual NLP systems but little consensus as to whether different approaches exhibit similar effects. Are the trends that we observe when adding more languages the same as those we observe when sharing more parameters? We focus on encoder representations drawn from modular multilingual machine translation systems in an English-centric scenario, and study their quality from multiple aspects: how adequate they are for machine translation, how independent of the source language they are, and what semantic information they convey. Adding translation directions in English-centric scenarios does not conclusively lead to an increase in translation quality. Shared layers increase performance on zero-shot translation pairs and lead to more language-independent representations, but these improvements do not systematically align with more semantically accurate representations, from a monolingual standpoint.

## introduction multilinguality, within the scope of neural nlp, can mean either ensuring that computations for different languages are homogeneous, or ensuring that models are trained with data coming from different languages. these two definitions are not as equivalent as they might appear: for instance, modular architectures, where some parameters are specific to a single language, can only be conceived as multilingual under the latter definition. both of these trends have been explored across multiple works. machine translation studies have authors listed alphabetically. corresponding author: timothee.mickus@helsinki.fi looked into sharing no parameters at all @xcite , sharing across linguistically informed groups @xcite , sharing only some components across all languages @xcite @xcite @xcite @xcite @xcite , and sharing the entire model @xcite . concerns about multilinguality have spearheaded research on how to make representations and systems more reliable for typologically and linguistically diverse data @xcite , the distinction between multilingual and monolingual representations @xcite , the specificity of massively-multilingual representations @xcite or the effects of having more diverse data @xcite @xcite @xcite . in this paper, we study whether these different implementations of multilinguality yield qualitatively different types of representations-in other words: are the effects of parameter sharing orthogonal to those of adding new languages? to broach this question, we make three simplifying assumptions. first, we only consider the task of multilingual machine translation-an exhaustive study of the impact of all multilingual nlp tasks is beyond the scope of this paper. moreover, massively multilingual language models are known to leverage parallel data to enhance semantic abstractions @xcite @xcite . second, we only consider parameter sharing in the last layers of the encoders: we focus on the intermediary representations acquired directly after the encoder and leave decoders for future study. as language selection tokens would compromise the language independence of the representations, this rules out fully shared decoders. third, we focus on an english-centric scenario: i.e., all translation directions seen during training contain english as a source or target language. while such an approach is not without issues @xcite , it makes it possible to select translation directions for zero-shot evaluations in a principled manner. furthermore, most multilingual translation datasets are highly skewed in any case and contain orders of magnitude more english examples (e.g., @xcite . we conduct our study by testing encoder outputs on three aspects: task fitness, language independence and semantic content. these features have been discussed in earlier literature: probing pretrained language models for semantic content in particular has proven very fecund (e.g., @xcite . as for machine translation, these studies are less numerous, although similar aspects have been investigated @xcite . for instance, @xcite study how the learned representations evolve in a multilingual scenario, whereas @xcite , @xcite or @xcite focus on the use of multilingual-mt as a signal for learning language. as we will show, studying representations under different angles is required in order to highlight the differences underpinning distinct implementations of multilinguality. foot_0 2 experimental setup ## conclusions we have studied whether different means of achieving multilinguality-sharing parameters and multiplying languages-bring about the same effects. what transpires from our experiments is that the two means are not equivalent: we generally observe higher performances and more reliable representations by setting the optimal number of shared parameters. crucially, this optimum depends on the criteria chosen to evaluate representations: machine translation quality (section 3.1), language independence (section 3.2) and semantic accuracy (section 3.3) all differed in that respect. these two approaches are not dichotomous: it is possible to both scale the number of languages and select optimal parameter sharing. what is possible may however not be practical. as guidance to nlp practitioners, we recommend spending effort on tuning the level of parameter sharing for the task at hand. sharing either too little (0-1 layers in our experiments) or too much (sharing the entire encoder) results in sub-optimal performance overall, but the optimal number of layers to share depends on the task. spending significant effort on acquiring data for additional language pairs may not yield improved representations past the initial stages of data collection (opus-03 in our experiments). changfeng zhu, heng yu, shanbo cheng, and weihua luo. 2020. language-aware interlingua for multilingual neural machine translation. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 1650-1655, online. association for computational linguistics.