Multilingual Machine Translation with Hyper-Adapters (2022)

Abstract: Multilingual machine translation suffers from negative interference across languages. A common solution is to relax parameter sharing with language-specific modules like adapters. However, adapters of related languages are unable to transfer information, and their total number of parameters becomes prohibitively expensive as the number of languages grows. In this work, we overcome these drawbacks using hyper-adapters – hyper-networks that generate adapters from language and layer embeddings. While past work had poor results when scaling hyper-networks, we propose a rescaling fix that significantly improves convergence and enables training larger hyper-networks. We find that hyper-adapters are more parameter efficient than regular adapters, reaching the same performance with up to 12 times less parameters. When using the same number of parameters and FLOPS, our approach consistently outperforms regular adapters. Also, hyper-adapters converge faster than alternative approaches and scale better than regular dense networks. Our analysis shows that hyper-adapters learn to encode language relatedness, enabling positive transfer across languages.

## introduction multilingual neural machine translation (mnmt) models @xcite reduce operational costs and scale to a large number of language pairs @xcite by using a shared representation space. this approach benefits low-resource languages through positive transfer from related languages, but introduces a transferinterference trade-off @xcite as the number of languages grows, the performance in more resource-rich languages starts to drop. prior work shows that constrained model capacity prevents models from representing all languages ⋆correspondence to c.baziotis@ed.ac.uk or shru@fb.com †work done during an internship at meta ai equally well @xcite . while naively increasing capacity is certain to improve performance @xcite , it comes with large computational costs. a common remedy for the capacity bottleneck is to relax the information sharing with language-specific parameters @xcite @xcite @xcite . adapter modules @xcite have been successfully employed in various natural language processing tasks to address similar capacity-related issues @xcite . in mnmt, adapters have been used to adapt (via finetuning) pretrained generic models to specific language-pairs or domains @xcite , to improve zero-shot performance @xcite , or to reduce interference @xcite . however, using regular language(-pair) adapters has certain limitations. first, they can be very parameter-inefficient. while each adapter layer might be small, the total number of layers is proportional to the number of languages. this quickly becomes very costly, in particular in massively multilingual settings. in addition, there is no information sharing between the adapters of related languages. for instance, an adapter for nepali cannot benefit from the more abundant hindi data, which prevents positive transfer between the two languages. in this work, we train mnmt models with extra language-specific modules generated by a hyper-network @xcite . we use adapters for the language-specific modules, dubbed hyperadapters, but it is trivial to replace them with any other architecture. hyper-adapters (figure 1 ) are a function of jointly trained language and layer embeddings. this approach naturally encodes language relatedness and enables knowledge transfer between related languages. it also substantially improves parameter efficiency, as the number of hyper-adapter parameters is invariant to the number of languages. we also address optimization obstacles @xcite overlooked by prior work @xcite , and propose a rescaling fix that improves convergence and enables us to successfully scale to large hyper-networks. we present experiments on a large multilingual translation benchmark. unlike prior work @xcite that finetunes adapters for language-specific adaptation, we train regular-and hyper-adapters jointly with the main network. we show that with the same parameter budget and flops, hyper-adapters are consistently better than other regular adapter variants. we also match the performance of regular adapters with hyper-adapters up to 12 times smaller. hyper-adapters, also converge faster than other approaches and improve scalability, as small dense networks with hyper-adapters yield similar results to larger regular dense networks. our analysis reveals that hyper-adapters do indeed exploit language similarity, unlike regular adapters. by comparing models on benchmarks with artificially constructed properties, we find that the gains of hyperadapters grow as the redundancy (e.g., language similarities) in the training data increases. our main contributions are: we verify that hyper-adapters encode language relatedness, unlike regular adapters. we also find that the gains of hyper-adapters are proportional to the redundancy in the training data. ## background: multilingual nmt in this work, we train universal mnmt models following @xcite . we prepend a special token ⟨2xx⟩ to the source and target sequences, that denotes the target language. given a source sentence @xmath0, x 2 , ..., x |x| ⟩, a target sequence @xmath1, y 2 , ..., y |y| ⟩ and a target language token t, we train our models as follows: we use the transformer architecture @xcite as the backbone of all our models. ## hyper-adapters we propose to use a hyper-network @xcite , a network that generates the weights of another network, to produce the weights of all adapter modules, dubbed hyper-adapters. as shown in figure 2 , we use a single hyper-network to generate adapters for all languages and layers by conditioning on (s, t, l) tuples, where s and t denote the source and target language and l denotes the encoder or decoder layer-id (e.g., enc3). unlike regular adapters, our approach enables information sharing across languages and layers, and the hypernetwork can learn to optimally allocate its capacity across them. our hyper-network has 3 components: input we first embed (s, t, l). we use a shared matrix for the source and target language embeddings, and a separate matrix for the layer-id embeddings for all encoder and decoder layers. encoder the language and layer embeddings are given as input to the hyper-network encoder. first, we concatenate the embeddings and project them with w in , followed by a non-linearity ϕ foot_1 , to obtain the hyper-network hidden representation @xmath2: where ∥ denotes the concatenation operation. we then pass h through n residual blocks, to encode high-level interactions between the input features: projections we feed the final representation h to separate projection heads to obtain (by reshaping their outputs) each weight matrix of a hyperadapter. specifically, we use to generate the weights for each down-projection @xmath3. we also generate the layernorm parameters γ ∈ r dz and β ∈ r dz , with the projection heads h γ ∈ r d h ×dz and h β ∈ r d h ×dz , respectively. ## experimental setup data we present results on ml50 @xcite , a multilingual translation dataset with 230m sentences between english and 50 other typologically diverse languages with data from different domains, and is larger than comparable publicly available mnmt datasets (e.g., 4x larger than opus100; @xcite . we concatenate the @xmath4, and group languages based on the amount of their training data into high (≥1m, 14 languages), med (≥100k, 17 languages) and low (<100k, 19 languages). we use sentencepiece 3 @xcite to obtain a joint vocabulary of 90k symbols. we explored smaller and larger vocabularies but empirically found that 90k strikes a good balance between the number of parameters and translation quality. finally, we filter out pairs with more than 250 tokens or with a length ratio over 2.5. ## results main results hyper-adapters-base consistently outperforms both regular adapter variants in all directions, while having the same parameter count as lang-adapters and half the parameter count of pair-adapters. we also find that our smaller variants yield very competitive results to regular adapters while being more parameter efficient. hyper-adapters-small outperforms both regular adapter variants with fewer parameters, and hyper-adapters-tiny yields comparable results with only 1/6th and 1/12th of the capacity of lang-adapters and pair-adapters, respectively. in the @xmath5, hyper-adapters-base outperforms lang-adapters by 0.9 bleu and pairadapters by 0.7 bleu. interestingly, we see gains even in high-resource settings up to +1.2 bleu, although regular adapters have dedicated capacity for these language(-pairs). in @xmath6, hyper-adapterbase has smaller improvements on medium-and high-resource languages, but we observe improvements of +1.2 bleu on low-resource languages. we hypothesize that the lower improvements on @xmath7. ## analysis this section investigates why hyper-adapters outperform regular adapters. specifically, we focus on how well each adapter variant encodes language relatedness and how it is affected by the redundancy in the training data (i.e., similarities between the data of different languages). we also explore how modifications in the hyper-network architecture affect the final model performance. ## conclusion in this work, we extend the capacity of mnmt models with hyper-adapters, which are languagespecific adapter modules generated from a hypernetwork. by resolving optimization issues not addressed by prior work, we successfully train large hyper-networks from scratch jointly with the rest of the main network on mnmt ( §3.1). we show that hyper-adapters consistently outperform other regular adapter variants across translation directions and model sizes ( §5), while improving parameter efficiency. we also observe computational efficiency gains, as a smaller transformer-base model with hyper-adapters gives similar results to the dense transformer-big model, which is computationally more expensive and requires more parameters. besides improvements in translation quality, hyper-adapters achieve faster training convergence as shown in §5. our analysis shows that, unlike regular adapters, hyper-networks enable positive transfer across the hyper-adapters of similar languages, by encoding language relatedness ( §6.1,6.3) and exploiting redundancies (i.e., language similarities) in the training data ( §6.2). finally, by manipulating the input of the hypernetwork we discover that there is a trade-off between the zero-shot and supervised translation performance of hyper-adapters ( §6.5).