Informative Language Representation Learning for Massively Multilingual Neural Machine Translation (2022)

Abstract: In a multilingual neural machine translation model that fully shares parameters across all languages, an artificial language token is usually used to guide translation into the desired target language. However, recent studies show that prepending language tokens sometimes fails to navigate the multilingual neural machine translation models into right translation directions, especially on zero-shot translation. To mitigate this issue, we propose two methods, language embedding embodiment and language-aware multi-head attention, to learn informative language representations to channel translation into right directions. The former embodies language embeddings into different critical switching points along the information flow from the source to the target, aiming at amplifying translation direction guiding signals. The latter exploits a matrix, instead of a vector, to represent a language in the continuous space. The matrix is chunked into multiple heads so as to learn language representations in multiple subspaces. Experiment results on two datasets for massively multilingual neural machine translation demonstrate that language-aware multi-head attention benefits both supervised and zero-shot translation and significantly alleviates the off-target translation issue. Further linguistic typology prediction experiments show that matrix-based language representations learned by our methods are capable of capturing rich linguistic typology features.

## introduction multilingual neural machine translation (mnmt) @xcite @xcite , unlike bilingual machine translation with task-specific engineering (language-specific features), uses a single learning system for multiple language pairs, which is jointly trained in a mult-task learning formalism @xcite . parameter sharing across different languages in mnmt models enables transferring of intermediate representations and knowledge among languages, which makes it beneficial to machine translation of low-resource and even zero-resource languages @xcite @xcite @xcite . according to the degree of parameter sharing across languages, multilingual neural machine translation (mnmt) approaches can be categorized into two strands: full parameter sharing and partial parameter sharing @xcite . the former uses a unified model where both the encoder and decoder are shared for all languages. to guide the translation direction, a special token is usually prepended to the beginning of the source or target sentence to indicate the target language @xcite . instead of sharing all model parameters for all translation directions, the latter uses language-specific components, e.g., separate encoders (in many-to-one translation), separate decoders (in one-to-many translation), or separate cross-attention networks @xcite @xcite @xcite . the key for full sharing models to distinguish target languages lies in the prepended tokens. 2 prior studies find that the embeddings of the prepended tokens encode typological properties of languages @xcite @xcite . hence the token embeddings are also referred to as language representations or language embeddings, guiding translation into different target languages whose typological features varies a lot. while the prepened tokens play a significant role in languagespecific knowledge learning, they are not usually working appropriately, making mnmt models generate translations in wrong languages, especially in zero-shot translation. such off-target translation issue @xcite implies that prepending special tokens to sentence pairs in full-sharing models is not adequate to learn sufficient language-specific information to guide the mnmt models to translate into right directions. partial sharing may be suitable for learning language-specific properties. nevertheless, it suffers from a rapid growth in the number of parameters with the increase in the number of languages. additionally, which modules should be languagespecific still remains to be further explored. in this work, we attempt to learn more informative language representations (beyond language embeddings) to improve translation quality of the mnmt models with the full parameter sharing strategy. we argue that the prepended token cannot provide sufficient target language information to guide the translation direction, which is detrimental to both supervised and zero-shot translation. we conjecture that the direction control supervision provided by the prepended token is becoming weaker as the translation information flows to deeper layers. therefore, instead of prepending special tokens to sentences, we propose a language embedding embodiment (lee) strategy that embodies language embeddings at critical switching points (e.g., in-between self-attention and ffn, or self-attention and cross-attention) across layers in both the encoder and decoder along the information flow from the source to the target, so as to amplify the translation direction guiding signal, shown in figure 1 . experiment results show a boosted translation performance compared with the standard full sharing mnmt model @xcite . inspired by the performance improvement, we further propose language-aware multi-head attention (laa) to model typological features of languages. laa estimates a matrix, instead of a vector, as the language representation for each language, which is able to encode more language information than a single fixed-length vector that is commonly adopted in previous studies. motivated by multi-head attention @xcite , we split the matrix along the column to learn information from different representation subspaces. in order to probe the typological properties encoded in language representations, we extract the language representations from the trained mnmt model and use them for linguistic typology prediction. the main contributions of our work can be summarized as follows: ## related work multilingual neural machine translation pioneering studies on multilingual neural machine translation mostly favor to extend the standard bilingual model to mnmt by designing language specific components @xcite @xcite , where the number of parameters grows rapidly with the number of languages. alternatively, @xcite ; @xcite propose to prepend an artificial target language token to source sentences without modifying the model architecture, which is parameter efficient and eases model design. it hence becomes the dominant approach to massively mnmt due to its simplicity and effectiveness @xcite @xcite @xcite . despite that, it usually lags behind the bilingual counterpart on high-resource language pairs and suffers from off-target translation issue @xcite on zero-shot translation. to alleviate these issues, subsequent studies propose approaches such as increasing the model capacity by deepening the model @xcite , increasing the model cardinality @xcite , inserting mixture-of-experts (moe) layers @xcite , designing lightweight language specific modules @xcite @xcite @xcite @xcite , reducing negative interference by clustering similar languages @xcite , resolving the gradient conflicts @xcite , dividing the model parameters into a shared part and language-specific part @xcite @xcite , bridging the representation gap by data argumentation @xcite and contrastive learning @xcite , introducing language-agnostic regularization @xcite and explicit word alignment supervision @xcite , to name a few. however, these studies neglect the capacity bottleneck in language representations as they all resort to a language embedding with the same dimension as word embeddings to encode the information of languages whose typological features diverse a lot. in contrast, laa adopts the matrix as the language representation to alleviate the bottleneck. moreover, the number of additional parameters brought by laa is manageable thus it can adapt to massive mnmt settings. linguistic typology prediction linguistic typology mainly studies the classification of languages based on their structural properties. our work is closely related to inferring typological features with language representations from the trained mnmt models. previous studies demonstrate that language representations in multilingual neural models can capture cross-lingual similarities between languages @xcite @xcite @xcite , which can potentially recover missing features in typological databases. @xcite fuse language representations in trained mnmt models with features from typological databases to enhance knowledge transfer in mnmt models. they perform typological feature prediction to analyze typological knowledge in the learned language representations. similarly, we conduct typological feature prediction to evaluate the quality of the language representations learned in lee and laa. ## language embedding embodiment the central idea for the language embedding embodiment strategy is shortening distance between the language embedding and target translation in order to enable language-embodied translation generation. in transformer-based mnmt models, instead of only feeding the language embedding into the first layer @xcite , we embody the language embedding into many different layers along the translation information flow from the source to the target, as illustrated in figure 1. since the prepended artificial token has been already in vocabulary, feeding the language embedding into other layers will not result in extra parameters. as shown in figure 1 , six candidate positions in the standard transformer can be used to embody the language embedding. consider that the language embedding is to be embodied in position 5. let the architecture of the language-aware multihead attention. w lang i is the trainable language-specific matrix of the i-th head. ## language-aware multi-head attention lee adopts a fixed-length language embedding to guide the mnmt model to generate translations of the target language. however, the fixed-length language vector may not be sufficient to capture all key linguistic features and diversities in languages, which are important for translation, due to its limited representational capacity. experiment results of lee on both zero-shot translation and linguistic typology prediction in section 5 have also empirically verified the capacity bottleneck in the fixedlength language embedding. to alleviate the capacity bottleneck, we further propose the language aware multi-head attention (shown in figure 2 ), representing a language with a matrix, rather than a vector. we incorporate the language-specific matrix into the attention modules (self-attention in the encoder/decoder or cross-attention) since they are the essential components in transformer. following the strategy of multi-head attention @xcite , we split the matrix along its column to learn information from different representation subspaces of languages. in the standard multi-head attention, three parameter matrices w q i , w k i , w v i are used in the i-th attention head to project q (queries), k (keys) and v (values). after projection, the scaled dot-product attention is applied on each head. the outputs from all heads are then concatenated and multiplied with and h is the number of heads. for simplicity, we set d @xmath0. in practice, the parameter matrices of all heads from q, k, v are concatenated into w q , w k , w v respectively and the projected results are split into h heads, thus benefiting parallel computation and simplifying implementation. in laa, we use the matrix for language representation to relax the capacity constraint. let matrix w lang be the representation of language lang, where to inject the target language information into mnmt models, which can be formulated as follows: let z i be the scaled dot-product attention output of the i-th head. z i is computed as: we split w o into h heads along the row and pack the outputs from different heads together as follows: ## experiments we conducted two types of experiments, many-tomany translation and linguistic typology prediction, aiming at: (1) systematically evaluating the traditional language token prepending method and (2) examing the effectiveness of the proposed language embedding embodiment and language-aware multihead attention. ## conclusion to improve massively mnmt, we have presented two approaches to learning informative language representations, language embedding embodiment and language-aware multi-head attention. we find that the ways to inject target language information into mnmt models have a significant impact on the translation performance, especially on zeroshot translation. we validate the effectiveness of the two approaches on two public datasets. we probe the typological features encoded in language representations learned by lee and laa through linguistic typology feature prediction. the superior prediction performance of the matrix-based language representations learned by laa on syntax features demonstrates its informativeness.