Vocabulary-informed Language Encoding (2022)

Abstract: A Multilingual model relies on language encodings to identify input languages because the multilingual model has to distinguish between the input and output languages or among all the languages for cross-lingual tasks. Furthermore, we find that language encodings potentially refine multiple morphologies of different languages to form a better isomorphic space for multilinguality. To leverage this observation, we present a method to compute a vocabulary-informed language encoding as the language representation, for a required language, considering a local vocabulary covering an acceptable amount of the most frequent word embeddings in this language. In our experiments, our method can consistently improve the performance of multilingual models on unsupervised neural machine translation and cross-lingual embedding.

## introduction with tied weights across required languages, a multilingual model is trained on non-parallel or/and parallel multilingual corpora. essentially, language encodings are required for cross-lingual tasks because the multilingual model has to distinguish between the input and output languages or among all the languages generally. we observe that, besides identifications of languages, language encodings can help the model build isomorphic space for multilinguality with the help of shared tokens. specifically, our hypothesis derives from the decomposition of attention mechanisms @xcite @xcite , and we observe explicit alignments and implicit alignments, where explicit alignments are key for language identifications, and implicit alignments promote multilinguality. furthermore, the implicit alignments are a special case of unsupervised bilingual/multilingual lexical induction, helping multilingual models learn multilingual and cross-lingual knowledge. our goal is to retain the explicit alignments for language identifications and improve the implicit alignments for multilinguality. in this work, we render an analysis of the attention mechanism in optimization, and then we find implicit alignments among a language encoding and other language encodings. stemming from morphology adaptation and the observation, we present a method to compute vle (vocabularyinformed language encoding) as the required language encodings, for multilingual models. each required language maintains a local vocabulary covering a subset of the most frequent tokens in this language from the shared vocabulary of the multilingual model. given one language and its vocabulary, when adapting to the multilingual model, we apply transformation layers to the average of token embeddings in the vocabulary for vle that can be used in either padding style or adding style. vle provides language characteristics for language identification and leverages implicit alignments for multilinguility. eventually, the multilingual model can identify languages and learn better multilinguality with the help of vle. ## experiment our code is implemented on tensorflow 2.2 @xcite with 2 nvidia titan xp 12g gpus. we accumulate gradients of 2 mini-batches per pretraining step. since we have only 2 gpus, this operation emulates 4 gpus. all the links of datasets, libraries, scripts, and tools marked with ⋄ are listed in §appendix. a preview version of the code is submitted, and we open the source code on github. ## conclusion in this work, we present a method to generate vle, a vocabulary-informed language encoding for the identification of a required language in multilingual models. we consider a frequency-based and local vocabulary for every language. for a required language, the required language encoding is obtained by applying transformation layers to the average of the token embeddings in its vocabulary. in our experiments, vle shows effectiveness on unmt and cross-lingual embedding tasks and is possible to improve language adaptation and multilinguality because vle can refine the morphologies of the languages to improve the isomorphic space. our method is simple but effective and compatible with any other extension for multilingual models.