Comparative Analysis of Cross-lingual Contextualized Word Embeddings (2022)

Abstract: Contextualized word embeddings have emerged as the most important tool for performing NLP tasks in a large variety of languages. In order to improve the cross- lingual representation and transfer learning quality, contextualized embedding alignment techniques, such as mapping and model fine-tuning, are employed. Existing techniques however are time-, data- and computational resource-intensive. In this paper we analyze these techniques by utilizing three tasks: bilingual lexicon induction (BLI), word retrieval and cross-lingual natural language inference (XNLI) for a high resource (German-English) and a low resource (Bengali-English) language pair. In contrast to previous works which focus only on a few popular models, we compare five multilingual and seven monolingual language models and investigate the effect of various aspects on their performance, such as vocabulary size, number of languages used for training and number of parameters. Additionally, we propose a parameter-, data- and runtime-efficient technique which can be trained with 10% of the data, less than 10% of the time and have less than 5% of the trainable parameters compared to model fine-tuning. We show that our proposed method is competitive with resource heavy models, even outperforming them in some cases, even though it relies on less resource

## introduction contextualized word representations generated from pre-trained language models have outperformed previously standard static embeddings. static distributional word representations offer a single representation for a word regardless of its current context @xcite . contrarily, a word's contextual representation is influenced by the context in which it is used. contextualized embeddings have demonstrated ground-breaking performance across sev-eral nlp tasks and languages, and accommodate many semantic and syntactic aspects of words @xcite @xcite . from the introduction of elmo @xcite and ulmfit @xcite to the present, different types of language models have been proposed @xcite @xcite @xcite @xcite of which the most influential is bert @xcite which initiated an era of transformer @xcite based language models. multilingual language models (mlms) can perform various tasks across different languages. previous works @xcite have showed that the mlm's performance in different transfer learning tasks can further be improved by alignment. the idea of aligning contextualized embeddings is to move the representations of words with similar meaning from different languages closer to each other. there are several ways to perform alignment on contextualized embeddings, such as anchor mapping @xcite and full model fine-tuning @xcite . however, all of these methods have several shortcomings. it is (1) time-consuming, taking about 24 hours to perform mapping. in contrast to static embeddings, in case of contextualized embeddings the generation of anchor embeddings is required to be able to perform mapping which is the majority of the required time @xcite . similarly, it takes about 8 hours to perform model fine-tuning @xcite on mbert. it is also (2) resource-intensive requiring a lot of gpu memory due to model size and (3) data-intensive requiring a huge collection of monolingual sentences for anchor generation, while fine-tuning requires around 250k pairs of parallel sentences to produce the best-reported alignment @xcite . as a result of these limitations anchor embeddings map-ping and fine-tuning are often difficult or expensive to perform, deploy and use in real-world scenarios. to the best of our knowledge there is no study available until now where different model architectures and alignment techniques on various downstream tasks are systematically compared other than on the most popular models such as mbert and xlm-roberta @xcite @xcite . in this paper our main goal is to fill this gap. we have compared five multilingual and seven monolingual models with three current alignment techniques (vecmap @xcite , rcsls @xcite and model fine-tuning @xcite ) from different perspectives such as multilingual vs. monolingual, big vs. small models and the effect of vocabulary. to assess the models and alignment techniques from different perspectives we used three different tasks: bilingual lexicon induction (bli), word retrieval @xcite and zero-shot cross-lingual natural language inference (xnli) on two language pairs: high-resource german-english and low-resource bengali-english. motivated by the shortcomings of current alignment methods discussed above, and inspired by the fine-tuning based alignment technique of @xcite , in addition to the comparative analysis we propose a parameter, data and time efficient alignment technique which requires 10% of the data, runs within less than 10% of the time and uses the amount of less than 5% of trainable parameters compared to model fine-tuning @xcite ). an overview of our proposed approach is given in figure 1 . the findings of our experiments demonstrate that 1) multilinguality always leads to better performance in cross-lingual transfer tasks. 2) we should choose bigger models over smaller models when the resources (computational and data) are available but 3) in case of unattainable resources smaller but specialized multilingual models, such as indic-bert @xcite , should be chosen, since they are capable of outperforming or performing similar to the big multilingual models, such as xlm-roberta @xcite , on a language the model is specialized for. 4) having a large vocabulary and language support is not an advantage of itself, instead the number of tokens allocated for a given language/script plays a more important role. 5) big language models are sensitive to batch size and learning rate. 6) model finetuning based alignment @xcite strengthens the quality of mlm's contextualized embeddings and 7) our proposed method is competitive with resource heavy models, even outperforming them in some cases despite having a significantly lower number of trainable parameters. our work shows that in specific cases (such as for bengali on xnli task) less resource intensive but more targeted solutions (e.g. indic-bert) can also be successfully employed. the paper is structured as the following: the related work is discussed in section 2. then section 3 contains required background knowledge followed by the explanation of our proposed approach in section 4. following that, section 5 contains all the information regarding the tasks, data, different pipelines of our experiments, training procedures and hyperparameters. in section 6 we discuss the results of different tasks and experiments. finally, we conclude our work in section 7. additionally, alignment approaches are resource intensive. performing anchor generation for mapping takes the majority of the required time @xcite . likewise, fine-tuning mbert takes more than 8 hours @xcite , and for xlm-roberta it is even longer. due to model size, they require a lot of gpu memory. also, they are data-intensive requiring a huge collection of monolingual sentences @xcite for anchor generation and during fine-tuning, around 250k pairs of parallel sentences are required to produce an alignment of good quality. focusing on these shortcomings we propose a parameter, data and time efficient alignment approach to tackle these issues. our proposed approach is lightweight compared to full model finetuning based alignment, as well as more time and data efficient than fine-tuning and anchor based alignment. ## proposed approach fao is data-intensive requiring 250k parallel sentences, it is time-consuming and resource-intensive. similarly, applying simple but efficient alignment techniques like vecmap and rcsls is too timeconsuming and resource intensive in the case of contextualized embeddings. inspired by these issues we propose a small alignment architecture which can be trained swiftly (less than 10% of the time required for fine-tuning the whole model) with a few thousand parallel sentences (10% of the data required for fine-tuning the whole model) and trainable parameters for all the proposed architectures are less than 3% of the language model's parameters. to achieve this we add small trainable modules to mlms and keep the rest of the network frozen. linear or transformer layer on top we add a single linear or transformer layer on top of the used mlm. an overview of our proposed method is provided in figure 1 . first, the sentences are fed into the language model then we extract the embeddings of all the words (we only take the embedding of the last subword following @xcite in case a word is split). these embeddings are then fed to the proposed linear or transformer layer, which outputs embeddings of the same size as the mlm. as mentioned, we train only the added layer and keep the mlm frozen. this way the number of parameters to be trained and the required time are significantly reduced compared to fao. additionally, unlike fao we do not use the regularizer loss which reduces computation and memory use since the initial model (f 0 ) is unnecessary. the rest of the procedure is the same as described in section 3. we named our method modified alignment objective (mao). adapters additionally, we leverage adapters @xcite in each of the mlm layers together with a transformer layer on top of the models. similarly as above, we only trained the transformer and the adapter parameters and kept the language model parameters frozen. ## results & discussion we show results for our word retrieval task in the results shown in these tables are the outcome of a single model per setup. we did not average the results across runs or seeds in order to reduce the required computational resources. next we discuss the comparison of various aspects of the selected models. multilingual vs. monolingual models from the results of the bli task in fine-tuned mbert-cased accompanied by rcsls outperformed all the models in the de-en and bn-en direction. monolingual models exhibited significantly poor performance for this word level bli task, which we did not anticipate. effect of vocabulary size and language support on the sentence level task of xnli shown in vecmap vs. rcsls in proposed alignment approach from the accuracy scores reported in this simple and smaller approach outperformed fine-tuned mbert, dbert on the german test set and indic-bert in the bengali test set in the xnli task. for the bli task our proposed approach with xlm-roberta and rcsls outperformed all the other models for both de-en and bn-en directions by achieving p@5 scores 75.03 and 32.90 for de-en and bn-en directions respectively. ## conclusion in this paper we have compared currently popular alignment techniques using multilingual and monolingual models of various architectures from different aspects by utilizing two word level tasks (bli and word retrieval) and one sentence level task xnli with one low resource (bengali-english) and one high resource language pair (german-english). we also have proposed a time, data and parameter efficient alignment technique. our experimental results demonstrate that multilinguality always lead to better performance in cross-lingual transfer tasks. when the resources (computational and data) are available, bigger models are always preferred over smaller models, but when the resources are not accessible, smaller but specialized multilingual models should be chosen, since they are capable of performing similarly to or better than the large multilingual models on the languages the model is specialized for. a large set of supported languages and a large vocabulary does not always assist in all types of tasks in contrast to models specifically trained for a limited number of target languages. large language models are sensitive regarding batch size and learning rate. finally, high resource languages and large multilingual models perform well with our proposed approach. in future work we aim to develop alignment techniques capable of performing well even on low resource unseen languages.