On the Language Neutrality of Pre-trained Multilingual Representations (2020)

Abstract: Multilingual contextual embeddings, such as multilingual BERT and XLM-RoBERTa, have proved useful for many multi-lingual tasks. Previous work probed the cross-linguality of the representations indirectly using zero-shot transfer learning on morphological and syntactic tasks. We instead investigate the language-neutrality of multilingual contextual embeddings directly and with respect to lexical semantics. Our results show that contextual embeddings are more language-neutral and, in general, more informative than aligned static word-type embeddings, which are explicitly trained for language neutrality. Contextual embeddings are still only moderately language-neutral by default, so we propose two simple methods for achieving stronger language neutrality: first, by unsupervised centering of the representation for each language and second, by fitting an explicit projection on small parallel data. Besides, we show how to reach state-of-the-art accuracy on language identification and match the performance of statistical methods for word alignment of parallel sentences without using parallel data.

## introduction multilingual bert (mbert; @xcite gained popularity as a contextual representation for many multilingual tasks, e.g., dependency parsing @xcite , cross-lingual natural language inference (xnli) or named-entity recognition (ner) @xcite @xcite . recently, a new pre-trained model, xlm-roberta (xlm-r; @xcite , claimed to outperform mbert both on xnli and ner tasks. we also study distilbert @xcite applied to mbert, which promises to deliver comparable results to mbert at a significantly lower computational cost. @xcite present an exploratory paper showing that mbert can be used cross-lingually for zero-shot transfer in morphological and syntactic tasks, at least for typologically similar languages. they also study an interesting semantic task, sentence-retrieval, with promising initial results. their work leaves many open questions regarding how well the cross-lingual mbert representation captures lexical semantics, motivating our work. in this paper, we directly assess the cross-lingual properties of multilingual representations on tasks where lexical semantics plays an important role and present one unsuccessful and two successful methods for achieving better language neutrality. multilingual capabilities of representations are often evaluated by zero-shot transfer from the training language to a test language @xcite . however, in such a setup, we can never be sure if the probing model did not overfit for the original language, as training is usually stopped when accuracy decreases on a validation set from the same language (otherwise, it would not be zero-shot), even when it would have been better to stop the training earlier. this overfitting on the original language can pose a disadvantage for information-richer representations. to avoid such methodological issues, we select tasks that only involve a direct comparison of the representations with no training: cross-lingual sentence retrieval, word alignment (wa), and machine translation quality estimation (mt qe). additionally, we explore how the language is represented in the embeddings by training language id classifiers and assessing how the representation similarity corresponds to phylogenetic language families. we find that contextual representations are more language-neutral than static word embeddings which have been explicitly trained to represent matching words similarly and can be used in a simple algorithm to reach state-of-the-art results on word alignment. however, they also still strongly carry information about the language identity, as demonstrated by a simple classifier trained on mean-pooled contextual representations reaching state-of-the-art results on language identification. we show that the representations can be modified to be more language-neutral with simple, straightforward setups: centering the representation for each language or fitting explicit projections on small parallel data. we further show that xlm-roberta (xlm-r; @xcite outperforms mbert in sentence retrieval and mt qe while offering a similar performance for language id and wa. ## related work multilingual representations, mostly mbert, were already tested in a wide range of tasks. often, the success of zero-shot transfer is implicitly considered to be the primary measure of language neutrality of a representation. despite many positive results, some findings in the literature are somewhat mixed, indicating limited language neutrality. zero-shot learning abilities were examined by @xcite on ner and part-of-speech (pos) tagging, showing that the success strongly depends on how typologically similar the languages are. similarly, @xcite trained good multilingual models but struggled to achieve good results in the zero-shot setup for pos tagging, ner, and xlni. @xcite draw similar conclusions for language-generation tasks. @xcite succeeded in zero-shot dependency parsing but required supervised projection trained on word-aligned parallel data. the results of @xcite on dependency parsing suggest that methods like structural probing @xcite might be more suitable for zero-shot transfer. pires et al. (2019) also assessed mbert on cross-lingual sentence retrieval between three language pairs. they observed that if they subtract the average difference between the embeddings from the target language representation, the retrieval accuracy significantly increases. we systematically study this idea in the later sections. xtreme @xcite and xglue @xcite , two recently introduced benchmarks for multilingual representation evaluation, assess representations on a broader range of zero-shot transfer tasks that include natural language inference @xcite and question answering @xcite . their results show a clearly superior performance of xlm-r compared to mbert. many works clearly show that downstream task models can extract relevant features from the multilingual representations @xcite @xcite . however, they do not directly show language-neutrality, i.e., to what extent similar phenomena are represented similarly across languages. thus, it is impossible to say whether the representations are language-agnostic or contain some implicit language identification. our choice of evaluation tasks eliminates this risk by directly comparing the representations. ## centering representations one way to achieve stronger language neutrality is by suppressing the language identity, only keeping what encodes the sentence meaning. it can be achieved, for instance, using an explicit projection. however, training such a projection requires parallel data. instead, we explore a simple unsupervised method: representation centering. following @xcite , we hypothesize that a sentence representation in mbert is additively composed of a language-specific component, which identifies the language of the sentence, and a language-neutral component, which captures the meaning of the sentence in a language-independent way. we assume that the language-specific component is similar across all sentences in the language. we estimate the language centroid as the mean of the representations for a set of sentences in that language and subtract the language centroid from the contextual embeddings. by doing this, we are trying to remove the language-specific information from the representations by centering the sentence representations in each language so that their average lies at the origin of the vector space. the intuition behind this is that within one language, certain phenomena (e.g., function words) would be very frequent, thus being quite prominent in the mean of the representations for that language (but not for a different language), while the phenomena that vary among sentences of the language (and thus presumably carry most of the meaning) would get averaged out in the centroid. we thus hypothesize that by subtracting the centroid, we re-move the language-specific features (without much loss of the meaning content), making the meaningbearing features more prominent. we analyze the semantic properties of the original and the centered representations on a range of probing tasks. for all tasks, we test all layers of the model. we test both the [cls] token vector and mean-pooled states for tasks utilizing a singlevector sentence representation. ## probing tasks we employ five probing tasks to evaluate the language neutrality of the representations. the first two tasks analyze the contextual embeddings. the other three tasks are cross-lingual nlp problems, all of which can be treated as a general task of a cross-lingual estimation of word or sentence similarities. supposing we have sufficiently language-neutral representations, we can estimate these similarities using the cosine distance of the representations; the performance in these tasks can thus be viewed as a measure of the languageneutrality of the representations. moreover, in addition to such an unsupervised approach, we can also utilize actual training data for the tasks to further improve the performance of the probes; this does not tell us much more about the representations themselves but leads to a nice by-product of reaching state-of-the-art accuracies for two of the tasks. ## probed models aligned static word embeddings. as a baseline in all our experiments, we use aligned static word embeddings @xcite . unlike hidden states of pre-trained transformers, they do not capture sentence context. however, they were explicitly trained to be language-neutral with respect to lexical semantics. we represent sentences as an average of the embeddings of the words. multilingual bert @xcite is a deep transformer @xcite encoder that is trained in a multi-task learning setup, first, to be able to guess what words were masked-out in the input and, second, to decide whether two sentences follow each other in a coherent text. we use a pre-trained mbert model that was made public with the bert release. 1 the model dimension is 768, the hidden layer dimension 3072, self-attention uses 12 heads, the model has 12 layers. it uses a vocabulary of 120k wordpieces shared for all languages. it is trained using a combination of a masked language model (mlm) objective and sentenceadjacency objective. for the mlm objective, 15% of input subwords are masked out, and the model predicts the masked subwords. for the sentence-adjacency objective, a special @xcite token is prepended to the input. the embedding corresponding to this token is used as an input to a classifier predicting if the input sentences are adjacent. therefore, for models based on mbert, we experiment both with @xcite vector and the meanpooled vector, i.e., average embeddings for the rest of the tokens. udify. the udify model @xcite uses mbert to train a single model for dependency parsing and morphological analysis of 75 languages. during training, mbert is finetuned, which improves accuracy. results on zero-shot parsing suggest that the finetuning leads to better language neutrality with respect to morphology and syntax. distillmbert. this model was inferred from mbert by knowledge distillation @xcite 2019). the model only has 6 layers instead of 12. the rest of the hyperparameters remain the same. it was initialized with a subset of the original mbert parameters and trained on similar training data and optimized towards cross-entropy of its output distribution with respect to the output of the teacher mbert model while keeping the mlm objective in the multi-task learning setup. as the model is forced to use smaller space to obtain the representation, it might leverage the similarities between languages and reach better language neutrality. xlm-roberta. @xcite claim that the original mbert is under-trained and train a similar model on a larger dataset that consists of two terabytes of plain text extracted from com-moncrawl @xcite . unlike mbert, xlm-r uses a sentencepiece-based vocabulary @xcite ) of 250k tokens. the rest of the architecture remains the same as in the case of mbert. we train the model using the mlm objective only, without the sentence adjacency prediction. ## experimental setup to train the language id classifier, for each of 73 languages covered both by mbert and xlm-r, we randomly select 110k sentences of at least 20 characters from wikipedia and keep 5k for validation and 5k for testing for each language. we also use the training data to estimate the language centroids and training the lng-free version of the model. for parallel sentence retrieval, we use a multiparallel corpus of test data from the wmt14 evaluation campaign @xcite with 3,000 sentences in czech, english, french, german, hindi, and russian. to compute the linear projection (for the special linear projection experimental condition), we used the wmt14 development data (500-3000 sentences per language pair). we use manually annotated wa datasets to evaluate word alignment between english on one side and czech (2.5k sent.; mareček, 2016) foot_1 , swedish (192 sent.; holmqvist and ahrenberg, 2011) foot_2 , german (508 sent.) foot_3 , french (447 sent.; och and ney, 2000) foot_4 and romanian (248 sent.; @xcite foot_5 on the other side. we compare the results with fastalign @xcite and efmaral ( östling and tiedemann, 2016) models, which were provided with 1m additional parallel sentences from paracrawl @xcite 7 . we use pre-trained tables provided by @xcite 8 for the static word embeddings. the embeddings were trained on wikipedia and aligned with a projection trained on small bilingual dictionaries. the number of word types captured in the embedding tables spans from 350k for romanian to 2.5m for english. the experiments with contextualized embeddings are implemented using the transformers package @xcite , which we also use for obtaining the pre-trained models, except for udify, which was obtained from @xcite ). 9 the lng-free mbert version was finetuned using the same data that was used for language identification. our source code is available at https://github. com/jlibovicky/assess-multilingual-bert . ## results language identification. for comparison, the state-of-the-art language id model from fasttext @xcite reaches 91.4% accuracy with a pre-trained model, and 91.8% when retrained on our training data, i.e., slightly worse than our best model based on mbert. langid.py @xcite reaches 90.1% when trained on the same dataset. adversarial finetuning prevented the language identification only from the @xcite vector and only marginally for mean-pooling. this supports the hypothesis that language identity is derived from the presence of function words and structures and representation centering suppresses these frequent phenomena. centering the representations within languages requires knowing the language in advance. it is therefore an oracle experiment. in a sense, centering adds language-specific information to the representation which the classifier might take advantage of. however, because the centering decreases the accuracy, we can interpret this as removing information about the language identity. for further comparison, we conduct the same experiment with aligned word embeddings for 44 languages @xcite . the language id accuracy is 99.5% but drops to 2.3% after centering (the same as assigning language by chance), which supports our intuition about centering functioning as removal of frequent patterns. note, however, that even the experiment without centering is an oracle experiment cannot be considered as language identification because we need to know the language identity in advance to use the matching embeddings table, so the accuracy is not comparable with other experiments. language similarity. figure 2 is a tsne plot @xcite of the language centroids, showing that the centroids' similarity tends to correspond to the similarity of the languages. ## conclusions using a set of semantically oriented tasks, we showed that unsupervised bert-based multilingual contextual embeddings capture similar semantic phenomena quite similarly across different languages. surprisingly, in cross-lingual semantic similarity tasks, employing cosine similarity of the contextual embeddings without any tuning or adaptation clearly and consistently outperforms cosine similarity of static multilingually aligned word embeddings, even though these were explicitly trained to be language-neutral using bilingual dictionaries. nevertheless, we found that vanilla contextual embeddings contain a strong language identity signal, as demonstrated by their state-of-the-art performance for the language identification task. we hypothesize this is due to the sentence-adjacency objective used during training because language identity is a strong feature for adjacency. we explored three ways of removing the language id from the representations in an attempt to make them even more cross-lingual. while adversarial finetuning of mbert did not help, a simpler unsupervised approach of language-specific centering of the representations managed to reach the goal to some extent, leading to higher performance of the centered representations in the probing tasks. the adequacy of the approach is also confirmed by a strong performance of the computed language centroids in estimating language similarity. still, an even stronger language-neutrality of the representations can be achieved by fitting a supervised linear projection on a small set of parallel sentences. although representation centering leads to satisfactory language neutrality, it still requires knowing in advance what the language is. the future work thus should focus on representations that are more language-neutral by default, not requiring subsequent language-dependent modifications. we hope that this work helps to establish how future language-neutral representation should be evaluated.