Inducing Language-Agnostic Multilingual Representations (2021)

Abstract: Cross-lingual representations have the potential to make NLP techniques available to the vast majority of languages in the world. However, they currently require large pretraining corpora or access to typologically similar languages. In this work, we address these obstacles by removing language identity signals from multilingual embeddings. We examine three approaches for this: (i) re-aligning the vector spaces of target languages (all together) to a pivot source language; (ii) removing language-specific means and variances, which yields better discriminativeness of embeddings as a by-product; and (iii) increasing input similarity across languages by removing morphological contractions and sentence reordering. We evaluate on XNLI and reference-free MT evaluation across 19 typologically diverse languages. Our findings expose the limitations of these approaches—unlike vector normalization, vector space re-alignment and text normalization do not achieve consistent gains across encoders and languages. Due to the approaches’ additive effects, their combination decreases the cross-lingual transfer gap by 8.9 points (m-BERT) and 18.2 points (XLM-R) on average across all tasks and languages, however.

## introduction figure 1 : zero-shot performance on xnli and rfeval vs. language similarity to english (top), and data sizes in wikipedia (bottom). each point is a language; brackets give the pearson correlation of points on the xand y-axis. zero-shot performance is based on the last layer of m-bert and is standardized (zero mean, unit standard deviation) for better comparison. here, we explore normalization as an alternative to re-mapping. to decrease the distance between languages and thus allow for better cross-lingual transfer, we normalize (i) text inputs to encoders before vectorization to increase cross-lingual similarity, e.g., by reordering sentences according to typological features, and (ii) the representations themselves by removing their means and standard deviations, a common operation in machine and deep learning @xcite . we evaluate vector normalization and posthoc re-mapping across a typologically diverse set of 19 languages from five language families with varying sizes of monolingual corpora. however, input normalization is examined on a smaller sample of languages, as it is not feasible for languages whose linguistic features cannot be obtained automatically. we investigate two nlp tasks, and two state-of-the-art contextualized cross-lingual encoders-multilingual bert @xcite and xlm-r @xcite . further, we provide a thorough analysis to investigate the effects of these techniques: (1) across layers; (2) to decrease the cross-lingual transfer gap, especially for low-resource and dissimilar languages; and (3) to eliminate language identity signals from multilingual representations and thus induce languageagnostic representations. we evaluate on two cross-lingual tasks of varying difficulty: (1) zero-shot cross-lingual natural language inference (xnli) measures the transfer ability of inference from source to target languages, where only the source language is annotated;and our contributions: we show that: (i) input normalization leads to performance gains of up to 4.7 points on two challenging tasks; (ii) normalizing vector spaces is surprisingly effective, rivals much more resource-intensive methods such as remapping, and leads to more consistent gains; (iii) all three techniques-vector space normalization, re-mapping and input normalization-are orthogonal and their gains often stack. this is a very important finding as it allows for improvements on a much larger scale, especially for typologically dissimilar and low-resource languages. ## language-agnostic representations analyses by @xcite indicate that random words are often assigned high cosine similarities in the upper layers of monolingual bert. we examine this in a cross-lingual setting, by randomly selecting 500 german-english mutual word translations and random word pairs within parallel sentences from europarl @xcite . fig. 2 (left) shows histograms based on the last layers of m-bert @xcite and xlm-r (conneau et al., 2019), respectively, which show that xlm-r wrongly assigns nearly perfect cosine similarity scores (+1) to both mutual word translations (matched word pairs) and random word pairs, whereas m-bert sometimes assigns low scores to mutual translations. this reaffirms that both m-bert and xlm-r have difficulty in distinguishing matched from random word pairs. surprisingly, vector space re-mapping does not seem to help for xlm-r, but better separates random from matched pairs for m-bert (fig. 2 (middle)). in contrast, the joint effect of normalization and re-mapping leads to adequate separation of the two distributions for both m-bert and xlm-r, increasing the discriminative ability of both encoders. ## results unlike re-mapping and vector space normalization, scaling input normalization to a large language sample is more difficult, as typological features differ across languages. thus, we report the results of re-mapping and vector space normalization across 19 languages, while text normalization is evaluated on a smaller sample of languages. re-mapping and vector space normalization. in tab. 2, we show results on machine translated test sets. the m-bert space modified by joint-align ⊕ norm achieves consistent improvements on rfeval (+10.1 points) and xnli (+7.6 points) on average. however, effects are different for xlm-r. the modified xlm-r outperforms the baseline xlm-r on rfeval by the largest margin (+33.5 points), but the improvement is much smaller (+2.8 points) on xnli. these gains are not an artefact of machine-translated test sets: we observe similar gains on human-translated data (see fig. 3 ). in tab. 3, we tease apart the sources of improvements. overall, the impacts of norm and joint-align are substantial, and their effect is additive and sometimes even superadditive (e.g., m-bert improves by 10.1 points on rfeval when both norm and joint-align are applied but only by 1.7 and 7.6 points individually). we note that the improvement from norm is more consistent across tasks and encoders, despite its simplicity and negligible cost. in contrast, joint-align has a positive effect for m-bert but it does not help for xlm-r on the xnli task, notwithstanding the minor difference of two encoders, e.g., much larger training data and a different tokenizer used in xlm-r. we believe the poor discriminative ability of xlm-r, viz., that it cannot distinguish word translations from random word pairs, leads to the inconsistent behavior of joint-align. as a remedy, negative examples such as random pairs could be included in eq. ( 3 ) during training so as to decrease the discriminative gap between m-bert and xlm-r. this suggests that future research efforts should focus on the robustness of cross-lingual alignments. batch vs. layer normalization. unsurprisingly, the choice of batch size greatly influences xnli performance when applying batch normalization for m-bert and xlm-r (fig. 4 ). we find that (i) the larger the batch size is, the smaller the impacts on xnli, and (ii) a batch size of 8 performs best. interestingly, layer normalization does not help for xnli, even though it yields batchindependent statistics and is effective in stabilizing the training process @xcite . we note that per batch sequences with varying time steps (i.e., sentence length) are often padded with zero vectors in practice. this leads to inaccurate batchindependent statistics, as they are computed across all time steps, unlike batch normalization with per batch statistics for individual time steps. in addition to batch and layer normalizations, other nor- ## conclusion cross-lingual systems show striking performance for transfer, but their success crucially relies on two constraints: the similarity between source and target languages and the size of pre-training corpora. we comparatively evaluate three approaches to address these challenges, removing language-specific information from multilingual representations, thus learning language-agnostic representations. our extensive experiments, based on a typologically broad sample of 19 languages, show that (vector space and input) normalization and re-mapping are oftentimes complementary approaches to improve cross-lingual performance, and that the popular approach of re-mapping leads to less consistent improvements than the much simpler and less costly normalization of vector representations. input normalization yields benefits across a small sample of languages; further work is required for it to achieve consistent gains across a larger language sample.