Discovering Low-rank Subspaces for Language-agnostic Multilingual Representations (2022)

Abstract: Large pretrained multilingual language models (ML-LMs) have shown remarkable capabilities of zero-shot cross-lingual transfer, without direct cross-lingual supervision. While these results are promising, follow-up works found that, within the multilingual embedding spaces, there exists strong language identity information which hinders the expression of linguistic factors shared across languages. For semantic tasks like cross-lingual sentence retrieval, it is desired to remove such language identity signals to fully leverage semantic information. In this work, we provide a novel view of projecting away language-specific factors from a multilingual embedding space. Specifically, we discover that there exists a low-rank subspace that primarily encodes information irrelevant to semantics (e.g., syntactic information). To identify this subspace, we present a simple but effective unsupervised method based on singular value decomposition with multiple monolingual corpora as input. Once the subspace is found, we can directly project the original embeddings into the null space to boost language agnosticism without finetuning. We systematically evaluate our method on various tasks including the challenging language-agnostic QA retrieval task. Empirical results show that applying our method consistently leads to improvements over commonly used ML-LMs.

## introduction large language models pretrained with selfsupervised objectives (e.g., masked language modeling) have become the de-facto standard for various nlp tasks @xcite @xcite . follow-up extensions to the multilingual setting inherit similar training objectives and show very promising results @xcite @xcite . despite these models are trained * corresponding author. while these ml-lms offer practical solutions for cross-lingual tasks, there is an enduring debate about why the ml-lms work. from a positive perspective, @xcite conduct an exploratory study on mbert @xcite , suggesting that cross-lingual transfer is possible even to languages in different scripts. @xcite probe mbert for structural phenomena and find that its representations can recover syntactic tree distances in languages other than english. these findings present shreds of evidence that the pretrained multilingual representations do capture cross-lingual properties in various aspects. on the flip side, a line of research shows that pretrained ml-lms encode strong language-specific signals. this causes their multilingual representations to cluster by language identities instead of semantic meaning @xcite @xcite . the property largely hinders the expression of linguistic signals shared across languages. for applications like cross-lingual sentence retrieval that mainly consider semantic information, ml-lms with strong language-specific signals tend to retrieve answers from specific languages, regardless of their semantic meaning @xcite . motivated by previous findings about language identity information, we aim to locate languagespecific factors captured by the pretrained ml-lms for recovering a language-agnostic embedding space. inspired by advances in domain generalization @xcite @xcite , we explore a simple but effective approach, lsar, to discover a low-rank subspace for language-agnostic representations within an ml-lm. the subspace primarily encodes information irrelevant to semantics, and can be identified without any translation pairs based on singular value decomposition. once the subspace is found, we can directly factor out language-specific factors from the multilingual embeddings by projecting them into the null space without finetuning. to evaluate lsar, we focus on semantic tasks for multilingual sentence embeddings. on standard cross-lingual zero-shot transfer tasks including classification and sentence retrieval, lsar consistently achieves significant improvements. especially, applying lsar leads to significant improvements for pretrained ml-lms on lareqa @xcite , a challenging benchmark targeting strong language agnosticism. we further examine what information exactly the subspace contains. by performing correlation analysis between structural language similarities obtained from the uriel database @xcite and the language similarities captured on the subspace, we observe that the subspace encodes a great deal of syntactic information. this implies that lsar successfully erases linguistic signals that are redundant to semantic tasks to facilitate language agnosticism. to conclude, our main contributions are: ## methodology in this section, we first introduce our method to identify the low-rank language-specific subspace in an unsupervised manner. once the subspace is found, we can then suppress the language identity from the original multilingual embeddings to achieve language agnosticism rectification by projecting them to the null space. this posttraining alignment procedure can largely benefit downstream tasks like cross-lingual retrieval which solely utilize semantic-related information. ## experiments we systematically evaluate our method on various tasks followed by further analyses 1 , with the purposes of understanding: 1) whether the proposed approach can benefit downstream tasks; 2) what exactly the identified low-rank subspace captures. to begin with, we describe our evaluation protocol for the alignment methods, which largely follows @xcite but with a broader scope to include more base models as listed in section b. given one of the pretrained ml-lms, we first randomly collect 10,000 sentences for each language from the oscar corpus @xcite covering all the evaluated languages and their web crawl texts 2 . the sentence embeddings extracted by the pretrained model are then used for finding mbert xlm xlm-r labse cross-lingual zero-shot transfer (w/o finetuning) original 37.53+00.00% 28.13+00.00% 57.68+00.00% 95.47+00.00% centered @xcite 39.57+05.43% 27.13-03.57% 61.08+05.89% 95.56+00.10% lir (@xmath0. the low-rank subspace described in equation 1. unless otherwise indicated, we consistently report lsar with @xmath1, where l is the number of the evaluated languages. we evaluate language agnosticism over pretrained ml-lms that are commonly used, as described in appendix b. detailed results are listed in appendix c.3. ## conclusion we present a simple yet effective approach called lsar to boost language agnosticism for pretrained multilingual encoders. lsar identifies a low-rank subspace residing in a pretrained model that primarily encodes language-specific signals in an unsupervised manner via singular value decomposition. once the subspace is discovered, it can be used to efficiently project away the language identity information. empirical results demonstrate the great effectiveness of lsar on semantic tasks and shed light on its ability to locate syntactic relations between languages.