A Simple and Effective Method To Eliminate the Self Language Bias in Multilingual Representations (2021)

Abstract: Language agnostic and semantic-language information isolation is an emerging research direction for multilingual representations models. We explore this problem from a novel angle of geometric algebra and semantic space. A simple but highly effective method “Language Information Removal (LIR)” factors out language identity information from semantic related components in multilingual representations pre-trained on multi-monolingual data. A post-training and model-agnostic method, LIR only uses simple linear operations, e.g. matrix factorization and orthogonal projection. LIR reveals that for weak-alignment multilingual systems, the principal components of semantic spaces primarily encodes language identity information. We first evaluate the LIR on a cross-lingual question answer retrieval task (LAReQA), which requires the strong alignment for the multilingual embedding space. Experiment shows that LIR is highly effectively on this task, yielding almost 100% relative improvement in MAP for weak-alignment models. We then evaluate the LIR on Amazon Reviews and XEVAL dataset, with the observation that removing language information is able to improve the cross-lingual transfer performance.

## introduction recently, large-scale language modeling has expanded from english to the multilingual setting (i.a., @xcite ; @xcite ; @xcite ). although these models are trained with language modeling objectives on monolingual data, i.e. without crosslingual information, these multilingual systems exhibit impressive zero-shot cross-lingual ability @xcite . these observations raise many questions and provide insight for multilingual representations learning. first, how is the language * work done during internship at google research. one of the first attempts in this research area, @xcite , proposed two concepts for language agnostic models: weak alignment v.s. strong alignment. for multilingual systems with weak alignment, for any item in language l 1 , the nearest neighbor in language l 2 is the most semantically "relevant" item. in the case of strong alignment, for any representation, all semantically relevant items are closer than all irrelevant items, regardless of their language. @xcite show sentence representations from the same language tend to cluster in weak-alignment system. similar phenomena can be observed on other pre-trained multilingual models like mbert, xlm-r @xcite and cmlm @xcite . @xcite provides carefully-designed training strategies for retrieval-like model to mitigate this issue in order to obtain language agnostic multilingual systems. we systematically explore a simple post-training method we refer to as language information removal (lir), to effectively facilitate the language agnosticism in multilingual embedding systems. first introduced in @xcite to reduce same language bias for retrieval tasks, the method ## language information removal for self language bias elimination in this section we describe language information removal (lir) to address the self language bias in multilingual embeddings @xcite . the first step is to extract the language identity information for each language space. given a multilingual embedding system e, e.g. multilingual bert, and a collection of multilingual texts {t i l }, where t i l denotes the ith phrase in the collection for the language l. we construct a language matrix m @xmath0, where n denotes the number of sentences in language l and d denotes the dimension of the representation. the row i of m l is the representation of t i l computed by e. second, we extract language identification components for each language. one observation in multilingual systems is that representations from the same language tend to cluster together (w.r.t representations in other languages), even though these representations have different semantic meanings. this phenomenon is also known as "weak alignment" @xcite . the mathematical explanation for this clustering phenomenon is that representations in the same language have shared vector space components. we propose that these shared components essentially represent the language identification information. removing these language components should leave semantic-related information in the representations. to remove the shared components, or the language identification from the representations, we leverage singular value decomposition (svd) which identifies the principal directions of a space. we use svd instead of pca since svd is more stable numerically (e.g. for läuchli matrix). specifically, the svd of a language matrix is m @xmath1, where the columns of v @xmath2. we take first r columns of v l as the language identification components, denoted as c @xmath3. different values of r are explored in the next experiments section. language identification components are removed as follows. given a multilingual representation e l in language l, we subtract the projection of e l onto c l from e l , i.e. in the following experiments, sentences used for extracting principle components are sampled from wiki-40b @xcite . we use 10,000 sentences per language. we notice performance initially increases as more sentences are used but then is almost unchanged after @xmath4, 000. we tried different samplings of {t i l } and text resources other than wiki-40b, e.g., tatoeba @xcite . the minimal differences in performance suggest language components are stable over different domains. ## related work & our novelty different training methods have been proposed to obtain language agnostic representations. laser @xcite leverages translation pairs and bilstm encoder for multilingual sen- the method is post-processing and requires bilingual word translation pairs to determine the projection vectors. in contrast, lir is post-training and does not require labeled data. @xcite build semantically specialized cross-lingual vector spaces. like cca, their methods requires the additional training to adjust the original embeddings using supervised data: cross-lingual synonyms and antonyms. @xcite propose that the language-specific information of mbert is the centroid of each language space (the mean of embeddings). @xcite propose several training techniques to obtain language-agnostic representations, including segmenting orthographic tokens in training data and aligning monolingual spaces by training. in contrast, lir is post-training and model-agnostic. critically, this means lir can be conveniently and easily applied to any trained multilingual systems without further training. previous explorations on principal components of the semantic space for sentence embeddings include @xcite and @xcite , whereby principal component removal is investigated for monolingual models and the evaluation is only conducted on semantic similarity benchmarks. in contrast, our work investigates the multilingual case and the evaluation is more diverse, e.g. cross-lingual transfer learning. @xcite explore removing top components from english representations. however, it was unclear prior to our work what purpose is served by removing principal components within multilingual and cross-lingual settings. we demonstrate these principal components represent language information for weak-alignment multilingual models. compared with @xcite , the novelty of this work is two-fold. first, it is unclear in @xcite whether the assumption (i.e. principal components contain language information) holds true for both weak and strong-alignment multilingual models. in this work we clearly show that it is is valid for weak-alignment models (section 3.1). however, for strong-alignment systems, the assumption is not quite true ( ## conclusion in this paper, we investigate the self-language bias in multilingual systems. we explore a simple method "language identity removal (lir)". this method identifies and removes the language information in multilingual semantic space by singular value decomposition and orthogonal projection. although as a simple and linear-algebra-only method, lir is highly effective in several downstream tasks, including zero-shot transfer learning, sentiment analysis, etc. especially for crosslingual retrieval, introducing lir increases the performance of weak-alignment multilingual systems by almost 100% relatively in map.