Isotropic Representation Can Improve Zero-Shot Cross-Lingual Transfer on Multilingual Language Models (2023)

Abstract: With the development of multilingual pre-trained language models (mPLMs), zero-shot cross-lingual transfer shows great potential. To further improve the performance of cross-lingual transfer, many studies have explored representation misalignment caused by morphological differences but neglected the misalignment caused by the anisotropic distribution of contextual representations. In this work, we propose enhanced isotropy and constrained code-switching for zero-shot cross-lingual transfer to alleviate the problem of misalignment caused by the anisotropic representations and maintain syntactic structural knowledge. Extensive experiments on three zero-shot cross-lingual transfer tasks demonstrate that our method gains significant improvements over strong mPLM backbones and further improves the state-of-the-art methods.

## introduction cross-lingual transfer aims to utilize the rich semantics and syntactic knowledge in high-resource source languages to improve the performance of low-resource target languages. benefiting from the "scaling effect", language models pre-trained on hundreds of languages have dominated crosslingual transfer for years owing to their superior performance and generalization capability, which can learn a unified representation space through self-supervised learning @xcite @xcite @xcite @xcite . these powerful multilingual pre-trained language models (mplms) not merely improve task performance with full supervised learning but even the zero-shot cross-lingual transfer @xcite @xcite . the core of cross-lingual transfer is to align representations among different languages @xcite @xcite . existing mplms can align the representation well for myriads of the cross-lingual transfer scenarios but fail to handle these language pairs differed significantly under the zero-shot setting, especially for languages with distinct morphological features @xcite . therefore, many works have explored the key factors affecting the alignment of language representations @xcite @xcite and proposed solutions accordingly @xcite @xcite @xcite . existing methods can be roughly divided into three categories: 1) using parallel corpus @xcite @xcite or bilingual dictionary @xcite to better align contextualized word embedding spaces; 2) utilizing morphological or syntactic features @xcite @xcite to eliminate misalignment; 3) leveraging robust training methods @xcite to tolerate misaligned representations. however, additional parallel corpora are difficult to obtain for many extremely low-resource languages, while annotating morphological or syntactic features requires considerable human effort. though the robust training methods are free from additional supervision signals, they ignore the misalignment caused by the distributional properties of representation. as found by @xcite , the representation distribution of mplms is highly anisotropic, where most words are squeezed into a narrow region in the representation space. as shown in figure 1a , even semantically irrelevant representations can appear in the neighborhood. thus, there are reasons to believe that highly anisotropic representations hurt cross-lingual alignment in the representation space. to alleviate the aforementioned obstacles, we propose an isotropy enhancement strategy that can improve the representation alignment at the semantic level while maintaining the essential syntactic knowledge, which is indispensable for crosslingual transfer @xcite . as an excessively isotropic representation space is risky to pull apart representations that should be aligned, we also introduce a constrained code-switching method to better utilize the readily available bilingual dictionaries. to verify the effectiveness of our proposed method, we launch experiments on three zero-shot cross-lingual tasks, i.e., paraphrase identification, natural language inference, and sentiment classification. experimental results demonstrate that our proposed method significantly improves the performance of zero-shot cross-lingual transfer upon strong mplms. further analytical exploration confirms that our method can alleviate the anisotropy problem of pre-trained representations while preserving syntactic knowledge implicit in the representations as much as possible. ## preliminary in this section, we briefly introduce the anisotropic problem of representation learning and the risk of undermining knowledge of syntactic structures in existing methods of mitigating anisotropy. ## method in this section, we introduce our method for zero-shot cross-lingual transfer, which comprises of isotropy enhancement and constrained codeswitching. figure 3 shows the overview of our proposed method. before elaborating our method in detail, we first present some necessary notations. given the training dataset @xmath0, y i )} n @xmath1, the loss of fine-tuning can be written by: ## conclusion in this paper, we propose a simple but effective method to improve the performance of zeroshot cross-lingual transfer. by introducing the isotropy enhanced fine-tuning and constrained code-switching, our proposed method can induce moderate isotropic representation and align multilingual representation. experimental results on three zero-shot cross-lingual transfer tasks demonstrate the performance superiority of our method over existing methods. extensive analytical experiments further confirm the effectiveness of our method for enhancing isotropic representations and reducing cross-lingual representation discrepancy.