A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning (2022)

Abstract: Existing zero-shot cross-lingual transfer methods rely on parallel corpora or bilingual dictionaries, which are expensive and impractical for low-resource languages. To disengage from these dependencies, researchers have explored training multilingual models on English-only resources and transferring them to low-resource languages. However, its effect is limited by the gap between embedding clusters of different languages. To address this issue, we propose Embedding-Push, Attention-Pull, and Robust targets to transfer English embeddings to virtual multilingual embeddings without semantic loss, thereby improving cross-lingual transferability. Experimental results on mBERT and XLM-R demonstrate that our method significantly outperforms previous works on the zero-shot cross-lingual text classification task and can obtain a better multilingual alignment.

## introduction in recent years, advances in multilingual models such as mbert @xcite , xlm (conneau and lample, 2019), xlm-r @xcite , etc., after being fine-tuned with annotated data, have enabled significant improvements in many cross-lingual tasks. however, due to the lack of annotated data, some tasks in low-resource languages have not enjoyed this technological advancement. to solve this issue, the academic and industrial community began to focus on zeroshot cross-lingual transfer learning @xcite , which aims to fine-tune multilingual models with annotated data in highresource languages and obtain a nice performance in low-resource language tasks. some works aligned word embeddings between high-and low-resource languages through additional parallel sentence pairs @xcite processing language natural 处理 语言 自然 próiseáil teanga nádúrtha english synonyms for "natural" english synonyms for "language" english synonyms for "processing" (b) the relative positions of "nature", "language" and "processing" are similar in english, chinese and irish @xcite . (c) using synonym augmentation to train a robust region covering words in other languages. (d) we align different languages and construct a suitable robust region by pushing the embeddings away and pulling the relative distance among words. 2019; @xcite @xcite or bilingual dictionaries @xcite @xcite , so that highresource fine-tuned models can be transferred to low-resource languages. although this approach has achieved excellent results in many languages, parallel corpora and bilingual dictionaries are still prohibitively expensive, rendering it impracticable in some minority languages. to disengage from the dependence on parallel corpora or bilingual dictionaries @xcite , some studies have found that syntactic features in high-resource languages can improve zero-shot cross-lingual transfer learning @xcite @xcite . @xcite found that the embeddings of different languages are clustered according to their language families, as shown in figure 1a and 1b, which demonstrated that different languages are not aligned perfectly in mbert @xcite . @xcite tried adversarial training and randomized smoothing with english synonym augmentation to build robust regions for embeddings in the multilingual models, as illustrated in figure 1c . in this way, models can output similar predictions for different language embeddings in the same robust region even they are not well aligned. however, the transferability of english synonym augmentation is limited because its robust region remains close to the english cluster, as shown in figure 1c . in this work, we select english as a highresource language and follow the studies that do not require additional parallel corpora or bilingual dictionaries to improve cross-lingual transfer learning performance with minimal cost. for this purpose, three strategies are proposed to enlarge the robust region of english embeddings. the first strategy is called embedding-push, which pushes the embedding of english to other language clusters. the second is attention-pull, which constrains the relative position of the word embeddings to prevent the meaning from straying. the last strategy, named robust target, introduces a virtual multilingual embedding (vme) to help the model build a suitable robust region, as shown in figure 1d . experimental results on mbert and xlm-r demonstrate that our method effectively improves the zero-shot cross-lingual transfer on classification tasks and outperforms a series of previous works. in addition, case studies show that our method improves the model through multilingual word alignment. compared with existing works, our method has the following advantages. first, our method only needs english resources, which is suitable for low-resource languages. second, our method can induce alignments in many languages without specifying the target language. finally, our method is simple to implement and achieves effective experimental results. our code is publicly available 1 . ## method given an english training batch b, for a specific @xmath0, x 2 , x 3 ), we first follow @xcite to generate an augmented example x @xmath1, x a 2 , x a 3 ) by randomly replacing x i with x a i from the pre-defined english synonym set @xcite . then, we introduce three objective functions to get the virtual multilingual embedding (vme) that provides a suitable robust region for zero-shot cross-lingual classification task as shown in figure 2 . we describe the details in the following subsections. ## analysis 4.1 case study to study the effects of vme, we do the t-sne visualization for the word embeddings of parallel sentences, as shown in figure 3 . compared with the rs-da, our fine-tuned model aligns better across languages, and words are closer to their translations, leading to correct predictions. this observation shows that the vme can effectively help cross-lingual word alignment and improve the performance of the model. we choose arabic for the case study because it can represent a class of languages far apart from english. ## conclusion to get rid of the dependence on parallel corpora, enable cross-lingual transfer to low-resource languages, we propose embedding-push, attention-pull, and robust targets to combat the influence of language clusters in multilingual models. experimental results demonstrate that our method outperforms previous works and obtains better-aligned embeddings when trained with only english.