E ntity CS : Improving Zero-Shot Cross-lingual Transfer with Entity-Centric Code Switching (2022)

Abstract: Accurate alignment between languages is fundamental for improving cross-lingual pre-trained language models (XLMs). Motivated by the natural phenomenon of code-switching (CS) in multilingual speakers, CS has been used as an effective data augmentation method that offers language alignment at word- or phrase-level, in contrast to sentence-level via parallel instances. Existing approaches either use dictionaries or parallel sentences with word-alignment to generate CS data by randomly switching words in a sentence. However, such methods can be suboptimal as dictionaries disregard semantics, and syntax might become invalid after random word switching. In this work, we propose EntityCS, a method that focuses on Entity-level Code-Switching to capture fine-grained cross-lingual semantics without corrupting syntax. We use Wikidata and the English Wikipedia to construct an entity-centric CS corpus by switching entities to their counterparts in other languages. We further propose entity-oriented masking strategies during intermediate model training on the EntityCS corpus for improving entity prediction. Evaluation of the trained models on four entity-centric downstream tasks shows consistent improvements over the baseline with a notable increase of 10% in Fact Retrieval. We release the corpus and models to assist research on code-switching and enriching XLMs with external knowledge.

## introduction cross-lingual pre-trained language models (xlms) such as mbert @xcite and xlm-r @xcite , have achieved state-of-the-art zero-shot cross-lingual code switching (cs) is a phenomenon when multilingual speakers alternate words between languages when they speak, which has been studied for many years @xcite @xcite . code-switched sentences consist of words or phrases in different languages, therefore they capture semantics of finergrained cross-lingual expressions compared to parallel sentences, and have been used for multilingual intermediate training @xcite and finetuning @xcite . nevertheless, since manually creating large-scale cs datasets is costly and only few natural cs texts exist @xcite @xcite , research has turned to automatic cs data generation. some of those approaches generate cs data via dictionaries, usually ignoring ambiguity @xcite . others, require parallel data and an alignment method to match words or phrases between languages @xcite . in both cases, what is switched is chosen randomly, potentially resulting in syntactically odd sentences or switching to words with little semantic content (e.g. conjunctions). on the contrary, entities contain external knowledge and do not alter sentence syntax if replaced with other entities, which mitigates the need for any parallel data or word alignment tools. motivated by this, we propose entitycs, a code-switching method that focuses on entities, as illustrated in figure 1 . resources such as wikipedia and wikidata offer rich cross-lingual entity-level information, which have been proven beneficial in xlms pre-training @xcite @xcite . we use such resources to generate an entity-based cs corpus for intermediate training of xlms. entities in wikilinks 2 are switched to their counterparts in other languages retrieved from the wikidata knowledge base (kb), thus alleviating ambiguity. using the entitycs corpus, we propose a series of masking strategies that focus on enhancing entity prediction (ep) for better cross-lingual entity representations. we evaluate the models on entity-centric downstream tasks including named entity recognition (ner), fact retrieval, slot filling (sf) and word sense disambiguation (wsd). extensive experiments demonstrate that our models outperform the baseline on zero-shot cross-lingual transfer, with +2.8% improvement on ner, surpassing the prior best result that uses large amounts of parallel data, +10.0% on fact retrieval, +2.4% on slot filling, and +1.3% on wsd. the main contributions of this work include: a) construction of an entity-level cs corpus, en-titycs, based solely on the english wikipedia and wikidata, mitigating the need for parallel data, word-alignment methods or dictionaries; b) a series of intermediate training objectives, focusing on entity prediction; c) improvement of zero-shot ## methodology we introduce the details of the entitycs corpus construction, as well as different entity-oriented masking strategies used in our experiments. ## experimental setup after preparing the entitycs corpus, we further train an xlm with wep, pep, mlm and the joint objectives. we use the sampling strategy proposed by @xcite , where high resource languages are down-sampled and low resource languages get sampled more frequently. since recent studies on pre-trained language encoders have shown that semantic features are highlighted in higher layers @xcite , we only train the embedding layer and the last two layers of the model foot_4 (similarly to @xcite ). we randomly choose 100 sentences from each language to serve as a validation set, on which we measure the perplexity every 10k training steps. details of parameters used for intermediate training can be found in appendix c. ## main results results are reported in ## analysis we conduct further analysis on different masking strategies, their impact across languages and training steps when performing intermediate training on the entitycs corpus. we primarily focus on wikiann, as it contains the largest amount of languages from the datasets we evaluate on. ## conclusions in this work, we improve zero-shot transfer on entity-oriented tasks via entity-level codeswitching. we make use of the english wikipedia and the wikidata kb to construct an entitycs corpus by replacing entities in wikilinks with their counterparts in other languages. we further propose entity-oriented training objectives to improve entity predictions. evaluation of the models on five datasets reveals consistent improvements on ner, fact retrieval and wsd over the baseline and prior work that uses large amounts of parallel data, as well as competitive results on slot filling. interestingly, we found that replacing masking candidates with random subwords in the conventional masking strategy is harmful for entity prediction, while different masking strategies are optimal for different downstream tasks. specifically, whole entity prediction performs best when emphasis is given on single-token factual knowledge. on the other hand, entity typing and multi-token factual retrieval benefit from partial entity prediction, i.e. when the model is given incomplete information on entities and is required to fill-in the gaps. concurrently jointly predicting non-entity and entity subwords can improve tasks where the entire input context is important for prediction. in addition, our method is particularly beneficial for languages with non-latin script. our corpus construction process is generic and can be scaled to many more languages, while the proposed masking strategies can be used with any existing language model. future work will focus on code-switching beyond entities, such as verbs and phrases, as well as using other sources beyond wikipedia and wikidata.