Toward the Limitation of Code-Switching in Cross-Lingual Transfer (2022)

Abstract: Multilingual pretrained models have shown strong cross-lingual transfer ability. Some works used code-switching sentences, which consist of tokens from multiple languages, to enhance the cross-lingual representation further, and have shown success in many zero-shot cross-lingual tasks. However, code-switched tokens are likely to cause grammatical incoherence in newly substituted sentences, and negatively affect the performance on token-sensitive tasks, such as Part-of-Speech (POS) tagging and Named-Entity-Recognition (NER). This paper mitigates the limitation of the code-switching method by not only making the token replacement but considering the similarity between the context and the switched tokens so that the newly substituted sentences are grammatically consistent during both training and inference. We conduct experiments on cross-lingual POS and NER over 30+ languages, and demonstrate the effectiveness of our method by outperforming the mBERT by 0.95 and original code-switching method by 1.67 on F1 scores.

## introduction recent studies @xcite @xcite have shown the success of multilingual corpus pre-training for cross-lingual knowledge transfer. some works @xcite further addressed the effectiveness of code-switching on improving the performance of multilingual models on zeroshot cross-lingual tasks @xcite . though code-switching has shown great potential and strong generalization ability on the semantic representation, the newly switched sequence fails to consider the token-level coherence. specifically, a code-switched sentence consists of tokens from various languages, and such words are likely to cause grammatical incoherence, resulting in an inconsistent context space for the newly substituted sentence. also, code-switched tokens are conditioned on the original context, for which, during the optimization phase, the embedding of substituted tokens is greatly affected. however, such updates do not consider the token-level dependency in their own language and will likely cause inconsistent embedding space during inference. therefore, even though code-switching benefits most of the sentence-level cross-lingual tasks, it still limits the performance of token-level prediction. to address such issues, we use an alignment strategy to reduce the gap between original context and code-switched tokens during training. specifically, code-switched tokens are mapped to the original context so that the newly switched sentences are expected to be grammatically coherent and maintain the same contextual space in the same language. meanwhile, our approach enables the substituted tokens not to significantly affect the token-level coherence in their own language and keep a consistent embedding space during inference. we conduct experiments on zero-shot cross-lingual pos and ner and outperforms both the mbert and the original code-switching method by 0.77 and 2.08 on ner and 1.13 and 1.27 on pos, respectively. we also make further analysis regarding the performance on low-resource languages, code-switch contribution, substitution strategy, and token-level coherence. ## approach we conduct cross-lingual tasks in the zero-shot setting, in which only english labeled sentences with code-switching augmentation are used for training, and evaluation is performed in all other languages. ## conclusion this paper introduces an alignment strategy to map the code-switched tokens to original context and solves the grammatically incoherence in the embedding space of code-switching. experimental results on pos and ner along with comprehensive analysis have demonstrated the effectiveness of our approach on the token-level classification. we think this work could further address other cross-lingual tasks and multilingual pretraining.