Bridging the Gap between Language Models and Cross-Lingual Sequence Labeling (2022)

Abstract: Large-scale cross-lingual pre-trained language models (xPLMs) have shown effective in cross-lingual sequence labeling tasks (xSL), such as machine reading comprehension (xMRC) by transferring knowledge from a high-resource language to low-resource languages. Despite the great success, we draw an empirical observation that there is an training objective gap between pre-training and fine-tuning stages: e.g., mask language modeling objective requires local understanding of the masked token and the span-extraction objective requires understanding and reasoning of the global input passage/paragraph and question, leading to the discrepancy between pre-training and xMRC. In this paper, we first design a pre-training task tailored for xSL named Cross-lingual Language Informative Span Masking (CLISM) to eliminate the objective gap in a self-supervised manner. Second, we present ContrAstive-Consistency Regularization (CACR), which utilizes contrastive learning to encourage the consistency between representations of input parallel sequences via unsupervised cross-lingual instance-wise training signals during pre-training. By these means, our methods not only bridge the gap between pretrain-finetune, but also enhance PLMs to better capture the alignment between different languages. Extensive experiments prove that our method achieves clearly superior results on multiple xSL benchmarks with limited pre-training data. Our methods also surpass the previous state-of-the-art methods by a large margin in few-shot data setting, where only a few hundred training examples are available.

## introduction sequence labeling (sl) tasks are common conditions and have considerable impact in natural language processing communities, such as named entity recognition (ner) @xcite , and * work done when interned at microsoft stca. ## related work pre-trained language models during the past few years, plms @xcite @xcite ) dominate natural language understanding and generating areas due to their promising results in various downstream tasks. taking a step forward a more real world, there is a growing body of communities extent plms to multi-lingual language (xplms), and some representative works have been done such as xlm-r (conneau et al., 2019) (short for xlm-roberta), info-xlm @xcite and xlm-align @xcite . almost these efforts trained with token-level pre-training tasks on large multi-lingual corpus. directly utilizing xplms lead high performances in the fine-tuning stage, but when transferred to downstream tasks can result the objective gap between pretrain-finetune, making xplms sub-optimized in downstream tasks especially for xsl @xcite . our proposed clism and cacr not only alleviate the pretrain-finetune discrepancy, but also strong the ability of xplms to capture alignment between cross-lingual representations in sentence-level. sequence labeling sequence labeling tasks are of great significance in web mining, such as named entity recognition @xcite , relation identification @xcite , event extraction @xcite , and even machine reading comprehension @xcite @xcite . however, when pushing the boundary of sl to low-resource languages, xsl can be very challenging, due to limited training data. to tackle this challenge, various efforts have been proposed @xcite @xcite . @xcite and @xcite utilized machine translation to get parallel data as the data argumentation method. but the performance of these methods are restricted to the quality of translation. hence, @xcite proposed to generate low-resource language weakly labeled data from wikipedia articles. similarly, we also collect pre-training parallel data for clism from web sources. we tackle the objective gap between language models and xsl tasks, an challenge overlooked by the previous methods. recent works currently, several studies have explored to bridge the gap between pre-training and fine-tuning, including two main categories: (1) knowledge-driven, which introduces external commonsense knowledge into the pre-training for improving model performance in down-stream tasks @xcite , ( 2 ) the latter is task-oriented, which includes several well designed pre-training or fine-tuning tasks @xcite . however, none of these works target cross-lingual areas, especially for xsl tasks. recently, concurrent with our research, ram et al. (2021) studied aligning the pre-training and fine-tuning stages in the context of question answering via introducing a novel pre-training task. nevertheless, ram et al. (2021) focused on few-shot question answering, and didn't explore the relation between different languages. ## methodology in this section, we first illustrate our two pretraining strategies: clism and cacr in detail. then we introduce how to apply our methods in down-stream tasks to obtain high performances. due to the excellent versatility of our proposed methods, which can build on top of any xplms. hence, we leverage m to represent a series of pre-trained language models. our goal aims at aiding m to eliminate the influence of objective gap between pretrain-finetune, and enhancing the consistency between parallel data representations. ## experiments in this section, we first present the pre-training data collection. then we illustrate experiments on xsl tasks, including specific implementation and detailed results. ## ablation study and analysis in this section, we first discuss the importance of each key component in our model. then we conduct an analysis of alleviating pretrain-finetune discrepancy by ours and independent contribution of the proposed clism at few-shot data seting. we further analyse the relation between pre-trainig data and our methods in the appendix g. ## conclusion in this paper, heading for bridging the discrepancy including the objective and representation gap between pre-training and fine-tuning stages in cross-lingual sequence labeling tasks. we propose two pre-training strategies: cross-lingual language informative span masking (clism) and contrastive-consistency regularization (cacr), reaching surprisingly superior results on various benchmarks even when only a hundred examples available. we further prove that even with less than a hundredth of the data needed to train other xplms, our model still achieves better performances. as an extension of our future work, we will explore how to apply our methods to other nlp tasks. in this component, we further conduct extensive experiments to explore the relation between pretraining data and model performances. as shown in figure 5 , we present the resulting model performances on xmrc datasets with different number of pre-training data: {1m, 2m, 4m, 8m, 12m }. obviously, ours achieve best results given 1 million pre-training data. with the amount of pre-training data increased, the performances drop gradually due to overfitting. the results prove the effectiveness of our proposed pre-training paradigm, that is, our methods could optimize the xplms sufficiently even if with limited pre-training data.