Structural Contrastive Pretraining for Cross-Lingual Comprehension (2023)

Abstract: To present, multilingual language models trained using various pre-training tasks like mask language modeling (MLM) have yielded encouraging results on a wide range of downstream tasks. Despite the promising performances, structural knowledge in cross-lingual corpus is less explored in current works, leading to the semantic misalignment. In this paper, we propose a new pre-training task named Structural Contrast Pretraining (SCP) to align the structural words in a parallel sentence, enhancing the models’ ability to comprehend cross-lingual representations. Concretely, each structural word in source and target languages is regarded as a positive pair in SCP. Since contrastive learning compares positive and negative pairs, an increase in the frequency of negative pairings could enhance the performance of the resulting model. Therefore, we further propose Cross-lingual Momentum Contrast (CL-MoCo) to increase the number of negative pairs by maintaining a large size of the queue. CL-MoCo extends the original Moco approach into cross-lingual training and jointly optimizes the source-to-target language and target-to-source language representations, resulting in a more suitable encoder for cross-lingual transfer. We conduct extensive experiments to validate the proposed approach on three cross-lingual tasks across five datasets such as MLQA, WikiAnn, etc, and results prove the effectiveness of our method.

## introduction following the promising results of the pre-training paradigm in the monolingual natural language domain, the efforts of multilingual pre-trained language models (xplms) @xcite @xcite @xcite have been proposed rapidly. in general, these xplms are always trained on large-scale multilingual corpora using various pretraining language modeling tasks, such as mlm here we present the same sentence in english and german. the words in red and blue box refers to the aligned verb and object words, separately. @xcite , nsp @xcite , clism @xcite , and trtd @xcite . in this manner, xplms acquire robust contextually relevant representations and, as a result, excel at a variety of downstream tasks, like question answering @xcite @xcite and name entity recognition @xcite . for instance, @xcite propose to train xplms with clism and mlm, achieving remarkable performances in multilingual sequence labeling tasks @xcite @xcite . although these pre-training tasks help xplms learn promising multilingual contextualized representations at hierarchical level (i.e., token or sentence-level) @xcite , they don't take structural knowledge into consideration. one obvious limitation of the above approaches is the semantic misalignment between structural words from different languages, leading to a bias in the understanding of the multilingual representations. we showcase the parallel sentences in english and german in figure 1 that are quite different in the syntax structure. the main components of this sentence are "ebydos ag" (subject), "founded" (verb), "subsidiary" (object) and "wroclaw" (entity). unfortunately, as one of the current state-of-the-art xplms: xlm-roberta (xlm-r) @xcite is incapable of capturing the alignment of these crucial words in german, leading to semantic deviation. specifically, xlm-r pays less attention to the corresponding words of "founded" and "subsidiary" in german due to the sentence structure barrier between these two languages. one step further, from the perspective of human behavior, when a language learner reads a sentence in another language, it can help him/her understand this sentence quickly and accurately by pointing out the structural words in a sentence, including subject, verb, object and entities. this effect will be more noticeable when the sentence is lengthy and complex. similarly, by providing the extra clues of aligned crucial/informative words in the parallel sentence, the model can benefit from a closer gap of cross-lingual representations. motivated by the above factors, we design a structural contrastive pretraining (scp) task to enhance xplms' comprehension ability via contrastive learning, bridging the misalignment between structural words in a parallel corpus. considering the facts that subject, verb, object (s-v-o) are the backbone of a sentence and aligned entities in cross-lingual parallel sentences convey coreference and information short-cuts @xcite , in this work, we consider s-v-o and entities as the structural words in a sentence, which are all insightful or crucial. concretely, we divide the parallel corpus into a number of smaller groups. each sub-group has two versions of the same sentence, one in the source language (high resource) and one in the target language (low resource). each structural word in the source and target languages is considered as a positive pair. due to the nature of contrastive learning, wherein comparisons are made between positive and negative pairs, an increase in the number of negative pairings may potentially improve performances of the resulting model @xcite . inspired by momentum contrast in computer vision @xcite , we keep a queue and employ the encoded embeddings from the previous mini-batch to increase the quantity of negative pairs. in this method, momentum contrast employs a pair of fast and slow encoders to encode the source language sentences and target language sentences, separately. and the fast encoder is saved for fine-tuning on down-stream datasets. however, directly applying this approach to cross-lingual pre-training could lead to another problem: as the fast encoder only sees the source language during pre-training, the training becomes insensitive to other target languages. as a consequence, the resulting model may underperform on cross-lingual transfer. to address this issue, we creatively incorporate the original momentum contrast into the cross-lingual setting, naming it cross-lingual momentum contrast (short for cl-moco). specifically, cl-moco utilizes two pairs of fast/slow encoders to jointly optimize source-to-target language and target-tosource language representations, further bridging the cross-lingual gap. in light of the fact that almost all down-stream cross-lingual understanding tasks only need one encoder, the two fast encoders share parameters in our pre-training. based on the above two proposed strategies for building positive and negative pairs in scp, our resulting model xlm-scp can accurately capture the alignment of sentence structures across different languages, improving the performances on crosslingual understanding tasks. as seen in figure 1 (b), ours successfully grasp the correspondence between sentence verbs ("founded"-"gegründet") and objects ("subsidiary"-"ableger") in english and german. we conduct experiments with two different xplms encoders on three multilingual tasks to test the effectiveness of our approach: name entity recognition (ner) @xcite , machine reading comprehension (mrc) @xcite and partof-speech tagging (pos) @xcite . extensive results show our method can improve the baseline performances across 5 datasets in terms of all evaluated metrics. for example, ours initialize from xlm-r improves the baselines from 61.35% to 63.39% on wikiann dataset @xcite . in general, our contributions can be summarized as follows: ## related work multilingual pre-trained language models to date, transformer-based large-scale plms have become the standard in natural language processing and generation @xcite @xcite . currently, more and more communities are working to bring plms into the actual world of various languages (xplms), and several efforts have been proposed such as xlm-roberta @xcite (short for xlm-r), info-xlm @xcite , clism @xcite . these works are pre-trained on a large multilingual corpus with token-level or sentence-level pre-training tasks. despite their promising performances in multiple down-stream tasks, they all don't explicitly consider structural knowledge in the parallel corpus. contrastive learning as a result of its potential to improve upon existing methods for learning effective representations, contrastive learning @xcite has gained popularity in recent years. it works by grouping representations that are semantically close together (positives) in an embedding space and then pushing apart others (negatives) that are not neighbors. contrastive learning objective has been particularly successful in different contexts of natural language processing @xcite . moreover, several efforts @xcite @xcite @xcite @xcite are well-designed for cross-lingual language understanding. for instance, @xcite proposed multi-level contrastive learning towards cross-lingual spoken language understanding. chen et al. (2022) employed contrastive learning to learn noise-invariant representation from multilingual corpora for downstream tasks. different from previous works, we utilize contrastive learning to learn the alignments of the structural words @xcite , leading to a more comprehensive and accurate understanding on the cross-lingual sentence. momentum contrast recently, several works @xcite have explored momentum contrast in natural language understanding tasks, such as sentence representation and passage retrieval. specifically, @xcite propose xmoco to learn a dual-encoder for querypassage matching via two pairs of fast/slow encoders. although we share a similar topic on mo-mentum contrast, our research questions, application areas, and methods differ. xmoco are designed for query-matching tasks while our proposed cl-moco is tailored for cross-lingual representation learning. moreover, @xcite employs two different encoders for query and passage, separately. however, we share parameters of the two fast encoders in our training. at last, we focus on the representation learning of cross-lingual transfer, but they only take monolingual into consideration. recent works recently, several works @xcite @xcite also focus on word alignment for multilingual tasks. for clarity, we list some key differences: all of them align each token in the parallel corpus in an "all-to-all" fashion, but we only consider structural words like s-v-o via contrastive learning. the motivations are: (1) in our pilot analysis and experiments, we have two different settings in the proposed scp: a. training the model with only structural words; b. training the model with all tokens in the sentences. experimentally, we observe that they achieve comparable performances on mrc tasks but the latter achieves slightly worse results on ner tasks. this is due to the fact that aligning some words with no precise meaning like stopwords may have visible side effects on token-level tasks like ner. (2) futhermore, the latter could result in more computation cost than the current method. (3) from a human perspective, structural words are the backbone of each sentence, and a solid grasp of them is sufficient to strengthen the management of the majority of situations. ## methodology in this section, we first illustrate our proposed structural contrastive pretraining (scp) in detail. then we introduce how to incorporate our method with momentum contrast. due to the fact that our proposed methods are flexible and can be built on top of any xplms, we leverage e to represent a series of pre-trained language models, where e could be the e f ast in section 3.2. we aim at enhancing e's ability to capture consistency between parallel structural representations via scp. the overview of our approach is illustrated in figure 2 . ## analysis aside from the high performances achieved by our proposed approaches, we are still concerned about the following questions: q 1 : what are the effects of each key component in our xlm-scp? q 2 : is cl-moco really superior to moco in cross-lingual understanding tasks? q 3 : does the size of the queue in cl-moco affect the performance of our model? q 4 : what are the model performances with different τ in eq.10? (seen in appendix c, figure 5 ) q 5 : within the chosen objects, verbs, objects, and entities in structural words, which part has the biggest effect on our xlm-scp's performance? (seen in appendix c, answer to q 1 : experiments are carried out to confirm the independent contributions of each component in our proposed pre-training scheme. on cross-lingual understanding tasks. we conduct ablation experiments on three tasks across four datasets and show the results in figure 3 . we can find that our proposed cl-moco can achieve better results on all these datasets when compared with the original moco. the results further prove cl-moco has a stronger ability to learn effective cross-lingual representations. answer to q 3 : the main assumption of cl-moco is that the size of negative samples is important in contrastive learning. here we empirically study this assumption in cross-lingual understanding tasks via varying the queue size of keeping negative pairs. as shown in figure 4 , we validate xlm-scp with @xmath0, 10k, 20k, 30k, 40k} on wikiann and mlqa datasets. we can draw the conclusion that the model performs slightly better as the queue size increases initially, especially for xmrc tasks. interestingly, the model achieves best results on wikiann when m is equal to 20k, and its performances slightly decrease when m passes 20k. one possible explanation is that larger size of the queue may introduce some "false negative samples", which could have a more obvious side effect on xner tasks. in light of the fact that the queue size has a negligible effect on training speed and memory use, we have chosen a queue size of 20k for all downstream datasets. ## conclusion in this paper, we observe that misalignment of crucial structural words occurs in the parallel sentences of current xplms. we propose a new pretraining task called structural contrastive pretraining (scp) to alleviate this problem, enabling the model to comprehend the cross-lingual representations more accurately. we further incorporate momentum contrast into cross-lingual pre-training, named cl-moco. in particular, cl-moco employs two sets of fast/slow encoders to jointly learn the source-to-target language and target-to-source language cross-lingual representations. because of this, the resulting model is better for cross-lingual transfer. extensive experiments and analysis across various datasets show the effectiveness and generalizability of our approach. as an extension of our future work, we will apply our method to other natural language understanding tasks and find a proper way to reduce data preprocessing costs.