Supervised Contrastive Learning for Cross-lingual Transfer Learning (2022)

Abstract: “Multilingual pre-trained representations are not well-aligned by nature, which harms their performance on cross-lingual tasks. Previous methods propose to post-align the multilingual pretrained representations by multi-view alignment or contrastive learning. However, we argue that both methods are not suitable for the cross-lingual classification objective, and in this paper we propose a simple yet effective method to better align the pre-trained representations. On the basis of cross-lingual data augmentations, we make a minor modification to the canonical contrastive loss, to remove false-negative examples which should not be contrasted. Augmentations with the same class are brought close to the anchor sample, and augmentations with different class are pushed apart. Experiment results on three cross-lingual tasks from XTREME benchmark show our method could improve the transfer performance by a large margin with no additional resource needed. We also provide in-detail analysis and comparison between different post-alignment strategies.”

## introduction cross-lingual transfer learning aims to transfer the learned knowledge from a resource-rich language to a resource-lean language. the main idea of crosss-lingual transfer is to learn a shared language-invariant feature space for both languages, so that a model trained on the source language could be applied to the target language directly. such generalization ability greatly reduces the required annotation efforts, and has urgent demand in real-world applications. recent multilingual pre-trained models, such as xlm-roberta(xlm-r) @xcite , have been demonstrated surprisingly effective in the cross-lingual scenario. by fine-tuning on labeled data in a source language, such models can generalize to other target languages even without any additional training. this has become a de-facto paradigm for cross-lingual language understanding tasks. despite their success in cross-lingual transfer tasks, multilingual pre-training commonly lacks explicit cross-lingual supervision, and the representations for different languages are not inherently aligned. to further improve the transferability of multilingual pre-trained representations, previous works propose different methods for cross-lingual alignment. @xcite and @xcite propose to augment the training set with different views, and align the pre-trained representations of different languages by dragging two views closer. however, simply bringing different views closer would easily lead to representation collapse and performance degradation @xcite . meanwhile, @xcite and @xcite propose to incorporate additional parallel data, and align the pre-trained representations by contrasting positive and negative samples. however, monotonously treating all random samples equally negative is inconsistent with the classification objective. in this work, we propose a simple yet effective method to better post-align the multilingual representations on downstream tasks, which can both avoid representation collapse and meanwhile induce classification bias. with only training data for the source language available, our method performs crosslingual fine-tuning by two steps. 1) firstly, the original training data is augmented with different views, including code-switching, full-translation and partial-translation. all views could provide cross-lingual supervision for post-alignment. 2) given one training sentence as the anchor point, the corresponding augmented view serves as the positive sample, and other augmented views with different labels serve as the negative samples, contrastive learning is performed by pulling positive samples together and pushing apart negative samples. this is called supervised contrastive learning (scl), and can be deemed as a cross-lingual regularizer to be combined with conventional fine-tuning. we perform experiments on two cross-lingual classification tasks, namely xnli (cross-lingual inference) and paws-x (cross-lingual paraphrase identification) @xcite . we compare different alignment methods, and our proposed method outperforms previous methods by a large margin, proving its effectiveness. besides, we also apply our method on the cross-lingual retrieval task of bucc 0 and tatoeba @xcite . we use the data from paws-x as supervision, and fine-tune the pretrained model by contrasting samples with their machine translation. our proposed method again outperforms other methods by a large margin. detailed analysis and discussion are provided to compare different post-alignment methods for pretrained representations, and to prove the necessity of label-supervision when performing cross-lingual contrastive learning. ## approach in this section, we first introduce the three cross-lingual data augmentation methods. based on that, we propose three paradigms to post-align the multilingual representations, and provide theoretical analysis and comparison for them. in this work, we do not want to incorporate any parallel data (which is inaccessible in a lot of scenarios, especially for a resource-lean language that we want to transfer to). therefore, to provide cross-lingual supervision for post-alignment, we propose three data augmentation methods: ## conclusion in this paper, we propose to improve cross-lingual fine-tuning with supervised contrastive learning. cross-lingual supervision is created by augmenting the training set, and different methods to post-align the multilingual pre-trained representation are compared. we propose to incorporate label-information when performing cross-lingual contrastive fine-tuning, and outperforms previous methods by a large margin on four cross-lingual transfer benchmark datasets. canonical cross-entropy has many intrinsic problems, especial when performing transfer learning tasks, and contrastive learning can be a decent supplementary. by alleviating the commonality and differences between different examples, representations are efficiently transferred from one domain or language to another. in the future, we would explore the application of supervised contrastive learning on other transfer learning tasks, including token-level classification, language generation, cross-domain transfer, etc.