Enhancing Cross-lingual Natural Language Inference by Prompt-learning from Cross-lingual Templates (2022)

Abstract: Cross-lingual natural language inference (XNLI) is a fundamental task in cross-lingual natural language understanding. Recently this task is commonly addressed by pre-trained cross-lingual language models. Existing methods usually enhance pre-trained language models with additional data, such as annotated parallel corpora. These additional data, however, are rare in practice, especially for low-resource languages. Inspired by recent promising results achieved by prompt-learning, this paper proposes a novel prompt-learning based framework for enhancing XNLI. It reformulates the XNLI problem to a masked language modeling problem by constructing cloze-style questions through cross-lingual templates. To enforce correspondence between different languages, the framework augments a new question for every question using a sampled template in another language and then introduces a consistency loss to make the answer probability distribution obtained from the new question as similar as possible with the corresponding distribution obtained from the original question. Experimental results on two benchmark datasets demonstrate that XNLI models enhanced by our proposed framework significantly outperform original ones under both the full-shot and few-shot cross-lingual transfer settings.

## introduction cross-lingual language understanding (xlu) plays a vital role in multilingual systems. it aims at training a model in a source language which is then applied to other languages. cross-lingual natural language inference (xnli) is a challenge task for evaluating xlu @xcite . natural language inference (nli) aims to determine the inferential relationship between the text of a premise and the text of a hypothesis while xnli upgrades nli to the cross-lingual scenarios. recently, prompt-learning based methods @xcite have shown to achieve promising results for few-shot natural language processing (nlp). these methods reformulate the text classification problem into a masked language modeling problem. in particular, the work @xcite demonstrates that prompt-learning outperforms fine-tuning in few-shot xnli. we argue that the effectiveness of prompt-learning in xnli still needs to be explored by a larger margin. the reasons are two-fold. on one hand, the effectiveness of prompt-learning in xnli under the full-shot setting is still unknown. on the other hand, the way to make the best of question templates is unexplored yet. the work (zhao and schütze, 2021) uses a uniform template in english for all examples in different languages. this way can hardly capture language-specific characteristics in xnli, especially for those languages that are right-to-left written such as arabic and urdu. we naturally expect that language-specific question templates lead to higher performance in xnli. figure 1 illustrates how language-specific question templates are used. the second sub-figure shows the uniform question template used in (zhao and schütze, 2021) to handle an arabic example, where the corresponding example in english is shown in the first sub-figure. the last sub-figure shows the arabic-specific question template used for the same arabic example, which is right-to-left written and conforms to the arabic grammar. in order to introduce language-specific characteristics in question templates while capturing correspondence between different languages, we propose a novel prompt-learning based framework named pct (shot for prompt-learning from crosslingual templates) for xnli. as illustrated in figure 2 , pct first constructs a cloze-style question by filling the template in the source language (namely english), then randomly samples a template in another language (such as chinese) to construct an augmented question, where the augmented question is written in two languages and thus its template is called a cross-lingual template. both the original question and the augmented question are fed into a pre-trained cross-lingual language model to calculate the answer probability distributions for inferential relationships that are represented by predefined tokens mapped from the mask token. to enforce answer consistency for the two questions, i.e., to make the two probability distributions of inferential relationships as similar as possible, the two probability distributions are regularized by the kullback-leibler divergence (kld) loss. the entire model is trained by minimizing the sum of the cross-entropy loss for classification accuracy and the kld loss for answer consistency. we employ pct to enhance pre-trained crosslingual language models xlm-r @xcite and infoxlm @xcite . experimental results on the xnli @xcite benchmark and the paws-x @xcite benchmark show that pct improves the original models by a significant margin under both the full-shot and few-shot cross-lingual transfer settings. main contributions of this work include: ## related work up to date xlu including xnli are widely addressed by pre-trained cross-lingual language models @xcite @xcite . multilingual bert (mbert) @xcite extends the basic pre-trained language model bert by training with multilingual corpora. xlm @xcite enhances mbert by introducing the translation language modeling (tlm) objective. xlm-roberta (xlm-r) @xcite trains xlm with larger corpora and more epochs. cross-lingual language models can further be enhanced by post-training tasks that rely on largescale parallel corpora. unicoder @xcite introduces several post-training tasks to utilize parallel corpora. infoxlm @xcite enhances xlm-r by introducing the cross-lingual contrastive learning task using 42 gb parallel corpora. xlm-align @xcite introduces a denoising word alignment pre-training task using several parallel corpora. these enhancements can hardly be applied to low-resource languages for which parallel corpora are rare. to alleviate the dependence on parallel corpora, some data augmentation strategies have been proposed for xnli. tman @xcite enhances xnli by exploiting adversarial training from translated data. the work @xcite proposes a data augmentation strategy for xnli by generating augmented data from a pre-trained sequence-tosequence model. uxla @xcite improves the performance of xnli by data augmentation and unsupervised sample selection. all these strategies require a large amount of external resources for data augmentation. in contrast, our proposal augments data by only predefined crosslingual templates. recently prompt-learning based methods have shown to achieve promising results in various fewshot nlp tasks. the key of these methods is reformulating the text classification problem into a masked language modeling problem by constructing cloze-style questions. the work @xcite applies prompt-learning to text classification (including nli) with manually defined templates. the work @xcite proposes to search for optimal discrete templates by a gradient based approach. several approaches @xcite @xcite have been proposed to search continuous prompts. the work (zhao and schütze, 2021) compares promptlearning with fine-tuning in few-shot xnli. different from (zhao and schütze, 2021), this work significantly advances prompt-learning in xnli further by introducing a new data augmentation strategy and a new consistency loss for regularization. the effectiveness of prompt-learning is also demonstrated further under both the full-shot and few-shot cross-lingual transfer settings. ## the pct framework the proposed pct framework is illustrated in figure 2. for every training triple (premise, hypothesis, label) in english, pct first constructs a cloze-style question by filling the template in english, then samples a predefined template from another language such as chinese to construct an augmented question. both the original question and the augmented question are fed into a pre-trained cross-lingual model to calculate the answer distributions of the mask token, through the masked language modeling (mlm) layer in the pre-trained cross-lingual model. the entire model is trained by minimizing the cross-entropy loss for classification accuracy and the kullback-leibler divergence (kld) loss for answer consistency. ## experiments to evaluate the effectiveness of the proposed pct framework, we applied pct to enhance several pretrained cross-lingual language models including xlm-r base , xlm-r large and infoxlm large . we call the enhanced models pct-x, where x denotes the original pre-trained cross-lingual model. ## conclusions in this paper we have proposed a prompt-learning based framework named pct for cross-lingual natural language inference. pct enhances pre-trained cross-lingual language models by augmenting data from cross-lingual templates and by introducing the consistency loss to regularize the answer probability distributions. experimental results on largescale benchmarks xnli and paws-x show that pct pushes existing models by a significant absolute gain in accuracy under both the full-shot and few-shot cross-lingual transfer settings. our ablation study and visualization analysis further confirm the contributions of different enhancements introduced by pct. future work will study pct further under the translate-train-all setting with real-world data in different languages. yuan zhang, jason baldridge, and luheng he. 2019. paws: paraphrase adversaries from word scrambling. in naacl-hlt, pages 1298-1308. mengjie zhao and hinrich schütze. 2021. discrete and soft prompting for multilingual models. in emnlp, pages 8547-8555.