Enhancing Cross-lingual Natural Language Inference by Soft Prompting with Multilingual Verbalizer (2023)

Abstract: Cross-lingual natural language inference is a fundamental problem in cross-lingual language understanding. Many recent works have used prompt learning to address the lack of annotated parallel corpora in XNLI.However, these methods adopt discrete prompting by simply translating the templates to the target language and need external expert knowledge to design the templates. Besides, discrete prompts of human-designed template words are not trainable vectors and can not be migrated to target languages in the inference stage flexibly. In this paper, we propose a novel Soft prompt learning framework with the Multilingual Verbalizer (SoftMV) for XNLI. SoftMV first constructs cloze-style question with soft prompts for the input sample. Then we leverage bilingual dictionaries to generate an augmented multilingual question for the original question. SoftMV adopts a multilingual verbalizer to align the representations of original and augmented multilingual questions into a unified semantic space with consistency regularization. Experimental results on XNLI demonstrate that SoftMV can achieve state-of-the-art performance and significantly outperform the previous methods under the few-shot and full-shot cross-lingual transfer settings.

## introduction multilingual nlp systems have gained more attention due to the increasing demand for multilingual services. cross-lingual language understanding (xlu) plays a crucial role in multilingual systems, in which cross-lingual natural language inference (xnli) is a fundamental and challenging task @xcite @xcite . nli is a fundamental problem in nlu that could help with tasks like semantic parsing @xcite , and relation extraction @xcite @xcite . in xnli settings, the model is trained on the source language with annotated data to reason the relationship between a pair of sentences (namely premise and hypothesis) and evaluated on the target language without parallel corpora. type prompt templates dp premise. question: hypothesis? answer: <mask>. sp premise. hypothesis? <v 1 >...<v @xmath0. mp premise. question: hypothesis? <v 1 >...<v @xmath1: <mask>. pre-trained multilingual language models, such as mbert @xcite , xlm @xcite , and xlm-r @xcite , have demonstrated promising performance in cross-lingual transfer learning. these language models learn a shared multilingual embedding space to represent words in parallel sentences. however, these models are trained on a large number of parallel corpora, which are not available in many low-resource languages. the major challenge of xnli is the lack of annotated data for low-resource languages. to address this problem, some works explored using prompt learning @xcite @xcite when adapting pre-trained language models to downstream tasks in cross-lingual scenarios. prompt learning reformulates the text classification problem into a masked language modeling (mlm) problem by constructing cloze-style questions with a special @xmath3. the model is trained to predict the masked word in the cloze-style questions. as shown in in this paper, we propose a novel soft prompt learning framework with the multilingual verbalizer (softmv) for xnli. first, we construct cloze-style questions for the input samples with soft prompts which consist of trainable vectors. second, we apply the code-switched substitution strategy @xcite to generate multilingual questions which can be regarded as cross-lingual views for the english questions. compared with discrete prompts, soft prompts perform prompting directly in the embedding space of the model and can be easily adapted to any language without human-designed templates. both the original and augmented questions are fed into a pre-trained cross-lingual base model. the classification probability distribution is calculated by predicting the masked token with the multilingual verbalizer to reduce the gap between different languages. finally, the two probability distributions are regularized by the kullback-leibler divergence (kld) loss @xcite to align the representations of original and augmented multilingual questions into the same space. the entire model is trained with a combined objective of the cross-entropy term for classification accuracy and the kld term for representation consistency. the well-trained soft prompt vectors will be frozen in the inference stage. experimental results on the xnli benchmark show that softmv outperforms the baseline models by a significant margin under both the few-shot and full-shot settings. our contributions can be summarized as follows: ## related work early methods for cross-lingual natural language inference are mainly neural networks, such as @xcite and @xcite . which encode sentences from different languages into the same embedding space via parallel corpora @xcite . in recent years, large pre-trained cross-lingual language models have demonstrated promising performance. devlin et al. (2019) extend the basic language model bert to multilingual scenarios by pre-trained with multilingual corpora. @xcite propose a cross-lingual language model (xlm) which enhances bert with the translation language modeling (tlm) objective. xlm-r @xcite is an improvement of xlm by training with more languages and more epochs. although these methods do not rely on parallel corpora, they still have limitations because fine-tuning needs annotation efforts which are prohibitively expensive for low-resource languages. to tackle this problem, some data augmentation methods have been proposed for xnli. @xcite propose to augment mbert with universal language syntax using an auxiliary objective for cross-lingual transfer. @xcite adopt reorder augmentation and semantic augmentation to synthesize controllable and much less noisy data for xnli. @xcite improve cross-lingual generalization by unsupervised sample selection and data augmentation from the unlabeled training examples in the target language. @xcite propose a cross-lingual finetuning method to better utilize four types of data augmentations based on consistency regularization. however, these methods do not perform well under the few-shot settings. recently, prompt learning @xcite @xcite @xcite @xcite has shown promising results in many nlp tasks under the few-shot setting. the key idea of prompt learning for xnli is reformulating the text classification problem into a masked language modeling problem by constructing cloze-style questions. @xcite propose a novel promptbased transfer learning approach, which first learns a prompt on one or more source tasks and then uses it to initialize the prompt for a target task. @xcite adopt separate soft prompts to learn embeddings enriched with domain knowledge. @xcite explore discrete prompt learning to nli with manually defined templates. @xcite demonstrate that prompt learning outperforms fine-tuning for fewshot xnli by simply applying soft, discrete, and mixed prompting with a uniform template in english. @xcite proposed a discrete prompt learning framework that constructs an augmented sample by randomly sampling a template in another language. however, discrete prompts of humandesigned template words require extensive external expert knowledge and are not flexible enough to adapt to different languages. in our work, we adopt trainable soft prompts to capture correspondence between different languages by comparing the augmented multilingual and original questions. ## framework the proposed softmv framework is illustrated in figure 1 . the training process of softmv is formalized in algorithm 1. for every training triple (premise, hypothesis, label) in english, softmv first constructs a cloze-style question with soft prompts initialized from the vocabulary. then, we apply the code-switched substitution strategy to generate multilingual questions which can be regarded as cross-lingual views for the english questions. both the original and augmented questions are fed into a pre-trained cross-lingual model to calculate the answer distributions of the mask token with a multilingual verbalizer. softmv is trained by minimizing the cross-entropy loss for classification accuracy and the kullback-leibler divergence (kld) loss for representation consistency. finally, the well-trained soft prompt vectors are frozen in the inference stage. ## conclusion in this paper, we propose a novel soft prompt learning framework with a multilingual verbalizer (softmv) for xnli. softmv applies the codeswitched substitution strategy to generate multilingual questions for original questions constructed with soft prompts. we adopt the multilingual verbalizer to align the representations of original and augmented samples into the same semantic space with consistency regularization. experimental results on xnli demonstrate that softmv significantly outperforms the previous methods under the few-shot and full-shot cross-lingual transfer settings. the detailed analysis further confirms the effectiveness of each component in softmv.