Enhancing Cross-lingual Prompting with Dual Prompt Augmentation (2023)

Abstract: Prompting shows promising results in few-shot scenarios. However, its strength for multilingual/cross-lingual problems has not been fully exploited. hao and Schütze (2021) made initial explorations in this direction by presenting that cross-lingual prompting outperforms cross-lingual finetuning. In this paper, we conduct an empirical exploration on the effect of each component in cross-lingual prompting and derive Universal Prompting, which helps alleviate the discrepancies between source-language training and target-language inference. Based on this, we propose DPA, a dual prompt augmentation framework, aiming at relieving the data scarcity issue in few-shot cross-lingual prompting. Notably, for XNLI, our method achieves 46.54% with only 16 English training examples per class, significantly better than 34.99% of fine-tuning. Our code is available at https://github.com/DAMO-NLP-SG/DPA .

## introduction although adapting pre-trained language models (plms) @xcite to downstream nlp tasks via finetuning is the de facto mainstream paradigm under fully supervised settings @xcite , prompting 1 @xcite @xcite has demonstrated its superiority over finetuning in low-resource scenarios. typically, prompting reformulates the classification task as a language modeling problem over manuallydesigned natural language prompts. despite the effectiveness of prompting on english tasks, its potential for cross-lingual problems, which assume the availability of the training data in high-resource languages (e.g., english) only, is still under-explored. @xcite is the pioneering work to apply prompting to crosslingual nlp. however, their major efforts are spent on comparing different training strategies for crosslingual prompting such as discrete prompting and soft prompting. they do not fully investigate the design choice of key components in prompting, i.e., prompt template and verbalizer. to provide a practical guide for designing cross-lingual prompting, we first conduct an empirical analysis to explore the effects of each prompting component on the performance of crosslingual transfer. our preliminary study shows that template-free prompting combined with englishonly inference, dubbed as language-agnostic "universal prompting" (up) in this paper, generally performs well across different few-shot settings. intuitively, up avoids the discrepancies between the source-language training and the target-language inference, which intrinsically better fits crosslingual tasks. the derived up is a concise solution with reasonable performance but does not take advantage of other available resources in the context of multilingual problems, e.g., the translation of verbalizers in target languages. motivated by this fact, we propose a dual prompt augmentation (dpa) framework to alleviate the data scarcity issue in few-shot scenarios. firstly, we introduce multilingual verbalizers as answer augmentation for prompting, where the translated label tokens are treated as additional target-language supervision. secondly, we propose prompt mixup as prompt input augmentation, which mixes the prompt representations in each batch. intuitively, given two prompt representations on real data, we can generate a virtual representation based on their interpolation, which encodes the semantics in between. our dpa framework is not task-dependent and does not require either external unlabeled data @xcite or massive text manipulation efforts @xcite compared with other data augmentation approaches. in summary, our contributions are as follows: ## language-agnostic universal prompting in this section, we first empirically investigate the importance of essential elements, i.e., template and verbalizer design, in cross-lingual prompting @xcite . based on our investigation, we derive a more competitive baseline called universal prompting. it is language-agnostic because it does not make assumptions about the input language in template design, and the verbalizer during training is taken for all other languages. note that, since soft prompting (sp) and mixed prompting (mp) rely on an external bidirectional lstm @xcite to create soft prompts and do not outperform discrete prompting (dp) significantly, we mainly discuss dp in this work for a clear comparison. as illustrated in to alleviate such discrepancies, we consider three possible variants. specifically, these three variants are derived by avoiding translation on the template and verbalizer tokens or removing the template words, see we follow the experimental setup (refer to section 4 for details) in @xcite to evaluate the impact of the above designs foot_0 . in note that the idea of removing template words in up is distinct to "null prompt" @xcite from the perspective of motivation. "null prompt" is proposed to simplify the manual prompt design on monolingual tasks. compared with "null prompt", the primary goal of up is to alleviate the source-target discrepancies in cross-lingual transfer. moreover, besides removing template words, our up also involves the design choice for targetlanguage inference (w/o verbalizer trans-lation), which proves to be a larger contribution according to the empirical results shown in ## dual prompt augmentation in prompting, the mask token is directly used for making predictions. in this section, we formalize a dual prompt augmentation (dpa) framework based on this crucial element of prompting. ## conclusion in this paper, we first derive language-agnostic universal prompting, a concise but competitive baseline for cross-lingual prompting. the proposed dpa framework can further enhance cross-lingual prompting as shown on two sentence-pair classification tasks. in the future, we will consider verifying the effectiveness of prompting and the dpa framework in cross-lingual sequence tagging or question-answering tasks @xcite . our work mainly focuses on cross-lingual sentencepair classification tasks. while it is directly applicable to single-sentence classification tasks @xcite but may require additional efforts to adapt our dpa framework to more complex cross-lingual tasks such as sequence tagging @xcite @xcite or question answering @xcite . another limitation is that the proposed multilingual verbalizer in the dpa framework requires an external machine translator to produce the translated verbalizers. finally, we limit the language set of the multilingual verbalizer to the set of target languages in a multilingual dataset. extending this language set might give us greater improvement for cross-lingual tasks.