Discrete and Soft Prompting for Multilingual Models (2021)

Abstract: It has been shown for English that discrete and soft prompting perform strongly in few-shot learning with pretrained language models (PLMs). In this paper, we show that discrete and soft prompting perform better than finetuning in multilingual cases: Crosslingual transfer and in-language training of multilingual natural language inference. For example, with 48 English training examples, finetuning obtains 33.74% accuracy in crosslingual transfer, barely surpassing the majority baseline (33.33%). In contrast, discrete and soft prompting outperform finetuning, achieving 36.43% and 38.79%. We also demonstrate good performance of prompting with training data in multiple languages other than English.

## introduction prompting strongly outperforms finetuning @xcite when adapting pretrained language models (plms; @xcite ; @xcite ) to downstream tasks in the low-resource regime @xcite @xcite @xcite , i.e., few-shot learning, a more realistic scenario than having tens of thousands of annotations, even for english @xcite @xcite . in contrast to finetuning, which learns discriminative classifiers for tasks like natural language inference (nli; @xcite ; @xcite ), prompting reformulates the classification task to generative text-to-text @xcite or cloze-style @xcite queries which are given to a plm to answer. for example, the nli task of assigning premise "they whinnied, eyes wide" and hypothesis "their eyes were open wide" to class "entailment" can be reformulated as: they whinnied, eyes wide . question: their eyes were open wide ? answer: . the plm is requested to fill in, for the blank ( ), the word "yes", which is mapped to "entailment". prompting makes a human description of the task available in learning. also, "filling in the blank" is well aligned with the pretraining objective (masked/autoregressive language modelling @xcite @xcite ), likely to deliver better performance in few-shot learning @xcite . in this paper, we investigate the effectiveness of prompting in multilingual tasks, which -despite the success of prompting in english -is largely unexplored. we address two main research questions: (rq1) does the strong few-shot performance of prompting transfer to other languages from english? (rq2) as the cost of few-shot non-english annotations is affordable @xcite @xcite , can we directly prompt plms in languages other than english or do we have to go through the (generally best resourced) intermediary of english? in this work, we systematically compare two popular prompting methods -discrete and soft prompting -with finetuning in the few-shot multilingual nli task and show that prompting is superior: (i) the strong few-shot learning performance of prompting transfers to other languages from english: it outperforms finetuning in crosslingual transfer (rq1; §5.1). (ii) directly querying the multilingual plm with few-shot non-english prompts achieves competitive performance, without relying on crosslingual transfer from english (rq2; §5.2). ## related work gpt3 @xcite succeeds in few-shot nlu tasks with "in-context learning": a natural language prompt describing the nlu task is prepended to an input example; gpt3 is then capable of making accurate predictions without updating its parameters. however, the number of parameters in gpt3 is prohibitively large (175b). integrating gradient descent into prompting, smaller (w.r.t. gpt3) plms also achieve good fewshot performance. like gpt3, discrete prompting uses natural language to describe nlu tasks. @xcite , @xcite , le scao and rush (2021) use human-designed prompts. @xcite leverage t5 @xcite to generate prompts. @xcite use extra training data to search tokens for constructing the prompts. discrete prompting naturally inherits interpretability from the task descriptions. soft prompting relaxes the constraint that a prompt needs to be composed of discrete tokens. instead, it learns the prompt in the continuous space with sgd. @xcite and @xcite learn soft prompts eliciting more knowledge @xcite from plms than discrete prompts. similar to soft prompting but with the plm being frozen, @xcite propose prefix-tuning to encourage plms to solve generation tasks with high parameter-efficiency @xcite . @xcite demonstrate that soft prompting benefits from scaling up the number of plm parameters. @xcite show that gpt @xcite can solve nlu tasks @xcite with soft prompting. all of this work focuses on english. we show that discrete and soft prompting perform better than finetuning in few-shot crosslingual natural language inference (xnli; @xcite ) with multilingual plms (xlm-roberta; @xcite ). we conduct experiments on nli because it is one of the most representative and challenging nlu tasks @xcite , and has been commonly used in prior work on prompting. ## dataset and setup dataset. we conduct our experiments on natural language inference datasets mnli and xnli @xcite . mnli provides multi-genre english nli sentence pairs. xnli provides development and test splits of human-translated parallel nli sentence pairs in 15 languages 2 and the machine-translated mnli training sets in 14 languages. for constructing the few-shot training set, we randomly sample without replacement @xmath0, 2, 4, 8, 16, 32, 64, 128, 256} shots per class from the en mnli training split. then we retrieve translations of this en training set from xnli to create the few-shot training sets in the other languages. to simulate a realistic low-resource regime @xcite , we use fewshot development sets. for en, we sample the same number of shots (as training) from the xnli development split. as a result, a 2-shot experiment uses 2 training and 2 development shots per class. for other languages, we retrieve the translations of the english development set from xnli. following @xcite , we report accuracy on xnli test. setup. we conduct all experiments using the pretrained xlm-roberta-base model @xcite containing 270m parameters trained on 2.5 tb commoncrawl data in 100 languages. we use pytorch @xcite and the hugprompt. specialized bilingual dictionaries can also be used. 2 the languages are english (en), french (fr), spanish (es), german (de), greek (el), bulgarian (bg), russian (ru), turkish (tr), arabic (ar), vietnamese (vi), thai (th), chinese (zh), hindi (hi), swahili (sw), and urdu (ur). gingface framework @xcite . 3 we use batch size 32 for finetuning and 24 for prompting methods due to resource limitations. following le scao and rush (2021), we use learning rate 1e-5 for both finetuning and prompting. following the suggestions of @xcite , @xcite , we train the model with a large number of epochs (50) and select the checkpoint that performs best on the development set. we repeat each experiment 5 times with different random seeds ({1, 2, 3, 4, 5}) and report mean and variance. appendix §a shows our reproducibility checklist. ## conclusion we showed that prompting performs better than finetuning in few-shot crosslingual transfer and inlanguage training of multilingual natural language inference. we hope our results will encourage more research about prompting multilingual tasks and models. future work may explore using text-to-text models like t5 @xcite or other autoregressive plms @xcite . investigating plms containing even more parameters is also promising as shown by @xcite .