Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt (2022)

Abstract: Prompt-based tuning has been proven effective for pretrained language models (PLMs). While most of the existing work focuses on the monolingual prompts, we study the multilingual prompts for multilingual PLMs, especially in the zero-shot cross-lingual setting. To alleviate the effort of designing different prompts for multiple languages, we propose a novel model that uses a unified prompt for all languages, called UniPrompt. Different from the discrete prompts and soft prompts, the unified prompt is model-based and language-agnostic. Specifically, the unified prompt is initialized by a multilingual PLM to produce language-independent representation, after which is fused with the text input. During inference, the prompts can be pre-computed so that no extra computation cost is needed. To collocate with the unified prompt, we propose a new initialization method for the target label word to further improve the model’s transferability across languages. Extensive experiments show that our proposed methods can significantly outperform the strong baselines across different languages. We release data and code to facilitate future research.

## introduction pre-trained language models (plms) have been proven to be successful in various downstream tasks @xcite @xcite . prompt-tuning is one of the effective ways to induce knowledge from plms to improve downstream task performance especially when the labeled data is not sufficient @xcite @xcite . the essence of prompttuning is to precisely design task input structure so that it can imitate the pre-training procedure of @xcite . food is great. @xcite . this restaurant is terrible.) 😊 (positive) plms and better induce the knowledge from them. for example, to classify the sentiment polarity of the source sentence "food is great", a template "it's @xcite ." is constructed before the source input where the accurate label is masked. in this way, the sentiment-related words like 'good', 'bad', and 'average' is predicted at the masked position with probabilities on the target side, over which a verbalizer is leveraged to project the final sentiment labels. previously, most work on prompt-based tuning @xcite ) mainly considers the monolingual prompts. however, it is not straightforward when applying to multilingual tasks due to absent multilingual prompts that heavily rely on native language experts to design both the templates and label words. an alternative way to build multilingual prompts is to machine-translate source prompts into target languages. but it is still infeasible for low-resource languages as the translation quality is hard to be guaranteed. other work also considers using soft prompts that consist of continuous vectors. although it reduces the cost of building prompts for multiple languages, the mismatch between the pro-cedures of pre-training and prompt-tuning brings many obstacles to the desired tasks, because the soft prompts never occur in the model pre-training stage. in this work, we focus on the zero-shot crosslingual transfer of prompt-based tuning. as shown in figure 1 , the model is trained on the source language (english), while tested on the other language (chinese). we explore the approaches to use a unified multilingual prompt that can transfer across languages. we propose a novel model, called uniprompt, which takes the merits of both discrete prompts and soft prompts. uniprompt is model-based and language-independent. it is initialized by a multilingual plm that takes english prompts as input and produces language-agnostic representation benefit from the transferability of multilingual plms. during inference, the prompts can be pre-computed so that no extra computation cost is introduced. in this way, we can alleviate the effects of prompt engineering for different languages, while reserving the ability of plms. to better collocate with the unified prompt, we propose a new initialization method for the label words instead of using the language model head from the plm. this proves to further improve the model's transferability across languages. we conducted extensive experiments on 5 target languages with different scales of data. experimental results prove that uniprompt can significantly outperform the strong baselines across different settings. we summarize the contributions of this paper as follows: ## analysis in this section, we will analyze the model in detail to verify the effectiveness of each module of the model. all experimental setups in this section are identical to the main experiments unless otherwise stated, and all experiments are based on 16-shot data. ## conclusion in this paper, we propose a new prompt-based tuning method for zero-shot cross-lingual text classification. for the two key elements of the prompt, we respectively give solutions under this task setting. for templates, we use a two-tower prompt encoder for encoding, which not only decouples specific languages but also preserves the ability of prompts to activate the latent knowledge of the language model. for label words, we use soft label words and dynamic initialization methods, which also achieve the goal of decoupling specific languages. the experimental results prove the utility of our model, and we also design experiments to carry out detailed analysis of the settings of our model.