MSP : Multi-Stage Prompting for Making Pre-trained Language Models Better Translators (2022)

Abstract: Prompting has recently been shown as a promising approach for applying pre-trained language models to perform downstream tasks. We present Multi-Stage Prompting, a simple and automatic approach for leveraging pre-trained language models to translation tasks. To better mitigate the discrepancy between pre-training and translation, MSP divides the translation process via pre-trained language models into three separate stages: the encoding stage, the re-encoding stage, and the decoding stage. During each stage, we independently apply different continuous prompts for allowing pre-trained language models better shift to translation tasks. We conduct extensive experiments on three translation tasks. Experiments show that our method can significantly improve the translation performance of pre-trained language models.

## introduction prompting @xcite , which refers to the approach of generating taskspecific outputs from language models (lms) by conditioning on extra information (known as prompts), has emerged as a new way of using lms to perform natural language processing (nlp) tasks @xcite . while being efficient in parameters @xcite , prompting can enable mixed-task inference, which is not possible for other related approaches like finetuning or adapter-based tuning @xcite . prompting also opens the possibility of using a single pre-trained lm to perform all nlp tasks @xcite . machine translation (mt), which involves transformations between two languages, is considered one of the most challenging tasks in nlp @xcite . while neural machine translation (nmt) @xcite @xcite is the current de facto approach for machine translation, using pre-trained lms as translators via prompting is appealing in several aspects. for example, for the method described in this paper, supporting a new translation direction with a pre-trained lm occupies disk spaces below 20m, which is much smaller than training a separate neural machine translation model, where the model size is typically larger than 60m per language pair for the transformer architecture. 2 furthermore, the pre-trained lm also retains the ability to perform other downstream tasks, which is an important characteristic that has not been validated available on neural machine translation models. however, it is challenging to leverage pre-trained lms to translation tasks via prompting. first, finding an appropriate prompt for a translation task is not trivial and requires specific designs @xcite @xcite . second, the prompting method with a single prompt may be sub-optimal for steering pre-trained lms to translation tasks, as there is a clear discrepancy between the objectives of translation and pre-training. translation imposes strict semantic equivalence and language space constraint, in which a source sentence must translate to a semantically equivalent sentence in the target language space. as the objective of pretraining is usually to reconstruct parts of the input sentence @xcite , the generation of a pre-trained lm conditioned on a source sentence will likely be in the source language space with non-equivalent semantics. therefore, using a single prompt to guide the lm for mitigating both the semantic and language gap is likely to be sub-optimal. third, prevalent figure 1 : overview of using prompts for steering a multilingual gpt (mgpt) model to machine translation tasks. note that we reset the position ids during each stage in multi-stage prompting for ease of implementation. all stages use the same mgpt model. in this paper, we present multi-stage prompting (msp) for addressing the challenges of steering pre-trained language models to translation tasks. msp encapsulates the idea of breaking translation tasks into simpler consecutive stages, allowing the pre-trained lm to learn "smoother transitions" to translation tasks by providing different prompts at different stages. for gpt-style pretrained lms, we design a three-stage prompting scheme for modeling the translation process, which consists of an encoding stage, a re-encoding stage, and a decoding stage. specifically, the pre-trained lm focuses on learning source representations at the encoding stage and learns refined bidirectional representations by re-encoding source sentences at the re-encoding stage. therefore, the lm can produce better translations with refined source representations at the decoding stage. following prefix-tuning (li and liang, 2021) and prompt tuning @xcite , we use independent trainable continuous prompts at different stages, which are learned through back-propagation. the difference between basic (single-stage) prompting and multi-stage prompting is illustrated in figure 1 . we demonstrate the effectiveness of our method with a multilingual gpt (mgpt) model on romanian-english, english-german, and english-chinese translation tasks. experiments verify that compared with prompt tuning or prefix-tuning, msp can significantly improve the translation per-formance of pre-trained lms. our method improves the translation performance of pre-trained language models via prompt tuning and prefixtuning by 18.6 and 4.1 bleu points on average over the three translation tasks, respectively, suggesting that msp is a more effective prompting method for translation tasks. ## multi-stage prompting we propose multi-stage prompting (msp), a simple and lightweight method for steering pre-trained lms to translation tasks. we first describe the concept of deep continuous prompts in section 3.1. then we detail the stages and training objective in section 3.2 and section 3.3, respectively. finally, we describe the reparameterization of deep continuous prompts in section 3.4. ## conclusion we have presented multi-stage prompting, a method for making pre-trained language models better translators. experiments show that with multi-stage prompting, pre-trained lms can generate better translations, showing the potential of using pre-trained lms for translation tasks.