PSP : Pre-trained Soft Prompts for Few-Shot Abstractive Summarization (2022)

Abstract: Few-shot abstractive summarization has become a challenging task in natural language generation. To support it, we developed a novel soft prompts architecture coupled with a prompt pre-training plus prompt fine-tuning paradigm, which is effective and tunes only extremely light parameters. To meet the structure of the generation models, the soft prompts comprise continuous input embeddings across an encoder and a decoder. Importantly, a new inner-prompt placed in the text is introduced to capture document-level information. The aim is to devote attention to understanding the document that better prompts the model to generate document-related content. In the training process, the prompt pre-training with self-supervised pseudo-data firstly teaches the model basic summarizing capability. Then, with few-shot examples, only the designed lightweight soft prompts are fine-tuned. Experimental results on the CNN/DailyMail and XSum datasets show that our method, with only 0.1% of the parameters, outperforms full-model tuning where all model parameters are tuned. It also surpasses Prompt Tuning by a large margin and delivers competitive results against Prefix-Tuning with 3% of the parameters.

## introduction given the high labor-costs of obtaining quality abstractive summaries, few-shot abstractive summarization is very demanding and highly challenging. a widely accepted paradigm for almost all nlp tasks is to fine-tune the entire set of parameters for a large pre-trained language model to suit the target task @xcite . however, the fine-tuning with few-shot examples usually leads to disappointing results, especially with generation tasks like abstractive summarization @xcite . the likely outcome is an overfit model. further, for * corresponding author. pre-trained language models are few-shot learners, i.e., @xcite that surprisingly perform generation tasks from a few examples without any further gradient updates. although it lacks a rigorously theoretical proof, prompt learning inherits the few-shot property @xcite @xcite . commonly, this type of learning is considered to retrieve relevant knowledge from frozen language models, only tuning continuous prompts to quickly adapt to new tasks with very few examples. more recently, prompt tuning @xcite has received much attention. with large frozen language models (say, >10 billion parameters), prompt tuning simply adds a tunable soft prompt to the input of the encoder, achieving results that are comparable to full-model tuning. yet, our empirical results, in section 2, demonstrate that prompt tuning for abstractive summarization yields simply abysmal performance. @xcite extends the use of prompt learning in the natural language generation area. with this technique, continuous prompts are applied to every layer of the pre-trained model and even shows increase in few-shot generation tasks over fine-tuning. yet the training process is not stable and updates are required that add to the memory and training costs. 1 given the shortcomings of these two methods, we have developed a soft prompts tuning method that is specifically designed for summarization. the structure is given in figure 1 . the method is capable of performing few-shot language generation task (i.e., abstractive summarization) with an efficient amount of training parameters. prompt tokens are added before the decoder input tokens to guide the generation process toward the target summary. moreover, we have designed three inner prompts -interval, sequential, and fixed-length -one of which is placed among the source input tokens. the aim is to capture the structure in the source document and aid in understanding its semantics, so as to better prompt the model to generate document-related content. each kind of inner prompts focuses on different semantic units (e.g., phrases, sentences, and etc.), differentiating important units from non-informative ones. to bolster the summarization ability of the model and assist the prompts to understand the documents, prompt pre-training is performed before the tuning process, and leveraged by self-supervised pseudo data. as a last step, all the prompts are fine-tuned with fewshot training examples. experiments conducted on two commonly used datasets -cnndm @xcite and xsum @xcite ) -demonstrate that our method outperforms full-model tuning under few-shot settings only with 0.1% of the parameters. it also surpasses naive prompt tuning by a large margin. our model also yields a performance competitive to prefix-tuning with 3% of the trainable parameters. a detailed analysis shows that the designed prompt-pre-training phase and the inner prompts are effective for few-shot text summarization. thus, the major contributions of this work include : 1) a novel soft prompt architecture for few-shot abstractive summarization. with the well-designed prompts in embedding layer, our model fulfills the task effectively and efficiently; 2) it is necessary to perform prompt pre-training strategy which benefits soft prompts model for fewshot summarization and shows excellent zero-shot capabilities; 3) experiments that investigate the effect of different prompts by probing the attention weights. the results show our model is able to: extract knowledge from the encoder language model; understand the discourse in the document; and guide the decoder language model to generate fluent summaries. ## pilot experiments in a pilot study, we experimented with using prompt tuning under 300-shots settings to find reasonable clues as to how to design summaryprompts for the task. our findings follow. consider an encoder-decoder language model p θ (y|x) based on the transformer architecture @xcite (e.g., bart @xcite ) and parameterized by θ. to conduct a few-shot summarization task, we have some few-shot training pairs of a document @xmath0, x 2 , . . . , x |x| } and a corresponding summary @xmath1, y 2 , . . . , y |y | }. specifically, we divided x into different subsets with sentences foot_1 as our unit, @xmath2, . . . x i j , . . . , x n m }, where x i j denotes the j th token in the i th sentence. first, original prompt tuning is applied by concatenating a series of prompt tokens p en , parameterized by θ pen , to the encoder input x @xmath3, . . . , e i j , . . . e n m }, where e represents the embedding of each token (the leftmost structure in figure 1 ). the gradients are backpropagated through the prompts and the weights θ of language model are frozen @xcite . in this way, the model maximizes the likelihood of the output y : the result of original prompt tuning is shown on the first line in ## method in light of our findings about the current architectures, we developed a new architecture of pretrained soft prompts, for few-shot abstractive summarization called psp. the framework includes continuous prompts across the encoder and decoder inputs, as well as inner-prompts to capture the dependencies between documents and target summaries. to better understand a given document, we add a prompt pre-training process before fewshot tuning. it also brings a good initialization for the prompting. the overall architecture and training scheme are illustrated in figure 3 . ## conclusion in this paper, we present a novel pre-trained soft prompts architecture (psp) specifically designed for few-shot abstractive summarization. we design continuous input embeddings across an encoder and a decoder alongside several kinds of innerprompts placed in the text, assisting the model better to understand documents and guide accurate generation. empirical results find the necessity of using prompt pre-training for few-shot/zero-shot abstractive summarization. extensive experiments and analyses show that the proposed psp provides the best effectiveness-efficiency trade off among all the baseline methods. cnndm xsum pseudo corpus pseudo corpus # of original passages 287,113 204,017 # of pre-training data 284,177 158,499