Domain-Oriented Prefix-Tuning: Towards Efficient and Generalizable Fine-tuning for Zero-Shot Dialogue Summarization (2022)

Abstract: The most advanced abstractive dialogue summarizers lack generalization ability on new domains and the existing researches for domain adaptation in summarization generally rely on large-scale pre-trainings. To explore the lightweight fine-tuning methods for domain adaptation of dialogue summarization, in this paper, we propose an efficient and generalizable Domain-Oriented Prefix-tuning model, which utilizes a domain word initialized prefix module to alleviate domain entanglement and adopts discrete prompts to guide the model to focus on key contents of dialogues and enhance model generalization. We conduct zero-shot experiments and build domain adaptation benchmarks on two multi-domain dialogue summarization datasets, TODSum and QMSum. Adequate experiments and qualitative analysis prove the effectiveness of our methods.

## introduction abstractive dialogue summarization task aims to distill the most critical information in a conversation to produce a concise version, involving chitchat @xcite , meeting @xcite , customer service @xcite , and task-oriented dialogue scenarios @xcite . compared to the single-speaker texts, summarizing a dialogue presents a unique set of challenges, such as unstructured expressions and information sparsity issues. recently, large-scale generative pre-trained models @xcite have promoted the development of abstractive dialogue summarization but they all require extensive human-annotated golden summaries, which makes them not scalable to new domains where only few/no labeled data is available. considering that real-world applications often face the problem u: i am looking for a train. the train should arriv e by 9:45 and should depart from cambridge. ## problem formulation domain adaptation of dialogue summarization aims to generate the summary y conditioned on the source dialogue x d , where the training and test are in different domains. we add the domain word prefix x dw and discrete prompt x dp as additional input which we will describe details later. note that we only update the prefix-related parameters and fix the parameters of bart. we train the model using the source data and test using the target data. ## methodology as figure 2 shows, our model is on the basis of the framework of bart, including a domain-oriented prefix module, a prompt encoder, and a decoder. ## challenges through the analysis of cases in appendix c, we summarize two challenges of low-resource domain adaption for abstractive dialogue summarization task: ## conclusion in this paper, we present a domain-oriented prefixtuning model to handle the domain adaptation for dialogue summarization based on an efficient and generalizable fine-tuning method. the domain word initialized prefix module disentangles the target domain knowledge from the source domain and the discrete prompts enhance the generalization ability of the model. the experiments in zero-shot and few-shot settings show that our methods have made great progress on two datasets. lead-3: this method is commonly used in news summarization task, which treats the first three sentences of the document as the summary. oracle: the method is used to obtain an oracle through a greedy way similar to @xcite , which treats the sentences that maximize the rouge-2 as the summary. bertext: proposed by @xcite , this model is extractive and its parameters are initialized with bert. pgn: proposed by @xcite , this model adopts the pointer mechanism to deal with the issue of out-of-vocabulary in the summary generation process. transformer: proposed by @xcite , this model captures long-distance information through the self-attention mechanism. bertabs: proposed by @xcite , this model is abstractive, which encoder is initialized with bert and is trained with a different optimizer than decoder. bart: proposed by @xcite , the model is a state-of-the-art abstractive summarization model pre-trained with a denoising autoencoding objective. bart w. ds/qr: proposed by @xcite , the model is a general summarization framework, with two encoders that share the underlying parameters and a decoder, which can fuse the input text and dialogue state/query. prefix-tuning: proposed by li and liang (2021), this model introduces a prefix matrix on the basis of fixed pre-training bart parameters and allows the prefix matrix to learn task information through training, which optimizes the summarization performance in the small parameters and few-shot scenarios. qmsum: this model is proposed by @xcite , which is a two-stage locate-thensummarize solution on query-based meeting summarization task.