Few-shot Query-Focused Summarization with Prefix-Merging (2022)

Abstract: Query-focused summarization has been considered as an important extension for text summarization. It aims to generate a concise highlight for a given query. Different from text summarization, query-focused summarization has long been plagued by the problem of lacking high-quality large-scale datasets. In this paper, we investigate the idea that whether we can integrate and transfer the knowledge of text summarization and question answering to assist the few-shot learning in query-focused summarization. Here, we propose prefix-merging, a prefix-based pretraining strategy for few-shot learning in query-focused summarization. Drawn inspiration from prefix-tuning, we are allowed to integrate the task knowledge from text summarization and question answering into a properly designed prefix and apply the merged prefix to query-focused summarization. With only a small amount of trainable parameters, prefix-merging outperforms fine-tuning on query-focused summarization. We further discuss the influence of different prefix designs and propose a visualized explanation for how prefix-merging works.

## introduction text summarization aims to compress the source document(s) into a shorter version that contains its important information. as a classic sub-topic for text summarization, query-focused summarization meets that situation that only a specific aspect of information is needed to be summarized. in other words, it aims to generate a summary based on the source content related to a given query. hence, this task requires not only to locate relevant content in a passage as question answering (qa) but also to summarize and generate a highlight as text summarization. although text summarization has been widely studied in recent years, there are fewer attempts on exploring query-focused summarization 1 corresponding author @xcite @xcite after the age of neural model. one main reason is the lack of generalized large-scale datasets. compared with the easily accessible nature reference summaries such as titles or headlines in text summarization, it is hard to collect large-scale data for query-focused summarization. meanwhile, human-written reference summaries have always been costly. the rapidly developed few-shot learning techniques provides potential cues to alleviate the problem of lacking large-scale dataset for query-focused summarization, and knowledge transferring is one of them. in fact, when facing unseen tasks, it is natural for human beings to integrate and transfer the knowledge of known tasks to relevant new tasks. inspired by this, we innovatively propose to decouple the query-focused summarization to two basic tasks, i.e. text summarization and question answering, and transfer the knowledge from these two tasks to query-focused summarization. however, in parameter-based knowledge learning, previous work are usually one-to-one (pre-train then fine-tune @xcite ) or one-to-many (domain/task adaption @xcite ), and seldom of them focus on manyto-one (integrate basic tasks to a complex one). in this case, the previous methods may not work well in this task. in this paper, we propose a pre-trained strategy, prefix-merging, for few-shot learning in queryfocused summarization. in recent prompt-based language models, the prompt/prefix is considered as containing the knowledge of the given task, which provides us an explicit way to control the task-specific knowledge previously dispersed in the language model (lm). for example, prefix-tuning @xcite achieved a similar result with fine-tuning by training only the task-specific prefix, a sequence of continuous vectors that prepend to the input. following the framework proposed by prefix-tuning, prefix-merging aims to integrate the task knowledge from text summarization and question answering into a properly designed prefix and apply the merged prefix to the more complex task, query-focused summarization. generally, there are two straightforward ideas for merging knowledge from multiple tasks into a prefix: concatenate the separated prefix for different tasks as a whole or adopt a shared prefix for all the tasks. considering there exist both similarities and differences across the tasks, a more flexible prefix design composed of both task-specific part and shared part is used in further investigation. moreover, we propose a self-adaptive prefix merging that allows the basic tasks themselves to decide the prefix design. drawn the inspiration from @xcite , we adopt fisher information to calculate the importance scores of the prefix embeddings (basic units for the prefix) for each basic task. for one task, only the prefix embeddings with top scores are activated in the following training. hence, different tasks can adapt to different parts of prefix automatically. after finishing training the merged prefix, it is transferred to a downstream task for fewshot learning. in the experiment, we explore prefix merging in the context of query-focused summarization, taking pubmedqa @xcite and duc @xcite as the evaluation dataset. prefix-merging provides a potential solution for the few-shot learning in complex tasks that can be integrated by the basic tasks. benefited by the universality of the prompt-based approach, prefixmerging is not limited by the model architecture and can be used in both autoregressive lm and encoder-decoder based lm. we believe this shows a possible direction to the application of promptbased approaches. our contribution can be summarized as follow: ## conclusion in this paper, we show that prefix-merging is an effective approach for transferring and integrating task knowledge from multiple auxiliary tasks to a target task with limited data. in the context of query-focused summarization, integrating text summarization and qa, our approach outperforms the traditional approach fine-tuning. we further discuss the influence of different prefix designs and propose a self-adaptive prefix-merging. we also provide a visualize explanation for how the merged prefix works. although this paper focuses on queryfocused summarization, we believe these findings suggest a new application for prompt-based approaches in multi-task situation, providing guidance for future progress in this field.