Towards Z ero- S hot C onditional S ummarization with A daptive M ulti- T ask F ine- T uning (2020)

Abstract: Automatic summarization research has traditionally focused on providing high quality general-purpose summaries of documents. However, there are many applications which require more specific summaries, such as supporting question answering or topic-based literature discovery. In this paper we study the problem of conditional summarization in which content selection and surface realization are explicitly conditioned on an ad-hoc natural language question or topic description. Because of the difficulty in obtaining sufficient reference summaries to support arbitrary conditional summarization, we explore the use of multi-task fine-tuning (MTFT) on twenty-one natural language tasks to enable zero-shot conditional summarization on five tasks. We present four new summarization datasets, two novel “online” or adaptive task-mixing strategies, and report zero-shot performance using T5 and BART, demonstrating that MTFT can improve zero-shot summarization quality.

## introduction transfer learning, in which a model is first pretrained on one or more data-rich tasks before being fine-tuned on a downstream task of interest, has been repeatedly shown to obtain remarkable performance on many natural language processing tasks @xcite @xcite . the most successful models resulting from this paradigm rely on self-supervised pretraining with prohibitively-large1 datasets to facilitate adaptation to new tasks (i.e., fine-tuning) with less abundant data @xcite @xcite . unfortunately, the benefits of pretraining are reduced for tasks in which there is little direct knowledge 1as estimated by @xcite , the cost for training the 11 billion parameter variant of t5 @xcite can exceed @xmath01 million to assess environmental damage caused by this week's devastating tsunami, as reports of destroyed coral reefs and uprooted mangrove forests began trickling in [...] ## background from its inception, automatic summarization aimed to condense documents either in a generic way -conveying the main points of the document to any user, or focusing on points tailored to specific users and applications, such as topic or querydriven summarization @xcite . our aims are even more specific than topic-driven summarization: we are interested in summarizing documents in response to ad-hoc natural language healthrelated questions asked by the general public. summarizing information to generate answers to such questions can only rarely be reduced to topicdriven summarization, e.g., if a person is looking for general information about a given health condition or treatment; in over 90% of cases, health questions are more specific and focus only on particular aspects of the topic @xcite . for example, people may be looking for medications for a specific condition or asking how to store a drug. the summary, therefore, has to be tailored not only to the topic of the question and task but must also be restricted only to the aspects of the topic that directly address the question. consider the following question from a user of a in the open domain, previous community efforts to focus on topic driven summarization include the document understanding conference (duc) and its successor, the text analysis conference (tac), both of which organized topic-based summarization tasks. in various iterations of these tasks, human assessors developed topic statements and documents cluster for those topics, and then manually authored summaries based on the topic statements. the tasks' participants were asked to develop automatic summarization approaches for generating single-or multi-document summaries that contained information relevant to the topic statement. other community efforts involving summarization include the bioasq2, cl-scisumm3, and scholarly document processing4 challenges that involve summarization of scientific articles. however, despite the attention that summarization has received in the natural language processing community and the recent development of more sophisticated summarization algorithms, the task of automatically generating human-quality still poses many challenges. a study of content selection across multiple domains, including medical articles, indicates that new forms of sentence representations and external knowledge sources are needed to identify the most suitable approaches to summarization @xcite . recent work has shown models with transformer-based architectures, coupled with unsupervised pretraining approaches, to achieve state of the art results in many text generation tasks. building on this, researchers have recently shown that these models can be conditioned on a prompt included in the input text. for example, this prompt can guide the content of the generated text towards either a desired topic @xcite or instruct the model to produce output for a specific task @xcite 2019). similar work on conditional generation includes @xcite , in which the authors condition an extractive transformer on control codes specifying position, importance, and diversity of the sentences in the source text. there have been relatively few publications focused on zero-shot learning specifically for summarization. @xcite experiment with zero-shot learning for cross-lingual sentence summarization, while @xcite explored zeroshot abstractive summaries of five-sentence stories. prior work indicates that topic and questiondriven summarization can be formulated as a textto-text, conditional generation problem in which content selection and source realization are explicitly conditioned on a user-specified prompt. the formulation of summarization in this way intuitively dovetails with the desired goal described above: question-driven summarization of answers to user's health-related questions. in this study, we extend previous work done with bart and t5 using multi-task fine-tuning using a large body of tasks and exploring multiple mixing strategies to advance topic and question-driven summarization in the open and medical domains. ## models several transformer-based models have been shown to generate high quality natural language @xcite @xcite . the majority of these models cast summarization as language modeling wherein the input to the model is the sequence of words in the source document followed by a mask token for each word in the desired summary @xcite . this substantially limits the length of summaries that can be generated due to the input sequence limits imposed during pretraining. fortunately, more recent approaches use separate transformers for encoding and decoding, allowing the generation of potentially arbitrary length sequences. in this work, we explored the two most notable of these approaches: bart and t5. bart (bidirectional and auto-regressive transformers) is pre-trained with sentence ordering and token in-filling tasks @xcite . bart uses a separate bidirectional encoder and autoregressive decoder similar to bert except that (1) bart's decoder incorporates cross attention over the final encoder layer and (2) bart's encoder does not use a feed-forward dense layer for word prediction. in our experiments, we used bart-large, which includes 12 transformer layers in the encoder and decoder. t5 (text-to-text transfer transformer) uses several pretraining objectives, including unsupervised fill-in-the-blank as well as supervised translation, summarization, classification, and reading comprehension tasks where each task is represented as a language generation task @xcite . t5 closely follows the originally-proposed transformer architecture @xcite except using relative positional embeddings rather than sinusoidal encoding. in this work, we used t5-base, which includes 12 transformer layers. ## adaptive multi-task fine-tuning we adapt the text-to-text setting used to pre-train t5 @xcite to enable fine-tuning on a large body of tasks with the intent of injecting knowledge from related natural language processing tasks to enable improved zero-shot conditional summarization. in this section, we describe (1) the fine-tuning tasks used in our experiments, (2) how these tasks are encoded as text-generation, and (3) approaches for task mixing. ## experiments in our experiments, we trained on the datasets described in section 4.1 and evaluated on five tasks originating from three datasets previously unseen by the model. all models were trained with a batch size of 8, maximum sequence length of 512 tokens, 3 warm-up epochs followed by 10 training epochs, and 1,000 batches-per-epoch, using single v100x gpus (32 gb vram) on a shared cluster. training took between four-to-six hours, depending on cluster load. additional implementation details are provided in appendix a. to reduce variance between runs, we report results with greedy decoding (i.e., no beam search). we measured the impact of (1) multi-task finetuning (mtft), (2) different task mixing strategies, and (3) excluding various tasks from fine-tuning on zero-shot summary quality. we report traditional summarization and generation metrics such as bleu @xcite , rouge @xcite and @xcite . because the reference summaries for many tasks are highly abstractive, we adopt the embeddingbased metrics proposed in @xcite , i.e., glove @xcite cosine similarity using embedding averages (eacs), vector extrema (vecs, @xcite , and greedy matching @xcite . ## results finally, ## discussion when analyzing the impact of different tasks on down-stream performance as indicated by while an in-depth manual assessment of all tasks is beyond the scope of this work, a shallow manual review suggests that conditional summarization would benefit from new metrics that emphasize the role of the conditional context (i.e., question or topic description) in the summary to ensure that summaries are not too generic. ## conclusions in this paper, we explored the impact of multitask fine-tuning (mtft) on zero-shot conditional summarization for consumer health questions (mediqa, @xcite as well as topicdriven news article summarization (i.e., the tac and duc summarization challenges). we introduced four new summarization datasets and proposed two online or adaptive methods for task mixing during fine-tuning. our experimental results indicate that mtft enables bart to produce higher quality summaries than t5, and that mtft improved summary quality on unseen tasks in terms of rouge-l by 35.50 % (relative; 11.20 % absolute) for consumer health and 35 %-241 % (relative; 3.80 %-11.46 % absolute) for tac. duc results were inconclusive, with mtft improving t5 results but hindering bart. ablation analysis indicates that all tasks are not created equal and careful consideration must be taken to ensure each task has transferable characteristics (even subtle semantic properties such as argumentation properties) to the down-stream zero-shot application. our proposed self-adaptive task mixing strategy was able to lessen the impact of irrelevant tasks on zero-shot performance by 8.25 % (relative; 2.75 % absolute) bleu-4 and 7.57 % (relative; 3.04 % absolute) rouge-l. in future work, we plan to explore automatic approaches for determining the optimal set of fine-tuning tasks, improving the robustness of task mixing strategies to accommodate sub-optimal task combinations, and exploring new evaluation metrics that better reflect the role of the summarization context (i.e., question or topic description).