LMGQS : A Large-scale Dataset for Query-focused Summarization (2023)

Abstract: Query-focused summarization (QFS) aims to extract or generate a summary of an input document that directly answers or is relevant to a given query. The lack of large-scale datasets in the form of documents, queries, and summaries has hindered model development in this area. In contrast, multiple large-scale high-quality datasets for generic summarization exist. We hypothesize that there is a hidden query for each summary sentence in a generic summarization annotation, and we utilize a large-scale pretrained language model to recover it. In this way, we convert four generic summarization benchmarks into a new QFS benchmark dataset, LMGQS, which consists of over 1 million document-query-summary samples. We thoroughly investigate the properties of our proposed dataset and establish baselines with state-of-the-art summarization models. By fine-tuning a language model on LMGQS, we achieve state-of-the-art zero-shot and supervised performance on multiple existing QFS benchmarks, demonstrating the high quality and diversity of LMGQS.

## introduction the field of generic summarization @xcite @xcite has made significant progress in recent years, thanks to the development of generative deep neural models @xcite and the availability of large-scale training data @xcite @xcite . however, query-focused summarization (qfs) presents a significant challenge due to the lack of data. most of the available qfs corpora @xcite @xcite contain only a few thousand documents or less, which is insufficient for training a robust neural model. for each summary, write a general question about the article that can be answered by it: article: a key rebel commander and fugitive from a u.s. drug trafficking … summary: 1. tomas medina caracas was a fugitive from a u.s. drug trafficking indictment … … 4. … the summary answers the following 4 questions from the article: 1. who was tomas medina caracas? … 4. … article: for many people, being happy at work is an important pursuit … summary: 1. naomi simson is a blogger, entrepreneur, speaker ... … 5. … the summary answers the following 5 questions from the article: 1. who is naomi simson? … 5. what is her book about? instruction one-shot example input document & @xmath0: generated query figure 1 : example prompt for query generation. the top part is the instruction, followed by the one-shot example consisting of document, summary, and query. the query for the input document (highlighted in yellow) and summary is generated by instructgpt. to address the lack of a large-scale qfs dataset. human annotation for qfs typically involves generating suitable queries and then writing corresponding summaries, which is both time-consuming and expensive. furthermore, it may necessitate a meticulous definition of the query scheme based on the domain of documents @xcite . we hypothesize that, for a pair of document and summary in a generic summarization dataset, hidden queries exist that represent the information needs associated with the summary. therefore, to efficiently scale up annotation, we prompt the largescale language model instructgpt @xcite 2022) with documents and summaries from four generic summarization datasets to generate the hidden queries. this approach results in the lmgqs dataset, which contains over 1.1 million triplets of document, query, and summary, encompassing a wide range of document and question types. to investigate the utility of our proposed lmgqs , we finetune a pretrained language model on it. the model accepts the concatenation of the original document and generated query as input and is trained to produce the original summary. we then compare the finetuned model with various query-focused summarization models on several existing qfs benchmarks that have no overlap with lmgqs under the zero-shot setting. empirical results demonstrate that the model finetuned on lmgqs achieves promising performance on both single-document and multi-document qfs benchmarks, surpassing strong baselines. similarly, when utilizing lmgqs for pre-finetuning, the model achieves state-of-the-art performance in the supervised setting. in summary, our contributions are three-fold: (1) we introduce a novel framework for constructing a qfs dataset by converting existing generic summarization datasets using language models as annotators. (2) we present lmgqs, a large-scale qfs benchmark, to foster future research on qfs 1 . (3) the model finetuned on lmgqs exhibits robust generalization capability and achieves remarkable zero-shot and supervised performance on other unseen qfs test sets. ## dataset creation we choose 4 generic datasets to build lmgqs: cnn/dailymail @xcite , xsum @xcite , samsum @xcite , and dialogsum @xcite . among 1 dataset will be released after the anonymous period them, cnn/dailymail and xsum are news summarization datasets, where both the documents and summaries are in formal written english. sam-sum and dialogsum are two recently proposed dialogue summarization datasets, whose inputs are the transcripts of multi-speaker conversations. ## lmgqs for qfs in this section, we demonstrate that by finetuning pretrained language models on lmgqs, one can obtain a qfs model that generalizes effectively to unseen tasks and domains. in particular, we finetuned a bart model @xcite , and the resulting model, lmgqs bart, exhibits promis- dataset size(train/test) query example multioped 1954/560 is protecting the environment incompatible with capitalism's values? newts -word 2400/600 snow, weather, cold, winter, temperatures, conditions, hot, morning, expected, parts newts -phrase 2400/600 winter temperatures, hot weather conditions, a cold morning, snow is expected later newts -sentence 2400/600 this topic is about winter temperatures as opposed to hot weather conditions, cold mornings, and weather forecasts like snow being expected later. debatepedia 12000/1000 would the election of a president make the eu a more accountable institution? duc 2006 0/200 identify computer viruses detected worldwide. include such details as how they are spread, what operating systems they affect, what damage they inflict, their country of origin, and their creators wherever possible. duc 2007 0/180 describe the state of teaching art and music in public schools around the world. indicate problems, progress and failures. ## conclusion & future works we introduce a novel large-scale dataset, lmgqs, for query-focused summarization (qfs), addressing the scarcity of large-scale benchmarks in this domain. in order to reduce the laborious human effort required for qfs, we utilize a large-scale language model to generate hidden queries from existing query-agnostic summarization datasets. by performing standard finetuning on lmgqs, we attain state-of-the-art zero-shot performance across multiple qfs benchmarks. the resulting zero-shot newts duc template template: what are {topical words, phrases, or sentences}? replace the instructions such as 'describe', 'indicate', 'include', 'summarize' with 'what are the' original query topics words: winter temperatures, hot weather, a cold morning describe the activities of morris dees and the southern poverty law center. templated query what are winter temperatures? what are hot weather conditions? what are a cold morning? what are the activities of morris dees and the southern poverty law center?