S um S urvey: An Abstractive Dataset of Scientific Survey Papers for Long Document Summarization (2024)

Abstract: With the popularity of large language models (LLMs) and their ability to handle longer input documents, there is a growing need for high-quality long document summarization datasets. Although many models already support 16k input, current lengths of summarization datasets are inadequate, and salient information is not evenly distributed. To bridge these gaps, we collect a new summarization dataset called SumSurvey, consisting of more than 18k scientific survey papers. With an average document length exceeding 12k and a quarter exceeding 16k, as well as the uniformity metric outperforming current mainstream long document summarization datasets, SumSurvey brings new challenges and expectations to both fine-tuned models and LLMs. The informativeness of summaries and the models supporting the evaluation of long document summarization warrant further attention. Automatic and human evaluation results on this abstractive dataset confirm this view. Our dataset and code are available at https://github.com/Oswald1997/SumSurvey.

## introduction as one of the major problems in natural language processing, summarization is about processing input documents and generating short texts that contain key information. its main applications include extracting main events of news, mining opinions on social media, and analyzing comments on shopping websites @xcite @xcite . these tasks use extractive or abstractive models for summarization. the former refers to the extraction of key words or sentences from input to form a summary @xcite , while the latter is to rewrite important information * corresponding author. on the basis of understanding @xcite . intuitively, abstractive tasks are more challenging, which require models to read the input document and present the most salient information within output length limits. in the process of improving and evaluating this capability of summarization models, abstractive datasets play an indispensable role. in addition, the distribution form of salient information is also an indicator of quality for summarization datasets, because a more uniform distribution means that obvious layout bias cannot be exploited @xcite . for some tasks, extracting only the first few sentences directly as the final summary can achieve good results, which is common in news domain @xcite . summarization models need to be able to extract key information distributed throughout the document and deal with relationship among different positions. with the explosion of data volume and increase of computing resources, language models can gradually deal with longer documents @xcite . however, even with the ability to handle long text, it is still difficult to evaluate the performance of these models due to the different lengths of documents in public datasets. the above problems bring a requirement to summarization task: to construct a new summarization dataset with high abstractiveness and uniformity of salient information distribution, while lengths of documents are in line with the processing capacity of current language models. we believe that collecting survey papers is a feasible approach, because such documents are usually long and enumerate many studies in a certain field in detail. besides, the abstract section naturally summarizes the full paper. compared with general papers, information in survey papers tend to be more evenly distributed. thus, we propose a long document summarization dataset called sumsurvey consisting of scientific survey papers. figure 1 shows a random summary example 1 from sumsurvey, different colored parts indicate that they are generated from different sections, corresponding to contents from chapter i to chapter vi. it can be seen that each section is reflected in this summary, and is presented in a highly general discourse, which is exactly what we need. we conduct a series of experiments using finetuned models @xcite @xcite and large language models @xcite . both automatic and human evaluation are conducted to explore models' ability to handle long documents. our main contributions are as follows: ## experiments we conduct a series of experiments to evaluate performance of baseline models on sumsurvey. ## conclusion we propose a new long document summarization dataset sumsurvey consisting of scientific survey papers. the average length of documents in sum-survey is longer than publicly available summarization dataset, and it has higher abstractiveness. in addition, salient information is more evenly distributed throughout documents. by benchmarking baseline models using automatic and human evaluation, we have a comprehensive view of how these models perform on sumsurvey. we hope sum-survey will contribute to evaluating and further improving fine-tuned models on longer inputs, and also expect llms to pay more attention to informativeness and factuality. moreover, we look forward to more comprehensive automatic evaluation models supporting long documents.