S ci R eview G en: A Large-scale Dataset for Automatic Literature Review Generation (2023)

Abstract: Automatic literature review generation is one of the most challenging tasks in natural language processing. Although large language models have tackled literature review generation, the absence of large-scale datasets has been a stumbling block to the progress. We release SciReviewGen, consisting of over 10,000 literature reviews and 690,000 papers cited in the reviews. Based on the dataset, we evaluate recent transformer-based summarization models on the literature review generation task, including Fusion-in-Decoder extended for literature review generation. Human evaluation results show that some machine-generated summaries are comparable to human-written reviews, while revealing the challenges of automatic literature review generation such as hallucinations and a lack of detailed information. Our dataset and code are available at [ https://github.com/tetsu9923/SciReviewGen ]( https://github.com/tetsu9923/SciReviewGen ).

## introduction scientific document processing has been a topic of interest in the frontiers of natural language processing (nlp) @xcite . although neuralbased nlp models have achieved remarkable success in diverse areas, scientific documents present distinct challenges, such as longer inputs, technical terms, and complex logic. these challenges have motivated nlp researchers to undertake various studies on scientific documents, such as scientific document summarization, retrieval, and information extraction @xcite @xcite . automatic literature review generation is one of the most attractive research topics in scientific document processing. a literature review is a summary of scientific papers written by experts to comprehend previous findings @xcite . @xcite investigated that the number of published scientific papers doubles every nine years, increasing the demand for literature reviews in diverse research areas. automatic literature review generation significantly benefits researchers by expanding their studies into new research fields. however, only a few studies have addressed automatic literature review generation. for example, @xcite recently proposed galac-tica, a large-scale language model trained on 48 million scientific papers. galactica was made publicly available to demonstrate its ability to generate literature reviews; however, it was shut down within a few days owing to the hallucination problem @xcite . as there are no large-scale literature review datasets, applying data-hungry supervised neural summarization models is difficult. the absence of large-scale datasets is a significant bottleneck in research on automatic literature review generation. in this study, we pioneer the research of automatic literature review generation by providing a large-scale dataset based on the semantic scholar open research corpus (s2orc; @xcite . we release scireviewgen, which consists of over 10,000 literature reviews in the field of computer science and 690,000 papers cited in the reviews. as our dataset is created in a domain-agnostic way, it is possible to create datasets in other scientific fields, such as medical and biological sciences. figure 1 shows an overview of the literature review generation task. we regard it as a query-focused multi-document summarization (mds) task. the inputs are the abstracts of papers cited in the reviews, and the queries are the titles of the reviews and chapters, which specify the topics in the reviews. in the actual writing process of a literature review, we need to decide on the papers to cite in the review and group them into several chapters. as the first step for automatic literature review generation, we exclude those processes from our scope and focus on summarization given the cited papers and chapter division. as scireviewgen and s2orc include bibliographic information on reviews and their cited papers (e.g., doi, citation, and chapter division), our dataset can be used for end-to-end literature review generation. based on our dataset, we evaluate recent transformer-based summarization models for the literature review generation task. as current summarization models cannot simultaneously generate the entire text of reviews, we split a review into chapters and evaluate each of the generated chapters. in addition to recent models, such as big bird @xcite and fusion-in-decoder (fid; @xcite , we propose queryweighted fusion-in-decoder (qfid), a simple extension of fid for query-focused mds. as shown in the experimental results, our proposed model outperforms the other models by focusing on the contents concerning the query. finally, we conduct a human evaluation of the generated reviews and compare them with humanwritten reviews. the human evaluation results clarified that we have not reached the fully automatic literature review generation stage due to issues such as hallucinations and less informativeness. however, we obtained promising results, showing that approximately 30% of the generated chapters are competitive or superior to human-written reviews. our dataset and evaluation results provide a basis for future research on automatic literature review generation. ## task definition & dataset we now describe the literature review generation task and the scireviewgen dataset, which is created using s2orc. the data collection process and dataset statistics are presented below. ## experiments we study the performance of the current document summarization models on the split version of scireviewgen (hereinafter refered to as scire-viewgen). we use the abstracts of the cited papers, the literature review titles, and the chapter titles as inputs. as mentioned in section 3.3, scire-viewgen has an average input length of longer than 1024 tokens and contains many novel n-grams. in addition, it contains literature review titles and chapter titles that can be used as summarization queries. therefore, we employ query-focused abstractive summarization models that can accept long sequences for the literature review generation task. we first experiment with several transformerbased models that simply concatenate queries into documents as encoder inputs. we then propose the query-weighted fusion-in-decoder (qfid) that extends fusion-in-decoder (fid) to explicitly consider each paper's relevance to queries. ## conclusion we propose scireviewgen, a large-scale dataset for automatic literature review generation. we also introduce an extension of fid @xcite for query-focused summarization and show that our qfid model outperforms naive fid. the human evaluation results show that some generated texts are comparable to human-written literature reviews. however, challenges such as hallucinations and a lack of detailed information still remain to be addressed. we hope that our study will serve as a basis for future research on the challenging task of automatic literature review generation.