Hierarchical Catalogue Generation for Literature Review: A Benchmark (2023)

Abstract: Scientific literature review generation aims to extract and organize important information from an abundant collection of reference papers and produces corresponding reviews while lacking a clear and logical hierarchy. We observe that a high-quality catalogue-guided generation process can effectively alleviate this problem. Therefore, we present an atomic and challenging task named Hierarchical Catalogue Generation for Literature Review as the first step for review generation, which aims to produce a hierarchical catalogue of a review paper given various references. We construct a novel English Hierarchical Catalogues of Literature Reviews Dataset with 7.6k literature review catalogues and 389k reference papers. To accurately assess the model performance, we design two evaluation metrics for informativeness and similarity to ground truth from semantics and structure. Our extensive analyses verify the high quality of our dataset and the effectiveness of our evaluation metrics. We further benchmark diverse experiments on state-of-the-art summarization models like BART and large language models like ChatGPT to evaluate their capabilities. We further discuss potential directions for this task to motivate future research.

## introduction today's researchers can publish their work not only in traditional venues like conferences and journals but also in e-preprint libraries and mega-journals, which is fast, convenient, and easy to access @xcite . this enables rapid development and sharing of academic achievements. for example, statistics show that more than 50,000 publications occurred in 2020 in response to covid-19 @xcite . therefore, researchers are overwhelmed by considerable reading with the explosive growth in the number of scientific papers, urging more focus on scientific literature review generation @xcite . pioneer studies on scientific literature review generation explore citation sentence generation @xcite @xcite and related work generation @xcite @xcite . however, these methods can only generate short summaries, while a literature review is required to provide a comprehensive and sufficient overview of a particular topic @xcite . benefiting from the development of language modeling @xcite , recent works directly attempt survey generation @xcite @xcite but usually suffer from a disorganized generation without hierarchical guidance. as illustrated in figure 1 (a), we take gpt-3.5-turbo foot_0 , a large language model trained on massive amounts of diverse data including scientific papers, as the summarizer to generate scientific literature reviews. direct generation can lead to disorganized reviews with content repetition and logical confusion. since hierarchical guidance is effective for text generation @xcite , as shown in figure 1 (b), we observe that the catalogue, representing the author's understanding and organization of existing research, is also beneficial for scientific literature review generation. therefore, the generation process can be divided into two steps. first is generating a hierarchical catalogue, and next is each part of the review. figure 1(c ) reveals that even state-of-the-art language models can not obtain a reliable catalogue, leaving a valuable and challenging problem for scientific literature review generation. to enable the capability of generating reasonable hierarchical catalogues, we propose a novel and challenging task of hierarchical catalogue generation for scientific literature review, named as hicatglr, which is a first step towards automatic review generation. we construct the first benchmark for hicatglr by gathering html for-introduction ... llms such as gpt-3 have revolutionized the field of nlp, enabling sophisticated language understanding and generating capabilities ... ## metrics to evaluate the quality of generated catalogues, we propose two evaluation methods, catalogue quality estimate (cqe) and catalogue edit distance similarity (ceds). the former takes a textual perspective, while the latter integrates text and structure. we use these evaluation methods to assess the informativeness of the generated catalogues and the gap between the generated catalogues and the 9 the rouge scores in this paper are all computed by rouge-1.5.5 script with the option "-c 95 -r 1000 -n 2 -a -m" oracle ones, respectively. ## models we now introduce our explorations in hierarchical catalogue generation for literature review, including end-to-end and step-by-step approaches. ## conclusion in this work, we observe the significant effect of hierarchical guidance on literature review generation and introduce a new task called hierarchical catalogue generation for literature review (hi-catglr) and develop a dataset hicad out of arxiv and semantic scholar dataset. we empirically evaluate two methods with eight implementations for catalogue generation and found that the end-to-end model based on led-large achieves the best result. cqe and ceds are designed to measure the quality of generated results from informativeness, semantics, and hierarchical structures. dataset and code are available at https://github.com/zhukun1020/hicad . our analy-sis illustrates how ceds resolves some of the limitations of current metrics. in the future, we plan to explore a better understanding of input content and the solution to headings level error, repetition, and semantic conflict between headings.