C ite B ench: A Benchmark for Scientific Citation Text Generation (2023)

Abstract: Science progresses by building upon the prior body of knowledge documented in scientific publications. The acceleration of research makes it hard to stay up-to-date with the recent developments and to summarize the ever-growing body of prior work. To address this, the task of citation text generation aims to produce accurate textual summaries given a set of papers-to-cite and the citing paper context. Due to otherwise rare explicit anchoring of cited documents in the citing paper, citation text generation provides an excellent opportunity to study how humans aggregate and synthesize textual knowledge from sources. Yet, existing studies are based upon widely diverging task definitions, which makes it hard to study this task systematically. To address this challenge, we propose CiteBench: a benchmark for citation text generation that unifies multiple diverse datasets and enables standardized evaluation of citation text generation models across task designs and domains. Using the new benchmark, we investigate the performance of multiple strong baselines, test their transferability between the datasets, and deliver new insights into the task definition and evaluation to guide future research in citation text generation. We make the code for CiteBench publicly available at https://github.com/UKPLab/citebench.

## introduction citations are a key characteristic of scientific communication. a paper is expected to substantiate its arguments and relate to prior work by means of exact bibliographic references embedded into the text, and the accumulated number of citations serves as a metric of publication importance. the acceleration of research and publishing in the past decades makes it increasingly challenging to both stay up-to-date with the recent publications and to figure 1 : citation text generation task, text in the grey box adopted from @xcite . the applications of related work generation would greatly reduce the reading and writing effort that accompanies research. yet, the importance of related work generation spans beyond practical applications. unlike most other genres, scientific writing enforces the use of citation markers that point to the information source, which in most cases is itself a highly structured scientific text backed by further citations. by circumventing the challenge of establishing information provenance, related work generation provides a unique opportunity to study knowledge aggregation and synthesis from textual sources, and contributes to our better understanding of text work in general. a key sub-task of related work generation is citation text generation -generating citation text for pre-selected cited papers given the context of the citing paper (figure 1 ). this task is actively studied by the nlp community and a plethora of approaches and datasets have been proposed in the past years: it has been cast as extractive @xcite and abstractive summarization @xcite @xcite @xcite @xcite ; taking one @xcite @xcite or multiple papers as input @xcite @xcite , and aiming to generate a single citation sentence @xcite @xcite or a paragraph @xcite . yet, the divergence in the existing task definitions and evaluation setups ( citebench adds substantial value to the research in citation text generation. it enables systematic comparison of existing approaches across a range of task architectures and domains, and provides a scaffolding for future citation text generation efforts. our proposed qualitative evaluation methodology based on citation intent and discourse structure advances the state of the art in assessing the performance and properties of citation text generation models beyond shallow evaluation metrics. ## conclusion citation text generation is a key task in scholarly document processing -yet prior work has been scattered across varying task definitions and evaluation setups. we introduced citebench: a benchmark for citation text generation that unifies four diverse task designs under a general definition, harmonizes the respective datasets, provides multiple baselines and a standard evaluation framework, and thereby enables a systematic study of citation text generation. our analysis delivered new insights about the baseline performance and the discourse composition of the datasets and model outputs. the baseline results show that simple extractive summarization models like lexrank perform surprisingly well. furthermore, we observe non-trivial ability for transfer learning among the baselines. the discourse analysis-based evaluation suggests that the models performing best in terms of rouge and bertscores capture the natural citation intent distribution in the generated texts. finally, our discussion outlines recommendations and promising directions for future work, which include detailed reporting of evaluation packages, the use of discourse-based qualitative metrics for evaluation, and further refinements to the citation text generation task definition. we argue that the current definitions for citation text generation overly simplify the complex process of composing a related work section, which hinders the development of successful real-world applications. in general, we view our work represents a crucial first step in addressing the extremely challenging problem of ai-assisted writing related work. we invite the community to build upon citebench by extending it with new datasets, trained models and metrics.