Cited Text Spans for Scientific Citation Text Generation (2024)

Abstract: An automatic citation generation system aims to concisely and accurately describe the relationship between two scientific articles. To do so, such a system must ground its outputs to the content of the cited paper to avoid non-factual hallucinations. Due to the length of scientific documents, existing abstractive approaches have conditioned only on cited paper abstracts . We demonstrate empirically that the abstract is not always the most appropriate input for citation generation and that models trained in this way learn to hallucinate. We propose to condition instead on the cited text span (CTS) as an alternative to the abstract. Because manual CTS annotation is extremely time- and labor-intensive, we experiment with distant labeling of candidate CTS sentences, achieving sufficiently strong performance to substitute for expensive human annotations in model training, and we propose a human-in-the-loop, keyword-based CTS retrieval approach that makes generating citation texts grounded in the full text of cited papers both promising and practical.

## introduction an automatic citation generation system aims to concisely and accurately describe the relationship between two scientific articles, a citing paper and a cited paper. due to the rigorous nature of academic research, even large language models (llms), such as the gpt family @xcite , still have to ground their outputs to the input cited papers to avoid non-factual hallucinations. given a set of papers to cite, prior extractive approaches @xcite @xcite @xcite selected salient sentences from the full text of the cited papers. however, existing abstractive approaches @xcite @xcite @xcite generate citation texts conditioned on only cited paper abstracts, since the full texts of the papers often exceed the length limit of fine-tuned transformer-based models @xcite . @xcite show that, when human judges are asked to rate the relevance of human-written gold citations with respect to the cited paper abstract (i.e. the generation system input), the scores are significantly lower than those of system-generated citations. we hypothesize that this discrepancy is caused by the gold citations referring to content from the cited paper body sections, not just the abstract, and these body sections are not available to either the generation model or the human judges. intuitively, the cited text span (cts), the specific span of the cited paper that the gold citation refers to, should be a better alternative to the abstract for grounding a citation. unfortunately, cts labels are difficult to obtain. existing manually-annotated sentence-level cts datasets @xcite @xcite suffer from two key issues: annotation is extremely labor-intensive, even for domain experts, so the datasets are small; and inter-annotator agreement is relatively low (cohen's κ of 0.16∼0.52). further, automatic cts retrieval systems trained on these datasets give low performance (@xmath0.2) and also use the gold citation text as input, making them inapplicable in a citation generation setting. given these obstacles, cts-based citation text generation is underexplored. in this work, we study scalable cts retrieval for citation text generation without expensive manual annotations. our main contributions are as follows: ## background and related work there are two main cts datasets: jaidka et al. @xcite proposed the cl-scisumm shared task and manually annotated cts, citation facets, and summaries for 364 citation texts that cite a set of 20 papers; abura'ed et al. ( 2020 ) manually annotated sentence-level cts for 20 related work sections. unfortunately, both suffer from low interannotator agreement, and automatic cts retrieval systems trained and evaluated on these datasets perform poorly @xcite @xcite . only a few studies have used cts for scientific document summarization. scisummnet (yasunaga et al., 2019) leveraged a rouge-based 1 https://github.com/jacklxc/ cts4citationtextgeneration cts retrieval approach @xcite to collect community-based evidence for summarizing a paper. @xcite trained a feature-based ensemble model for cts retrieval using scisummnet and used the retrieved cts for extractive related work generation. @xcite used bm25 and scibert @xcite to retrieve cts to provide more contexts for scientific paper summarization. however, all of these works used the target citation text as the query for cts retrieval, which cannot be used in the citation generation setting that we tackle in this paper. ## conclusion in this work, we show that cts-based citation text generation is more accurate and faithful than the abstract-based approaches used in prior work. we address the challenges that previously prevented cts from being used effectively for citation text generation. distant labeling, such as rougebased labeling, achieves sufficiently strong performance to substitute for expensive manual cts annotations, allowing us to build a dataset large enough to train a supervised cts retriever. to avoid the pitfall of using the gold citation text as the retrieval query, which has been used in prior work but would not be possible in a real application, we propose a human-in-the-loop, keyword-based cts retrieval approach that achieves strong performance in a practical and realistic setting. finally, our experimental results show that, while retrieving cts using only the citation context sentences is challenging and does not achieve the best performance using automated metrics, it is preferred by human judges and is a good direction for further investigation.