BACO : A Background Knowledge- and Content-Based Framework for Citing Sentence Generation (2021)

Abstract: In this paper, we focus on the problem of citing sentence generation, which entails generating a short text to capture the salient information in a cited paper and the connection between the citing and cited paper. We present BACO, a BAckground knowledge- and COntent-based framework for citing sentence generation, which considers two types of information: (1) background knowledge by leveraging structural information from a citation network; and (2) content, which represents in-depth information about what to cite and why to cite. First, a citation network is encoded to provide background knowledge. Second, we apply salience estimation to identify what to cite by estimating the importance of sentences in the cited paper. During the decoding stage, both types of information are combined to facilitate the text generation, and then we conduct a joint training for the generator and citation function classification to make the model aware of why to cite. Our experimental results show that our framework outperforms comparative baselines.

## introduction a citation systematically, strategically, and critically synthesizes content from a cited paper in the context of a citing paper @xcite . a paper's text that refers to prior work, which we herein refer to as citing sentences, forms the conceptual basis for a research question or problem; identifies issues, contradictions, or gaps with state of the art solutions; and prepares readers to understand the contributions of a citing paper, e.g., in terms of theory, methods, or findings @xcite . writing meaningful and concise citing sentences that capture the gist of cited papers and identify connections between citing and cited papers is not trivial @xcite . learning how to write up information about related work with appropriate and meaningful citations is particularly challenging for new scholars @xcite . to assist scholars with note taking on prior work when working on a new research problem, this paper focuses on the task of citing sentence generation, which entails identifying salient information from cited papers and capturing connections between cited and citing papers. with this work, we hope to reduce scientific information overload for researchers by providing examples of concise citing sentences that address information from cited papers in the context of a new research problem and related write up. while this task cannot and is not meant to replace the scholarly tasks of finding, reading, and synthesizing prior work, the proposed computational solution is intended to support especially new researchers in practicing the process of writing effective and focused reflections on prior work given a new context or problem. a number of recent papers have focused on the task of citing sentence generation @xcite @xcite , which is defined as generating a short text that describes a cited paper b in the context of a citing paper a, and the sentences before and after the citing sentences in paper a are considered as context. however, previous work has mainly utilized limited information from citing and cited papers to solve this task. we acknowledge that any such solution, including ours, is a simplification of the intricate process of how scholars write citing sentences. given this motivation, we explore two sets of information to generate citing sentences, namely background knowledge in the form of citation networks, and content from both citing and cited papers, as shown in figure 1 . using citation networks was inspired by the fact that scholars have analyzed such networks to identify the main themes and research developments in domain areas such as information sciences @xcite , busi-figure 1 : an example from our dataset (source: acl anthology network corpus @xcite . the red text in the citing paper is the citing sentence, and the special token #refr indicates the citation of the cited paper. our framework aims at capturing information from two perspectives: background knowledge and content. the background knowledge is learned by obtaining structural features of the citation network. the content information entails estimated sentence salience (higher salience is highlighted by darker color) in the cited paper and the corresponding citation function of the cited paper to the citing paper. we use the content of citing and cited papers as a second set of features to capture two more in-depth content features: (1) what to cite -while the overall content of a cited paper needs to be understood by the authors of the citing paper, not all content is relevant for writing citing sentences. therefore, we follow the example of estimating salient sentences @xcite and use the predicted salience to filter crucial information that should be integrated into the resulting citing sentence; (2) why to cite -we define "citation function" as an approximation of an author's reason for citing a paper @xcite . a number of previous research on citation functions has used citing sentences and their context for classification @xcite . our paper involves citation functions into citing sentence generation so that the generated citing sentences can be coherent given their context, and can still contain the motivation for a specific citation. in this paper, we propose a background knowledge-and content-based framework, named baco. specifically, we encode a citation network based on citation relations among papers to obtain background knowledge, and the given citing and cited papers to provide content information. we extend a standard pointer-generator @xcite to copy words from cited and citing papers, and determine what to cite by estimating sentence salience in the cited paper. the various pieces of captured information are then combined as the context for the decoder. furthermore, we extend our framework to include why to cite by jointly training the generation with citation function classification and facilitate the acquisition of the content information. as for the dataset, we extended the acl anthology network corpus (aan) @xcite with extracted citing sentences by using regex. we then hand-annotated the citation functions on a subset of the dataset, and trained a citation function labeling model based on scibert @xcite . the resulting labeling model was then used to automatically label the rest data to build a large-scale dataset. we summarize our contributions as follows: ## related work several studies on citing sentence generation have used keyword-based summarization methods @xcite @xcite . to that end, they built keyword-based trees to extract sentences from cited papers as related work write-ups. these studies have two limitations: first, since related work sections are not simply (chronological) summaries of cited papers, synthesizing prior work in this manner is insufficient. second, extractive summarization uses verbatim content from cited papers, which implies intellectual property issues (e.g., copyright violations) as well as ethical problems, such as a lack of intellectual engagement with prior work. alternatively, abstractive summarization approaches, such as methods based on linear programming @xcite and neural seq2seq methods @xcite , have also been explored. these approaches mainly focus on utilizing papers' content information, specifically on the text of cited papers directly. a recent paper that went beyond summarizing the content of cited papers @xcite ) used a multi-source, pointer-generator network with a cross attention mechanism to calculate the attention distribution between the citing sentences' context and the cited paper's abstract. our paper is based on the premise that citation network analysis can provide background knowledge that facilitates the understanding of papers in a field. prior analyses of citation networks have been used to reveal the cognitive structure and interconnectedness of scientific (sub-)fields @xcite , and to understand and detect trends in academic fields @xcite . network analysis has also been applied to citation networks to identify influential papers and key concepts @xcite , and to scope out research areas. while previous studies have shown that using text from citing papers is useful to generate citing sentences, the benefit of other content-based features of a citation (e.g., reasons for citing) is insufficiently understood @xcite . extant literature on citation context analysis @xcite , which focused on the connections between the citing and cited papers with respect to purposes and reasons for citations, has found that citation function @xcite is an important indicator of why a paper chose to cite specific paper(s). based on a content analysis of 750 citing sentences from 60 papers published in two prominent physics journals, @xcite identified 11 citation functions, such as questioned, affirmed, or refuted cited paper's premises. similarly, @xcite qualitatively coded the citation context of 30 articles on high energy physics, finding 10 citation functions grouped into 5 pairs: conceptual-operational, organic-perfunctory, evolutionary-juxtapositional, confirmative-negational, valuable-redundant. citation context analysis has also been used to study the valence of citing papers towards cited papers @xcite by classifying citation context as positive, negative, or neutral. in this paper, we adopt abu-jbara et al. (2013)'s definition of a positive citation as a citation that explicitly states the strength(s) of a cited paper, or a situation where the citing paper's work is guided by the cited paper. in contrast to that, a negative citation is one that explicitly states the weakness(es) of a cited paper. a neutral citation is one that objectively summarizes the cited paper without an additional evaluation. in addition to these three categories, we also consider mixed citation contexts @xcite , which are citations that contain both positive and negative evaluations of a cited papers, or where the evaluation is unclear. given that our paper is a first attempt to integrate citation functions into citing sentence generation, we opted to start with a straightforward valence category schema before exploring more complex schemas in future work. ## dataset and annotation we first extended the aan foot_0 @xcite with the extracted citing sentences using regex. we followed the process in @xcite to label 1,200 randomly sampled citing sentences with their citation functions. the mark-up was done by 6 coders who were provided with definitions of positive, negative, neutral, and mixed citation functions, and ample examples for each valence category. our codebook including definitions and examples of citation functions is shown in our labeling model was built upon scibert @xcite , a pre-trained language model based on bert @xcite but trained on a large corpus of scientific text. we added a multilayer perceptron (mlp) to scibert, and finetuned the whole model on our dataset. as for the input, we concatenated each citing sentence with its context in the citing paper, and inserted a special tag [cls] at the beginning and another special tag [sep] to separate them. the final hidden state that corresponded to [cls] was used as the aggregate sequence representation. this state was fed into the mlp, followed by the softmax function for predicting the citation function of the citing sentence. we report details of test results and dataset statistics in the appendix, section a.1. ## methodology our proposed framework includes an encoder and a generator, as shown in figure 2 . the encoder takes the citation network and the citing and cited papers as input, and encodes them to provide background knowledge and content information, respectively. the generator contains a decoder that can copy words from citing and cited paper while retaining the ability to produce novel words, and a salience estimator that identifies key information from the cited paper. we then trained the framework with citation function classification to enable the recognition of why a paper was cited. ## conclusions and future work we have brought together multiple pieces of information from and about cited and citing papers to improve citing sentence generation. we integrated them into baco, a background knowledge-and content-based framework for citing sentence generation, which learns and uses information that relate to (1) background knowledge; and (2) content. extensive experimental results suggest that our framework outperforms competitive baseline models. this work is limited in several ways. we only demonstrated the utility of our model within the standard rnn-based seq2seq framework. secondly, our citation functions scheme only contained valence-based items. finally, while this method is intended to support scholars in practicing strategic note taking on prior work with respect to a new literature review or research project, we did not evaluate the usefulness or effectiveness of this training option for researchers. in future work, we plan to investigate the adaptation of our framework into more powerful models such as transformer @xcite . we also hope to extend our citation functions scheme beyond valence of the citing sentences to more fine-grained categories, such as those outlined in @xcite and @xcite .