Automatic Generation of Citation Texts in Scholarly Papers: A Pilot Study (2020)

Abstract: In this paper, we study the challenging problem of automatic generation of citation texts in scholarly papers. Given the context of a citing paper A and a cited paper B, the task aims to generate a short text to describe B in the given context of A. One big challenge for addressing this task is the lack of training data. Usually, explicit citation texts are easy to extract, but it is not easy to extract implicit citation texts from scholarly papers. We thus first train an implicit citation extraction model based on BERT and leverage the model to construct a large training dataset for the citation text generation task. Then we propose and train a multi-source pointer-generator network with cross attention mechanism for citation text generation. Empirical evaluation results on a manually labeled test dataset verify the efficacy of our model. This pilot study confirms the feasibility of automatically generating citation texts in scholarly papers and the technique has the great potential to help researchers prepare their scientific papers.

## introduction a scientific paper usually needs to cite a lot of reference papers and introduce each reference paper with some text. in this study, the text describing a reference paper is called citation text. a researcher usually needs to find relevant papers he wants to cite and write some text to introduce them when writing a scientific paper. however, the process of writing citation texts is tedious and time-consuming. in order to reduce the burden of researchers, we propose and try to address the task of automatic citation text generation. automatic generation of citation texts in scholarly papers is a challenging and meaningful task, however, there are very few studies investigating this problem. given a cited paper b and the context in a citing paper a (i.e., the sentences before and after a specific position in paper a), the task aims to generate a short text to describe b with respect to the given context in a. the task is like the task of scholarly paper summarization @xcite @xcite . both of the two tasks aim to produce a text to describe the cited paper b. the major difference between the two tasks is that the citation texts reflect not only the salient content of b, but also the context of a. different citing papers usually have different descriptions of the same cited paper. sometimes one paper may cite another paper several times in different positions but give different descriptions because the specific contexts are different. another difference between the two tasks is the length of the text. a citation text is usually much shorter than a paper summary. generally, citation text generation can be considered as a task of generating a very short summary of paper b given the context of paper a. the difficulty lies in that given different a or different contexts of a, the task aims to produce different citation texts for the same b. most commonly, the citation text is a single sentence, but sometimes it may consist of several sentences @xcite @xcite . like (small, 2011), we define citation text as a block of text composed of one or more consecutive sentences surrounding the reference sign. each citation sentence can be classified as explicit or implicit @xcite @xcite . explicit citation is a citation sentence that contains explicit reference to the cited paper. an implicit (or non-explicit) citation sentence appears around the explicit citation sentence and it does not attach any explicit reference to the cited paper but supplies additional information about the cited paper. the citation text generation task in this study aims to generate both explicit and implicit citation sentences. we build a citation text generation dataset based on the acl anthology network corpus (aan) @xcite . we first perform human annotation and get 1,000 citation texts (including explicit and implicit citation sentences). we randomly select 400 citation texts as test set, and use the other 600 citation texts to first train a citation text extraction model and then use the extraction model to automatically extract many more citation texts to build a large-scale training dataset. with the training dataset we construct, we can train our citation generation model. in this paper, we use pointer-generator network @xcite as the baseline model. we believe that the key to dealing with citation text generation problem is modelling the relationship between the context of citing paper a and the content of cited paper b. so we encode the context of paper a and the abstract of paper b separately, and add cross attention mechanism by making context and abstract attend to each other. we call our model multi-source pointergenerator network with cross attention mechanism. the evaluation results show that our model outperforms the baseline models. our contributions are summarized as follows: evaluation results demonstrate the efficacy of our proposed model. ## related work firstly, we introduce some studies on citation extraction. @xcite proposed a method based on coreference-chains for citation extraction. @xcite first independently trained a separate hmm for each citation in the article and then performed a constrained joint inference to label non-explicit citing sentences. @xcite proposed a framework based on probabilistic inference to extract implicit citations. @xcite proposed an unsupervised approach which is based on topic modeling and word embedding for implicit citation extraction. @xcite introduced method based on neural network but it did not give out convincing evaluation results. a few studies have investigated the task of summarizing single scholarly paper, i.e., single document summarization in the scientific domain, which is relevant to the citation text generation task. early works include @xcite @xcite , and they tried to use various features specific to scientific articles for summary extraction. later on, citation information has shown its usefulness for scientific paper summarization @xcite @xcite @xcite . several benchmark tests have been set up for scientific summarization, including tac 2014 biomedical summarization track and the cl-scisumm shared task @xcite . a few other studies have investigated the task of summarizing multiple scholarly papers, i.e., multi-document summarization in the scientific domain @xcite @xcite . related work generation is a special case of multi-document scientific summarization @xcite @xcite . however, the above related work about scholarly paper summarization is different from the task of citation text generation, which aims to generate a usually very short text to describe the cited paper in the given context of the citing paper. ## problem and corpus formally, given a citing paper a, a cited paper b and the context c in a, the task aims to generate the citation text t to describe b. the context c refers to the sentences surrounding the target citation text in a and it is provided to distinguish different mentions of b in different positions of a. the following example shows a paragraph of @xcite and this article cites paper @xcite . in this example, a refers to @xcite and b refers to @xcite . the sentence underlined (i.e., the second sentence) is an explicit citation, and the sentence in italics (i.e., the third sentence) is an implicit citationand both of them compose the citation text. the remaining two sentences (i.e., the first and last sentences) compose the context c of a. the phrase in bold which indicates the explicit citation to paper b is called reference sign. and the explicit citation text can be defined as the sentence with a reference sign to the cited paper. the implicit citation text can be defined as the sentences that provide information about the cited paper but do not have any reference sign. in this study, we build a citation generation dataset based on the acl anthology network corpus (aan) @xcite . the acl anthology is a collection of papers from the computational linguistics journal, and proceedings from acl conferences and workshops. in particular, we download and use the 2014 version of the aan corpus which includes almost 23594 papers. after removing papers containing many garbled characters and papers without abstracts, there remains 16675 papers. the metadata of each paper and the paper citation network have been extracted and stored. we find all the mentions of each reference paper in a citing paper by using manually designed regular expressions to match the corresponding reference signs. lastly, we extract 86052 explicit citations for further use. ## implicit citation text extraction model after the annotation process, we have 400 citation texts as test dataset and 600 citation texts for training. however, we need large-scale training data to train a feasible citation text generation model. so we decide to use the 600 human annotated citation texts to train an implicit citation text extraction model to expand our training dataset. we treat implicit citation text extraction as a sequence labeling problem and use bert @xcite to deal with this problem. we add a classification layer on the final hidden representation of bert and fine-tune the whole model on our dataset. we concatenate all the candidate sentences, the explicit citation sentence and the abstract of the cited paper as the input of bert. we add a special tag '[s]' at the beginning of all sentences, a special tag '[explicit]' at the beginning of the explicit citation sentence and a special tag '[abs]' at the be our model is compared with these baseline models: all one: it labels all candidate sentences with 1. random: it labels all candidate sentences randomly. cosine sim: it first uses bag of words model to represent all texts as vectors. then it calculates the cosine similarity between candidate sentence and cited paper's abstract, and the cosine similarity between candidate sentence and the explicit citation sentence. when the two similarities are both greater than the threshold, the sentence is labelled with 1. w2v sim: this model is also based on similarity. the similarity in this model is calculated based on word2vec model. with two sequence of words, it first gets the corresponding two sequences of results of all these baseline models are shown in as shown in these tables, our extraction model outperforms all the baseline models. the f-value of our extraction models with α=0.1 and α=0.9 are very close. this indicates that they have close performance. the precision of extraction model with α=0.9 is higher, while the recall of extraction model with α=0.1 is higher. so we can get two different extraction models with two different α. and with the two different extraction models, we can construct two different datasets for further training citation generation model. to get the two different datasets, we use all 600 data to train two final extraction models. we call the extraction model with α=0.1 ext α=0.1 and call the extraction model with α=0.9 ext α=0.9 . the results on external test data when using full training data are shown in ## final evaluation datasets with the two implicit citation extraction models we trained in the previous section, we construct three datasets for experiments. in each dataset, a data example is a triple: [citing paper's context, cited paper's abstract, gold citation text]. the first dataset is an explicit citation text generation dataset (explicit dataset). the gold citation text in the training data and test data is single explicit citation sentence. note that the explicit citation sentences in the training data are automatically extracted with rules and the explicit citation sentences of test data are human annotated. the second dataset is a full citation text generation dataset. the gold full citation texts of test data are human annotated. the gold full citation text of training data is constructed as follows: the gold explicit citation text is extracted with rules and the gold implicit citation text is extracted with ext α=0.1 . this extraction model gets higher recall, so we call this dataset high-recall full citation text generation dataset (hr dataset). the third dataset is also a full citation text generation dataset, and it is constructed in the same way with the second dataset except that the gold implicit citation text of training data is extracted with ext α=0.9 and we call it high-precision full citation text generation dataset (hp dataset). the cited paper's abstract in all the three datasets refers to the abstract of the cited paper b. we use it to represent the content of paper b because the whole article is too long to encode. the citing paper's context in all the three datasets refer to the sentences around the gold citation text in citing paper a. we take three sentences before the gold citation text and three sentences after it as the context. note that all the context sentences must be in the same section as the gold citation text. finally, we have three datasets for experiments: the average lengths of explicit citation texts in the training and test sets are 29.64 words and 27.14 words, respectively. ## citation generation model our citation text generation model is a multisource pointer-generator network with cross attention mechanism. because the citation generation task has two input sequences, we use two encoders to encode them separately and allow the model to copy words from both input sequences. such a multi-source pointer-generator network does not have the ability to model the relationship between two input sequences, so we add a cross attention mechanism on them. the cross attention mechanism calculates the attention distribution of every word to the other sequence of words. these attention distributions are used to help the decoder. we believe that the citing paper's context can tell the model what information in cited paper's abstract is important and vice versa. the structure of the whole model is shown is figure 1 . ## conclusion and future work in this paper we investigate the challenging task of automatic generation of citation texts in scholarly papers. we annotate a dataset and train an implicit citation extraction model to automatically enlarge the training data. we then propose the multi-source pointer-generation network with cross attention mechanism to deal with this task. empirical evaluation results on three datasets verify the efficacy of our proposed method. in future work, we will consider introducing more information like the citation texts to the cited paper in other papers to help the generation.