S mart C ite C on: Implicit Citation Context Extraction from Academic Literature Using Supervised Learning (2020)

Abstract: We introduce SmartCiteCon (SCC), a Java API for extracting both explicit and implicit citation context from academic literature in English. The tool is built on a Support Vector Machine (SVM) model trained on a set of 7,058 manually annotated citation context sentences, curated from 34,000 papers from the ACL Anthology. The model with 19 features achieves F1=85.6%. SCC supports PDF, XML, and JSON files out-of-box, provided that they are conformed to certain schemas. The API supports single document processing and batch processing in parallel. It takes about 12–45 seconds on average depending on the format to process a document on a dedicated server with 6 multithreaded cores. Using SCC, we extracted 11.8 million citation context sentences from ~33.3k PMC papers in the CORD-19 dataset, released on June 13, 2020. We will provide continuous supplementary data contribution to the CORD-19 and other datasets. The source code is released at https://gitee.com/irlab/SmartCiteCon .

## introduction citations are ubiquitous in scientific publications. with proper citations, statements in research papers are supported by existing works, and readers obtain relevant information beyond the current paper. citations also form graphs, which provide unique models for ranking, sentimental classification, and plagiarism detection. therefore, citation analysis plays an important role in helping to understand the deep connection between literature. accurate citation context recognition is the prerequisite of many downstream applications. recently, citation context, the text segment that appears around the citation mark in the body text, has been used for enhancing and improving keyphrase extraction @xcite and document summarization @xcite . there are two types of citation context. explicit citation contexts (ecc) are sentences containing ci-tation marks. each citation thus corresponds to one explicit citation context sentence. implicit citation contexts (icc) are sentences that are semantically relevant to the cited articles but do not contain citation marks. icc may appear before or after but may not immediately precede or follow the ecc sentence. one paper could be cited multiple times and each time may have different citation contexts. in the example below, the ecc, containing the citation mark " @xcite )", is highlighted in green. the icc sentences are highlighted in yellow. the nonhighlighted sentence is not a citation context for the given citation. we investigate the impact of semantic constraints on sta tistical word alignment models as prior knowledge. in @xcite ), bilingual se mantic maps are constructed to guide word alignment. the framework we proposed seamlessly integrates derived semantic similarities into a statistical word alignment model. and we extended monolingual latent se mantic analysis in bilingual application. most existing tools extract ecc, i.e., sentences containing citation marks. although the results are highly relevant, the method omits icc if the author uses multiple sentences to summarize the results. to our best knowledge, there are no offthe-shelf tools dedicated to icc extraction. unlike ecc sentences with citation marks, the lack of explicit marks makes citation context recognition challenging. in this work, we develop a java api that implements a supervised machine learning model trained on 7058 manually labeled sentences to extract both ecc and icc. the model achieves an f 1 -measure of 85.6%. the java api can be deployed on a local machine or as a web service. several citation context extraction methods have been developed. in @xcite , the scope of the citation context covered several consecutive sentences before and after the sentences with citation marks (i.e., citation sentence), identified based on a referential relationship with the citation sentence. in another work, markov model was used for identifying citation context @xcite . @xcite described a support vector machine (svm) and maximum entropy (me) model for identifying citation sentences using shallow features such as proper nouns and contextual classification of the previous and next sentence @xcite . they found that the performances of svm and me do not exhibit significant differences. the positive samples were selected as sentences including citation marks using regular expression matching, icc extraction was not covered. parscit is an open-source software commonly used for citation parsing and citation context extraction @xcite . parscit parses citation strings using a conditional random field (crf) model. the citation context extraction was performed by extracting a fixed window size of 200 characters on either side of the citation mark. gro-bid @xcite is a library to extract information from scholarly documents. the documentation reports the f 1 -measure of citation context resolution is around 75%, which counts both the correct identification of citation marks and its correct association with bibliographic references. in summary, existing citation context extraction tools focus on ecc but ignore icc, the latter of which includes more sentences semantically related to the cited papers. ## supervised machine learning model our system is based on a supervised machine learning model proposed in @xcite , which classifies a sentence into icc and non-icc. we adopted the ground truth built by @xcite containing 130 articles from 34,000 computational linguistics conference proceedings in acl anthology. the original pdf files were converted to xml format using ocr @xcite . the training set was labeled by 13 graduate students majoring in information management. the labeling agreement was tested using cohen's kappa coefficient (κ = 0.937). the fi-nal ground truth contains 3,578 positive and 3,480 negative samples. the preprocessing uses apache opennlp for sentence segmentation. citation marks are identified using regular expressions. citation marks are then removed, and the original sentences are converted into regular sentences for following analyses such as part-of-speech (pos) tagging. each sentence is represented by up to 19 features of four types ( ## architecture the scc system completes the extraction in four steps (figure 1 ): (1) file type recognition, (2) preprocessing, (3) feature extraction, and (4) sentence classification. the output is a json file containing ecc and icc and other citation-related information. the api was written based on the springboot framework in java. the machine learning model was implemented with weka. ## scc api performance we test the scc api on a computer with 16gb ram and an intel core i7-8570h cpu@2.20ghz, which has 6 hyperthreaded cores (12 threads in total). in a preliminary experiment, we compare the runtime of processing 10 xml documents using a single process under different jvm heap sizes. the runtimes corresponding to 12gb, 8gb, 4gb, and 2gb are 23.8 min, 12.7 min, 7.3 min, and 7.6 min, respectively. higher heap does not boost processing speed probably due to garbage collection. based on the results, in the following experiments, 4gb heap was allocated to jvm. the experiments were set to extract citation context from randomly selected documents in different formats. the datasets include 10 pdf documents from plos one, 10 xml documents corresponding to the pdf documents, and 10 json documents from the cord-19 dataset. we monitor the system using jprofiler (version 11) and calculate the median time it takes for processing one document as we vary the number of processes n p . figure 2 shows that the cpu utilization increases from about 10% and saturates when n p reaches 8. the memory utilization climbs up slowly as n p increases but are mostly well below the maximum allocated heap, because processed documents are not stored in memory anymore. the average processing time for all three types gradually decreases as n p increases but in general, it takes longer to process pdf files than json and xml files. the maximum and minimum processing time are shown in ## lessons learned the results in ## conclusions and future works we developed smartcitecon (scc), a java api to extract explicit and implicit citation context from academic literature. the api implements an svm model achieving an f 1 = 85.6%. scc accepts xml (in plos one schema or grobid schema), pdf, and json (in s2orc schema) formats. the output of scc is a json file containing marked citation contexts and paper metadata if available. we applied scc on the pmc subset of the cord-19 dataset and obtained about 11.8 million citation context sentences in which 10.2 million are implicit citation context. one limitation of scc is that the model was trained on papers in computational linguistics, so more careful evaluation and feature distribution analysis should be performed when applying the model to other domains. in the future, we will explore word embedding models to enrich semantic features and improve scalability by overcoming performance bottlenecks.