Predicting the presence of inline citations in academic text using binary classification (2023)

Abstract: Properly citing sources is a crucial component of any good-quality academic paper. The goal of this study was to determine what kind of accuracy we could reach in predicting whether or not a sentence should contain an inline citation using a simple binary classification model. To that end, we fine-tuned SciBERT on both an imbalanced and a balanced dataset containing sentences with and without inline citations. We achieved an overall accuracy of over 0.92, suggesting that language patterns alone could be used to predict where inline citations should appear with some degree of accuracy.

## introduction providing accurate, relevant citations is an essential part of academic writing. not only do citations allow authors to better contextualize the results of the paper, but they also lend credibility and authority to the claims made in the article. failing to give credit to existing research when credit is due, on the other hand, is taken to show a lack of academic integrity, and is strongly frowned upon by the academic community. appropriately adding citations, however, is not trivial: even humans sometimes struggle to determine where inline citations should go, and what should or should not be cited. this is particularly true in the case of junior academics and students @xcite @xcite . in the context of automatic text evaluation, determining where citations should go is even less straightforward. one way in which one could automatically determine whether a given paragraph requires (additional) inline citations is through automatic plagiarism detection systems. however, processing a document to determine whether some sections of it have been plagiarized can require a considerable amount of time, particularly if the document exceeds a certain length. building a plagiarism checker is also complicated, as the process requires scanning the full web for documents, and possibly obtaining access to research articles that might lay behind a paywall. finally, results might not always be accurate @xcite , as the checker might fail in finding similarities between concepts simply because sentences that are identical in meaning have been expressed through a different formulation. because of these downsides, we were interested in exploring how much mileage we could get out of a simple binary classification experiment trying to predict whether or not a given sentence should include an inline citation. in particular, we reasoned that it should be possible to predict at least to some extent whether a sentence should contain an inline citation simply by looking at the presence vs. absence of specific lexical cues. for example, verbs such as "claimed", nouns such as "authors" and phrases such as "as seen in" tend to appear together or in the vicinity of inline citations. the same holds true of some capitalized nouns (e.g. "attention", "minimalism"). ## preparing the data to determine what types of inline citation styles are used in different research disciplines, we randomly selected two articles for each of the following 18 research fields: medicine, biology, chemistry, engineering, computer science, physics, math, psychology, economics, political science, business, geology, sociology, geography, environmental science, art, history, philosophy. after analyzing these 36 articles, we concluded that most of the articles adopted the ieee, apa or the chicago reference styles. we first created an initial dataset consisting of 2000 research articles; these were randomly selected from the arxiv and pubmed datasets @xcite that are freely available on the huggingface datasets library @xcite ( https://huggingface.co/ datasets/scientific_papers). these 2000 articles were subsequently processed to discard articles with a citation pattern other than the ieee, apa or chicago reference styles. the pre-processing task of detecting inline citations was handled through a simple python script. using regular expressions, different kinds of citation styles were mapped to corresponding regex capture patterns. we started by writing regexes that would match the three citation styles that we identified as the most frequently used: ieee, apa and chicago. later on, we also decided to include the alpha bibtex style, as that appears to be quite frequently used in arxiv papers. the python script did the following: first, every given citation pattern was extracted from the article's plain text. then, the style with the highest capture count was set as the article's default style. this means that even when the extraction process found inline citations that matched a style that was not the article's primary citation style, the script was still able to identify the primary style. finally, the inline citations matching the primary style were substituted with an -add-citationtoken; this step is important as it allowed us to generalize across different referencing styles. if for some reason no citation style was detected, the token replacement failed, and the article was discarded from further analysis. we then created a second dataset by taking all the articles with ieee, apa or chicago as reference styles and by (i) breaking down the original text into sentences, and assigning each sentence to a separate entry, (ii) assigning different labels to entries containing inline citations and entries not containing inline citations, and (iii) removing the -add-citation-token throughout the dataset. this second dataset features 411'992 sentences (entries), of which 54'735 contain an inline citation (see ## classification model in our research, we intend to train a classification model that can determine whether a sentence should contain a citation (positive class) or not (negative class) depending on the text input. in the first column of the in recent years, bert-based language models @xcite have achieved state-of-theart performance in numerous nlp classification tasks. due to their pre-training on massive corpora and fine-tuning for a specific downstream purpose, these models can acquire accurate language representations. our inline citation dataset includes scientific data containing science-specific terminology. because of that, we decided to encode texts for the classification task using the bert architecture that has been pre-trained on scientific texts, i.e. the scibert model @xcite . exactly like bert, scibert contains 30k wordpiece tokens, but unlike bert its vocabulary is pertinent to the scientific area. in the scientific domain, scibert outperforms bert in a variety of tasks @xcite and achieves sota performance in multi-class text classification on the scicite dataset @xcite . it has been demonstrated that fine-tuned uncased scibert with scivocab followed by a linear layer produces the best results for scientific data @xcite or for citation context classification @xcite . therefore we use this model in each experiment. ## fine-tuning scibert research papers generally contain more sentences without inline citations than sentences with citations, which leads to having more examples for the "no citation" class. performing classification tasks using imbalanced datasets poses multi- figure 2 : roc curve on testing balanced dataset ple challenges, the most prominent being the bias towards the most represented class @xcite . there are multiple studies that try to counteract this phenomenon by bringing more balance in the distribution of classes within the same dataset (see for example @xcite or @xcite ). two well-known techniques in direction of balancing are undersampling and oversampling. undersampling however also presents drawbacks, the most important one being the loss of information that might be captured by the most represented class. with this in mind, we decided to run classification experiments on both the full (imbalanced) dataset and a more balanced subset of the dataset which we obtained by undersampling the data. we divided both the balanced and the imbalanced dataset into a training subset (60%), a validation subset (20%) and a test subset (20%), resulting in a "60:20:20" split. the split was then modified so that the proportion of positive (sentences containing a citation) to negative (sentences not containing a citation) texts in each subset would not be altered following the split (see next, we fine-tuned all scibert parameters end-to-end utilizing the training and validation subsets. for fine-tuning, we adhered primarily to the similar design and optimization decisions utilized in articles @xcite . we used the relu activation function in linear one-layer feed-forward classifier which inputs the last hidden state of the [cls] token. in other words, this last hidden state of the [cls] token is utilized as the sequence's features to feed the classifier. we experimented with numerous hyper ## discussion in our work, we always consider positive labels as a class of those input texts that contain an inline citation. this means that we always understand true-positives (tp) as correctly predicted texts that contain an inline citation (see graph 1 and graph 2). this is also analogous to the precision and recall calculations and the derived fscore in graphs 3 and 4, and the metrics in we report the results of our two experiments in since we used both balanced and imbalanced datasets, useful performance indicators include the area under the curve auc for the precision-recall curve pr or the receiver operating characteristic curve roc @xcite . figure 1 and figure 2 reveal that the roc curves are nearly comparable in both datasets, with the imbalanced dataset having a slightly lower auc value of 0.94 against that of 0.96 for the balanced dataset. for imbalanced data, however, a pr plot is advised @xcite ; our pr plots are depicted in figure 3 and 4 . the imbalanced dataset's pr curve follows a different path than the balanced dataset's pr curve, which is also reflected in its considerably lower auc value (=0.84) compared to that of the balanced dataset (=0.96). ## conclusion the goal of this paper was to determine how effective binary classification models can be at predicting whether or not sentences appearing in academic articles should contain an inline citation. to that end, we used regular expressions to identify inline citations in published research papers, and then created a dataset composed of 411k sentences, where approximately 54k contained inline citations. we then ran a fine-tuned scibert classifier on both a balanced and imbalanced dataset, achieving an overall accuracy of over 0.92. this result shows that language patterns alone could be used to predict the presence of inline citations in academic text with a reasonable degree of accuracy. we presented the problem as a binary clas-sification task on the sentence level, i.e. we only considered the target sentence and did not consider the context in which the sentence appeared, for example by also looking at the sentences appearing before and after the target sentence. taking into account the previous and the following sentence could be worthwhile in that some inline citations scope over multiple contiguous sentences rather than just refer to a single sentence (i.e. the concept of "citing area" first mentioned in @xcite ). the sentences contained in the inline citation dataset however are all sequential: they come in the same sequence as they were found in the original paper. this means that information on the context in which a given target sentence appears is already available in our dataset. this paves the path for further experiments that take contextual sentential information into account, such as using transformers to predict in which position inline citations should appear.