Citation Context Classification: Critical vs Non-critical (2022)

Abstract: Recently, there have been numerous research in Natural Language Processing on citation analysis in scientific literature. Studies of citation behavior aim at finding how researchers cited a paper in their work. In this paper, we are interested in identifying cited papers that are criticized. Recent research introduces the concept of Critical citations which provides a useful theoretical framework, making criticism an important part of scientific progress. Indeed, identifying critics could be a way to spot errors and thus encourage self-correction of science. In this work, we investigate how to automatically classify the critical citation contexts using Natural Language Processing (NLP). Our classification task consists of predicting critical or non-critical labels for citation contexts. For this, we experiment and compare different methods, including rule-based and machine learning methods, to classify critical vs. non-critical citation contexts. Our experiments show that fine-tuning pretrained transformer model RoBERTa achieved the highest performance among all systems.

## introduction in scientific papers, citations acknowledge the sources and help the reader to find more information about the citation context. citations are also an important indicator exploited to identify significant publications in a specific scientific field @xcite . they are used for different purposes, e.g. referring to state of the art, to a specific method or result, and they reflect how authors frame their work and this diversity impacts future academics' adoption @xcite . according to @xcite , the study of critical citation appears to give an applicable theoretical framework, making criticism a vital phenomenon for scientific development. we believe that classifying citation contexts into critical/noncritical categories could be essential to downstream process, such as identifying scientific claims or observing controversial papers. @xcite identifies three different functions for critical citation context : "to criticize," "to compare," and "to question" where : there have been numerous researches on citation analysis in nlp, with for instance determining citation sentiments @xcite . in addition to citation sentiment, there have been research to define citation function which refers to the specific purpose a citation plays with respect to the citing paper @xcite @xcite . these researches have been conducted to find the real reason behind the citation. nevertheless, how citation might be utilized to point out criticism and encourage correction have not been studied yet. given a set of citation contexts, our work aims at determining critical ones using nlp methods. first, we present the construction process of the corpus, which contains citation contexts annotated with critical and non-critical labels. then, we experiment different methods to classify citation contexts into critical/non-critical labels using our constructed corpus. indeed, we compare and discuss rule-based methods and machine learning ones. ## related works in this section, we present different existing works for citation analysis. some of them are rule-based methods, while others are based on machine learning methods. since 2000, several researches on automation citation classification have been using rule-based approaches @xcite @xcite . the rule creation process is generally composed of 2 steps. in the first step, cue words/phrases are extracted from dataset samples. in the second step, rules are created based on the extracted cue words/phrases. these rules are the bases to classify citation contexts. for instance, in (avanço, 2020) a rule-based method is used to identify negative or contradictory citation contexts. the authors built citaneg corpus ( more recently, several approaches relying on machine learning have been proposed. for example, @xcite used ibk, a form of k-nearest neighbor (knn), to classify citation contexts into 4 polarities (weakness, positive, contrast and neutral) and obtained an f1-score of 0.61 using athar corpus ( ## experimental setup we present our methods used for critical/non critical classification in section 3.1. then, we describe our corpus in section 3.2. ## results and discussion to improve the quality of the rb+ system, we need to add more rules to identify critical citation contexts. for instance, we could analyze in depth the grammar or cue words/phrases to define more patterns for critical citation contexts. we could also analyze the concept "to question" of critical citation context that has not been taken into account by our rulebased system rb+ yet. if we take a look at confusion matrix of lr and cnn systems in figure 1d and figure 1c respectively, the number of misclassified non-critical examples is greater than the number of misclassified critical ones. it could be explained by imbalanced training set. indeed, critical class represents around 60% of the training set. being aware of imbalanced training set, we might enhance cnn and lr performances by assigning class weights while training. however, the cnn model does not exhibits a strong bias towards a particular class, so it is likely that a class weighting strategy would have a marginal impact on the performance. f-roberta (figure 1a ) predicts well noncritical examples, only 1 non-critical and 3 critical examples are misclassified. this could be explained by the class weights' assignment while training f-roberta in order to deal with class imbalance. to go further, we analysed these 4 misclassified examples. one of them is "it is based on modal logic and owes much to the work of blackburn 1994." has been classified critical while it should not. if we check out the linguistic aspect of the citation context above, the use of the word "owes" may reflect critical citation aspect. but, we still can argue if "owes" here did not be used to criticize the cited paper, it seems like an incomplete context. in this case, we might need more investigation of corpus. the misclassified critical citation contexts are reported in critical citation contexts doc_1: the morphological processing in pair-class @xcite ) is more sophisticated than in @xcite . doc_2: in particular, we showed that using a general purpose machine translation (mt) system such as systran, or a general purpose parallel corpus -both of which perform very well for news stories (peters, 2003) -dramatically fails in the medical domain. doc_3: in particular, these problems affect the processing of predicate argument structures annotated in @xcite or framenet @xcite . ## conclusion and future work in this paper, we were interested in identifying critical citation contexts in scientific papers. we proposed and tested five methods for citation context classification into critical/non-critical labels. the methods rb and rb+ were rule-based. the three others, lr, cnn and f-roberta, were machine learning based. we also built a corpus to evaluate and compare these methods. our task-specific corpus was composed of 2643 citation contexts labeled as being critical or non-critical. machine learning based systems outperformed rulebased ones. the best system f-roberta gave 98.84% of accuracy and 98.52% of f1-score. the performances could be explained by the use of transfer learning in f-roberta. class weight assignment while training might also explain the good accuracy of f-roberta's performance compared to other systems, since our training set was imbalanced. some improvements can be made to the proposed systems. in particular, we will assign class weights while model training to solve imbalanced datasets. moreover, we could operate the data itself (and not the model) to balance the corpus by applying sampling methods either oversampling or undersampling. dealing with scientific documents, it could be crucial to train our best system f-roberta, initially trained on standard corpora, on scientific texts by using for example scibert embeddings @xcite . another perspective consists on expanding corpus. to go further, we would extend this work of identifying critical citation contexts in nlp field and study field portability. indeed, we would identify critical citations in other fields, such as biology or medicine.