S ci BERT Sentence Representation for Citation Context Classification (2021)

Abstract: This paper describes our system (IREL) for 3C-Citation Context Classification shared task of the Scholarly Document Processing Workshop at NAACL 2021. We participated in both subtask A and subtask B. Our best system achieved a Macro F1 score of 0.26973 on the private leaderboard for subtask A and was ranked one. For subtask B our best system achieved a Macro F1 score of 0.59071 on the private leaderboard and was ranked two. We used similar models for both the subtasks with some minor changes, as discussed in this paper. Our best performing model for both the subtask was a finetuned SciBert model followed by a linear layer. This paper provides a detailed description of all the approaches we tried and their results.

## introduction recent years have witnessed a massive increase in the amount of scientific literature and research data being published online, providing revelation about the advancements in different domains. the introduction of aggregator services like core has enabled unprecedented levels of open access to scholarly publications. the availability of the full text of the research documents facilitates the possibility of extending the bibliometric studies by identifying the context of the citations @xcite . the shared task organized as part of the sdp 2021 focuses on classifying citation context in research publications based on their influence and purpose. subtask a aims at identifying the purpose of the citation. subtask a involves a multiclass classification of citations into one of six classes: background, uses, compare and contrast, motivation, extension, and future. subtask b aims at identifying the importance of the citation. it is a binary classification of citations into one of two classes: incidental and influential. ## related work in this section, we briefly describe other work in this and closely related areas. several machine learning-based frameworks have been developed in the past which attempt to classify scientific citation text based on its context. before the machine learning era, garfield (1965) introduced the idea of identifying the reasons for a particular citation and presented a list of 15 reasons why an author would include a citation. there have been other works and discussions about the classification of citations and its importance in this classification in developing models of referencing @xcite . however, despite these works, the literature on automating the task of classifying citations has been limited. @xcite treated the citation classification as a task of sentence categorization. they extracted a sentence that incorporated citations and then applied manually curated lexical and grammar rules to assign categories to the citations. @xcite used supervised machine learning techniques to classify citations into 12 categories. the authors annotated 2,829 citation contexts from 116 articles, using linguistic features, including the cue phrases. @xcite used algorithms like linear discriminant analysis(lda) and support vector machine(svm) to develop an automated citation classifier specific to the domain of biomedical text classification. @xcite proposed structural scaffolds, a multitask model to incorporate structural information of scientific papers into citations for effective classification of citation intents. their model did not rely on external linguistic resources or handengineered features as done in previous methods. they also introduced a new dataset of citation in-tents (sci-cite) which covers multiple scientific domains. a pretrained language model based on bidirectional encoder representations from transformers(bert) @xcite , scibert (beltagy et al., 2019) was introduced, which leverages unsupervised pretraining on a large multi-domain corpus of scientific publications and can be used for multiple downstream tasks like this one. ## methodology bert-based models have shown significant performance in different natural language understanding(nlu) tasks. this paper uses sentence representation from the bert-based model and experiments with different machine learning and deep learning models. the training data contains nine different fields viz. unique identifier, coreid of citing paper, citing paper title, citing paper author, cited paper title, cited paper author, citation context, citation class label, citation influence label. for a machine learning or deep learning solution, we identify that only the citation context is significant. we also experiment by adding cited paper title information with citation context and observe a drop in macro f1 score. thus, for all our experiments, unless specified otherwise, we use citation context only. ## dataset the labeled training dataset contains 3000 instances. the training data includes nine different fields viz. the unique identifier, coreid of citing paper, citing paper title, citing paper author, cited paper title, cited paper author, citation context, citation class label, citation influence label. we randomly divided 500 instances into validation data and used the remaining 2500 instances for training the model. it was verified that the validation and training data shared similar class distribution. during inference, we trained the model by combining training and validation data. fig 1 shows the class distribution for sub-task a and sub-task b. the data is imbalanced, so we use the weighted loss function. the balanced heuristic is inspired by @xcite . we use sklearn implementation to compute class weight foot_0 . the weighted loss function outperformed the unweighted loss function for the model described in section 3.1, as shown in ## results and discussion ## conclusion and future work this work shows how scibert embeddings can capture nuances of scientific documents, and simple deep learning and machine learning models could give competitive results. despite a small dataset, good results could be achieved. there is much noise in the training data, like numbers, html elements, etc. a more detailed study of features of the training data could improve the results. since the dataset is small, data augmentation techniques could be used as future work.