IITP - CUNI @3 C : Supervised Approaches for Citation Classification (Task A) and Citation Significance Detection (Task B ) (2021)

Abstract: Citations are crucial to a scientific discourse. Besides providing additional contexts to research papers, citations act as trackers of the direction of research in a field and as an important measure in understanding the impact of a research publication. With the rapid growth in research publications, automated solutions for identifying the purpose and influence of citations are becoming very important. The 3C Citation Context Classification Task organized as part of the Second Workshop on Scholarly Document Processing @ NAACL 2021 is a shared task to address the aforementioned problems. In this paper, we present our team, IITP-CUNI@3C’s submission to the 3C shared tasks. For Task A, citation context purpose classification, we propose a neural multi-task learning framework that harnesses the structural information of the research papers and the relation between the citation context and the cited paper for citation classification. For Task B, citation context influence classification, we use a set of simple features to classify citations based on their perceived significance. We achieve comparable performance with respect to the best performing systems in Task A and superseded the majority baseline in Task B with very simple features.

## introduction citations are crucial in a research paper and the community for various reasons, including scientific and administrative. over the years, citation analysis techniques are used to track research in a field, discover evolving research topics, and measure the impact of research articles, venues, researchers, etc. correctly identifying the intent of the citations finds applications ranging from predicting scholarly impact, finding idea propagation, to text summarization, to establishing more informative citation indexers. authors use citations to frame their contributions and connect to an intelligent lineage @xcite . however, not all citations are created equal, nor they play similar roles. citations have different intents depending upon the citation context, the section under which they occur, etc. for example -a citation might indicate motivation or usage of a method from a previous work or a comparison of results of various works. and not even all citations are equally @xcite effective in finding relevant research. a majority of the papers cite a work contextually @xcite for providing additional background context. such background contextual citations help in the broader understanding; however, they are not central to the citing paper's theme. while some papers use the methods or ideas from the previous works, build upon them, and further progress the research in the field. these papers are expected to acknowledge them by citing them duly. these citations, which heavily rely on a given work or build upon that work, are significant. in this paper, we describe our team, iitp-cuni@3c's entry for the 3c citation context classification task. this shared task consisted of two subtasks. the goal of task a was to identify the purpose of the citations. the task b intended to classify the citations based on their importance into either influential or incidental. for identifying the purpose of the citations (task a), we employed a deep multi-task learning (mtl) framework that incorporates three scaffolds, including a cited paper title scaffold that leverages the relationship between the citation context and the cited paper title. the other two scaffolds are the structural scaffolds to leverage the relationship between the structure of the research papers and the intent of the citations. we utilize the scicite @xcite dataset apart from the training data available for this task. we achieve a public f1 macro score of 30.258% and a private macro f1 score of 26.059%. for the task of assessing the importance of the citation (task b), we pursue a feature-engineering approach to curate certain features from cited-citing paper pairs. in this task, we achieve a public f1 macro score of 50.396% and a private macro f1 score of 53.588%. to guarantee the reproducibility of our solution, our code and pretrained models are available on github. foot_0 2 related works one of the early contributions for automated classification of citation intents was from @xcite , a rule-based system where the authors used a classification scheme with 35 categories. later on, works included using machine learning systems based on the linguistic patterns of the scientific works. for example, the use of "cue phrases" along with fine-grained location features such as the location of citation within the paragraph and the section in @xcite . @xcite engineered pattern-based features, topic-based features, and prototypical argument features for the task. recently, @xcite proposed that features based on the structural properties related to scientific literature are more effective than the predefined hand-engineered domain-dependent features or external resources. in this work, we utilize the cited paper information as additional context and leverage the structural information related to the scientific discourse to improve the results. measuring academic influence has been a research topic since publications associate with academic prestige and incentives. several metrics (impact factor, eigen factor, h-index, citation counts, alt metrics, etc.) came up to comprehend research impact efficiently. still, each one is motivated on a different aspect and has found varied importance across disciplines. @xcite did pioneering work on academic influence prediction leveraging on citation context. @xcite explored citation classification into influential and incidental using machine learning techniques. ## task and dataset description the 3c citation context classification shared task organized as part of the second workshop on scholarly document processing @ naacl 2021 is a classification challenge, where each citation context is categorized based on its purpose and influence. it consists of 2 subtasks: ## methodology 4.1 task a we propose a multitask learning framework @xcite with the main task of citation intent classification along with three auxiliary tasks. these tasks help the model to learn optimal parameters for better performance on the main task. our model architecture is shown in figure 1 . let c be the tokenized citation context of size n. we pass it onto the scibert @xcite model with pre-trained weights to get the word embeddings of size (n, d 1 ) i.e. we have the output as @xmath0, x 2 , x 3 , . . . . . . ..x n } where x @xmath1. then we use a bidirectional long shortterm memory @xcite (bilstm) network with a hidden size d 2 to get an output vector h of size (n, 2d 2 ). we pass h to the dot-product attention layer with query vector w to get an output vector z which represents the whole input sequence here, α i represents the attention weights. for each task, we use a multi layer perceptron (mlp) followed by a softmax layer to obtain the class with the highest class probability. each mlp consists of a series of dense layers, a dropout @xcite layer and a relu @xcite activation layer. the parameters of a task's mlp are the specific parameters of that task and the parameters in the lower layers (parameters till the attention layer) are the shared parameters. we pass the vector z to m mlps related to the m tasks with t ask 1 as the main task and t ask i as the m-1 scaffold tasks, where @xmath2, m], to get an output vector @xmath3, y 2 , y 3 , .......y m }. we compute the loss function as : in each training epoch, we formulate a batch with an equal number of instances from all the tasks and calculate the loss as specified in equation 5 , where l @xmath4, task k where @xmath5. we use the adadelta optimizer for updating the parameters after backpropagation. the training is done in two stages: we use the pre-trained scibert scivocab uncased model trained on a corpus of 1.14m papers and 3.1b tokens to get the 768-dimensional word embeddings. we have tuned the hyperparameters λ i based on the performance on the validation set. ## results for task b, we analyze different machine learning algorithms and choose the model that performs the best on our validation data. we have also analyzed the impact of including each feature on the performance of the model. in task a, we ranked 6th (difference of 0.009 in the final private f1 score between our model and the best performing system) and we ranked 17th in task b (difference of 0.0644 in the final private f1 score between our model and the best performing system). ## analysis figure 2 and 3 show the confusion matrix of our proposed model on the validation data (400 instances) for task a and task b respectively. we investigate the type of errors made by our proposed model on the validation data for both tasks. we found out that for task a, our model makes many false-positive errors in the background category. to overcome this problem of overfitting, we decided to use some oversampling techniques like smote, but we still did not get any significant improvements. we also found out that some errors are due to ambiguity in the citation context as well as the title of the cited paper. we can avoid them by providing some additional context apart from the cited paper title information (for example, providing abstract from the cited paper, etc). figure 4 shows the significance of each feature in terms of information gain for task b. we can see that the word mover's distance (measures semantic similarity) features are the most contributing, unlike the vader (measures sentiment intensity) and the lexical tf-idf features. this trend might be true because in general, research articles have a style of writing that involves significantly less subjective content and follows a more objective discourse. hence, measurement of semantic similarity becomes the most important feature. we are yet to try many crucial features, but the current list of very simple features that we employed performed better than the majority baseline. ## conclusion and future work in this work, we demonstrate our team iitp-cuni@3c's submission to the 3c citation context classification task that includes two shared tasks. for task a, we show that the structural information related to a research paper and additional context (title information) of the cited article can be leveraged to classify the citation intent effectively. for task b, we use some simple features in a traditional machine learning setup to identify how significantly a paper cites the other. a future line of research would be to use the abstract and full-text information of the cited-citing paper pairs as additional contextual information.