Multitask Learning for Citation Purpose Classification (2021)

Abstract: We present our entry into the 2021 3C Shared Task Citation Context Classification based on Purpose competition. The goal of the competition is to classify a citation in a scientific article based on its purpose. This task is important because it could potentially lead to more comprehensive ways of summarizing the purpose and uses of scientific articles, but it is also difficult, mainly due to the limited amount of available training data in which the purposes of each citation have been hand-labeled, along with the subjectivity of these labels. Our entry in the competition is a multi-task model that combines multiple modules designed to handle the problem from different perspectives, including hand-generated linguistic features, TF-IDF features, and an LSTM-with- attention model. We also provide an ablation study and feature analysis whose insights could lead to future work.

## introduction the influence of a scientific article has often been measured by its citation count. citation counts can be highly impactful, influencing the overall ratings of journals (e.g., their impact factor and eigenfactor) and are used to quantify the productivity and influence of authors (e.g., h-index, i-index). however, these methods all operate under the false premise that all citations should be counted equally. treating all citations with equal weight ignores the wide variety of functions that citations perform. citations can provide information about the methodology used in the work, point to a contrasting perspective, provide motivation or background information on a field, describe a previous work that the present work extends, or suggest that future work should consider a specific direction. because citations are so influential, there is a clear need * equal contribution. for automated methods to judge both the role and degree of importance of citations within articles. the 3c shared task citation context classification based on purpose competition, organized by scholarly document processing @ naacl 2021, aims to improve the state-of-the-art in automated citation classification @xcite . using the full text of an article, the competition's goal is to identify the purpose of each citation. in this paper, we describe our high-scoring entry into this competition, achieving third place. the paper is organized as follows. in section 2, we discuss related work. we describe the competition task and available data in section 3. in section 4, we present our approach and methodology. section 5 contains our results and an ablation study. finally, we close the paper with a discussion about our findings in section 6. ## related work past work on automated citation classification can be divided into two main categories: approaches centered around hand-generated linguistic features and approaches that utilize deep learning models. works that fall into the first category analyze the citation context or the full text of the cited article in order to design strong linguistic features. more specifically, teufel et al. @xcite focus on semantic-based features, identifying cue words, phrases, verb tense, voice, and so on. hou et al. @xcite introduce count-based features, i.e., the number of times an article is cited in various sections of the research paper. zhu et al. @xcite combines these approaches, presenting a list of 40 features split into five classes: count-based, similarity-based, contextbased, position-based, and miscellaneous features. among those, count-based features were the most significant. similarly, @xcite construct a set of twelve features, most of which were also similar to features generated by @xcite , and studied their importance. @xcite provided their own analysis of features from @xcite and @xcite , narrowing down to a list of three relatively predictive features (total number of citations, abstract similarity, and author overlap) for classifying the influence of a citation. while the above methods detect the influence of a citation, abu-jbara et al. @xcite explores vocabulary, number of citations in the sentence, and grammatical construction to classify the purpose of a citation. we draw upon each of these works in constructing our own features for our hand-generated features module. more recent approaches for citation classification use deep learning techniques from natural language processing @xcite . these methods rely on word embeddings from pretrained linguistic models such as bert @xcite , elmo @xcite , or glove @xcite to transform the citation context or full-text data into vector representations. jurgens et al. feeds these embeddings into a random forest for classification @xcite , but others have used these vectors as input for neural network classifiers. for example, structural scaffolds @xcite feeds the embedded representation of the data into a bidirectional lstm-with-attention. we adopt a similar approach in our lstm-withattention module. ## task description the 3c shared task citation context classification based on purpose competition is a supervised multiclass classification challenge, where each citation context must be categorized based on its purpose. the categories (i.e., the possible labels) are: back-ground, uses, compares_contrasts, motivation, extension, and future. the training dataset consists of 3000 labelled observations, each including the citing title, citing author, cited title, and cited author, along with the sentence in which the citation occurred. labels, which were constructed by humans, were provided for each training example. the test dataset contains 1000 unlabelled samples, 500 of which were used for public scoring, and the other 500 of which were used for the final ranking. ## methodology our model was trained on three different tasks: classifying purpose, section, and worthiness similar to @xcite . the model consists of three modules, as shown in figure 1 , which will be discussed in this section. ## results