Overview of the 2020 WOSP 3 C Citation Context Classification Task (2020)

Abstract: The 3C Citation Context Classification task is the first shared task addressing citation context classification. The two subtasks, A and B, associated with this shared task, involves the classification of citations based on their purpose and influence, respectively. Both tasks use a portion of the new ACT dataset, developed by the researchers at The Open University, UK. The tasks were hosted on Kaggle, and the participated systems were evaluated using the macro f-score. Three teams participated in subtask A and four teams participated in subtask B. The best performing systems obtained an overall score of 0.2056 for subtask A and 0.5556 for subtask B, outperforming the simple majority class baseline models, which scored 0.11489 and 0.32249, respectively. In this paper we provide a report specifying the shared task, the dataset used, a short description of the participating systems and the final results obtained by the teams based on the evaluation criteria. The shared task has been organised as part of the 8th International Workshop on Mining Scientific Publications (WOSP 2020) workshop.

## introduction citation analysis for research evaluation has been a subject of interest for the past several decades. the conventional one dimensional perspective of citation analysis, based on the pure citation frequency, which treats all citations equally, has endured a lot of criticism way back @xcite . subsequently, researchers have emphasised the need for developing new methods that consider the different aspects of the citing sentences. one such qualitative way for measuring the scientific impact is to analyse the citation context for discovering the author's reason for citing a paper. the text containing the reference to the cited document, the citation context, has proved to be a valuable signal for characterising the citation intent @xcite . the increase in the accessibility of the scientific publications, as well as the availability of full text of the research documents, from various services like core @xcite facilitates the possibility of exploring citation contexts, thereby further extending the bibliometric studies for research assessment @xcite . understanding the intent of citation has an essential role in measuring the scientific impact of the research papers. the possibility of knowing why a citation is included in one's work and how influential it is offers an excellent measure for evaluating the impact of a scientific publication. previous approaches for citation context classification employed a variety of annotation schemes ranging from low to high granularity. due to the lack of standard methods and annotation schemes, a comparison of the earlier systems is practically difficult. earlier systems used datasets with very limited size and this is probably because of the difficulties in manually annotating the citation contexts. besides, most of the research on citation context classification is not extensive enough and mainly reduced to specific domains of application, for instance, computer science and biomedical fields. this raises questions related to the generalisability of the presented models. the 3c shared task aims to create a platform encouraging researchers to participate in research in this area so that we can more reliably measure the performance of methods that have been tried in this area, establish the state-of-the-art and understand what works and what doesn't. two subtasks associ-ated with this shared task provide the participating teams the possibility to explore the new academic citation typing (act) dataset @xcite for analysing the citation context and classify the associated citations based on their purpose (subtask a) and influence (subtask b). a total of four teams participated in subtask a, and five teams participated in the subtask b. we used kaggle inclass competitions 1 for organising this shared task and the participating systems were evaluated using the macro f-score. this overview paper presents the 2020 3c shared task organisation. section 2 describes the related work; section 3 discusses the shared task setup, the data used, the baselines, followed by task evaluation in section 4. section 5 summarises the participating system description. section 6 and 7 presents the results and the conclusion. ## related work several supervised machine learning based frameworks that inspect the language used in scientific discourse have been developed in the past to categorise citations based on their context. @xcite used an annotation scheme with 12 categories and applied machine learning techniques on 2,829 citation contexts from 116 articles, using linguistic features including the cue phrases. these 12 classes belonged to four top-level categories; citations explicitly mentioning weakness, citations that compares or contrasts, citations which agrees or uses or is compatible with the citing work and finally a neural class. a more finegrained classification scheme introduced by jurgens et.al @xcite contains six categories and 1,941 instances from papers in computational linguistics(acl-arc dataset). the authors applied three novel features: pattern-based, topicbased and prototypical argument-based features besides the structural, lexical and grammatical, field and usage features. the above mentioned approaches all used handengineered features for classification. @xcite proposed a neural multi-task learning method using non-contextualised (glove) and contextualised word embeddings (elmo) along with bilstm and attention mechanism for citation intent classification. to achieve multi-task learning, the authors used two auxiliary tasks to aid the main 1 https://www.kaggle.com/c/about/ inclass classification task. the new dataset (scicite) @xcite contains 11,020 instances belonging to computer science and medicine domains and only three citation categories. a pre-trained model using 1.14m papers from semantic scholar foot_0 , called scibert @xcite , was released in 2019 and achieved a macro f-score of nearly 85% with fine-tuning using the scicite dataset. 3 the 3c shared task to address the limitations of citation context classification from the previous studies, we introduce a unified task to compare different citation classification methods on the same dataset. the shared task for the citation context classification, called, the "3c shared task", is organised as part of the international workshop on mining scientific publications (wosp), 2020 foot_1 , collocated with the joint conference on digital libraries (jcdl) 2020 foot_2 . as organisers, we believe, this shared task will provide the opportunity for comparing different classification systems and help progress the state-of-the-art. the competing systems in the 3c shared task will serve as a standard benchmark for future research in this direction. ## evaluation the evaluation was based on the test set of 1,000 examples. the test dataset was partitioned into public and private sets in kaggle. 50% of the test set was used for the initial evaluation, and the evaluation results against it appeared on the public leaderboard as the competition progressed. the rest of the data, which is the private partition on the test file, was used for the final scoring. the private leaderboard was visible only to the shared task organisers during the competition period. we used macro f-score for evaluating the submissions. ## participating system description this section presents the overview of the systems used by the participated teams, ufmg, paul larmuseau, scubed and amrita cen nlp in the 3c shared task. except for paul larmuseau, rest of the teams participated in both the tasks. the teams that participated in both tasks used the same approach while making submissions to subtask a and subtask b. ## results since the dataset for purpose classification task was highly skewed, with the majority of the classes belonging to the background class and the fact that we used macro f-score for evaluating the systems, all the systems submitted for this task scored less when compared to the subtask b. the results for the final evaluation of systems submitted for subtask b is shown in ## discussion the 3c shared task is the first open competition for citation context classification. this shared task could be considered as a new benchmark for these tasks as we release both the data and the source code of all the submitted systems. all the teams that participated in this shared task used simple as with feature representation, the conventional approach used by the majority of the teams was tf.idf. the prospect of employing word vectors developed using wikipedia, the shared task dataset and the use of pre-trained embeddings like glove were explored by the teams. figure 3 shows the confusion matrix for the best systems submitted by the teams scubed, ufmg, and amrita cen nlp for the subtask a. the most successfully classified category is back-ground. the winning team, ufmg, classified nearly 80% of the background class instances correctly. the number of true positives for the minority class future is zero, which implies that none of the above mentioned teams could successfully categorise the instances to this class. the imbalanced nature of the subtask a dataset significantly affects the performance of the systems submitted by teams, which is one of the challenging aspects as far as citation function classification task is concerned. tables 4 and 5 displays the public and private scores obtained by teams for the different systems they submitted for subtask a and subtask b respectively. all the teams for both tasks used the data field, citation context as the main source of semantic information for feature extraction, and classification. two teams also examined citing title and the cited title fields for extracting useful features. since kaggle allows late submissions for the hosted competitions, the participants can still submit results to get better scores, although this will not be visible on the public and the private leaderboard. both the tables also contain the post-evaluation results obtained by some of the teams. the current deep learning based state-of-the-art language models like scibert could not achieve better results on our dataset, and as the leaderboard indicates, such sophisticated models are beaten by more simpler methods, that are significantly less computationally expensive on this task. one possible reason for this could be the lesser number of training instances we provided to the participants. ## conclusion citations, which act as a connection between the cited and the citing articles, cannot be treated equally and serve different purposes. traditional citation analysis based on mere citation counts take into consideration just the quantitative factors. analysing the citation context for classifying citations based on their function and influence has many applications and the most important being its implementation in the research quality evaluation. one of the greatest challenges faced in the citation context analysis for identifying the citation function and its influence is the absence of multidisciplinary datasets and unavailability of medium to fine grained schemes which sufficiently captures information for citation classification @xcite . although previous works on the problem of citation context classification exist, lack of shared datasets, common conventions and annotation schemes caused the benchmarking of systems on the same tasks difficult. the 3c shared task constitutes the first systematic effort to a) compare different methods on the same data, b) on the same classification taxonomy across two previously reported tasks, and c) on multi-disciplinary data. we propose the unifying framework of the 3c shared task to be used as a standardised benchmark for this task, as we make all the submitted systems to this shared task, publicly available. we believe this will allow future comparison of participating systems head-to-head on the same data and task. the results obtained by the teams indicate the relevance of the simple machine learning based models over complex deep learning based approaches. the winning team for the subtask a, ufmg obtained an overall score of 0.19425. the team, paul larmuseau finished at first position on the leaderboard with a macro f score of 0.55565 for subtask b.