Overview of the 2021 SDP 3 C Citation Context Classification Shared Task (2021)

Abstract: This paper provides an overview of the 2021 3C Citation Context Classification shared task. The second edition of the shared task was organised as part of the 2nd Workshop on Scholarly Document Processing (SDP 2021). The task is composed of two subtasks: classifying citations based on their (Subtask A) purpose and (Subtask B) influence. As in the previous year, both tasks were hosted on Kaggle and used a portion of the new ACT dataset. A total of 22 teams participated in Subtask A, and 19 teams competed in Subtask B. All the participated systems were ranked based on their achieved macro f-score. The highest scores of 0.26973 and 0.60025 were reported for subtask A and B, respectively.

## introduction authors cite scholarly publications for a wide variety of reasons @xcite . as a result, citations in a research paper cannot be considered the same. hence, evaluating research requires the substitution of existing citation frequency based research assessment methods and adoption of new practises 1 . one such qualitative method that can be considered is to use the sentences surrounding the citation, known as citation context, for identifying the author's intent. characterising citations by examining citation context from the citing paper to evaluate the cited paper has been proposed as a promising direction for research assessment by previous studies @xcite . towards this end, several methods and taxonomies have been devised by manually assessing and analysing samples of research papers. this includes the classification schemes introduced by @xcite , @xcite , @xcite . the low generalisability due to limited dataset sizes and practical difficulties in using the above mentioned earlier schemes resulted in developing new general purpose classification taxonomies. in a pioneering work by @xcite , a new taxonomy consisting of 12 function classes has been created and used as part of an experiment to train models for automatic classification of citation function. other prominent schemes developed in similar contexts include @xcite @xcite . despite these efforts and more than a decade of research, it remains hard to compare and contrast existing solutions in this field due to the existence of multiple classification schemes and the absence of a benchmark suite. this is the primary motivation for introducing the 3c shared task. the goal of the 3c shared task is to analyse the text containing the citation, known as the citing sentence, in order to classify the reference anchor according to their purpose or function (subtask a) and influence or impact (subtask b). following the last year, we used a fraction of the academic citation typing (act) dataset @xcite for both the subtasks. we used the kaggle inclass competitions 2 environment to run the shared task, because we found it to be a highly suitable environment in the previous iteration of the 3c task in 2020. all the submitted systems were ranked using macro f-score. this overview paper is organised as follows: section 2 describes the related work; section 3 discusses the shared task setup, the data used, the baselines, followed by task evaluation in section 4. section 5 summarises the participating system description. section 6 and 7 present the results and the conclusion. ## related work existing automated systems mostly use taxonomy and ai-based methods to identify the citation function and influence from the citation context. some of the prevalent citation function classification schemes developed in the past use pre-defined categories, with number of classes ranging from small to medium to large @xcite @xcite . some of the function taxonomies also considered the polarity aspect of the citations @xcite . all classification methods developed for influence detection uses a binary scheme (incidental/non-important/non-influential and influential/important) @xcite . earlier methods for citation classification used feature based supervised machine learning models like random forest (rf) @xcite ., 2 https://www.kaggle.com/c/about/ inclass date event 15/12/2020 release of 1000 practise instances 26/02/2021 kaggle competition start date + release of train, test and full text 30/04/2021 kaggle competition end date with the aim to enhance research in this field and to provide a general platform for the competing systems, we introduced the new 3c citation context classification shared task in 2020 @xcite . three teams participated in the citation purpose classification task and four teams contested in the influence task @xcite @xcite . all the teams used simpler approaches such as tf-idf for feature representation and other machine learning based models for classification. the highest score obtained were 0.205 (subtask a) and 0.555 (subtask b). 3 the 2021 3c shared task the second version of the 3c shared task is organised by the researchers at the knowledge media institute (kmi), the open university, uk and oak ridge national laboratory (ornl), us. the 2021 3c shared task was part of the second workshop on scholarly document processing (sdp) foot_0 , collocated with the 2021 annual conference of the north american chapter of the association for computational linguistics (naacl) foot_1 . ## evaluation in order to rank the submitted results by the participating teams, we used macro f-score: the submission file has the following fields in the csv format: (1) unique id and (2) citation class/influence label. to avoid possible over-fitting due to large number of submissions, we restricted the per day number of submissions to 5. the final evaluation is based on the private score obtained by the teams. the number of scores eligible for the final evaluation was limited to 5. ## participating system description this edition of the 3c shared task witnessed the active participation of more teams compared to the last year's tasks. 22 teams participated in the purpose classification task, and 19 teams competed in the influence classification task. the following subsections describe some of the submitted systems in this shared task. ## discussion the sdp 3c shared task is a continuation of last year's wosp 3c shared task. the significant difference, however, is the inclusion of a full-text dataset for both subtasks. another highlight of this year's shared task is the active participation of 27 teams from the start, out of which 14 of them competed in both subtasks in kaggle. the overall scores obtained for both tasks shows considerable improvement when compared to the previous year's results ( the paper and code submissions for the purpose classification tasks shows the use of deep learning based methods by all the teams, as opposed to the last year, when the successful teams relied primarily on more traditional non-deep machine learning methods. the winning team, irel, used a pre-trained scibert model for predicting citation purpose. interestingly, using just the citation context information, the team achieved the highest private score. team duke data science, which finished third on the leaderboard, however, used external datasets like acl-arc and scicite as well for the purpose task. they also used the full-text dataset to extract the previous and next sentences for generating the tf-idf. iitp@3c emphasised the need for exploiting additional information related to the cited paper. like duke data science, iitp@3c, too, used a similar state-of-the-art bi-lstm attention model and external dataset. the team amrita_cen_nlp, who competed in the previous competition, improved their results this year using a bi-lstm model. for the influence classification task, we received the paper and source code from three teams. except for team irel, both iitp@3c and am-rita_cen_nlp used a feature-based machine learning approach. iitp@3c evaluated the feature importance and found that the similarity-based features computed between the citation context and cited/citing paper title resulted in increased performance. amrita_cen_nlp obtained the highest score with rf and fasttext embedding for the influence task. team irel, however, used the same scibert based model for this task too. figure 2 represents the public vs private macro f-scores plotted for both the subtasks. for the influence classification task, the graph 2b shows that the scores obtained by the teams are clustered more around the diagonal. however, for the purpose classification task, as shown in the graph 2b, there is significant deviation for the majority of points, indicating the possibility of over-fitting or under-fitting on the public data. although this year witnessed significantly more teams participating than in the previous shared task, despite our systematic encouragement, we only received research papers and source code submissions from four teams for subtask a and and three teams for subtask b. for the purpose classification task, out of the top five teams, only two teams submitted their methods. we did not receive the solution used by the top team on the leaderboard for the influence task. the open nature of the competi-tion hosting platform resulted in the participation of several kaggle enthusiasts, who competed with the other teams, without formally registering for the shared task. ## conclusion this paper describes the overview of the 2nd 3c citation context classification shared task organised as part of the sdp workshop and run at naacl. as in the previous year, two subtasks were organised to classify citations based on their purpose and influence. in addition to the act dataset, we provided the full-text data dump to the participants. this year, we have observed a substantial improvement in the private macro f-score values. we believe this is attributed to the use of advanced deep learning methods, the full-text dataset and external sources by the participating teams. however, the overall lower scores still indicate the need to address the challenges caused by the multi-domain nature of the dataset and the presence of more than one citation in the citation context for achieving better performing systems.