Scubed at 3 C task A - A simple baseline for citation context purpose classification (2020)

Abstract: We present our team Scubed’s approach in the ‘3C’ Citation Context Classification Task, Subtask A, citation context purpose classification. Our approach relies on text based features transformed via tf-idf features followed by training a variety of models which are capable of capturing non-linear features. Our best model on the leaderboard is a multi-layer perceptron which also performs best during our rerun. Our submission code for replicating experiments is at: https://github.com/napsternxg/Citation_Context_Classification .

## introduction the number of research papers has increased exponentially in recent years. in order to efficiently access this scientific resource, we need automated solutions for extracting information from these records. citations in research papers are important for multiple reasons e.g. comparing novelty @xcite , expertise @xcite , and self-citation patterns @xcite . for people new to the field, they are an important resource to increase their knowledge whereas for experts in the field they act as useful pointers to summarize the paper. citations are also used to measure various indexes which showcase the influence and reach of the researchers in their field. however, these indexes give equal weight to each citation. it has been established that all citations are not equal @xcite . in many cases, cited papers are used as examples. often, they are not influential to the paper itself. in this paper we describe our team, scubed's entry for the citation context purpose classification shared task (n. @xcite . this work aims to develop models that can identify the purpose of citations in the research papers, and hence can then be used to produce better indexes and make research more easily accessible to everyone. ## task and data description this paper focuses on the wosp 3c shared subtask b. in this sub-task, we were required to classify the citation context in research papers on the basis of their influence and purpose in the paper. for this shared task we used the acl-arc dataset @xcite . the dataset consisted of 3000 labeled data-points annotated using the act platform @xcite . the data provided contains the following fields: to identify the citation being considered a #au-thortag is placed in the citation. for this task the citation class label field was ignored. this was a multi-label classification task, where the following target labels were used : to evaluate the models the macro-f1 score was used on the test data. the final score that was used to rank was not the public score but a different subset of data that was not visible to the participating teams. the teams were advised to make submissions that would perform the best overall and not just on the public subset. ## methodology we utilize a simple approach based on text classification baseline methods. for the original submission we utilized a limited set of models. however, we trained additional models to conduct exhaustive evaluation for this paper. below, we describe our workflow for pre-processing, feature extraction, and model-training. ## results ## discussion our results show that tradition tf-idf features give good performance for this shared task resulting in a strong baseline to compare against. simple machine learning models like logistic regression, random forests, and gradient boosted trees perform well for this task but are superseeded by multi-layer prerceptron models. furthermore, the citation context contains the maximum signal for predicting citation usage. we were able to achieve one of the top performances in the task within the number of submissions required in the task. due to the small dataset, multiple submissions increase the likelihood of the models to over-fit to the test set. furthermore, our methods show that deep learning methods (e.g. mlp and mlp-3) do give significant advantage over simpler machine learning methods. the minor loss in performance is acceptable compared to the increased speed and low computation of simple machine learning models. further analysis reveals that mlp based models are indeed over-fitting to the training data as shown by near perfect f1-score on the training data (see 2). additionally, gbt models consistently achieve much better performance on the test set compared to other models, including rf model which was our best entry on the leader board. furthermore, the highest performing label is the influential label. all models (except lr) perform the worse on the incidental when using all text features but when only using citation context, the label performance is similar across labels. ## conclusion our team 'scubed' submitted 3 models for the citation context classification based on purpose task. out of the submitted models the multi-layer perceptron classifier performed the best on the test set achieving third position in this task. this model gave a private score of 0.18146 on the test set. we were able to achieve competitive results under minimum trials using fast and computationally cheap machine learning models.