Scubed at 3 C task B - A simple baseline for citation context influence classification (2020)

Abstract: We present our team Scubed’s approach in the 3C Citation Context Classification Task, Subtask B, citation context influence classification. Our approach relies on text based features transformed via tf-idf features followed by training a variety of simple models resulting in a strong baseline. Our best model on the leaderboard is a random forest classifier using only the citation context text. A replication of our analysis finds logistic regression and gradient boosted tree classifier to be the best performing model. Our submission code can be found at: https://github.com/napsternxg/Citation_Context_Classification .

## introduction the number of research papers has increased exponentially in recent years. in order to efficiently access this scientific resource, we need automated solutions for extracting information from these records. citations in research papers are important for multiple reasons e.g. comparing novelty @xcite , expertise @xcite , and self-citation patterns @xcite . for people new to the field, they are a way to increase knowledge whereas for experts in the field they act as useful pointers to summarize the paper. citations are also used to measure various indexes which showcase the influence and reach of the researchers in their field. however, these indexes give equal weight to each citation. it has been established that all citations are not equal (n. @xcite . in many cases, cited papers are used as examples or are not influential to the paper itself. in this paper we describe our team, scubed's entry for the citation context influence classification shared task (n. @xcite . this work aims to develop models that can identify the influence of citations in the research papers, and hence can then be used to produce better indexes and make research more easily accessible to everyone. ## task and data description this paper focuses on the wosp 3c shared @xcite . in this sub-task, we were required to classify the citation context in research papers on the basis of their influence and purpose in the paper. for this shared task we used the acl-arc dataset @xcite . the dataset consisted of 3000 labeled data-points annotated using the act platform @xcite . the data provided contains the following fields: to identify the citation being considered a #au-thortag is placed in the citation. for this task the citation class label field was ignored. this was a binary classification task, where the following target labels were used : to evaluate the models the macro-f1 score was used on the test data. the final score that was used to rank was not the public score but a different subset of data that was not visible to the participating teams. the teams were advised to make submissions that would perform the best overall and not just on the public subset. ## methodology we utilize a simple approach based on text classification baseline methods. for the original submission we utilized a limited set of models. however, we trained additional models to conduct exhaustive evaluation for this paper. below, we describe our workflow for pre-processing, feature extraction, and model-training. ## results second, in ## discussion our results show that tradition tf-idf features give good performance for this shared task resulting in a strong baseline to compare against. simple machine learning models like logistic regression, random forests, and gradient boosted trees perform well for this task compared to other submissions. furthermore, the citation context contains the maximum signal for predicting citation influence. we were able to achieve one of the top performances in the task within the number of submissions required in the task. due to the small dataset, multiple submissions increase the likelihood of the models to over-fit to the test set. furthermore, our methods show that deep learning methods (e.g. mlp and mlp-3) do not give significant advantage over simpler machine learning methods. the minor loss in performance is acceptable compared to the in-creased speed and low computation of simple machine learning models. further analysis reveals that mlp based models are indeed over-fitting to the training data as shown by near perfect f1-score on the training data (see 2). additionally, gbt models consistently achieve much better performance on the test set compared to other models, including rf model which was our best entry on the leader board. furthermore, the highest performing label is the influential label. all models (except lr) perform the worse on the incidental when using all text features but when only using citation context, the label performance is similar across labels. ## conclusion our team 'scubed' submitted 5 models for the citation context classification based on influence task. out of the submitted models the random forest classifier performed the best on the test set achieving second position in this task. it achieved a private score of 0.55204 on the test set which was not only 0.003 behind the best performing model. we were able to achieve competitive results under minimum trials using fast and computationally cheap machine learning models.