Combining Representations For Effective Citation Classification (2020)

Abstract: In this paper, we describe our participation in two tasks organized by WOSP 2020, consisting of classifying the context of a citation (e.g., background, motivational, extension) and whether a citation is influential in the work (or not). Classifying the context of an article citation or its influence/importance in an automated way presents a challenge for machine learning algorithms due to the shortage of information and inherently ambiguity of the task. Its solution, on the other hand, may allow enhanced bibliometric studies. Several text representations have already been proposed in the literature, but their combination has been underexploited in the two tasks described above. Our solution relies exactly on combining different, potentially complementary, text representations in order to enhance the final obtained results. We evaluate the combination of various strategies for text representation, achieving the best results with a combination of TF-IDF (capturing statistical information), LDA (capturing topical information) and Glove word embeddings (capturing contextual information) for the task of classifying the context of the citation. Our solution ranked first in the task of classifying the citation context and third in classifying its influence.

## introduction data science is becoming more and more popular with its largest data community being kaggle foot_0 , a platform that hosts several data mining and machine learning tasks and challenges. in 2020, the 8th international workshop on mining scientific publication (wosp) foot_1 , through kaggle, promoted two challenges consisting of: 1) classifying the context of a citation in one of the six existing classes (e.g., background, motivational, extension) and 2) a binary task where the goal is to identify the importance of a citation for a given work. the competition overview was presented by @xcite . an example of the citation context taken from the dataset is:"in the future we are planning to experiment with different ways of calculating relatedness of the sequences to the descriptions, such as with computing similarity of embeddings created from the text fragments using approaches like doc2vec (#author tag and mikolov, 2014)", where #author tag tag means the quote being classified. in this case, for the context classification challenge, this citation belongs to the future class, and for the influence classification task, this citation is considered influential. for the sake of the defined classification tasks, the citation text can be represented in several ways (e.g. tf-idf, word embeddings, text graph), each representation capturing or focusing on a different aspect of the task. for instance, the traditional tf-idf representation captures statistical aspects of the text and the specificity of certain words in the collection (idf component). topic modeling strategies such as lda identify patterns of recurrent topics (i.e., clusters of words) in the text. word embeddings are vectorial representations of words, sentences and whole documents aimed at capturing word co-occurrence patterns and contextual information. our main hypothesis here is that these different sources of information are somewhat complementary and, when combined, have the potential to enhance classification effectiveness. by exploring such ideas, we were able to reached the first place in the multiclass classification task promoted by wosp and third in the binary influence classification task, with further improvements after the closing of the challenge, as we shall see. in the competition there are two types of scores: for the task of assessing the importance of the citation, we obtained an improvement of 1.39% for the public score and 3.07% for the private one, by adding a feature that identifies that the author of the quote is the same author who is making the citation. to guarantee the reproducibility of our solution, all the code is available on github 34 and we create an image through the docker with the entire process configured. ## related work some papers addressed the problem of classifying the citation for context and influence. @xcite aimed at classifying the context of a citation, similarly to the challenge, into six possible classes. to train the model, they used structural information from the text, lexical, grammatical, etc. in that work, specific terms have higher weights in relation to the others, for example, "we extend" have more importance in the classification of the context "extends". unlike their work, we exploit only information from titles and the context of the citation with combinations of representations. in @xcite , if the citation is connected to related works or is used for comparison between methods, the author considers it as an incidental citation. if the cited method is used in the work or the current work is an extension of the cited one, the citation is considered important. these are the two possibilities in the challenge of the influential classification competition. in that work, the authors used 12 features, among them, the citation count per section, similarity between abstracts, etc. @xcite also dealt with the binary task of classifying an influen-tial citation. in that work, the authors expose the problem of extracting data from pdf when there is no structured data. their work analyzed the features of previous works and the impacts of adding specific features to the model. unlike valenzuela and pride's works, we do not use any information other than the titles and context of the citation, data provided through the kaggle. ## methodology in this section we present our methodology, consisting of of applying preprocessing to the data, creating different representations to explore the importance of individual words (tf-idf), group similar concepts shared by terms (lda) and explore the semantics and context of terms (glove). after this step, we join (concatenate) the representations to train the classifier, aiming to predict the class of the test set. the dataset contains eight fields, for each document we combine fields citing paper title, cited paper title and citation context (hereafter called citation text or simply citation (cit)) on a single line, separated by space. we use this file as input to the data preprocessing algorithm, which consists of: 1) turning the text into lowercase, 2) removing accents from words and 3) applying a parser that replaces specific strings with tags fixes, for example, number by "parsernumber". after preprocessing the citation text, we initially use the tf-idf @xcite representation. it exploits both, unigrams and bigrams. where the bigram is the combination of the current term and the next one. in more details, tf quantifies the frequency of the terms (unigrams or bigrams) and the idf measures their inverse frequency of document, giving higher weights to terms that occur less frequently in documents (higher discriminative power). another representation we use is the latent dirichlet allocation (lda) with online variational bayes algorithm (lda) @xcite . it groups terms into similar concepts called topics. topics are learned as a probability distribution on the words that occur in each document, using as input the original tf-idf representation. for each document, a score is associated with each topic and a citations may be seen as a combination of topics with different likelihoods. we tested some values for the hyperparameter that defines the number of topics in the dataset. we tested 6, 10, 50 and 100 topics. the one that produced the best results in the training data was 6 topics. that is, the algorithm creates 6 groups of words in the training and at the moment it receives a new test citation cit i , it assigns a score (likelihood) to each of the groups for cit i . the last exploited representation was glove embeddings @xcite , an unsupervised learning algorithm to obtain vector representations for words. for each term t present in a citation cit, we average (pool) their respective glove vectors to create a representation for the whole citation. each textual representation generates a number of features for each citation. in tf-idf, this number corresponds to the amount of unique terms existent in the entire dataset. for lda, this is the number of defined topics, in our case 6. for the glove embeddings we use the vector representation with 300 dimensions, thus generating 300 features for each citation. figure 1a presents the features for each citation cit i , where m represents the number of citations and n, l and g the number of features generated by each representation. finally, to combine the representations, we use a simple concatenation of all available representations (feature union foot_4 ) resulting in a single vector, as shown in figure 1b . ## experiment the dataset was created based on the methodology developed by @xcite . by means of an automatic system, authors (or external evaluators) can select to which group the citations are most related to (context) and whether the citation is described as central to the work. based on this methodology, 3000 citations with defined classes were made available through the kaggle for training along with 1000 test citations that should be classified according to the defined classes. ## conclusion in this paper we described our participation in the citation classification tasks organized by wosp