Team MLU @ CL - S ci S umm20: Methods for Computational Linguistics Scientific Citation Linkage (2020)

Abstract: This paper describes our approach to the CL-SciSumm 2020 shared task toward the problem of identifying reference span of the citing article in the referred article. In Task 1a, we apply and compare different methods in combination with similarity scores to identify spans of the reference text for the given citance. In Task 1b, we use a logistic regression to classifying the discourse facets.

## introduction the cl-scisumm shared task focuses on automatic paper summarization in the domain of computational linguistics research. given a document set with a reference papers and citing papers that all contain citations to the reference paper. in task 1a we should identify the spans of text (cited text spans) in the reference paper that most accurately reflect the citance. in task 1b for each cited text span, we should identify what facet of the paper it belongs to, from a predefined set of facets: hypothesis, aim, method, results, and implication. task 2 is to generate a summary of the reference paper. in this work, we focus on task 1. for comparison purposes, we experimented with the following approaches: svm, logistic regression, decision tree (cart), voting, and calculated a set of similarity metrics between reference spans and citance: tf-idf approach, cosine similarity, jaccard similarity, wordnet similarity. the best results are obtained by method which combined similarity scores using tf-idf approach, cosine similarity, wordnet similarity, bigram distance and svm. we also analyzed a dataset to identify features and highlights, which help us in solving the task. we found that most reference spans contain only one sentence. also, we conduct a semantic analysis to extract the named entities such as persons, organizations, products or locations. the most common named entities such as organizations and persons can be used for feature extraction. ## dataset the dataset contains 40 topics with citation sentences and human-annotated reference summaries. each topic is composed of a reference paper (rp) and some citing papers (cp). we have separate the data into two sets: 30 for training and 10 for testing. for this analysis all uppercase letters in train dataset were transformed into lowercase letters and all words that included non-alphabetical characters were removed. there are 46451 unique words among the reference documents. there are 648 reference sentences and 559 citances. an average number of reference sentences is 1.2, that means, a citation text can be linked to many sentences in the reference paper. the fig. 1 shows the distribution of the number of sentence in reference spans. the horizontal axis represents the ## experiment the dataset training-set-2018 provided by cl-scisumm shared task are training data and test data in our system. the dataset contains 40 topics with citation sentences and human-annotated reference summaries. as described in section "dataset", we separated data into two sets: 30 for training and 10 for testing. the documents were selected in alphabetical order. before we use dataset in our system we preprocessed the dataset to reduce some xmlcoding errors. formatting problems such a missed tags, broken words, non-ascii characters in xml files are some examples of these problems. we manually fixed broken words and automatically removed all non-ascii characters. we also fixed some missed closing tags. to remove the effect of using words to their different words we used lemmatization. this process is used to return the word to its origin. stopwords were removed for all configurations. we have also limited reference sentences by number of words. the average number of words in sentences is 47, minimal number is 6, maximal 282. in order to reduce noise we consider sentences with more than 10 and less than 70 words. in our baseline method, first we analyzed results of similarity calculations in order to choose a threshold. in approach method, we train our feature-based classifiers on all the relevant sentences pairs. ## results we implement our system and use official scripts to evaluate the training data. the evaluation of our approaches is done by comparison of several metrics which are presented in tables below. from the second best score is represented by svm method in combination with tf-idf approach (tfidf), bigram distance (bd), jaccard similarity (js), count cosine similarity (cs), wordnet similarity and sentence position (sid). method precision tfidf+threshold 0.0507 svm+tfidf+bd+js+cs 0.0648 svm+tfidf+bd+js+cs+ws 0.375 svm+sgd+tfidf+bd+js+cs+ws 0.5 svm+sid+tfidf+bd+js+cs+ws 0.1389 svm+sid+tfidf+bd+js+cs+ws+inq+inqs 0.1111 svm+tfidf+bd+js+cs+ws+me+mel 0.25 lr+tfidf+bd+js+cs 0.0806 lr+tfidf+bd+js+cs+ws 0.0694 lr+tfidf+bd+js+cs+ws+me+ mel 0.0139 dt+tfidf+bd+js+cs 0.0745 vt+tfidf+bd+js+cs 0.0926 vt+tfidf+bd+js+cs+ws 0.0556 vt+tfidf+sid+bd+js+cs+ws 0.0388 our set of experimental results, shown in figure 4 , present the performance of baseline method and approach method with respect to the metrics of precision, recall and f1-score. the vertical axis represent metric values, the horizontal axis represent the evaluation metrics, namely precision, recall and f1-score. compares to the baseline, the increases in approach method are 640% (0.3750 vs 0.0507), 85% (0.0783 vs 0.0423) and 180% (0.1295 vs 0.0462) respectively. ## conclusion and future work in this paper, we transform our work into binary classification problem and apply various methods to identify the spans of text in the reference paper reflecting the citance. we compare feature-based classifiers in combination with different features. although results show an improvement over the baseline, it is important to improve the performance of imbalanced data classification.