U ni HD @ CL - S ci S umm 2020: Citation Extraction as Search (2020)

Abstract: This work presents the entry by the team from Heidelberg University in the CL-SciSumm 2020 shared task at the Scholarly Document Processing workshop at EMNLP 2020. As in its previous iterations, the task is to highlight relevant parts in a reference paper, depending on a citance text excerpt from a citing paper. We participated in tasks 1A (citation identification) and 1B (citation context classification). Contrary to most previous works, we frame Task 1A as a search relevance problem, and introduce a 2-step re-ranking approach, which consists of a preselection based on BM25 in addition to positional document features, and a top-k re-ranking with BERT. For Task 1B, we follow previous submissions in applying methods that deal well with low resources and imbalanced classes.

## introduction scientific papers are among the most important means of communication between researchers that enable scholars to share their knowledge and progress, and provide other scientists with retrospective documentation and starting points for further improvements. a crucial part of this scientific exchange is citations, which refer to prior academic work that helped researchers put their work in the context of a broader scientific vision. the cl-scisumm shared task aims to construct meaningful summarization of this scientific communication by utilizing information extracted from such citations. the key contributions of a paper are identified by investigating which parts of the paper are cited, since citations usually highlight the critical points and main contributions of a paper. additionally, citations often target several aspects of a paper, and hence, can complement each other @xcite . as a result, a paper's contributions may be outlined by summarizing the parts of the paper that other researchers cited. another essential aspect of citations is the manner in which they cite another work: some may refer to results obtained in previous work, some build on top of the reference paper's methodology or propose modifications, and some debate claims hypothesized in a prior paper @xcite . therefore, particular citations of the same paper may refer to different text spans in various sections of the reference paper. if the authors use the cited work as a basis or starting point, they often refer to the methodology section. at the same time, a citation comparing the goals or results with that of prior work mainly refers to the introduction or evaluation of a paper. building on the ideas presented above, the cl-scisumm shared tasks @xcite @xcite split up the task of scientific summarization into multiple sub-tasks. these sub-tasks are formulated as follows: given a set of reference papers (rp) and a set of corresponding citing papers (cp) that contain citations to one of the reference papers, and in which text spans (so called citances) have been identified that pertain to a particular citation to the respective rp, participants of the shared task have to develop methods to solve the following tasks: our team participated in tasks 1a and 1b, and hence, we do not construct a final summarization of the respective rps. as the quality of a pre-selection can significantly improve the results of downstream tasks @xcite , we focus primarily on improving selection results in tasks 1a and 1b. we formulate task 1a as a search problem modeled in two steps: first, a set of cited sentences is extracted by employing a search using bm25 in combination with the sentence position in the document. here, the query term is the citation itself, and each sentence in the reference paper is treated as a single document in the search process. in the second step, a top-k re-ranking is applied that utilizes bert to extract the most relevant sentences. for task 1b, we follow previous work by zerva et al. @xcite in implementing oneversus-rest classifiers, but base them on perceptron classifiers instead of random forests or bert-based models. ## related work in previous editions of the cl-scisumm shared task, various effective strategies were proposed to solve task 1 @xcite . to find the relevant reference sentence, most systems from 2019 focused on sentence similarity. similarities are either obtained by methods such as tf-idf and jaccard or embedding-based methods to mine more semantic information @xcite or by designing specific features and learning sentence similarities in a supervised manner @xcite . task 1a can also be framed as a classification task and solved via a single or ensemble of multiple classifiers. an ensemble of regression and classification models are trained on the reference and citation sentences @xcite . the best performing system from 2019 @xcite uses a bert-based model to solve the task in two ways, first by using sentence similarity of bert vectors trained on citation and reference sentence pairs, and second by using bilateral multi-perspective matching model. the authors also perform extensive data cleaning and mine additional data from the pdf version of the papers to fine-tune their language model. the most similar work to our approach is by @xcite , where the authors propose a two-stage similarity-based unsupervised ranking method. in the first stage, they use modified jaccard similarity to select the top-5 relevant sentences. these top-5 selected sentences are ranked again using a listwise ranking model and the features from the first stage. in contrast, we utilize a variant of bm25 for our original ranking, a larger candidate set of 10 sentences, and a pair-wise ranking for our second stage using bert. since neural models pretrained on language modeling such as bert have achieved impressive results for several nlp tasks, many have applied them to search-related tasks. bert is used in ad-hoc document ranking @xcite and also for multi-stage ranking, where the original ranking is often performed by an efficient method like bm25 and the results are ordered by their relevancy score by single or multiple re-ranking stages @xcite . ## methodology in our approach, we aim to solve task 1a and 1b independently. we formulate the citation linkage in task 1a as a search problem, where the cp sentence is the query, and each sentence in the respective rp is considered a separate "document" in an indexed collection. for task 1b, we mainly follow existing work to deal with the unbalanced data and a low number of samples. in the following, we explain the framework in more detail. ## experimental settings and evaluation aside from the pure evaluation of results, we further verified the correctness of the existing training data. for the manually annotated samples from 2018, we were able to identify several citations that were either duplicated or mismatched (differing reference text and reference offset, or similar for the citance). further, we deleted around 30 empty documents from the scisummnet corpus (containing non-empty "sentences" for less than 10% of the document). a pull request with several changes is currently awaiting approval for the main shared task repository. ## future work as indicated in the results on the different test sets for task 1a, a high-quality sub-selection of potentially relevant sections can significantly boost the performance of more learning-based methods. despite some optimization, we still saw relatively low recall values for some citation spans, which can have several causes. we believe that improvements to the initial recall can still significantly boost results, but require measures that do not entirely rely on simple word frequency measures such as bm25 or tf-idf. while the re-ranker seemed to struggle with the limitations of the current setup, future exploration with different re-ranking approaches might ultimately yield improvements over the results returned by solr. we intend to utilize contextualized results, or "snippets" of several sentences within a single result in future submissions to the workshop to increase pre-selection recall. similarly, for task 1b, replacing prediction for lowresource classes with rule-based approaches could balance classification scores on the unseen test set.