C it R et: A Hybrid Model for Cited Text Span Retrieval (2022)

Abstract: The paper aims to identify cited text spans in the reference paper related to the given citance in the citing paper. We refer to it as cited text span retrieval (CTSR). Most current methods attempt this task by relying on pre-trained off-the-shelf deep learning models like SciBERT. Though these models are pre-trained on large datasets, they under-perform in out-of-domain settings. We introduce CitRet, a novel hybrid model for CTSR that leverages unique semantic and syntactic structural characteristics of scientific documents. This enables us to use significantly less data for finetuning. We use only 1040 documents for finetuning. Our model augments mildly-trained SBERT-based contextual embeddings with pre-trained non-contextual Word2Vec embeddings to calculate semantic textual similarity. We demonstrate the performance of our model on the CLSciSumm shared tasks. It improves the state-of-the-art results by over 15% on the F1 score evaluation.

## introduction citations are an integral part of scientific literature as they help better understand the relationships between scientific documents. authors cite other papers to acknowledge their contributions, compare to their work, criticize, and improve upon their work. citances often focus on the most important components of a scientific document. moreover, citance-based summarization is also a widely studied field because it covers some insights that might not be present in abstract-based summarization @xcite . however, a citance depends on the intention and opinion of the citing author and can be affected by epistemic value drift 1 @xcite . also, a citance in itself lacks sufficient details to capture the exact content of the referenced paper. hence, identifying the correct context of the cited text can enable us to verify the biases @xcite , overcome epistemic value drift, build dense knowledge graphs, and generate better summaries @xcite . furthermore, it also helps in qualitative analysis of the citations @xcite . motivated by these, research tasks and tracks such as biomedsumm foot_0 and clscisumm lay significant emphasis on this fundamental and challenging problem of finding the exact cited text span. we refer to this task as cited text span retrieval @xcite . most of the current methods targeting this problem are centered around fine-tuning deep neural networks. in this regard, transformer @xcite based encoders such as bert @xcite and scibert @xcite have proven to be very effective and have outperformed standard baselines like lda and tf-idf. however, a major drawback of these methods is that they require large domain-specific datasets, often exceeding 1 million documents, to fine-tune. this paper proposes citret, a hybrid ctsr model that performs well even in low-resourced domain-specific settings. we model the problem as a semantic textual similarity (sts) task. we exploit the distinctive semantic and syntactic structural characteristics of scientific literature, i.e., when a paper is cited, the cited text of the reference paper is often paraphrased in such a way that it still expresses the same central idea while also preserving certain keywords. hence, we use these keywords, which are common to both the citance and the cited sentence, to find weighted contextual embeddings for the sentences. to find these weighted contextual embeddings, we use sentence-bert (sbert) @xcite fine-tuned to minimize cosine similarity loss on training data. however, when the training data is scarce, these contextual embeddings fail to cap-ture out-of-domain knowledge. to overcome this, we further leverage pre-trained non-contextual embeddings like word2vec @xcite to capture the general domain knowledge. we use word mover's distance (wmd) @xcite to find (dis)similarity scores based on these non-contextual embeddings. this hybrid approach of utilizing contextual and non-contextual embeddings enables citret to generalize well over unseen datasets. definitions of the terms used throughout the paper are: reference paper (rp): a scientific document of which one or more sentences have been cited by another paper(s). citing paper (cp): a document that contains one or multiple citations to an rp. citance: a sentence in cp that contains the reference to the rp. cited sentence: the exact piece of the text belonging to the rp that a citance refers to. cited text span: span of the cited sentence(s) belonging to the rp corresponding to a citance. the major contributions of this work are: 1) proposing a simple yet effective ctsr model that requires less data for fine-tuning and is computationally inexpensive. we train only on the cl-scisumm training dataset that consists of 40 manually annotated articles and 1000 automatically annotated articles. 2) advancing the state-of-the-art (sota) to identify cited text span by over 15%. 3) empirically validating the advantage of using the semantic and syntactic structure for ctsr. ## related work the task of ctsr requires modeling the relationship (similarity) between a citing and a candidate cited sentence. early systems proposed using features based on tf-idf @xcite @xcite and n-grams or sentence graph overlap @xcite in order to calculate similarity scores between the citing sentence and candidate sentences. similarity measures such as jaccard similarity and cosine similarity were commonly used to solve this task. @xcite @xcite . the problem has also been posed as a binary classification problem in @xcite ; @xcite ; @xcite . in addition to traditional features such as tf-idf and n-grams, prior methods have also proposed using learned distributed vector space representation (word embeddings) based features since they contain the seman-tic similarity information at the word level. models using both non-contextual embeddings such as word2vec and contextual embedding methods like bert have been utilized to find these word embeddings. these extracted features are further used as an input to machine learning algorithms like svm @xcite , random forests @xcite , word mover's distance @xcite , cnn @xcite or xgboost @xcite . furthermore, many approaches even adopted voting mechanisms and ensemble techniques on top of their models to improve their metrics @xcite @xcite @xcite . the current best performing models exploit transformers fine-tuned on very large datasets @xcite . @xcite also experimented with adding document level features to the model using special tokens. other noteworthy approaches, like aumiller et al. (2020) formulated the task as a search problem and used a two-step approach for retrieving relevant sentences for a given citation. they first find candidate sentences using apache solr and bm25 and then re-rank the retrieved sentences using a computationally expensive bert-based reranker. ctsr as semantic textual similarity: we model the problem as a semantic textual similarity (sts) task. to this end, learning sentence embeddings, instead of word embeddings, has shown promise and improvement in performance @xcite . using pooling strategies such as mean or max pooling of word embeddings has proven to be an efficient way of obtaining sentence embeddings. sbert @xcite by default uses mean pooling. @xcite further explored generalized pooling strategies to enhance sentence embeddings. cnn-based models have also been used to encode sentences into fixed length vectors @xcite . to improve performance on sentence matching tasks, @xcite proposed syntaxand semantics-aware bert(ss-bert), which implicitly integrates syntactic and semantic information of sentences. @xcite showed that sentence embeddings could be further improved by employing principal component removal based denoising as a post-processing step. the wces are then denoised using the common component removal technique. these denoised wces are used to find cosine similarity between the sentences of the input pair. finally, wmd and cosine scores are added, and top k similar sentences in an rp for a citance are retrieved. ## methodology we formulate this task of ctsr as finding semantic textual similarity between a citance and all the sentences of an rp, i.e., to find the cited text span for a given citance, we pick the top k similar sentences in the rp. we refer to @xmath0, a sentence in the @xmath1. as shown in figure 1 , an input pair is first pre-processed by lowercasing the tokens, removing the stop words, and removing the special characters. then to find the final similarity scores, citret employs a mix of cosine scores using weighted contextual embeddings (contextual distance) and word mover's distance scores (non-contextual distance) using pre-trained non-contextual embeddings. now, we explain each component of the pipeline in detail. ## experiments and results we demonstrate the performance of the proposed method on cl-scisumm shared task @xcite @xcite task 1(a), where for each citance, we need to identify the spans of text (cited text spans) in the rp that most accurately reflect the citance. these cited text spans range from the granularity of a sentence fragment to several consecutive sentences. we pick top @xmath2. citret is trained only on the cl-scisumm training dataset that consists of 40 manually annotated articles and 1000 low quality document sets that were automatically annotated using neural networks. we do not use any external corpora to fine-tune our model. we evaluate our model's performance against gold label annotations for the cl-scisumm test set of 20 documents. we consider the sota models of 2019 and 2020 cl-scisumm tasks as baselines. ## discussion as can be observed from the proposed approach is in line with the recommendation made by the task organisers to exploit the structural and semantic characteristics that are unique to scientific documents to enrich the embeddings. the paper proposes a simple and computationally inexpensive alternative to the current state-of-art model in the form of citret. it leverages both contextual and non-contextual embeddings. citret also combines a supervised model and an unsupervised model. this hybrid architecture provides performance and robustness against noisy training samples. the components of the model are lightweight (do not require extensive fine-tuning), faster, explainable, and intuitive. this highlights how other statistical machine learning techniques can be leveraged along with modern deep neural network architectures to compensate for the lack of quality training data and outperform computationally expensive architectures. it may also be noted that while our method beats the baselines by large margins and achieves a new sota, the absolute values are still rather low because of the non-triviality of the task. the task becomes particularly challenging because of the low-quality training data and subjectivity of the annotators. hence, we believe that there is a scope for further improvement, and the problem demands greater exploration. ## conclusion in this paper, we propose citret, a novel model for cited text span retrieval. citret outperforms the current sota models by significant margins (15% f1). the proposed model is quite simple, computationally inexpensive, improves generalization, and does not require any large external datasets to fine-tune. however, considering the non-triviality of the task, this paper proposes a new approach for further exploration of the task.