Unsupervised Partial Sentence Matching for Cited Text Identification (2022)

Abstract: Given a citation in the body of a research paper, cited text identification aims to find the sentences in the cited paper that are most relevant to the citing sentence. The task is fundamentally one of sentence matching, where affinity is often assessed by a cosine similarity between sentence embeddings. However, (a) sentences may not be well-represented by a single embedding because they contain multiple distinct semantic aspects, and (b) good matches may not require a strong match in all aspects. To overcome these limitations, we propose a simple and efficient unsupervised method for cited text identification that adapts an asymmetric similarity measure to allow partial matches of multiple aspects in both sentences. On the CL-SciSumm dataset we find that our method outperforms a baseline symmetric approach, and, surprisingly, also outperforms all supervised and unsupervised systems submitted to past editions of CL-SciSumm Shared Task 1a.

## introduction the goal of a sentence-matching task is to extract a sentence that is most relevant to the query sentence from a collection of candidate sentences. in addition to information retrieval (ir) methods, a common unsupervised approach to sentence-matching tasks is to represent the query and candidate sentences by dense vectors, each computed by averaging the (contextualized) word embeddings corresponding to all constituent words in the sentence @xcite . in this way, all semantic aspects of each sentence are collapsed into a single embedding representing the entirety of its semantics. by applying a cosine similarity to each query-candidate pair of these embeddings to evaluate affinity, the implicit assumption is that the most similar pair of sentences should contain exactly the same set of semantic aspects. while em has worked quite well for a few tasks, notably machine translations (starting with the ibm models 1-5 @xcite , it has not had success in most others, such as part-ofspeech tagging @xcite , named-entity recognition @xcite and context-free-grammar induction (numerous attempts, too many to mention). the expectation maximization (em) algorithm (dempster, laird and rubin 77) is a common approach for unsupervised training; in this section we describe its application to the named entity problem. however, this approach is suboptimal for some applications, such as the task of identifying the "target" cited sentence(s) from a reference academic paper given a "query" citing sentence. as the example in figure 1 shows, the target matching cited sentence may contain extra semantic aspects in addition to those that are shared, perhaps providing further details. similarly, the query citing sentence might contain extra aspects referring to other work or to the relation of the cited information to the citing paper. motivated by this observation, we propose a simple and efficient unsupervised method that can accommodate extra semantic aspects in both query and candidates in the cited sentence identification task. to achieve this, our method employs an asymmetric sentence similarity measure to ignore words in the candidate that have little similarity to any query words, and we introduce a scaling function that de-emphasizes the unmatched words in the query citing sentence as well. on f1 of cl-scisumm shared task 1a @xcite , our method outperforms the corresponding symmetric similarity baseline, a strong unsupervised ir approach (aumiller et al., 2020), and the best supervised approach among the past submissions between 2018 and 2020, which ensembles four bert-based models @xcite . ## method figure 2 illustrates our similarity estimation method given a pair of sentences. in section 2.1, we ignore the details in the cited sentence candidate and only consider its matched words. in section 2.2, we softly remove the stop words because the similarity score should not consider the number of matched stop words. finally, we reduce the influence of irrelevant words in the query citing sentence and let the similarity score be determined more by the exactly matched words in section 2.3. @xcite perform extractive summarization by extracting the summary sentences that cover the original document best. inspired by their work, we extract the cited sentences that cover the query citing sentence best, which means not penalizing the details or extra words in the cited sentences. ## experiments we evaluate our method on the cl-scisumm dataset @xcite , comparing our results to past submissions to shared task 1a. in the official evaluation, performance is measured based on sentence overlap f1 and rouge-s* 1 @xcite , both micro-averaged over all sentences selected by all annotators. we train 300-dimensional word embeddings using word2vec skip-gram @xcite . we use the acl anthology reference corpus version 2 @xcite as our training corpus, because the papers in cl-scisumm are sampled from the computational linguistics domain. for each query citing sentence in the corpus, we select the top @xmath0. all the hyperparameters are experimentally chosen to maximize average f1 scores on the training set. ## related work a variety of unsupervised approaches to sentencematching tasks have been proposed. a traditional method uses an average (contextualized) word embedding as a sentence representation and computes a cosine similarity between query and candidate embeddings @xcite . another approach solves optimal transportation to match the words between two sentences @xcite . in addition, skip-thought @xcite , bert @xcite 2019), and simcse @xcite encode the sentence into a single embedding to predict the nearby sentences or augmented original sentence. these methods assume that all the semantic aspects in a sentence should be matched and lack a way to emphasize the matched aspects. @xcite propose an asymmetric similarity measure to be used in unsupervised extractive summarization. bertscore @xcite automatically evaluates generated text using similar asymmetric similarity scores. the coverage score from the generation to reference is its recall, and the score with the reverse direction is its precision. however, they do not use the asymmetric similarity to solve partial sentence matching tasks such as cited text identification. there are also many supervised approaches for estimating the relevancy of two sentences. for example, the approaches built on bert include the cross-encoder model @xcite , biencoder model (sentence-bert) @xcite , and the model that maximizes the coverage score from the retrieved document to the query (colbert) @xcite . although effective, these approaches often require a large training dataset to learn a good sentencematching. thus, such methods might not perform well in scientific sentence-matching tasks where annotations are very limited and expensive. ## conclusion we observe that many target cited sentences and query citing sentences are only partially matched, which motivates us to propose a simple asymmetric sentence similarity measurement that downweights or masks the unmatched words, stop words, and citation markers. with only a few training labels, learning the prior weighting on contextualized word embeddings could be challenging, and we suspect that this is the main reason that our simple unsupervised approach could outperform a well-tuned bert-based supervised approach. there are several potential applications of our approach. for example, it could be used to accelerate the labeling process, trace the claims made by the citing sentence to verify their correctness, or serve as a baseline for future supervised cited text identification approaches. one potential risk of our approach is that its assumptions might not be always valid and might create biases in downstream applications. for example, we assume that high-frequency words or unmatched words are less important in cited text identification tasks. this assumption could bias our method toward outputting longer sentences with more low-frequency words, which might be less comprehensible to users.