Multi-Vector Models with Textual Guidance for Fine-Grained Scientific Document Similarity (2022)

Abstract: We present a new scientific document similarity model based on matching fine-grained aspects of texts. To train our model, we exploit a naturally-occurring source of supervision: sentences in the full-text of papers that cite multiple papers together (co-citations). Such co-citations not only reflect close paper relatedness, but also provide textual descriptions of how the co-cited papers are related. This novel form of textual supervision is used for learning to match aspects across papers. We develop multi-vector representations where vectors correspond to sentence-level aspects of documents, and present two methods for aspect matching: (1) A fast method that only matches single aspects, and (2) a method that makes sparse multiple matches with an Optimal Transport mechanism that computes an Earth Mover’s Distance between aspects. Our approach improves performance on document similarity tasks in four datasets. Further, our fast single-match method achieves competitive results, paving the way for applying fine-grained similarity to large scientific corpora.

## introduction the ability to identify similarity across documents in large scientific corpora is fundamental for many applications, including recommendation @xcite , exploratory or analogical search @xcite @xcite , paper-reviewer matching @xcite and many more uses. @xcite , centrally important discourse structures of scientific texts. in this paper, we propose a new model for document similarity that makes aspect-level matches across papers and aggregates them into a documentlevel similarity. we focus on sentence-level aspects of paper abstracts, and train multi-vector representations of papers in terms of their contextualized sentence embeddings. to train our models, we leverage a readily available data source: sentences that co-cite multiple papers. unlike recent work that used citation links for learning scientific document similarity @xcite , we observe that papers cited in close proximity provide a more precise indication of relatedness. furthermore, the citing sentences typically describe how the co-cited papers are related, in terms of shared aspects (e.g., similar methods or findings, related challenges or directions, etc.). building on this observation, we leverage these textual descriptions as a novel source of textual supervision, using them to guide our model to learn which sentence-aspects match without any direct sentence-level supervision. guidance for the document similarity model is obtained via an auxiliary sentence encoder model that is used for aligning abstract sentences by finding pairs most similar to the citing sentence text. our document similarity objective is modeled as a function of similarity between sentence-level matches. we explore two strategies to aggregate over sentence-level distances between documents. first, a single-match method with minimum l2 distances between document aspect vectors. this approach readily supports approximate nearest neighbor search methods for large-scale retrieval. second, a multi-match method that computes an earth mover's distance between documents' aspect vectors by solving an optimal transport problem. this yields a soft sparse matching of aspect vectors, which when combined with their l2 distances gives a document-level distance. finally, as an additional benefit of our representation, our models also support a finer aspectconditional retrieval task @xcite @xcite where aspects can be specified by selecting abstract sentences -for example, selecting sentences describing methods and retrieving papers using similar methods. as we show, naively encoding sentences without their context leads to subpar results in this task, and our representation that does take context into account dramatically improves results. extensive empirical evaluation on four english scientific text datasets and seven similarity tasks at the level of documents and sentences demonstrates the effectiveness of our models. these include biomedical document retrieval tasks and a recent faceted query-by-example corpus of computer science papers @xcite . this latter dataset is used for evaluating retrieval conditioned on specific aspects in context (e.g., for finding papers with similar methods to a query document), demonstrating that our model can be used in this challenging and important setting. in summary, we make the following main contributions: we present aspire 2 , a multi-vector document similarity model that flexibly aggregates over fine-grained sentence-level aspect matches. 2. co-citation context supervision: we exploit widely-available co-citation sentences as a new source of training data for document similarity and provide a method using a novel form of textual supervision to guide representation learning for aspect matching. 3. state of the art results: our aspire models outperforms strong baseline methods across four datasets for the abstract and aspectconditional similarity tasks. ## problem setup given query document q and a candidate amongst a set of documents @xmath0, where documents consists of n sentences s 1 , s 2 , . . . s n we aim to leverage fine-grained document similarity in two problem settings. an abstract level retrieval task @xcite and an aspect-level retrieval task @xcite : def 1. retrieval by abstracts: given query and candidate documents -q and c a system must output the ranking over c. ## aspire: aspectual scientific paper relations. def 2. aspect-level retrieval by sentences: given query and candidate documents -q and c, and a subset of sentences s q ⊆ q conditional on which to retrieve documents, a system must output the ranking over c. modeling desiderata: next, we also outline key desired properties we require from models developed for task definitions 1 and 2. we follow these desiderata when building our methods ( §3.1). ## proposed approach: aspire in this section we describe our approach to document similarity -aspire. we model finer-grained matches between documents at the level of sentences via contextual representations and aggregating over matches to obtain similarities between whole documents. we leverage co-citation sentences as a source of document similarity and also as implicit textual supervision describing related aspects of co-cited documents. we formulate our multi-vector models @xcite ) that can support scalable inference as novel multiple-instance learning (mil) models. ## experiments and results evaluation data: we evaluate the proposed methods on datasets for whole abstract document similarity and fine-grained document similarity. we overview these below. appendix b provides detail. baselines: we compare the proposed approaches to three classes of methods. we overview these classes and associated models below, with appendix d presenting further detail: 1. sentence models: sentence embedding models present reasonable baselines since we consider fine-grained matches at the sentence level. these are represented by mpnet-1b, a sentence model trained on over 1 billion text pairs foot_1 , sentence-bert (sentbert) @xcite , sim-cse @xcite , cosentbert of §3.2.2, and ictsentbert @xcite . sentence models use the same inference procedure as our single match method, abstract models rank using l2 distances between papers embeddings, and the modified sentence model uses the multi match inference procedure. all reported model hyper-parameters are tuned, trained on 1.3m co-citation triples, and initialized with specter unless noted otherwise. 4 in reporting results, we report standard retrieval metrics mean average precision (map) and ndcg at rank k. for ndcg@k we follow @xcite , and choose @xmath1, 1). ndcg %20 therefore refers to ndcg computed at 20% of the pool size for a query. this is apt since queries have varying pool sizes. appendices a, e, and f detail training data, algorithms, and hyper-parameters. next, we present our main results comparing proposed approaches to baselines. ## conclusions we presented aspire, a scientific document similarity model that is trained by leveraging co-citation contexts for learning fine-grained similarity. we use co-citation contexts as a novel form of textual supervision to guide the learning of multivector document representations. our model outperformed strong baselines on seven document similarity tasks across four english scientific text datasets. moreover, we showed that a fast singlematch method achieves competitive results, enabling fine-grained document similarity in largescale scientific corpora. a future direction is the interactive use of our methods, with a system allowing users to highlight specific aspects of papers and retrieve contextually-relevant matches. another promising application is for finding analogiesstructural matches between texts describing ideas, as in scientific papers, to boost discovery @xcite @xcite . of co-cited papers from the s2orc corpus foot_3 @xcite . since our evaluation datasets draw on text from different domains we build training sets with co-cited papers for each: biomedicine for relish and treccovid rf , computer science for csfcube, and a 60/40 mix of biomedicine and cs for scidocs. each dataset contains 1.3m training triples. next we describe construction of our co-citation data given 8.1 million english full text articles in the s2orc corpus which have been parsed for citation mentions and linked to cited papers in the corpus using automatic tools @xcite ): abstract level training triples for the biomedical and computer science sets are built by treating all unique pairs of papers as positives. 1.3 million triples were used for each domain -these were sampled from larger sets at random.