Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings (2022)

Abstract: Learning scientific document representations can be substantially improved through contrastive learning objectives, where the challenge lies in creating positive and negative training samples that encode the desired similarity semantics. Prior work relies on discrete citation relations to generate contrast samples. However, discrete citations enforce a hard cut-off to similarity. This is counter-intuitive to similarity-based learning and ignores that scientific papers can be very similar despite lacking a direct citation - a core problem of finding related research. Instead, we use controlled nearest neighbor sampling over citation graph embeddings for contrastive learning. This control allows us to learn continuous similarity, to sample hard-to-learn negatives and positives, and also to avoid collisions between negative and positive samples by controlling the sampling margin between them. The resulting method SciNCL outperforms the state-of-the-art on the SciDocs benchmark. Furthermore, we demonstrate that it can train (or tune) language models sample-efficiently and that it can be combined with recent training-efficient methods. Perhaps surprisingly, even training a general-domain language model this way outperforms baselines pretrained in-domain.

## introduction large pretrained language models (llms) achieve state-of-the-art results through fine-tuning on many nlp tasks @xcite . however, the sentence or document embeddings derived from llms are of lesser quality compared to simple baselines like glove @xcite , as their embedding space suffers from being anisotropic, i.e. poorly defined in some areas @xcite . one approach that has recently gained attention is the combination of llms with contrastive finetuning to improve the semantic textual similarity between document representations @xcite . these contrastive methods learn sample induced margin easy negatives figure 1 : starting from a query paper in a citation graph embedding space. hard positives are citation graph embeddings that are sampled from a similar (close) context of , but are not so close that their gradients collapse easily. hard (to classify) negatives (red band) are close to positives (green band) up to a sampling induced margin. easy negatives are very dissimilar (distant) from the query paper . to distinguish between pairs of similar and dissimilar texts (positive and negative samples). as recent works show @xcite @xcite , the selection of these positive and negative samples is crucial for efficient contrastive learning. this paper focusses on learning scientific document representations (sdrs). the core distinguishing feature of this domain is the presence of citation information that complement the textual information. the current state-of-the-art specter by @xcite uses citation information to generate positive and negative samples for contrastive fine-tuning of a scibert language model @xcite . specter relies on 'citations by the query paper' as a discrete signal for similarity, i.e., positive samples are cited by the query while negative ones are not cited. however, specter's use of citations has its pitfalls. considering only one citation direction may cause positive and negative samples to collide since a paper pair could be treated as a positive and negative instance simultaneously. also, relying on a single citation as a discrete similarity signal is subject to noise, e.g., citations may reflect politeness and policy rather than semantic similarity @xcite or related papers lack a direct citation @xcite . this discrete cutoff to similarity is counter-intuitive to (continuous) similarity-based learning. instead, the generation of non-colliding contrastive samples should be based on a continuous similarity function that allows us to find semantically similar papers, even without direct citations. with scincl, we address these issues by generating contrastive samples based on citation embeddings. the citation embeddings, which incorporate the full citation graph, provide a continuous, undirected, and less noisy similarity signal that allows the generations of arbitrary difficult-to-learn positive and negative samples. ## methodology our goal is to learn citation-informed representations for scientific documents. to do so we sample three document representation vectors and learn their similarity. for a given query paper vector d q , we sample a positive (similar) paper vector d + and a negative (dissimilar) paper vector d -. this produces a 'query, positive, negative' triplet (d q , d + , d -) -represented by ( , , ) in fig. 1 . to learn paper similarity, we need to define three components: ( §3.1) how to calculate document vectors d for the loss over triplets l; ( §3.2) how citations provide similarity between papers; and ( §3.3) how negative and positive papers (d -, d + ) are sampled as (dis-)similar documents from the neighborhood of a query paper d q . ## experiments in the following, we introduce our experiments including the data sets and implementation details. ## overall results tab. 1 shows the results, comparing scincl with the best validation performance against the baselines. with replicated specter training data (w/ leakage), scincl achieves an average performance of 81.8 across all metrics, which is a 1.8 point absolute improvement over specter (the next-best baseline). when trained without leakage, the improvement of scincl over specter is consistent with 1.7 points but generally lower (79.4 avg. score). in the following, we refer to the results obtained through training on the replicated specter data (w/ leakage) if not otherwise mentioned. we find the best validation performance based on specter's data when positives and hard negative are sampled with knn, whereby positives are k + =25, and hard negatives are k - @xmath0. easy negatives are generated through filtered random sampling. scincl's scores are reported as mean over ten random seeds (@xmath1, 9]). for mag classification, specter achieves the best result with 82.0 f1 followed by scincl with 81.4 f1 (-0.6 points). for mesh classification, scincl yields the highest score with 88.7 f1 (+2.3 compared to specter). both classification tasks have in common that the chosen training settings lead to over-fitting. changing the training by using only 1% training data, scincl yields 82.2 f1@mag (tab. 2 ). in all user activity and citation tasks, scincl yields higher scores than all baselines. moreover, scincl outperforms sgc on direct citation prediction, where sgc outperforms specter in terms of ndcg. on the recommender task, specter yields the best p@1 with 20.0, whereas scincl achieves 19.3 p@1 (in terms of ndcg scincl and specter are on par). when training specter and scincl without leakage, scincl outperforms specter even in 11 of 12 metrics and is on par in the other metric. this suggests that scincl's hyperparameters have a low corpus dependency since they were only optimized on the corpus with leakage. regarding the llm baselines, we observe that the general-domain bert, with a score of 63.4, outperforms the domain-specific bert variants, namely scibert (59.6) and biobert (58.8). llms without citations or contrastive objectives yield generally poor results. this emphasizes the in summary, we show that scincl's triplet selection leads on average to a performance improvement on scidocs, with most gains being observed for user activity and citation tasks. the gain from 80.0 to 81.8 is particularly notable given that even oracle scidocs yields with 83.0 an only marginally higher avg. score despite using test and validation data from scidocs for the triplet selection. appendix h shows examples of paper triplets. ## impact of sample difficulty in this section, we present the optimization of scincl's sampling strategy ( §3.3). we optimize the sampling for positives and hard or easy negatives with partial grid search on a random sample of 10% of the replicated specter training data (sampling based on queries). our experiments show that optimizations on this subset correlate with the entire dataset. the validation scores in fig. 2 and 3 are reported as the mean over three random seeds. ## conclusion we present a novel approach for contrastive learning of scientific document embeddings that addresses the challenge of selecting informative positive and negative samples. by leveraging citation graph embeddings for sample generation, scincl achieves a score of 81.8 on the scidocs benchmark, a 1.8 point improvement over the previous best method specter. this is purely achieved by introducing tunable sample difficulty and avoiding collisions between positive and negative samples, while existing llm and data setups can be reused. this improvement over specter can be also observed when excluding the scidocs papers during training (see w/o leakage in tab. 1). furthermore, scincl's improvement from 80.0 to 81.8 is particularly notable given that even oracle triplets, which are generated with scidocs's test and validation data, yield with 83.0 only a marginally higher score. our work highlights the importance of sample generation in a contrastive learning setting. we show that language model training with 1% of triplets is sufficient to outperform specter, whereas the remaining 99% provide only 1.0 additional points (80.8 to 81.8). this sample efficiency is achieved by adding reasonable effort for sample generation, i.e., graph embedding training and knn search. we also demonstrate that in-domain llm pretraining (like scibert) is beneficial, while general-domain llms can achieve comparable performance and even outperform specter. this indicates that controlling sample difficulty and avoiding collisions is more effective than indomain pretraining, especially in scenarios where training an llm from scratch is infeasible. consider the increased compute effort as the major limitation of the scincl method. especially the training of the graph embedding model accounts for most of the additional compute effort. this is also the reason for us providing only a shallow of evaluation of the graph embeddings (appendix d). for example, we did not evaluate the effect of different graph embeddings on the actual scidocs performance. moreover, evaluations with smaller subsets of the s2orc citation graph are missing. such evaluations could indicate whether also less citation data can be sufficient, which would lower the compute requirements but would make scincl also applicable in domains where less graph data is available. tion graph of s2orc @xcite . more specifically, s2orc with the version identifier 20200705v1 is used. the full citation graph consists of 52.6m nodes (papers) and 467m edges (citations). tab. 4 presents statistics on the datasets and their overlap with specter and scidocs. the steps to reproduce both datasets are: replicated specter (w/ leakage) in order to replicate specter's training data and do not increase the leakage, we exclude all scidocs papers which are not used by specter from the s2orc citation graph. this means that apart from the 110,538 specter papers not a single other scidocs paper is included. the resulting citation graph has 52.5m nodes and 463m edges and is used for training the citation graph embeddings. for the scincl triplet selection, we also replicate specter's query papers and its corpus from which positive and negatives are sampled. our mapping and the underlying citation graph allows us to use 227,869 of 248,007 specter's papers for training. regarding query papers, we use 131,644 of 136,820 specter's query papers. to align the number training triplets with the one from specter, additional papers are randomly sampled from the filtered citation graph.