S ci R ep E val: A Multi-Format Benchmark for Scientific Document Representations (2023)

Abstract: Learned representations of scientific documents can serve as valuable input features for downstream tasks without further fine-tuning. However, existing benchmarks for evaluating these representations fail to capture the diversity of relevant tasks. In response, we introduce SciRepEval, the first comprehensive benchmark for training and evaluating scientific document representations. It includes 24 challenging and realistic tasks, 8 of which are new, across four formats: classification, regression, ranking and search. We then use this benchmark to study and improve the generalization ability of scientific document representation models. We show how state-of-the-art models like SPECTER and SciNCL struggle to generalize across the task formats, and that simple multi-task training fails to improve them. However, a new approach that learns multiple embeddings per document, each tailored to a different format, can improve performance. We experiment with task-format-specific control codes and adapters and find they outperform the existing single-embedding state-of-the-art by over 2 points absolute. We release the resulting family of multi-format models, called SPECTER2, for the community to use and build on.

## introduction learning representations of documents is critical for a variety of nlp tasks such as classification, search, and recommendation @xcite . recent work has shown how pre-trained language models (e.g. @xcite @xcite ) can be tailored to produce high-quality document representations with contrastive learning @xcite @xcite . in the scientific domain, contrastive learning of cross-document links (e.g. citations) has led to improved document-level representations @xcite @xcite . these representations can be indexed and consumed later by lightweight downstream models without additional fine-tuning. while there has been significant progress in evaluating generalizability of nlp models @xcite , evaluation of scientific document representations has remained limited. existing benchmarks either focus on document similarity @xcite or tasks that are highly correlated and not diverse @xcite . further, as shown in our experiments, a model's good performance on general-purpose text embedding benchmarks like the mteb @xcite may not translate as well to scientific tasks. we introduce scirepeval, the first benchmark for comprehensive evaluation of documentrepresentation models in the scientific domain. unlike prior work, scirepeval is large and includes a collection of highly diverse tasks, thus encouraging research on generalization (instance-level, crosstask and cross-domain). it consists of 24 realistic tasks that reflect practical use cases of scientific document representations across four formats: text classification, regression, proximity-based ranking (e.g., nearest-neighbor), and ad-hoc search. eight of these are new contributions. scirepeval contains standard sets of both training and evaluation datasets to simplify and standardize comparisons between methods evaluated on the benchmark. further, we use this benchmark to investigate and improve the generalization ability of document representation models. following recent work @xcite @xcite we pre-fine-tune a transformer model originally trained on citation triplets to produce high-quality representations for downstream tasks. we hypothesize that condensing all relevant information of a document into a single vector might not be expressive enough for generalizing across a wide range of tasks. prior work addresses a similar challenge in the context of document similarity and learns multiple representations associated with different aspects of a paper (e.g. task, method, results) @xcite . in contrast, we aim to learn effective representations for multiple downstream task formats. with the success of multi-task learning in nlp @xcite , we explore it in the context of scientific document representations by optimizing a suitable objective for every task format in scirepeval, i.e., cross-entropy for classification, triplet margin for proximity/adhoc search, and mean squared error for regression. we explore two state-of-the-art methods to generate format-specific representations: control codes @xcite as input signal; and parameter-efficient adapter methods @xcite , where a separate network module is introduced for every task format as in figure 1 . our experiments investigate (i) if existing document representation methods generalize to a highly diverse set of tasks, (ii) if multi-task training on diverse data can improve document representation models, and (iii) if task-format-specific representations can improve generalization. through extensive analysis we find that existing state-of-the-art scientific document representation models such as specter @xcite and scincl @xcite struggle with generalizing to multiple task types. we interestingly find that simple multi-task training on a large set of tasks does not lead to significant improvements. however, we learn that multiple task format-specific representations can substantially improve generalization. to summarize, our contributions are: (i) scirepeval, a new comprehensive benchmark of 24 highly diverse and practical tasks for scientific document representation techniques across four different formats, of which 8 are made available for the first time, and six are explicitly designed for training. both @xcite and @xcite are evaluated on scidocs. however, as discussed in section 3 and appendix g, this benchmark has important limitations. in contrast, scirepeval provides more challenging and diverse tasks to help motivate methods for producing scientific document representations that can generalize well across tasks. we attempt to learn task-specific embeddings of documents by pre-fine-tuning on multiple objectives simultaneously. @xcite and @xcite study the orthogonal task of generating multiple embeddings per paper for different "facets," while we aim to learn general embeddings for mutliple task formats. multi-task learning across formats multitask learning @xcite with deep neural networks has shown improvements upon singletask training for related objectives @xcite . though unrelated tasks can lead to negative transfer, recent work shows that simply increasing number of tasks yields better performance in multi-task learning @xcite @xcite . @xcite pre-fine-tune language models on 46 tasks from 4 task types before finetuning on the downstream task. @xcite pre-train t5 @xcite on c4 span denoising and 107 other tasks across 8 task families. @xcite introduce an ontology of 160 tasks for few shot multi-task training. unlike these task families, which are divided primarily by semantics (e.g. classifying sentiment vs entailment), the training tasks in scirepeval consist of 8 large scientific datasets across the four task formats. we wish to evaluate final document representations, so rather than fine-tune on downstream tasks as in above approaches, we directly apply the representations as features to the tasks @xcite . adapters for multiple tasks adapters were introduced by @xcite for parameter efficient training of transformers @xcite . a small number of trainable parameters are added to each layer, while freezing the base encoder. this is similar to elmo @xcite , which learned task-specific weightings for the bil-stm layers. to use adapters in a multi-task setup, @xcite define a two-step process they call fusion. first, individual adapter modules are trained for every task. the second step adds fusion modules at each layer which attend to (i.e. fuse) all the previously pre-fine-tuned adapters, keeping them fixed. similarly, stickland and @xcite introduced projected attention layers (pals) with adapters and self-attention modules for every task, but the entire network is trained simultaneously. control codes control codes can be defined as token(s) pre-pended to the input as additional signals for the model. @xcite use control codes as prompts to govern style, content, and taskspecific behavior for conditional text generation. @xcite use control codes to switch between three de-noising modes during pre-training, and associate every downstream task with a mode during fine-tuning. zhang et al. (2022) apply control codes to dense retrieval and produce multiple representations covering different aspects of the same document, allowing them to match queries written from multiple perspectives. in contrast to this, we use control codes to indicate the target task format for an embedding output by the model, and demonstrate how this is effective for producing paper embeddings across different formats. ## scirepeval we introduce scirepeval, a benchmark of 24 tasks across four formats to train and evaluate multi-task embeddings of scholarly papers. scirepeval aims to enable comprehensive evaluation of paper embeddings with: (1) a highly diverse set of tasks spanning multiple formats such as classification, regression, proximity and ad-hoc search to challenge the general-purpose applicability of embeddings, (2) realistic tasks that reflect practical use cases of paper embeddings, and (3) a standard set of training and evaluation datasets to simplify comparisons between methods evaluated on the benchmark. the previous scholarly paper embedding benchmark scidocs @xcite includes two classification, one recommendation, and four nearest neighbors tasks. scirepeval includes scidocs as a subset, but addresses several key limitations: (i) the four nearest neighbor tasks in scidocs are built to distinguish related papers from random negatives given a query paper, which might be too easy and not representative of real tasks in scholarly information retrieval. scirepeval has more realistic tasks like search, author disambiguation, and paper-reviewer matching among others. the tasks in scirepeval are summarized in next, we briefly describe each of the task formats and their component tasks. full details are provided in appendix a. all the tasks save one use paper embeddings created from a combination of paper title and abstract as the input. search requires additional metadata (subsection 4.1) which is concatenated to the title and abstract. ad-hoc search in ad-hoc search tasks, we are given a short textual query and the aim is to rank a task format name train + dev test eval metric source in-train clf mesh descriptors 2,328,179 258,687 macro f1 this work fields of study (fos) 676,524 s 471 g macro f1 this work rgn citation count 202,774 30,058 kendall's t this work year of publication 218,864 30,000 kendall's t this work prx same author detection q: 76,489 p: 673,170 q: 13,585 p: 123,430 map (subramanian et al., 2021) highly influential citations q: 65,982 p: 2,004,688 q: 1,199 p: 58,255 map this work citation prediction triplets 819,836 -*not used for eval (cohan et al., 2020) srch search q: 528,497 p: 5,284,970 q: 2,585 p: 25,850 ndgc this work out-of-train clf biomimicry -10,991 binary f1 (shyam et al., 2019) drsm -7,520 s; 955 g macro f1 (burns, 2022) scidocs mag -23,540 macro f1 (cohan et al., 2020) scidocs mesh diseases -25,003 macro f1 (cohan et al., 2020) rgn peer review score -10,210 kendall's t this work h-index of authors -8,438 kendall's t this work tweet mentions -25,655 kendall's t (jain and singh, 2021) prx s2and -x: 68,968 y: 10,942 b 3 f1 (subramanian et al., 2021) paper-reviewer matching -q:107 p: 1,729 p@5, p@10 (mimno and mccallum, 2007) (liu et al., 2014) (zhao et al., 2022) relish -q: 3190 p: 191,245 ndcg (brown et al., 2019) scidocs co-view -q: 1,000 p: 29,978 map, ndcg (cohan et al., 2020) scidocs co-read -q: 1,000 p: 29,977 map, ndcg (cohan et al., 2020) scidocs cite -q: 1,000 p: 29,928 map, ndcg (cohan et al., 2020) scidocs co-cite -q: 1,000 p: 29,949 map, ndcg (cohan et al., 2020) srch nfcorpus -q: 323 p: 44,634 ndcg (boteva et al., 2016) trec-covid -q: 50 p: 69,318 ndcg (voorhees et al., 2021) to evaluate the ad-hoc search tasks, candidate papers are ranked by increasing euclidean distance between the query and the candidate embeddings. pytrec_eval (van gysel and de rijke, 2018) is used to calculate the ranking metrics. normalized discounted cumulative gain (ndcg) is used for search and trec-covid as the true relevance score can @xmath0. proximity similar to ad-hoc search, proximity tasks involve ranking a set of candidate papers by their relatedness to a query, except the query in this case is a paper as well. proximity-based tasks form a basis for paper-based retrieval and recommendation, and for estimating paper similarity for use in applications like author disambiguation. we include a total of ten proximity-based tasks, including four nearest neighbor tasks from scidocs (predicting citations, co-citations, co-viewed or coread papers), and three others from previous work: the s2and author disambiguation task @xcite with paper similarity features; paper-reviewer matching @xcite @xcite , where candidate reviewers are ranked by expert annotators based on the similarity of their papers to the query paper to be reviewed; and finally relish @xcite : a document recommendation task consisting of ground truth labels collected by a consortium of over 1500 scientists worldwide. we also introduce three new large-scale training datasets aimed at predicting same-authors, citations (via triplets) as in @xcite , and influential citations, which we define as four or more citations of the same paper in the text of a single paper. for evaluation, we again rank candidate embeddings by euclidean distance, using map and ndcg as the scoring metric except for s2and with b 3 f1 @xcite , and paper-reviewer matching, which uses precision@5 and @10. classification classifying papers into topical categories is a foundational task for document organization and discovery. apart from the two scidocs tasks (mag and mesh diseases), we have four others; including a binary task to predict whether a paper is relevant to biomimicry @xcite , two biomedical classification tasks, namely drsm from burns (2022) and mesh descriptors classification @xcite , and a new largescale field of study (fos) multi-label training set of more than 500k papers with silver fos labels based on publication venues. we evaluate embeddings on classification by scoring their performance as features within linear support vector classifiers. results for these tasks are evaluated using binary/macro f1 score. to better understand how embeddings perform in data-scarce regimes, we also construct two fewshot versions each from biomimicry, drsm and fos dataset subset for which we have manually annotated gold labels. regression we also consider regression tasks where the goal is to predict a continuous quantity for a given paper. for evaluation, we consider predicting three numeric attributes related to prominence or quality: tweet mentions (jain and singh, 2021), and two new datasets predicting peer review rating and maximum h-index of authors for a set of iclr papers from openreview foot_0 . for training, we introduce two additional datasets; predicting citation count and publication year of papers. we evaluate embeddings on regression by scoring their performance as features within linear support vector regression models. the reported results are computed as kendall's τ rank correlation between the true and predicted labels. foot_1 4 multi-format representation learning typical approaches for learning document embeddings produce a single embedding for every task @xcite . we hypothesize that a single embedding is insufficient for generalizing across multiple downstream tasks. at the other extreme, learning embeddings for each task separately limits generalization to new tasks to help build intuition for why different embedding sets for different task formats may be helpful, figure 1 illustrates the qualitative distinctions between the task formats. in general, an embedding space yielding good results for one task format may be less suited to others; for example, the classification space provides an error-free linear classifier, but its nearest neighbor pairs are not always of the same class. empirically, we find that learning specialized embeddings per format improves performance, and that embeddings trained on a format tend to perform better on held-out tasks with the same format (see ## experiment setup training data our multi-format models are trained on the 8 large in-train tasks detailed in transformer baselines as a first step, we evaluate existing document representation methods on scirepeval. these include scibert @xcite , a language model pre-trained on scientific corpora, and paper-embedding methods specter @xcite , aspire @xcite , and scincl @xcite which is the state-of-the-art on scidocs. aspire produces representations for aspect-based matching between query and candidate papers which is a similar setting to our proximity tasks, so we only evaluate and report its results for that subset in appendix c. finally, we also evaluate two recent general-purpose text embedding methods, e5 v2 @xcite and mpnet @xcite . these methods are pre-trained on a set of over 1b query-candidate pairs including scientific text, and are two of the best bert-base sized models on the recent mteb benchmark @xcite for general-purpose text embeddings. ## results further, as shown in alternative base models to test the consistency of our findings for the proposed training techniques, we also train the mtl cls, mtl ctrl and adapters variants with specter and scincl as the base models. ## analyses specialization of control code embeddings our hypothesis is that by training embedding spaces on particular task formats, they will become more accurate for tasks of that format than for others. we test this by sampling one in-train and one as an extension to this we also analyze how well the control code representations work when the training tasks are randomly grouped together as opposed to by task format. from an evaluation across 5 random partitions, we found that task-formatbased partitions were better by over 2.7 points on average (both in-train and out-of-train) across the formats, suggesting that representations specific to each task format do lead to better results overall. finally, to study training affinity among the task formats themselves, we pre-fine-tune on at most two formats at once. appendix h reveals that combined multi-task training on similar task formats like proximity/adhoc-search results in performance gains, but only on the related tasks. training on all the tasks yields better results on average across the task formats. efficiency while the variants producing taskformat based representations are strong baselines on scirepeval as shown in ## conclusion we introduce scirepeval, a benchmark for scientific document representation methods with 24 tasks across four task formats. on this benchmark, we show that learning a separate document representation for each task format substantially improves task performance compared to learning a single representation for all tasks. future work could address limitations of our work by evaluating partitioning schemes beyond task format, crafting higher-fidelity metrics to account for the diversity of tasks in scirepeval (which vary in sensitivity and in relevance to downstream applications), or further exploring how accuracy varies with computational and storage cost.