SPECTER : Document-level Representation Learning using Citation-informed Transformers (2020)

Abstract: Representation learning is a critical ingredient for natural language processing systems. Recent Transformer language models like BERT learn powerful textual representations, but these models are targeted towards token- and sentence-level training objectives and do not leverage information on inter-document relatedness, which limits their document-level representation power. For applications on scientific documents, such as classification and recommendation, accurate embeddings of documents are a necessity. We propose SPECTER, a new method to generate document-level embedding of scientific papers based on pretraining a Transformer language model on a powerful signal of document-level relatedness: the citation graph. Unlike existing pretrained language models, Specter can be easily applied to downstream applications without task-specific fine-tuning. Additionally, to encourage further research on document-level models, we introduce SciDocs, a new evaluation benchmark consisting of seven document-level tasks ranging from citation prediction, to document classification and recommendation. We show that Specter outperforms a variety of competitive baselines on the benchmark.

## introduction as the pace of scientific publication continues to increase, natural language processing (nlp) tools that help users to search, discover and understand the scientific literature have become critical. in recent years, substantial improvements in nlp tools have been brought about by pretrained neural language models (lms) @xcite @xcite . while such models are widely used for representing individual words or sentences, extensions to whole-document embeddings are relatively underexplored. likewise, methods that do use inter-document signals to produce whole-document embeddings @xcite ) have yet to incorporate stateof-the-art pretrained lms. here, we study how to leverage the power of pretrained language models to learn embeddings for scientific documents. a paper's title and abstract provide rich semantic content about the paper, but, as we show in this work, simply passing these textual fields to an "off-the-shelf" pretrained language model-even a state-of-the-art model tailored to scientific text like the recent scibert @xcite -does not result in accurate paper representations. the language modeling objectives used to pretrain the model do not lead it to output representations that are helpful for document-level tasks such as topic classification or recommendation. in this paper, we introduce a new method for learning general-purpose vector representations of scientific documents. our system, specter, foot_0 incorporates inter-document context into the transformer @xcite language models (e.g., scibert @xcite ) to learn document representations that are effective across a wide-variety of downstream tasks, without the need for any task-specific fine-tuning of the pretrained language model. we specifically use citations as a naturally occurring, inter-document incidental supervision signal indicating which documents are most related and formulate the signal into a triplet-loss pretraining objective. unlike many prior works, at inference time, our model does not require any citation information. this is critical for embedding new papers that have not yet been cited. in experiments, we show that specter's representations substantially outperform the state-of-the-art on a variety of document-level tasks, including topic classification, citation prediction, and recommendation. as an additional contribution of this work, we introduce and release scidocs 3 , a novel collection of data sets and an evaluation suite for documentlevel embeddings in the scientific domain. sci-docs covers seven tasks, and includes tens of thousands of examples of anonymized user signals of document relatedness. we also release our training set (hundreds of thousands of paper titles, abstracts and citations), along with our trained embedding model and its associated code base. ## scidocs evaluation framework previous evaluations of scientific document representations in the literature tend to focus on small datasets over a limited set of tasks, and extremely high (99%+) auc scores are already possible on these data for english documents @xcite . new, larger and more diverse benchmark datasets are necessary. here, we introduce a new comprehensive evaluation framework to measure the effectiveness of scientific paper embeddings, which we call scidocs. the framework consists of diverse tasks, ranging from citation prediction, to prediction of user activity, to document classification and paper recommendation. note that specter will not be further fine-tuned on any of the tasks; we simply plug in the embeddings as features for each task. below, we describe each of the tasks in detail and the evaluation data associated with it. in addition to our training data, we release all the datasets associated with the evaluation tasks. ## results for document classification, we report macro f1, a standard classification metric. we observe that the classifier performance when trained on our representations is better than when trained on any other baseline. particularly, on the mesh (mag) dataset, we obtain an 86.4 (82.0) f1 score which is about a ∆= + 2.3 (+1.5) point absolute increase over the best baseline on each dataset respectively. our evaluation of the learned representations on predicting user activity is shown in the "user activity" columns of finally, for recommendation task, we observe that specter outperforms all other models on this task as well, with ndcg of 53.9. on the recommendations task, as opposed to previous experiments, the differences in method scores are generally smaller. this is because for this task the embeddings are used along with several other informative features in the ranking model (described under task-specific models in §4), meaning that embedding variants have less opportunity for impact on overall performance. we also performed an online study to evaluate whether specter embeddings offer similar advantages in a live application. we performed an online a/b test comparing our specter-based recommender to an existing production recommender system for similar papers that ranks papers by a textual similarity measure. in a dataset of 4,113 clicks, we found that specter ranker improved clickthrough rate over the baseline by 46.5%, demonstrating its superiority. we emphasize that our citation-based pretraining objective is critical for the performance of specter; removing this and using a vanilla scib-ert results in decreased performance on all tasks. ## analysis in this section, we analyze several design decisions in specter, provide a visualization of its embedding space, and experimentally compare specter's use of fixed embeddings against a finetuning approach. ablation study we start by analyzing how adding or removing metadata fields from the input to specter alters performance. the results are shown in the top four rows of with paper topics). the fact that specter does not require inputs like authors or venues makes it applicable in situations where this metadata is not available, such as matching reviewers with anonymized submissions, or performing recommendations of anonymized preprints (e.g., on openreview). one design decision in specter is to use a set of hard negative distractors in the citation-based finetuning objective. the fifth row of visualization figure 2 shows t-sne (van der maaten, 2014) projections of our embeddings (specter) compared with the scibert baseline for a random set of papers. when comparing specter embeddings with scibert, we observe that our embeddings are better at encoding topical information, as the clusters seem to be more compact. further, we see some examples of crosstopic relatedness reflected in the embedding space (e.g., engineering, mathematics and computer science are close to each other, while business and economics are also close to each other). to quantify the comparison of visualized embeddings in figure 2 , we use the dbscan clustering algorithm @xcite on this 2d projection. we use the completeness and homogeneity clustering quality measures introduced by @xcite . for the points corresponding to figure 2 , the homogeneity and completeness values for specter are respectively 0.41 and 0.72 compared with scibert's 0.19 and 0.63, a clear improvement on separating topics using the projected embeddings. comparison with task specific fine-tuning while the fact that specter does not require finetuning makes its paper embeddings less costly to use, often the best performance from pretrained transformers is obtained when the models are finetuned directly on each end task. we experiment with fine-tuning scibert on our tasks, and find this to be generally inferior to using our fixed representations from specter. specifically, we finetune scibert directly on task-specific signals instead of citations. to fine-tune on task-specific data (e.g., user activity), we used a dataset of coviews with 65k query papers, co-reads with 14k query papers, and co-citations (instead of direct citations) with 83k query papers. as the end tasks are ranking tasks, for all datasets we construct up to 5 triplets and fine-tune the model using triplet ranking loss. the positive papers are sampled from the most co-viewed (co-read, or co-cited) papers corresponding to the query paper. we also include both easy and hard distractors as when training specter (for hard negatives we choose the least non-zero co-viewed (co-read, or co-cited) papers). we also consider training jointly on all task-specific training data sources in a multitask training process, where the model samples training triplets from a distribution over the sources. as illustrated in there have been other efforts in document representation learning such as extensions of word vectors to documents @xcite @xcite @xcite , convolution-based methods @xcite , and variational autoencoders @xcite . relevant to document embedding, sentence embedding is a relatively well-studied area of research. successful approaches include seq2seq models @xcite , bilstm siamese networks @xcite , leveraging supervised data from other corpora @xcite , and using discourse relations @xcite , and bert-based methods @xcite . unlike our proposed method, the majority of these approaches do not consider any notion of inter-document relatedness when embedding documents. other relevant work combines textual features with network structure @xcite @xcite @xcite . these works typically do not leverage the recent pretrained contextual representations and with a few exceptions such as the recent work by @xcite , they cannot generalize to unseen documents like our specter approach. context-based citation recommendation is another related application where models rely on citation contexts @xcite to make predictions. these works are orthogonal to ours as the input to our model is just paper title and abstract. another related line of work is graphbased representation learning methods @xcite @xcite . here, we compare to a graph representation learning model, sgc (simple graph convolution) @xcite , which is a state-of-the-art graph convolution approach for representation learning. specter uses pretrained language models in combination with graph-based citation signals, which enables it to outperform the graph-based approaches in our experiments. specter embeddings are based on only the title and abstract of the paper. adding the full text of the paper would provide a more complete picture of the paper's content and could improve accuracy @xcite @xcite . however, the full text of many academic papers is not freely available. further, modern language models have strict memory limits on input size, which means new techniques would be required in order to leverage the entirety of the paper within the models. exploring how to use the full paper text within specter is an item of future work. finally, one pain point in academic paper recommendation research has been a lack of publicly available datasets @xcite . to address this challenge, we release scidocs, our evaluation benchmark which includes an anonymized clickthrough dataset from an online recommendations system. ## conclusions and future work we present specter, a model for learning representations of scientific papers, based on a transformer language model that is pretrained on cita-tions. we achieve substantial improvements over the strongest of a wide variety of baselines, demonstrating the effectiveness of our model. we additionally introduce scidocs, a new evaluation suite consisting of seven document-level tasks and release the corresponding datasets to foster further research in this area. the landscape of transformer language models is rapidly changing and newer and larger models are frequently introduced. it would be interesting to initialize our model weights from more recent transformer models to investigate if additional gains are possible. another item of future work is to develop better multitask approaches to leverage multiple signals of relatedness information during training. we used citations to build triplets for our loss function, however there are other metrics that have good support from the bibliometrics literature @xcite ) that warrant exploring as a way to create relatedness graphs. including other information such as outgoing citations as additional input to the model would be yet another area to explore in future.