Large-scale Evaluation of Transformer-based Article Encoders on the Task of Citation Recommendation (2022)

Abstract: Recently introduced transformer-based article encoders (TAEs) designed to produce similar vector representations for mutually related scientific articles have demonstrated strong performance on benchmark datasets for scientific article recommendation. However, the existing benchmark datasets are predominantly focused on single domains and, in some cases, contain easy negatives in small candidate pools. Evaluating representations on such benchmarks might obscure the realistic performance of TAEs in setups with thousands of articles in candidate pools. In this work, we evaluate TAEs on large benchmarks with more challenging candidate pools. We compare the performance of TAEs with a lexical retrieval baseline model BM25 on the task of citation recommendation, where the model produces a list of recommendations for citing in a given input article. We find out that BM25 is still very competitive with the state-of-the-art neural retrievers, a finding which is surprising given the strong performance of TAEs on small benchmarks. As a remedy for the limitations of the existing benchmarks, we propose a new benchmark dataset for evaluating scientific article representations: Multi-Domain Citation Recommendation dataset (MDCR), which covers different scientific fields and contains challenging candidate pools.

## introduction the introduction of large pre-trained language models (lms) @xcite @xcite based on the transformer architecture @xcite has improved performance on numerous nlp tasks. the adaptation of lms to scientific corpora @xcite @xcite laid the foundation for applying transformer-based lms to various scholarly document processing (sdp) tasks, such as named-entity recognition @xcite , article summa-rization @xcite , scientific fact-checking @xcite , describing relationships between articles @xcite , and citation recommendation (cr) @xcite , among others. while some of the sdp tasks rely on word-or sentence-level representations, others, such as cr and article summarization, require document-level representations. to obtain such representations, recent work has proposed various transformer-based article encoders (taes), i.e., lms that are finetuned using citation or co-citation information as a training signal, such as specter @xcite , aspire @xcite , and scincl @xcite . representations obtained with these models can then be used in various downstream recommendation tasks where a user searches for articles that are in some way relevant to a given query article. to date, article representations obtained with taes have been evaluated against recommendation benchmarks such as scidocs @xcite , relish, @xcite or trec-covid @xcite . while scidocs focuses mainly on the field of computer science, relish and treccovid cover articles from the biomedical field. these benchmarks contain a set of query articles, where each query is paired with a candidate pool consisting of both relevant and irrelevant articles for that query. the difference between the benchmarks, apart from the domains they cover, is how the candidate pools are constructed: relish and treccovid contain expert-annotated relevance labels for each candidate in a pool, while scidocs uses random sampling of negative candidates for pool construction. however, they all contain relatively small candidate pools (e.g., 25 in scidocs). such small and, in some cases, randomly sampled candidate pools do not resemble typical use-case scenarios in which query articles are compared to millions of candidate articles from public databases. thus, evaluating taes on such benchmarks can lead to an overly optimistic performance estimation as the candidate pool does not constitute a representative sample of the population of candidate articles in realistic use cases. in this work, we turn to a more realistic evaluation of taes and evaluate them on large (≥200k) candidate pools and across different scientific fields. although emulating such a realistic setup has so far been avoided due to the prohibitive computational cost of nearest neighbor search on millions of embeddings, research on gpu-based nearest neighbor (nn) search @xcite has given rise to efficient techniques that enable embedding-based search in large-scale setups. to make use of fast nn search, we focus on the bi-encoder models @xcite , that can be easily coupled with fast gpu-based nn search. we evaluate taes on the task of cr, in which a model outputs a list of articles as recommendations for citing in a given article. alongside taes, we evaluate the traditional lexical retrieval model bm25 @xcite , which, in spite of its simplicity, still stands as a hard-to-beat baseline in many retrieval tasks. our evaluation shows that bm25 performs on par with taes in this setup, especially as candidate pools grow. building on the results of our large-scale evaluation of taes, we then construct a new benchmark dataset for evaluating scientific article representations on the task of cr. our multi-domain crbased benchmark dataset (mdcr), albeit comparable in size to previous benchmarks, spans different scientific fields and consists of challenging candidate pools. more precisely, candidate pools in mdcr contain different candidate types, ranging from those obtained from the large-scale evaluation of state-of-the-art taes to candidates from the citation graph neighborhood. to summarize, the contribution of our work is twofold: (1) we conduct a large-scale evaluation of state-of-the-art taes on pools of varying sizes, and (2) present a new and challenging multi-domain benchmark dataset for evaluating scientific article representations that contains challenging candidates identified in the large-scale evaluation. 1 the rest of the paper is organized as follows. in section 2, we describe the models we evaluate and give an overview of the existing benchmarks 1 our code, the data splits and the new benchmark data are publicly available at the following link: https://github. com/zoranmedic/mdcr . for scientific article recommendation. section 3 presents the results of a large-scale evaluation of taes and bm25 in two evaluation setups. in section 4 we describe the construction of a new and more challenging multi-domain benchmark and present the initial results for the models we considered. section 5 concludes the paper and proposes future work. ## large-scale evaluation we performed the large-scale evaluation in two setups: dataset-and field-level. dataset-level evaluation resembles a basic evaluation setup -a random sampling of both queries and articles in the candidate pool. the field-level evaluation focuses on specific scientific fields using queries and candidate pools comprised of articles from specific fields. in both setups, we evaluated the chosen models on the task of global cr, in which a model is trained to produce a list of articles as recommendations for citing in a given query article. although cr is not the only task on which taes can be evaluated, it is arguably the most accessible among the article retrieving tasks. whereas other tasks (e.g., user activity tasks) might require data that is typically not publicly available (e.g., search engine logs), cr datasets are easily obtained through parsing reference lists of publicly available articles. previous research on global cr has proposed many features that could be used to represent the input articles @xcite . however, in this work, we only use the article's title and abstract as input, as our focus is not on improving the state-of-the-art in global cr but rather on evaluating the tae-produced article representations in a retrieval scenario. for a detailed overview of the various tasks and methods in cr, we refer the reader to @xcite . for each tae that we consider, the input was constructed by concatenating the input article's title and abstract (separated with a [sep] token). for scibert, specter, and scincl, we used the final layer's [cls] token embedding as input article's representation, while for aspire we meanpooled token embeddings across all layers for each sentence in the input. we used huggingface's foot_1 implementations of taes, while for bm25 we used lucene's implementation, i.e., its python toolkit pyserini. 4 for nearest neighbor search across article embeddings, we used faiss @xcite . 5 we used the s2orc dataset @xcite in all our experiments. s2orc is a recently released large dataset of 81.1m scientific articles covering dozens of scientific fields. together with the metadata and article's title and abstract, the dataset contains citation links between the articles. therefore, we consider it appropriate for the large-scale evaluation, not just due to its size and coverage but recency as well. we perform initial filtering of articles and remove all those with (1) empty publication year field, (2) empty title field, (3) abstract shorter than 30 characters, or (4) less than three citations in s2orc. this filtering leaves us with a prefiltered set of around 16m articles that we use for both sampling of queries and candidate pool construction in both evaluation setups. for both evaluation setups, we report the standard metrics used in prior work on scientific article recommendation: map, ndcg, and r@30. we set k in r@k to 30, since on average there are 29 positives (cited articles) for each query in the query set. all metrics range from 0 to 1, where higher is better. although defined differently, all the metrics yield higher values when relevant articles are positioned higher in the list of retrieved articles. ## multi-domain citation recommendation benchmark we now present our newly constructed multi-domain citation recommendation benchmark -mdcr. as queries in mdcr, we use the same 200 queries per field as in the field-level evaluation setup ( §3.2). for the candidate pools, we start with a random sampling of 5 articles cited in the query article and then select negative candidates. ## conclusion we evaluated transformer-based article encoders in large-scale citation recommendation scenarios across different scientific fields and candidate pool sizes. together with transformer-based encoders, we evaluated the performance of a robust lexical retrieval baseline bm25 and demonstrated that it still performs competitively with recent neural-based models. in the case of large field-specific candidate pools, bm25 outperformed transformer-based models in most fields. furthermore, to promote a more realistic and a more diverse evaluation across different fields in comparison to the existing benchmarks used for evaluating scientific article representations, we presented a new multi-domain benchmark dataset based on citation recommendation task, which we call mdcr. evaluation on mdcr demonstrated the difficulty of specific candidate types and set the ground for evaluating future scientific article encoders. our evaluation demonstrated the varying performance across scientific fields, which we believe should be analyzed in future work to improve encoders' performance across all fields, not just those prevailing in the datasets. given that our bench-mark dataset is not expert-annotated but rather based on citations as relevance signals, we propose constructing an expert-annotated dataset with articles from different scientific fields. we hope our contributions will stimulate the community to work on more realistic and challenging evaluation setups of scientific article recommendation models.