L it S earch: A Retrieval Benchmark for Scientific Literature Search (2024)

Abstract: Literature search questions, such as “where can I find research on the evaluation of consistency in generated summaries?” pose significant challenges for modern search engines and retrieval systems. These questions often require a deep understanding of research concepts and the ability to reason over entire articles. In this work, we introduce LitSearch, a retrieval benchmark comprising 597 realistic literature search queries about recent ML and NLP papers. LitSearch is constructed using a combination of (1) questions generated by GPT-4 based on paragraphs containing inline citations from research papers and (2) questions about recently published papers, manually written by their authors. All LitSearch questions were manually examined or edited by experts to ensure high quality. We extensively benchmark state-of-the-art retrieval models and also evaluate two LLM-based reranking pipelines. We find a significant performance gap between BM25 and state-of-the-art dense retrievers, with a 24.8% difference in absolute recall@5. The LLM-based reranking strategies further improve the best-performing dense retriever by 4.4%. Additionally, commercial search engines and research tools like Google Search perform poorly on LitSearch, lagging behind the best dense retriever by 32 points. Taken together, these results show that LitSearch is an informative new testbed for retrieval systems while catering to a real-world use case.

## introduction finding literature via a specific search query-for example, collecting related work, checking if a method has been proposed before, or recalling a previously seen paper-is a critical task for researchers. developing systems that recommend citations pertinent to such inquiries holds the potential to enhance researchers' productivity and expedite scientific discovery @xcite . however, this task is inherently challenging as it often requires deep domain expertise and reasoning through lengthy papers. prior to this study, the task of citation recommendation was often formalized by using inline citation mentions from existing papers as queries, and the cited papers as targets @xcite . for instance, given the citation mention "roberta and t5 are based on recent advances in masked language modeling @xcite ," the text surrounding the citation mention is used as a retrieval query, and the cited paper is the target literature. however, directly using inline citations often leads to queries that are noisy, overly broad (e.g., "large language models [citation]"), or highly context-dependent (e.g., "we follow the hy- source paper title: roberta: ... : ... : unlike , ... we do not train with a reduced sequence length for the first 90% of updates. source paper abstract citation mention @xcite sample an inline citation mention question target paper : is there a pre-trained lm that is trained with reduced sequence length for the first 90% steps? : (bert: pre-training ...) @xcite figure 2 : the pipeline for generating inline-citation questions. we first sample a citation mention and prompt gpt-4 to generate a question. next, we filter questions based on word overlap with the target paper title and perform manual inspections to annotate their specificity and quality (see rubrics in ## litsearch our benchmark litsearch consists of (a) a large corpus of scientific papers p and (b) pairs of literature search questions and one or more target papers from p. our desiderata are scientific questions that researchers may use while conducting literature surveys. we use two different strategies to collect such questions: (1) we construct questions using the surrounding context from inline citations in published papers (section 2.1), and (2) we invited the authors of recent conference publications to manually write questions about their own papers (section 2.2). for both subsets, we ensure high question quality via manual inspection and filtering conducted by the authors of this work (section 2.3). ## related work citation recommendation. the community has proposed a number of citation recommendation datasets @xcite , including global citation recommendation datasets (directly using a paper as the query and papers it cites as target papers; @xcite , and local citation recommendation datasets (using inline citation mentions as queries; @xcite @xcite . there are also language models and retrieval models specifically trained for scientific document understanding and retrieval tasks, such as scibert @xcite and specter @xcite . compared to existing citation recommendation datasets, litsearch is comprised of manually annotated, natural language literature search questions, providing a more realistic and challenging evaluation for citation recommendation systems. retrieval benchmarks. there have been numerous datasets evaluating retrieval systems from wikipedia @xcite , web queries @xcite , biomedical questions @xcite , and more. recently, there have been several benchmarks combining multiple datasets and evaluating retrieval or embedding models across different domains and different use cases, such as kilt @xcite , beir @xcite , and mteb @xcite . litsearch offers a unique perspective by exploring the novel literature search question type, effectively complementing the existing benchmarks. contemporary with our work, citeme @xcite introduces a benchmark for identifying references based on claims made in a paper's inline text (as opposed to research questions in our case). their focus differs from ours in that it is aimed more at llm-based agents rather than retrieval systems. retrieval systems. traditional retrieval systems rely on bag-of-word algorithms such as tf-idf and bm25. dense retrieval (embedding) models have gained more popularity due to their abilities to do semantic search without relying on exact keyword matches @xcite . state-of-the-art dense models are mostly adopted by fine-tuning pre-trained language models @xcite with a contrastive learning objective on either supervised or unsupervised data @xcite @xcite @xcite . recent development introduces "instructions" when encoding queries and documents, which significantly improves the versatility of embeddings across tasks @xcite @xcite @xcite . ## conclusion in this paper, we propose litsearch, a new retrieval benchmark comprising 597 manually-curated literature search questions. litsearch includes an inline-citation question set and an author-written question set, both undergoing manual inspection from the authors of litsearch. we conduct extensive experiments with bm25, state-of-the-art embedding models, and llm reranking. our experiments demonstrate the superior performance of state-of-the-art instruction-finetuned embedding models, with additional improvement via gpt-4obased reranking. we also verify that commercial search engines like google struggle with litsearch questions. the comparison with existing retrieval benchmarks shows that litsearch better differentiates the performance of retrieval systems.