Scientific Paper Retrieval with LLM -Guided Semantic-Based Ranking (2025)

Abstract: Scientific paper retrieval is essential for supporting literature discovery and research. While dense retrieval methods demonstrate effectiveness in general-purpose tasks, they often fail to capture fine-grained scientific concepts that are essential for accurate understanding of scientific queries. Recent studies also use large language models (LLMs) for query understanding; however, these methods often lack grounding in corpus-specific knowledge and may generate unreliable or unfaithful content. To overcome these limitations, we propose SemRank, an effective and efficient paper retrieval framework that combines LLM-guided query understanding with a concept-based semantic index. Each paper is indexed using multi-granular scientific concepts, including general research topics and detailed key phrases. At query time, an LLM identifies core concepts derived from the corpus to explicitly capture the query’s information need. These identified concepts enable precise semantic matching, significantly enhancing retrieval accuracy. Experiments show that SemRank consistently improves the performance of various base retrievers, surpasses strong existing LLM-based baselines, and remains highly efficient.

## introduction scientific paper retrieval is a crucial task to facilitate literature discovery and accelerate scientific progress @xcite . unlike generalpurpose information retrieval, scientific paper retrieval is more challenging because queries often involve theme-specific intent and specialized terminology. in addition, acquiring labeled querypassage pairs for supervised fine-tuning is costly and requires domain expertise, making it impractical to continuously annotate more data to adapt the fast-evolving scientific domains. recently, dense passage retrieval methods have been widely studied in various ad-hoc searches @xcite . these methods encode the overall semantics of queries and passages into the same vector space and measure relevance using vector similarity. although being effective in different general domain applications, they still face challenges in scientific paper retrieval. specifically, general-purpose semantic representations learned by dense retrievers often fail to capture fine-grained scientific concepts that are crucial for accurately understanding and satisfying a scientific query @xcite . for example, the query "can you point me to studies discussing methods for evaluating text generation models on various dimensions?" involves not only general topics like "natural language generation" and "automatic evaluation" that need to be inferred from the text, but also specific details like "multidimensional evaluation". a dense retriever, however, only encodes the text in a holistic view, while it lacks the ability and controllability to focus on the scientific concepts which are the core need of the query. with the advancements of large language models (llm) such as gpt @xcite and claude @xcite , recent studies also explore how to utilize llms in query understanding to help retrieval tasks. for example, hyde @xcite uses an llm to generate a hypothetical passage for encoding, and csqe @xcite prompts an llm to select a set of relevant sentences to expand the original query. however, these methods still rely on a pre-trained dense retriever to encode overall semantics on the document or sentence levels, lacking the ability to explicitly capture what the query is asking for. in addition, llms are not inherently retrieval models. they do not have the vast and dynamic knowledge in the scientific literature necessary to understand scientific queries. therefore, when used in a zero-shot manner, llms cannot identify domain-specific terminology and may generate hallucinated content. thus, how to effectively augment llms with corpus-based knowledge while capturing specific query information remains a challenge. to overcome these limitations, we propose to utilize llms' text understanding ability in the scientific retrieval task with the help of a scientific concept-based semantic index. unlike earlier studies which construct domain-specific semantic index at topic level @xcite , we build concept-based semantic index at various granularities: from broad research topics such as "natural language generation" to specific terms such as "multidimensional evaluation metrics". these multi-granular concepts capture the essential content of the paper. then, we use the semantic index to improve any existing retrievers with the help of llms. specifically, we prompt an llm to identify a set of core concepts to explicitly represent what the scientific query is asking for. we augment the prompt with candidate concepts derived from the corpus, which helps the llm to reduce hallucination and ensure the generated content align with the semantic index of corpus. figure 1 shows an example that, by accurately capturing the multi-granular concepts of the corpus and query, we can improve the scientific paper retrieval results by focusing on the core need of the query. we propose the semrank, llm-guided semantic-based ranking, a plug-and-play framework for scientific paper retrieval. first, during the indexing time, we build scientific concept-based semantic index for the corpus by identifying a set of research topics and key phrases for each paper. to ensure the topic labels in their canonical forms, we train an auxiliary topic classifier model to identify a set of candidate topics from a large label space foot_1 . then, we prompt an llm to select the core topic labels and extract key phrases to build the semantic index. then given a query during retrieval, we first construct a set of candidate concepts from the corpus using a base retriever, from which an llm can be augmented with corpus-based knowledge and identify a set of core concepts for the query. these concepts, serving as an explicit guidance on what the query is asking for, will be used for conceptlevel semantic matching to improve the retrieval results. our method can be easily integrated with any dense retriever and improve their retrieval quality without relying on any annotated query data. experiments show that our method is both effective and efficient. during retrieval time, semrank only needs one llm prompting per query, no additional call to the base retriever, and all computation can be done easily on cpus. yet it can significantly improve the ranking performance of a wide range of base retrievers and outperforms various llmbased baselines. the contributions of this paper are as follows: • we propose semrank, a plug-and-play framework for scientific paper retrieval, which utilizes concept-based semantic index and llm guidance to explicit capture the information need for scientific queries. sec 3.1: semantic index construction (offline) sec 3.2: llm-guided semantic-based retrieval q: can you point me to studies about methods for evaluating text generation models on various dimensions? retriever corpus concept frequency ranking … … … … … initial top-ranked papers feq topic corpus concept-based semantic score final retrieved papers 0.9, 0.78, …, 0.14 key phrase top-k selection llm & topic classifier … doc id concept-based semantic index … … … … doc i feq query concepts: natural language generation, multi-dimensional evaluation, summarization, dialogue response generation, … llm corpus with semantic index topic: nlg, auto evaluation… phrase: multidimensional, dialogue response generation… semrank consistently improves the base retrievers' performance and outperforms existing baselines while being more efficient. ## problem formulation given a corpus d of scientific papers, our goal is to retrieve and rank the papers according to their relevance to a given query q. specifically, we will build concept-based semantic index for the corpus by identifying a set of scientific concepts representing the core information of each paper. we assume a scientific topic label space t , which is widely available and normally contains a large number of research topics. in this work, we use microsoft academic graph @xcite , which contains 13,613 computer science topics. then, we utilize llms and the semantic index to improve the scientific paper retrieval performance. ## methodology in this section, we will present our semrank framework. we first introduce the offline semantic index construction module (sect. 3.1), then we introduce the llm-guided semantic-based ranking for scientific paper retrieval (sect. 3.2). figure 2 shows an overview of semrank. ## conclusion we present semrank, a novel scientific paper retrieval method that integrates llm-guided query understanding with a concept-based semantic index. to overcome the limitations of existing methods, semrank identifies multi-granular scientific concepts to explicitly understand scientific queries at the concept level. by augmenting llms with corpus knowledge, semrank also facilitates llm's understanding of query and context while reducing hallucination. experiments demonstrate that semrank consistently improves retrieval performance across various base retrievers and outperforms various baseline methods while remaining highly efficient.