Literature discovery with natural language queries (2025)

Abstract: Literature discovery is a critical component of scientific research. Modern discovery systems leveraging Large Language Models (LLMs) are increasingly adopted for their ability to process natural language queries (NLQs). To assess the robustness of such systems, we compile two NLQ datasets and submit them to nine widely used discovery platforms. Our findings reveal that LLM-based search engines struggle with precisely formulated queries, often producing numerous false positives. However, precision improves when LLMs are used not for direct retrieval but to convert NLQs into structured keyword-based queries. As a result, hybrid systems that integrate both LLM-driven and keyword-based approaches outperform purely keyword-based or purely LLM-based discovery methods.

## introduction scientific research heavily relies on the ability to discover and assimilate relevant literature @xcite . traditional literature search methods, whether through publisher-specific databases (e.g., nature 1 ) or generic academic search engines (e.g., google scholar foot_0 ), primarily use keywordbased queries processed via inverted indexes and ranking algorithms such as bm25 @xcite . while these conventional approaches are effective, they often struggle with nuanced, conceptdriven queries. recent advancements in artificial intelligence, particularly the rise of large language models (llms) and llm-powered chatbots (e.g. consensus 3 ), have enabled a more intuitive and contextaware search experience. however, despite their convenience, llm-driven retrieval systems lack formal guarantees of accuracy @xcite ). 1 nature.com/search/advanced 2 scholar.google.com 3 consensus.app this limitation can lead to erroneous search results, including false positives (retrieving irrelevant papers) and false negatives (overlooking relevant papers), potentially impacting the reliability of literature discovery. to systematically assess these challenges, we conduct an evaluation of various literature search engines using natural language queries (nlqs). we find that existing platforms are not equipped to handle nlqs, with most papers retrieved being incorrect, but that using llms to parse nlqs into structured queries interpretable by these platforms highly boosts retrieval performance. our contributions are: 1. we introduce two manually curated datasets 4 designed for systematic evaluation of literature discovery platforms on nlqs. 2. we benchmark the performance of nine popular literature discovery platforms on our datasets. 3. we investigate the ability of llms to transform nlqs into structured formats and analyze their impact on retrieval effectiveness. gines are prone to hallucination @xcite , grounding llm-powered literature discovery platforms by referencing a paper database remains key. retrieving information from most databases involves submitting structured queries that are interpretable by said databases. to reduce the need for learning about the structure accepted by each database, much research has gone into automatically translating nlqs into structured queries @xcite . a typical approach was to manually craft a mapping table between mentioned keywords and database properties @xcite , but these tables are often too rigid to effectively handle query ambiguity and complexity. in contrast, recent works have successfully used llms for this mapping problem @xcite , and in this work, we explore whether this effectiveness extends to the specific context of precise nlqs for literature discovery. ## methods a key principle of effective retrieval systems is to provide users with fine-grained control over retrieval behaviour @xcite . in literature discovery, this control would include enabling researchers to specify authors, venue, publication year, and topics of interest. alternatively, researchers might seek papers on multidisciplinary topics defined by one or several keywords that are rarely used together. such complex queries are well-suited to classical search engines but may pose challenges for llms, which we seek to explore. ## conclusion our findings highlight key strengths and limitations of llm-based literature discovery systems. while these systems struggle with precise nlqs, llm parsing significantly enhances retrieval performance, particularly when integrated with structured search engines. this suggests that hybrid approaches combining llm-based and structured retrieval methods are more promising for literature discovery and could bridge the gap between the flexibility of human-like queries and the structured nature of conventional search engines, effectively mitigating the challenges posed by ambiguous or instruction-based nlqs. additionally, llm-based systems prove valuable in scenarios where structured queries are not feasible or when queries do not conform to strict database formats. future work should explore refining hybrid methodologies to further optimize retrieval accuracy and relevance.