A uto E volve: Automatically Evolving Queries for Applicable and Scalable Retrieval-Augmented Generation Benchmarking (2025)

Abstract: Retrieval-augmented generation (RAG) enables large language models (LLMs) to address queries beyond their internal knowledge by integrating domain knowledge in specialized corpus, which necessitates the generation of benchmarks on specific corpus to evaluate RAG systems. However, existing automated generation methods exhibit Weak Applicability and Weak Scalability. Weak Applicability refers to the reliance on metadata from specific corpora for query generation, constraining applicability to other corpora. Weak Scalability is characterized by fixed query content after generation, unable to dynamically increase difficulty, limiting scalability of the query. To overcome these issues, we propose AutoEvolve, an applicable approach for dynamically evolving queries to construct scalable RAG benchmarks. Our approach is grounded in three key innovations: (i) a corpus-agnostic method for constructing the universal entity-document graph; (ii) a suite of evolution operations designed to dynamically update queries; and (iii) a difficulty-guided metric that directs query evolution process. Through experiments on three generated benchmarks, we demonstrate that AutoEvolve evolves queries that are significantly more challenging, paving the way for more applicable and scalable RAG evaluations.

## introduction retrieval-augmented generation (rag) systems are attracting growing attention @xcite @xcite due to their capacity to enable large language models (llms) to answer queries beyond their internal knowledge by integrating domain knowledge in specialized corpus. therefore, it is necessary to automatically construct an effective benchmark on a specific corpus rather than relying on generic benchmarks to evaluate the rag system. however, existing methods @xcite for automatically generating benchmarks face issues of weak applicability and weak scalability. weak applicability pertains to reliance on metadata derived from specific corpora, which lacks the ability to apply to other corpora. specifically, @xcite utilize link attributes from wikipedia corpus to generate multi-hop queries, whereas @xcite leverages schema extracted from specific domains to assist in generating a diverse array of query types. weak scalability refers to the fixed query content after generation, which cannot dynamically increase the difficulty, thereby limiting scalability of the query. to overcome these issues, we propose autoevolve, an applicable approach for automatically evolving queries to construct scalable rag benchmarks, as illustrated in figure 1 . specifically, to alleviate the issue of weak applicability, we utilize a corpus-agnostic method to construct a universal entity-document relationship graph to establish the relationships among documents by extracting consistent entities from documents. simultaneously, we propose an evolutionary framework to dynam-ically enhance the difficulty of queries based on the graph, thereby addressing the issue of weak scalability. specifically, we introduce a suite of evolution operations to dynamically update the existing queries and present a difficulty-guided metric without the intervention of llms for assessing query difficulty to guide the direction of evolution in the framework. in experiments, we demonstrate the applicability of autoevolve by generating benchmarks from three corpora and evaluating seven retrievers and llms. furthermore, we compare the performance of retrievers before and after evolution, as well as the overall performance of the rag system, demonstrating the scalability of autoevolve. specifically, the evolution of the queries leads to a hit@6 decline of 21.4% for bge-m3 on booksum-e and 17.3% on multihoprag-e. for multihoprag-e, recall-kp for gpt-4o and qwen2.5-7b-ins decreases by 12.4% and 14.6%, respectively. these findings indicate that autoevolve can dynamically increase the difficulty of queries across different corpora, opening the door for more transferable and scalable rag evaluations. overall, our contributions are as follows: ## conclusion in this paper, we investigate the automatic generation of effective benchmarks for evaluating rag systems on a specific corpus. we observe that existing methods suffer from two issues: weak applicability and weak scalability. to address these issues, we propose autoevolve, an applicable approach for automatically evolving queries to construct scalable rag benchmarks. for weak applicability, we propose a corpus-agnostic method for constructing the universal entity-document relationship graph. for weak scalability, we utilize the automatic evolution framework to dynamically enhance the difficulty of queries based on the graph. through experiments on three benchmarks that autoevolve generates from three different corpora, we demonstrate that evolved queries are more challenging, opening the door for more transferable and scalable rag evaluations.