R^3 : “This is My SQL , Are You With Me?” A Consensus-Based Multi-Agent System for Text-to- SQL Tasks (2025)

Abstract: Large Language Models (LLMs) have demon- strated exceptional performance across diverse tasks. To harness their capabilities for Text- to-SQL, we introduce R3 (Review-Rebuttal- Revision), a consensus-based multi-agent sys- tem for Text-to-SQL tasks. R3 achieves the new state-of-the-art performance of 89.9 on the Spider test set. In the meantime, R3 achieves 61.80 on the Bird development set. R3 out- performs existing single-LLM and multi-agent Text-to-SQL systems by 1.3% to 8.1% on Spi- der and Bird, respectively. Surprisingly, we find that for Llama-3-8B, R3 outperforms chain-of- thought prompting by over 20%, even outper- forming GPT-3.5 on the Spider development set. We open-source our codebase at https: //github.com/1ring2rta/R3.

## introduction text-to-sql, the task of converting natural language to sql queries, enables non-technical users to access databases with natural language @xcite . recently, large language models (llms) have made significant progress on various tasks @xcite . although various methods were proposed to enhance the reasoning abilities of llms @xcite @xcite , they are still facing challenges with text-to-sql tasks @xcite . the llmbased multi-agent system leverages collective intelligence from a group of llms and has achieved exceptional performance across various tasks @xcite @xcite , but little work explores using them on text-to-sql. the existing multi-agent text-to-sql system first decomposes the task into multiple subtasks, which are then accomplished step-by-step in a pipeline by agents @xcite . while achieving remarkable performance, such decomposition-based systems necessitate extensive prompt engineering and logic design. we propose r 3 , a consensus-based multi-agent system for text-to-sql tasks that draws inspiration from the peer-review mechanism. in our designed framework, the llm does not need to be divided into sub-tasks such as column selection, schema linking, and so on. instead, it is split into an sql writer and multiple reviewers who provide feedback based on the execution results. once the generated sql query is confirmed to be executable, the system enters a review process, where the execution results guide the sql writer and reviewers to refine the sql. through rounds of "review," "negotiation or rebuttal," and "revision," the sql writer and reviewers ultimately reach a consensus and deliver a solution with collective agreement (see figure 1 ). we test r 3 on the popular spider and bird benchmarks. r 3 outperforms the existing single llm as well as the multi-agent tex-to-sql systems by 1.3% to 8.1% on spider and bird, and set new stateof-the-art (sota) performance of 89.9 on spider dataset. surprisingly, we find that for llama-3-8b, r 3 outperforms chain-of-thought prompting by over 20%, even outperforming gpt-3.5 on the spider-dev set. in summary, our contributions are several-fold: in the meantime, r 3 achieves 61.80 on the bird-dev dataset. in addition, r 3 effectively helps open-source llms such as llama-3-8b on sql generation. when using llama-3-8b as the backbone model, r 3 outperforms direct cot prompting llama-3-8b by 20%, and outperforms gpt-3.5 on the spider-dev set. 3. we provide a detailed error analysis of r 3 on the existing text-to-sql benchmarks, shedding light on future research on the text-to-sql task. llms for text-to-sql. recent years have witnessed llms' breakthroughs in many fields @xcite @xcite . moreover, @xcite ; @xcite ; @xcite have observed that these llms can learn in context with a few examples during their inference time. the strong reasoning and in-context learning capabilities of these llms have brought a paradigm shift to the textto-sql community, which now focuses on leveraging llms' ability to handle text-to-sql tasks. for instance, @xcite propose din-sql to few-shot prompt gpt-4; @xcite introduce c3, which zero-shots gpt-3.5 with hints and checks output consistency; dail-sql @xcite comprehensively evaluates the efficiency and effectiveness of various prompting techniques. output consistency. recent works have applied the consistency principle @xcite to enhance the reasoning ability of llms through incontext learning, such as chain-of-thought (cot) @xcite or tree-of-thoughts (tot) @xcite . in addition, chen et al. (2023) adopt program-of-thoughts (pot), which uses python code to assist llms in the reasoning process and surpasses cot on math reasoning. 3 r 3 architecture sql-writer we task sql-writer (sw) agents to: (1) compose the original sql query based on the user question and database schema; (2) ensure that the sql query is executable, and correct it when errors occur; (3) respond to reviewer agents' feedback and revise the sql query accordingly. specifically, we prompt sw agent through prompt 1 in appendix a.6. for task (1), we feed the prompt 1 to sw agent directly. given a user question x and the database schema s, task (1) can be formalized as: we then concatenate y ′ to the history, resulting in the update h ← h + y ′ . furthermore, in consideration of the context window limitations of llms, we truncate the dialogue history h when the length of the prompt exceeds the model's context limit. reviewers. we generate the reviewer agent's (res) professions using an llm (see prompt 3 in appendix a.6) based on the database schema and the content of the sql query, for instance, "senior database engineer specialized in writing various clauses" and "data analyst in the automotive industry", etc. we incorporate these professions in the system prompt for the res to make them focus on different aspects of the sql query. these res are prompted to provide their professional comments based on the database schema, the user's question, the predicted sql, and its execution result in the table format. overall architecture. after several rounds of "negotiation" between the sql-writer and res, we decide whether there is a consensus by checking if the sql-writer agent generates the same sql query as in the previous round. when there is a consensus, we terminate the negotiation loop and output the final sql query. algorithm 1 depicts the overall process of our system. appendix a.6 provides the detailed prompts we use in r 3 . in addition, we incorporate: ## experimental setup spider-dev spider-test bird-dev @xcite ( datasets. we conduct experiments on two crossdomain text-to-sql benchmarks, spider and bird, detailed in baselines. we conduct our experiments based on llms including gpt-3.5-turbo, gpt-4 (openai, 2023) and llama-3 (ai@meta, 2024). as for the compared methods, the raw performance for gpt-3.5 ("-") was evaluated by @xcite ; c3 employs schema linking filtering @xcite ; dail selects few-shot demonstrations based on their skeleton similarities @xcite , and "sc" represents self-consistency @xcite ; pet uses cross-consistency @xcite ; din decomposes the text-to-sql task into smaller subtasks @xcite ; mac, as previously mentioned, is the first to apply a multi-agent system to text-to-sql tasks @xcite . metrics. we employ test-suite execution evaluation foot_0 @xcite , the standard evaluation protocol for spider, and the official sql execution accuracy evaluation for bird foot_1 . ## conclusion in this paper, we propose r 3 , a consensus-based multi-agent system for text-to-sql generation. r 3 sets the new sota performance on spider (89.9) and achieves 61.80 on the bird dev set. in addition, we find that r 3 significantly enhances open-source llms such as llama-3-8b (over 20% improvement on spider dev set). last but not least, we conduct a comprehensive error analysis and identify issues with the current text-to-sql evaluation, underscoring the necessity for a more refined evaluation protocol, as the llms and llm-based methods become more powerful than ever.