Graph-Reward- SQL : Execution-Free Reinforcement Learning for Text-to- SQL via Graph Matching and Stepwise Reward (2025)

Abstract: Reinforcement learning (RL) has been widely adopted to enhance the performance of large language models (LLMs) on Text-to-SQL tasks. However, existing methods often rely on execution-based or LLM-based Bradley–Terry reward models. The former suffers from high execution latency caused by repeated database calls, whereas the latter imposes substantial GPU memory overhead, both of which significantly hinder the efficiency and scalability of RL pipelines. To this end, we propose a novel reward model framework for RL-based Text-to-SQL named Graph-Reward-SQL, which employs the GMNScore outcome reward model. We leverage SQL graph representations to provide accurate reward signals while significantly reducing time cost and GPU memory usage. Building on this foundation, we further introduce StepRTM, a stepwise reward model that provides intermediate supervision over Common Table Expression (CTE) subqueries. This encourages both functional correctness and readability of SQL. Extensive comparative and ablation experiments on standard benchmarks, including Spider and BIRD, demonstrate that our method consistently outperforms existing reward models.

## introduction text-to-sql @xcite @xcite aims to translate natural language into structured database queries and plays a crucial role in democratizing data access by enabling nontechnical users to interact with relational databases more effectively. a significant body of work has focused on fine-tuning foundational models, with recent studies showing that reinforcement learning (rl) can effectively enhance model performance @xcite @xcite . among these efforts, the careful * work was done during the internship at bytedance. ## methodology we introduce graph-reward-sql, a novel reward model framework designed to enhance sql generation through two key innovations. first, we propose gmnscore, which replaces ex, reducing time costs while maintaining the accuracy of reward signals without requiring database execution. second, we introduce steprtm, a stepwise reward model based on the relational operator tree (rot) representation of cte sql, which provides sup-plementary intermediate feedback. ## results we benchmark our method against recent sota text-to-sql systems, including in-context learning and multi-agent approaches. as shown in ## discussion the gmnscore introduced in this paper offers an alternative to ex while remaining fully compatible with other reward models. as detailed in appendix i, we extend our investigation beyond the steprtm integration (section 6.1) by applying hybrid outcome reward models, which further improve performance. this finding is consistent with previous work using multiple outcome rewards @xcite . ## conclusion we propose graph-reward-sql, a reward model framework for rl-based text-to-sql that replaces execution-based rewards with the gmnscore outcome reward model and the steprtm stepwise reward model. by eliminating the dependency on execution during training, it significantly improves training efficiency. extensive experiments demonstrate that our proposed reward models achieve superior performance. the graph-reward-sql framework establishes a new direction toward scalable and efficient rl-based text-to-sql tasks.