FLEX : Expert-level False-Less EX ecution Metric for Text-to- SQL Benchmark (2025)

Abstract: Text-to-SQL systems have become crucial for translating natural language into SQL queries in various industries, enabling non-technical users to perform complex data operations. The need for accurate evaluation methods has increased as these systems have grown more sophisticated. However, the Execution Accuracy (EX), the most prevalent evaluation metric, still shows many false positives and negatives. Thus, this paper introduces **FLEX(False-Less EXecution)**, a novel approach to evaluating text-to-SQL systems using large language models (LLMs) to emulate human expert-level evaluation of SQL queries. Our metric improves agreement with human experts (from 62 to 87.04 in Cohen’s kappa) with comprehensive context and sophisticated criteria. Our extensive experiments yield several key insights: (1) Models’ performance increases by over 2.6 points on average, substantially affecting rankings on Spider and BIRD benchmarks; (2) The underestimation of models in EX primarily stems from annotation quality issues; and (3) Model performance on particularly challenging questions tends to be overestimated. This work contributes to a more accurate and nuanced evaluation of text-to-SQL systems, potentially reshaping our understanding of state-of-the-art performance in this field.

## introduction text-to-sql systems, which translate natural language questions into sql queries, have become vital across various industries by democratizing data access and facilitating data-driven decisionmaking @xcite . as these systems grow more complex, accurate and efficient evaluation methods are increasingly important. while initial evaluations relied on human experts, this approach proved too time-consuming and costly for large-scale assessments. this, benchmarks like spider @xcite introduced the execution accuracy (ex) metric, which has been widely adopted by others such as bird @xcite . however, the ex metric, which evaluates queries based on their execution result, has significant limitations that can lead to assessments diverging from human experts. our analysis reveals that ex incorrectly validates queries due to coincidental database states or unfairly penalizes correct ones because of ambiguous questions, thus misestimating a model's ability to generate correct and valid sql queries @xcite . although llm-based evaluation methods @xcite @xcite have been proposed, they exhibit suboptimal performance due to limited context. they fail to utilize crucial contextual information, struggle with noisy ground truth issues, and employ ambiguous criteria inappropriate for text-to-sql evaluation. consequently, they even underperform the traditional ex metric when evaluated against established text-to-sql benchmarks. to address these issues, we propose flex (false-less execution), a novel approach leveraging llms to emulate expert-level evaluation of sql queries. flex is specifically designed for evaluating text-to-sql tasks and overcomes existing limitations through comprehensive context analysis, considering the natural language question, database schema, and external knowledge. it also exploits elaborately devised evaluation criteria with detailed guidelines for assessing query correctness. our approach is validated through strong consistency between flex judgments and human expert evaluations, demonstrating significantly higher agreement (cohen's kappa of 87.04) than the existing ex metric (62.00) and outperforming previous llm-based methods. to demonstrate the practical impact of flex, we re-evaluated 50 publicly available text-to-sql models on the spider and bird foot_0 benchmarks, revealing several key insights. as shown in our key contributions are as follows: 1) we identify key limitations in current text-to-sql evaluation metrics, especially the execution accuracy (ex) metric, and 2) we introduce flex, a novel llm-based evaluation method that aligns with human expert reasoning and shows better agreement with expert assessments. 3) by re-evaluating 50 text-to-sql models using flex, we uncover significant ranking changes and provide a clearer understanding of benchmarks and state-of-the-art models, emphasizing the need for advanced evaluation methods. flex framework is publicly available foot_2 . llm-based evaluation has gained popularity as an alternative to human evaluators for assessing human preference in natural language generation models. frameworks like mt-bench @xcite and alpacaeval @xcite utilize llms as judges to provide faster and more cost-effective evaluations than human assessments. however, these frameworks primarily rely on the proprietary @xcite , which poses reproducibility risks since providers may alter or discontinue models without notice. in response, @xcite introduced the open-source llm judge prometheus, offering a reproducible and cost-free alternative at gpt-4's performance level. nonetheless, prometheus is mainly focused on general human preference evaluations rather than specialized tasks like text-to-sql evaluation. to specifically leverage llms for text-to-sql evaluation, zhao et al. proposed llm-sql-solver. without distinguishing between the false positives and false negatives, this approach showcases the capability of llms to assess execution equivalence between two queries using two prompting strategies: miniature-and-mull, evaluating seman-tic equivalence between two queries, and explainand-compare, asking llms to compare their significant logical differences. despite its advancements, llm-sql-solver remains insufficient to fully replace ex as an evaluation metric. we will investigate these detailed limitations in section 4.3. ## preliminaries this section introduces the notation and definitions used throughout the paper. we begin by establishing the following conventions: let d denote a database, s its schema, and x a set of natural language questions. for each question @xmath0, we define contextual information c to assist in the evaluation of metrics, including q gt (x) as the ground truth sql query for question x, and q gen (x) as the sql query generated by a text-to-sql model for x. we then execute these queries on the database d to obtain their execution result sets. specifically, we define r gt (x) = execute(q gt (x), d) as the result of the ground truth query and r gen (x) = execute(q gen (x), d) as the result of the generated query. we define evaluation metrics: em, ex, and llm-based evaluation (lx) as: for em, the condition checks if q gen (x) exactly matches the q gt (x) syntactically. for ex, the condition checks if the r gen (x) exactly matches the r gt (x). however, lx handles semantic match between q gen (x) and question x under contextual information. c may vary depending on the evaluation method. these metrics produce a score between 0 and 1, where 1 indicates perfect performance (100% accuracy) and 0 indicates complete failure (0% accuracy). ## analyzing the limitations of current text-to-sql evaluation methods in this section, we address key research questions to investigate the shortcomings of existing text-to-sql evaluation metrics. we aim to understand the types of errors these metrics produce and explore whether llms can offer a better alternative. ## experiments to validate the effectiveness of our flex approach, we conducted a comprehensive study comparing the agreement between llm judgments and human evaluations. we tested various state-of-theart language models to determine which aligns most closely with human judgment. ## leaderboard re-evaluation in this section, we evaluated the published result in the spider and bird benchmark using robust capabilities of the flex. three key findings were observed to demonstrate crucial limitations of current text-to-sql evaluation. we first introduce experimental settings and demonstrate three crucial ## conclusion in this paper, we found the limitations in text-to-sql evaluation, the vulnerability of the ex metric to false positives and negatives, which can result in inaccurate model assessments. although llmbased evaluations have emerged as alternatives to human evaluators, they still reveal several limitations. to overcome this, we introduce a novel metric, flex (false-less execution). it utilizes the comprehensive context and elaborate criteria based on the advanced language understanding of llms to achieve more expert-level evaluations. our comprehensive experiments show the effectiveness of our proposed metric and provide several empirical findings to promote future studies on text-to-sql evaluation.