P rob G ate at EHRSQL 2024: Enhancing SQL Query Generation Accuracy through Probabilistic Threshold Filtering and Error Handling (2024)

Abstract: Recently, deep learning-based language models have significantly enhanced text-to-SQL tasks, with promising applications in retrieving patient records within the medical domain. One notable challenge in such applications is discerning unanswerable queries. Through fine-tuning model, we demonstrate the feasibility of converting medical record inquiries into SQL queries. Additionally, we introduce an entropy-based method to identify and filter out unanswerable results. We further enhance result quality by filtering low-confidence SQL through log probability-based distribution, while grammatical and schema errors are mitigated by executing queries on the actual database.We experimentally verified that our method can filter unanswerable questions, which can be widely utilized even when the parameters of the model are not accessible, and that it can be effectively utilized in practice.

## introduction in recent years, the field of natural language processing (nlp) has witnessed remarkable progress driven by transformer-based large language models (llms) @xcite @xcite . a prevailing approach involves fine-tuning pre-trained language models with new data across various tasks, facilitating transfer learning @xcite . this methodology has proven effective in tasks like document summarization, entity-relationship extraction, document classification, and sentiment analysis. one of the main tasks where these language models are increasingly leveraged is text-to-sql (text2sql), which converts natural language queries into sql queries @xcite . text2sql presents unique challenges distinct from conventional nlp tasks. firstly, it demands grammatical correctness, as even minor errors can render sql queries unexecutable. unlike document summarization, where semantic correctness compensates for grammatical inaccuracies, sql queries must adhere strictly to syntax rules @xcite . secondly, schema awareness is crucial; understanding the database structure is essential for generating accurate sql queries @xcite . finally, discerning unanswerable queries is vital, especially in domains like healthcare where incorrect or incomplete information can have severe consequences @xcite . if users do not inspect the sql queries themselves, but only receive the results of the execution, the results of an incorrect sql execution can be fatally misleading. in the domain of text2sql, effectively filtering out unanswerable questions presents a significant challenge @xcite , particularly within the medical field where accuracy is paramount. existing methodologies for identifying unanswered queries have primarily targeted cases where such queries exhibit discernible patterns @xcite . however, these methods are often tailored to specific model architectures and learning methods, thereby constraining their direct applicability to llm services accessible via apis, such as chat-gpt. given the recent widespread adoption of such managed llms across various industries, the need for a more versatile and adaptable approach to filtering unanswered questions becomes increasingly pronounced. this underscores the necessity for innovative solutions that can seamlessly integrate into existing llm services, ensuring robust performance in diverse application scenarios, including medical contexts. we address solutions that effectively solve the challenges of text2sql tasks through a subset focusing on electronic health records(ehr), utilizing medical questions and corresponding sql queries relevant to medical systems used in real hospitals @xcite . specifically, we participate in the ehrsql shared task on reliable text-to-sql modeling on electronic health records @xcite . a distinctive feature of this shared task is that under the basic premise of generating appropriate sql statements for given natural language queries, not all questions are answerable; some are unanswerable. moreover, beyond merely generating suitable sql statements for questions, this task is complex as it requires distinguishing between answerable and unanswerable questions and considering the high penalties for incorrectly identifying the questions are answerable or not, thus necessitating both reliability and accuracy in execution. in this paper, we introduce probability gate (probgate), a novel probability-based filtering approach designed for seamless integration with diverse generative language models, without requiring direct access to the model's parameters. figure 1 illustrates the concept of probgate, which leverages the logarithmic probability of individual tokens to assess the uncertainty associated with generated sql queries. we consider the log probability of specific target tokens as an indicator of how confident the model is and how well it can perform the task without hallucinations. we found that utilizing logarithmic probability-based confidence to identify answerable and unanswerable questions was very effective, which is a key aspect of this task. we evaluate the efficacy of probgate through experimentation with electronic health record (ehr) sql dataset @xcite . specifically, we apply probgate to both t5-based @xcite text2sql models and gpt-3.5-turbo finetuned models, comparing their performance against conventional binary classifiers. additionally, we train binary classifiers based on t5 and gpt-3.5-turbo model to filter out unaswerable questions. our experimental findings reveal that prob-gate outperforms binary classifiers in terms of both performance and resilience to shifts in data distribution. these results underscore the potential of probgate as a versatile and robust filtering solution for a wide range of applications. our contributions and methods can be summarized as follows: ## backgrounds text2sql databases serve as powerful tools for efficiently querying extensive datasets. however, accessing this data often requires users to possess knowledge of query languages like sql. to democratize this process and render it accessible across proficiency levels, significant research efforts have focused on techniques for interpreting natural language questions and autonomously translating them into sql queries. recent strides in deep learning methodologies, particularly transformerbased language models, have spurred the development of text-to-sql techniques. these approaches aim to bridge the gap between natural language queries and sql commands, thereby enhancing accessibility and usability in database querying tasks @xcite . early text2sql research relied on rule-based and template-based methods, but more recently, deep learning-based methodologies have become mainstream @xcite . deep learning methodologies exhibit robustness on the data they are trained on but often struggle to generalize to unseen database schemas. to mitigate this challenge, researchers have explored approaches to encode database relationships and leverage column relationships using self-attention mechanisms @xcite . in text2sql, ensuring the accuracy of generated sql statements is crucial as even minor errors can lead to failures in query execution. recent studies have demonstrated the effectiveness of utilizing llms like gpt-3.5-turbo to rectify sql statements derived from natural language queries, addressing the challenge of proofreading sql output @xcite . one of the main applications of text2sql is its utilization in the healthcare domain, specifically to handle complex tasks within electronic health records @xcite . recent research has shown that decomposing these tasks into manageable pieces can improve the performance of multi-table reasoning within ehrs. the authors proposed to iteratively improve sql queries by incorporating interactive coding and execution feedback mechanisms to learn from the error messages encountered. this iterative improvement process proved to be effective and resulted in noticeable improvements in sql performance in the healthcare domain @xcite . in a closely related investigation, researchers observed that ehr data is commonly stored in relational databases, which can be represented as directed acyclic graphs. leveraging this insight, they employed a graph-based methodology to capture the intricate relationships between tables, entities, and values within relational databases @xcite . ## methods from this section, we cover the contents related to the methods. in §3.1, there is a detailed description of the shared task dataset; in §3.2, the main metrics used in the shared task are discussed; and from §3.3 to §3.5, detailed information on the main methods is provided. the entire architecture can be referenced in figure 2 . ## conclusion we participate in the ehrsql shared task on reliable text-to-sql modeling on electronic health records, as detailed in @xcite , aiming to develop a reliable and high-performance text2sql method. this encompasses the chal-lenge of generating appropriate sql for answerable questions while also distinguishing unanswerable questions within datasets that include them. to solve this, we fine-tune llms on the training dataset and then employ a filtering pipeline called probgate, which consists of a combination of probabilistic threshold filtering and grammatical errors filtering, effectively executing the task. additionally, through an ablation study and detailed analysis, we demonstrate that our method can be effectively used for tasks with a high sensitivity to errors. ultimately, using this method, we conclude the shared task with a team ranking of 3rd place.