P rompt M ind Team at EHRSQL -2024: Improving Reliability of SQL Generation using Ensemble LLM s (2024)

Abstract: This paper presents our approach to the EHRSQL-2024 shared task, which aims to develop a reliable Text-to-SQL system for electronic health records. We propose two approaches that leverage large language models (LLMs) for prompting and fine-tuning to generate EHRSQL queries. In both techniques, we concentrate on bridging the gap between the real-world knowledge on which LLMs are trained and the domain-specific knowledge required for the task. The paper provides the results of each approach individually, demonstrating that they achieve high execution accuracy. Additionally, we show that an ensemble approach further enhances generation reliability by reducing errors. This approach secured us 2nd place in the shared task competition. The methodologies outlined in this paper are designed to be transferable to domain-specific Text-to-SQL problems that emphasize both accuracy and reliability.

## introduction text-to-sql technology translates natural language questions into executable sql queries that can answer the questions using a provided database. a robust text-to-sql system could significantly increase productivity for anyone using databases by providing an easy-to-use natural language interface and reducing the need for expertise in different sql dialects. these systems are particularly more valuable in domains where sql knowledge is not essential, such as healthcare, where healthcare professionals like doctors, nurses, and hospital administrators spend a significant amount of time interacting with patient health records stored in databases. in the era of large language models (llms), the field of text-to-sql is gaining prominence as these models demonstrate impressive text generation capabilities without the need for fine-tuning. introduced in 2017, wikisql @xcite remains one of the largest datasets for text-to-sql and primarily caters to relatively simple queries. subsequently, the spider @xcite and multi-spider @xcite datasets were developed. these datasets posed challenges with complex queries that required an understanding of the database schema and support for various languages. bird-bench was introduced to bridge the gap between research and real-world applications by providing large and imperfect databases @xcite . these datasets are good representations of typical text-to-sql tasks. however, the healthcare domain differs from these generic datasets for the following reasons: these differences present unique challenges for developing a reliable text-to-sql system for the healthcare domain. ehrsql is the first dataset that closely captures the needs of hospital staff and serves appropriately for building and testing text-to-sql systems in the healthcare domain @xcite . our solution aims to create a text to sql system that emphasizes both reliability and accuracy. to achieve this, we divide the task into two phases: ## related work prior to the advent of llms, the primary focus of research in natural language processing involved refining specialized models using innovative strategies @xcite . additionally, substantial efforts were devoted to developing sophisticated pre-training methodologies, such as those proposed by star @xcite , and exploring decoding strategies, as exemplified by picard @xcite . however, these approaches typically require substantial computational resources and novel techniques. large language models (llms) have been trained extensively on textual data, which has equipped them with vast knowledge. as a result, they exhibit exceptional probabilistic reasoning abilities and can excel at various tasks even without explicit training. zero-shot prompting techniques, when used with llms, have not only narrowed the performance gap on text-to-sql but have also surpassed specialized pre-trained or fine-tuned models. several prompt techniques have been developed based on this zero-shot approach for textto-sql tasks, leading to remarkable achievements on datasets such as spider @xcite , @xcite . zero-shot generation capabilities can be further enhanced through techniques like in-context learning (icl) and few-shot prompting. din-sql @xcite adopts an in-context learning approach to break down complex sql generation into manageable subtasks, leading to improved performance on intricate queries. another technique, retrieval-augmented generation, provides relevant and helpful examples as a few-shot to guide sql generation @xcite . these approaches have proven effective on general text-to-sql tasks but they have not yet been studied rigorously on domain-specific text-to-sql problems. retrieval augmented finetuning (raft) introduces a novel fine-tuning technique that improves the in-domain performance of rag while integrating domain-specific knowledge @xcite . through our work, we delve into the application of these techniques for the ehrsql-2024 task. ## shared task and dataset the ehrsql-2024 shared task @xcite is aimed at creating a reliable sql for answering questions posed in natural language on the mimic-iv demo database. the mimic-iv database consists of anonymized electronic health records of patients admitted to the beth israel deaconess medical center. these records primarily cover two modules: hospital records and icu records. the publicly available demo version of the database contains a subset of patient records for 100 individuals. it consists of 17 tables from both modules, encompassing a total of 109 columns. ## approach the reliable text-to-sql solution is decomposed into two stages as follows. ## results in this section, we present the comparison of the reliability scores of the individual models followed by ensembles. ## ablation study to assess the significance of each parameter in the final prompt employed for icl and fine-tuning, we conduct an ablation study. in these experiments, we focus solely on pre-trained models because finetuning experiments are more expensive and timeconsuming. to accelerate the process and maintain costs, we leverage gpt-3.5, a compact and less potent yet faster variant of the gpt family. through these experiments, we extrapolate the efficacy of each parameter for prompting using more robust and advanced models such as gpt-4 and claude-3 opus. ## conclusion our work primarily aims to enhance the reliability of sql generation, which is of paramount importance for the ehrsql-2024 shared task. although in-context learning with advanced llms such as gpt-4, claude-3 opus, or fine-tuning gpt-3.5 yields excellent rs0, errors still seem inevitable. the model's ability to solve a specific task is heavily influenced by the training data. repeatedly generating using the same prompt (or) the same model to validate often fails to minimize errors since hallucinations mainly originate from the training data. fine-tuning gpt-3.5 resulted in different error tendencies compared to pre-trained models, even when using the same prompt. therefore, ensemble llms, particularly those with a fine-tuned model, offer a superior approach for sql validation, improving robustness and reliability. this approach has also secured us 2nd place in the competition.