KU - DMIS at EHRSQL 2024 : Generating SQL query via question templatization in EHR (2024)

Abstract: Transforming natural language questions into SQL queries is crucial for precise data retrieval from electronic health record (EHR) databases. A significant challenge in this process is detecting and rejecting unanswerable questions that request information outside the database’s scope or exceed the system’s capabilities. In this paper, we introduce a novel text-to-SQL framework that focuses on standardizing the structure of questions into a templated format. Our framework begins by fine-tuning GPT-3.5-turbo, a powerful large language model (LLM), with detailed prompts involving the table schemas of the EHR database system. Our approach shows promising results on the EHRSQL-2024 benchmark dataset, part of the ClinicalNLP shared task. Although fine-tuning GPT achieves third place on the development set, it struggled with the diverse questions in the test set. With our framework, we improve our system’s adaptability and achieve fourth position in the official leaderboard of the EHRSQL-2024 challenge.

## introduction electronic health records (ehrs) are crucial elements of the contemporary healthcare system, storing patients' medical histories in relational databases. however, retrieving data from ehrs can be challenging, requiring specialized training in structured query language (sql). to bridge this gap, previous works build ai-powered systems that parse the user's question @xcite or convert it into an sql query that the database can process. @xcite identify an essential component in this text-to-sql task; recognizing and adequately handling unanswerable questions that seek information beyond what the database contains. hence, to ensure reliability and trustworthiness, systems should be able to refrain from answering unanswerable questions. to further encourage research in this field, the clinical nlp 2024 workshop has introduced a new shared task called ehrsql-2024 @xcite to motivate the development of more reliable question-answering (qa) systems. the ehrsql-2024 dataset involves the real-world needs of medical personnel, incorporating templates of their most common questions. in this challenge, systems are tasked to generate sql queries that accurately return the desired information from tables from the mimic-iv @xcite . additionally, the dataset includes inherently unanswerable questions, either due to the restrictions of the database schema or the request for information not contained within the databases. on the other hand, the test set presents distracting question types that contain noisy words, further testing the robustness of participants' systems. in this paper, we introduce a novel framework created to convert natural language questions to sql queries for ehr databases. this framework transforms free-form questions into a templated format to handle distracting questions. we finetune gpt-3.5-turbo @xcite , one of the most performant large language models (llms), optimizing it to effectively interpret intricate medical queries and produce the corresponding sql queries. we also provide detailed prompts that describe the tables in the ehr database. for sql generation, given the task's complexity and relationships between tables, we break it down into two steps: selecting relevant tables and then generating sql by reflecting in-depth on the selected tables. we enhance the accuracy and reliability of the generated sql queries by correcting any errors in table names and applying ensemble techniques with majority voting. our empirical results of fine-tuned gpt-3.5 on the ehrsql-2024 benchmark highlight its capability, achieving third place on the development set. however, it revealed a limitation in generalizing to questions in the test set that diverged from the predefined templates. by using our framework, we successfully address this gap between free-form questions, resulting in a notable improvement of 26.5 in the rs(10) metric in the test set. additionally, we find that decomposing the task into two steps contributed to this success, with a significant improvement in rs (10) in the test set. furthermore, by employing further verification and ensemble techniques, we attained fourth place in the ehrsql-2024 challenge's official leaderboard. we conduct in-depth analyses of the questions to uncover disparities in each split. in particular, we apply n -gram counting of the questions to highlight the distribution gaps. this variation emphasizes the need to develop a resilient model capable of adapting to and performing consistently across datasets with diverse word distributions. additionally, we manually categorize the unanswer-able questions into three distinct types. ## methods figure 2 presents an outline of our proposed methodology. our process starts with the templatization of questions, transforming free-form inquiries into a standardized format to ensure consistency in how queries are represented. additionally, we enrich the model's understanding by supplying detailed information about the database tables, thereby improving its capacity to formulate precise queries. to further elevate the accuracy of the generated sql queries, we introduce a verification phase to confirm that the queries accurately correspond to the intended data retrieval objectives. ## analysis in this section, we analyze the word distribution of questions for each dataset split: training, development, and test sets. the objective is to identify variations in question composition among these datasets. furthermore, we investigate the distribution of unanswerable questions in the training set to better understand questions that yield an null response. to focus solely on word analysis and minimize noise, we eliminate punctuation marks such as ".", ",", and "?", remove stop words such as "the", "a", and "an" from the questions, and convert all letters to lowercase. after eliminating these elements, we analyze the processed questions using n-grams. the analysis is limited to 1 to 3-grams, which is sufficient for understanding the context of questions while excluding the aforementioned noise. appendix d details the ten most prevalent words alongside their respective frequencies within each dataset arranged in non-increasing order, including the collection of unanswerable queries labeled as unanswerable train set. ## conclusion throughout the challenge, we noticed that differences in the way data is distributed across training, development, and test sets can make it hard for our model to determine which questions are answerable or not. to tackle this issue, we templatized questions to make the word distribution of development and test data more similar to the training data. this method aimed to bridge the gap between the datasets, helping the model better understand the features of unanswerable questions within the test dataset. although we did not address this in this study, we anticipate that future research could see performance improvements by augmenting the training dataset to more closely match the distribution of unanswerable questions in the development and test sets. focusing on refining the test data to align more closely with the characteristics observed in the training datasets, we expect to increase model performance in identifying unanswerable questions. such data augmentation strategies could bridge the remaining gaps between datasets and ensure a more robust model performance across varied datasets. also, we utilized finetuned gpt-3.5-turbo, which is expensive and unusable for other researchers. thus, further study should be done with open sourced models, like llama or gemma. appendix d word frequencies: 3-gram