Overview of the EHRSQL 2024 Shared Task on Reliable Text-to- SQL Modeling on Electronic Health Records (2024)

Abstract: Electronic Health Records (EHRs) are relational databases that store the entire medical histories of patients within hospitals. They record numerous aspects of patients’ medical care, from hospital admission and diagnosis to treatment and discharge. While EHRs are vital sources of clinical data, exploring them beyond a predefined set of queries requires skills in query languages like SQL. To make information retrieval more accessible, one strategy is to build a question-answering system, possibly leveraging text-to-SQL models that can automatically translate natural language questions into corresponding SQL queries and use these queries to retrieve the answers. The EHRSQL 2024 shared task aims to advance and promote research in developing a question-answering system for EHRs using text-to-SQL modeling, capable of reliably providing requested answers to various healthcare professionals to improve their clinical work processes and satisfy their needs. Among more than 100 participants who applied to the shared task, eight teams completed the entire shared task processes and demonstrated a wide range of methods to effectively solve this task. In this paper, we describe the task of reliable text-to-SQL modeling, the dataset, and the methods and results of the participants. We hope this shared task will spur further research and insights into developing reliable question-answering systems for EHRs.

## introduction electronic health records (ehrs) store all types of medical events that occur in the hospital, including hospital admissions, diagnoses, procedures, prescriptions, and discharges. they replace traditional paper-based records and provide a centralized repository for patient data. over the years, the widespread adoption of ehrs in hospitals has been shown to improve patient care, increase efficiency, and enhance coordination among healthcare professionals @xcite @xcite . although ehrs are a valuable source of patient data, the complexity of their data structures and the need for specialized skills, such as query languages like sql, to extract and analyze the information, often hinder their effective utilization by healthcare professionals @xcite . these barriers lead to the underutilization of the full potential of ehrs in clinical practice and research. an alternative way to utilize data stored in ehrs is to develop a question-answering (qa) system. qa systems provide a user-friendly interface that allows healthcare professionals to ask questions in natural language and receive relevant answers from the ehr data, without needing to know query languages or ehr database strctures. specifically, text-to-sql modeling is an effective approach for building qa systems for ehrs, which are typically relational databases. these models automatically convert natural language questions into their corresponding sql queries, and then execute these queries on the database to obtain the final answer. with the impressive advances in large language models (llms), various high-performance textto-sql models have been introduced, which are accomplished through model fine-tuning @xcite or llm prompting with demonstrations @xcite @xcite . if deployed with reliable performance, these models could significantly benefit healthcare professionals by allowing them to explore patient data more freely from the ehrs through natural language interactions. several datasets on question-answering for ehrs have been introduced, including mimic-sql @xcite , emrkbqa @xcite , and ehrsql @xcite . ehrsql, in particular, poses unique challenges. it is the first dataset to compile a collection of questions that reflect the diverse needs of healthcare professionals, including physicians, nurses, and what medication was last prescribed to patient 10027602? tell me the length of the last hospital stay of patient 10027602. what are the side effects of long term insulin administration? reliable text-to-sql model select drug from prescriptions where @xmath0; select strftime('%j', admissions.dischtime)strftime('%j', admissions.admittime) from admissions where admissions.@xmath1.dischtime is not null order by admissions.admittime desc limit 1; not answerable readi-cat 2 (barium sulfate 2% suspension) 21.1 days i am not able to answer based on the ehr natural language question answer sql execution ehr figure 1: overview of reliable text-to-sql modeling on ehrs. for any input questions, a reliable text-to-sql model should accurately predict sql queries for what it can and abstain from what it cannot, such as for intrinsically unanswerable questions or ones that are likely incorrect by the model. successfully developing such a model can serve as a valuable tool for healthcare professionals in hospitals for better accessibility of patient data and assistance of clinical decision-making. in this paper, we present the ehrsql 2024 shared task and its dataset in sections 2 and 3, respectively. section 4 introduces the evaluation metric and baseline model for the task. section 5 describes methods proposed by the participating teams and discusses interesting findings from the official results. the goal of the ehrsql 2024 shared task is to develop a reliable qa system for ehrs, specifically through text-to-sql modeling. reliability is crucial for the deployment of ai systems, especially in critical domains like hospitals, where incorrect predictions can have severe consequences. the term reliability in question answering refers to the system's preference for abstention over providing an incorrect answer @xcite @xcite . in this shared task, we adopt the definition of reliable text-to-sql from trustsql @xcite , which first expands the scope of reliability to include unanswerable questions. a reliable text-to-sql model should not only correctly generate sql queries, providing utility, but also abstain from answering questions that are likely to be incorrect or are unanswerable, thereby minimizing harm. this objective contrasts with most other text-to-sql tasks, where the primary focus is to maximize sql generation performance for answerable questions only. further discussion on specific scenarios of measuring reliability for text-to-sql is explained in section 4.1. ## dataset construction in this section, we outline the key steps to generate data for the shared task. the overall data construction pipeline is illustrated in figure 2 . each subsection provides a detailed explanation of each step. ## conclusion with the increasing volume of data stored in ehrs and the impressive advances in llms, the ehrsql 2024 shared task offered an opportunity to develop and test participants' creative methods to building reliable qa systems on ehrs using text-to-sql modeling. the dataset for this shared task presents unique challenges, including questions that extensively use time expressions and the increased complexity of sql queries, which more accurately reflect the real needs of a hospital setting. it also includes challenging unanswerable questions that should be avoided. this distinguishes the task from most other text-to-sql challenges, as reliable text-to-sql models must not only generate correct sql queries, providing utility, but also abstain from answering questions that are likely incorrect or unanswerable, thereby minimizing harm. the shared task attracted over 100 participants from academia and industry, with 8 teams ultimately submitting their code and fact sheets. as a novel task at the intersection of the nlp and clinical domains, it inspired a variety of proposed methods. these included self-training llms through pseudo-labeling, ensembling of different llms, generating synthetic question-sql pairs to handle distribution shifts from training to test sets, leveraging log-probabilities for abstention, and pipelinebased approaches with specialized models for correct sql generation and abstention. we hope that this shared task, emphasizing reliability, will encourage further research into building qa systems for ehrs that can truly serve as valuable tools for healthcare professionals in hospitals, improving clinical decision-making, facilitating research, and enhancing patient care quality. future research directions include expanding reliable question answering for ehrs to multimodal settings by incorporating clinical notes, x-ray images, and ecg signals.