LTRC - IIITH at EHRSQL 2024: Enhancing Reliability of Text-to- SQL Systems through Abstention and Confidence Thresholding (2024)

Abstract: In this paper, we present our work in the EHRSQL 2024 shared task which tackles reliable text-to-SQL modeling on Electronic Health Records. Our proposed system tackles the task with three modules - abstention module, text-to-SQL generation module, and reliability module. The abstention module identifies whether the question is answerable given the database schema. If the question is answerable, the text-to-SQL generation module generates the SQL query and associated confidence score. The reliability module has two key components - confidence score thresholding, which rejects generations with confidence below a pre-defined level, and error filtering, which identifies and excludes SQL queries that result in execution errors. In the official leaderboard for the task, our system ranks 6th. We have also made the source code public.

## introduction electronic health records (ehrs) have revolutionized healthcare by serving as comprehensive digital repositories of medical histories of patients. they capture every step, from initial admission and diagnosis to treatment plans and discharge summaries. while ehrs are invaluable for clinical data storage and retrieval, unlocking their full potential goes beyond basic searches. traditional methods often necessitate proficiency in structured query language (sql), a complex hurdle for many healthcare providers, especially those pressed for time. to bridge this gap and make ehr data more accessible, researchers are exploring the development of question-answering systems that leverage text-to-sql models. these systems empower users to ask questions in plain natural language and receive answers directly retrieved from the ehr data, streamlining the process of extracting valuable insights from patient data. in this task @xcite , we tackle the problem of developing a reliable text-to-sql model tailored for an ehr database, ensuring accurate responses while abstaining from providing incorrect answers. this model must handle a diverse range of topics relevant to clinical settings, such as patient demographics, vital signs, and disease survival rates. the model should accurately generate sql queries for answerable questions, abstain from providing erroneous answers, and recognize and abstain from addressing unanswerable questions, whether they extend beyond the database schema or are impossible to solve using sql alone. the spectrum of unanswerable questions also encompasses adversarially crafted queries designed to mislead text-to-sql models. successfully tackling this task will yield a robust and scalable questionanswering system for ehrs, significantly enhancing how clinicians leverage clinical knowledge. ## related work with the advent of deep learning models, there has been a renewed interest in generating text-to-sql models in the medical domain. @xcite tackle the problem by developing a deep learningbased approach that adapts a sequence-to-sequence architecture to directly generate sql queries for a given question. the model further performs the necessary edits using an attentive copying mechanism and task-specific lookup tables. additionally, they release a large-scale dataset called mimic-sql that generates sql queries from questions in this domain. several papers utilize pre-trained bert @xcite ) models, as their foundation blocks for their text-to-sql systems. these models leverage large amounts of text data to learn effective representations of language in their pre-training stage, which are then fine-tuned for the specific task of ehr-based question answering. @xcite reliability module abstention module text-to-sql generation module sql query null confidence score thresholding error filtering answerable unanswerable null sql query, confidence score confidence less than threshold execution error question figure 1: system workflow develop a bert-based model to convert medical text into an intermediate representation that can then be translated into sql. gao et al. (2023) explore using open-source large language models (llms) and supervised fine-tuning methods for text-to-sql tasks. tarbell et al. (2023) investigate the generalizability of the text-to-sql models across different ehr systems and data formats. they also introduce a data augmentation approach to improve generalizability. there has been limited prior work in improving the reliability of text-to-sql systems, and this task aims to address this gap. ## dataset the dataset provided for the shared task is the medical information mart for intensive care iv (mimic-iv) demo version of ehrsql with additional unanswerable questions @xcite . mimic-iv @xcite @xcite is a large database containing deidentified patient information from beth israel deaconess medical center. it is a relational database consisting of twenty-six tables, which stores the data collected during routine clinical care, including patient demographics, vital signs, diagnoses, medications, and procedures. ehrsql @xcite is a dataset designed to evaluate and enhance text-to-sql systems specifically tailored for ehrs. it contains real-world questions gathered from various hospital staff, including doctors, nurses, insurance specialists, and record-keeping personnel. this ensures that the dataset incorporates practical queries healthcare professionals ask daily. the questions range from simple data lookups to complex calculations, such as determining patient survival rates. recognizing that not all questions have answers within the ehr data, ehrsql empowers text-to-sql systems to identify cases when low confidence or missing information necessitates abstaining from a response. the given dataset consists of 5,124 samples in the training set, 1,163 samples for validation, and 1,167 samples for testing. the database encompasses eighteen different tables adapted from mimic-iv. the training data is exclusively used for model training, while the validation data is not utilized during training. no external data sources are incorporated and no data augmentation techniques are employed. ## system description our system contains three modules -abstention module, text-to-sql generation module, and reliability module. ## experiments the system is developed incrementally, allowing us to evaluate each module after its introduction. we begin with a baseline system consisting solely of the text-to-generation module and an experiment is conducted with different representations of the table metadata. error filtering is introduced through the execution of the query. next, we integrate the abstention module and compare its performance to the multi-task model trained on both text-to-sql generation and abstention tasks. finally, we incorporate the reliability check of confidence score thresholding, experimenting with different threshold values to optimize performance. ## results the evaluation results of the different experiments can be seen in table -1. we achieve the best performance with the system of abstention module + text-to-sql generation module + reliability module, with a confidence score threshold of -0.4 (see fig 2 for confidence score distribution). our submission ranks 6th on the leaderboard. ## conclusion in this work, we present a system consisting of several layers that contribute to reliable text-to-sql modeling. by filtering out unanswerable questions based on the provided database schema, the system avoids generating incorrect sql queries and focuses on its strengths. this makes the system robust to unanswerable questions. the reliability module ensures a certain level of confidence in the generated query before using it. error filtering avoids errors during execution. fine-tuning the pre-trained models on ehr data specifically helps the system understand the medical language and schema, leading to more accurate sql generation for ehr-related queries in comparison to models trained on generic text-to-sql tasks. using pre-trained text-to-sql models as a starting point helps the system leverage existing knowledge and reduces the amount of training data required for fine-tuning. this has led to resource efficiency as no external training data is used. the modular design allows for easier development, maintenance, and potential future improvements to each specific module. this facilitates adaptation to evolving requirements or changes in the dataset or task. overall, this system presents a promising approach for reliable text-to-sql generation in the ehr domain. however, the potential limitations need to be managed to ensure the system's robust and reliable performance in real-world applications.