AIRI NLP Team at EHRSQL 2024 Shared Task: T5 and Logistic Regression to the Rescue (2024)

Abstract: This paper presents a system developed for the Clinical NLP 2024 Shared Task, focusing on reliable text-to-SQL modeling on Electronic Health Records (EHRs). The goal is to create a model that accurately generates SQL queries for answerable questions while avoiding incorrect responses and handling unanswerable queries. Our approach comprises three main components: a query correspondence model, a text-to-SQL model, and an SQL verifier.For the query correspondence model, we trained a logistic regression model using hand-crafted features to distinguish between answerable and unanswerable queries. As for the text-to-SQL model, we utilized T5-3B as a pretrained language model, further fine-tuned on pairs of natural language questions and corresponding SQL queries. Finally, we applied the SQL verifier to inspect the resulting SQL queries.During the evaluation stage of the shared task, our system achieved an accuracy of 68.9 % (metric version without penalty), positioning it at the fifth-place ranking. While our approach did not surpass solutions based on large language models (LMMs) like ChatGPT, it demonstrates the promising potential of domain-specific specialized models that are more resource-efficient. The code is publicly available at https://github.com/runnerup96/EHRSQL-text2sql-solution.

## introduction electronic health records (ehrs) play a critical role in storing comprehensive medical histories within hospital settings, capturing everything from patient admissions to treatment and discharge. however, efficiently retrieving relevant information from these records remains a significant challenge, particularly from complex medical relational databases. this paper focuses on enhancing a text-to-sql system specifically designed for the medical domain. the aim is to improve the retrieval pro-figure 1 : the schema intersection algorithm. we match normalized n-grams of an input natural language question "how much is the cost for the drug nystatin cream?" against normalized database (db) content. as we can see on schema intersection count distribution, the null questions have much less schema intersection elements in comparison to sql questions. for more details, please refer to section 3. cess of patient information and enable better clinical decision-making. the objective is to develop a model capable of accurately generating sql queries for answerable questions while effectively handling unanswerable queries and avoiding incorrect responses. in other words, when faced with unanswerable questions, the model should refrain from generating any sql prediction and indicate the absence of an answer by returning null. to conduct experiments, we utilize the text-to-sql benchmark @xcite provided by the organizers of the clinical nlp 2024 task @xcite . this benchmark consists of pairs of input utterances and expected sql queries, including cases where generating an sql query is impossible for a given question. this dataset is linked to two open-source ehr databases-mimic-iii @xcite and eicu @xcite . this benchmark includes questions that address the actual needs of a hospital and incorporate various time expressions crucial to daily healthcare work. in this paper, we describe our solution for the clinical nlp 2024 shared task on reliable text-to-sql. as shown in figure 2 , our system consists of three components -query correspondence model, text-to-sql model, and sql result inspector. to sum up, the system takes the user's query as input and goes through the following steps: feature extraction, query scoring using the query correspondence model and alignment check, sql generation using a text-to-sql model with question and schema input representation, sql results inspection, execution of the generated query, and checking if the execution result meets the requirements. if the requirements are met, the system returns the result to the user. the paper is organized as follows. section 2 presents related work on text-to-sql corpora and state-of-the-art (sota) models. we describe our model with three components in section 3. experiments with baselines and our model are presented in section 4. finally, we discuss errors and conclude the work in sections 5 and 6, respectively. ## related work text-to-sql currently is one of the most developing and promising research areas in the field of semantic parsing. well-known public leaderboard spider @xcite popularized the task, and text-to-sql domain developed many directions and specializations. spider dataset is a complex and cross-domain text-to-sql dataset which consists of 10181 questions with 5693 sql queries on 200 databases. the main goal of the dataset is to generalize to new databases. however, the spider dataset does not contain unanswerable questions. spider also gave rise to more complex datasets like bird @xcite , which paid attention not only to sql query complexity (introduction of window functions, etc.) but also to the optimality of the generated query. bird databases are close to real-world examples, with tables consisting of millions of data rows; hence, the optimal sql is required. another dataset named cosql @xcite ) raised questions about ambiguity and the system's ability to handle such questions. it is a dialoguebased text-to-sql benchmark, which consists of the following dialogue acts -answering user questions with sql, double checking the user intent if the questions are ambiguous, or the system reminder to the user that the question is not related to the database. spider leaderboard gave rise to specialized textto-sql architectures. naturally, sota solutions adapted the following text-to-sql solutionsschema linking stage, encoding of question and schema, and subsequent decoding. starting from most notable solutions like bridge, which induced database content into training process @xcite and rat-sql, which modified transformer architecture for question with schema interaction and specialized grammar-based decoding process @xcite coming to the finetuning approaches which reached its peak in res-dsql @xcite approach and pickard @xcite . llms are also present in the leaderboard in the form of in-context learning few-shot approaches @xcite and sql debugging stages @xcite . most solutions utilize chatgpt-4 as a core model and experiment with different prompt strategies for stages of schema linking, query generation, and sql debugging. increased attention towards text-to-sql domain detected the problem of generalization in semantic parsing. the spider dataset focused on crossdomain generalization, but the work of @xcite made the challenges more visible, introducing the challenges of single database split compared to cross-database setting. recently, somov and tutubalina (2023) evaluated the generalization capabilities of supervised models on the original, multilingual, and target length splits of the improved version of the spider dataset called pauq @xcite . results indicate that the models can generalize well to unseen simple sql's, while multilingual split shows that some models benefit from learning on the translated task. overall, the ongoing progress in dataset development and the advancement of specialized architectures have significantly contributed to a deeper understanding of the text-to-sql task and its applications across various domains, including medicine. ## main method our final solution consists of 3 components -query correspondence module, fine-tuned text-to-sql model, and sql result inspector, which checks the result of the generated query. the system pipeline is presented in figure 2 . the system output can be null if the system considers the query unanswerable or results if the system can answer the query. this section will describe our validation schema and all the system components in detail. ## experiments this section describes our most successful attempts on the test leaderboard. solutions feature schema intersection algorithm explained in sec. 3.2 and sql inspector explained in sec. 3.4. in ## error analysis we manually checked the errors of our final system components on our validations sets from 3.1. the text-to-sql t5-3b errors mostly consist of regular errors -asc to desc mismatch, wrong column, missed comparison expression. sometimes the model shows the overfitting signs -looping the prediction output, adding wrong syntactic constraints (like adding not needed group by) or extra symbol to value ('10-31' -→ '10-31\'). the query correspondence errors come mostly from the starting word feature -although it helps to identify questions with starting phrases that were in the training set, it does not help to combat novel starting phrases that occur in the test set. as we pointed out, we evaluated our query inspector on sensitivity and specificity metrics. specificity is 99%, and sensitivity is 94%. we can see that we are stricter than necessary to generated queries, and sometimes correct sql can return the result of none or 0, but we will not return it to the user. ## discussion and conclusion in this work, we have built a reliable text-to-sql solution. we have developed our validation schema for model evaluation and submitted our final system results to the ehrsql leaderboard. our solution consists of 3 components -query correspondence model, text-to-sql generation model, and sql inspector. during validation, we measured our performance based on sensitivity and specificity metrics to account for null queries and execution match for the query generation model. the sensitivity and specificity metrics for the query correspondence model and sql inspector are 81%/99% and 94%/99% accordingly. the execution accuracy of text-to-sql model is 99%. our components are independent of each other; therefore, we can calculate the product probability that the null question will be discarded is 98%, while the probability that the sql question will be answered correctly is 75%. the advantages of our system are the following: the disadvantages of the system are: as a future work direction, we see the necessity of developing reliable, robust, and lightweight specialized solutions. these solutions can be run and maintained on-premise without exposing personal data to external llms.