Saama Technologies at EHRSQL 2024: SQL Generation through Classification Answer Selector by LLM (2024)

Abstract: The EHRSQL task aims to develop a dependable text-to-SQL model for Electronic Health Records (EHR) databases, which are crucial sources of clinical data that store patients’ medical histories in hospitals. Large language models (LLM) have been proven to exhibit state-of-the-art performance for text-to-SQL tasks across various domains. To this end, we have developed a framework, SQL Generation through Classification Answer Selector by LLM (SCAS), which comprises two modules. The CAS module determines the answerability of the question, while the SG model generates the SQL query exclusively for answerable questions. Our system ranked 7th on the leaderboard with a Reliability Score of 53.21 on the official test set.

## introduction electronic health records (ehrs) are an essential component of modern healthcare. they store a patient's complete medical history, allowing hospital staff to make better clinical decisions @xcite by quickly accessing relevant patient information. however, accessing this information can be time-consuming, especially when complex queries are involved. the traditional way of accessing ehrs involves using a predefined rule conversion system to convert user queries to sql and retrieve the relevant information. this process can become a bottleneck for users who need to build custom queries or deal with complex queries. to address this issue, the ehrsql @xcite task aims to develop a system that can automatically translate user questions into corresponding sql queries, making retrieving the information they need easier and quicker. the system's objective is to build a text-to-sql system that converts natural language queries to sql and informs users whether their queries are answerable. text-to-sql tasks @xcite involve mapping natural language questions onto a given relational database into sql queries. early studies @xcite tackled this task with pre-defined rules or as a sequence-to-sequence task. however, recent advancements in large language models (llms) @xcite @xcite have become a milestone for natural language processing and machine learning. llms are pre-trained on massive text corpus, which enables them to perform various natural language tasks, and their ability to do in-context learning @xcite makes them most suitable for text-to-sql generation. in this paper, we present our approach to tackling the ehrsql 2024 @xcite shared task, which involves a complex dataset of electronic health records. our proposed framework, the sql generation through classification answer selector by llm (scas), helps avoiding incorrect sql generation by using the classification answer selector (cas) module. the cas module uses an llm prompting method that incorporates the output of other classification models to generate the final classification output, thereby abstaining from incorrect responses. the scas framework generates sql queries only for necessary questions by utilizing other llm models. our system achieved a 7th position on the leaderboard, with a reliability score of 53.21 on the official test set. the code to reproduce the experiments mentioned in this paper is publicly available foot_0 . the reliability score (rs) is a new evaluation metric for text-to-sql models used in the ehrsql task. it rewards accurate sql generation for certain types of questions while penalizing incorrect sql generation for others. it does not assign any reward or penalty for abstaining from answering certain questions. the competition uses several scoring systems, including rs(0), rs(5), rs(10), and rs(n), with rs(10) being the primary metric for the leaderboard. in rs(10), correct predictions receive one positive point, while incorrect predictions receive -10 points. n in rs(n) represents the size of the test set. ## finetuning we fine-tuned the model using the efficient parameter tuning method lora @xcite and the huggingface library @xcite . the finetuning process for both the cas and sql generation modules involved using the adamw optimizer and a cosine learning rate scheduler, targeting all linear layers within the model. we em-exp no development phase rs(0) rs( 5 ) rs( 10 ## results we tried both fine-tuning and in-context learning of llm for this task. we established a baseline for our experiment using the gpt-3.5-turbo model, which received a rs(10) score of -561.56 points. to enable in-context learning for the model, we used few-shot prompting. in the second experiment, we fine-tuned the codellama model, and despite having only 7b parameters, it outperformed the gpt-3.5turbo model. notably, both the question classifier and sql generation in both 1 & 2 experiments used the same model. experiment 3 showcased the robustness of our sql generation module, as it achieved an impressive rs(0) score of 74.97. this indicates that our system can correctly generate executable queries 74.97% of the time. experiment 4 focused on improving the question classification model by adjusting the threshold to identify unanswerable questions better. while this enhanced the rs(10) score by 23.48 points, it caused a decrease in the rs(0) score due to misclassification. experiment 5 evaluated the performance of the codellama-based question classifier, which showed no significant improvement over classical models. finally, in experiment 6, we used the cas mod-ule that combines the result of two classification models to enhance the gpt-3.5-turbo model's performance. the input the cas module is detailed in section 2. after completing the development phase, we submitted our top-performing model for the testing phase, scoring rs(0) of 53.21 and rs(10) of 36.07 points. the sql generator module achieved an impressive rs(0) score of 77.03 points during the test phase, without any adjustments made to the question classifier threshold, which shows the capabilities of the sql generator module. after performing an error analysis on the cas module, it was discovered that false negatives were higher. this indicates that some answerable questions were incorrectly classified as unanswerable. since our system is designed as a pipeline model, only the questions classified as answerable will advance to the sql generation model. this resulted in a decrease in the rs(0) score. notably, most of the false negative predictions were observed in queries related to test procedures, hospitals and departments. ## conclusion we developed a sophisticated system that can generate sql queries from user queries in the ehr dataset, provided they are convertible to sql. our system was able to achieve an impressive rank of 7 on the ehrsql task. our experiments have shown that fine-tuning an llm for task-specific sql generation significantly enhances its performance compared to in-context learning. however, we acknowledge that our system needs improvement in identifying which user queries can be successfully converted to sql. this is crucial for ensuring the reliability of our sql converter system. to facilitate the reproducibility of our work, we have made available instruction templates, code, and pre-trained models as open-source resources.