Project PRIMUS at EHRSQL 2024 : Text-to- SQL Generation using Large Language Model for EHR Analysis (2024)

Abstract: This paper explores the application of the sqlcoders model, a pre-trained neural network, for automatic SQL query generation from natural language questions. We focus on the model’s internal functionality and demonstrate its effectiveness on a domain-specific validation dataset provided by EHRSQL. The sqlcoders model, based on transformers with attention mechanisms, has been trained on paired examples of natural language questions and corresponding SQL queries. It takes advantage of a carefully crafted prompt that incorporates the database schema alongside the question to guide the model towards the desired output format.

## introduction electronic health records (ehrs), a large collection of data related to digital medical records, serve as the backbone of modern healthcare, storing a wealth of patient information. this data, encompassing diagnoses, procedures, medications, and more, offers invaluable insights for clinical decision-making and research. @xcite however, effectively utilizing this vast resource is often hampered by the complexity of querying the underlying relational databases. traditionally, hospital staff relies on predefined rule conversion systems to interact with ehr databases. these systems, while functional, limit access to information beyond pre-configured rules. modifying and extending these systems requires specialized training, creating a bottleneck for users seeking broader data access. this paper explores the potential of natural language processing (nlp) to bridge this gap. we present a system that leverages the power of large language models (llms) to automatically translate natural language questions into corresponding sql queries. this approach empowers users to directly query the ehr database using natural language, eliminating the need for complex sql syntax and significantly streamlining data retrieval. the core of our system lies in a pre-trained llm, specifically the sqlcoders model. this model, trained on paired examples of natural language questions and their corresponding sql queries, has the remarkable ability to understand the user's intent and translate it into the appropriate database query language. we delve into the inner workings of the sqlcoders model and the concept of prompt engineering, a crucial aspect of guiding the llm towards generating accurate sql statements. by focusing on open-source llms like sqlcoders, our work contributes to the broader exploration of readily available resources for nlp tasks. we aim to demonstrate the effectiveness of supervised fine-tuning in enhancing the performance of open-source llms for the challenging task of text-to-sql translation in the specific domain of healthcare. this paper is structured as follows. first, we discuss related work on text-to-sql translation, highlighting the advantages of llm-based approaches and the importance of prompt engineering. subsequently, we introduce the sqlcoders model and its methodology. we then present our approach and implementation details, followed by an evaluation of the system's performance. finally, we conclude by discussing the implications of our work and outlining future directions. ## related work extracting sql queries from natural language questions has been a well-studied area within nlp, with applications spanning various domains. this section explores relevant research directions and highlights how the sqlcoders model aligns with these methodologies. ## methodology this section delves into the research methodology employed to investigate the effectiveness of the sqlcoders model for automated sql query generation from natural language questions in the healthcare domain. we exploit the model's capability to learn intricate relationships between natural language and database structures, coupled with the power of prompt engineering, to achieve this goal. ## results this section dives deeper into the model's performance based on the reward scoring (rs) schemes employed for evaluation. ## conclusion and future direction this investigation evaluated the sqlcoder model's performance in generating sql queries using various reliaibilty scoring (rs) schemes. though the model shows a basic capability to generate some correct results (evident in the positive rs(0) score), its overall accuracy and robustness require significant improvement. .by focusing on exploration of different model architectures, enhanced error handling and incorporating human expertise, future investigations hold promise for significant advancements in this domain.