README : Bridging Medical Jargon and Lay Understanding for Patient Education through Data-Centric NLP (2024)

Abstract: The advancement in healthcare has shifted focus toward patient-centric approaches, particularly in self-care and patient education, facilitated by access to Electronic Health Records (EHR). However, medical jargon in EHRs poses significant challenges in patient comprehension. To address this, we introduce a new task of automatically generating lay definitions, aiming to simplify complex medical terms into patient-friendly lay language. We first created the README dataset, an extensive collection of over 50,000 unique (medical term, lay definition) pairs and 300,000 mentions, each offering context-aware lay definitions manually annotated by domain experts. We have also engineered a data-centric Human-AI pipeline that synergizes data filtering, augmentation, and selection to improve data quality. We then used README as the training data for models and leveraged a Retrieval-Augmented Generation method to reduce hallucinations and improve the quality of model outputs. Our extensive automatic and human evaluations demonstrate that open-source mobile-friendly models, when fine-tuned with high-quality data, are capable of matching or even surpassing the performance of state-of-the-art closed-source large language models like ChatGPT. This research represents a significant stride in closing the knowledge gap in patient education and advancing patient-centric healthcare solutions.

## introduction throughout the extensive history of natural language processing (nlp), enhancing individuals' ability to read and comprehend complex texts has been a significant research goal that remains only partially achieved @xcite . among the various tasks in this field, improving the comprehension of medical electronic health records 1 our data is released at https://github.com/ seasonyao/noteaid-readme and https://huggingface. co/datasets/bio-nlp-umass/noteaid-readme with cc-by-nc 4.0 license. figure 1 : a visualization of the noteaid pipeline, where nlp tools first identify jargon that may be challenging for patients to understand. the lay definitions corresponding to these jargon terms are then retrieved from relevant dictionaries and presented to the patients, enhancing their comprehension and engagement with their health information. however, despite advancements in ehrs management @xcite , one significant barrier persists in the form of medical jargon foot_0 in ehrs, impeding patient understanding and selfcare @xcite @xcite . as shown in figure 1 , tools like noteaid @xcite , which employs nlp to demystify complex medical terms, have been instrumental in bridging the communication gap between healthcare professionals and patients @xcite @xcite . however, lay language dictionary resources like the consumer health vocabulary (chv) @xcite @xcite are limited in scale, posing a challenge to noteaid. for instance, only a fraction (about 4%) of the medical terms in noteaid have been annotated with lay definitions, highlighting the need for a more scalable solution to address this knowledge gap effectively. addressing this issue requires shifting our focus to online health education materials such as unified medical language system (umls) @xcite , medlineplus @xcite , wikipedia, and google. however, as indicated in yet, the critical aspect of generating lay definitions remains largely unexplored. as patients gain more access to their ehrs, the demand for lay definition resources is escalating. despite our efforts to expand them, they are inevitably destined to surpass the capacity of current expert-annotated resources. moreover, the dynamic nature of "jargon" based on individual and context makes preannotated expert resources less adaptable to reallife scenarios. the model-driven automatic genera-tion of lay definitions from medical jargon emerges as a viable solution. recent research highlighted chatgpt's potential in its integration with the field of medicine @xcite @xcite , including generating humanreadable definitions for biomedical terms @xcite . nonetheless, our evaluation of open-source models (refer to figure 3 ) shows a significant performance degradation compared to chatgpt. using open-source large language models like llama2 @xcite and small language models such as gpt-2 @xcite is crucial because proprietary llms accessed via third-party apis may not always be feasible, especially in fields like healthcare with strict privacy requirements and economic constraints. open-source models offer the necessary privacy, while smaller models provide economic and infrastructural benefits, addressing distinct concerns about effectively deploying nlp tools in healthcare scenarios. to bridge this gap, we aim to train an in-house system using open-source models for automatic lay definition generation to provide reliable lay definitions for jargon in patient education tools like noteaid. inspired by research on retrieval-augmented generation (rag) in general and medical domains @xcite @xcite @xcite , we designed to use external resources to overcome the limitations of these open-source models in medical knowledge @xcite @xcite . we are positioning automatic lay definition generation as a form of text simplification, where language models are prompted to generate context-aware, jargon-specific, and layperson-friendly definitions based on general definitions retrieved from external knowledge resources. specifically, in this work, we use the umls to retrieve general definitions of jargon terms and construct a dataset upon the readme that includes context, jargon terms, general definitions, and lay definitions. to improve the initial readme dataset's data quality for model training, we developed a data-focused process called examiner-augmenter-examiner (eae), as illustrated in figure 2 . drawing inspiration from the human-in-the-loop concept (monarch, 2021), we employed human experts to guide ai in both the examiner and augmenter stages. the examiner filters high-quality training data, which may come from expert an-notations before the augmenter stage or from aigenerated content after. the augmenter generates potentially high-quality synthesized data to increase data points. in the end of eae, human annotators review filtered data to ensure its quality. after obtaining high-quality expertannotated and ai-synthesized datasets, we employed the ai-synthesized dataset to augment the expert-annotated dataset, aiming to explore the effectiveness of ai-synthesized data in training. we implemented a range of heuristic data selection strategies to integrate ai synthetic data, allowing us to incorporate suitable data points into our training process. in summary, our contributions are as follows: ## problem statement consider a dataset @xmath0, y, z + } comprising t ehrs, where @xmath1, x 2 , . . . , x t } represents the contexts of these ehrs, @xmath2, y 2 , . . . , y t } denotes the corresponding jargon terms, and z + = {z 1 + , z 2 + , . . . , z t + } are the ground truth expert lay definitions. each ehr context x i is a sequence of n tokens, expressed as x @xmath3, x i 2 , . . . , x i n }, and each lay definition z i + consists of m tokens, given by z i + = {z i +,1 , z i +,2 , . . . , z i +,m }. the readme lay definition generation task t aims to train a reference model m ref such that m ref (z i + | x i , y i ) is optimized. the standard approach for fine-tuning m ref on t involves using the crossentropy loss ℓ ce (z i + , m ref (x i , y i )) over the dataset d. to enhance the training of m ref , we introduce an additional set of general definitions z -= {z 1 -, z 2 -, . . . , z t -}, where each z i -corresponds to the general definition of the jargon term y i , generated using openly available data sources (umls) or gpt-3.5-turbo. our proposed eae pipeline is designed to acquire high-quality general definition data z -, culminating in the augmented dataset d @xmath4, y, z + , z -}. the readme lay definition generation task t is then formalized as a text simplification task, where m ref is trained to produce z + based on x, y, and z -. this process utilizes a selected subset d sel ⊆ d simp , chosen according to one of the selection criteria: ran-dom, syntax, semantic, or model. ## conclusions our study underscores the potential of nlp to democratize medical knowledge, enabling patientcentric care by simplifying complex medical terminology. developing the readme dataset and implementing a data-centric pipeline has improved dataset quality and expanded ai training possibilities. our experiments show that small open-source models can match advanced close-source llms like chatgpt with well-curated data. readme will be open to the community as an important lay dictionary for patient education. we hope our work can help the innovative patient education community advance toward a future where all patients can easily understand their health information.