Knowledge-Infused Prompting: Assessing and Advancing Clinical Text Data Generation with Large Language Models (2024)

Abstract: Clinical natural language processing faces challenges like complex medical terminology and clinical contexts. Recently, large language models (LLMs) have shown promise in this domain. Yet, their direct deployment can lead to privacy issues and are constrained by resources. To address this challenge, we delve into synthetic clinical text generation with LLMs for clinical NLP tasks. We propose an innovative, resource-efficient approach, ClinGen, which infuses knowledge into the process. Our model involves clinical knowledge extraction and context-informed LLM prompting. Both clinical topics and writing styles are drawn from external domain-specific knowledge graphs and LLMs to guide data generation. Our extensive empirical study across 8 clinical NLP tasks and 18 datasets reveals that ClinGen consistently enhances performance across various tasks by 7.7%-8.7% on average, effectively aligning the distribution of real datasets and enriching the diversity of generated training instances.

## introduction clinical natural language processing (nlp) emerges as a distinct subfield including the extraction, analysis, and interpretation of unstructured clinical text @xcite . despite its significance, unique challenges exist for methodology development in clinical nlp. for example, clinical texts are often dense with abbreviations and specialized medical terminologies can be perplexing to standard nlp models @xcite . fortunately, recent advances in large language models (llms) @xcite @xcite provide a promising way to resolve these issues, as they contain billions of parameters and have been pretrained on massive corpora, thus inherently capture a significant amount of clinical knowledge @xcite . these progresses inspire the need for designing specialized approaches for adapting llms to clinical settings, which both address the terminology complexities and improve models through clinical data finetuning @xcite . despite the strong capacity of general llms, directly applying them to infer over clinical text data is often undesired in practice. firstly, these llms often have billions of parameters that translate to significant computational resources even for inference, leading to increased infrastructure costs and long inference time. furthermore, the sensitive patient information in the clinical text naturally raises privacy and regulatory compliance concerns @xcite . to combat these challenges, generating synthetic training data using llms serves as a promising solution, as it leverages the capability of llms in a resourceefficient and privacy-centric way. when trained with synthetic data mimicking real-world clinical data, models can achieve high performance while obeying data protection regulations. synthetic data generation with llms is a popular research area in nlp @xcite @xcite , with a focus on gemeral-domain data. however, adapting llms trained on general texts for generating high-quality clinical data poses distinct challenges. to assess the quality of data generated by existing methods, we carry out an evaluation centered on distribution and diversity, detailed in section 3, which indicate a noteworthy data distribution shift. we further examine the clinically-related entity quantities and frequencies in synthetic data, where a notable decline is observed when contrasting synthetic data with ground truth data. while some research has delved into clinical data generation with language models, many of these efforts are tailored to specific tasks. examples include medical dialogues @xcite , clinical notes @xcite , and electronic health records @xcite . these studies often directly adopt language models for text generation, and sometimes on excessive training data. till now, a unified principle to better adapt llms for generating synthetic text for facilitating clinical downstream applications is still missing. motivated by the above analysis, we propose clingen, a clinical knowledge-infused framework for high-quality clinical text generation in few-shot scenarios. our ultimate goal is to bridge the gap between synthetic and real data while enhancing topic diversity. towards this end, we propose to utilize clinical knowledge extraction to contextualize the prompts. this includes generating clinical topics on entity and relation information from both kgs and llms and deriving writing style suggestions from llms. by doing this, clin-gen integrates both non-parametric insights from external clinical knowledge graphs with the intrinsic parametric knowledge encoded in llms and enjoys higher diversity via dynamically composing different topics and writing styles together during the data generation process. it is worth noting that, clingen only relies on minimal additional human efforts, and can be readily applied to a wide array of core tasks in clinical nlp. our contributions can be summarized as follows: ## related work generating additional training data enables a more precise analysis of medical text, and has gained more attention in the past years. earlier research has employed data augmentation techniques to generate similar samples to existing instances with word substitution @xcite , back translation @xcite , pretrained transformers @xcite . but they often yield rigid transformations and the quality of the augmented text cannot be always guaranteed. the emergence of llms has presented new possibilities for synthetic data generation @xcite @xcite . however, these methods often use generic and simple prompts that may not fully capture domain-specific knowledge, thus potentially limiting the quality of the generated data. @xcite ; @xcite ; @xcite employ interactive learning to generate instances, at the cost of additional human efforts. several recent studies explore llmbased synthetic data generation for clinical nlp. @xcite rely on a much larger training set to generate candidate entities, which disregards the practical low-resource setting @xcite . moreover, these studies often concentrate on specific target tasks, thus lacking generality for diverse clinical nlp scenarios. on the other hand, several works aimed at optimizing prompts using llms @xcite or knowledge graphs @xcite ), yet they mainly focus on refining prompts to obtain the answer for the given input, and the prompt template often remains unchanged. instead, we focus on the different task of generating training instances. by composing different topics and styles together, we can generate diverse templates for prompting llms to improve the quality of the synthetic data. ## preliminary study this section first presents the foundational setup of synthetic data generation. then, we provide an in-depth investigation into the pitfalls of existing synthetic data generation methods. ## knowledge infused data generation section 3 highlights the necessity of domaintailored knowledge for clinical synthetic data generation. in pursuit of this, we present clingen, a knowledge-informed framework for clinical data generation. the overview of clingen is shown in figure 2 . this two-step methodology harnesses the emergent capabilities of llms and external knowledge from kgs to facilitate the synthesis of clinical data, even with few-shot examples only. ## empirical evaluation given our focus on data generation, our major interest lies in faithfully evaluating different synthetic text generation approaches under few-shot scenarios, rather than competing in a "state-of-the-art" race with general few-shot nlp methods. the following questions particularly intrigue us: rq1: how does clingen perform when compared with baselines on different downstream tasks? rq2: what impact do factors like llm generators and synthetic data size have on the performance of clingen? rq3: how is the quality of the synthetic data generated by clingen and baselines? ## llm-based data generation techniques. see appendix d for their descriptions. implementation details. for implementation, we use pytorch @xcite and hugging-face @xcite . for each dataset, we randomly sample 5 examples from each class to provide few-shot demonstrations and keep a validation set of the same size. during the data generation process when we call the chatgpt apis (openai, 2023b), we set the parameter @xmath0.0 and temperature @xmath1.0 to balance between the quality of the generated text as well as diversity @xcite foot_1 . in the experiments, we generate 5000 synthetic training data for both clingen and the baselines and report the average performance over 3 random seeds for all the results. with the generated synthetic dataset, we follow the common few-shot learning setting @xcite to train all the models for 6 epochs and use the model with the best performance on the validation set for evaluation. during the pubmed-bert fine-tuning, we adopt adamw @xcite for optimization with a linear warmup of the first 5% steps and linear learning rate decay. the learning rate is set to 2e-5 for base and 1e-5 for large, and the maximum number of tokens per sequence is 256. ## conclusion in this work, we study clinical text data generation using llms. we thoroughly assess existing methods for clinical data generation and identify issues including distribution shifts and limited diversity. to tackle these challenges, we introduce clin-gen, a framework that leverages clinical knowledge from non-parametric kgs and parametric llms. this empowers data generation by utilizing clinical topic knowledge and real-world writing styles in domain-specific prompts. our extensive empirical evaluations across 8 clinical nlp tasks and 18 datasets, compared to 10 baseline methods, consistently show that clingen improves task performance, aligns closely with real data, and enhances data diversity. we expect clingen can be seamlessly incorporated into a broad suite of clinical text tasks to advance clinical nlp research.