Two Directions for Clinical Data Generation with Large Language Models: Data-to-Label and Label-to-Data (2023)

Abstract: Large language models (LLMs) can generate natural language texts for various domains and tasks, but their potential for clinical text mining, a domain with scarce, sensitive, and imbalanced medical data, is under-explored. We investigate whether LLMs can augment clinical data for detecting Alzheimer’s Disease (AD)-related signs and symptoms from electronic health records (EHRs), a challenging task that requires high expertise. We create a novel pragmatic taxonomy for AD sign and symptom progression based on expert knowledge and generated three datasets: (1) a gold dataset annotated by human experts on longitudinal EHRs of AD patients; (2) a silver dataset created by the data-to-label method, which labels sentences from a public EHR collection with AD-related signs and symptoms; and (3) a bronze dataset created by the label-to-data method which generates sentences with AD-related signs and symptoms based on the label definition. We train a system to detect AD-related signs and symptoms from EHRs. We find that the silver and bronze datasets improves the system performance, outperforming the system using only the gold dataset. This shows that LLMs can generate synthetic clinical data for a complex task by incorporating expert knowledge, and our label-to-data method can produce datasets that are free of sensitive information, while maintaining acceptable quality.

## introduction clinical text holds a large amount of valuable information that is not recorded by the structured data fields in electronic health records @xcite . clinical text mining, which aims to extract and analyze information from medical records, such as diagnosis, symptoms, treatments, and outcomes, has various applications, such as clinical decision support, disease surveillance, patient education, and biomedical research @xcite . however, clinical text mining faces two major obstacles: the scarcity and sensitivity of medical data. medical data is often limited in quantity and diversity, due to the high cost and difficulty of data collection and annotation, which require expert knowledge and consent from patients and providers. on the other hand, medical data is highly sensitive and confidential, due to the ethical and legal issues of data privacy and security, which impose strict regulations and restrictions on data collection and usage @xcite . these obstacles hinder the development and evaluation of clinical text mining methods, especially those based on data-hungry deep learning models. recently llms have demonstrated impressive performance on many natural language processing (nlp) benchmarks, @xcite @xcite , as well as in medical domain applications @xcite @xcite . however, they also face some common problems like hallucination, homogenisation, etc @xcite . hallucination means that llms produce factual errors or inconsistencies in their outputs that do not match the input or the real world. this can damage the reliability and credibility of llms, especially for applications in clinical domain that need high accuracy and consistency. in addition, data generated by llms tends to be highly homogeneous and fails to capture the diversity and realism of real data, which are essential for many downstream tasks. for example, for clinical text analysis, we need the generated data to cover different types of clinical texts, such as patient histories, diagnoses, or treatment plans. this presents a huge challenge for llm. the difference between llm generated data and real data makes people doubt llms' practical application value. in this paper, we investigated whether the outputs of llms can be a valuable data source for clinical text mining despite all these aforementioned drawbacks. we focus on alzheimer's disease (ad) signs and symptoms detection from electronic health records (ehrs) notes. alzheimer's disease (ad) is a progressive neurodegenerative disorder that affects millions of people worldwide @xcite . it can cause cognitive impairment, behavioral changes, and functional decline. detecting ad-related signs and symptoms from ehr is a crucial task for early diagnosis, treatment, and care planning @xcite . in addition to the scarcity, sensitivity, and imbalance of clinical data, this task is highly challenging due to the high expertise required to interpret the complex and diverse manifestations of ad @xcite . we propose a novel pragmatic taxonomy for ad sign and symptom progression based on expert knowledge, which consists of nine categories that capture the cognitive, behavioral, and functional aspects of ad @xcite . we created three datasets following the taxonomy: (1) a gold dataset annotated by human experts on longitudinal ehrs of ad patients; (2) a silver dataset created by the data-to-label method which labels sentences from a public ehr collection with ad-related signs and symptoms; and (3) a bronze dataset created by the label-to-data method which generates sentences with ad-related signs and symptoms based on the label definition. the "data-to-label" method employs llms as annotators and has been widely adopted in many tasks. the "label-to-data", on the other hand, relies on the llm's generation ability to produce data with labels based on instructions. we performed experiments of binary classification (whether the sentence is related to ad signs and symptoms or not) and multi-class classification (assign one category from the nine pre-defined categories of ad signs and symptoms to an input sentence), using different data combinations to finetune pre-trained language models (plms), and we compared their performance on the human annotated gold test set. we observed that the system performances can be significantly improved by the silver and bronze datasets. in particular, combing the gold and bronze dataset, which is generated by the label-to-data method, outperform the model trained only on the gold or gold+silver dataset for some categories. the minority classes with much fewer gold data samples benefit more from the improvement. we noticed slight degradation of performances for a small proportion of categories. but the overall increases in results demonstrates that llm can be applied to medical data annotation, and even its hallucinations can be leveraged to create datasets that are free of sensitive information, while preserving acceptable quality. the contributions of this paper are as follows: ## conclusion and future work in this paper, we examined the possibility of using llms for medical data generation, and assessed the effect of llms' outputs on clinical text mining. we developed three datasets: a gold dataset annotated by human experts from a medical dataset, which is the most widely used method for clinical data generation, a silver dataset annotated by the llm from mimic (data-to-label) and a bronze dataset generated by the llm from its hallucinations (labelto-data). we conducted experiments to train classifiers to detect and categorize alzheimer's disease (ad)-related symptoms from medical records. we discovered that using a combination of gold data plus bronze and/or silver achieved better performances than using gold data only, especially for minority categories, and that the llm annotations and hallucinations were helpful for augmenting the training data, despite some noise and errors. our findings suggest that llm can be a valuable tool for medical data annotation when used carefully, especially when the data is scarce, sensitive, or costly to obtain and annotate. by using llm hallucinations, we can create synthetic data that does not contain real patient information, and that can capture some aspects of the clinical language and domain knowledge. however, our approach also has some ethical and practical challenges, such as ensuring the quality, diversity, validity, and reliability of the llm annotations and hallucinations, protecting the privacy and security of the data and the model, and avoiding the potential harms and biases of the llm outputs. for future work, we will investigate other methods and techniques for enhancing and regulating the llm annotations and hallucinations, such as using prompts, feedback, or adversarial learning. and we would also tackle the ethical and practical issues of using llm for medical data annotation, by adhering to the best practices and guidelines for responsible and trustworthy ai. we also intend to apply our approach to other clinical text processing tasks, such as relation extraction, entity linking, and clinical note generation.