How to use Language Models for Synthetic Text Generation in Cerebrovascular Disease-specific Medical Reports (2024)

Abstract: The quantity and quality of data have a significant impact on the performance of artificial intelligence (AI). However, in the biomedical domain, data often contains sensitive information such as personal details, making it challenging to secure enough data for medical AI. Consequently, there is a growing interest in synthetic data generation for medical AI. However, research has primarily focused on medical images, with little given to text-based data such as medical records. Therefore, this study explores the application of language models (LMs) for synthetic text generation in low-resource domains like medical records. It compares the results of synthetic text generation based on different LMs. To achieve this, we focused on two criteria for LM-based synthetic text generation of medical records using two keywords entered by the user: 1) the impact of the LM’s knowledge, 2) the impact of the LM’s size. Additionally, we objectively evaluated the generated synthetic text, including representative metrics such as BLUE and ROUGE, along with clinician’s evaluations.

## introduction the performance of artificial intelligence (ai) is greatly influenced by the quantity and quality of data, including computational resources @xcite . for example, massive language model (lm) services like @xcite , with 20 billion parameters, require large-scale datasets and high-performance computing resources. however, building extensive datasets in the biomedical domain is challenging for the following reasons @xcite : 1) inclusion of sensitive information such as personal details require ensuring privacy protection, information security, transparency, and related considerations. 2) caution must be exercised due to concerns such as the potential failure to maintain anonymity and the possibility of data misuse. consequently, despite the existing demand, accessing the necessary data in the biomedical domain can be difficult, with only a limited subset being publicly available. nevertheless, medical ai has consistently advanced, primarily in fields with less sensitive information, such as medical image analysis. however, text data containing extensive information, like electronic medical records (emr), is organized in different formats depending on regions or institutions and poses a challenge for effective utilization due to the inclusion of a considerable amount of sensitive information @xcite . therefore, there is a need for a technique of synthetic text generation, one of the text augmentation methods, to enable the effective utilization of medical records in medical ai. in this study, we explored methods to synthetic text generation using lms for text augmentation in low-resource medical records. we compared the following aspects: 1) the impact of the lm's knowledge, and 2) the impact of the lm's size. to do so, we compared chatgpt, biogpt @xcite , distilgpt2 @xcite , and our own gpt2 (we called cerebrogpt). at this time, we conducted experiments using low-resource medical records related to cerebrovascular diseases. finally, for objective evaluation, we performed clinician's evaluations along with four quantitative metrics (blue, rouge, cosine similarity with sentence embeddings and tf-idf). this paper is structured as follows: in chapter 2, we provide a review of prior research on text augmentation methods. chapter 3 describes the medical records we utilized in this study. chapter 4 details the methodology, while chapter 5 details the experimental setup and results. in chapter 6, we summarize our observations and present the conclusion. ## related work rule-based methods involve traditional techniques such as synonym replacement, random word insertion and deletion, collectively known as easy data augmentation @xcite . due to their simple principles, they have addressed good performance when applied to small-scale datasets. however, it may not be suitable for the medical domain when applied to text, as they could incorrectly represent the location of abnormalities or describe diseases inaccurately in medical records. model-based methods augment text based on the language understanding derived from the knowledge of lms. rakshit et al. ( 2022 ) enhanced text generation by fine-tuning gpt2 using the generative abilities of the lm. it involved adding new special tokens ("
") to the model during fine-tuning, enabling it to generate text learned based on knowledge of lms. after the release of chatgpt, prompt engineering based on chatgpt has also been attempted @xcite @xcite . in these efforts, suitable prompts for specific tasks were explored when using chatgpt for generating synthetic training data in low-resource tasks. they focused on the fact 3 medical reports in cerebrovascular disease the dataset consists of radiologist's reports on ct images collected from hallym university sacred heart hospital foot_0 and chuncheon sacred heart hospital 2 from 2012 to 2020, involving a total of 35,511 individuals which is intracerebral hemorrhage (ich), non-ich, and normal patients. while it includes a total of 90,489 reports, but there are many duplicate reports with the same content. therefore, we utilized 47,675 reports after removing duplicates. additionally, to standardize different report formats by radiologists and to address sensitive information such as personal details, we conducted preprocessing using regular expressions. all procedures in this study were performed in accordance with the decla-ration of helsinki, and it was approved by the institutional review board at chuncheon sacred heart hospital (irb no. 2021-10-012). the examples of medical reports are shown in appendix a. ## language models for synthetic text generation as shown in figure 1 , we compared the following lms for lm-based synthetic text generation of medical records: • chatgpt (20b): chatgpt utilizes the gpt3.5-turbo version, a lm that has been pretrained on diverse datasets for various domains and fine-tuned using methods like rlhf (reinforcement learning of human feedback) in instructgpt @xcite . therefore, we explored suitable prompts through prompt engineering and generated synthetic text using this prompt. it is a lm that underwent knowledge distillation from gpt2 (124m) trained on the openwebtext corpus @xcite , a corpus from a general domain, to achieve a smaller size and faster inference speed. therefore, we fine-tuned a distilgpt2 using medical record corpus. synthetic text generation requires producing diverse expressions or sentence structures for the same content. therefore, we compared four generation strategies (grid search, beam search, top-k sampling, top-p sampling) and employed the most suitable strategy for each lm. and all lms are designed to generate synthetic text based on two keywords inputted by the user. ## conclusion in this study, we explored the use of lms for synthetic text generation based on two keywords provided by the user to augment low-resource medical records. we compared chatgpt, biogpt, distilgpt2, and cerebrogpt (in-house). the experimental results indicated that, overall, distilgpt2 exhibited superior performance in semantic aspects and the expression of cerebrovascular disease-related vocabulary. moreover, distilgpt2 demonstrated the most suitable quality of synthetic text in the evaluation by clinicians. therefore, we conclude that selecting a lm tailored to the dataset size of the specific domain, rather than relying on large-scale lms (llms), is crucial for augmenting low-resource domain text. additionally, it is considered that generating high-quality synthetic text is possible by preprocessing the text into the desired format. however, as data augmentation ultimately involves generating synthetic dataset, annotating for the synthetic data must also be conducted. therefore, we will develop an annotation model for cerebrovascular disease-related synthetic text. in addition, we plan to obtain additional permission to build a database of not only radiologist's reports on ct images but also emrs from a wide range of medical fields with sensitive information removed. through this, we plan to use retrieval-augmented generation @xcite to conduct research on an advanced text augmentation algorithm based on medical facts in emr. these results are expected to play an important role in further improving synthetic text generation in the medical domain and utilizing the rich information contained in emr in the future medical ai field.