Can Synthetic Text Help Clinical Named Entity Recognition? A Study of Electronic Health Records in F rench (2023)

Abstract: In sensitive domains, the sharing of corpora is restricted due to confidentiality, copyrights or trade secrets. Automatic text generation can help alleviate these issues by producing synthetic texts that mimic the linguistic properties of real documents while preserving confidentiality. In this study, we assess the usability of synthetic corpus as a substitute training corpus for clinical information extraction. Our goal is to automatically produce a clinical case corpus annotated with clinical entities and to evaluate it for a named entity recognition (NER) task. We use two auto-regressive neural models partially or fully trained on generic French texts and fine-tuned on clinical cases to produce a corpus of synthetic clinical cases. We study variants of the generation process: (i) fine-tuning on annotated vs. plain text (in that case, annotations are obtained a posteriori) and (ii) selection of generated texts based on models parameters and filtering criteria. We then train NER models with the resulting synthetic text and evaluate them on a gold standard clinical corpus. Our experiments suggest that synthetic text is useful for clinical NER.

## introduction the lack of specific resources might be the most common hurdle encountered when working in natural language processing (nlp), whether it be the lack of task-specific resources, domain-specific resources, or both. a major challenge for biomedical nlp in languages other than english is the lack of available clinical corpora that can be shared with the community @xcite . one way of addressing this problem is to generate new data automatically. the performance of recent language models based on the transformer architecture @xcite on various tasks that involve generating text (translation, summarization, etc.) makes it possible to use the transformer's power to generate text to address resource sparsity. more specifically for generation, large generative language models such as gpt-2 @xcite and gpt-3 @xcite have the ability to generate well-written text containing a wealth of information. the knowledge gained during the pre-training of those large language models could be helpful in more specialized domains where there is not enough data to train a generative model from scratch. dealing with medical data brings one particular issue. the generated data should not contain sensitive information from the data it is inspired by while remaining as close as possible to it. this means that the evaluation of this task is different from the usual evaluation of text generation. the evaluation of text generation is a difficult problem @xcite . there is a distinction between intrinsic and extrinsic evaluation @xcite . for intrinsic evaluation in generation tasks such as translation and summarization, the quality is usually first automatically measured using metrics such as bleu @xcite , rouge @xcite , or bertscore @xcite . those metrics and many others compare the system's output to one or several human-written references @xcite . however, a gold reference does not exist for an open-generation task. human evaluation is usually the best way of intrinsically evaluating text generation. still, there can be several problems. first, human evaluation should be conducted once the system's output can be considered "good enough" to avoid wasting human evaluation costs. this means there is a need to measure if the system is ready to be manually evaluated. second, it might not be possible for a human to assess the quality of generated data at a larger scale, if the generated data is not considered as a set of independent sentences but as a whole self-sufficient corpus. this is why extrinsic evaluation of the generated data is also important, meaning evaluating the usefulness of the generated data @xcite . however, to our knowledge, methodologies to extrinsically evaluate open text generation are rare. in this work, we explore text generation using a pre-trained language model and we focus our evaluation on named entity recognition (ner). this paper offers the following contributions: ## related work generating data to address resource sparsity is an active field of research. pre-trained transformerbased models have been used as data augmentation tools, such as in @xcite , where the authors fine-tuned the pre-trained auto-regressive english language model gpt-2 to create labeled examples for training classification models in french and english. in the medical domain, amin-nejad et al. (2020) use an encoder-decoder transformer architecture to generate discharge summaries given a context. the context gives a target to the generation that is used to evaluate the generated texts. @xcite train an lstm-based language model to generate shareable clinical notes using differential privacy @xcite . word embeddings are trained on the generated data and compared to word embeddings trained on the original data on two existing benchmarks. @xcite used an entity extraction method to build a generative model with entities as input and the document as output and generated artificial mental health records in english. several efforts exploit generative adversarial networks (gans) @xcite to seamlessly augment original data. among them, @xcite use gans to generate artificial structured data in the medical domain. @xcite created medgan, a framework to generate artificial structured electronic health records (ehr) with discrete values. @xcite rely on a combination of autoencoders @xcite and gans to generate artificial structured data with both discrete and continuous values. ## results tags llf bloom no il s'agit d'un patient âgé de 52 ans, sans antécédents pathologiques notables. admis aux urgences pour douleurs abdominales diffuses associées à des vomissements postprandiaux évoluant depuis deux semaines en rapport avec une obstruction du canal cystique dont l'examen clinique a objectivé un ictère cutanéo-muqueux généralisé associé à quelques signes inflammatoires non spécifiques (extériorrhée et urétéro-hydronéphrose). l'examen ophtalmologique n'a révélé aucun autre signe oculaire ou ostéo-nerveux associés. il s'agissait d'une patiente âgée de 55 ans ayant comme antécédents une polyarthrite chronique touchant toutes les articulations métacarotidiennes évoluant depuis plus de 10 ans associée à une insuffisance surrénalienne fonctionnelle qui s'est aggravée après l'installation d'un diabète insulino-résistant non insulinodépendant nécessitant son hospitalisation six mois avant sa consultation chez notre confrère ophtalmologue afin d'être mis sous traitement médicamenteux associant : imipramine 250 mg / jour ; dexaméthasone 150 mg/jour; corticostéroïde prednisolone 5mg/kg/24h) yes une femme âgée de 55 ans, sans antécédents pathologiques particuliers était hospitalisée pour une dyspnée d'effort évoluant depuis 6 jours. l' examen clinique a montré un état général altéré avec des signes inflammatoires du membre supérieur gauche ainsi qu'un syndrome grippal (une fièvre à 39 °c) et le reste de l' examen neurologique était sans particularités. la radiologie standard n'a objectivé ni troubles sensitivo-moteur ni convulsions fébriles. il s'agissait d'un patient de 50 ans, sans antécédents pathologiques particuliers, admis aux urgences pour des douleurs épigastriques aiguës associées à une distension abdominopelvienne évoluant depuis deux jours. l' examen clinique trouvait un patient en assez bon état général (apgar: 10/10). le bilan préopératoire objectivait une fonction rénale normale et la crp était à 12 mg/l. ## conclusion we presented four models for generating synthetic clinical cases in french. our experiments suggest that synthetic text is useful for training ner models in the clinical domain. while text generated with bloom e3c+t seems more natural, text gener-