Generalizing Clinical De-identification Models by Privacy-safe Data Augmentation using GPT -4 (2024)

Abstract: De-identification (de-ID) refers to removing the association between a set of identifying data and the data subject. In clinical data management, the de-ID of Protected Health Information (PHI) is critical for patient confidentiality. However, state-of-the-art de-ID models show poor generalization on a new dataset. This is due to the difficulty of retaining training corpora. Additionally, labeling standards and the formats of patient records vary across different institutions. Our study addresses these issues by exploiting GPT-4 for data augmentation through one-shot and zero-shot prompts. Our approach effectively circumvents the problem of PHI leakage, ensuring privacy by redacting PHI before processing. To evaluate the effectiveness of our proposal, we conduct cross-dataset testing. The experimental result demonstrates significant improvements across three types of F1 scores.

## introduction in the realm of healthcare data management, the protection of sensitive information embedded within medical records is essential. protected health information (phi) encompassing identifiers, such as names, addresses, and contacts, requires safeguards to prevent unauthorized access and ensure patient confidentiality. the u.s. health insurance portability and accountability act (hipaa) lists a comprehensive set of 18 categories of phi that must be de-identified in electronic health records (ehrs) for secondary use. the advent of automated systems for the deidentification (de-id) of phi within medical data has been facilitated by advances in rule-based methods, machine learning, and deep learning models @xcite . for deep-learning-based approaches, a de-id task is basically a sequence tagging task of named entity recognition (ner). currently, i2b2 2006 @xcite figure 1 : performance degradation in cross-dataset settings using bio-clinicalbert @xcite . binary, token, and entity refers to binary token-level f1, token-level micro f1, and entity-level micro f1. they are types of f1 scores used to evaluate sequence tagging of de-id models. see section 4.3 for the details regarding these metrics. and i2b2 2014 de-id challenge datasets @xcite are widelyused for training de-id models. these datasets contain clinical notes with phi replaced with surrogate phi for privacy and annotated with multi-classes of phi labels, such as patient, doctor, etc. lstm-based models @xcite and bert-based models @xcite have achieved micro f1 scores exceeding 0.97, demonstrating the effectiveness of deep-learningbased models to address the critical task of phi de-id in medical records. however, de-id tasks present unique challenges compared to other ner tasks, primarily due to the difficulty in accessing suitable datasets. moreover, making a publicly available dataset requires expertise and efforts to anonymize and annotate phi @xcite . this leads to poor generalization of deep-learning-based models. while the i2b2 2006 and i2b2 2014 corpora offer clinical de-id datasets both annotating phi with surrogate phi, the nuanced differences in annotation standards and the variability of institutes (e.g., hospitals) across datasets can significantly impact cross-dataset test performance in cross-institute settings @xcite . figure 1 shows the performance decrease in three types of f1 scores in a cross-dataset setting when training on the i2b2 2006 dataset and testing on the i2b2 2014 dataset, and vice versa. data augmentation can be a solution to the problem of acquiring sensitive data and enhancing generalization, and some studies have explored this approach. @xcite propose rule-based ehr data augmentation with phi replacement and context augmentation (i.e., synonym replacement). they improve token-level micro f1 on a cross-dataset setting of i2b2 datasets. however, rule-based augmentation lacks context diversity. this paper proposes a data augmentation method using gpt-4 @xcite that can enrich the context of augmented datasets while ensuring privacy. as illustrated in figure 2 , our framework preemptively conducts phi-scrubbing (i.e., replacing phi with placeholders like "john" → "[pa-tient]," and "16th of november" → "[date]") before providing samples to gpt-4. this can effectively circumvent the problem of phi leakage through apis. our method includes zero-shot and one-shot prompt engineering for generating synthetic de-id clinical corpus with i2b2 2006 and 2014 datasets. zero-shot generation does not require any preobtained dataset. on the other hand, one-shot augmentation requires a pre-obtained dataset. with the one-shot augmented dataset, we conduct a crossdataset test on i2b2 2006 and i2b2 2014 and achieve performance boosts up to 6.48%, 29.83%, and 36.50% across three types of f1 scores: binary token-level f1, token-level micro f1, and entitylevel f1, respectively. overall, the contributions of this paper are summarized as follows: ## related work this section briefly describes the related work to this paper. ## proposed methods this section outlines our approach for enhancing de-id sequence tagging datasets, such as i2b2 2006 and i2b2 2014, at the document level. we employ both one-shot and zero-shot prompting techniques with gpt-4-turbo (openai, 2023b) via the openai api. the distinction between zero-shot and oneshot prompts is based on their use of the original datasets; one-shot prompts incorporate sample data from these datasets to guide the model, whereas zero-shot prompts do not. one-shot is a type of few-shot that uses only one example to guide the model's response. in our case, we use the term one-shot as we provide one example (a clinical record) for each prompt. figure 2 illustrates the overall pipeline of our one-shot augmentation. excluding "i2b2 2006" in the figure also demonstrates the pipeline for our zero-shot augmentation. our methodology uses the advanced nlp capabilities of gpt-4-turbo to augment the datasets, potentially improving the performance of models when finetuned on them for de-id tasks. ## experimental results this section provides the experimental results for the proposed method to boost the generalization capability of the de-id models. ## conclusion this paper explores and analyzes the potential of one-shot and zero-shot data augmentation for deidentification in cross-dataset testing and extends the result to models pretrained with medical data. the proposed methods leverage gpt-4 to significantly enhance data scalability and diversity, which is particularly vital in healthcare domains where publicly available data is often limited. moreover, the proposed methods of employing synthetic data generation with phi-scrubbed datasets offer a strategic solution to addressing privacy concerns for gpt-4 apis, aligning with hipaa regulations. overall, our methodology not only enriches the dataset quality for more effective model training but also aligns with ethical standards and privacy regulations, highlighting its potential to transform the data preparation process in sensitive domains.