Revisiting De-Identification of Electronic Medical Records: Evaluation of Within- and Cross-Hospital Generalization (2023)

Abstract: The de-identification task aims to detect and remove the protected health information from electronic medical records (EMRs). Previous studies generally focus on the within-hospital setting and achieve great successes, while the cross-hospital setting has been overlooked. This study introduces a new de-identification dataset comprising EMRs from three hospitals in China, creating a benchmark for evaluating both within- and cross-hospital generalization. We find significant domain discrepancy between hospitals. A model with almost perfect within-hospital performance struggles when transferred across hospitals. Further experiments show that pretrained language models and some domain generalization methods can alleviate this problem. We believe that our data and findings will encourage investigations on the generalization of medical NLP models.

## introduction de-identification is a natural language processing (nlp) task to detect and remove the protected health information (phi) from electronic medical records (emrs). it is a prerequisite to the distribution of emrs outside their original institutions for medical nlp research @xcite . previous studies generally focus on the withinhospital setting, where the training and test data are from a same hospital or institution. this includes english tasks like 2006 i2b2 @xcite , 2014 i2b2/uthealth @xcite , 2016 cegs n-grid @xcite , and others in swedish @xcite and french @xcite . deidentification may be regarded as an easy task relative to other nlp tasks, because simple rule-based or shallow neural models can achieve 95%+ f1 scores @xcite . however, the cross-hospital setting has been largely overlooked.this setting corresponds to a realistic scenario that a de-identification model is deployed across hospitals. this study aims to formally fill the gap. we introduce a new deidentification dataset comprising emrs from three hospitals in china, establishing a benchmark for evaluating both within-and cross-hospital generalization. the latter poses a challenging domain generalization (dg) task, where hospitals are referred as domains. we find significant domain discrepancy existing between hospitals. a model with almost perfect within-hospital performance encounters dramatic degradation when transferred to other hospitals. using pretrained language models (plms) like bert @xcite , or some existing dg methods @xcite ) can improve the cross-hospital generalization, but by limited margins. this paper contributes in twofold: in addition, the hm database provides the largescale clinical text for pretraining word2vec embeddings and bert, named hm-word2vec and hm-bert, respectively. this mitigates the potential domain gap of transferring word embeddings or plms trained on common corpora to clinical text. see section 3 for more details. ## models and experimental settings model architecture. figure 1 displays the architecture of our model. we employ the neural sequence tagging framework @xcite , a widely-used and mature solution for the de-identification task. specifically, the tokens are first mapped to embeddings and fed into the encoder. the encoder can be a 1d cnn @xcite , a bil-stm @xcite , or a pretrained transformer @xcite , which transforms the embeddings to hidden representations. finally, the dense layer classifies the representations into the pre-defined bio tag space, and the resulting bio tags can be parsed to identify the boundaries and categories of phi mentions. an optional conditional random field (crf) layer can be inserted after the dense layer, which may improve the consistency of the predicted bio tags. pretraining on emrs. most publicly available chinese word embeddings or plms are trained on common corpora, which may encounter domain gaps and result in sub-optimal performance when transferred to the clinical domain. to alleviate this issue, we pretrain word2vec embeddings @xcite and a chinese bert @xcite on the large-scale clinical text from the hm database. the resulting embeddings and model are named hm-word2vec and hm-bert, respectively. after data parsing, cleaning and deduplication, the remaining hm corpus consists of 21.4k emrs, including clinical text of 2.8 gb (1.1b tokens). hm-word2vec has a character-level vocabulary of size 5.7k and embedding size of 100. it is trained for 5 epochs by the gensim ( řehůřek and sojka, 2010) package with window size 15 and learning rate 1e-3. following @xcite , we initialize hm-bert from the bert-base-chinese checkpoint released by hugging face. 5 we then pretrain the model with the optimizer adamw @xcite , learning rate 1e-4 and batch size 384 for 20 epochs on the whole corpus. a scheduler of linear warmup in the first 20% steps followed by linear decay is applied. the masking rate is 15% for the masked language modeling (mlm) task; the maximum input length is 512. in addition, we apply the whole word masking @xcite and dynamic masking @xcite strategies. within-and cross-hospital evaluation. we train the models on the training set of hm, and use the development set for hyperparameter tuning. we perform the within-hospital evaluation, i.e., evaluating the trained models on the test set of hm; and the cross-hospital evaluation, i.e., evaluating the models on sy and ccks. hyperparameters. 6 for cnn or bilstm models, the embedding layer is 100-dimensional, and optionally initialized from hm-word2vec; the encoders have one layer with 200 hidden states and dropout rate of 0.5. the kernel size is 15 for cnn. the models are trained for 100 epochs by the adamw @xcite optimizer with learning rate 1e-3 and batch size 32. for models with plms, we use bert-wwm @xcite , mc-bert @xcite and hm-bert, all of a base size (768 hidden size, 12 layers). the mc-bert is pre-trained on chinese medical corpora including biomedical question answering, medical encyclopedia and emrs. the models are trained for 100 epochs by the adamw optimizer with learning rate 2e-5 and batch size 32. evaluation. a predicted phi mention is considered correct if its boundaries and category exactly match the ground truth. the evaluation metrics are micro precision rate, recall rate and f1 score on the test sets. all the experiments are repeated for five times and the average metrics are reported. ## conclusion in this paper, we revisit the emr de-identification task and create a new dataset. it consists of emrs from three hospitals and thus asks evaluation for both within-and cross-hospital generalization. the latter poses a challenging dg task, which corresponds to a realistic scenario that a de-identification model is required to deploy across hospitals. with this new benchmark, we find significant domain shift between hospitals. a model with perfect within-hospital performance struggles when transferred across hospitals. using plms or some existing dg methods can alleviate but not address this problem.