Handling Name Errors of a BERT -Based De-Identification System: Insights from Stratified Sampling and M arkov-based Pseudonymization (2024)

Abstract: Missed recognition of named entities while de-identifying clinical narratives poses a critical challenge in protecting patient-sensitive health information. Mitigating name recognition errors is essential to minimize risk of patient re-identification. In this paper, we emphasize the need for stratified sampling and enhanced contextual considerations concerning Name Tokens using a fine-tuned Longformer BERT model for clinical text de-identifcation. We introduce a Hidden in Plain Sight (HIPS) Markov-based replacement technique for names to mask name recognition misses, revealing a significant reduction in name leakage rates. Our experimental results underscore the impact on addressing name recognition challenges in BERT-based de-identification systems for heightened privacy protection in electronic health records.

## introduction clinical narratives and unstructured documentation within electronic health records (ehrs) are considered valuable assets in epidemiological research @xcite and the creation of prognostic clinical prediction models @xcite . the advancement of these applications is frequently impeded by the limited availability of de-identified clinical text corpora. clinical notes must remove protected health information (phi) to safeguard patient privacy and align with privacy regulations exemplified by the united states' hipaa safe harbor privacy guidelines @xcite . bidirectional encoder representations from transformers (bert) models have shown promise in automatically identifying sensitive phi in clinical texts @xcite . previous research has explored bert variants with hyper-parameter tuning, comparing their efficacy in clinical text de-identification across phi subcategories, including dates, phone numbers and names @xcite . less focus has been given to the shrinking but persistent margins of error that plague even the best-performing deidentification pipelines. for instance, our base deidentification model uses a longformer bert variant @xcite fine-tuned using discharge summaries from a us-based tertiary healthcare institution. in preliminary work as a part of @xcite , we discovered that the longformer model performed exceptionally well during preliminary testing among the pretrained models we sampled. our base model achieved an impressive but far-from-optimal overall f1 score of 0.90 for names ( this study aims to contribute to characterizing name errors in bert-based transformer models on real-world discharge summaries. the goal is to provide insights into the development of comprehensive de-identification strategies capable of both correcting bias and tolerating mistakes related to names. we first compare the effect on recognizing names using a fine-tuned longformer model with a stratified sample that uses demographic information against a model fine-tuned with a standard randomized sample. we suggest that enhancing the recognition of name tokens in our de-identification system is achieved by including sets of name tokens found in stratified training samples compared to those in random training samples. next, we investigate a hidden in plain sight (hips) markovbased replacement technique for names. using real-world discharge summaries from an academic healthcare institution, we compare the effectiveness of a markov-based replacement strategy against a random replacement strategy in reducing name leakage. ## discussion while previous research has emphasized the effectiveness of strategies such as data augmentation for improving name diversity and contextual patterns (yue and zhou, 2020), we focused on enhancing name performance through stratification. we aimed to increase the proportion of urm patients in the train dataset by using a stratified sampling approach. surprisingly, this approach did not significantly improve the name performance of our de-identification system. this lack of improvement could be attributed to a relatively small sample size, where the change in urm representation was not substantial enough to impact name performance in a predominately white patient population. additionally, higher urm representation in the training sample through stratification may not perfectly correlate with a more robust set of name tokens. urm patients may have names more commonly associated with white demographics and white patients may also have rare or unique names. in our discharge summaries, there is a single patient with two notes. to prevent name token leakage, we deliberately included both notes from this patient exclusively in the training sets. this ensures that the names associated with this particular patient do not leak between the training and test sets through their notes. however, this method does not address the potential leakage of provider names between training and test sets, as a provider's name may appear in notes from multiple patients. additionally, our stratification rule for creating the white and urm strata does not consider the demographics of the providers. consequently, not only may provider names leak between the training and test sets, but the set of provider name entities could also remain similar between the stratified and random training samples, posing a challenge to the effectiveness of our methodology." the higher performance for provider names in future work could expand on efforts to explore how shifts in training data effect embedding or attention mechanisms @xcite to better understand causal mechanisms for targeting name distributions as a parameter to improving de-identification system performance on names. nevertheless, achieving complete elimination of name errors through incremental model improvements is unlikely, especially as the model is applied to different note types or health system contexts. in such scenarios, our markov-based hips strategy holds theoretical efficacy in diminishing documentlevel leakage associated with false negatives related to name tokens. however, missed names that ap-pear more than the allowable repeats in the markov method remain at risk for phi leakage. our results propose that implementing such a strategy would be beneficial for bert-based de-identification systems, effectively masking name errors and providing enhanced privacy protection for patients. ## conclusion our study aims to address errors in name recognition of a longformer-based de-identification model fine-tuned on discharge summaries. stratified sampling for fine-tuning did not significantly improve name recognition. however, the introduction of a hips markov-based pseudonymization strategy showed promising results, significantly reducing name leakage rates compared to random replacement. this research contributes to the ongoing efforts to address the persistent challenges associated with de-identification of clinical texts, offering valuable guidance for the development of robust and privacy-conscious clinical text deidentification.