The Impact of De-identification on Downstream Named Entity Recognition in Clinical Text (2020)

Abstract: The impact of de-identification on data quality and, in particular, utility for developing models for downstream tasks has been more thoroughly studied for structured data than for unstructured text. While previous studies indicate that text de-identification has a limited impact on models for downstream tasks, it remains unclear what the impact is with various levels and forms of de-identification, in particular concerning the trade-off between precision and recall. In this paper, the impact of de-identification is studied on downstream named entity recognition in Swedish clinical text. The results indicate that de-identification models with moderate to high precision lead to similar downstream performance, while low precision has a substantial negative impact. Furthermore, different strategies for concealing sensitive information affect performance to different degrees, ranging from pseudonymisation having a low impact to the removal of entire sentences with sensitive information having a high impact. This study indicates that it is possible to increase the recall of models for identifying sensitive information without negatively affecting the use of de-identified text data for training models for clinical named entity recognition; however, there is ultimately a trade-off between the level of de-identification and the subsequent utility of the data.

## introduction there is a growing demand for access to large amounts of healthcare data in order to facilitate research and development of tools for healthcare management and clinical decision support, not least as a result of the increasing application of ai and machine learning in healthcare. however, to enable large-scale secondary use of sensitive healthcare data, there is a need for automatic privacyprotecting methods for clinical text; manual deidentification to ensure that the data does not con-tain personal information is often prohibitively expensive. privacy-protecting methods for de-identification of data generally address three privacy risks: (i) the risk that someone's records can be uniquely identified in a dataset, (ii) preventing linkage from one dataset to another, and (iii) the risk of inferring sensitive information about an individual from the dataset @xcite . de-identification of unstructured clinical text commonly focuses on identifying potentially identifiable information within predefined classes in an approach based on named entity recognition (ner), where protected health information (phi) is first identified and subsequently obscured in some fashion @xcite . this technique addresses the first of the mentioned privacy risks, and is the focus of this study. de-identification techniques for structured data generally address all three mentioned privacy risks, and their impact on data quality and data utility are more thoroughly studied than the impact of de-identification on unstructured text @xcite . research on structured data shows that de-identification techniques applied to structured data may lead to reduced data quality @xcite . for both structured and unstructured data, information necessary for answering a research question may be identifying, and therefore required to be removed or altered to ensure the privacy of data subjects. furthermore, for automatic de-identification, relevant information may be changed or removed unintentionally due to non-sensitive tokens being misclassified as sensitive. studies have indicated that de-identification may erroneously alter or hide data significant for other tasks; however, no significant impact has so far been observed when comparing models for downstream tasks trained with original versus deidentified text data @xcite @xcite . it is, however, still not clear how the impact is related to various forms and levels of de-identification, in particular the trade-off between the precision and recall of the de-identification system. common measures for evaluating deidentification models for unstructured data are precision, recall and f 1 -score. high recall is generally preferred over high precision, as the privacy of the data subjects is prioritised over potential loss of document interpretability @xcite . there is, at the same time, a concern that poor precision would have a negative impact on the quality of the data. in practice, precision and recall are typically evaluated as equally important when using f 1 -score as the primary evaluation metric. however, if precision of the de-identification model is of less importance and it turns out that this does not have a clear impact on using the data for some downstream task, it would entail that de-identification systems can be adapted for high coverage -but also that recall should carry more weight than precision when evaluating de-identification systems. on the other hand, if de-identification impacts the text data quality negatively, the concern is that it would lower the possibility to use the data for building models for various downstream tasks, i.e. data utility would decrease once it has been deidentified. examples of downstream tasks are detection of healthcare-associated infections, adverse drug events and early cancer symptoms. this study intends to answer the question of how the precision of a de-identification system affects data quality and, in particular, data utility by investigating how different levels of de-identification affects the performance of downstream clinical ner. in particular, the study aims to investigate the impact of the trade-off between precision and recall, as well as different methods for concealing phis. ## related research to our knowledge, no studies have specifically focused on the trade-off between precision and recall and its impact on downstream tasks. there are, however, studies that have investigated the impact of de-identification. a study by @xcite showed that deidentification reduces the information content, and leads to the possible introduction of misleading information if tokens are replaced with pseudonyms. in the study, only 0.81% of clinical named entity annotations were erroneously detected as phi, but between 10 and 49% of all eponyms foot_0 were misclassified as phi. fewer snomed-ct concepts were also found, but this was largely explained by incorrectly labelled snomed-ct concepts in the original dataset. studies on downstream tasks have, however, not shown that de-identification has a negative impact. no significant differences could be seen between using original text or de-identified text as training data for clinical text classification @xcite . in contrast, one study observed potentially significant benefits of training on de-identified data for medication name extraction @xcite . the reduction of dimension in de-identified text has been hypothesised to potentially improve machine learning performance @xcite . while previous studies have compared different systems with various levels of precision @xcite , no study has explicitly compared the impact the precision of a deidentification system has on downstream tasks. in @xcite , a recall bias was introduced in two versions and those models were compared to the original version. the systems with a slightly lower precision performed similarly to the original one in terms of performance on a medication name extraction task. ## methods and materials in this paper, the impact of various forms of deidentification -where a trade-off is made between precision and recall of the model used for identifying phi, as well as using different levels of de-identification -on downstream ner tasks, in this case identifying various clinical entity types, is studied. this paper uses four corpora: one for development of de-identification models and three for evaluating the impact of de-identification on downstream clinical ner. the method is presented first, followed by a description of the corpora. the experiments include five models for identifying phi, each one trading off precision for increased recall to different extents, as well a four concealment strategies that hide the identified phi to different degrees. these phi models and concealment strategies are used to de-identify three clinical corpora 2 , which, in turn, are used for training downstream clinical ner models. the impact of using different phi models and concealment strategies is then analysed from a number of different perspectives. ## results the results for phi identification cross-validated on the phi corpus are first presented to provide an estimate of the performance of each phi model. the impact of de-identification on the three clinical ner corpora used for the downstream tasks are then analysed in three different ways: (i) the overlap between predicted phi and manually annotated clinical entities, (ii) the co-occurrence of predicted phi and manually annotated clinical entities in the same sentences, and (iii) the impact on downstream clinical ner performance. finally, the misclassification of eponyms as phi is specifically studied. ## discussion in contrast to the findings by both @xcite , a small negative impact was indeed observed regardless of the precision and recall of the phi model used for deidentification. however, the impact is very small unless the f 40 phi model or the remove concealment strategy is used. the worst performing model in @xcite had a token-based binary precision of 76% (and a recall of 52%). the only models with an estimated precision above this in our study is f 1 and f 4 . despite the clear difference in precision between f 1 and f 10 , f 10 gives a high recall 99.5% for phi identification, see ## conclusion this study demonstrates that corpora de-identified using phi models with a moderate to high precision lead to similar performance when used for downstream clinical ner tasks. the impact is, however, affected by both the choice of concealment strategy and the trade-off between precision and recall, in particular when the precision is low. optimising the phi identification model for f 4 gives a relatively lower precision of 77.82% and a higher recall of 97.55%. compared to using a standard f 1 -optimised model for de-identification, this results in higher privacy and a relatively small negative impact on downstream clinical named entity recognition. this study indicates that it is possible to increase the recall of models for identifying sensitive information without negatively affecting the use of de-identified text data for training models for clinical named entity recognition furthermore the overlap analysis showed that with lower precision, there is an increase of overlap between clinical information and automatically labelled phi. some phi labels overlap with clinical entities more than others. different clinical entities are also more likely to be affected than others, like drug. eponyms may also risk being misclassified as last names, and studies interested in those may need to take extra precautions to handle those properly.