INCOGNITUS : A Toolbox for Automated Clinical Notes Anonymization (2023)

Abstract: Automated text anonymization is a classical problem in Natural Language Processing (NLP). The topic has evolved immensely throughout the years, with the first list-search and rule-based solutions evolving to statistical modeling approaches and later to advanced systems that rely on powerful state-of-the-art language models. Even so, these solutions fail to be widely implemented in the most privacy-demanding areas of activity, such as healthcare; none of them is perfect, and most can not guarantee rigorous anonymization. This paper presents INCOGNITUS, a flexible platform for the automated anonymization of clinical notes that offers the possibility of applying different techniques. The available tools include an underexplored yet promising method that guarantees 100% recall by replacing each word with a semantically identical one. In addition, the presented framework incorporates a performance evaluation module to compute a novel metric for information loss assessment in real-time.

## introduction the digitization of medical textual data has allowed for a whole new range of possibilities, such as the development of tools for the summarization of clinical notes or the automated identification of the international classification of diseases (icd) codes in clinical text. however, clinical data contains sensitive information regarding both patients and health professionals. these entities are protected by the general data protection regulation (gdpr), which grants equal data protection rights to all european union (eu) citizens @xcite . therefore, information systems must be compliant to maintain such information private. this poses a challenge when considering the publication of clinical data for secondary usage. the gdpr (gdpr, 2018) defines anonymization as the process through which "personal data (is) rendered anonymous in such a manner that the data subject is not or no longer identifiable" (recital 26). however, achieving true anonymization is not a simple task, especially when considering unstructured data such as clinical notes. in fact, despite the fact that many literature works propose strategies for the automated anonymization of clinical text, their implementation in real contexts is still scarce. consequently, the problematic access to clinical text data for secondary usage remains a barrier to scientific research. this demonstration paper presents the incog-nitus platform for automated clinical notes anonymization. the beta version is available online at https://tospe-incognitusfhp. streamlit.app/ . an overview video can be watched at https://www.youtube.com/watch? @xmath0. besides offering a wide range of methods to perform anonymization tasks, incog-nitus was designed to address two common flaws of similar systems: (i) the inability to provide truly anonymized outputs (100% sensitive information removal) @xcite and (ii) the lack of an assessment on the quantity of relevant information that gets lost in the anonymization process @xcite . the remainder of the manuscript is organized as follows. the next section contextualizes text anonymization literature, mainly focusing on the clinical domain. then, the platform proposed is described, along with its composing methods and models. the experimental setup followed to develop each component is detailed in section 4. the results associated with these experiments are then discussed in section 5. in the last section, the main conclusions of this work are presented. ## background the list of literature publications focused on clinical text anonymization is extensive. this section presents a chronological overview of the developments achieved in this area of research, culminating in a final discussion on the main findings regarding the strengths and flaws of current techniques and future research tendencies. ## incognitus framework incognitus is a flexible and intuitive toolbox aimed at transparent and accountable automated text anonymization. this framework offers the possibility to employ different techniques while providing performance measures relating to the anonymization task itself and the resultant loss of information. all this content is displayed and accessible through an interactive, user-friendly interface, which can be seen in figure 1 . with these insights, the user can consciously select the adequate approach to anonymization, considering the specifications of each application context. the flow of information circulating through incognitus is represented in the diagram of figure 2 . first, the user uploads or writes the content of a clinical note. the anonymization may then be performed through any of the available techniques. as the anonymization is completed, an icd-10 classification model runs in the background, and its outputs are leveraged to estimate the ratio of information that got lost in the anonymization process. by the end of the calculation, this measure is displayed in the user interface, as well as the values of the standard performance metrics (recall, precision, and f1-score) associated with the technique selected. the user can then choose to (i) download the anonymized content or (ii) select another technique and repeat the process. ## results to estimate the performance of the offered strategies in the anonymization task, they were tested against the working test sets. for each experiment, the f1-score regarding the binary task of classifying sensitive information was calculated. the mean il associated with anonymizing each clinical note was also measured, based on equation 1 by looking at the f1-score values in furthermore, while f1-scores of 95% are high, these values can never comply with the gdpr definition of anonymization. even if these rose to 100%, the underlying risk of existing particular identifiers not being considered during training is still a threat. in this regard, the traditional methods of automated anonymization (ner-based) cannot compete with the kneo strategy, since the latter replaces every single token from the original text, guaranteeing no such occurrences. since every original token is exchanged, the f1-score calculation becomes inappropriate for the kneo methodology. on the one hand, by replacing every token, it is guaranteed that none of the original content gets overlooked, ensuring 100% recall. on the other hand, since the replacement of every original word is the fundamental ideology behind this method, the concept of false positives makes little sense. one could argue that every non-sensitive token replaced might be regarded as a false positive. still, such an interpretation ignores the role of neighbor word embeddings in preserving the information encoded. even so, although kneo outperforms both ner-based strategies in preventing sensitive information leaks, the quantity of relevant information lost in the process is undeniably higher than what is observed for the other methods. looking at the information loss values, this tendency is clear: around 60% of the original content is lost when applying the kneo strategy. this means that, on average, the obfuscator hides the information related to six of the ten codes identified in the original notes. this becomes even more concerning if we consider that the classification task used to compute the il metric simplifies a much more complex one: identifying individual icd10 codes. therefore, applying this strategy before performing advanced text processing tasks, such as clinical note summarization, could be problematic. the rates of information lost for the ner-based strategies are much lower, although it is visible that some information also gets hidden (around 20%). in sum, since no technique is flawless, it is fundamental to understand (i) the context of application and (ii) the pros and cons of applying each technique. incognitus answers to the second necessity by providing various solutions and estimating performance during anonymization. in this way, it allows the users to switch between strategies according to their needs and the characteristics of each method. ## conclusions this paper introduced incognitus, a userfriendly platform to prompt conscious automated anonymization of clinical text. it provides two distinct ner-based methods and a recently proposed alternative that guarantees proper anonymization (100% recall) at the cost of information loss and low readability. employing a pre-trained classifier of icd-10 code categories, incognitus brings a new way of estimating the information lost during anonymization. this framework moves the research around clinical text anonymization forward, towards accountable strategies and fair performance assessments.