Creating and Evaluating a Synthetic N orwegian Clinical Corpus for De-Identification (2021)

Abstract: Building tools to remove sensitive information such as personal names, addresses, and telephone numbers - so called Protected Health Information (PHI) - from clinical free text is an important task to make clinical texts available for research. These de-identification tools must be assessed regarding their quality in the form of the measurements precision and re- call. To assess such tools, gold standards - annotated clinical text - must be available. Such gold standards exist for larger languages. For Norwegian, how- ever, there are no such resources. Therefore, an already existing Norwegian synthetic clinical corpus, NorSynthClinical, has been extended with PHIs and annotated by two annotators, obtaining an inter-annotator agreement of 0.94 F1-measure. In total, the corpus has 409 annotated PHI instances and is called NorSynthClinical PHI. A de-identification hybrid tool (machine learning and rule-based meth- ods) for Norwegian was developed and trained with open available resources, and obtained an overall F1-measure of 0.73 and a recall of 0.62, when tested using NorSynthClinical PHI. NorSynthClinical PHI is made open and available at Github to be used by the research community.

## introduction the data contained within electronic health records (ehrs) are of significant value to medical researchers and for administrative purposes, but privacy and patient confidentiality legislation restricts access. however, de-identification of such data -removing the protected health information (phi) within -allows it to be shared between researchers @xcite . this process can be done manually; however, manual deidentification has proven to be inefficient with regards to cost, time and quality @xcite . tools for automatic de-identification of clinical data have been studied extensively. however, most of the published research is concerned with structured records and not clinical free-text, and few de-identification tools are made publicly available @xcite . furthermore, most research focus on english and other languages with many native speakers. despite the fact that the norwegian language has comparatively few native speakers foot_0 , hospitals and organisations like the cancer registry of norway are in possession of comprehensive collections of clinical data. enabling research on this valuable and unique information could reveal new discoveries and would be of great importance for the future health care. to ensure that de-identification applications can successfully de-identify clinical texts, they must be evaluated in a quantitative manner @xcite . for this purpose, verified, annotated corpora are used to test and score the applications @xcite . these corpora are referred to as gold standards (or reference standards), and are typically made by domain experts or linguists -following specific guidelines. a gold standard does not need to contain real phi, and it can be developed using synthetic data. consequently, a gold standard developed with synthetic data can be made publicly available. this study describes the efforts of creating and evaluating the first publicly available gold standard for de-identification of norwegian bokmål 2 clinical text, describing and discussing the devel-opment and evaluation of the gold standard. the lack of publicly available clinical text in norwegian places limitations on the development of gold standards and tools for de-identification of norwegian clinical text. recently, there have been developments of open datasets for named entity recognition (ner) of the norwegian language, most notably norsynthclinical @xcite and norne @xcite . norsynthclinical is a small dataset of synthetic clinical text, focusing on family history information (further described in section 3) @xcite . while the development of norne resulted in a sizeable dataset with approximately 300,000 tokens for each written variant of norwegian and a rich entity set, most phi entity types are missing @xcite . only a few attempts aiming at developing deidentification tools focusing on the norwegian language have previously been made. one of these was conducted by @xcite . they tackled de-identification of norwegian free text clinical notes for their master's thesis project, employing a combination of pattern recognition and simplistic statistical methods, reporting an f 1measure of 0.72. furthermore, they developed a reference in order to evaluate their developed tool, consisting of 225 records manually annotated and de-identified. it was, however, not evaluated further or made publicly available @xcite . as previously mentioned, most of the existing tools and gold standards for de-identification of clinical text are written in, and for, the english language @xcite . one of the most well-known gold standards is the multiparameter intelligent monitoring in intensive care (mimic ii) corpus @xcite . in sweden, the development of both deidentification tools and gold standards has come further than in norway. in 2008, a group of swedish researchers developed a gold standard corpus for de-identification of swedish clinical text @xcite . the researchers manually annotated and de-identified 100 electronic patient records (eprs) deriving from five different clinics (neurology, orthopaedia, infection, dental surgery and nutrition) at karolinska university hospital. the gold standard consists of unstructured text (around 174,000 tokens in total) and is known as the stockholm epr phi corpus. it has 4,700 annotated instances distributed over 8 phiclasses. it has been further developed to stockholm epr phi pseudo corpus, which contains only surrogate names, addresses, phone numbers, etc., and is partly available for research @xcite . ## method the development of the gold standard involved two main steps: extension and annotation. the gold standard was evaluated by measuring the inter-annotator agreement (iaa) and by testing it on a hybrid de-identification tool. ## analysis 6.1 the norsynthclinical phi corpus the amount of phi in the extended corpus, norsynthclinical phi, constitutes around 5% of ## discussion and conclusion what makes the norsynthclinical phi special and valuable is the fact that it is synthetic. as it does not contain any real personal information, the gold standard can be accessed by anyone and utilized in the development of tools for deidentification of norwegian clinical text. hopefully, this will facilitate more research on the content of clinical notes, and eventually a better health care. the major weakness of the created gold standard is its small size. the english corpus mimic ii consists of 412,509 clinical notes and the stockholm epr phi corpus consists of 100 patient records @xcite . as mentioned in @xcite , the latter contributes 174,000 tokens. in comparison, the norsynthclinical phi, which consists of 8,270 tokens, is very small. besides, it is very specific to the area of cardiology, written by one cardiologist, and extended by a layman. therefore, there might be a lack of linguistic variety. furthermore, the gold standard is written in norwegian bokmål and not in nynorsk. however, it would be relatively uncomplicated to translate the gold standard from bokmål to nynorsk. the de-identification tool used for evaluating norsynthclinical phi corpus was initially designed for another purpose 13 and trained on publicly available data. the effect of fundamental incompatibilities between the training set and the gold standard, like the disparity between org and health care unit, is difficult to estimate. however, no other de-identification system for norwegian is available. the final evaluation of the modified hybrid deidentification tool for norwegian using norsynth-clinical phi gave an f 1 -measure of 0.731 and a recall of 0.619. a de-identification tool is aiming on a higher recall to remove all possible phis, also on the cost of lower precision. further improvements could be made to the de-identification tool. implementing dictionarybased algorithms could improve the accuracy of certain entity types. task-specific dictionaries for norwegian health care units and/or medications are feasible implementations and would likely improve accuracy on clinical texts. furthermore, implementing tokenization directly in the de-identification tool would allow for deidentification of untokenized text, and minimize incompatibilities between the input and deidentification algorithm. the gold standard has its limitations and cannot alone decide whether a specific tool provides sufficiently de-identified outcomes. therefore, we encourage to further expansions of the gold standard corpus, in addition to more evaluation research, in order to make it more reliable and improve its quality.