A Privacy-Preserving Corpus for Occupational Health in S panish: Evaluation for NER and Classification Tasks (2024)

Abstract: Annotated corpora are essential to reliable natural language processing. While they are expensive to create, they are essential for building and evaluating systems. This study introduces a new corpus of 2,869 medical and admission reports collected by an occupational insurance and health provider. The corpus has been carefully annotated for personally identifiable information (PII) and is shared, masking this information. Two annotators adhered to annotation guidelines during the annotation process, and a referee later resolved annotation conflicts in a consolidation process to build a gold standard subcorpus. The inter-annotator agreement values, measured in F1, range between 0.86 and 0.93 depending on the selected subcorpus. The value of the corpus is demonstrated by evaluating its use for NER of PII and a classification task. The evaluations find that fine-tuned models and GPT-3.5 reach F1 of 0.911 and 0.720 in NER of PII, respectively. In the case of the insurance coverage classification task, using the original or de-identified corpus results in similar performance. The annotated data are released in de-identified form.

## introduction text plays a relevant role in healthcare since it is one of the richest forms of information inside electronic health records @xcite . therefore, developing tools for processing and analyzing clinical text is an important goal of clinical natural language processing (nlp). however, one of the challenges when processing clinical text is the appearance of pii, such as names, locations, and identification numbers. to develop tools that can help the clinical community process text, researchers and developers need to access clinical text in a privacypreserving manner for the patients involved. otherwise, patients' rights are being violated. a common way to share clinical text without violating patients' rights is to publish a deidentified version of a clinical corpus. some of the most known clinical datasets are the mimic (multi-parameter intelligent monitoring for intensive care) databases @xcite @xcite . these databases contain not just clinical text from critical care units but also the whole structure and data from their databases. the previously described datasets are uncommon in languages other than english @xcite . in particular, for spanish, few clinical annotated corpus have been released. some examples are: cantemist @xcite , an annotated corpus of oncology reports; ct-ebm-sp @xcite , an annotated corpus of clinical trials; nubes @xcite an annotated corpus with negation and uncertainty entities in anonymized health records; and the chilean waiting list corpus @xcite , an annotated corpus of referrals for the chilean waiting list. this work presents a corpus for occupational health in spanish. occupational health is an area of work in public health to promote and maintain the highest degree of physical, mental, and social well-being of workers in all occupations @xcite . occupational insurance and health providers collect patient data whenever patients face a work-related accident or disease. this data is used to deliver better treatment and to decide if an occupational insurer will cover a patient. the corpus presented in this work is similar to meddoprof @xcite , an annotated corpus of occupations in clinical texts in spanish. however, meddoprof focuses on annotating only occupations, while our corpus also annotates pii. in that sense, our corpus is comparable to meddocan @xcite , one of the few freely available clinical datasets for pii identification in spanish. but there are two main differences, meddocan is a synthetic corpus, while our corpus uses actual data, and pii are masked. our team holds an agreement with one of the biggest occupational insurance and health providers in chile, giving us access to their data for research purposes. hence, the corpus introduced in this paper is a clinical corpus containing information that must be protected. the annotation procedures outlined in section 3 describe the precautions taken to minimize the privacy risks described in section 2. later, section 4 reports the experiments run with the original and de-identified corpus, including ner and classification tasks, and section 5 shows the results and discussion of the experiments. finally, section 6 states the main conclusion and future work that can be done. the main contributions of this work are: ## corpus in chile, occupational insurance and health providers actively address work-related health problems during commuting or within the workplace. the core of this procedure involves creating a document known as an admission report, in which an administrative employee compiles a narrative summary of the events surrounding the incident. after this process, a medical report called anamnesis is generated. this new document is a clinical report where healthcare professionals register the clinical details of the affected patient and the specifics of the problem from a medical per-spective. in this work, we compiled a dataset of 3,000 work-related accidents. typically, each case includes both a medical report and an admission report; however, there are instances where only one of the reports is available. as a result, we constructed an annotated corpus consisting of 2,869 documents, divided into 1,383 medical reports (anamnesis) and 1,486 admission reports. these documents are presented in a free-text format, enabling a rich and diverse range of textual content (it is noteworthy that many contain pii). conversely, within our dataset, we noted that the admission report typically demonstrates a more pronounced structural organization than to the medical report. this results in a reduced lexical variety, as illustrated in ## experiments the three experiments described in this section demonstrated the value of the corpus: ## results this section shows the results of the experiments and discusses the implications for ner and downstream tasks. ## conclusions this work introduces a novel corpus of admission and medical reports retrieved from an insurance and health provider specialized in occupational health. the annotation of pii and the subsequent de-identification process highlight the importance of releasing data considering ethical and privacy matters. this corpus is released foot_5 in de-identified form, where all sensitive entities are replaced with their class names. our exploration of the corpus has revealed its inherent value in named entity recognition and classification tasks. the insights gained through these analyses not only contribute to the existing body of knowledge but also hold practical implications for improving information extraction within the specified domain. as future work, one promising avenue involves exploring synthetic data to replace masked pii entities. this approach has the potential to not only safeguard privacy but also the possibility of building robust models that can be applied in diverse real-world scenarios with synthetic data.