Automatic Detection and Labelling of Personal Data in Case Reports from the ECHR in S panish: Evaluation of Two Different Annotation Approaches (2024)

Abstract: In this paper we evaluate two annotation approaches for automatic detection and labelling of personal information in legal texts in relation to the ambiguity of the labels and the homogeneity of the annotations. For this purpose, we built a corpus of 44 case reports from the European Court of Human Rights in Spanish language and we annotated it following two different annotation approaches: automatic projection of the annotations of an existing English corpus, and manual annotation with our reinterpretation of their guidelines. Moreover, we employ Flair on a Named Entity Recognition task to compare its performance in the two annotation schemes.

## introduction one of the reasons why research on the automatic detection and labelling of personal information in legal texts (such as case reports) is important is that many countries, including spain, have the legal requirement of removing sensitive information from these texts before publishing them. however, @xcite have argued that most research on sensitive entities detection and classification has focused on clinical data, and publicly available evaluation datasets outside this domain are scarce. moreover, as indicated by @xcite , carrying out this process manually is extremely inefficient. regarding datasets for automatic detection and labelling of personal data in the legal domain in spanish, few datasets have been published, and often they have been built from texts subjected to previous anonymization, thus making the evaluation performed on this data less realistic. this is the case of the work of @xcite , which introduces very detailed annotation guidelines @xcite . in contrast, the privacy-oriented annotated corpus in english built by @xcite stands out due to the use of case reports from the european court of human rights (echr), which are publicly available in the hudoc database foot_0 in full-text with the consent of the applicants involved in the cases. additionally, they annotated the corpus by taking into consideration not just direct identifiers, following the usual procedure, but also quasi or indirect identifiers. nevertheless, their annotation approach presents some ambiguity, given that it leaves room for interpretation and some of their selected entities cover a broad variety of forms. the ambiguity of annotation guidelines is an important factor to take into consideration when investigating automatic detection and labelling of personal information because it might cause annotations to be non-homogeneous. as argued by @xcite , non-homogeneous annotations may decrease the performance of language models (lms) on named entity recognition (ner) tasks (the tasks that precede the masking or replacement of sensitive information for anonymization or pseudonymization in legal texts). drawing from the work of @xcite , in this paper we intend to contribute to the field of automatic detection and labelling of personal information by (1) building one evaluation corpus of case reports from the echr in spanish, and annotating it following two different annotation approaches; and (2) employing this corpus for assessing the performance of flair @xcite on a ner task and comparing the results of the two annotation schemes. on the one hand, we annotate the corpus via automatic annotation projection, respecting the annotation approach of @xcite . separately, we annotate the same corpus by following our own reinterpretation of their annotation approach, inspired by the guidelines of @xcite . our goal is to observe the effects of the different level of ambiguity of the annotation approaches on the results of the ner task per entity type. we publish the code employed as well as the annotated datasets on github foot_2 under a mit license. ## the tab corpus in order to build an evaluation corpus of case reports from the echr in spanish, we departed from the test set of the english corpus built by @xcite . their corpus is called the text anonymization benchmark (tab) corpus and it is available on github foot_3 in json format under a mit license. this json file contains both the annotations as well as the texts from the echr, taken from the hu-doc database. in regards to the reproduction of its website content, the ehcr (n.d.) claims that: the information and texts available on the court's website may be reproduced provided the source is acknowledged ( c ⃝ echr-cedh) and the reproduction is made for private use or for the purposes of information and education in connection with the court's activities. this authorisation is subject to the condition that the source is indicated and that any such reproduction is free of charge. as it is explained by @xcite , the texts included in their annotated corpus only contain judgments from the "grand chamber" and the "chamber", and they are restricted to the document sections called "introduction" and "statement of facts", given that they contain the largest quantity of personal identifiers. in addition to that, they selected the judgements by ensuring that their annotators would have knowledge of the national language of the country accused of human rights violations. it is important to note that the tab corpus was annotated by 12 annotators. in the work of @xcite , annotators were instructed to annotate all sensitive entities and their semantic types in a first step. in a second step, they were asked to use their interpretation to determine whether to mask each sensitive entity for protecting a person's identity while preserving data utility. moreover, annotators were instructed to indicate whether the entities to be masked were direct or quasi-identifiers. in a third step, they added a second attribute to the entity mentions indicating whether they corresponded to confidential information (such as religious beliefs, ethnicity or health data). the tab corpus maintains the masking decisions by all the annotators given that @xcite consider that there are often multiple correct masking choices in the same text. ## dataset creation in this section we explain our workflow for creating the spanish corpus. ## experimental setup once we built and annotated the corpus of legal texts in spanish language, we used it for assessing flair @xcite , which has provided good results in the identification of sensitive information in legal texts in previous studies @xcite . we used flair version 0.12.2. for all the experiments, the corpus was split into train, dev, and test set as shown in we trained a bi-lstm-crf sequence tagger foot_6 with default hyper parameters for 18 epochs in both our experiment on the corpus with projected annotations as well as our experiment with the manually annotated corpus. we used pre-trained embeddings @xcite from flair "ner-multi" model. 7 the metrics used in the assessment of flair are precision, recall, and f1-score, computed at the mention level, but for brevity we will only focus on the f1-scores in section 5 and section 6. ## evaluation on corpus with projected annotations on the corpus with projected annotations, flair achieves a micro average f1 of 0.73. by looking at the results per entity type (shown in ## evaluation on corpus annotated with our reinterpreted guidelines on the corpus annotated with our reinterpreted guidelines, flair outperforms the results of the previous experiment, achieving a micro average f1 of 0.80. by looking at the results per entity type (shown in on the other hand, the performance of flair on the date label (0.98 f1-score) is slightly higher than it was on the datetime label (0.95 f1score). with a similar number of tags of this type, the increase in performance could indicate that it was beneficial to make a distinction between the annotation of dates, times, and durations. in particular, the annotation of durations was covered by the datetime label according to the guidelines created by @xcite . on the contrary, in our reinterpretation of their guidelines, durations were covered by the quantity label, which also shows an increase in performance (0.75 f1-score vs. 0.50 f1-score in the previous experiment). there is also a slight increase in flair's performance on the code label (1 f1-score vs. 0.92 f1-score in the previous experiment). regarding the labels per and legal professional, the performance of flair decreases when compared to the person label. in regards to the nationality label included in the manually annotated corpus, which replaces the previous dem label, the performance of flair is higher (0.85 f1-score). this is especially interesting considering that the dem label included in the corpus with projected annotations got a 0 f1-score and the number of tags is similar in both corpora. furthermore, while the performance of flair on the org label decreases, its performance on the loc label is much higher (0.87 f1-score vs. 0.67 f1-score in the previous experiment). in this case, the main reason for the increase seems to be related to the larger number of loc tags in the corpus annotated according to our reinterpreted guidelines (454 tags vs. 163 tags). the larger number of loc tags is due to our indication of annotating all the occurrences of the entity types included in our reinterpreted guidelines. finally, the performance of flair on the cur-rency label is low (0.60 f1-score). other than having few occurrences (32 tags in the whole cor-pus), this low performance could also indicate that this label is still ambiguous. ## conclusions and future work throughout this paper, we have evaluated two annotation approaches for the automatic detection and labelling of personal information in case reports from the echr in spanish language. our goal was to observe the differences in the performance of flair @xcite in relation to the ambiguity of the selected entity types. we performed this evaluation by building one evaluation corpus of case reports from the echr in spanish, and annotating it by following two different annotation approaches: automatic projection of the annotations of the english corpus built by @xcite , and manual annotation with our reinterpretation of their guidelines, also based on the work of @xcite . we used this newly-built corpus for assessing flair on a ner task and comparing the results of the two annotation schemes. we make both the corpus and the code public under a mit license to encourage research on automatic detection and labelling of personal data in legal texts in spanish. the results showed that our reinterpreted guidelines partly succeeded in getting less ambiguous labels and more homogeneous annotations. this idea is reinforced by the higher iaa obtained with our reinterpreted guidelines, which suggests that the more detailed approach of our guidelines might also help human annotators to be consistent in their annotations. as we mentioned, the manual annotation of entities may be very time-consuming. an automatic system that yields a good performance in the task will help decreasing the burden. trying to make a more consistent annotation has proven to be a sensible approach to improve the performance of flair. in the near future an anonymization analysis should be conducted to see whether our approach effectively reduces the risk of reidentification while not compromising the readability of the document. in the future research, we will expand our corpora, adapt and apply our reinterpreted guidelines to other languages, and include new specific labels. we also plan to test other language models and other techniques such as zero-shot or few shot. additionally, we intend to test privacy models such as c-sanitized @xcite for a comprehensive risk analysis.