Building a corpus for the anonymization of R omanian jurisprudence (2024)

Abstract: Access to jurisprudence is of paramount importance for both law professionals (judges, lawyers, law students) and for the larger public. In Romania, the Superior Council of Magistracy holds a large database of jurisprudence from different courts in the country, which is updated daily. However, granting public access requires its anonymization. This paper presents the efforts behind building a corpus for the anonymization process. We present the annotation scheme, the manual annotation methods, and the platform used.

## introduction the astonishing advancement of machine learning (ml) and artificial intelligence (ai) during the last decade has generated a global rush for collecting more and diverse data, as clean as possible, with an eye to the general data protection regulation (gdpr) foot_0 compliance. large language models (llms), fueling the most successful ai applications, are built from data collected from various sources, the web being the most frequent one, but not the only one. when access to the data is open to the public, according to global gdpr requirements, any personal/private information must be hidden. the procedure of hiding/obscuring/obfuscating personal data in documents released to the public is known as documents anonymization. it has to be performed so that the remaining context could not unveil the purposely hidden information. this is, generally speaking, a hard task, but, for specialized texts/language, it gains in accuracy. the judicial systems all over the world are under the scrutiny of people, who naturally claim the right to have access to information on the decisions affecting their lives. transparency of judicial decisions, as well as consistency of national courts decisions among themselves and with the international practices and recommendations are highly sensitive topics. to tackle these issues, the council of europe is implementing the project "fostering transparency of judicial decisions and enhancing the national implementation of the european court on human rights" (tjeni) foot_1 , which aims to improve the transparency and consistency of national judicial decisions, to strengthen the quality of their judicial decision-making and to streamline information on human rights jurisprudence to national judiciaries. the protection of human rights and the rule of law are strengthened by transparency of the judicial process, increasing the consistency of national courts decisions with european human rights and rule of law standards. these objectives can be supported through the publication of court decisions, which requires their prior anonymization. this can be done by applying specific technical solutions meant to automate the preparation of the documents for publication. romania is the only tjeni beneficiary that publishes decisions of all courts in romania through the portal specifically developed by the superior council of magistracy (scm), with the exception of the high court of casation and justice, which maintains its own case law database. the case law database of the scm, the official beneficiary of our project, contains all court decisions from criminal, civil, commercial and administrative case types. the only exceptions from publication are documents marked in ecris (the case management system) as confidential, such as the judgements or other decisions related to minors, sexual harassment, divorces/family matters, decisions acknowledging mediation settlements, verdicts on offences as treason, espionage, rape and child pornography and upon a request of a party in the proceeding. so far, more than 40 million decisions have been published since 2011 in the romanian case law database and the database is updated daily. initially, decisions were anonymized before publication by means of regular expressions. a new anonymizer, developed based on this work, more accurate and much faster, will replace the previous one. the new tool is being built with out-of-thebox scalability, by means of parallelism and containerization mechanisms, allowing for highperformance processing of an increasing number of documents. furthermore, the new system employs state-of-the-art llms, such as romanian language bert @xcite models, for detecting named entities (nes) that need to be anonymized, thus increasing the recognition performance. the corpus described in this work will be used for training the tool. the system still employs dictionaries and regular expression lists for certain types of nes, that are particularly suited for such recognition processes (like personal identification codes, vehicle registration, email addresses). the rest of this paper is structured as follows: section 2 presents related work, section 3 describes the annotation scheme, section 4 presents the manual annotation process in the relate platform, section 5 introduces preliminary statistics on the corpus, and we conclude in section 6. ## related work the anonymization process implies named entity recognition (ner). however, not all nes require anonymization (as detailed in section 3). given this link between anonymization and ner, this section will cover corpora built either for ner or for anonymization in the legal domain. @xcite admit that anonymization of court decisions presupposes proper identification of more than just person names, while too much anonymization reduces the text readability and usability. @xcite are concerned with the identification of lawyer names in historical legal text. they acknowledge problems arising from using nicknames or initials instead of complete names. @xcite perform fine-grained ner on a corpus @xcite legal-es @xcite ) is a large spanish corpus covering different types of legislative, administrative and jurisprudential texts. @xcite describe a ner corpus for indian court judgements, covering 14 ne classes. considering the romanian language, pȃis , et al. (2021a) constructed the legalnero corpus, covering persons, locations, organizations, time expressions, and legal references (5 ne classes). the corpus consists of a manually annotated subset of the larger marcell-ro corpus @xcite , containing legal domain texts, primarily legislation. this corpus, without the legal reference annotations (which do not have a corresponding class for the current project), could be used to augment the jurisprudence corpus described in the rest of this paper. all these papers admit the necessity to annotate more types of entities in corpora from the legal domain. as detailed in section 3, we also considered the annotation of more types of entities, as well as the necessity of their annotation (see ## annotation scheme our annotation scheme is based on ne labels commonly used in the ner field (like loc, org, per and date) and is extended to accommodate entities specific to the legal domain, especially the jurisprudence context of our project (with labels like ecli, case, decision, etc.). the scheme contains 17 entity types (see the guidelines for using the annotation scheme were adapted to the anonymization task, which is different from ner in the sense that not all nes need being anonymized. the anonymization task particularities resulted in a classification of the entity types according to their consistency of annotation: all occurrences of certain entity types (e.g., ecli, email, cui, iban, etc.) in the target documents are annotated, while for other types (such as per, date, loc, org, etc.), the decision to annotate is based on the type of reference those entities have (see below). to have a sense of the degree of distinction between the anonymization and the ner task in our framework, we randomly selected 20 documents from the set already manually annotated for anonymization and supplemented the annotation to include all the ne occurrences corresponding to our entity types of interest. date entities (used to annotate time expressions) are in the opposite situation, i.e. they must be annotated on a few specific occasions, namely when they refer to the date of birth of a person; the jurisprudence documents are rich in occurrences of date nes which are not to be anonymized: dates when the trial takes place, when a decision is made, a document is issued, etc. loc entities are anonymized only if otherwise they could disclose the identity of persons or organizations. therefore, they are always anonymised when referring to residence, place of birth, headquarters and buildings/land in possession. locations where accidents and events take place are not normally annotated, but there are exceptions to this rule: e.g., when the event takes place in a small town/village or a specific geographical location that is very close to the individual residence and whereabouts information could endanger the anonymity of the parties. the decision to annotate such occurrences is made for each case separately. org entities include all groups defined by a formal organisational structure, whether public or private. while private organizations are always anonymized, public organisations are annotated only when they are parts in the trial. by their nature, the jurisprudence documents abound in org named entities, with many of them representing law institutions that do not require anonymization. this is reflected in the tripling of org entities in our evaluation trial (see all remaining ne types (see ## manual annotation in the relate platform relate (pȃis , et al., 2020) is a modern platform incorporating a large number of tools (pȃis , , 2020) for processing the romanian language. it was previously used for automatic annotation of large corpora, such as the marcell @xcite legislative corpus and the curli-cat @xcite corpus and for creating romanian language named entity corpora, such as microbloggingnero @xcite and @xcite . for the purpose of this work, we use a number of relate platform's modules, including: corpus management, manual annotation and basic language annotation resource kits. for security reasons, with regard to data access, the modules were deployed in the secure network of the superior council of magistracy, and the web interface was made available to annotators via vpn connections. for manual annotation, the relate platform integrates the brat rapid annotation tool @xcite , connected to the platform's corpus management component. annotators are shown one document at a time and must select, using the mouse, each ne text span. the platform remembers the last document worked on by each annotator, as well as documents not finalized, allowing smooth transition between documents. at this stage, the nes are only marked in the documents and no anonymization takes place, as the corpus is intended for training automatic processes later. following the span-level annotation, documents are processed using udpipe @xcite with a custom model @xcite trained on the romanian roreftrees (rrt) corpus version 2.7 @xcite , available in the universal dependencies project. the resulting tokenized version is automatically aligned with the span-level ne annotations, using a bio (begin/inside/outside) annotation format. ## preliminary corpus statistics the manual annotation task (which is currently in progress and done by 38 annotators) is intended to cover 1,500 documents and double annotation was taken in consideration for inter-annotator agreement (iaa) analysis. at the moment, based on 5,563,617 tokens from documents doubly anno- ## conclusion and future work this paper introduced the work carried out for creating a corpus for the purpose of anonymizing the romanian jurisprudence. it is a very challenging task, with its own peculiarities when it comes to automatic processing. even though it is suitable for an ner approach, the fact that only entities requiring anonymization (not all the entities) are annotated makes it difficult to use readily available ner applications. thus, a combination of different methods are explored, including a combination of algorithms based on traditional techniques (dictionary, regular expressions) and large language models. even though completely annotating the corpus for nes and marking those that require anonymization would have enabled additional uses for the corpus, given the annotators experience in anonymization (and not ner or other corpus building activities) it was decided to focus only on the anonymization task. system development, including algorithm's implementation, is realized open source 4 , but the corpus itself cannot be publicly released, due to the sensitivity of the information. we are also considering releasing pre-trained models (when this would not compromise privacy) and anonymized corpus samples. the end-result of the project, the anonymized jurisprudence, will be available from the rejust portal 5 .