PSILENCE : A Pseudonymization Tool for International Law (2024)

Abstract: Since the announcement of the GDPR, the pseudonymization of legal documents has become a high-priority task in many legal organizations. This means that for making public a document, it is necessary to redact the identity of certain entities, such as witnesses. In this work, we present the first results obtained by PSILENCE, a pseudonymization tool created for redacting semi-automatically international arbitration documents in English. PSILENCE has been built using a Named Entity Recognition (NER) system, along with a Coreference Resolution system. These systems allow us to find the people that we need to redact in a clustered way, but also to propose the same pseudonym throughout one document. This last aspect makes it easier to read and comprehend a redacted legal document. Different experiments were done on four different datasets, one of which was legal, and the results are promising, reaching a Macro F-score of up to 0.72 on the legal dataset.

## introduction although the redaction of sensitive information in different types of documents is a common practice in multiple domains, since the announcement of the gdpr and especially after its implementation, the need to find automatic or semi-automatic ways to redact documents has become a priority in many organizations. historically, the redaction of documents has been done mostly by hand, following guidelines and, in some cases, pattern-matching tools. however, due to its nature, the redaction process is not only slow, but it is also expensive as in many cases an expert needs to be consulted. in certain domains, like biomedicine, the automatic redaction of documents is well-known thanks to shared tasks, e.g. @xcite . however, in the legal domain, as in many others, the automatic redaction of documents is still a challenge. for instance, legal documents tend to be long and they have multiple types of entities, e.g. parties, witnesses, experts, judges, lawyers, and citations. furthermore, some of these entities can be either individuals or organizations. finally, as we get farther from the beginning of the document, entities become less clear to identify correctly. currently, in the legal domain, we can find two different redaction processes: anonymization and pseudonymization, and while both terms are similar, they differ in key aspects. @xcite summarizes gdpr definition of pseudonymization as the task that "prevents direct identification through attribution, but not through any other mean". certain organizations add to the definition of pseudonymization the use of a unique identifier for each individual across multiple data sources, that hides their actual identity @xcite . furthermore, it is a process, that if necessary, can be reversed @xcite as only the individuals are substituted, regardless of the occurrence of other elements which could reveal, for example, the gender or age of a person @xcite . in contrast, the goal of anonymization is to remove the complete link between individuals and data @xcite . moreover, it is a process that should make the re-identification of people hard to achieve, sometimes by doing additional alterations to the source @xcite . we present in this work the first results of psi-lence (pseudonymization of international law cases using ner and coreference resolution), a tool created to pseudonymize international arbitration documents in english. these first results come from multiple experiments done over four different datasets, one of which has been created by a group of legal experts for this specific task. the rest of the paper is organized as follows. we present the scope and objectives of this work in section 2. in section 3, we present the most relevant works found in the literature related to the automatic redaction of documents, i.e. methods and data, as well as some additional relevant tasks. then, we present the methodology of our system in section 4. the data collection explored in this work is described in section 5 while the evaluation setup is detailed in section 6. the experimental results and their discussion are presented in section 7 and section 8 respectively. finally, we conclude and propose our future work in section 9. ## scope and objectives psilence has been developed to semi-automatize the pseudonymization process of english documents within jus mundi foot_0 . currently, it focuses only on entities of type people, however, we are aware of the existence of other types of information that need to be hidden, such as emails and addresses. furthermore, from all the entities of type people, only those of type witnesses are redacted. this means that we do not redact lawyers, judges, or parties. 2 therefore, psilence has two main goals. first, to propose to a legal expert a list of people that should be redacted in the document to keep the sensitive information hidden. secondly, to cluster the names of people to provide a unique identifier to each redacted person within a document. this means that different name variations of the same person are grouped together. for instance, "mariano puerta" and "mr. puerta" will compose one cluster, while "laura puerta" would be put into a different one. in this way, we can simplify the redaction process and improve the readability and comprehension of a redacted document. ## related work in the health and biomedical domains, we can find multiple tools developed for the anonymization, pseudonymization, and deidentification of information, as presented by @xcite and @xcite . however, in the legal domain, there is a reduced number of works. for instance, we can name anoppi @xcite , a pseudonymization tool for finnish court documents that makes use of multiple ner systems, based on rules and machine learning. it uses regular expressions and dictionaries to find elements such as registration plates or specific names. as finnish inflects pronouns and nouns, they perform morphological analysis to correctly inflect pseudonyms. individuals are not grouped, this means that each occurrence of them is assigned a different identifier. in @xcite , the authors present an anonymization tool for german court rulings. specifically, the authors create a ner system by using bert embeddings @xcite through a bilstm and crf architecture. @xcite compare different tools for anonymyzing legal documents: presidio foot_2 , a generic ner system based on roberta @xcite and a specialized ner based on longformer @xcite . outside the legal domain, we can highlight the work of @xcite . in this paper, the authors present a full anonymization system for german financial documents. the system considers the anonymization task as a sequence tagging problem, thus, they make use of ner for detecting entities. they explored elements such as word embeddings, contextual embeddings, and different neural network architectures for creating the ner system. similarly, @xcite use knowledge graphs and k-anonymity @xcite to generate a weakly supervised dataset. then, the generated dataset is used to fine-tune roberta @xcite and create an anonymization tool following a ner architecture. regarding pseudonymization and anonymization data there are not many publicly available datasets. the documents that need this kind of tool have to be pseudonymized or anonymized before becoming public, due to privacy reasons, and, annotating documents is an expensive and timeconsuming task. thus many of the legal datasets used in the literature are private, such as the works of @xcite and @xcite . one exception is the tab corpus @xcite , which is a collection of publicly available documents from the european court of human rights that have been annotated for evaluating anonymization tasks. outside the legal domain, there are clinical datasets such as the 2014 i2b2/uthealth corpus @xcite and the 2016 cegs n-grid shared task @xcite . @xcite created an anonymization dataset using a collection of wikipedia biographies. as the pseudonymization and anonymization tasks can be seen as a ner one @xcite @xcite it is not uncommon for researchers to use general ner datasets, such as conll 2003 @xcite , and then apply the (pre-) trained models through a zero-shot approach into the legal domain. this is the case of the works presented by @xcite and @xcite . however, as @xcite conclude, the zero-shot results are not the best as, in some cases, the entities to mask are different than those available in the original tagset. although the clustering of individuals for the pseudonymization and anonymization tasks has been considered relevant in some works @xcite , the amount of available resources regarding this aspect is scarce. for instance, in tab @xcite only 1.7k of 24k entities of type person belong to a cluster, the rest are singletons foot_3 . in the case of conll 2012 coreference corpus @xcite there are no singletons. the best exception is litbank @xcite , a collection of 100 fiction documents in english that are annotated with coreference resolution, and presents singletons and clusters. finally, we can find some additional tools in the literature related to the pseudonymization task. in @xcite , the authors present a tool for identifying parties of legal cases using ner and coreference resolution. moreover, in @xcite , the authors present a ner system for annotating indian legal documents on which they reconcile types of named entities using rules and coreference resolution. booknlp foot_4 a spacy-based tool created for processing long documents, especially fiction books. among booknlp's tools, we can name a character clustering and a coreference resolution module. finally, petra @xcite is a model based on bert @xcite which uses memory modules to keep track of people within short documents. ## methodology in figure 1 , we present psilence's architecture, which is composed of four modules. in the first module, we make use of a python-based html parser and spacy @xcite to preprocess the documents. during the pre-processing of documents, we convert html documents into plain text divided into paragraphs and we extract citations that were defined as html spans. the second module is a hybrid ner system, i.e. based on a machine learning model and rules; its goal is to detect different types of named entities within a document. the third module is a simplified coreference resolution model that only clusters names of entities and does not consider any kind of pronouns. 6 the fourth module is a reconciliation system, similar to the one used in @xcite , which tries to determine the exact type of entity in a document, even if the context in which it occurs, is not clear. at the end of the pipeline, we create a pseudonymization dictionary, which is a json file, see figure 2 , indicating the different clusters found for each type of person entity. each cluster contains all the variations found for the same person with their occurrences based on character positions. based on the example presented in figure 2 , the occurrences of the names "bill scott", "william scott" and "scott", would be replaced with "witness_1" while the name "mcconnell" would be replaced with "witness_2" in the pseudonymized document. although in this work we focus on clusters of type witness, we provide other types of clusters in the json output in case we make a mistake in the grouping or classification of entities. the second, third, and fourth modules will be described in detail in the following subsections. ## data for training psilence coreference resolution system, we use three different corpora. litbank (bamman et al., 2020) is the main training corpus because it contains singletons, documents are long and it is one of the biggest coreference resolution corpora. however, due to its literary nature, we decided as well to use two entity-linking-related corpora, in media res (brasoveanu et al., 2020) and aida-conll-yago dataset @xcite ; both of these corpora focus on news articles. even though these two corpora are not annotated with coreference resolution groups, as we focus only on the clustering of people, we make use of their entity-linking annotations to determine clusters. in other words, named entities of type person can be grouped thanks to common knowledge-base links. for example, in aida-conll-yago, in a document talking about the signer "johnny allen hendrix", all his name variations, e.g. "hendrix" and "jimi hendrix", are linked to the same knowledge base yago id. to improve the quality of these two last corpora, we manually validated some of the clusters, and in the case of aida-conll-yago, we also included some heuristics to match some people that were not linked correctly. 13 for the three corpora, we use a training, development, and testing partition; therefore, we can fine-tune the models and evaluate their performance. besides, we have a collection of 140 international arbitration documents written in english covering different types of cases: sports (121), commercial (8), inter-state (5), iran-us claims (2), and investor-state (4); see ## evaluation in this paper, we use the evaluation framework proposed in conll 2012 coreference shared task @xcite . it asses in the first place whether all the named entities have been found within a document. and, in the second place, it evaluates how well these entities have been grouped into clusters. this evaluation framework is composed of three metrics, b 3 @xcite , ceaf @xcite and muc @xcite . however, instead of using muc as defined by @xcite , we make use of a modified version that takes singletons into account. specifically, for singletons, we define the minimum number of correct links as |k(s)|, instead of |k(s)| -1, and the number of missing links as |p(s)| instead of |p(s)| -1; where |k|(s) is the size of the key cluster for mention s, i.e. 1, and p(s) is the intersection of the predicted cluster and the key cluster of mention s. in simple words, muc for singletons becomes a binary metric. this change was necessary as the conll 2012 coreference shared task did not consider singletons but our four corpora do. as indicated in section 3, there are not many available tools for pseudonymizing legal documents. however, we compare psilence coreference resolution tool with booknlp foot_13 . specifically, we assess the clustering performance of psi-lence and booknlp when they are provided with the gold standard entities, i.e., those that have to be pseudonymized. we use booknlp because it was designed to process long documents and it is capable of performing coreference resolution @xcite . 16 the comparison is done on the legal documents, thus, we can assess how well the tools behave in the legal domain when all the correct entities are given. ## results we present in in in ## discussion regarding the results presented in as we observed in we performed a manual analysis of certain clusters found in our legal dataset to better understand some of the previous errors might be able to be fixed by changing the embeddings type, from word to contextual ones like those provided by bert @xcite or roberta @xcite . however, this would mean that the architecture of the coreference resolution system would need to change completely, as contextual embeddings are not designed for single-word analysis, and have to be trained differently for calculating cosine similarity @xcite . moreover, mod- @xcite have been created to find similar sentences and not similar words, unlike fasttext. as we observed in to solve the aforementioned issues, we can propose certain solutions. first, we need to reduce our dependency on rules for the ner by training a specialized legal ner rather than using a generic one in a zero-shot way. secondly, to reduce the number of entities with wrong boundaries, the new ner should be trained with a crf layer, like in ma and hovy (2016), and use an iobes encoding, as in @xcite . also, we might need to use data augmentation methods, such as in cabrera-diego and gheewala (2023), where a frustratingly easy domain adaption method is used to mix different legal ner corpora. moreover, some of the detected errors were caused by the reconciliation module. in other words, the rules used in this module were not robust enough to detect or solve issues generated by the ner model. for example, in one document a law firm was incorrectly tagged as a person rather than as an organization; in this case, the reconciliation module determined that the entity was of type person because it was the most frequent type, thus it was an entity that had to be pseudonymized. some other errors found during the analysis were caused by a wrong splitting of sentences. this was particularly noticeable when a paragraph contained citations that were not tagged in the html document, which in consequence made a paragraph be split into wrong sentences. in consequence, authors found in these undetected and wrongly split citations were considered many times as witnesses because specialization rules could not be applied. other splitting errors in sentences come from the fact that spacy, was not trained to analyze legal documents, thus it is not aware of specialized abbreviations such as hon'ble and q.c. moreover, we found out that in general, spacy is bad at processing long sentences, such as those that are found in legal documents. therefore, when a paragraph is wrongly split into sentences, it has a consequence not only on the ner system but also on the coreference resolution one. to solve these errors, one option is to train our model for splitting sentences, although it can be complicated to achieve due to the number of data necessary to train this kind of model. another option is to stop using sentences for delimiting the context in which a named entity occurs. however, this would mean that it could be harder to determine the actual context of a named entity in the coreference resolution system. despite the complexity of the pseudonymization task and the use of multiple deep learning models through a zero-shot approach, we consider the macro results shown in ## conclusions in this paper, we presented the first results of psi-lence, a pseudonymization tool for the semiautomatic redaction of international arbitration documents in english, where people are clustered, to accelerate the human validation step and improve the readability of the document. experiments were done on different datasets, including one composed of legal documents. the obtained results were promising, especially for the clustering of people through coreference resolution. for instance, we got a macro f-score of 0.95, when clustering gold standard named entities, and a macro f-score of 0.72 when we use the ner. an analysis of the results showed that some of the errors come from the fact that we use multiple rules at different levels. but also, because the current implementation of psilence is based on multiple zero-shot approaches, meaning that the training data did not come from the legal domain. therefore, to improve psilence, it will be necessary to work on a specialized legal corpora. in the future, we will work on the improvement of the psilence system as discussed in section 8. moreover, we would like to cluster named entities through multiple documents to assign them the same pseudonym. this would be useful when a case has multiple documents and certain people occur in several of them, allowing us to increase the readability of complex cases. finally, we will train psilence using multilingual language models on legal documents in other languages than english, especially those from the european union where legal documents are subject to gdpr rules.