Automatic Anonymization of S wiss Federal S upreme C ourt Rulings (2023)

Abstract: Releasing court decisions to the public relies on proper anonymization to protect all involved parties, where necessary. The Swiss Federal Supreme Court relies on an existing system that combines different traditional computational methods with human experts. In this work, we enhance the existing anonymization software using a large dataset annotated with entities to be anonymized. We compared BERT-based models with models pre-trained on in-domain data. Our results show that using in-domain data to pre-train the models further improves the F1-score by more than 5% compared to existing models. Our work demonstrates that combining existing anonymization methods, such as regular expressions, with machine learning can further reduce manual labor and enhance automatic suggestions.

## introduction the swiss federal supreme court (sfsc) is the highest judicial authority in switzerland. it is the final arbiter in legal disputes and ensures the uniform application of federal law throughout the country. it consists of several divisions specialized in different areas of law, including civil, criminal, administrative, and social security matters @xcite . in a year, the sfsc roughly handles 7k cases and publishes its rulings. in this process, personal information must be anonymized from the rulings in order to protect involved parties. in the traditional setting, court rulings are anonymized by skilled experts. this task is highly complex, as the removal/anonymization of a word is dependent on the context it is written in. for example, zuerich needs to be removed if it is part of the name of the legal entity "zurich insurance group", but not if it is a reference to the city. at the sfsc, experts are ## related work for identifying elements that might require anonymization, a process called named entity recognition (ner) is employed. traditionally, ner recognizes and categorizes text parts according to a set of semantic categories like location (loc), organization (org), or person (per) @xcite . as these classes are not enough for the anonymization of court cases @xcite suggested expanding this list to seven coarse and 19 fine-grained classes, including entities such as judge (rr), or lawyer (an). using this dataset @xcite finetuned germanbert @xcite , clearly outperforming a bilstm-crf+ model. similar approaches have been applied and tested in other languages, such as romanian @xcite , greek @xcite , portuguese (luz de @xcite , and multilingually @xcite . domain-specific pretraining has flourished in the legal domain recently. chalkidis et al. (2020) pretrained legalbert on eu and uk legislation, echr and us cases, and us contracts. zheng et al. (2021) pretrained caseholdbert on us case law, while henderson et al. (2022) trained pol-bert on the 256 gb pile of law corpus. niklaus and giofré (2022) pretrained longformer (beltagy et al., 2020) models using the replaced token detection (rtd) task on the pile of law. hua et al. (2022) used rtd to pretrain reformer (kitaev et al., 2020) models on 6 gb of us case law. finally, niklaus et al. (2023b) released a large multilingual legal corpus and trained various legal models. we continue pretraining the german, french, and italian models for 800k and 300k steps more for base and large models, respectively. rasiah et al. (2023) pretrain models on swiss legal data, termed legal-swiss-roberta. document anonymization has a long tradition in the medical domain, where personal data need to be removed from documents. initially, this task was handled using methods such as semantic lexicons @xcite or regular expressions to replace text occurrences. recently, this has been expanded to include bert-style models as well @xcite . in the legal domain, @xcite worked on 1400 anonymized german rulings. using already anonymized rulings, they trained different recurrent neural networks (rnn) using bert embeddings. using this approach, they achieved a maximum of 68.9% precision and 79.1% recall rates. @xcite performed similar work on 80k documents from uruguayan courts. our work specifically tackles court decisions by the sfsc. we compare the generic cased mbert model @xcite with models pre-trained on in-domain data (such as legal-swiss-roberta-base @xcite ). we also investigate monolingual model performance in the three languages of the sfsc rulings: german, french, and italian. ## dataset we used 119156 rulings (77262 german, 40099 french, 6795 italian) supreme court decisions and split them into sentences using spacy @xcite . we prepared the decisions for ner based on the manual labels from the paralegals who performed manual anonymizations. the histograms in figure 2 illustrate the distribution of four key measures, namely, number of tokens, anonymized tokens, entities, and anonymized entities, in three languages: german (de), french (fr), and italian (it). different color schemes for each language enhance the visual interpretability of the plots. measures concerning tokens and entities exhibit a long-tailed distribution, signifying a concentration of instances at the lower end of the value spectrum. specifically, the distribution of number of tokens and number of entities is examined within a 10 to 100,000 range, capturing their broad spread. in contrast, anonymized tokens and entities are evaluated within a 1 to 10,000 range, reflecting their constrained distribution. ## legal pretraining to improve the sfsc anonymization system, we pretrained legal-specific models on diverse legal text in german, french, and italian. (a) we warm-start (initialize) our models from the original xlm-r checkpoints (base or large) of @xcite . model recycling is a standard process followed by many @xcite to benefit from starting from an available "well-trained" plm, rather from scratch (random). xlm-r was trained on 2.5 tb of cleaned commoncrawl data in 100 languages. (b) we train a new tokenizer of 32k bpes on the training subsets to better cover legal language. however, we reuse the original xlm-r embeddings for all lexically overlapping tokens @xcite , i.e., we warm-start word embeddings for tokens that already exist in the original xlm-r vocabulary, and use random ones for the rest. (c) we continue pretraining our monolingual models on our pretraining corpus with batches of 512 samples for an additional 1m/500k steps for the base/large model. we do initial warm-up steps for the first 5% of the total training steps with a linearly increasing learning rate up to 1e-4, and then follow a cosine decay scheduling, following recent trends. for half of the warm-up phase (2.5%), the transformer encoder is frozen, and only the embeddings, shared between input and output (mlm), are updated. we also use an increased 20/30% masking rate for base/large models respectively, where also 100% of the predictions are based on masked tokens, compared to devlin et al. (2019b) foot_0 , based on the findings of wettig et al. (2023). (d) we consider mixed cased models, i.e., both upper-and lowercase letters covered, similar to recently developed large plms (conneau and lample, 2019; raffel et al., 2020; brown et al., 2020). (e) this leaves us with two models for each language (base and large). additionally, we consider the multilingual legal models pretrained by niklaus et al. (2023b) and the swiss legal models pretrained by rasiah et al. (2023). ## anonymization system the sfsc employs an anonymization system, anom2, to assist paralegals in anonymizing rulings for public access. the main ui is shown in figure 1 . upon loading a ruling, the application auto-identifies terms requiring anonymization and lists them on the left, along with replacement text. the search function allows direct term marking for anonymization. anom2 uses different algorithms for the search for text that needs to be anonymized: conventional is based on a statistical analysis of the loaded ruling. using polyglot 2 an initial set of named entities is detected. using the specific knowledge of the format, the rubrum is dynamically detected, allowing for the labelling of important names and addresses. bert performs the recognition of entities to be anonymized using a bert @xcite model fine-tuned for ner. entity recognition is performed on the sentence level, as the rulings are often too long for the model. this approach could lead to inconsistencies in recognition, as a term identified in one sentence might not be identified in another. this is solved in post-processing, where any identified term is automatically anonymized in the whole document. legal-swiss-roberta-base works analogously to the bert method, but uses a fine-tuned legal-swiss-roberta-base @xcite model. ## experimental setup we used the following hyperparameters for all evaluated models: batch size of 64, learning rate of 5e-5, and weight decay of 0.01. we employed the seqeval metric for evaluation. we set the maximum sequence length to 192 tokens, which we deter-2 see: https://polyglot.readthedocs.io mined to be the optimal trade-off between average sentence size and training time for computational efficiency. we used early stopping based on the f1-score of the validation set, which constitutes 10% of the entire dataset, following an 80-10-10 split for the training, validation, and test sets, respectively. training ceases once the f1-score on the validation set starts to decline. due to resource constraints (we only had two tesla t4 gpus) we could only run one random seed per model. we define and configure two special parameters: 1) truncationstrideratio: we set this parameter to 0.5. when a sentence exceeds 192 tokens, we truncate it using a specific overlap strategy. the overlap consists of half of the previous snippet and half of the next snippet. 2) nonanonymizedsentencesratiotoanonymized-sentences: we set the ratio at 1.5, including only 150% of sentences without anonymization examples compared to those with examples. this minimizes data redundancy and maximizes utility. ## results ## discussion we pretrained models on swiss legal data and performed a detailed comparison of legal and generic models, both multilingually and monolingually in the ruling anonymization task. our experiments indicate that pretraining on legal data improves the performance of models significantly compared to generic multi-or monolingual models. to reduce errors in sentence splitting, we suggest future work to use legal specific sentence splitters @xcite . due to computational constraints we only experimented with base size encoder models. future work may expand this by also testing larger models.