Anonymity at Risk? Assessing Re-Identification Capabilities of Large Language Models in Court Decisions (2024)

Abstract: Anonymity in court rulings is a critical aspect of privacy protection in the European Union and Switzerland but with the advent of LLMs, concerns about large-scale re-identification of anonymized persons are growing. In accordance with the Federal Supreme Court of Switzerland (FSCS), we study re-identification risks using actual legal data. Following the initial experiment, we constructed an anonymized Wikipedia dataset as a more rigorous testing ground to further investigate the findings. In addition to the datasets, we also introduce new metrics to measure performance. We systematically analyze the factors that influence successful re-identifications, identifying model size, input length, and instruction tuning among the most critical determinants. Despite high re-identification rates on Wikipedia, even the best LLMs struggled with court decisions. We demonstrate that for now, the risk of re-identifications using LLMs is minimal in the vast majority of cases. We hope that our system can help enhance the confidence in the security of anonymized decisions, thus leading the courts to publish more decisions.

## introduction the swift advancements in natural language processing (nlp) @xcite @xcite have introduced new challenges to the security of traditional legal processes @xcite . as public access to data increases in tandem with digital advancements @xcite @xcite , the potential risks associated with data disclosure have become increasingly significant. larger and more capable language models (lms), more powerful vector stores and potent embeddings together have the capacity to extract unintended information from pub- * equal contribution. figure 1 : re-identification framework lic data @xcite @xcite @xcite . this poses a security risk, as identifying individuals in legal proceedings can lead to privacy breaches, leading to inequity in insurance, enabling extortion, and even risking public defamation. over the past decade, at least 18 requests for name changes following re-identification of convicts have been registered in switzerland, indicating existing issues due to imprudent media coverage @xcite . the number of cases involving unlawful disclosure of personal information is likely to rise. therefore, the prevention of re-identification is critical not only for the protection of the accused, but also for the courts. @xcite even suggests that the state could be held accountable for non-monetary damages to judged persons, underscoring the urgent need for courts to address the re-identification issue proactively. vokinger and @xcite and @xcite have shown that companies can be re-identified by simply extracting information from the court decisions with regular expressions and matching it with public databases. we see strong parallels between re-identification and penetration testing, where cyber-security experts attempt to find and exploit vulnerabilities in a computer system @xcite . to the best of our knowledge, we are the first to study the re-identification task of anonymized persons from court decisions. we provide a framework for anonymization teams in courts and researchers alike to battle-test anonymizations of cases (illustrated in figure 1 ). in this work, we investigate to what extent large language models (llms) like llama-2, gpt-4 or bloom @xcite @xcite can re-identify individuals in swiss court decisions. our main findings reveal that while top models identify persons from masked wikipedia articles, they struggle with the harder task of court decision re-identification. only in cases we manually re-identified in a painstaking process and thus know re-identification is possible, and using a highly curated set of manually identified relevant news articles, they are capable of identifying the anonymized defendants from cases. finally, in detailed ablations, we identify three main factors influencing the re-identification risk: input length, model size, and instruction tuning. with our research, we are testing whether affected parties in rulings could still be identified despite anonymization. thus the results from our research can guide legal entities, data privacy advocates, and nlp practitioners in devising strategies to mitigate potential re-identification risks. this is relevant beyond switzerland, as anonymization of court rulings became mandatory across the eu with the introduction of the gdpr (see appendix f.4). the german supreme court even ruled that all rulings should be anonymized and published. however, in 2021 barely one percent of rulings were being published (hamann, 2021) (see appendix f.4). this may be partially caused by fears that publications are insufficiently anonymized and courts could be held accountable. we hope that our framework will be used to ensure privacy for anonymized documents and will therefore lead to more cases being published across europe. in the spirit of open science, we release all datasets and code for reproducibility with permissive licenses. 123 ## collaboration with the supreme court to ensure responsible research and maximize downstream usability, we collaborated closely with the federal supreme court of switzerland (fscs). the fscs currently uses regular expressions and a bert-based @xcite token classifier to provide suggestions to human anonymizers for what entities should be masked. in a prior project, we improved their system's recall on anonymization tokens from 83% to 93% by pre-training a legal specific model. in this work, we partner with their anonymization team for testing. ## datasets to perform our case study, we select switzerland for its richness in published data -both newspapers and court decisions -and its high privacy standards. we created three datasets: first, the court decisions dataset consisting of anonymized swiss case law serves as a substantial benchmark for evaluating re-identification risks in court rulings. only the fscs can assess the outcomes on this dataset, as they exclusively possess knowledge of the involved individuals. the second dataset called legal-news linkage provides a small sample of manually re-identified court rulings, elucidating potential re-identification cases by llms. finally, we curated a dataset consisting of wikipedia biographical pages and automatically anonymized it. this extensive dataset facilitates a broad analysis of re-identification techniques using llms. ## metrics re-identification of persons is a known problem for imaging @xcite , but comparable metrics for re-identifications within texts are, to the best of our knowledge, not established. unlike memorization verification @xcite the re-identification of persons requires the model to be able to connect knowledge over multiple datapoints (see section 4.2). this means that information does not always exist in a single knowledge triple, but is connected over several ones or requires several ones to lead to a re-identification. to allow the quantification of produced results, we introduce the following four novel metrics to measure reidentification performance of a person in a text: partial name match score (pnms) evaluates predictions against a regular expression requiring any part of a persons's name to be a match for the prediction to be considered as correct. for example, "max orwell" would match "george orwell". this allows for matches with predictions that only contain a part of the name. manual experimentation suggested that persons can be re-identified by using just a part of their name. the predicted name might be near exact, hence the allowance for partial matches. the metric accepts n predictions and deems any collection of predictions correct if at least one of the n predictions is correct. normalized levenshtein distance (nld) is introduced to assess the precision of predictions deemed correct by pnms. given that there is no clear-cut distinction between correct and incorrect, using the levenshtein distance provides a more nuanced perspective on how close the predictions are to the target. for the top five predictions, the smallest distance of all five was used. using the best distance of n given predictions, the distance was normalized against the length of the target name to avoid distortions in results. as example, the distance between "alice cooper" and "alina cooper" would be two, and with the normalization by len("alina cooper") applied result in 0.16. last name match score (lnms) works the same way as pnms, but only the last name is considered. the last name is defined as the last whitespace-separated part of a full name string. partial matches are accounted as correct as well meaning that the name "mill" would also be counted as correct if the target was "miller". this overlap might cause a very slight imprecision but does not lead to problems in evaluations as all models have the same advantage. weighted partial name match score (w-pnms) blends pnms and the lnms using a weighted sum, emphasizing the significance of last names for re-identification. let α = 0.35 be the weight for pnms. thus, w-pnms is calculated as the metrics are designed to recognize both exact and partial name matches. we prompted our models to predict full names, yet texts often contain name variations such as "j. doe" or "mr. doe" prompting us to accommodate partial name matches and measure the nld. our methodology overlooks spelling variations and multilingual representations, which, in our experience, are rare enough to safely de-prioritize. ## experimental setup we ran models using the huggingface transformers library on two 80gb nvidia a100 gpus, using default model configurations in 8-bit precision. for efficiency, we only used the first 1k characters of each wikipedia page. for court rulings, we extended input length to 10k characters, maximizing model sequence lengths. sequences exceeding maximum input length were automatically truncated. we used temperature 1 and considered the top 5 predictions. see figure 9 for a high level overview of our code architecture.