Automated Anonymization of Parole Hearing Transcripts (2024)

Abstract: Responsible natural language processing is more and more concerned with preventing the violation of personal rights that language technology can entail (CITATION). In this paper we illustrate the case of parole hearings in California, the verbatim transcripts of which are made available to the general public upon a request sent to the California Board of Parole Hearings. The parole hearing setting is highly sensitive: inmates face a board of legal representatives who discuss highly personal matters not only about the inmates themselves but also about victims and their relatives, such as spouses and children. Participants have no choice in contributing to the data collection process, since the disclosure of the transcripts is mandated by law. As researchers who are interested in understanding and modeling the communication in these hierarchy-driven settings, we face an ethical dilemma: publishing raw data as is for the community would compromise the privacy of all individuals affected, but manually cleaning the data requires a substantive effort. In this paper we present an automated anonymization process which reliably removes and pseudonymizes sensitive data in verbatim transcripts, while at the same time preserving the structure and content of the data. Our results show that the process exhibits little to no leakage of sensitive information when applied to more than 300 hearing transcripts.

## introduction the growing need for anonymized datasets in computational social science such as nlp applications in law, criminology, sociology and political science is driven by the importance of ethical compliance, legal requirements, reduction of bias, and, ultimately, by the necessity for data sharing. in the context of spoken and transcribed dialogue data, anonymized datasets are particularly scarce. this holds especially for dialogues in legal set-tings, such as parole suitability hearings, where inmates who were originally sentenced to life-long imprisonment engage in discussions with a board of commissioners, requesting to be released from prison before the completion of their sentence. in california, the verbatim transcripts of these hearings can be requested at the california board of parole hearings, but sharing them publicly raises ethical concerns: they include personally identifiable information (pii), such as names, inmate ids, dates and other sensitive details about the people involved, and the participants do not have a choice as to whether they want to take part in the data collection process. researchers who wish to make these transcripts available to ensure academic transparency face ethical dilemmas, as publishing the data would compromise the privacy of those affected. the contribution of our work is twofold: first, we introduce a robust automatic anonymization process for dialogue transcripts in criminal law, ensuring consistent entity replacement throughout each transcript. second, we provide an evaluation of our process based on a subset of 100 manually anonymized parole hearing transcripts and show our system's minimal risk of data leakage demonstrated by the systems high precision. ## related work while anonymization has mainly been applied to data in the legal and clinical domain, researchers from other disciplines also feel the need to protect sensitive information in their datasets. in the specific case of californian parole hearings, the dataset has not been made available to the public in its entirety @xcite , with the available data only restricted to individual examples @xcite . in a similar case for german, the entire dataset was manually redacted @xcite and double-checked by a second person to ensure correctness. regarding automatic anonymization, previous work mainly relies on written legal and clinical documents, applying methods such as masking, falsification or pseudonymization. anoppi uses a combination of automatic and semi-automatic processes, utilizing statistics and rule-based named entity recognition (ner) methods to identify and remove personal information in finnish court documents @xcite . it consistently replaces sensitive data with categorical labels, preserving both the semantic meaning and readability of the documents. psilence uses a combination of ner tools and coreference resolution to ensure consistent labelling of entities in written legal documents @xcite . @xcite proposes a customization solution to anonymize german legal court rulings using domain-specific ner in order to mask entities according to predefined rules. in the clinical domain, @xcite proposes incogni-tus to automatically anonymize clinical notes by using a combination of ner tools like conditional random field. for the anonymization of spoken language, @xcite use an automatic speech recognition (asr) system to generate transcripts from phone and video conversations and then enhance google data loss prevention service to improve pii detection. the data underlying this paper falls between audio data (unpredictable and unstructured) and written legal documents (structured and manually curated): parole hearing transcripts exhibit some inherent structure with clearly identified speakers and roles, while dialogue is characterized by repair sentences, filled pauses, elliptical content as well as cut-off and spelled-out names with pii. while the aforementioned work on anonymization of written language does not directly translate to verbatim transcripts, the approach we take in the paper is similar in that we combine named entity recognition and regular expressions with rule-based postprocessing to identify sensitive information in the transcripts and replace it with categorical tags foot_0 . in this way we adjust the approach by incognitus @xcite and apply it to dialogues from criminal law. ## data the dataset comprises 334 parole suitability hearing transcripts in pdf format, which we officially requested from the california department of corrections and rehabilitation (cdcr) foot_1 . all hearings took place between august and september 2021. the corpus consists of 21,874 pages and 5,013,156 words in total, with an average of 15,009 words and 65 pages per file. the transcripts share a standardized format, which we can exploit for anonymization: the first page of the pdf contains the names of the participants present in the hearing, the location such as the prison or facility of the inmate, as well as the date and time of the hearing. the second page contains the index, indicating the page number of each section (such as pre-commitment factors, post-commitment factors and decision) of the hearing. the subsequent pages comprise the main body of the transcript, containing the verbatim transcription of the dialogue, uniformly formatted with the speaker tag and the according text. the document also includes the closing statements of all participants and the final decision. the last pages of the document are reserved for a declaration of the transcriber as well as their signature. for anonymization, all transcripts are converted to text files. the index of each transcript is ignored during conversion as it does not contain pii. the final text file is formatted so that each utterance appears on a separate line to facilitate reading and processing. ## pseudonymization pseudonymization involves assigning unique labels to each distinct entity within the dataset for the purpose of anonymization, meaning that reappearing entities are consistently replaced by the same tag. this is done through the use of a dictionary, which stores the original entity along with their pseudonymized category label. the primary function of this dictionary is to ensure the correct tag is consistently assigned throughout the transcript to the specific entities. in practice, entities are anonymized by combining the entity type with a sequential number. for example, names within the transcript are replaced by tags like [person_1], [person_2] and so forth. this approach is applied not only to names but to all recurring entities, ensuring consistent labeling across the dataset. appendix a (step 4) showcases a practical illustration of how the anonymized final transcript appears after the replacement of pii by categorical tags. for person entities, all full names are extracted and each part of the full name is assigned a unique tag, which is then stored in a dictionary to ensure consistency across the transcript. we decided to use this approach, as commissioners often refer to the inmates by their last name alone. each part of a person entity is labeled with a sequential number, and the corresponding spelled_name is normalized and assigned the same sequential number as the matching per-son entity. for cdcr id entities, each unique value is assigned an individual tag. example (1) demonstrates how these direct identifiers (in 1a) are anonymized within the transcripts (in 1b): we have a john doe and the victim is jane smith. that's d-o-e. case id m23515. b. anonymized: we have a [person_1] [per-son_2] and the victim is [per-son_3] [person_4]. that's [spelled_name_2]. case id @xcite . given that specific information such as the offender's name, hearing date and time are publicly available online on the cdcr's hearing calendar web page 6 , we remove numerical values, such as date, time and age, by replacing the original data with fine-grained labels, using manually crafted rules consistently across all transcripts. the full date value is split into individual components and replaced by a type-specific label based on specific conditions. for instance, ordinal numbers ("1st", "2nd", ...) in the context of dates are replaced with the label @xcite . months and days of the week are detected using regular expressions and replaced with [month] and [day_of_week] labels respectively. four digit numbers under the date entity represent years and are replaced with @xcite . decades ("20s", "30s", ...) are replaced with the label @xcite . formatted dates that resemble patterns like "mm/dd/yyyy" are simply replaced by @xcite . any other numbers that do not satisfy the aforementioned conditions are replaced with @xcite . example (2) illustrates the handling of these specific entities. to generate more fine-grained and accurate pseudonymization labels for nrp, location and organization, we use bart, a zero shot classification model by facebook @xcite . bart is trained on the multi-genre natural language inference (multinli) dataset @xcite , which includes a diverse range of written and spoken data sources, including letters, oxford university press, press releases from government websites as well as transcriptions of faceto-face conversations and telephone calls. the nrp entity, derived from presidio, combines an individual's nationality, religion, and political group affiliations -three distinct yet interrelated types of sensitive information. similarly, lo-cation entities are complex, including diverse geographical information such as states, counties, cities, and countries. bart's role is to distinguish and categorize the data within these multifaceted entities. by doing so, it enhances the contextual relevance and overall utility of the final dataset. this approach allows for more precise and meaningful pseudonymization while maintaining the analytical value of the data. example (3) illustrates the conversion of each entity type into appropriate category labels, demonstrating the granularity and accuracy achieved through this method. original: he lived in connecticut but then moved to california. he is a canadian citizen from canada and works with the california city police department. b. anonymized: he lived in [state_1] but then moved to [state_2]. he is a [na-tionality_1] citizen from [coun-try_1] and works with the [po-lice_department_1]. for the remaining entities, each unique occurrence is replaced with the specific category label and their corresponding sequential number. appendix c shows that person entities are the most frequent throughout parole hearing transcripts, accounting for 34.92% per 1000 tokens. this result is none of a surprise, as discussions typically revolve around the inmate, their victims, accomplices and family members. date, age and location entities, while important, appear at varying frequencies with only date occurring just over 3% per 1000 tokens. this reflects the importance of discussing the inmate's age at specific life events, such as the crime or key moments during incarceration, as well as references to locations related to their past or to future parole plans. similarly, the frequent mention of date and lo-cation entities can be attributed to discussions about important milestones in the inmate's history or potential locations for future parole plans. ## conclusion we introduced a novel approach for the anonymization of direct and indirect identifiers in parole hearing transcripts, offering a way to protect sensitive personal information while preserving the utility of the data for different kind of research purposes. our methodology combines named entity recognition tools with pseudonymization techniques and addresses the challenges posed by this specific type of legal dialogue. despite the limitations of current ner tools, leading to misclassified entity types and errors arising from misspelled names, our approach successfully cleans the transcripts from sensitive data in the majority of cases. while our approach provides a strong methodology to reliably pseudonymize parole hearing transcripts, a thorough manual review of the transcripts before publication is still mandatory in order to avoid any unintended data leakage. future research should focus on addressing the remaining limitations, with the ultimate goal of enhancing data privacy without sacrificing data utility.