Behind the Mask: Demographic bias in name detection for PII masking (2022)

Abstract: Many datasets contain personally identifiable information, or PII, which poses privacy risks to individuals. PII masking is commonly used to redact personal information such as names, addresses, and phone numbers from text data. Most modern PII masking pipelines involve machine learning algorithms. However, these systems may vary in performance, such that individuals from particular demographic groups bear a higher risk for having their personal information exposed. In this paper, we evaluate the performance of three off-the-shelf PII masking systems on name detection and redaction. We generate data using names and templates from the customer service domain. We find that an open-source RoBERTa-based system shows fewer disparities than the commercial models we test. However, all systems demonstrate significant differences in error rate based on demographics. In particular, the highest error rates occurred for names associated with Black and Asian/Pacific Islander individuals.

## introduction in a time of extensive data collection and distribution, privacy is a vitally important but elusive goal. in 2021, the us-based identity theft resource center reported a 68% increase in data breaches from the previous year, with 83% involving sensitive information foot_0 . the exposure of personally identifiable information (pii), such as names, addresses, or social security numbers, leaves individuals vulnerable to identity theft and fraud. in response, a growing number of companies provide data protection services, including pii detection, redaction (masking), and anonymization. pii masking offers assurances of security. however, this paper considers whether the models pow-ering these services perform fairly across individuals, regardless of race, ethnicity, and gender. historically, the us "right to privacy" concept has been centered around whiteness, initially to protect white women from the then-emergent technology of photography and visual media @xcite . black individuals have had less access to privacy and face greater risk of harm due to surveillance, including algorithmic surveillance @xcite . in this paper, we evaluate the detection and masking of names, which are the primary indexer of a person's identity. we sample datasets of names and demographic information to measure the performance of off-the-shelf pii maskers. although model bias or unfairness can be the result of a number of factors, including training data or presuppositions encoded in the algorithms themselves, the commercial systems we examine fail to provide details about training data or implementation. therefore, we do not hypothesize a causal relationship between these factors and our findings. our work quantifies disparities in the name detection of pii masking systems where poor performance can directly and negatively impact individuals. we demonstrate significant disparities in the recognition of names based on demographic characteristics, especially for names associated with black and asian/pacific islander groups. ## pii masking this study analyzes personally identifiable information (pii) masking systems which aim to detect and redact sensitive personal information, particularly names, from text. this has been an important problem in the biomedical domain, in terms of preparing de-identified patient data for research @xcite , but is also increasingly important in an age of language models trained from web-scraped data, which have been shown to reveal private information that was not removed from the underlying training data @xcite . since early efforts masking data by hand, automated methods have been employed, from using word lists or dictionaries @xcite , which do not generalize to unseen names and locations, to rule-based or regular expression systems @xcite , which are generalizable, but can be brittle. these have been replaced with machine learning systems @xcite and most recently neural networks @xcite . modern pii maskers rely on named entity recognition (ner) to identify entities (e.g. name and location) for redaction. ner has had recent success with hybrid bi-directional long short term memory (bilstm) and conditional random field (crf) models @xcite , and following the general trend in nlp, fine-tuning on large language models such as bert @xcite . additional discussion on ner architectures can be found in @xcite . previous research in named entity recognition (ner) has illuminated race and gender-based disparities. @xcite evaluates a number of ner models which consider performance according to gender and race/ethnicity. the analysis considers 15 names per intersectional group, finding that white-associated names are more likely to be recognized across all systems. our work differs from and extends this work in key aspects: focusing on off-the-shelf pii masking, providing analysis on over 4k names, and reporting on significance and additional metrics. recent pii masking models perform extremely well in certain contexts. the recurrent neural network of @xcite achieves 99% recall overall and just below 98% for names on patient discharge summaries in the medical domain. the commercial models we consider do not advertise performance metrics, and as shown in section 7, do not achieve such high performance across our datasets. it is important to note that removing names alone is insufficient to fully protect individuals from being identified from data. data sets can still reveal just enough information to re-identify individuals, as in the case of massachusetts governor william weld, whose medical records, although not con-nected directly to his name in a de-identified data set, were traceable back to him by matching information from an easily attained external data resource @xcite . here we focus on names as they are a primary identifier for an individual. ## what's in a name? the primary goal of this paper is to understand whether, and to what degree, the performance of pii masking models is influenced by correlates of race, ethnicity, and gender. we frame bias in terms of significant discrepancies in performance based on race/ethnicity and gender, looking specifically to instances where private information was not masked (false negative rates, described in section 6.2). pii masking is a primary mechanism for protecting personal data, and a systematic failure to mask information belonging to marginalized subgroups can cause undue harm to those populations, through identity theft, identity fraud, and loss of privacy. names are not a proxy for gender or race/ethnicity, but our rationale is as follows: if most of the people with name n have self-identified as belonging to group g 1 , and name n is frequently miscategorized by pii systems at a rate that is higher than that for a name more commonly used by individuals in group g 2 , then we argue that members of group g 1 bear a higher privacy risk. we focus our analysis on given names (sometimes known as 'first names') and family names (sometimes known as 'surnames' or 'last names'). naming conventions vary in different cultural and linguistic contexts. in many cultures, given names and/or family names can be gendered, or disproportionately associated with a particular gender, religious or ethnic group. in the present study, gender, race and ethnicity are considered with respect to a defined set of categories for the purpose of analysis, but we acknowledge that such labels are socially constructed and mutable over time and space @xcite . previous research has uncovered racial and gender discrimination based on individual names. @xcite found that, given identical resumes with only a change in name, resumes with black-associated names received fewer callbacks than white-associated names. @xcite found that internet searches for black (in contrast to white) names were more likely to trigger advertisements that suggested the existence of arrest records for people with those names. we do not attempt to infer personal information tied to names in our data, but rather, rely on real, self-reported information. however, there are limitations to using standardized gender and racial categories in studying algorithmic fairness, even when individuals are able to self-identify @xcite . within each racial/ethnicity category made available on the standardized forms in the data we use (described in section 4), for example, there is a large variety in the linguistic cultures and naming practices encompassed in each group. our intent is not to conflate race and ethnicity and language, but rather to get a coarse-grained look at performance of pii masking systems on names that are strongly associated with the demographic groupings that are available. similarly, the available data limits gender categories to the binary 'male' and 'female,' and while names are not a good proxy for gender, we look for strong associations in the data, as described further in section 4. ## data in this section, we describe our method for creating test sentences for evaluating name detection in pii masking models. in our evaluation, we use a sentence perturbation technique which is employed in previous studies to test model performance across sensitive groups @xcite . using a variety of templates, we fill slots with names from the datasets, allowing us to measure performance across race/ethnicity and gender. reliable sources of demographically labeled names are difficult to find and using real names is an issue of privacy. therefore, we consider datasets of names with aggregate demographic information as a proxy. we also evaluate on the names of us congress members, whose identity and self-reported demographic information is publicly available. templates and source datasets are described in the following sections. ## sampling process this section describes the process of sampling the source names. the lar and nyc datasets aggregate name counts and frequencies per race/ethnicity. we sample names which have a strong 'association' with a particular race/ethnicity and gender. because frequency (i.e. popularity) of a name could contribute to spurious performance disparities between groups, we sample the lar data so that all names are frequency matched across groups. ## experiment setup the following sections discuss the pii masking systems we evaluate. we use several metrics to investigate the pii masking performance across name subsets. foot_3 ## results we present the results of the evaluation, considering overall performance and performance related to race/ethnicity, gender, and intersectional factors. the section concludes with an analysis of errors. ## discussion this paper considers differences in the performance of three pii maskers on recognizing and redacting names based on demographic characteristics. supported by quantitative results and error analysis, we find disparities in the fairness of name masking across groups. in terms of race and ethnicity, api-associated names are often poorly masked. disparities are shown to be significant for aws and gcp systems. this is not simply a result of the popularity of the names, as the frequency-controlled lar1 dataset revealed disparities between api and whiteassociated names. name length is considered as a performance factor, but it does not entirely account for the gap between api and white-associated names. several systems and datasets show poor performance on the masking of black-associated names. gcp and presidio revealed significant differences between black and white-associated names. error rates are especially high on the nyc dataset, and are highest for black women. this is in line with previous research which demonstrates the poor performance of nlp systems on black women (see inter alia @xcite . race and ethnicity were the strongest factors related to pii masking performance, but gender-based differences were also noted. names which were not strongly associated with gender had the highest error rates. this underscores the importance of considering categories outside the traditional gender binary when evaluating systems for bias. of all pii masking systems, the presidio model (with roberta ner) shows fewer significant discrepancies based on demographics. however, all systems demonstrate some significant disparities. across datasets, the performance difference between groups is not consistent. for instance, the aws model has poor performance on api names in the lar2 dataset but not in nyc. we consider this not an issue, but a feature of our evaluation across datasets. the datasets we've chosen contain variety in age groups, locations, and contexts. we argue that evaluating nlp systems responsibly requires careful curation of data, including steps to consider the context of the system and the diverse set of system users and stakeholders. the aggregate name data used here is openly available and can be used for testing on pii masking, ner, and related systems. we are releasing ## conclusion this work considers the performance of pii masking systems on names sourced from real data. we find disparities related to demographic characteristics, especially race and ethnicity, across all systems. while features such as name length and ambiguity play a role in recognition, they do not fully account for performance differences. disparities in the performance of pii masking systems reflect historical inequities in the "right to privacy". the nlp community, as a commodifier of both models and data, has a responsibility to develop more equitable systems to protect the data privacy of all individuals.