Detecting Personal Information in Training Corpora: an Analysis (2023)

Abstract: Large language models are trained on increasing quantities of unstructured text, the largest sources of which are scraped from the Web. These Web scrapes are mainly composed of heterogeneous collections of text from multiple domains with minimal documentation. While some work has been done to identify and remove toxic, biased, or sexual language, the topic of personal information (PI) in textual data used for training Natural Language Processing (NLP) models is relatively under-explored. In this work, we draw from definitions of PI across multiple countries to define the first PI taxonomy of its kind, categorized by type and risk level. We then conduct a case study on the Colossal Clean Crawled Corpus (C4) and the Pile, to detect some of the highest-risk personal information, such as email addresses and credit card numbers, and examine the differences between automatic and regular expression-based approaches for their detection. We identify shortcomings in modern approaches for PI detection, and propose a reframing of the problem that is informed by global perspectives and the goals in personal information detection.

## introduction the problem of identifying personal information (pi) on the web is increasingly critical as larger and larger datasets, built by scraping data from the internet, are made publicly available and used to train machine learning (ml) models @xcite @xcite . while the extent to which this information is memorized by natural language processing (nlp) models is largely under-explored, recent work has shown that it is possible to extract specific examples of pi from trained language models such as email addresses, phone numbers, and physical addresses via prompting @xcite , while complementary work has shown that it is also possible to steer * both authors contributed equally to this research pretrained models to generate arbitrary sequences without modifying the underlying weights at all via steering vectors @xcite @xcite and prompting @xcite . this suggests that it is necessary to better understand the types of pi contained in training corpora and the types of harms that they can cause, and to propose ways for automatically detecting @xcite the most high-risk types of pi from nlp corpora. we endeavor to address both of these directions in the current article: we start with defining different types of pi and propose a novel categorization in section 2 and discuss the risks of different types of pi. then, in section 3, we explore the difficulty in detecting one of the highest-risk and easiest-to-identify types of pi, character-based identifiers, comparing a model-based pi detection tool, presidio @xcite and a simple regular-expression-based approach on the colossal clean crawled corpus (c4) @xcite and the pile @xcite . we present our results in section 4 -these suggest that some of the highest risk pi are currently not well-captured in modern tools, opening immense risk to individuals who require anonymity in data. we discuss related efforts and promising research directions in section 5, and conclude with a discussion of our results and propose ways forward to improve the extent to which our field takes pi into account in section 6. ## case study: personal information in the common crawl and the pile to estimate the quantity of high-risk character-based personal information in two popular corpora, we run both an out-of-the-box personal information detection tool and a regular-expression based approach on them. we present the methodology that we adopt, our evaluation approach, and our results in the current section. ## results we first ran presidio and our set of regular expressions to detect the different kinds of personal information listed in the previous section on the entirety of c4 and the pile. in order to validate these results, we then manually verified the top 100 documents with the most detected pi, as well as a random sample of 2800 entities detected by the two approaches. we present our results in the sections below. ## future work given the results of our case study on the c4 corpus and the pile, we propose several recommendations for dataset creators and users that can help reduce the risks and harms due to the dissemination of personal information, whether it be via dataset sharing or model training.