Quantifying Association Capabilities of Large Language Models and Its Implications on Privacy Leakage (2024)

Abstract: The advancement of large language models (LLMs) brings notable improvements across various applications, while simultaneously raising concerns about potential private data exposure. One notable capability of LLMs is their ability to form associations between different pieces of information, but this raises concerns when it comes to personally identifiable information (PII). This paper delves into the association capabilities of language models, aiming to uncover the factors that influence their proficiency in associating information. Our study reveals that as models scale up, their capacity to associate entities/information intensifies, particularly when target pairs demonstrate shorter co-occurrence distances or higher co-occurrence frequencies. However, there is a distinct performance gap when associating commonsense knowledge versus PII, with the latter showing lower accuracy. Despite the proportion of accurately predicted PII being relatively small, LLMs still demonstrate the capability to predict specific instances of email addresses and phone numbers when provided with appropriate prompts. These findings underscore the potential risk to PII confidentiality posed by the evolving capabilities of LLMs, especially as they continue to expand in scale and power.

## introduction the accelerated development of large language models (llms) has resulted in substantial progress in natural language understanding and generation @xcite @xcite @xcite @xcite . however, as these models continue to scale up and incorporate increasingly larger training data, the issue of personally identifiable information (pii) leakage has become a growing concern @xcite @xcite . language models may unintentionally expose sensitive information from their training data, raising privacy concerns and posing legal and ethical challenges. to ensure the responsible development and deployment of language models, it is crucial for researchers to gain a comprehensive understanding of the risks related to pii leakage and implement strategies to mitigate them effectively. @xcite identify two key capabilities of language models that contribute to the issue of pii leakage: memorization and association. memorization refers to the ability of a language model to retain verbatim training data, which can potentially allow the extraction of pii present in the training set when provided with contextual prefixes. for example, if "have a great @xmath0.com" foot_1 is part of the training set, and the language model accurately predicts john doe's email address when given the prompt "have a great @xmath1, we would consider this a case of pii leakage due to memorization. association, on the other hand, is the ability to connect different pieces of information about an individual, enabling adversaries to recover specific pii by providing other aspects of a person. for instance, if the language model correctly predicts john doe's email address given the prompt "the email address of john doe is", then we consider this a case of pii leakage due to association. previous studies have demonstrated that models possess significant memorization capabilities @xcite . however, there remains a limited understanding of how these models perform in terms of association, a capability that poses a greater risk as it enables attackers to extract specific pii more effectively @xcite , e.g., by providing a prompt such as "the email address of {name} is" instead of an exact prefix from the training data preceding the target information. although @xcite offer a preliminary exploration of privacy leakage caused by the association capabilities of language models, their focus is limited to one dataset and the analysis primarily centers around relatively small language models. a more comprehensive examination is necessary. in this regard, we conduct an extensive analysis of the association capabilities of language models across varying sizes in two distinct domains, utilizing two distinct datasets: one containing commonsense knowledge, and the other comprising email exchanges. our experimental results elucidate both commonalities and divergences in the association capabilities of language models across the two domains. both datasets corroborate that larger models exhibit stronger association capability, and that association accuracy positively correlates with co-occurrence frequency and negatively with co-occurrence distance. nevertheless, a notable performance disparity exists between the two domains. language models exhibit strong association capabilities on the commonsense dataset but struggle to maintain the same level of performance on the email dataset. the performance gap may be attributed to the complexity of the prediction tasks and the quality of the training data. from a privacy standpoint, there are two findings regarding pii leakage risks in llms: 1) the association capability of llms is generally weaker than their memorization capacity @xcite ; 2) the association of pii is less potent than that of common knowledge. however, potential risks cannot be overlooked. namely, llms do manage to predict a portion of email addresses and phone numbers correctly when prompted with a specific owner's name. for instance, a 20b model can accurately predict approximately 3% of email addresses and 1% of phone numbers. additionally, as our analysis suggests, the model's proficiency in associating beneficial information such as common knowledge improves, it may parallelly associate more pii. therefore, maintaining vigilance is critical, given the potential for pii leakage issues to intensify as language models continue to scale. petroni et al. (2019, 2020); jiang et al. (2020); huang et al. (2022a) test the factual and commonsense knowledge across different language models. kandpal et al. (2022a) show llms' ability to answer fact-based questions and analyze how this ability relates to the number of documents associated with that question during pre-training. zheng et al. (2023) observe that sometimes chatgpt cannot associate the relevant knowledge it memorized with the target question. huang et al. (2022b); lehman et al. (2021) find that the association capability of language models plays a negligible role in pii leakage compared to their memorization capabilities. these studies provide an initial investigation into the association capabilities of language models, concentrating on a narrow range of datasets or focusing their analysis on relatively small lms. however, the understanding of llms' performance in terms of association and its implication on privacy leakage remains limited. ## background and problem formulation as highlighted by @xcite , two key capabilities of language models-association and memorization-may potentially contribute to privacy leakage. drawing from @xcite ; @xcite , we define them as follows: definition 1. (memorization) a model, denoted as f , is considered to have memorized an entity, x, if a sequence, p, present in the training data can prompt f to produce x. definition 2. (association) a model, f , is considered to have the ability to associate a pair of entities, (x, y), if it can successfully generate y when provided with a prompt p that includes x but excludes y. it is important to note that the individual designing the prompt should not have access to the model's training data and the entity y. entities in this context include pii such as phone numbers and email addresses. @xcite conduct a thorough investigation into the memorization abilities of language models. in our work, we shift our focus to investigating language models' association capabilities, as these capabilities pose a greater risk for pii leakage compared to memorization alone @xcite . specifically, we test language models' ability to recover a target entity by prompting with a related entity. to evaluate the risks of privacy leakage, we impersonate adversaries to attack lms aiming to extract as much pii as possible. it is crucial to acknowledge that association cannot entirely divorce itself from memorization, given that association processes might inherently depend on some level of memorization. in our study, our aim is not to completely eliminate the role of memorization in testing association. instead, our purpose is to test a more insidious form of attack where attackers operate without access to the training data. this means they are not just trying to match sequence prefixes to recover suffixes, but are executing more realistic attacks grounded in association capabilities. this constitutes a more realistic threat scenario compared to previous evaluations @xcite which primarily centered around verbatim recovery or direct memorization. ## overview of results in this section, we provide an overview of our results. we reserve in-depth analysis of the results for section 7 and section 8. accuracy vs. co-occurrence distance. figures 2 and 3 depict how prediction accuracy fluctuates in response to various distance thresholds set for counting co-occurrences-that is, only pairs whose distance is less than the threshold are categorized as "co-occurring". each data point signifies the mean accuracy achieved when we aggregate all pairs that co-occur within a given distance range. in computing the accuracy, we view each co-occurrence as a discrete pair. for instance, (x, y) that co-occurs 6 times within a distance of 20 and 15 times within a distance of 50 will be counted 6 and 15 times, respectively, when calculating the average accuracy for thresholds of 20 and 50. accuracy vs. co-occurrence frequency. figures 4a and 4b illustrate the relationship between model prediction accuracy and the co-occurrence frequencies. in each figure, we divide the cooccurrence frequencies into logarithmic bins and plot the average prediction accuracy of each bin. for the lama dataset, bins with fewer than 100 samples and, for the enron email dataset, bins with fewer than 10 samples are excluded. this rule also applies to all other figures that include bins. ## analysis: association capability in this section, we explore the factors influencing the association capabilities of language models. ## analysis: privacy risks on association in this section, we focus on the analysis of pii leakage related to lms' association capabilities. ## conclusion in this paper, we measure the association capabilities of language models. our results highlight that language models demonstrate enhanced association capabilities as their scale enlarges. additionally, we reveal that lms can better associate related entities when target pairs display shorter co-occurrence distances and/or higher co-occurrence frequencies within the training data. however, there's a noticeable threshold beyond which the association does not improve. moreover, other factors such as the complexity of prediction tasks and the quality of the training data also play crucial roles in influencing the association of language models. furthermore, we investigate the potential risks of pii leakage in llms due to their association capabilities. from a privacy standpoint, it is crucial to remain vigilant, as the challenges associated with pii leakage may intensify as llms continue to evolve and grow in scale. we hope our findings can help researchers and practitioners to develop and deploy llms more responsibly, taking into account the privacy risks and potential mitigation strategies.