Information-Guided Identification of Training Data Imprint in (Proprietary) Large Language Models (2025)

Abstract: High-quality training data has proven crucial for developing performant large language models (LLMs). However, commercial LLM providers disclose few, if any, details about the data used for training. This lack of transparency creates multiple challenges: it limits external oversight and inspection of LLMs for issues such as copyright infringement, it undermines the agency of data authors, and it hinders scientific research on critical issues such as data contamination and data selection. How can we recover what training data is known to LLMs? In this work we demonstrate a new method to identify training data known to proprietary LLMs like GPT-4 without requiring any access to model weights or token probabilities, by using information-guided probes. Our work builds on a key observation: text passages with high surprisal are good search material for memorization probes. By evaluating a model’s ability to successfully reconstruct high-surprisal tokens in text, we can identify a surprising number of texts memorized by LLMs.

## introduction for proprietary, legal, and reputational reasons, it has become common practice for companies to release few, if any, details about the secret ingredient -training data -powering their large language models (llms). for example, the data used to train gemini is described only at a high level as containing "data from web documents, books, and code" @xcite , while llama-2 apparently uses a "new mix of data from publicly available sources, which does not include data from meta's products or services" @xcite . and even though training data is widely regarded as one of the most valuable components in building high-performing llms @xcite ), 1 code/data available at https://github.com/ abhilasharavichander/information-probing token surprisal scores input text someone reconstruction probes someone tapped on my front door, and both [mask] and i sat perfectly still with our radar humming… stranger radar someone tapped on my front door, and both jack and i sat perfectly still with our [mask] humming… tapped on my front door and both someone tapped on my front door, and both jack and i sat perfectly still with our radar humming… 1. ## methodology our work aims to surface evidence of text samples that have been memorized by models, and we show that we can recover examples at higher precision compared to previous approaches such as prefix probing. information-guided probes offer a complementary view of identifying training data that is memorized by models by identifying tokens that are challenging to predict without memorization. concretely, information-guided probing involves first identifying high-surprisal tokens, then constructing probes, and finally quantifying the reconstruction capacity of target models. we discuss details of this probing pipeline. ## experiments we are interested in identifying whether some text d has been memorized by a model m . we investigate two distinct risks associated with model memorization: (1) the memorization of copyrighted content such as works of fiction or news articles (fiction and new york times), and (2) the contamination of evaluation metrics through direct memorization of test samples, which undermines the assessment of a model's capabilities by allowing it to succeed through recall rather than by applying intended skills (dataset contamination). we examine the performance of two closed models: gpt-3.5 (gpt-3.5-turbo-0125), gpt-4 (gpt-4-0613), and an open-weight model: llama-2-70b @xcite . note that as of the writing of this paper, llama-2-70b is open-weight, but details of its training data remain unknown. for all the described experiments, we use bert (110m parameters) as the reference model. we hold out 1870 instances from bookmia @xcite to tune hyperparameters such as the thresholds for selecting surprising tokens. we select low-probability or highly ranked tokens for our two information measures (please refer to appendix a for details of all hyperparameters). to evaluate detection methods, we primarily use precision (the proportion of correctly identified memorized samples among all samples flagged as memorized). when comparing methods with similar precision scores, we prefer those that identify more memorized samples. therefore, we report the f β=0.1 score which weights precision more highly than recall. for tasks which examine the memorization of copyrighted content (fiction and new york times), we compare against prefix probing due to it's widespread use @xcite . prefix probing evaluates a model's ability to generate similar continuations to a piece of text given the first n tokens as context. in practice, to overcome the limitations of a model not generating an exact continuation, evaluation can be undertaken by measuring number of words in the longest common subsequence between the original text and the model generation (lcs) @xcite . see appendix e for examples of model-generated continuations and details about our prefix probing setup. for dataset contamination where samples are typically shorter and less amenable to prefix probing, we compare against ts-slot guessing @xcite , which probes contamination in blackbox llms by first asking chatgpt to identify "informative" words and then using these to make masked-prompts for the target model. ## conclusion in the current landscape of closed llms, the lack of documentation surrounding training data remains a major obstacle for model auditing and for scientific exploration. in this work, we consider one of the most restrictive (and yet common) access scenarios: models which do not permit accessing pretraining data, model weights, or logits. we construct a probing strategy that only requires input-output access to a model, and that can be applied using much smaller and cheaper language models. we show that we can use these probes to identify documents in the training data of commercial llms. we hope this effort leads to greater transparency in the llm ecosystem, and empowers data contributors to have greater agency when interacting with ai systems.