Probing Language Models for Pre-training Data Detection (2024)

Abstract: Large Language Models (LLMs) have shown their impressive capabilities, while also raising concerns about the data contamination problems due to privacy issues and leakage of benchmark datasets in the pre-training phase. Therefore, it is vital to detect the contamination by checking whether an LLM has been pre-trained on the target texts. Recent studies focus on the generated texts and compute perplexities, which are superficial features and not reliable. In this study, we propose to utilize the probing technique for pre-training data detection by examining the model’s internal activations. Our method is simple and effective and leads to more trustworthy pre-training data detection. Additionally, we propose ArxivMIA, a new challenging benchmark comprising arxiv abstracts from Computer Science and Mathematics categories. Our experiments demonstrate that our method outperforms all baselines, and achieves state-of-the-art performance on both WikiMIA and ArxivMIA, with additional experiments confirming its efficacy.

## introduction large language models (llms) trained on massive corpora of texts demonstrate extraordinary abilities to understand, reason, and generate following natural language instructions @xcite . meanwhile, the open-source of llms has significantly contributed to the advancement and collaborative development within the llm community @xcite @xcite @xcite . despite this progress, the lack of transparency raises ethical and legal questions, particularly about the use of potentially private data sourced from the internet, and threatens the reliability of benchmark evaluations due to the risk of leaking test data. therefore, determining if certain texts have been utilized during the pre-training phase of the target llm becomes a critical task. recent efforts to detect pre-training data in llms have attracted significant attention. several studies have been proposed to investigate dataset contamination, including prompting llms to generate data-specific examples or using statistical methods to detect contamination in test sets @xcite @xcite . concurrently, membership inference attacks (mias) in natural language processing have been extensively explored for their potential to discern whether specific data was used in llms' pre-training @xcite @xcite . the above solutions have achieved a certain success. however, they all rely on the model's superficial features, such as generated texts or loss metrics, which may not always be reliable indicators of pretraining data. different from these conventional approaches, we propose a simple yet effective pre-training data detection method that utilizes the probing technique to examine the model's internal activations. this approach is based on the assumption: texts that have been seen during the model's pre-training phase are represented differently in the model's internal activations compared to texts that have not been seen, so we could train a linear probe classifier to distinguish them. as illustrated in figure 1 , our method consists of three main steps: (1) we initiate our process by gathering a training dataset that the llm has not previously been trained on, splitting the data into member and non-member subsets. we then inject the member data into the target model through a fine-tuning process on the member data alone. this step enables us to create a proxy model that retains the memory of the member data from the training member data @xmath0... ... layer 1 fine-tune training dataset [text, label] training dataset [text, label] training training dataset [activation, label] dataset. in other words, the probe classifier could assess whether the target text has been seen during the pre-training phase. in order to evaluate various pre-training data detection methods in a more challenging scenario, we introduce arxivmia, a difficult benchmark in the academic domain. in contrast to the existing wikimia @xcite benchmark, which primarily utilizes event data from wikipedia pages, arxivmia presents a more challenging scenario. the academic abstracts within arxivmia are rarer on the internet compared to wikipedia content, naturally resulting in a lower duplication rate. furthermore, the inherent complexity of texts targeted at researchers adds another layer of difficulty for model memorization. this combination of rarity and complexity makes it exceedingly challenging for large models to memorize such content during the pre-training process, making its detection through conventional methods markedly tougher. therefore, arxivmia stands as an especially rigorous benchmark, highlighting the need for more sophisticated pre-training data detection methods. our contributions can be summarized as follows: ## experiments we evaluate the performance of our method and other baselines against open-source language models trained to predict the next word, including pythia-2.8b @xcite , opt-6.7b @xcite , tinyllama-1.1b @xcite and openllama-13b @xcite . ## results and analyses in this section, we report our main result and conduct ablation studies to analyze the impact of model size and training data number for our method. we also compare the performance of various detection methods on pubmedqa @xcite and commonsenseqa @xcite in the contamination detection challenge proposed by @xcite . ## conclusion in summary, this paper investigates the pre-training data detection problem in large language models. we propose a simple and effective approach that determines whether a target text has been included in a model's pre-training dataset by analyzing the internal activations using the probe technique. additionally, we introduce a more challenging benchmark, arxivmia. the experiments demonstrate that our method outperforms all baselines across various benchmarks, achieving sota performance. we further analyze the impact of target model size and the number of training data on our method. additionally, we validate the effectiveness of our approach through a downstream task datasets contamination detection challenge. future work could extend our methods to larger model scales or apply them to multi-modal models.