Koala: An Index for Quantifying Overlaps with Pre-training Corpora (2023)

Abstract: In very recent years more attention has been placed on probing the role of pre-training data in Large Language Models (LLMs) downstream behaviour. Despite the importance, there is no public tool that supports such analysis of pre-training corpora at large scale. To help research in this space, we launch Koala, a searchable index over large pre-training corpora using lossless compressed suffix arrays with highly efficient compression rate and search support. In its first release we index the public proportion of OPT 175B, GPT-3, GPT-Neo, GPT-Neo, LLaMA, BERT, ELECTRA, RoBERTA, XLNet pre-training corpora. Koala provides a framework to do forensic analysis on the current and future benchmarks as well as to assess the degree of memorization in the output from the LLMs. Koala is available for public use at https://koala-index.erc.monash.edu/.

## introduction large language models (llms) have achieved state-of-the-art results in nlp and on many benchmarks have reached the performance ceiling @xcite . this evergrowing success has been facilitated by the algorithmic and computational progress in scaling up model sizes @xcite @xcite , integrating human feedback @xcite , adopting modes of instructional inference at both zero-or few-shot settings @xcite @xcite , as well as the ability of feeding them massive volumes of free text during pre-training. recent works exhibit various cases which highlight the sensitivity of downstream behaviour of llms (and their smaller variants) to the frequency of observed overlap between pre-training corpora * corresponding author and test set @xcite @xcite @xcite . in the generative setting, several issues such as hallucination @xcite , undesired biases @xcite , or toxicity @xcite have been attributed partly or fully to the characteristics of the pre-training data, while a parallel line of works have emphasised on the positive role of filtering the pre-training data for safety and factual grounding @xcite . the above observations are not a comprehensive list but echo the undeniable role of pre-training data in how these models would function in practice. understanding the limitations imposed by pre-training data would also lead to more informed algorithmic and computational innovations @xcite . however, these forensic studies are done either at a small scale or by using surrogate sources such as web search hit counts. this is mainly due to the absence of reliable tools for supporting deeper analyses in this space at large scale. our work attempts to fill this gap. we launch the koala project, a service backed by lossless compressed suffix arrays (csa) @xcite , with efficient compression rate and query support. koala contains a searchable index over the public portion of the pre-training corpora foot_0 of several existing pre-trained language models from opt 175b @xcite to bert @xcite . koala is intended to provide various overlap statistics for text query files provided by researchers. we foresee several areas of impact for koala; (i) as a tool to measure data leakage between existing benchmarks and pre-training corpora of llms, (ii) and evaluate the degree of memorisation or creativity in generative models' output, (iii) and to support designing harder benchmarks by reducing the overlap with pre-training corpora. we present an overview of the koala pipeline for pre-processing and constructing the index. we also provide examples of the types of analyses that could be done via koala by looking at a few commonly used test benchmarks. ## conclusion and future work we presented koala, a web-based service powered by a compressed data structure backbone that facilitates efficient search over large collections of texts. koala is a tool for comprehensive overlap analysis with potential use-cases including but not limited to assessing leakage of test benchmarks, measuring the degree of memorization in genera-tive llms outputs. additionally, koala not only provides a public tool for forensic analysis of these phenomena it could also help benchmark designers towards constructing more challenging testbeds for llms.