On the data requirements of probing (2022)

Abstract: As large and powerful neural language models are developed, researchers have been increasingly interested in developing diagnostic tools to probe them. There are many papers with conclusions of the form “observation X is found in model Y ”, using their own datasets with varying sizes. Larger probing datasets bring more reliability, but are also expensive to collect. There is yet to be a quantitative method for estimating reasonable probing dataset sizes. We tackle this omission in the context of comparing two probing configurations: after we have collected a small dataset from a pilot study, how many additional data samples are sufficient to distinguish two different configurations? We present a novel method to estimate the required number of data samples in such experiments and, across several case studies, we verify that our estimations have sufficient statistical power. Our framework helps to systematically construct probing datasets to diagnose neural NLP models.

## introduction while modern deep neural language models achieve impressive performance on various benchmarking datasets, the question of how this is achieved is gaining increased attention. this line of inquiry includes a new avenue of research: probing. there are many methods to probe a neural network. among these, diagnostic classification is by far the most common. to probe a neural network in a classification configuration, we specify a classification task that examines an ability (e.g., detecting verb tense). we encode the texts with a deep neural network and apply a post-hoc classifier to the encoded representations. if the classifier can easily predict an attribute, we consider this deep neural network capable of encoding the specified ability. researchers have expanded the targets of probing classifications to a wide range of abilities including syntax and semantics @xcite figure 1 : sizes of the datasets in some common probing suites. depending on the tasks, they vary from hundreds to larger than 10 5 . 2019; @xcite @xcite , discourse @xcite @xcite and commonsense reasoning @xcite . these probing papers are associated with datasets of varying sizes, as shown in figure 1 . what is a suitable size for probing datasets? larger training datasets lead to tighter generalization bounds. with other conditions fixed, larger testing datasets allow for higher statistical power in comparing probing models and classifiers. that said, it is neither realistic nor desirable to increase the probing dataset sizes arbitrarily. it is therefore essential to find a balanced size for probing classifications. we propose a framework to formally estimate the data requirements of probing configurations ( §3). our framework considers the scenario of comparing probing configurations given some data. how many additional data samples may be necessary to reliably reproduce this comparison effect? we propose a novel method to estimate the required data samples by adapting a generalization bound. we evaluate our framework on various probing tasks. first, we verify that the choice of generalization bound agrees with the probing results ( §4.2). on a case study recognizing synthetic gaussian noise, we verify that larger datasets provide higher statistical power ( §4.3). we evaluate the utility of our framework on a collection of comparison problems ( §4.4 - §4.5), where the probing data sizes recommended by the theoretical framework always supports power larger than 0.8. this paper helps to formalize probing experiments. our framework can be used by the research community in collecting probing datasets. ## related work understanding the datasets there is increased research attention paid to the datasets used for deep learning models. one way to study the dataset is to visualize each of the dataset samples in a map. @xcite mapped the data samples in nlp datasets to regions such as "hard-tolearn" and "easy-to-learn" using signals observed during the training process. @xcite mapped the difficulty of data samples by the data-dependent complexity @xcite . @xcite , a statistical framework from psychometrics to describe various attributes related to the difficulty of test set items. similarly, researchers also describe the effects of datasets as a whole. several papers attempted through information theory @xcite . le scao and rush (2021) compared the effectiveness of prompts to those of classification samples. when collecting datasets for probing, showing the effect of data is an important goal. our framework considers the classification data samples, but we do not impose restrictions on the signal to probe in the classification tasks. ## discussion what does a high accuracy entail? our framework implicitly considers the probing classification performances, but the causal relationship between, e.g., the accuracy, the data requirements, and the reliability can be explored further in future frameworks. a high probing accuracy indicates a small empirical risk r(f ). this could result from a small |r(f ) -r( f )| (the probe "learns the task"), or a small r( f ) -r(f * ) (the distribution of the data samples represent the "true distribution" well). the two possibilities resemble the dichotomy raised by @xcite , but do they describe the same phenomenon? we leave this as an open question to future researchers. ## conclusion this paper presents a novel framework to estimate the data requirements for probing experiments. this framework uses generalization bounds from formal learning theory to determine minimum training set sizes. in a series of comparison problems, we verify that our recommendations provide sufficient power. our framework describes an actionable procedure to double check if an experiment needs additional data samples to be scientifically meaningful. additionally, this paper calls for further attention to complete a systematic methodology in evaluating probing datasets and methods.