Benchmark Transparency: Measuring the Impact of Data on Evaluation (2024)

Abstract: In this paper we present an exploratory research on quantifying the impact that data distribution has on the performance and evaluation of NLP models. We propose an automated framework that measures the data point distribution across 6 different dimensions: ambiguity, difficulty, discriminability, length, noise, and perplexity.We use disproportional stratified sampling to measure how much the data distribution affects absolute (Acc/F1) and relative (Rank) model performance. We experiment on 2 different datasets (SQUAD and MNLI) and test a total of 135 different models (125 on SQUAD and 10 on MNLI). We demonstrate that without explicit control of the data distribution, standard evaluation frameworks are inconsistent and unreliable. We find that the impact of the data is statistically significant and is often larger than the impact of changing the metric. In a second set of experiments, we demonstrate that the impact of data on evaluation is not just observable, but also predictable. We propose to use benchmark transparency as a method for comparing datasets and quantifying the similarity between them. We find that the “dataset similarity vector” can be used to predict how well a model generalizes out of distribution.

## introduction with the growing popularity and widespread adoption of end-to-end nlp solutions, more emphasis is put on designing and maintaining high-quality evaluation frameworks @xcite @xcite . the two key components of model evaluation are data and metrics. an extensive body of research explores the significance of choosing appropriate metrics @xcite in various supervised tasks. in this paper, we present benchmark transparency: an automated framework for quantifying the data distribution and measuring the impact data can have on model evaluation. figure 1 : the impact of data distribution on model f1. we report the δ in f1 caused by re-sampling the test set across each dimension. we report the mean δ of 125 models on squad. we include random baseline and the impact of changing the "metric" from f1 to "exact". figure 1 illustrates the variance of model performance caused by different data dimensions in the squad dataset @xcite . to put the results in perspective, we also include the variance caused by uniform random re-sampling and by changing the evaluation metric. it is evident that all data features impact the evaluation more than the random baseline and 4 out of the 6 features are more impactful than changing the metric. a change in f1 by 6 -12 points is substantial and statistically significant and puts in question the validity of standard evaluation approaches. we argue that a reliable evaluation framework needs to identify the factors in the environment that largely affect the reported model performance. these factors must be quantified and explicitly incorporated in the evaluation report. we propose benchmark transparency as a way to incorporate scalable data-centric features in model evaluation and subsequently measure and predict the impact of data on reported model performance. figure 2 : comparing datasets using benchmark transparency. we measure the data distribution and obtain a "dataset similarity vector". the vector can successfully predict the out-of-distribution change of model performance. the complexity of linguistic tasks and the importance of data sampling has been discussed before in isolated studies. lack of sufficient data analysis can lead to discrimination @xcite , overestimation of model performance on challenging examples @xcite , and can hide the errors of the model on particular phenomena @xcite @xcite . we take a more holistic approach, focusing on the overall impact of data on model evaluation. we choose six data dimensions that can be measured automatically: ambiguity, difficulty, discriminability, length, noise, and perplexity. we pose two research questions: 1) what is the observable variance in model performance w.r.t. different data dimensions; and 2) can data distribution be used to directly compare datasets and predict the variance in model performance. we experiment with two datasets: squad and multinli @xcite and evaluate a total of 135 ml models (125 on squad and 10 on mnli). for the first research question we use disproportional stratified sampling to determine how the absolute (f1/accuracy) and relative (ranking) performance of models changes as a function of the data. for the second research question, we split squad and mnli by domain, using the available meta-data. we then appply benchmark transparency to directly compare the different data splits (figure 2 ) and use the resulting "dataset similarity vector" to predict how model performance will change when applied to unseen out-of-distribution data. we demonstrate that: benchmark transparency provides clear benefits to various groups of stakeholders and opens promising new lines of research. incorporating data-centric features can increase the reliability of evaluation, improve the use of nlp benchmarks, and provide a more accurate approximation for ood model performance. for model developers, the additional feedback on model performance can be used to identify and address model blindspots. our approach scales well as it uses simple proxy models to assign data features. it also generalizes to two different nlp tasks: text classification (mnli) and extractive question answering (squad). ## related work the increased complexity and lack of interpretability of end-to-end neural models makes the design of robust and exhaustive evaluation frameworks a key issue in nlp. large-scale benchmarks such as glue @xcite , super-glue @xcite , big-bench @xcite , and helm @xcite have been created to address that gap. however, existing datasets and evaluation procedures still have issues and limitations. imbalanced data can lead to issues with bias and fairness @xcite @xcite . state-of-the-art models often perform poorly on adversarially generated input @xcite @xcite . some benchmarks can be solved using heuristics and spurious correlations @xcite . models often underperform on linguistic phenomena such as negation @xcite , conjunction @xcite , or coreference @xcite . however standard benchmarks are often ill equipped to capture detailed nuances of model performance @xcite . popular benchmarks typically rely on uniform random sampling and statistical aggregation, which can hide model blind-spots on under-represented populations and phenomena. various strategies have been proposed to improve the evaluation and explicitly identify and address the limitations of the models. @xcite discuss different metrics that can be used to quantify the bias and fairness of models. the large multi-task benchmarks @xcite rely on testing a single model across multiple tasks. datasets designed to test one or more concrete phenomena @xcite @xcite can be used for diagnostics, and @xcite propose an approach for unit-testing nlp models based on predefined capabilities. @xcite suggest the use of "beat the machine" human-in-the-loop approach to gather datasets with increasing difficulty @xcite . more recently, approaches for automatic dataset analysis grow in popularity. @xcite analyze the process of model learning and identify patterns in the training set. @xcite use evaluation approaches borrowed from the educational domain to improve relative model ranking. @xcite propose a measure for dataset "difficulty" based on information theory. while promising, many of the existing approaches for dataset analysis have limited scope and scalability. some of them are not directly applicable for measuring absolute or relative model performance. we combine and improve existing data-centric techniques and propose new ones with the goal of designing a framework for data-centric and data-informed evaluation of nlp models. ## benchmark transparency in this paper, we adopt the popular claim that evaluation instances are qualitatively different @xcite . for example, some instances are more frequent and popular than others, some are more difficult for humans or models, and some are more useful for distinguishing between strong and weak models. instances can also come from different domains and can refer to different subpopulations. evaluation frameworks in nlp typically ignore the differences between instances and treat them equally. they rely on uniform random sampling and make the implicit assumption that the resulting dataset is representative for the task. in the cases when differences in the data are made explicit, it is often done across a single axis, such as target demographics @xcite . in this paper, we want to quantify the qualitative differences between data instances across multiple (independent) dimensions. we aim to externalize the implicit data assumptions and present an explicit analysis of the data distribution. the objectives of this process that we call benchmark transparency are twofold: 1) to better understand and compare the content of datasets; and 2) to provide a "dataset representation" that can be used to objectively measure the impact of data on model evaluation. we propose to use six data dimensions: ambiguity ambiguous examples are "instances whose true class probabilities fluctuate frequently during training" @xcite . note that in our framework ambiguous examples express high variability with respect to the model. they are not necessarily ambiguous to a human. to calculate ambiguity, we adapt the code from @xcite for extractive qa and use a bert-base model to score squad and mnli. difficulty intuitively, some instances are more difficult than others and processing them requires different capabilities and world knowledge. to measure instance-level difficulty, we adapt pointwise v-information (pvi) @xcite for extractive qa. for each of the two datasets, we train two bert-large models. the first model is trained normally, using the full input and the gold label. the second model is trained on the gold labels but without input (mnli) or with partial input (squad). we obtain pvi by comparing the label probabilities of the two models. discriminability the concept comes from the domain of education. in item response theory (irt) @xcite ) "discriminability" indicates how useful an instance can be in differentiating between models with varying ability. the underlying idea is that instances where models of different ability disagree are more important for evaluation than instances where the models make the same prediction. for squad, we use the implementation and data of @xcite . for mnli, we use the implementation of lalor and @xcite and analyze the data ourselves. length we introduce length as a non-trivial baseline to determine the degree to which simple quantifiable dimensions of the data can affect the evaluation outcome. we count the number of tokens in the context (squad) or the sum of tokens in the premise and the hypothesis (mnli). perplexity perplexity measures the likelihood of a text sequence, given a (neural) language model. intuitively, some examples are more likely to appear in a context. we link perplexity to the colloquial notions of frequency and popularity, which may be important to various stakeholders. we use a pretrained gpt2-large model to measure the perplexity of a question given a context (squad) and of a hypothesis given a premise (mnli). ## data impact on evaluation the evaluation frameworks of ml and nlp typically report two types of model performance: 1) absolute performance (i.e.: "how well can we expect a model to perform on unseen data") and 2) relative performance (i.e.: "how good is each model compared to the alternatives?"). in this section, we quantify the impact of data distribution on both types of model performance. we use disproportionate stratified data sampling and statistical analysis to address our first research question rq1: "what is the observable variance in model performance w.r.t. different data dimensions?". disproportionate stratified data sampling for each of the 6 dimensions, we sub-sample the original data to obtain 10 test sets with strictly increasing feature intensity. for example, "len_0" contains the 10% shortest examples, and any instance in "len_2" is longer than any instance in "len_1". we then perform model evaluation on each new test set. as the model parameters and evaluation metrics remain fixed, any difference in model performance can be attributed to the data distribution. expected random variance to put the results in perspective and to calculate the statistical significance of any observed change in reported performance we introduce "expected random variance" baseline. we randomly sample 200 test sets with size equal to 10% of the original data. we test the models on all 200 "random" sets and use bootstrapping to determine the two-tailed @xmath0.05 thresholds for change in absolute or relative performance. the random baseline allows us to filter any fluctuations due to noise or to reducing the test size foot_1 . evaluated models for squad, there are publicly available instance-level predictions of over 100 different models foot_2 . we use that data as-is to calculate absolute and relative model performance without having to re-train the models. we use the data from 125 submissions, with performance between 77 and 92 f1. for mnli, we implement and evaluate 10 different models foot_3 . ## predicting changes in performance in section 4 we demonstrated the inconsistency of evaluation frameworks caused by changes in the data distribution. with the aim of designing reliable evaluation frameworks, we want to go further and use benchmark transparency to predict the changes of model performance as a function of the distribution shift. this will allow nlp practitioners to directly incorporate data features in the benchmark design and in the evaluation metrics. it will also provide a more accurate approximation for model generalizability to unseen data. ## discussion and conclusions in this paper we emphasize and quantify the importance of data in nlp evaluation. there are various popular ways of calculating model performance: precision, recall, f1, accuraccy, and auroc for absolute performance; global ranking, pairwise "duels" @xcite , or complex statistical models @xcite for ranking. we argue that the specifics of the test data are no less important than the choice of adequate distance and aggregation metrics. the effect that data has on model performance is, no doubt, known intuitively by most researchers. however, to the best of our knowledge, this is the first systematic and multidimensional approach towards quantifying data distribution and measuring its impact on evaluation across multiple tasks and models.