Are All the Datasets in Benchmark Necessary? A Pilot Study of Dataset Evaluation for Text Classification (2022)

Abstract: In this paper, we ask the research question of whether all the datasets in the benchmark are necessary. We approach this by first characterizing the distinguishability of datasets when comparing different systems. Experiments on 9 datasets and 36 systems show that several existing benchmark datasets contribute little to discriminating top-scoring systems, while those less used datasets exhibit impressive discriminative power. We further, taking the text classification task as a case study, investigate the possibility of predicting dataset discrimination based on its properties (e.g., average sentence length). Our preliminary experiments promisingly show that given a sufficient number of training experimental records, a meaningful predictor can be learned to estimate dataset discrimination over unseen datasets. We released all datasets with features explored in this work on DataLab.

## introduction in natural language processing (nlp) tasks, there are often datasets that we use as benchmarks against which to evaluate machine learning models, either explicitly defined such as glue @xcite and xtreme @xcite or implicitly bound to the task (e.g., dpedia @xcite has become a default dataset for evaluating of text classification systems). given this mission, one important feature of a good benchmark dataset is the ability to statistically differentiate diverse systems @xcite . with large pre-trained models consistently improving state-of-the-art performance on nlp tasks @xcite , the performances of many of them have reached a plateau @xcite . in other words, it is challenging to discriminate a better model using existing datasets @xcite . in this context, we ask the question: are all benchmark's datasets necessary? we use the text classification task as a case study and try to answer the following two sub-questions: rq1: how can we quantify the distinguishing ability of benchmark datasets? to answer this question, we first design measures with varying calculation difficulties ( §4) to judge datasets' discrimination ability based on top-scoring systems' performances. by exploring correlations among different measures, we then evaluate how reliable a dataset's discrimination is when discrimination is calculated solely based on overall results that top-scoring systems have achieved and generalize this measure to other nlp tasks. fig. 1 illustrates how different text classification datasets are ranked (the bottom one) based on measures devised in this work (a smaller value suggests lower discrimination) and the corresponding citations of these datasets (the upper one). one can observe that: (i) the highly-cited dataset dbpedia @xcite (more than 3,000 times since 2015) shows the worst discriminative power. (ii) by contrast, dataset like ade @xcite (less than 200 times since 2012) does better in distinguishing top-scoring systems, suggesting that some of the relatively neglected datasets are actually valuable in distinguishing models. this phenomenon shows the significance of quantifying the discriminative ability of datasets: it can not only help us to eliminate those with lower discrimination from commonly-used datasets (e.g., dbpedia), but also help us to recognize the missing pearl in seldom used datasets (e.g., ade and atis @xcite ). rq2: can we try to predict the discriminative power of the dataset? given a dataset, we investigate if we can judge its ability to distinguish models based on its characteristics (e.g., average sentence length), which is motivated by the scenario where a new dataset has just been constructed without sufficient top-scoring systems to calculate discrimination defined in rq1. to answer this question, inspired by recent literature on performance prediction @xcite @xcite @xcite , we conceptualize this problem as a discrimination regression task. we define 11 diverse features to characterize a text classification dataset and regress its discrimination scores using different parameterized models. preliminary experiments ( §5.4) indicate that a meaningful regressor can be learned to estimate the discrimination of unseen datasets without actual training using top-scoring systems. we brief takeaways in this work based on our observations: our contributions can be summarized as: ## related work benchmarks for nlp in order to conveniently keep themselves updated with the research progress, researchers recently are actively building evaluation benchmarks for diverse tasks so that they could make a comprehensive comparison of systems, and use a leaderboard to record the evolving process of the systems of different nlp tasks, such as squad @xcite , glue @xcite , xtreme @xcite , gem @xcite and ge-nie @xcite . despite their utility, more recently, bowman and dahl (2021) highlight that unreliable and biased systems score so highly on standard benchmarks that there is little room for researchers who develop better systems to demonstrate their improvements. in this paper, we make a pilot study on meta-evaluating benchmark evalu-ation datasets and quantitatively characterize their discrimination in different top-scoring systems. performance prediction performance prediction is the task of estimating a system's performance without the actual training process. with the recent booming of the number of machine learning models @xcite and datasets, the technique of performance prediction become rather important when applied to different scenarios ranging from early stopping training iteration @xcite , architecture searching @xcite , and attribution analysis @xcite . in this work, we aim to calculate a dataset's discrimination without actual training top-scoring systems on it, which can be formulated as a performance prediction problem. ## how to characterize discrimination? to achieve this goal, we design measures based on the performance of different models for a dataset. ## discussion & implications discussion given a leaderboard of a dataset, metrics explored in this paper can be easily used to calculate its discrimination, while some limitations still exist. we make some discussion below to encourage more explorations on new measures: (a) interpretability: current metrics can only identify which datasets are of lower indiscriminability while don't present more explanation why it is the case. (b) functionality: a dataset with lower discrimination doesn't mean it's useless since the supervision signals provided there can not only help us directly train a system for the specific use case but also provide good supervised transfer for related tasks. metrics designed in this work focus on the role of discriminating models. calls based on observations obtained from this paper, we make the following calls for future research: (1) datasets' discrimination ability w.r.t top-scoring systems could be included in the dataset schema (such as dataset statement (bender and friedman, 2018)), which would allow researchers to gain a saturated understanding of the dataset. (2) leaderboard constructors could also report the discriminative ability of the datasets they aim to include. (3) seldom used datasets are also valuable for model selection, and a more fair dataset searching system should be investigated, for example, relevance-and scientifically meaningful first, instead of other biases, like popularity.