Comparing Test Sets with Item Response Theory (2021)

Abstract: Recent years have seen numerous NLP datasets introduced to evaluate the performance of fine-tuned models on natural language understanding tasks. Recent results from large pretrained models, though, show that many of these datasets are largely saturated and unlikely to be able to detect further progress. What kind of datasets are still effective at discriminating among strong models, and what kind of datasets should we expect to be able to detect future improvements? To measure this uniformly across datasets, we draw on Item Response Theory and evaluate 29 datasets using predictions from 18 pretrained Transformer models on individual test examples. We find that Quoref, HellaSwag, and MC-TACO are best suited for distinguishing among state-of-the-art models, while SNLI, MNLI, and CommitmentBank seem to be saturated for current strong models. We also observe span selection task format, which is used for QA datasets like QAMR or SQuAD2.0, is effective in differentiating between strong and weak models.

## introduction many datasets have been created to evaluate various aspects of natural language understanding (nlu) in english. these datasets are useful to measure progress; however, it is evident from various leaderboards @xcite @xcite ) that many of them are no longer challenging or discriminative enough to differentiate strong models such as those based on transformers @xcite . 1 even if these benchmarks are sound tests of important (and potentially unsolved) tasks, their usefulness is limited if they cannot measure further progress. in this paper, we ask: which datasets are best in distinguishing current and possible future strong models? we aim to compare datasets using a single metric that accounts for their effectiveness in separating current stronger and weaker models. to that end, we use item response theory (irt; @xcite , a statistical framework from psychometrics that is widely used for the evaluation of test items in educational assessment. irt assumes that the probability that a model will correctly handle an example in a test set depends on the model's latent ability parameter and three example-specific parameters, typically measuring example difficulty (how strong does a model have to be to get it right), discrimination (how effective the example is for differentiating between similar models), and guessing (how likely a weak model is to get the example right for spurious reasons). this paper presents a large-scale irt analysis of existing english nlu datasets. unlike previous work which focuses on example-level analysis within individual datasets @xcite ), here we analyze example characteristics from a larger perspective by comparing individual examples across datasets. we evaluate test sets from 29 datasets in different formats-classification, multiple-choice qa, and span-selection qa. as responses, we use model predictions from 18 transformer-based models, including some limited-capacity models chosen to expose better the dataset's ability to discriminate weaker from stronger predictors. we then fit a single irt model on these responses using a variational inference method. 2 figure 1 : distribution of test examples according to our proposed locally estimated headroom (leh) scores ( § 4.1.1), which measure the local slope of the item characteristic curve (icc) for an example at the ability level corresponding to the best model, and thus reflect the effectiveness of that single example at distinguishing between near-state-of-the-art models. datasets are grouped by task format: classification (green), sentence-level multiple-choice (blue), paragraph-level multiple-choice (red), and span selection (grey). within each format, the datasets are sorted by their release date. more details on the datasets are given in ## item response theory baker and @xcite introduce item response theory (irt), a statistical framework to measure the probability of a responder (human or ai system) predicting a correct answer for a given item (test example). the probability of a responder i answering an item j correctly is estimated as a function of the responder's latent ability θ i and the item characteristics, referred to as the item characteristic curve (icc). we use the 3-parameter (3pl) irt model, where item behavior is governed by discrimination, difficulty, and guessing parameters. the discrimination figure 2 : an example of item characteristic curves (iccs) with different values for discrimination (α), difficulty (β), and guessing (γ) parameters. p(θ) is the probability of a correct answer for a given θ. θ measures a model's ability level (higher is better). α governs the steepness of the function, β determines the θ value at which the curve is the steepest, while γ defines the baseline likelihood that an arbitrarily weak model can guess correctly. parameter (α) defines how effective an item is for distinguishing predictors along the ability axis. the difficulty parameter (β) defines a minimum level of ability at which we expect to see high responder performance. the guessing parameter (γ) defines the probability of correctly answering an item by random guessing. figure 2 shows example iccs with different parameter values. formally, the probability of individual i answering item j correctly is modeled as: ## results and analysis figure 3 shows the performance of roberta large , albert-xxl-v2, and one of the low performing minibertas (roberta-med-small-1m-2) on all validation sets. unsurprisingly, albert-xxl-v2 and roberta large are the best-performing models, while the small miniberta model achieves much lower performance. full results using all 18 models can be found in the appendix ( ## related work prior work on using irt to evaluate nlp systems mostly relies on human responses. @xcite use irt to estimate the relative ability of a set of machine translation systems using responses from pairwise comparison of system outputs by human judges. @xcite extend this work by including a baseline translation to the pairwise comparison. @xcite use irt to identify hard examples in natural language inference data based on human responses. in a follow-up study, @xcite compare human versus model responses and find that both are positively correlated and demonstrate the use cases of irt parameters in training set filtering. sedoc and ungar (2020) use irt to evaluate chatbot systems. the work by @xcite is the first to study the idea of using model responses (as opposed to human responses) for irt in machine learning research. for nlu, @xcite use model responses to estimate difficulty parameters of several glue datasets for dynamic data selection in curriculum learning. in concurrent work, @xcite study how irt can be used for more nuanced leaderboard evaluations. their experiments demonstrate that irt can produce a more reliable ranking of models than the traditional metrics. they also show that irt is not only useful for better understanding of individual examples in the dataset and task, but also effective in identifying annotation errors. for other dataset evaluations, in addition to providing a benchmark, the superglue paper also compares a set of candidate datasets using a fixed pool of machine learning models and human annotators @xcite . @xcite investigate pretraining tasks and paradigms for effective transfer learning methods. @xcite study when and why intermediate-task training is useful for a given target task. @xcite introduce task embeddings to predict the most beneficial source task for a given target task. @xcite propose an evaluation framework for machine reading comprehension (mrc) datasets and reveal some concerns regarding factual correctness and the presence of linguistic cues in existing mrc gold datasets. ## conclusion given the large number of nlu datasets introduced in recent years, what kinds of datasets are effective to measure near-future progress? our analysis on 29 test sets using irt gives us reason to believe that, among the datasets we evaluate, quoref, hellaswag, and mc-taco are best able to discriminate among current (and likely future) strong models. meanwhile, snli, mnli, and commit-mentbank seem to be saturated and ineffective for measuring future progress. our analysis of examples' difficulty and discrimination parameters shows that datasets with many hard examples do not always contain examples that can discriminate between strong and weak models. we find that qa datasets are more difficult than other datasets. we also find span selection as the most effective task format for discriminating between strong and weak models. according to our leh score, datasets that seem to be solved are unlikely to see improvements with future pretrained models. therefore, the skills they intend to test are either largely solved, to the extent that they are solvable, or not well isolated (e.g., due to data artifacts). focusing on the skills for which these solved test sets are originally designed to evaluate would most likely require a new dataset that better isolates the reasoning ability of interest. on the other hand, datasets that perform well according to our leh metric show the best signs of being amenable to future hill-climbing. this does not entail that we should focus future research on these benchmarks, since we do not evaluate whether they test the skills they mean to test, or whether these skills are important for scientific or practical progress on natural language understanding. finally, we argue that this evaluation should be done periodically, as datasets and models improve over time. for future work, one can study multidimensional variables for both model ability and item parameters, which could reveal a factorization of datasets by skills. other potential directions include expanding our analysis to a broader range of tasks and analyzing the relationship between the estimated irt parameters and the human-model gap.