Clustering Examples in Multi-Dataset Benchmarks with Item Response Theory (2022)

Abstract: In natural language processing, multi-dataset benchmarks for common tasks (e.g., SuperGLUE for natural language inference and MRQA for question answering) have risen in importance. Invariably, tasks and individual examples vary in difficulty. Recent analysis methods infer properties of examples such as difficulty. In particular, Item Response Theory (IRT) jointly infers example and model properties from the output of benchmark tasks (i.e., scores for each model-example pair). Therefore, it seems sensible that methods like IRT should be able to detect differences between datasets in a task. This work shows that current IRT models are not as good at identifying differences as we would expect, explain why this is difficult, and outline future directions that incorporate more (textual) signal from examples.

## introduction understanding and describing the data in natural language processing (nlp) benchmarks is crucial to ensuring their validity and reliability @xcite @xcite . this is even more important as multi-dataset task benchmarks have-for better or worse-become the norm @xcite . for example, superglue incorporates eight natural language inference (nli) datasets @xcite , and mrqa incorporates twelve question answering (qa) datasets @xcite . to better understand benchmark data, there are methods for analyzing examples in isolation @xcite , characterizing a dataset's data distribution @xcite , using individual models to glean insight about datasets and examples @xcite , and using many models to do the same @xcite . this paper investigates how effectively one method-item response theory (irt)-gives insight into multi-dataset benchmarks. outside of nlp, irt provides insight into educational test questions @xcite and political ideologies of legislators @xcite . in nlp, irt is used to identify helpful training examples @xcite , detect errors in evaluation examples @xcite , and estimate the future utility of examples in benchmarks @xcite . the goal of this paper is to identify the characteristics of multidataset benchmarks that irt methods focus on. are certain datasets easier than others? can clustering highlight dataset or example properties? we hypothesize that examples from similar datasets will cluster together as they should have similar irt characteristics (such as difficulty level) compared to examples from other datasets. however, we do not see any distinct dataset-based clusters in our results. instead, we find that irt characteristics tend to group the examples of similar labels in the same clusters, suggesting that some label types are more difficult or more discriminating regardless of the datasets they belong to. in the rest of this paper, we describe irt methods for benchmark analysis ( §2), our clustering methods ( §3), and our experimental results ( §4). foot_0 ## irt for benchmark analysis in this paper, we adapt irt methods to explain why benchmarks examples are difficult, rather than solely assigning them difficulty values. this section describes the irt models in our experiments and the test-bed we use in our experiments. ## interpreting irt parameters this section explains the methods that our experiments ( §4) use to interpret irt parameters. these methods fall into two categories: (1) methods that correlate examples' irt parameters with dataset or label features and (2) methods that correlate derived textual information with irt parameters (e.g., topic models or embeddings). ## conclusion and future work in this work, our expectation was that datasets would be separable by irt-learned parameters. however, we found that clustering was more interpretable at the label level than the dataset level. future work in irt should better jointly model the characteristics of nlp data as opposed to our methods that train these components in isolation. for example, it may be that the signal provided by dataset properties is second order to labels and our methods may not effectively model this (potential) multi-level relationship. multidimensional irt models that encode relationships between difficulty dimensions ought to better fit the data (e.g., predicting sentiment of restaurant reviews should overlap with hotel reviews, as they both involve service). if these models succeed, they should aid the interpretation of benchmarks. lastly, as models provide more information through initiatives like model cards @xcite , irt could jointly model these properties with latent ability parameters to glean insights into which differences in models yield empirical impacts. 103