The Validity of Evaluation Results: Assessing Concurrence Across Compositionality Benchmarks (2023)

Abstract: NLP models have progressed drastically in recent years, according to numerous datasets proposed to evaluate performance. Questions remain, however, about how particular dataset design choices may impact the conclusions we draw about model capabilities. In this work, we investigate this question in the domain of compositional generalization. We examine the performance of six modeling approaches across 4 datasets, split according to 8 compositional splitting strategies, ranking models by 18 compositional generalization splits in total. Our results show that: i) the datasets, although all designed to evaluate compositional generalization, rank modeling approaches differently; ii) datasets generated by humans align better with each other than with synthetic datasets, or than the latter among themselves; iii) generally, whether datasets are sampled from the same source is more predictive of the resulting model ranking than whether they maintain the same interpretation of compositionality; and iv) specific lexical items in dataset impacts the measurement consistency. Overall, our results demonstrate that much work remains to be done when it comes to assessing whether popular evaluation datasets measure what they intend to measure, and suggests that elucidating more rigorous standards for establishing the validity of evaluation sets could benefit the field.

## introduction over the past few years, nlp has made astonishing progress on almost all language-related tasks proposed by the community. concurrently, a plethora of benchmark datasets has emerged for evaluating the skills of nlp models and exposing their strengths and weaknesses @xcite . these datasets focus on a variety of different aspects of model capabilities, that are increasingly not mutually exclusive: oftentimes, multiple benchmarks are available that target the same capability or skill, using (slightly) different metrics, design choices, and/or conceptual approaches. for instance, @xcite report that many recent studies on generalization used different shift sources to study the same types of generalization (see figure 2 ). 2 however, somewhat surprisingly, despite a wealth of work in the domain of evaluation and generalization, there is very little research that assesses whether multiple datasets designed to measure the same ability also yield the same conclusions. this makes it difficult for practitioners to conduct informed evaluation dataset selection and, perhaps even more concerning, impedes our understanding of how well different datasets measure what they intend to measure. while establishing construct validity and construct reliability -for instance through comparing the results of tests with other tests that intend to measure the same thingis common practice in the social sciences @xcite , it is not the standard in the field of nlp. in this work, we argue that establishing such standards is much needed in our field, and we present a detailed set of experiments that assesses construct validity in the domain of compositional generalization. following liu et al. (2021), we use concurrence to measure the extent to which 8 different compositional splitting strategies for 4 different datasets -scan, geoquery, cogs, and spider -provide similar rankings for 6 different modeling approaches -bart, t5, transformer, uni-and bilstms, and neural-btg. we find that, in general, the conclusions drawn from one dataset split typically do not align with the results from another dataset split. in a range of experiments, we explore if that could be attributed to whether the underlying data are synthetic or human-generated, to the compositional splitting strategy is used to create the data (a.k.a. what interpretation of compositionality), or to uncontrolled exposure to lexical items that also occurred during pretraining. we find that concurrence values are generally low: only 10 out of 153 pairs of dataset splits have a concurrence value that surpasses the threshold for high concurrence. furthermore, results from human-authored datasets concur much more than results from synthetic datasets. on the contrary, dataset splits that share the same interpretation of compositionality -as defined by their splitting strategy -hardly concur with each other: the underlying data plays a more important role in model rankings. lastly, aligned with the findings of @xcite , we find that carefully controlling the lexical items in a compositional split has a positive impact on concurrence. overall, our results suggest that much work remains to be done to evaluate compositional generalization, and more generally that having more rigorous standards for establishing the validity of evaluation sets should be prioritized in the future. 2 related work in this section, we provide an overview of datasets commonly used for assessing compositional generalization, and we discuss previous attempts to compare performance across benchmarks. ## methodology we compare how the conclusions drawn from 18 different compositional generalization splits -defined over 4 different datasets with 8 compositional splitting strategies -compare across 6 modeling approaches. in this section, we describe the datasets and modeling approaches we consider and provide details on training and hyperparameter selection. ## results we now present our results, starting with a discussion of the performance of models on the datasets ( §4.1) and the concurrence scores between the performances ( §4.2), we then proceed to look at the relationship between synthetic and natural compositionality datasets ( §4.3), and how this interacts with the choice of definition of compositionality and underlying dataset ( §4.4). we finish our results section with a short investigation into the impact of the choice of lexical items in data ( §4.5). ## conclusion in this paper, we explored how different evaluation choices impact the conclusions drawn from the experiments evaluating compositionality. using compositional generalization datasets and models ranging from trained-from-scratch to pretrained, we conduct a series of experiments to understand whether datasets consistently rank models in terms of their generalizability, and we find little consistency. when we perform further analysis to try to better understand this inconsistency, we find that comparing within the training setting (pretrained v.s. trained-from-scratch) or data creation type (synthetically generated v. naturally generated) does 2 . the color of dots indicates the type of split pairs. the triangle-shape dots indicates the values of self-concurrence.