ECBD : Evidence-Centered Benchmark Design for NLP (2024)

Abstract: Benchmarking is seen as critical to assessing progress in NLP. However, creating a benchmark involves many design decisions (e.g., which datasets to include, which metrics to use) that often rely on tacit, untested assumptions about what the benchmark is intended to measure or is actually measuring. There is currently no principled way of analyzing these decisions and how they impact the validity of the benchmark’s measurements. To address this gap, we draw on evidence-centered design in educational assessments and propose Evidence-Centered Benchmark Design (ECBD), a framework which formalizes the benchmark design process into five modules. ECBD specifies the role each module plays in helping practitioners collect evidence about capabilities of interest. Specifically, each module requires benchmark designers to describe, justify, and support benchmark design choices—e.g., clearly specifying the capabilities the benchmark aims to measure or how evidence about those capabilities is collected from model responses. To demonstrate the use of ECBD, we conduct case studies with three benchmarks: BoolQ, SuperGLUE, and HELM. Our analysis reveals common trends in benchmark design and documentation that could threaten the validity of benchmarks’ measurements.

## introduction while benchmarking has long been seen as critical to gauging progress in natural language processing (nlp) and guiding model selection for downstream applications, assessing the quality of a benchmark remains a persistent challenge. do benchmark measurements-most often in the form of numerical scores-provide meaningful insights about the evaluated models and their capabilities? how valid are these measurements? for what purposes are they useful? the field of nlp lacks a systematic way of reflecting on these important questions. at the same time, as nlp models are increasingly believed to be more performant and to exhibit a wider range of capabilities, evaluation in nlp has shifted from measuring model performance on a specific dataset for a single task to using large benchmarks that cover multiple tasks (e.g., glue @xcite , superglue @xcite , big-bench @xcite , helm @xcite ). these benchmarks are increasingly larger and more ambitious (e.g., helm aims to "assess language models in their totality"), covering ever-growing numbers of tasks, datasets, and metrics aimed at measuring an increasing number of capabilities-i.e., abilities or behaviors that researchers believe the models exhibit or might exhibit. this trend further increases the complexity of assessing benchmark quality. such issues with assessing measurement quality do not only concern nlp practitioners. researchers and practitioners in educational testing often face similar questions: do students' exam results provide meaningful insights about their ability in, for example, reading comprehension? can these results be used to determine whether a student needs remedial classes? in this work, we take inspiration from the evidence-centered design (ecd) framework in educational testing @xcite . ecd views testing as the process of gathering evidence about students' abilities, and provides guidance on the creation, documentation, and validation of educational tests. we draw an analogy between educational tests and nlp benchmarks and propose the evidence-centered benchmark design (ecbd) framework, in which we view benchmarking as the process of gathering evidence from objects of evaluation (e.g., language models) about whether or to what degree they have some capabilities of interest. ecbd unpacks and formalizes benchmark design decisions into five modules, each having a specific role in supporting the process of collecting necessary evidence (see figure 1 ). for each module, we provide guiding questions that help benchmark creators document, justify, and validate their design choices. these same questions can also guide benchmark users in analyzing and documenting existing benchmarks to better understand their limitations and how to appropriately use them: what are the design decisions shaping the benchmark? why did its creators make these decisions? and what evidence do they provide to support their decisions? by doing so, ecbd also supports added transparency about what benchmarks measure, and when and how they can be used. in turn, this transparency can also help our community contest and mitigate issues with current benchmarks. to illustrate ecbd's usage for benchmark analysis, we organize these questions into a worksheet (appendix a) and apply it to three different benchmarks: boolq @xcite , superglue @xcite , and helm @xcite . 1 through this exercise, we uncover common issues, such as poor conceptualization of capabilities, that threaten the validity of these benchmarks' measurements. in general, we find that these benchmarks lack justification and validation, as well as appropriate documentation of the many of their design choices made during their creation. this trend of evaluating models across an increasing number of datasets and tasks continues, with recent benchmarks such as gem @xcite , covering 40 language generation tasks. collaborative and evolving benchmarks such as big-bench @xcite , now counting more than 200 tasks in its repository, foot_0 call on the research community to contribute new tasks. we designed ecbd to encourage more critical analyses of these increasingly complex benchmarks and a deeper reflection on how benchmark design decisions might affect the validity of benchmark measurements. critiques and meta-analyses prior work has also surveyed and critiqued both nlp and machine learning (ml) evaluation more generally. bowman and dahl (2021) outline a list of criteria they argue that useful benchmarks for natural language understanding (nlu) should meet, including validity. similarly, @xcite highlights the disconnect between benchmark results and real world impacts-e.g., does a given increase on the benchmark actually lead to positive impact in the tested domain of application?-while liao and xiao (2023) argue for centering language models' evaluation on how models will be used in practice. analyses of benchmarks in nlp evaluation have raised concerns about annotation artifacts @xcite , threats to validity @xcite , lack of justification surrounding design choices @xcite , inconsistent results from benchmarks aimed at measuring similar things @xcite , and benchmarks' lack of robustness @xcite . nlp and ml documentation various documentation guidelines have been proposed for nlp and ml datasets, models, and systems @xcite @xcite @xcite . datasheets for datasets @xcite provides a standardized process for documenting ml datasets, formulated as a list of questions (e.g., "does the dataset contain data that might be considered confidential?"). in nlp, data statements for nlp @xcite ) contains guidelines more specific to speech and text data, asking practitioners to document details about how data is curated, such as the demographics of the speakers included. similarly, model cards @xcite and factsheets @xcite have been proposed to support better model and ai service documentation. our work contributes to these efforts by providing a set of guidelines for documenting nlp benchmark design choices. our focus, however, is on design choices that affect how benchmarks are used to gather the necessary evidence about whether, or to what degree, an evaluated model has some capabilities of interest. our framework also guides the process of gathering the required validity evidence for benchmark measurements. ## evidence-centered benchmark design we consider benchmarking as the process of gathering capability evidence from objects of evaluation (e.g., lms)-i.e., evidence about whether or to what degree those objects have some capabilities of interest. evidence-centered benchmark design (ecbd) structures this process into five modules, foot_1 each of which has a specific role in helping collect the necessary capability evidence: the capability module ( §3.1), the content module ( §3.2), the adaptation module ( §3.3), the assembly module ( §3.4), and the evidence module ( §3.5). see figure 2 for an overview of ecbd. for each module, ecbd breaks down the design process into three required actions. to guide benchmark creation, ecbd requires benchmark creators to i) describe their design choices; ii) justify these choices, forming hypotheses about how they ensure that the module accomplishes its role; and iii) provide support for these hypotheses, which requires gathering either evidence showing that underlying constructs are well-defined and well-grounded, or validity evidence-i.e., evidence about how benchmark design choices might support or threaten the validity of the resulting benchmark measurements. such evidence can be theoretical (e.g., work conceptualizing a capability) or empirical (e.g., experiments correlating benchmark measurements with some ground truth). in addition to helping benchmark creators reflect on their design choices, ecbd can also support benchmark analysis by benchmark users or third parties by drawing attention to whether and how benchmark creators describe and justify their design decisions, and to what extent there is validity evidence supporting these decisions. we also organize ecbd in a worksheet of 20 questions to facilitate its use (appendix a). benchmark creators can use this worksheet while constructing a benchmark, with each question meant to encourage them to reflect on their decisions and make their assumptions explicit. when analyzing existing bench-marks, the worksheet can be completed by referring to available documentation of the benchmark under analysis (e.g., papers, blog posts). benchmark intended use while establishing the intended use of a benchmark is not a ecbd module per se, it is a critical step that should precede benchmark design or analysis using ecbd modules. this first step is important as the validity of the resulting benchmark measurements often concerns whether the benchmark can be used as intended. articulating a benchmark's intended use requires specifying: i) what are the intended objects of evaluation (analogously, "test takers")? ii) who are the intended users of the benchmark? and iii) how should they interpret and use the benchmark results? a benchmark's design choices should be assessed with respect to its intended use. this intended use might be closely connected to the use context of the evaluated model (e.g., aiming to inform deployment decisions of automatic summarizers for medical records), or it might not be directly connected to any downstream applications (e.g., aiming to compare language models' and humans' linguistic abilities). if the intended use is not clearly stated, benchmark creators risk making choices simply because they are convenient or common practices, likely resulting in a benchmark that does not serve any particular purpose. furthermore, if the intended use is not clearly communicated to potential users, they could unintentionally misuse the benchmark (e.g., use it to evaluate other objects than those intended), or misinterpret the resulting measurements. ## case studies to illustrate how our framework guides benchmark analysis and helps foreground possible validity concerns, we apply the ecbd worksheet to analyze helm @xcite , superglue @xcite , and boolq @xcite . ## conclusion to guide nlp benchmark creation and analysis, we take inspiration from the evidence-centered design framework from the field of educational testing to propose ecbd (evidence-centered benchmark design). our framework formalizes the benchmark design process into five modules that each play a critical role in gathering valid and useful capability evidence-i.e., evidence about whether or to what degree objects of evaluation have some capabilities of interest. we demonstrated its utility by using it to analyze boolq, superglue, and helm, finding common practices that threaten the validity of their measurements. for example, for these benchmarks we found more of a focus on describing design choices (e.g., which dataset/metric is used), and less on justifying them and their role in the benchmark. gathering validity evidence is also rare. future directions include further analyses of our framework's utility in guiding the creation of benchmarks. it is also important to understand how ecbd helps increase transparency and supports practitioners in achieving a greater understanding of benchmark results and their limitations. future work might also examine this, for example through user studies with benchmark creators and users. as ecbd does not constrain the model inputs and outputs to be textual, we also see it to be applicable or adaptable to multi-modal nlp benchmarks, or to other areas in ml and ai.