It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performance (2023)

Abstract: Progress in NLP is increasingly measured through benchmarks; hence, contextualizing progress requires understanding when and why practitioners may disagree about the validity of benchmarks. We develop a taxonomy of disagreement, drawing on tools from measurement modeling, and distinguish between two types of disagreement: 1) how tasks are conceptualized and 2) how measurements of model performance are operationalized. To provide evidence for our taxonomy, we conduct a meta-analysis of relevant literature to understand how NLP tasks are conceptualized, as well as a survey of practitioners about their impressions of different factors that affect benchmark validity. Our meta-analysis and survey across eight tasks, ranging from coreference resolution to question answering, uncover that tasks are generally not clearly and consistently conceptualized and benchmarks suffer from operationalization disagreements. These findings support our proposed taxonomy of disagreement. Finally, based on our taxonomy, we present a framework for constructing benchmarks and documenting their limitations.

## introduction claims of progress in nlp are often premised on how models perform on benchmarks for various nlp tasks foot_0 (e.g., coreference resolution, question answering) @xcite @xcite . benchmarks instantiate a task with a specific format, dataset of correct input-output pairs, and an evaluation metric @xcite , and they are intended to serve as measurement models for performance on the task. on the one hand, benchmarks allow for performance results to be easily compared across a rapidly-rising number of nlp models @xcite . additionally, many nlp benchmarks are easily accessible via open-source platforms @xcite , which reduces the need of practitioners to construct new evaluation datasets and metrics from scratch. however, prior research has identified numerous threats to the validity of benchmarks (i.e., how well benchmarks assess the ability of models to correctly perform tasks). these threats include spurious correlations and poorly-aligned metrics (refer to however, little literature has surfaced sources of disagreement among nlp practitioners about benchmark validity, which is paramount to contextualize progress in the field. hence, we develop a taxonomy of disagreement based on measurement modeling (from the social sciences @xcite ). our taxonomy critically distinguishes between disagreement in how tasks are conceptualized and how measurements of model performance are operationalized @xcite . it thereby goes beyond prior examinations of nlp benchmarking methodology, which assume that tasks are generally clearly and consistently understood from person to person @xcite . this is important because our taxonomy captures that practitioners may perceive a benchmark for a task to have poor validity because they conceptualize the task differently than the benchmark creators do, and not simply because of the creators' oversights or mechanistic failures when constructing the benchmark. (we validate this hypothesis empirically in § 5.1.) furthermore, our taxonomy addresses that benchmarks can shape practitioners' conceptualization of a task. ultimately, our taxonomy equips practitioners with a language to structure their thinking around and communicate their perceptions of benchmark validity. to provide evidence for our taxonomy, we conduct a survey of practitioners (@xmath0: how contested tasks are and the quality of common benchmark formats, datasets, and metrics for tasks. we further conduct a metaanalysis of relevant literature to understand how tasks are conceptualized. our meta-analysis and survey across eight tasks, ranging from coreference resolution to question answering, uncover that tasks are generally not clearly and consistently conceptualized and benchmarks suffer from operationalization disagreements. these findings support our taxonomy of disagreement. finally, based on our taxonomy, we present a framework for constructing benchmarks and documenting their limitations. ## taxonomy of disagreement we present our taxonomy of disagreement about the validity of nlp benchmarks (displayed in figure 1 ). drawing from measurement modeling @xcite , our taxonomy critically distinguishes between disagreement in: 1) how a task τ is conceptualized, and 2) how a benchmark b τ operationalizes measurements of model performance on τ . we provide evidence for our taxonomy in § 5, via our survey results and a meta-analysis of relevant literature. ## survey methodology with our taxonomy in mind, we conduct a survey of nlp practitioners 3 (@xmath1, practitioners' perceptions of: (1) the extent to which the tasks appear to have a clear and consistent conceptualization and (2) the quality of benchmarks (with respect to task format, dataset, and metric). we ultimately chose to include the following tasks in our survey: sentiment analysis (sent), natural language inference (nli), question answering (qa), summarization (sum), machine translation (mt), named-entity recognition (ner), coreference resolution (coref), and dependency parsing (dep). we detail our task selection protocol in appendix d. ## a framework for nlp benchmarks towards better documenting benchmarks' conceptualization and operationalization, we encourage benchmark creators to answer the questions in we hope that this reflection will benefit the nlp community in the following ways: • reduce overhyping: by being transparent about and defining the model capabilities that benchmarks are intended to assess, as well as documenting benchmark validity issues, benchmark creators will: 1) not misrepresent model capabilities, and 2) remind people to be careful about extrapolating benchmark performance results. ## conclusion we develop a taxonomy of disagreement (based on measurement modeling) which distinguishes between how tasks are conceptualized and how measurements of model performance are operationalized. to provide evidence for our taxonomy, we conduct a survey of practitioners and meta-analysis of relevant literature. based on our taxonomy, we propose a framework for the creation of benchmarks and the documentation of their limitations. future work includes studying task conceptualization via benchmark inter-annotator disagreement.