Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarks (2024)

Abstract: The evaluation of natural language processing (NLP) systems is crucial for advancing the field, but current benchmarking approaches often assume that all systems have scores available for all tasks, which is not always practical. In reality, several factors such as the cost of running baseline, private systems, computational limitations, or incomplete data may prevent some systems from being evaluated on entire tasks. This paper formalize an existing problem in NLP research: benchmarking when some systems scores are missing on the task, and proposes a novel approach to address it. Our method utilizes a compatible partial ranking approach to impute missing data, which is then aggregated using the Borda count method. It includes two refinements designed specifically for scenarios where either task-level or instance-level scores are available. We also introduce an extended benchmark, which contains over 131 million scores, an order of magnitude larger than existing benchmarks. We validate our methods and demonstrate their effectiveness in addressing the challenge of missing system evaluation on an entire task. This work highlights the need for more comprehensive benchmarking approaches that can handle real-world scenarios where not all systems are evaluated on the entire task.

## introduction benchmarking and system evaluation are critical processes for assessing the performance of ai systems, providing a standardized means of comparing various models and techniques while keeping track of technological advancements @xcite @xcite . however, evaluating general-purpose systems, such as foundation models used for generative tasks @xcite @xcite @xcite , presents unique * main authors challenges. a single task, metric, or dataset may not be sufficient to effectively gauge their capabilities @xcite @xcite . therefore, it is crucial to develop tools that can benchmark these systems on a multitude of tasks @xcite , enabling a comprehensive assessment of their overall performance @xcite . in recent years, the field of nlp has made significant strides, with frequent emergence of new models @xcite @xcite @xcite @xcite and techniques @xcite . to evaluate the performance of these systems across various tasks, datasets, and metrics @xcite have been created. however, with the increasing complexity of these benchmarks, missing scores has become a significant challenge. missing data can arise from a variety of sources, such as benchmarks that are too large or time-consuming to run (e.g., bigbench has recently introduced minibench for these reasons @xcite ), high costs associated with reproducing experiments (e.g., see in this work, we address benchmarking nlp systems when one or several systems cannot be evaluated on a specific task. we propose the de-velopment of effective methods for aggregating metrics that can handle missing data and enable a comprehensive assessment of system performance. our approach will ensure the reliability and validity of nlp system evaluations and contribute to the creation of benchmarks that can be used to compare and evaluate nlp systems effectively. specifically, our contributions are listed below. ## ranking with missing system evaluation in this section, we will outline our methodology for ranking multiple systems in multi-task benchmarks, even if some systems have not been evaluated on one or more tasks. we use the ranking and ordering notation interchangeably. ## empirical experiments in this section, we benchmark our methods on real rankings. we introduce a dataset with over 100 million scores, surpassing previous datasets by several orders of magnitude (see ssec. 5.1 and ap. c). ## conclusions our study sheds light on the limitations of the conventional mean-aggregation when dealing with missing data. our novel statistical perspective and aggregation procedures that are both robust and grounded in social choice theory. overral, the onelevel aggregation method (σ l ) stands out as the most robust approach.