Vote’n’Rank: Revision of Benchmarking with Social Choice Theory (2023)

Abstract: The development of state-of-the-art systems in different applied areas of machine learning (ML) is driven by benchmarks, which have shaped the paradigm of evaluating generalisation capabilities from multiple perspectives. Although the paradigm is shifting towards more fine-grained evaluation across diverse tasks, the delicate question of how to aggregate the performances has received particular interest in the community. In general, benchmarks follow the unspoken utilitarian principles, where the systems are ranked based on their mean average score over task-specific metrics. Such aggregation procedure has been viewed as a sub-optimal evaluation protocol, which may have created the illusion of progress. This paper proposes Vote’n’Rank, a framework for ranking systems in multi-task benchmarks under the principles of the social choice theory. We demonstrate that our approach can be efficiently utilised to draw new insights on benchmarking in several ML sub-fields and identify the best-performing systems in research and development case studies. The Vote’n’Rank’s procedures are more robust than the mean average while being able to handle missing performance scores and determine conditions under which the system becomes the winner.

## introduction benchmarking has evolved as a conventional practice for accelerating the development of generalisable systems in different applied areas of machine learning (ml). benchmarks are typically designed as a collection of datasets, corresponding task-specific evaluation metrics, and a criterion for summarising the overall performance on the tasks @xcite . the benchmark holders provide public leaderboards, which are utilised by ml researchers and practitioners for comparing novel systems against one another, and, if applicable, human baselines, as well as selecting the bestperforming ones for practical purposes. according to the benchmark sharing platform paperswith-code foot_0 , the community has put much effort into creating more than 10, 000 influential benchmarks in natural language processing (nlp), computer vision, and knowledge graphs, to name a few. criticism of the benchmark pillars. the benchmark methodological foundations have received wide criticism from academic and industrial communities @xcite . the criticism covers various aspects of benchmarking, raising concerns about the construct validity @xcite , fragility of the design and task choices @xcite , data leakage and annotation artifacts @xcite , sota-chasing tendencies at the cost of large carbon footprints @xcite , and low reproducibility of the reported results @xcite , inter alia. recommendations proposed in these studies are of utmost importance to benchmark holders, system users, and developers. however, little attention has been paid to a more nuanced methodological question: how to aggregate performance scores in multi-task benchmarks? limits of canonical aggregation. the appropriateness of mean aggregation in multi-task ml problems is an ongoing debate in the community. the mean aggregation procedure implies that all task metrics are homogeneous @xcite . otherwise, it is recommended to evaluate the statistical significance of differences between models with non-parametric tests @xcite . in practice, the nlp glue-style benchmarks @xcite @xcite use arithmetic average to rank models over heterogeneous metrics, which may lead to biased evaluation and subjective outcomes @xcite . the top-leading systems may dominate the others only on the outlier tasks @xcite , and their ranking is inconsistent with other pythagorean means @xcite . at the same time, the mean aggregation ignores the relative ordering and relies on the absolute score difference @xcite , equally treating tasks of different complexity @xcite and from different domains @xcite . ## case studies this section describes four case studies on three nlp and multimodal benchmarks. our main objective here is to re-interpret the benchmarking trends under the social choice theory. we provide a brief description of the benchmarks below. ## recommendations for rules choice the information about the voting rules' properties can be used to choose the most suitable one to the user's preferences @xcite . we also provide the following recommendations. ## conclusion and future work this paper introduces novel aggregation procedures to rank and select the best-performing systems under the social choice theory principles. our approach provides an alternative perspective of system evaluation in benchmarking and overcomes the standard mean aggregation limitations. our case studies show that vote'n'rank provides interpretable decisions on the best and worst systems whilst accounting for missing performance scores and potential user preferences. the framework allows for finding scenarios in which a given system dominates the others. we provide recommendations based on the rules' properties and scenarios of the intended framework's application. the application scope of vote'n'rank is not limited and may be easily extended to other applied ml areas. in our future work, we hope to explore applications of the social choice theory in the multilingual and multimodal benchmarking.