Utility is in the Eye of the User: A Critique of NLP Leaderboards (2020)

Abstract: Benchmarks such as GLUE have helped drive advances in NLP by incentivizing the creation of more accurate models. While this leaderboard paradigm has been remarkably successful, a historical focus on performance-based evaluation has been at the expense of other qualities that the NLP community values in models, such as compactness, fairness, and energy efficiency. In this opinion paper, we study the divergence between what is incentivized by leaderboards and what is useful in practice through the lens of microeconomic theory. We frame both the leaderboard and NLP practitioners as consumers and the benefit they get from a model as its utility to them. With this framing, we formalize how leaderboards – in their current form – can be poor proxies for the NLP community at large. For example, a highly inefficient model would provide less utility to practitioners but not to a leaderboard, since it is a cost that only the former must bear. To allow practitioners to better estimate a model’s utility to them, we advocate for more transparency on leaderboards, such as the reporting of statistics that are of practical concern (e.g., model size, energy efficiency, and inference latency).

## introduction the past few years have seen significant progress on a variety of nlp tasks, from question answering to machine translation. these advances have been driven in part by benchmarks such as glue @xcite , whose leaderboards rank models by how well they perform on these diverse tasks. performance-based evaluation on a shared task is not a recent idea either; this sort of shared challenge has been an important driver of progress since muc @xcite . while this paradigm has been successful at driving the creation of more accurate models, the historical focus on performancebased evaluation has been at the expense of other attributes valued by the nlp community, such as fairness and energy efficiency @xcite . for example, a highly inefficient model would have limited use in practical applications, but this would not preclude it from reaching the top of most leaderboards. similarly, models can reach the top while containing racial and gender biases -and indeed, some have @xcite @xcite . microeconomics provides a useful lens through which to study the divergence between what is incentivized by leaderboards and what is valued by practitioners. we can frame both the leaderboard and nlp practitioners as consumers of models and the benefit they receive from a model as its utility to them. although leaderboards are inanimate, this framing allows us to make an apples-to-apples comparison: if the priorities of leaderboards and practitioners are perfectly aligned, their utility functions should be identical; the less aligned they are, the greater the differences. for example, the utility of both groups is monotonic non-decreasing in accuracy, so a more accurate model is no less preferable to a less accurate one, holding all else constant. however, while the utility of practitioners is also sensitive to the size and efficiency of a model, the utility of leaderboards is not. by studying such differences, we formalize some of the limitations in contemporary leaderboard design: we contextualize these limitations with examples from the ml fairness @xcite , green ai @xcite , and robustness literature @xcite . these three limitations are not comprehensive -other problems can also arise, which we leave to be discussed in future work. what changes can we make to leaderboards so that their utility functions better reflect that of the nlp community at large? given that each practitioner has their own preferences, there is no way to rank models so that everyone is satisfied. instead, we suggest that leaderboards demand transparency, requiring the reporting of statistics that are of practical concern (e.g., model size, energy efficiency). this is akin to the use of data statements for mitigating bias in nlp systems @xcite @xcite . this way, practitioners can determine the utility they receive from a given model with relatively little effort. @xcite have suggested that model creators take it upon themselves to report these statistics, but without leaderboards requiring it, there is little incentive to do so. ## utility functions in economics, the utility of a good denotes the benefit that a consumer receives from it (mankiw, 2020). we specifically discuss the theory of cardinal utility, in which the amount of the good consumed can be mapped to a numerical value that quantifies its utility in utils (mankiw, 2020). for example, a consumer might assign a value of 10 utils to two apples and 8 utils to one orange; we can infer both the direction and magnitude of the preference. leaderboards we use the term leaderboard to refer to any ranking of models or systems using performance-based evaluation on a shared benchmark. in nlp, this includes both longstanding benchmarks such as glue @xcite and one-off challenges such as the annual semeval sts tasks @xcite @xcite . this is not a recent idea either; this paradigm has been a driver of progress since muc @xcite . all we assume is that all models are evaluated on the same held-out test data. in our framework, leaderboards are consumers whose utility is solely derived from the rank of a model. framing leaderboards as consumers is unorthodox, given that they are inanimate -in fact, it might seem more intuitive to say that leaderboards are another kind of product that is also consumed by practitioners. while that perspective is valid, what we ultimately care about is how good of a proxy leaderboards are for practitioner preferences. framing both leaderboards and practitioners as consumers permits an apples-to-apples comparison using their utility functions. if a leaderboard were only thought of as a product, it would not have such a function, precluding such a comparison. unlike most kinds of consumers, a leaderboard is a consumer whose preferences are perfectly revealed through its rankings: the state-of-the-art (sota) model is preferred to all others, the second ranking model is preferred to all those below it, and so on. put more formally, leaderboard utility is monotonic non-decreasing in rank. still, because each consumer is unique, we cannot know the exact shape of a leaderboard utility function -only that it possesses this monotonicity. ## utilitarian critiques our criticisms apply regardless of the shape taken by practitioner utility functions -a necessity, given that the exact shapes are unknown. however, not every criticism applies to every leaderboard. stere-oset @xcite is a leaderboard that ranks language models by how unbiased they are, figure 1 : a contrast of possible utility functions for leaderboards and nlp practitioners. note that for leaderboards, utility is not smooth with respect to accuracy; there is only an increase if the model is sota / #2 / #3 on the leaderboard. for practitioners, utility is smooth -any improvement in accuracy yields more utility. the utility functions can take on a variety of shapes but are always monotonic non-decreasing. so fairness-related criticisms would not apply as much to stereoset. similarly, the snli leaderboard @xcite reports the model size -a cost of making predictions -even if it does not factor this cost into the model ranking. still, most of our criticisms apply to most leaderboards in nlp, and we provide examples of well-known leaderboards that embody each limitation. ## conclusion in this work, we offered several criticisms of leaderboard design in nlp. while it has helped create more accurate models, we argued that this has been at the expense of fairness, efficiency, and robustness, among other desiderata. we were not the first to criticize nlp leaderboards @xcite @xcite ), but we were the first to do so under a framework of utility, which we used to study the divergence between what is incentivized by leaderboards and what is valued by practitioners. given the diversity of nlp practitioners, there is no one-size-fits-all solution; rather, leaderboards should demand transparency, requiring the reporting of statistics that may be of practical concern. equipped with these statistics, each user could then estimate the utility that each model provides to them and then re-rank accordingly, effectively creating a custom leaderboard for everyone.