Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards? (2021)

Abstract: Leaderboards are widely used in NLP and push the field forward. While leaderboards are a straightforward ranking of NLP models, this simplicity can mask nuances in evaluation items (examples) and subjects (NLP models). Rather than replace leaderboards, we advocate a re-imagining so that they better highlight if and where progress is made. Building on educational testing, we create a Bayesian leaderboard model where latent subject skill and latent item difficulty predict correct responses. Using this model, we analyze the ranking reliability of leaderboards. Afterwards, we show the model can guide what to annotate, identify annotation errors, detect overfitting, and identify informative examples. We conclude with recommendations for future benchmark tasks.

## leaderboards are shiny leaderboard evaluations-for better or worse-are the de facto standard for measuring progress in question answering @xcite and in many nlp tasks @xcite ). an unfortunate side effect of leaderboard popularity is sota-chasing, often at the expense of carefully inspecting data and models @xcite . for example, the same "super-human" models that top question answering leaderboards @xcite often fail spectacularly @xcite ) by learning non-generalizable statistical patterns @xcite . finally, focusing solely on metrics conflates progress on a specific task with progress on realworld nlp problems behind the task @xcite . plainly, focusing on headline sota numbers "provide(s) limited value for scientific progress absent insight into what drives them" and where they fail @xcite . in this work we take leaderboards "as they are," and imagine how they might better support research. leaderboards establish differences between models on a fixed task. hence, leaderboards should enable and encourage the comparison of models and inspection of examples. and leaderboards should also signal when they have outlived their usefulness @xcite . ## a generative story for leaderboards leaderboards are a product of the metrics, evaluation data, and subjects (machine or human) who answer items (figure 2 ). for concreteness, let's assume that we have a question-answering task and two subjects: ken, who is good at trivia, and burt, who is not. in the simplest irt models, each subject j has a random variable θ j corresponding to their skill: ken's is big, burt's is small. but you cannot know that until you start asking them questions of varying difficulty β i . harder questions have a higher difficulty ("what is the airspeed of an unladen swallow") than easy ones ("who is buried in grant's tomb"). the bigger the margin between a subject's skill θ j and an item's difficulty β i , θ j -β i , the more likely that subject j responds correctly p i,j (r i,@xmath0. this is the simplest irt model, which we call irt-base. generally, given n test items @xmath1, . . . , x n ) and m subjects @xmath2, . . . , s m ), where each subject answers every item, we want to estimate subject skills and item difficulties. to discover the random variables that best explain the data, we turn to probabilistic inference @xcite . two additional random variables further improve dad: discriminability γ i and feasibility λ i . we first consider discriminability and the margin between a question's difficulty β i and a subject's skill θ j . a discriminative question is challenging but can still be answered correctly by a strong subject. if ken's ability is higher than most items' difficulty (θ j -β i is large), item discriminability multiplies this gap by γ i in a model called irt-disc. questions with low γ i are low quality: they have annotation error or do not make sense. another way of capturing poor quality questions is the feasibility λ i . for example, if the question "who was the first president" has the answer rajendra prasad, the question has an unstated implicit assumption that subjects must guess what country or company the question is about. in the model irt-feas, if a large fraction of subjects all get an item wrong, everyone's probability of getting the item right is capped at λ i . in nlp terms, 1 -λ i corresponds to the prevalence of annotation errors that lead to unsolvable items. having introduced all of the constituent elements of the model, we can now present the full generative model: ## ranking and comparing subjects fundamentally, the objective of comparative evaluations like leaderboards is to decide whether model a is better than model b. a thread of nlp has rightfully advocated for adding rigor to these decisions using statistics @xcite where the objective is to infer a true score t from the observed test score @xmath3, uniform across subjects. however, in educational testing-a field measuring skill and knowledge in humans-irt is a primary measurement instrument (hambleton, 1991, p. 2). a major motivation for irt is that subjects of different skill have different errors. irt explicitly accounts for the bandwidth-fidelity dilemma @xcite : items can either accurately measure a narrow ability range (fidelity) or inaccurately measure large ability ranges (bandwidth). 6 this section and the next contrast methods for identifying the best model and advocate for irt. implicit in nearly all leaderboard evaluations is ranking models by a statistic such as the average accuracy. as we show in §4, naïve rankings are noisier than irt rankings. ## irt for leaderboards leaderboards should: (1) reliably and efficiently rank better models ahead of worse models @xcite and (2) guide inspection of items and subjects ( §5). the first ameliorates the unavoidable randomness of finite evaluations while the second enables error analysis @xcite and model probing @xcite . first we verify that irt models accurately predict the responses of subjects ( §4.2). next, a ranking stability analysis shows that irt has modestly better reliability than classical rankings ( §4.2.3). lastly, using irt to actively sample items for annotation yields rankings with better correlation to complete test data ( §4.4). ## qualitative insights on leaderboards dad also helps qualitative analysis of items and subjects. first, irt identifies overfitting and generalizes partitioning datasets by difficulty. then we show that-like in educational testing-irt identifies good and bad items. ## related work dad draws together two primary threads: we use irt to understand datasets, which has been applied to other nlp tasks, and apply it to improving leaderboards. finally, we explore how the insights of irt can improve not just the analysis of test sets but to improve the construction of test sets. irt in nlp irt is gaining traction in machine learning research @xcite where automated metrics can be misleading @xcite : machine translation (hopkins and may, 2013) and chatbot evaluation (sedoc ## conclusion this paper advocates incorporating decades of research in crafting education tests to improve how we evaluate the capabilities of nlp models. we propose and validate an alternate irt ranking method for leaderboard evaluations, show it can guide annotation, detect annotation error, and naturally partition evaluation data. just as educators moved from classical testing to irt, the nlp community should consider future evaluations with irt.