Elo Uncovered: Robustness and Best Practices in Language Model Evaluation (2023)

Abstract: In Natural Language Processing (NLP), the Elo rating system, well-established for ranking dynamic competitors in games like chess, has seen increasing adoption for evaluating Large Language Models (LLMs) through “A vs B” paired comparisons. However, while popular, the system’s suitability for assessing entities with constant skill levels, such as LLMs, remains relatively unexplored. Our study investigates the sensitivity and reproducibility of Elo scores for LLMs, integrating both synthetic and human feedback. We show that Elo ratings for LLMs stabilize with 100 or more comparison permutations. A lower K-factor is preferable for closely matched models, whereas a higher K-factor better distinguishes models with clear performance differences. We also report that transitivity (A B and B C implies A C) does not consistently hold, particularly when models demonstrate similar performance. Our empirical findings provide guidelines for more reliable LLM evaluation.

## introduction in the rapidly evolving field of natural language processing (nlp), the task of accurately and reliably evaluating llms has become increasingly challenging @xcite @xcite @xcite . human feedback has emerged as an indispensable tool in this performance assessment process, serving as a qualitative metric that captures nuances that automated scoring mechanisms often fail to address @xcite @xcite @xcite . these human-centered evaluations, highly valuable to the overall progress of the nlp field, typically adopt an "a vs b" comparative setup, turning evaluations into a zero-sum game between language models. this paired feedback structure @xcite naturally lends itself to the elo rating system @xcite , originally designed for ranking chess players for better matchmaking. variants such as glicko @xcite @xcite and trueskill ™ @xcite have incorporated more complex statistical methods into the original elo framework, to address some of the limitations of the elo system, particularly in the context of games with more than two players or teams, or games with more complex outcomes than just win or loss. there is ongoing research into the efficacy of these systems in diverse and dynamic environments, and new methods continue to be developed @xcite . despite these limitations, the core principles of elo have proven to be incredibly resilient and adaptable. as a result, the elo system has found diverse applications, from predicting sports events outcomes @xcite @xcite , and facilitating matchmaking in massively multiplayer online games like starcraft ii and dota @xcite @xcite , to its recent use in the evaluation of llms @xcite @xcite @xcite @xcite . however, its application to llm evaluations landscape has been insufficiently studied. unlike dynamic competitors that evolve, llms have static capabilities and operate in a time-agnostic context. in this setting, not only are llm evaluations unconstrained by tournament timelines or predefined match sequences, but the ordering of matches can also significantly influence the final elo scores and, consequently, models rankings. this oversight is especially concerning, given the direct impact of elo system rankings on both research directions and real-world applications in nlp. this study aims to close this research gap by scrutinizing both the reliability and limitations of the elo rating system when applied to llms. through theoretical and empirical analyses grounded in collected human feedback data, our contributions provide a comprehensive understanding of when and how to reliably employ the elo system for llm evaluation, thus offering valuable guidelines for researchers and practitioners in the nlp field. we find that elo ratings are far from stable, and are highly sensitive to permutation of ordering and hyperparameter choice. desirable properties such as transitivity are not always guaranteed, and can be unreliable unless there is comprehensive human feedback data for all unique pairwise comparisons among models in the feedback pool. the sensitivity of elo ratings becomes more pronounced when dealing with models that exhibit similar performance levels. we illustrate the best practices for mitigating these sensitivities by offering guidelines for hyperparameter selection and matchmaking scenarios. ## elo algorithm explained we provide the mathematical formulation of the elo algorithm, contextualized to the setting of llm evaluation. in this formulation, let m be a set of models and each model @xmath0. ## synthetic human feedback this time-agnostic and independent setup of llm evaluations resembles a bernoulli process @xcite , a sequence of independent experiments, each with two possible outcomes; one model outperforming the other. we use this synthetic setting where we can control characteristics of the distribution to evaluate different desirable properties of a rating system. in this controlled setting where we can precisely control the data distribution, we ask whether the elo score respects transitivity and quantify the degree of sensitivity to ordering of models and hyperparameter choices like the k-factor. ## validation on real-world human feedback building on the insights gained from our synthetic data experiments, we extend our validation efforts to include real-world human feedback. our objective is two-fold: first, to ascertain how the demonstrated properties established using synthetic data generalize to real human annotations; and second, to evaluate the elo rating system's utility for assessing large language models (llms) under practical conditions. experimental setup. our study leverages human feedback data previously collected to explore data prioritization in language model evaluations. for details about our pool of prompts and models, completion generation, and annotation collection process, we refer the reader to the experimental setup section of our previous work @xcite . we focus on models from the well-established dolly @xcite and @xcite families, ensuring relevance to the broader nlp community. the evaluation dataset consists of 400 prompts, with 100 randomly chosen from the soda @xcite dataset and 100 from each of the common-senseqa @xcite , commongen @xcite , and adversarialqa @xcite subsets, all of which are part of the public pool of prompts (p3) dataset @xcite . this ensures a diverse set of evaluation scenarios for a comprehensive assessment of the models' capabilities. consistent with our synthetic data methodology, tie outcomes have been excluded from this analysis to focus specifically on the implications for the robustness of elo scores. in line with our previous analyses, we continue to explore the influence of variations in n @xmath1, 100} and the k-factor (ranging from 1 to 36) on the robustness and reliability of elo scores. the win rates for each model, derived from human evaluations, are summarized in key findings. our analysis of real-world human feedback data reveals that the stability of elo ratings is influenced by the disparities in win rates and the choice of hyperparameters k-factor and n perms . in situations where win rates show a significant discrepancy, such as in our flan family experiment, elo ratings remain notably consistent across different k-factors and n perms configurations (see figure 7 ). on the other hand, in cases like the dolly family experiment where win rates are closely matched, the elo rating system exhibits higher volatility at n @xmath2. regarding the conservation of transitivity, our findings indicate that this property is not universally maintained in real-world human evaluations, as observed in synthetic data in section 4. the relative rankings of models that perform similarly are sensitive to the choice of hyperparameters k-factor and n perms . consequently, one should exercise caution in drawing conclusions from the elo scores when comprehensive paired comparison data, as dictated by the combination formula 5, is not available. our observations are in line with the trends seen in our synthetic data experiments. ## empirical guidelines for robust elo-based evaluation we consolidate the following best practices for a reliable and robust elo-based evaluation of language models: these guidelines serve as empirically-grounded recommendations to improve the robustness and interpretability of elo-based evaluations for llms. following these best practices will help in yielding more reliable conclusions on models' performance via human judgment. ## conclusion this paper provides a comprehensive study on the reliability of the elo rating system for evaluating llms using human feedback. we identify various factors that influence the robustness of elo ratings and offer guidelines for their effective application in real-world scenarios. while our findings lay down an essential framework, they are by no means exhaustive. future work could extend the present study by considering tie outcomes and adopting multi-category bernoulli synthetic data to more closely simulate the varied landscape of human feedback. such extensions could provide additional insights into the convergence properties of the elo rating system in the fast-evolving landscape of language models.