Post T uring: Mapping the landscape of LLM Evaluation (2023)

Abstract: In the rapidly evolving landscape of Large Language Models (LLMs), introduction of well-defined and standardized evaluation methodologies remains a crucial challenge. This paper traces the historical trajectory of LLM evaluations, from the foundational questions posed by Alan Turing to the modern era of AI research. We categorize the evolution of LLMs into distinct periods, each characterized by its unique benchmarks and evaluation criteria. As LLMs increasingly mimic human-like behaviors, traditional evaluation proxies, such as the Turing test, have become less reliable. We emphasize the pressing need for a unified evaluation system, given the broader societal implications of these models. Through an analysis of common evaluation methodologies, we advocate for a qualitative shift in assessment approaches, underscoring the importance of standardization and objective criteria. This work serves as a call for the AI community to collaboratively address the challenges of LLM evaluation, ensuring their reliability, fairness, and societal benefit.

## introduction alan turing began his famous article "computing machinery and intelligence" @xcite by stating that it is extremely difficult to formulate objective definitions of the terms "machine" and "think" in the context of the question: can machines think? instead, he proposed looking for an answer to another question: can machines reliably imitate human dialogue? back then, in 1950, the answers to both questions were so far apart from us that the difference between them was insignificant, and this substitution helped to set the "north star metric" for a long time, the direction of development for the entire field of research, including dialog systems, human-machine interfaces, and various kinds of ai. a possible reason for this success is that a practical solution to this imitation task implies the need to fulfill (to some extent) several complex conditions simultaneously, including natural language proficiency, interactivity, and effective grasp on the context of the conversation. moreover, since the initial setup does not specify the fixed protocol, other strong requirements may be implied, such as common knowledge of the world, reasoning, abstract or creative thinking, concept of causality, and so on, depending on the particular interviewer's questions. now, 73 years after turing's paper, modern systems have greatly evolved, successfully mimicking human-like behaviors and interactions. the first officially documented machine passed the turing test in 2014 @xcite , long before the era of large language models. since then, the quality of dialog simulation and text generation in general has increased even more, so the turing test has long since ceased to serve as a reliable proxy for evaluation of modern systems. instead, a wide variety of approaches are used in practice, aimed to assess different individual abilities and properties of a system. however, we have neither a unified system of criteria nor clear formulation of the evaluation goals. in the meantime this new evaluation methodology will not only influence the trajectory of ai research but will also have broader implications. thus, it is paramount to ensure that llms are reliable, unbiased, and beneficial for society. this paper does not set a general goal for further development of llms but tries to provide a comprehensive overview of the evaluation methodologies for large language models and dialog agents. in section 3, we present a chronological overview of the recent history of llm development and their evaluation methods. specifically, we explore benchmarks, human assessments, and model assessments, among others, that are prevalent in both academic research and practical applications. in section 4, we propose a primary taxonomy of these approaches and discuss their strengths and internal issues, including noticeable errors, problems, and contradictions. section 5 examines which specific aspects of llms are commonly evaluated in contemporary studies. finally, in section 6 we use the proposed taxonomy to discuss current challenges and possible directions for further progress in the field. one has to state that the current evaluation approaches have are not effective and do not meet modern requirements. moreover, further extensive development of the existing approaches (for example, increasing the number of benchmarks and creating new tasks within existing benchmarks) cannot address these issues. we drastically need a qualitative rather than quantitative leap in evaluation. in our opinion, the first step towards a solution should be the survey of the existing evaluation taxonomy, and a detailed discussions of the weaknesses of the available methods that we try to provide in this paper. ## "cambrian explosion" of large language models lately the landscape of language models has expanded remarkably (figure 1 ). as of october 2023, the number of generative text models on hugging face (hf) has reached a remarkable 25 000+ and 86 59 models are based explicitly on the llama model @xcite . this explosion can also be observed in real time 2 . with such an abundance of models, it becomes essential to evaluate and compare their quality however, the various works in this field employ different methodologies for assessing quality. expansion at such a rate brings inevitable confusion foot_0 within the field. so, common evaluation methodologies are not only far from consistent but are also contradictory sometime. this paper has no intention to provide a complete and comprehensive survey of the field. we suggest focusing on one aspect of llm development that we personally see as the most crucial for the future progress of the field, namely, evaluation. however, even in this narrowed context, it is hardly possible to guarantee any form of a complete review due to the number of relevant papers on the topic (figure 2 ). we address the reader to chang et al., 2023 for an example of such a survey. in this paper, we instead discuss selected examples to illustrate the trends and challenges we are facing. we believe those examples are relevant to the field and had a high impact at the moment of their release. we do not claim we can provide a full review of all evaluation techniques used for llms, but to the best of our knowledge, this paper lists all significant conceptual approaches. ## evolution of llm evaluation let us review the trends in llm evaluation. subjectively, we split llm development into three core periods with specific properties. we list some of the models for every period and briefly describe the methods used for performance evaluation. we do not imply that the list of the models is complete. we also list only some of the evaluation methods used for every model since they are numerous and tend to overlap. nevertheless, we enumerate the primary evaluation methodologies so the reader can have a fair and complete representation of the spectrum of evaluation methods available today. let us briefly discuss each period and highlight some of the methods that were used for evaluation. ## prevalent evaluation methodologies as the field evolved, several generalized approaches to evaluation established themselves. these include comparing the models on a set of benchmarks, assessment by humans, and modeling human evaluation (either using heuristics, dedicated models, or a superior llm model). each of these approaches has its advantages, limitations, and potential drawbacks. let us analyze them sequentially to understand their specifics. ## what are we evaluating? with dozens of actively used benchmarks with hundreds of task types, researchers naturally tend to group them into general aspects of the model's performance, so providing several high-level scores becomes standard practice. often, researchers present them as so-called radar diagrams to highlight the advantages and disadvantages of the given model over baselines. however, an overview of recent papers reveals no structure or system of these aspects, even on the highest level (see figure 3 ). sometimes, they remind the famous fiction animals classification @xcite , mixing different types and principles altogether. building a proper taxonomy for these aspects is a complex and extensive endeavor, far beyond the scope of this paper. for deeper insights on this topic, we address the reader, for example, to @xcite . here, we just mention some commonly used approaches and group them intuitively, then discuss the results. these diverse evaluation dimensions highlight the multifaceted nature of assessing language models, each with unique considerations and challenges. for example, the precise definition of text style remains challenging @xcite , while storytelling evaluation needs a deeper understanding of the concept of narrative @xcite . indeed, the evaluation guidelines proposed in @xcite for creative, generative systems are relevant for the llm evaluation in general: "clearly defining the goal of the generative system, asking questions as concrete as possible, testing the evaluation setup, using multiple different evaluation setups, reporting the entire evaluation process and potential biases clearly, 25 https://bit.ly/3rkzwlm 26 see the survey by yao et al., 2023 27 https://huggingface.co/huggingfaceh4 28 though, @xcite recently showed the flaws of available explainability metrics. and finally analyzing the evaluation results more profoundly than merely reporting the most typical statistics." a well-defined and structured list of aspects we want to evaluate llm on is essential to optimize and prioritize the evaluation of language models. do we really need them all? how do they interrelate? without a clear understanding of what aspects we are assessing and why, it becomes difficult to focus on specific areas for improvement or to allocate resources effectively. ## discussion let us now try to sketch the main trends in evaluation approaches and hypothesize their further development in the context of the multiple challenges we highlighted above. ## conclusion this paper provides an overview of the current state of evaluation techniques used for llms and analyzes them. we trace the progress of llms in the last few years and create a taxonomy of the methods used to evaluate llm performance. one by one, we analyze significant approaches and highlight challenges that arise with them, including insufficient standardization, poor robustness to noise, and test set leakage of benchmarks; frequent cases of disagreement between benchmark-based evaluations, humans' and superior models' preferences; humans' and superior models' biases; dead ends of pareto optimization and non-transitive results in the absence of global criteria; no structure or system of aspects of evaluations, even on the highest level. based on these observations, the current evaluation approaches have lost their effectiveness and do not meet modern requirements, and there is no clear way to patch them. in our opinion, the first step towards a solution should be the standardization of tasks and evaluation methods, including a precise formulation of the assessed aspects. we still do not know whether there is a new single "turing question" that can set the main direction of the industry for the following decades. what is certain is that to figure out how to move forward, we need to precisely articulate what we want to measure and for what reason.