Beyond Probabilities: Unveiling the Misalignment in Evaluating Large Language Models (2024)

Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities across various applications, fundamentally reshaping the landscape of natural language processing (NLP) research. However, recent evaluation frameworks often rely on the output probabilities of LLMs for predictions, primarily due to computational constraints, diverging from real-world LLM usage scenarios. While widely employed, the efficacy of these probability-based evaluation strategies remains an open research question. This study aims to scrutinize the validity of such probability-based evaluation methods within the context of using LLMs for Multiple Choice Questions (MCQs), highlighting their inherent limitations. Our empirical investigation reveals that the prevalent probability-based evaluation method inadequately aligns with generation-based prediction. Furthermore, current evaluation frameworks typically assess LLMs through predictive tasks based on output probabilities rather than directly generating responses, owing to computational limitations. We illustrate that these probability-based approaches do not effectively correspond with generative predictions. The outcomes of our study can enhance the understanding of LLM evaluation methodologies and provide insights for future research in this domain.

## introduction large language models (llms) have significantly advanced the field of natural language processing (nlp), reshaping the paradigms in nlp research and application @xcite @xcite @xcite @xcite . as the scale of model parameters of language models expands from the million to billion or even trillion levels, a proficient llm is expected to exhibit a broad mastery across various tasks. recent works aim to assess llms comprehensively by aggregating a substantial array of nlp benchmarks @xcite @xcite . additionally, there exists a line of research that curates human exam questions to challenge llms @xcite @xcite . the collected questions and nlp benchmarks are adapted into prompts via standardized templates. due to computational constraints, recent evaluation frameworks commonly adopt the approach of selecting the option with the highest probability as the prediction of llms, as illustrated in figure 1 . these frameworks employ either labelbased prediction, which assesses the probability of the next token output, or sequence-based prediction, which evaluates the probability of an entire option, ultimately selecting the option with the highest probability as the llm's prediction. however, these probability-based evaluation methodologies introduce a misalignment between evaluation procedures and real-world application scenarios, where llms are typically tasked with generating responses to user queries. this misalignment raises an important question: is the probability-based evaluation method sufficient to accurately assess the capabilities of llms? in this position study, we argue that the current llm evaluation and leaderboard misalign the actual llm capabilities. we examine three prediction methodologies: generation-based, label-based, and sequence-based predictions. we conducted extensive experiments across llms with varying model sizes on three prominent benchmarks: mmlu @xcite , truthfulqa @xcite , and belebele @xcite . our findings reveal a significant disconnect between probability-based methods and generation-based predictions. even when predictions are correct, the consistency between probability-based methods and generation-based predictions remains notably low. we additionally find that many of these multiple-choice nlp benchmark rankings do not agree with human preference for free-text generation output. consequently, these results raise serious doubts about the reliability of evaluation outcomes derived from popular benchmarks reliant on probability-based methods. in conclusion, our research emphasizes the urgent need for an evaluation approach that ensures accurate and reliable assessments of llm capabilities, more closely aligned with real-world usage scenarios. in next section, we will discuss the course of the development and paradigm of the evaluation of llms. ## empirical evidence in this section, we empirically show that mcq performance does not reflect free-text generation performance. ## moving forward to make sure the future research in llms more reliable, it is crucial to reevaluate our current benchmarks and evaluation methodologies. our analysis indicates a misalignment between these traditional evaluation mechanisms, primarily mcqbased benchmarks and output probability metrics, and the practical usage of generative text appli-cations in llms. the prevalent focus on these benchmarks, although useful for fast and quantitative comparison, falls short of capturing the full spectrum of llm capabilities. in response to these challenges, we propose several forward-looking recommendations for the llm research community: do not take leaderboard scores at face value: the emphasis on leaderboard rankings, while serving as a proxy for llm performance, often overlooks the complexity of tasks that llms are now being developed to perform. as a community, we should not be easily over-hyped with leaderboard chasing, especially considering the limitations on either mcq-based, or voting-based leaderboards as discussed in this paper. develop comprehensive evaluation protocols: future research should focus on creating evaluation frameworks that encompass a broader range of llm capabilities. the discrepancy between evaluation measures and real-world applicability underscores the necessity for a more holistic approach to llm evaluation. this includes not just traditional benchmarks but also metrics that evaluate free-text generation, contextual understanding, and conversational engagement. crafting these comprehensive evaluation protocols will be challenging yet essential for a deeper understanding of llm performance and applicability. ## conclusion this work critically examines the alignment between probability-based evaluation methods for llms and their actual performance in generating text, particularly on benchmarks such as mmlu, truthfulqa, and belebele. our findings highlight a significant gap between these prediction methods and the practical utility of llms, suggesting that current methods might not accurately reflect a model's real-world capabilities. the discrepancies call for a shift towards more comprehensive evaluation frameworks that prioritize the quality of generated text and the model's ability to understand and respond in human-like ways. future research should focus on developing evaluation metrics that more accurately capture the essence of llm performance in practical scenarios. in summary, our study underscores the need for revising llm evaluation practices to ensure they accurately estimate the models' effectiveness in real-world applications. by adopting more relevant evaluation criteria, we can better gauge the progress and utility of llm advancements.