A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations (2024)

Abstract: Large Language Models (LLMs) have recently gained significant attention due to their remarkable capabilities in performing diverse tasks across various domains. However, a thorough evaluation of these models is crucial before deploying them in real-world applications to ensure they produce reliable performance. Despite the well-established importance of evaluating LLMs in the community, the complexity of the evaluation process has led to varied evaluation setups, causing inconsistencies in findings and interpretations. To address this, we systematically review the primary challenges and limitations causing these inconsistencies and unreliable evaluations in various steps of LLM evaluation. Based on our critical review, we present our perspectives and recommendations to ensure LLM evaluations are reproducible, reliable, and robust.

## introduction the evolution of llms has transitioned from simple generative models predicting the next word to advanced systems capable of following instructions and solving complex problems @xcite . early models like gpt @xcite could generate coherent text but were limited to simple tasks, whereas instruction-tuned llms @xcite like chatgpt 1 greatly enhanced their versatility and ability to execute specific commands. this shift has revolutionized the development of realworld applications powered by llms. with the advancements and broad applicability of llms, it is essential to properly evaluate them to ensure they are safe to use. this is indeed important not only for academic benchmarks but also for business use cases. consequently, understanding the bottlenecks of current evaluation methods, and developing strategies to address these challenges are crucial for standardizing evaluations and enabling reliable use of llms in practical applications. nonetheless, evaluating llms is as complex and resource-intensive as their development, involving multiple levels or aspects. existing reviews @xcite @xcite @xcite related to the evaluation of llms often focus only on benchmark tasks, datasets, and evaluation criteria, neglecting the broader complexities. this oversight can undermine the reliability of evaluation by ignoring issues like robustness and reproducibility. while some recent studies @xcite have investigated data contamination @xcite and evaluation malpractices in llm evaluation, their focus is limited to only assessing chatgpt, overlooking other llms, as well as the entire evaluation pipeline. more recently, @xcite discussed the reproducibility problem in existing evaluations of llms and introduced a library to address this. however, their work lacks comprehensive discussions on how aspects like reliability or robustness impact llm evaluation and how to address them. hence, existing llm evaluation studies often focus on individual aspects in a scattered manner, resulting in findings that are only sparsely useful. to mitigate this gap, this paper brings together the discussions to address the fundamental challenges and limitations in llm evaluations that emerge from diverse evaluation setups. first, we craft a schematic workflow of the evaluation pipeline in practical settings (presented in section 2) for a systematic study. we then examine each step in the evaluation workflow, uncovering various inconsistencies and decision-making complexities affecting repro-figure 1 : typology of the llm evaluation workflow. a more detailed description of the challenges and the limitations can be found in ## overview of llm evaluation process the following components are crucial for llm evaluation: evaluation setup, response generation, and evaluation methodology @xcite . each component has its own challenges, which we discuss in section 3. these components in an evaluation workflow are shown in figure 1 . ## challenges in evaluating llms we examine challenges and limitations in the evaluation process of llms based on three dimensions: reproducibility, reliability, and robustness. ## recommendations and best practices so far, we've outlined the primary challenges in evaluating llms. in light of these challenges, a crucial question arises: how can we enhance the evaluation of llms? crafting a structured framework that's both practical and easy to implement is daunting, given the complexities of generative llm development. previous studies tended to focus on specific evaluation aspects without offering comprehensive guidelines for the entire evaluation cycle, leaving researchers without clear guidance. before diving into recommendations for each evaluation stage, it's important to acknowledge three key factors shaping current llm evaluation practices: inherent randomness in generative models, significant computational demands, and insufficient documentation across stages. evaluation setup: selecting benchmarks for model assessment is crucial. rather than simply replicating past choices, researchers should align datasets with required capabilities. to ensure robustness, datasets should vary across expected llm capabilities (e.g., long-context understanding), tasks (e.g., summarization), and language complexity (e.g., vocabulary coverage). ideally, a metric should measure dataset diversity. for model selection, conduct contamination tests between the chosen model and benchmarks using relevant techniques @xcite . this acts as an additional filter for benchmarking datasets, ensuring selection of unseen ones measuring intended capabilities. meanwhile, for reproducibility, document any subset use of benchmarking datasets, along with the selected model version. in addition, throughout scientific history, intelligence progress has evolved across generations. tests from a decade ago may appear simplistic compared to today's standards (e.g., math olympiads, icpc programming contests). refreshing llm evaluations periodically can effectively communicate standard capabilities in both open and closedsource llm markets and ecosystems (e.g., chatbots). hence, to ensure reliability, verify if the dataset has updated versions and incorporate them if available (e.g., humanevalplus @xcite , mmlu-pro @xcite , gsm-1k @xcite ) response generation: for reproducibility, thorough documentation of prompts (e.g., explaining the selection of few-shot samples) and parameter settings (e.g., use tools like mlflow 7 or weights & biases foot_6 (w&b)) is essential. to ensure reliability, it's crucial to justify why specific prompts and parameters are chosen over others by providing comparisons with alternative options. as for robustness, experimenting with diverse prompts and parameters is the key to showcasing their effectiveness and limitations in different scenarios. in resource-constrained environments, conducting experiments with diverse evaluation settings may pose challenges, yet it remains vital to perform robust evaluations on at least a subset of samples. ## conclusions and future work in this paper, we systematically survey the challenges and limitations in evaluating llms. we identified significant inconsistencies and complexities at various stages of the evaluation pipeline, impacting the reproducibility, reliability, and robustness of the results. these issues underline the necessity for a standardized and systematic approach for llm evaluation to ensure their reliable usage in real-world applications. by comprehensively reviewing the current evaluation practices, we have provided a set of recommendations aimed at enhancing the consistency and fairness of llm evaluations. therefore, future work should focus on developing and adopting standardized evaluation protocols for llms to address the identified limitations. this includes creating benchmark datasets, evaluation metrics, and proper documentation of the evaluation settings to ensure reproducibility, reliability, and robustness.