The Good, The Bad, and The Greedy: Evaluation of LLM s Should Not Ignore Non-Determinism (2025)

Abstract: Current evaluations of large language models (LLMs) often overlook non-determinism, typically focusing on a single output per example. This limits our understanding of LLM performance variability in real-world applications. Our study addresses this issue by exploring key questions about the performance differences between greedy decoding and sampling, identifying benchmarks’ consistency regarding non-determinism, and examining unique model behaviors. Through extensive experiments, we observe that greedy decoding generally outperforms sampling methods for most evaluated tasks. We also observe consistent performance across different LLM sizes and alignment methods, noting that alignment can reduce sampling variance. Moreover, our best-of-N sampling approach demonstrates that smaller LLMs can match or surpass larger models such as GPT-4-Turbo, highlighting the untapped potential of smaller LLMs. This research shows the importance of considering non-determinism in LLM evaluations and provides insights for future LLM development and evaluation.

## introduction when evaluating a large language model (llm), two common generation configurations are commonly used: greedy decoding and nucleus sampling @xcite ). it's important to note that given a particular input, the same llm may generate significantly different outputs under various decoding configurations, a phenomenon known as non-determinism in generation. however, most evaluations of llms are based on a single output per example. this practice is primarily due to practical considerations, as llm inference and evaluation can be computationally expensive. neglecting non-determinism in generation significantly limits our comprehensive understanding of llms. additionally, without reporting the standard deviation in most current llm evaluations, it is difficult to measure the variability and dynamics of llms in real-world applications. for certain capabilities such as math reasoning @xcite and coding, greedy generation is preferred to ensure fair comparisons. nonetheless, it remains unclear whether there are significant differences in performance between greedy decoding and sampling. recent investigations have also highlighted potential issues of instability in llms @xcite . in a study where the best answer was selected from 256 random generations, the llama-2-7b model achieved an impressive 97.7% accuracy in solving gsm8k questions, even surpassing gpt-4 @xcite . this phenomenon further underscores the enormous potential of llms in their non-deterministic outputs. previous studies @xcite @xcite have extensively examined the influence of different sources of variance on llm generation, including prompts and in-context examples. however, the impact of different decoding configurations on llm performance has not been fully explored. herein, we aim to investigate a series of critical questions regarding the non-determinism of llm generations: • q1: how does the performance gap between greedy decoding and sampling differ? • q2: when is greedy decoding better than sampling, and vice versa? why? • q3: which benchmark is most/least consistent with respect to non-determinism? • q4: do any models possess unique patterns? apart from q1-q4 in sec. 3, we also explore the scaling effect on non-determinism (sec. 4.1), the alignment effect on non-determinism (sec. 4.2), and how temperature and repetition influence on generation (sec. 4.3, 4.4). our extensive results reveal these findings: • for most benchmarks we evaluated, a notable performance gap is observed between greedy generation and the average score of multiple sampling. in certain cases, the performance ranking under different generation configurations differs. given the non-deterministic nature of llm generation, it is essential to explore how to leverage this characteristic. in sec. 5, we enable llms to generate multiple responses for a task and pick the optimal answer with off-the-shelf reward models. in this best-of-n sampling setting, we observe that 8b-level lms exhibit the potential to surpass gpt-4-turbo on several benchmarks. these findings underscore the importance of scaling inference time compute through repeated sampling @xcite for further enhancing the performance of existing llms. ## experimental setup benchmarks. we select multiple benchmarks for our experiments, encompassing abilities of general instruction-following, knowledge, math reasoning, coding, etc. as summarized in alpacaeval 2 @xcite , arena-hard @xcite and wildbench v2 (lin et al., 2024) are general instruction-following benchmarks. al-dataset instance num. sample num. metric alpacaeval 2 805 16 lc arena-hard 500 16 wr mixeval 4000 16 score wildbench v2 1024 16 wb-score mmlu-redux 3000 32 acc gsm8k 1319 128 em humaneval 164 128 pass@1 mixeval @xcite is a comprehensive mixture of various off-the-shelf groundtruth style benchmarks. since the original mmlu @xcite benchmark is huge and contain numerous ground truth errors @xcite , we use mmlu-redux @xcite which is a subset of 3000 manually re-annotated questions across 30 mmlu subjects. we also include gsm8k @xcite , and hu-maneval @xcite , two popular benchmarks for evaluating the math and code generation abilities of llms. ## experimental results we present our experiment results in q1. how does the performance gap between greedy decoding and sampling differ? from the results, we observe a consistent performance gap between greedy decoding and the sampling method (significance test in appendix a). this disparity holds true across vari-ous llms, whether they are proprietary or opensource, and across multiple benchmarks encompassing instruction-following, language understanding, math reasoning, and code generation. for wildbench, which enables fine-grained analysis of llm capabilities, the performance gap is also evident across all task categories, as shown in for most evaluated tasks and models, greedy decoding outperforms sampling. however, alpacaeval serves as a notable exception, where sampling demonstrates superior performance. gsm8k and humaneval are reasoning tasks requiring llms to solve specific math or coding problems with definite solutions. mixeval also follows a deterministic pattern with its groundtruth-based benchmarks. while alpacaeval, arena-hard, and wildbench are open-ended instructionfollowing benchmarks, alpacaeval exhibits a contrasting behavior compared to the others. the potential reasons are two folds: firstly, the task category distributions vary across different benchmarks. as highlighted by lin et al. (2024), 50% of instances in alpacaeval are information-seeking, whereas more than 50% in arena-hard are related to coding and debugging. furthermore, the difficulty of instances might play an important role. the tasks in both arena-hard and wildbench, sourced from real users, pose substantial challenges. on the other hand, instances in alpacaeval are comparatively simpler. in summary: 1) greedy decoding generally proves more effective for most tasks. 2) in the case of alpacaeval, which comprises relatively simpler open-ended creative tasks, sampling tends to generate better responses. q3. which benchmark is most/least consistent with respect to non-determinism? in terms of the performance gap between two decoding configurations and the standard deviation across different samplings, mixeval and mmlu exhibit the highest stability. this stability can be attributed to the constrained answer space of these benchmarks. specifically, mmlu is structured in a multiple-choice format, and mixeval, comprising various ground-truth-based benchmarks, prompts llms to generate short answers, further limiting the output space. in contrast, gsm8k and humaneval are relatively less stable with respect to non-deterministic generations. the performance gap between the best and worst samplings can exceed 10.0 points. to address this instability, the llm community has adopted specific evaluation protocols. for gsm8k, all models are evaluated using greedy decoding. for humaneval, models are sampled multiple times, and pass@k is used as the final metric to ensure reliable comparison. different decoding configurations can even alter the model rankings in some cases. for example, on arena-hard, qwen2-7b is slightly better than llama-3-8b when both use greedy decoding; however, llama-3-8b may outperform qwen2-7b when both decode by sampling. to measure the change in ranking induced by nondeterminism, we compute kendall's τ @xcite : n n ×n @xmath0, 1], a non-parametric statistic which measures the correspondence between two rankings r 1 , r 2 (greedy decoding and average score of multiple samplings, in our case). τ is formally defined as: q4. do the models possess distinctive characteristics? gpt-4-turbo shows consistent performance across multiple tasks, with a smaller performance gap between greedy decoding and sampling, as well as improved sampling quality. some openweight llms, however, exhibit unique characteristics. for example, mistral-7b-instruct-v0.2 displays inverse behavior on open-ended instruction following tasks like alpacaeval and arena-hard when compared to other models. similarly, llama-3-8b-instruct performs better by sampling than by greedy decoding on mmlu, which is unlike the behavior of other models. these observations raise intriguing questions for future research. why do certain models exhibit inverse behavior on specific tasks? can these unique characteristics be leveraged to develop more robust llms? these questions highlight the need for deeper explorations into the underlying mechanisms of llms. such research could significantly enhance our understanding of how different models and training impact model behavior. ## how various factors influence non-determinism? in this section, we further investigate how various factors, such as scaling, alignment, and several decoding parameters, influence non-determinism. ## conclusion & future directions we investigate a series of critical yet overlooked questions around non-determinism of llm generations. after evaluating several llms across seven commonly used benchmarks, we have answered several intriguing research questions. further analysis also provides insights on how scaling and alignment will effect on non-determinism generation. we hope this work can enhance our comprehension of the generation methods and the widely used benchmarks. our evaluation results can also be used for improving future research. for example, our best-of-n results can serve as a benchmark for assessing reward models @xcite .