Statistical Multicriteria Evaluation of LLM -Generated Text (2025)

Abstract: Assessing the quality of LLM-generated text remains a fundamental challenge in natural language processing. Current evaluation approaches often rely on isolated metrics or simplistic aggregations that fail to capture the nuanced trade-offs between coherence, diversity, fluency, and other relevant indicators of text quality. In this work, we adapt a recently proposed framework for statistical inference based on Generalized Stochastic Dominance (GSD) that addresses three critical limitations in existing benchmarking methodologies: the inadequacy of single-metric evaluation, the incompatibility between cardinal automatic metrics and ordinal human judgments, and the lack of inferential statistical guarantees. The GSD-front approach enables simultaneous evaluation across multiple quality dimensions while respecting their different measurement scales, building upon partial orders of decoding strategies, thus avoiding arbitrary weighting of the involved metrics. By applying this framework to evaluate common decoding strategies against human-generated text, we demonstrate its ability to identify statistically significant performance differences while accounting for potential deviations from the i.i.d. assumption of the sampling design.

## introduction large language models (llms; @xcite @xcite rely on decoding strategies-algorithms that select subsequent tokens based on probability distributions over the vocabulary. as these models advance, numerous decoding methods have emerged, including deterministic @xcite @xcite and stochastic approaches @xcite @xcite , necessitating robust benchmarking protocols for systematic evaluation. in this work, we present a methodological contribution: we adapt the generalized stochastic dominance (gsd) framework @xcite to evaluate llm-generated text. rather than exhaustively comparing all decoding strategies, we demonstrate how gsd enables rigorous multicriteria evaluation while preserving distinct measurement scales. we focus on open-ended text generation as an illustrative example, though the framework generalizes to other tasks. current benchmarking for open-ended text generation typically uses curated datasets like wikitext @xcite or wikinews to assess decoding strategies through automatic metrics and human evaluation. while valuable for practitioners and researchers, these methodologies face three fundamental challenges: contributions. we introduce a gsd-based benchmarking framework that: (i) incorporates multiple quality metrics simultaneously (ii) integrates human and automatic evaluations while preserving measurement scales (iii) provides statistical inference beyond descriptive evaluation (iv) quantifies robustness when i.i.d. assumptions are violated we validate our approach using wikitext and wikinews prompts, comparing five decoding strategies (beam search, contrastive search, temperature sampling, top-𝑘, and nucleus sampling) against human completions. our evaluation combines q*text (garces arias et al., 2025b) with human assessments, demonstrating that human-written text maintains superior or equivalent quality. code and data are publicly available 1 . ## generalized stochastic dominance and the gsd-front to address the challenges outlined in section 1, we propose a framework based on generalized stochas-1 https://github.com/hannahblo/statistical_ multicriteria_evaluation_of_llm-generated_text tic dominance (gsd). rather than imposing a complete ranking-which would require potentially unjustified assumptions about the relative importance of quality dimensions-our framework identifies a minimal set of non-dominated strategies. this set, which we term the "gsd-front," represents potentially optimal choices under varying preference structures. in this section, we present the theoretical foundation of our approach. ## application: automatic and human quality evaluation in this section, we present an application where we investigate whether human text generation potentially still offers superior quality compared to alternative automatic decoding strategies. please note that our method can be applied to any set of competing decoding methods. this investigation addresses the three key benchmarking challenges outlined in section 1: analyzing multiple quality metrics with different measurement scales simultaneously (challenges i and ii), and quantifying the robustness of inferential statements under potential deviations from the i.i.d. sampling assumption (challenges iii and iv). the latter arises specifically from potential dependencies in human evaluations and the use of prompts from two distinct datasets. ## how can the field benefit from the gsd framework? current text generation research overwhelmingly optimizes individual metrics in isolation, leading to systems that excel along one dimension while potentially degrading others. for instance, recent advances in decoding strategies-including locally typical sampling @xcite -rely primarily on mauve scores for hyperparameter tuning and benchmarking. while mauve captures distributional similarity, optimizing solely for this metric may inadvertently compromise other dimensions of quality, such as coherence or diversity. our gsd framework addresses this limitation by enabling system design guided by a holistic view of non-dominated methods, rather than single-metric optimization. instead of declaring one decoding strategy "best" based on isolated metrics, practitioners can identify the set of competitive approaches across multiple quality dimensions simultaneously. this shift-from descriptive metric reporting to statistical assessment of method dominance-provides actionable guidance for both research and deployment decisions. ## related work benchmarks serve as critical platforms for methodological validation in machine learning @xcite @xcite . however, recent studies have exposed significant challenges: @xcite show that performance improvements often fail to replicate, while @xcite demonstrate that minor variations in initialization or sampling can alter rankings @xcite . these findings underscore the need for more statistically rigorous evaluation methodologies. in response, researchers have developed frameworks that explicitly acknowledge benchmark datasets as finite samples from larger populations @xcite . this has led to multi-criteria benchmarking paradigms across diverse domains @xcite , from predictive ml balancing accuracy against efficiency @xcite to optimization tasks requiring simultaneous performance and speed considerations @xcite . for neural text generation, multiple metrics assess different quality dimensions: diversity measures lexical richness, mauve evaluates distributional similarity, coherence calculates prompt-continuation likelihood, and perplexity assesses predictability @xcite @xcite . single-metric optimization proves inadequate-coherence optimization yields repetitive outputs (degeneration), while diversity maximization compromises semantic integrity @xcite . despite this, multicriteria benchmarking has declined since 2022 (figure 1 ). moreover, the discrepancy between automatic metrics (cardinal) and human evaluations (ordinal) presents additional challenges @xcite @xcite . integrated frameworks like huse (hashimoto et al., 2019) combine human judgments with model probabilities. recently, (garces arias et al., 2025b) proposed a multicriteria framework using the bradley-terry model for pairwise comparisons and introduced q*text-a weighted mean of coherence, diversity, and perplexity. our work builds on these advances, proposing a framework that supports rigorous statistical inference while seamlessly integrating both automatic and human evaluations. ## conclusion we introduced a framework based on generalized stochastic dominance (gsd) that addresses three critical limitations in current methodologies for evaluating llm-generated text: (1) the inadequacy of single-metric assessment, (2) the incompatibility between cardinal automatic metrics and ordinal human judgments, and (3) the absence of robust statistical guarantees. the gsd-front approach integrates multiple quality dimensions while preserving their distinct measurement scales and enables quantifying the robustness of inference under potential deviations from i.i.d. assumptions. to validate this framework, we conducted a comparative analysis of five common decoding strategies against humanwritten text, though the method generalizes to any set of generation approaches. the gsd-front enables statistically sound multicriteria evaluation without requiring arbitrary metric weighting or compromising measurement scale integrity. by incorporating techniques from robust statistics, our approach extends beyond descriptive benchmark analysis to provide inferential guarantees that account for potential dependencies in human evaluations. this advancement provides researchers and practitioners with a more rigorous methodology for evaluating text generation systems. future work could extend the gsd approach to other generation tasks such as summarization and translation, investigate additional quality dimensions, and further enhance statistical robustness for complex evaluation dependencies.