Towards Better Open-Ended Text Generation: A Multicriteria Evaluation Framework (2025)

Abstract: Open-ended text generation has become a prominent task in natural language processing due to the rise of powerful (large) language models. However, evaluating the quality of these models and the employed decoding strategies remains challenging due to trade-offs among widely used metrics such as coherence, diversity, and perplexity. This paper addresses the specific problem of multicriteria evaluation for open-ended text generation, proposing novel methods for both relative and absolute rankings of decoding methods. Specifically, we employ benchmarking approaches based on partial orderings and present a new summary metric to balance existing automatic indicators, providing a more holistic evaluation of text generation quality. Our experiments demonstrate that the proposed approaches offer a robust way to compare decoding strategies and serve as valuable tools to guide model selection for open-ended text generation tasks. We suggest future directions for improving evaluation methodologies in text generation and make our code, datasets, and models publicly available.

## introduction large language models (llms, e.g., @xcite have demonstrated remarkable capabilities in generating coherent and contextually appropriate text across diverse domains. however, the quality of llm outputs is fundamentally determined not only by the underlying model architecture but also by the decoding strategies employed during inference-the algorithms that transform the model's output probability distributions into actual text sequences. as the landscape of both llms and decoding strategies continues to expand rapidly, the need for robust evaluation frameworks has become increasingly critical @xcite . figure 1 : multicriteria evaluation framework for benchmarking models and decoding strategies, i.e., decoding methods. we distinguish two scenarios for benchmarking ( §1) and two ranking objectives ( §4), giving rise to three use-case tailored, distinct methods ( §4.1, 4.3 and 5). scope and problem definition. this paper specifically addresses the challenge of multicriteria evaluation in open-ended text generation, where we must simultaneously consider multiple, often conflicting quality dimensions @xcite . we focus on developing principled methods for both relative and absolute rankings of decoding methods. our approach centers on a subset of automatic evaluation metrics-coherence, diversity, and generation perplexity-that capture fundamental trade-offs in text generation quality. while numerous other metrics exist (e.g., relevance, informativeness, style consistency), we deliberately limit our scope to these three core dimensions to establish a foundational framework that can be systematically extended. current evaluation approaches face remarkable limitations when assessing the quality of text generations within this multicriteria context. traditional methods typically rely on either human judgments-considered the gold standard, but resourceintensive, and dependent on carefully designed protocols @xcite @xcite )-or individual automatic metrics. while automatic metrics such as mauve @xcite , coherence @xcite , diversity, and generation perplexity @xcite provide valuable insights into specific aspects of generation quality, an isolated consideration of these measures offers only an incomplete perspective on overall performance and fails to address the fundamental multicriteria nature of the evaluation problem. in the context of open-ended text generation, this evaluation challenge is particularly acute because decoding strategies inherently involve trade-offs between competing objectives such as coherence and diversity. a method that excels in coherence may underperform in diversity, and vice versa, making it difficult to establish consistent relative rankings among different approaches or provide meaningful absolute assessments of their quality. the fundamental challenge addressed in this work lies in developing principled approaches for both relative and absolute multicriteria evaluation that can balance our selected subset of automatic metrics within a comprehensive framework. this enables reliable comparison of different models and decoding strategies-collectively referred to as decoding methods throughout this work (fig. 1 )-while acknowledging the inherent trade-offs between the chosen evaluation criteria. addressing this challenge is essential for advancing the field of open-ended text generation evaluation and providing practitioners with evidence-based guidance for selecting optimal decoding methods within the multicriteria landscape we define. research gap. when evaluating decoding methods based on multiple quality criteria in several scenarios (i.e., datasets), a method may excel in one area while lagging in another. aggregating such multicriteria evaluation results for different scenarios is still an open problem. existing approaches comprise the pareto front or weighted sums. while the former is hardly informative for large-scale benchmarking (cf. §4), the latter depends on (ar-bitrarily) selected weights. in this work, we offer two alternative approaches while distinguishing two foot_1 prototypical practical benchmarking scenarios with associated research questions (rq): scenario 1 (ranking). first, consider a practitioner using open-ended text generation for a specific task, e.g., a customer support chatbot. this practitioner might primarily be interested in a complete scenario-specific relative ranking of existing methods. this motivation renders metric information about the methods' performances a means to an end. thus, an ordinal ranking of methods will do. rq1: can we exploit novel statistical methodologies for partial orders to establish multicriteria rankings that potentially allow for incomparability? scenario 2 (cardinal assessment). second, for researchers interested in designing new decoding methods (i.e., model, decoding strategy, or both), it is of utmost importance to know how much better one method is compared to another, i.e., having an absolute ranking on a cardinal scale. knowledge of the performance of existing methods on different tasks will help derive new methods. rq2: can we aggregate multiple automatic evaluation metrics in a meaningful and statistically valid way? contributions. we address rq1 ( §4) and rq2 ( §5) by proposing appropriate aggregation methods (cf. fig. 1 ), including a novel summary metric to balance multiple assessments. we further provide experimental results by applying all introduced methods to over 1.8m stories generated by six llms on real-world datasets (cf. §3 for the setup and §4.2, §4.4, §5.2 for the results). ## related work benchmarks are ubiquitous in applied machine learning (ml) research @xcite @xcite @xcite @xcite , being used to make informed decisions and to demonstrate the superiority of newly proposed methods over concurrent ones @xcite @xcite . in recent years, the focus has shifted towards multicriteria and multi-task benchmarking problems @xcite @xcite @xcite @xcite . in a multitude of domains, there are several criteria concerning which methods need to be compared. classical examples include runtime and accuracy in predictive ml @xcite or performance and speed in optimization @xcite , to name only a few. modern llms require evaluation across multiple metrics due to their broad capabilities (see, e.g., @xcite . assessing models on diverse tasks -from reasoning and comprehension to creativity and ethics -provides better understanding of their strengths and limitations @xcite . these comprehensive evaluation frameworks advance model performance while ensuring alignment with real-world applications and ethical standards @xcite @xcite . multicriteria benchmarking has thus become essential for guiding both theoretical progress and practical deployment of llms. decoding methods for open-ended text generation are no exception. several metrics to evaluate the quality of decoding strategies have been proposed and discussed in recent years @xcite @xcite @xcite @xcite . diversity, mauve, coherence, and generation perplexity are among the most popular metrics. diversity measures lexical variation using n-gram repetition rates, with higher scores indicating less repetition. mauve is a distribution similarity metric between generations and reference texts. coherence is defined as the averaged log-likelihood of the generated text conditioned on the prompt and rewards logical flow. finally, generation perplexity @xcite measures the predictability of the generated text under the language model; lower perplexity indicates that the text is more likely according to the model's own probability distribution. this multitude of quality metrics naturally raises the question of how to aggregate them, i.e., how to account for multiple dimensions of text quality to compare decoding methods holistically. it is selfevident that focusing on single metrics has obvious shortcomings. exclusively optimizing for coherence will favor decoding methods with only moderate diversity, leading to degenerate, i.e., repetitive and uncreative generations @xcite . on the other hand, focusing solely on diversity will eventually result in incoherent text only slightly -if at all -related to the prompt. in this work, we offer a fresh perspective on the problem of multicriteria evaluation, adopting recent developments in the theory of depth functions and order theory (cf. §4). ## experimental setup we evaluate six model architectures that generated over 1.8 million stories based on prompts sourced from three distinct datasets, utilizing five decoding strategies across 59 hyperparameter configurations. models. we employ gpt2-xl (1.5b, @xcite , mistral 7b v0.3 @xcite , llama 3.1 8b @xcite , qwen 2 7b @xcite , and falcon 2 11b @xcite . evaluation metrics. building upon su and collier (2023), we select diversity, coherence, and generation perplexity foot_2 as automatic metrics to assess the quality of the generated texts individually. based on this subset of possible instance-level metrics, we construct partial orders for multicriteria rankings ( §4) and develop a cardinal assessment that collapses all metrics into one single score ( §5). since both approaches require instance-level metrics, we exclude mauve in this study as it assesses distributional similarities between samples of machine-generated text and human-written continuations, i.e. it relies on aggregated data, which would prevent us from applying the methods proposed in §4 and §5. ## scenario 1: ranking methods to benchmark decoding methods according to multiple criteria (cf. §2) aiming for a ranking of methods (scenario 1 and rq1 in §1), we adopt very recent developments in the theory of multicriteria and multitask benchmarking @xcite @xcite @xcite @xcite , some of them grounded in decision theory (social choice theory), some in the theory of data depth. in this section, we propose benchmarking of decoding methods in terms of an ordinal ranking along (i) the extended bradley-terry model ( §4.1; @xcite ) and (ii) the unionfree-generic (ufg) depth ( §4.3; @xcite as an alternative approach. both approaches deliver ordinal rankings of decoding methods rather than a cardinal quality assessment (cf. left and middle column of use case to illustrate our evaluation methodology, we apply it to the wikitext-103 dataset, which comprises 1,314 human-written prompts. we assess decoding methods by analyzing their text generations across three quality metrics: coherence, generation perplexity, and diversity. our benchmarking approach produces partial rankings by determining whether one decoding method outperforms another, without quantifying the magnitude of performance differences. given the use of multiple quality metrics, we employ a dominance-based comparison framework. a decoding method is considered superior to another if and only if all three metrics either support this preference or remain neutral (i.e., do not contradict it). consider, for example, the performance of mistral 3 cs with hyperparameter configurations (('0.2', '1')) and (('0.8', '1')) on the first wikitext prompt. we observe that the coherence metric demonstrates a strict preference for (('0.2', '1')) over (('0.8', '1')), while the perplexity and diversity metrics show no contradictory evidence. consequently, we conclude that mistral 3 cs (('0.2', '1')) dominates mistral 3 cs (('0.8', '1')) for this particular prompt. 4 overall, for each prompt, we derive pairwise comparisons for 6 models × 59 decoding @xmath0, one for each decoding method. ## scenario 2: cardinal assessment while multicriteria analysis provides ordinal rankings among decoding methods, many applications require a single unified metric for benchmarking and optimization. use case we compute q*text scores for over 1.8m text continuations, as described in ## discussion first, we examine the extended bradley-terry model and the union-free generic depth approach, both of which are based on pairwise comparisons. a first impression can be seen in appendix c. we observe that out of a total of 124,962 pairwise comparisons among methods, only a very small percentage shows a clear dominance structure. that is, for the majority of the method comparisons, the metrics contradict each other (or imply indifference), e.g., a method is better with respect to coherence but worse with respect to diversity. for a small minority of comparisons, however, methods multilaterally outperform other methods, e.g., method 1 outperforms method 2 for at least 90% of the generations with respect to all metrics. moving on to q*text results, we observe that it shares a preference for larger architectures with the extended bradley-terry model, though smaller we now examine the advantages and disadvantages of the three proposed benchmarking methods within our established framework. as highlighted in section 1, benchmarking serves different purposes: scenario 1 requires only an ordering of decoding methods, while scenario 2 additionally demands a cardinal assessment of quality. while scenario 2 naturally encompasses scenario 1, the ordering focus in scenario 1 enables the utilization of partial ranking theory, leading to fundamentally different procedures than those based on mean transformations and incorporating concepts such as method incomparability. both scenario 1 methods build upon a data transformation, where metric scores are translated into ordinal values. the extended bradley-terry model offers computational efficiency with o(n 2 m) complexity, making it scalable to large numbers of methods and generations. it provides interpretable worth parameters representing estimated preference probabilities and addresses incomparabilities and ties in observed data. however, this approach forces a total order in results, potentially oversimplifying complex dominance structures where methods may genuinely be incomparable. the model assumes independence between pairwise comparisons, which is questionable when comparing methods on fixed datasets, and relies strictly on dominance agreements across all evaluated metrics. the union-free generic depth method preserves incomparabilities through partial orderings, providing more realistic representations of method relationships while offering insights into entire performance distribution structures. unlike the extended bradley-terry approach, it does not assume independence between pairwise comparisons, making it more suitable for fixed-dataset evaluations. nevertheless, this method suffers from computational intensity with worst-case complexity o(2 m ), limiting applicability to smaller methods and dataset subsets. the approach is more complex to interpret than traditional rankings and, like the extended bradley-terry method, may be overly conservative in establishing dominance relationships. q*text provides cardinal assessment with meaningful score differences, enabling quantification of performance gaps. it incorporates penalization of extreme values to prevent degenerate solutions such as repetitive or erratic text, automatically balances multiple criteria through mean aggregation, and remains computationally efficient and straightforward to implement. however, the method relies on normalization bounds and penalization parameters that may not generalize across different contexts. by collapsing multiple metrics into a single score, it may obscure important trade-offs between individual metrics and prove less interpretable than separate metric examination, potentially masking insights about specific strengths and weaknesses. ## conclusion in this work, we analyze the challenge of evaluating open-ended text generation by introducing a multicriteria benchmarking framework that supports both relative and absolute rankings of decoding methods. we present three complementary approaches-the extended bradley-terry model, the union-free generic (ufg) depth, and q*text, a unified metric that harmonizes coherence, diversity, and perplexity into a single score. moreover, we show that our framework captures nuanced tradeoffs among metrics and avoids misleading comparisons when methods excel on different criteria. extensive experiments involving six large language models, three distinct domains (news, wikipedia, stories), and over 1.8 million generated continuations demonstrate the practical benefits of our approach. the extended bradley-terry model yields interpretable "worth" parameters that reflect overall preference probabilities, while ufg-depth uncovers central and atypical ranking structures, highlighting when decoding methods are genuinely incomparable. q*text further enables direct comparison and quantification of performance gaps, revealing that balanced hyperparameter settings outperform extreme configurations and that smaller models can rival larger ones under appropriate decoding choices. taken together, these contributions provide practitioners and researchers with a more reliable, data-driven basis for selecting and designing decoding methods in open-ended text generation, paving the way for more holistic benchmarking practices.