(Towards) Scalable Reliable Automated Evaluation with Large Language Models (2025)

Abstract: Evaluating the quality and relevance of textual outputs from Large Language Models (LLMs) remains challenging and resource-intensive.Existing automated metrics often fail to capture the complexity and variability inherent in LLM-generated outputs.Moreover, these metrics typically rely on explicit reference standards, limiting their use mostly to domains with objective benchmarks.This work introduces a novel evaluation framework designed to approximate expert-level assessments of LLM-generated content.The proposed method employs pairwise comparisons of outputs by multiple LLMs, reducing biases from individual models.An Elo rating system is used to generate stable and interpretable rankings.Adjustable agreement thresholds—from full unanimity to majority voting—allow flexible control over evaluation confidence and coverage.The method’s effectiveness is demonstrated through evaluating competency profiles extracted from scientific abstracts.Preliminary results show that automatically derived rankings correlate well with expert judgments, significantly reducing the need for extensive human intervention.By offering a scalable, consistent, and domain-agnostic evaluation layer, the framework supports more efficient and reliable quality assessments of LLM outputs across diverse applications.

## introduction large language models (llms) are machine learning-based models capable of understanding, analyzing, and generating human language @xcite . their advanced capabilities stem from extensive training on large-scale datasets, enabling them to develop a profound understanding of syntax, semantics, and contextual language aspects @xcite . consequently, natural language processing has become a core component of llms. recent advancements have significantly improved their capacity for semantic analysis and textual data comprehension @xcite . as a result, llms are broadly employed across numerous domains, including software test generation @xcite , question answering @xcite , and text summarization @xcite . evaluating the quality of textual outputs generated by llms, however, poses significant methodological challenges, primarily due to the inherently subjective and task-specific nature of text evaluation @xcite . traditional evaluation approaches typically depend on either human judgment-which is resource-intensive, inconsistent, and difficult to scale-or predefined metrics that are often insufficient to capture nuanced variations in quality across diverse tasks @xcite . these limitations highlight a critical gap in current evaluation methodologies, underscoring the necessity for more robust and scalable alternatives. to address these evaluation challenges, this paper proposes a robust and scalable evaluation framework that leverages llms themselves to perform systematic pairwise comparisons. in contrast to conventional methods dependent solely on single-llm judgments or fixed metrics, the presented approach integrates multiple llm judgments and aggregates them using the elo rating system. this aggregation method produces reliable and consistent rankings, substantially reducing the need for extensive human evaluation. thus, the proposed method serves effectively as a universal evaluation layer applicable to a wide range of tasks involving free-form text generation. the remainder of this paper is structured as follows: section 2 introduces foundational concepts, including llms, the elo rating system, and correlation metrics. section 3 provides an overview of related work. section 4 describes the proposed evaluation framework in detail, followed by a pro-totypical implementation in section 5. section 6 demonstrates the framework's applicability by evaluating its performance in extracting competency profiles from scientific abstracts and discusses the results. section 7 summarizes the main contributions and concludes the paper. finally, section 8 highlights the limitations of the proposed approach. ## background this section briefly introduces foundational concepts of llms, the elo rating system, and correlation metrics, which are essential for understanding the proposed evaluation framework presented subsequently. ## related work evaluation of llms has become increasingly crucial due to their widespread application. reliable assessment methods are necessary to ensure outputs meet quality standards, motivating the development of various evaluation strategies. existing methodologies typically fall into two categories: reference-based metrics and reference-free methods. reference-based metrics, such as bleu @xcite , rouge @xcite , and bertscore @xcite , assess outputs by comparing them to predefined reference texts. however, their dependence on static references limits their applicability, especially for creative or open-ended tasks @xcite . to overcome this limitation, reference-free methods like gptscore @xcite have emerged, directly evaluating outputs based on token probabilities and task-specific dimensions. although these approaches are promising, they sometimes exhibit limited correlation with human judgments @xcite , highlighting the ongoing need for more accurate evaluation techniques. an alternative approach, known as llm-as-a-judge @xcite , utilizes llms themselves to perform evaluations. this can be implemented either through single-llm scoring or through more robust multi-llm frameworks, such as debates or peer reviews @xcite . within multi-llm evaluation frameworks, the elo rating system has gained popularity as a structured method for dynamically ranking models based on pairwise comparisons. despite its widespread use, elo ratings are sensitive to factors such as evaluation order and hyperparameter selection, leading to reliability concerns @xcite . recent work by @xcite proposes guidelines to enhance reliability, including a permutation oversampling approach to mitigate order effects, thereby enabling a more robust and dependable model performance assessment. ## approach overview and motivation. this section presents a methodology for utilizing llms to assess diverse free-text responses to a given task (e.g., summarization) through a pairwise comparison methodology. evaluating free-text outputs with llms poses several challenges: to address these challenges, our pipeline is organized into three distinct stages: (i) generation of items to compare, (ii) systematic pairwise comparison using multiple llms, and (iii) ranking the items with an elo rating system to clearly identify the best-performing candidates. figure 1 outlines this pipeline. the methodology begins with generating multiple items intended for comparison. this step may include various methodologies to ensure diverse inputs for evaluation. for example, different hyperparameter configurations, distinct llms, or alternative wording styles can be employed. the methodology then systematically assesses and ranks the items, enabling the identification of the methodology with the best results for the given task. typical applications include hyperparameter optimization, method comparison, and llm selection, where the objective is to determine the most effective configuration or llm. based on the final elo ranking, the performance of different methods is assessed, and the bestperforming item is identified. this highest-ranked item can subsequently be deployed in production environments or research settings. ## implementation the proposed evaluation pipeline has been implemented and is demonstrated through a specific use case: generating competency profiles from research abstracts (see section 5.1). this scenario illustrates how the framework can be applied to real-world data and highlights its effectiveness in evaluating complex, task-specific outputs. competency profiles serve as a concrete example of evaluable items throughout the following sections. the implementation leverages widely adopted tools and frameworks to ensure scalability, usability, and reliability. this section details the technical stack, the integration of llms, data sources, and the experimental setup. additionally, it discusses implementation challenges and the strategies used to address them. ## evaluation to evaluate the proposed method for automated evaluation using llms, an experimental study was conducted. this section outlines the evaluation strategy, introduces the dataset used, and presents the metrics and results related to llm quality. ## conclusion this paper presented a scalable and reliable automated evaluation framework utilizing multiple llms in combination with an elo rating system, significantly enhancing the efficiency and consistency of assessments of llm-generated texts. the conducted evaluation demonstrated a strong alignment between automated rankings and expert judgments, thereby validating the multi-llm approach. the versatility of the presented framework supports broad applicability across diverse domains requiring nuanced textual evaluation, substantially reducing dependency on extensive human intervention. further research is encouraged, particularly focusing on optimizing computational efficiency to fully leverage the framework's potential at scale.