O pe NLG auge: An Explainable Metric for NLG Evaluation with Open-Weights LLM s (2025)

Abstract: Large Language Models (LLMs) have demonstrated great potential as evaluators of NLG systems, allowing for high-quality, reference-free, and multi-aspect assessments. However, existing LLM-based metrics suffer from two major drawbacks: reliance on proprietary models to generate training data or perform evaluations, and a lack of fine-grained, explanatory feedback. We introduce OpeNLGauge, a fully open-source, reference-free NLG evaluation metric that provides accurate explanations based on individual error spans. OpeNLGauge is available as a two-stage ensemble of larger open-weight LLMs, or as a small fine-tuned evaluation model, with confirmed generalizability to unseen tasks, domains and aspects. Our extensive meta-evaluation shows that OpeNLGauge achieves competitive correlation with human judgments, outperforming state-of-the-art models on certain tasks while maintaining full reproducibility and providing explanations more than twice as accurate.

## introduction evaluating natural language generation (nlg) systems remains a challenging research problem. traditional overlap-based metrics, such as bleu @xcite and rouge @xcite , are still widely used but exhibit limited correlation with human judgments, particularly when assessing modern nlg systems @xcite . with the rise of pre-trained language models, the research community began to shift toward model-based metrics that better capture semantic similarity, yet their performance was still unsatisfactory @xcite . recently, large language models (llms) have demonstrated remarkable potential in imitating human evaluation of generated text @xcite @xcite . llm-based metrics are often general enough to evaluate diverse nlg tasks and can provide high evaluation performance without the need for reference texts @xcite . they also offer better adaptability to evaluate specific aspects of generated text and are able to evaluate beyond semantic correctness, taking into account aspects such as factual consistency or relevance @xcite . however, existing llm-based nlg metrics lack two important features: (1) they are not fully open-source, with most relying on data generated by proprietary llms such as @xcite , and (2) they do not provide precise explanations for their evaluations, with most of them producing only overall scores, or explanations limited to a single short comment (see in this paper, we introduce openlgauge -a versatile, reference-free metric for nlg tasks that provides precise, error-span-based explanations (figure 1 ). unlike existing llm-based metrics, openlgauge is built entirely on open-weight models and does not rely on human-annotated datasets. it supports a wide range of nlg tasks, including data-to-text, summarization or story generation. moreover, it allows for fine-grained evaluation of specific, customizable aspects of the generated text such as faithfulness, fluency or coherence. it can also evaluate other user-defined aspects appropriate for a given application. our contributions are as follows: our experiments show that openlgauge achieves higher explanation quality than other metrics trained on data generated by proprietary llms, while also delivering strong evaluation performance. on tasks such as summarization, openlgauge achieves higher correlations with human judgments than metrics based on the strong proprietary gpt-4 model. while the state-of-theart metric, themis -which leverages both humanannotated data and data from proprietary llms -remains superior in terms of correlations with human judgments, openlgauge proves highly competitive and even outperforms it on certain tasks, while also providing more detailed error explanations. all our experiments were conducted using only quantized open-weight models and two gpus with 48gb of vram each, ensuring that the results can be reproduced in many ai research labs. our code and data are available on github. foot_0 ## related work although nlg has traditionally been evaluated using simple word-overlap-based metrics such as bleu, these are known to have low correlations with human judgments @xcite . this improved somewhat with the use of trained models for metrics in the past few years @xcite @xcite , but the correlations remained moderate. recently, numerous studies explored the application of llms in nlg evaluation. a prominent line of research focuses on leveraging proprietary llms such as @xcite , with direct prompting for an overall score @xcite or even annotating error spans with categories, which has been explored in machine translation @xcite @xcite . however, using proprietary models is costly and comes with a reproducibility penalty, as some llm versions become unavailable or are modified in a non-transparent way @xcite ). an- ## problem statement we formulate the problem of evaluating the output of an nlg system while providing error-based explanations as follows. given an input x, an output y, and an evaluation aspect a, the task is to return a tuple ⟨z, {e 1 , ..., e n }⟩, where z is a numeric score assigned to y, and {e 1 , ..., e n } represents a set of error annotations. each error annotation e @xmath0, t i , l i ⟩ includes a span of text s @xmath1, a textual explanation t i , and a severity level l i . although the term error is used throughout this work, it should be understood in a broader sense as any issue in the text related to the evaluated aspect a. examples of outputs provided by openlgauge are presented in figure 2 and in appendix l. ## open llm ensemble as evaluator to achieve a high-quality evaluation of nlg outputs, we propose openlgauge ens , a two-stage ensemble of open-weight llms. the ensemble consists of n annotator models, which perform independent analyses of the provided nlg output, and a consolidator model, which is responsible for merging their results and filtering inaccuracies. a high-level overview of the approach is presented in figure 1 . although our approach requires multiple llms, using the ensemble with a handful of models (@xmath2, as required by some metrics, e.g. g-eval (@xmath3, liu et al. ( 2023) ). furthermore, we only use quantized llms to limit the computational requirements. ## summary in this work, we present openlgauge -a versatile method for evaluating nlg that uses only openweight models and provides fine-grained explainability. the method provides a much better explanation quality than previous methods and achieves competitive correlations with human judgments.