Leveraging Large Language Models for NLG Evaluation: Advances and Challenges (2024)

Abstract: In the rapidly evolving domain of Natural Language Generation (NLG) evaluation, introducing Large Language Models (LLMs) has opened new avenues for assessing generated content quality, e.g., coherence, creativity, and context relevance. This paper aims to provide a thorough overview of leveraging LLMs for NLG evaluation, a burgeoning area that lacks a systematic analysis. We propose a coherent taxonomy for organizing existing LLM-based evaluation metrics, offering a structured framework to understand and compare these methods. Our detailed exploration includes critically assessing various LLM-based methodologies, as well as comparing their strengths and limitations in evaluating NLG outputs. By discussing unresolved challenges, including bias, robustness, domain-specificity, and unified evaluation, this paper seeks to offer insights to researchers and advocate for fairer and more advanced NLG evaluation techniques.

## introduction natural language generation (nlg) stands at the forefront of ai-driven communication, with advancements in llms @xcite @xcite . these models demonstrate exceptional text generation proficiency, highlighting the need for robust evaluation. traditional metrics such as bleu @xcite and rouge @xcite ) mainly focus on surface differences, inadequately capturing semantic quality @xcite . embedding-based methods @xcite @xcite suffer from limited scope @xcite , low alignment with human judgment @xcite , and lack of interpretability @xcite . these underscores the urgent need for more effective and flexible evaluation techniques in nlg. ## generative evaluation amidst the rapid evolution of llms, a burgeoning body of research has directed its focus toward leveraging llms as nlg evaluators, which we refer to as generative evaluation. this category, broadly classified into prompt-based and tuningbased evaluation, hinges on whether the parameters of llm evaluators require fine-tuning. the former typically involves directly prompting llms to assess generated text through prompt engineering, while the latter relies on open-source llms that are specifically calibrated for nlg evaluation. both approaches cater to diverse evaluation protocols for measuring the quality of generated texts. mt: machine translation, ts: text summarization, dg: dialogue generation, ic: image captioning, d2t: data-to-text, sg: story generation, ge: general generation. we adopted the evaluation aspects for different tasks from @xcite . specifically, for each evaluation aspect, con: consistency, coh: coherence, rel: relevance, flu: fluency, inf: informativeness, cov: semantic coverage, ade: adequacy, nat: naturalness, eng: engagement, gro: groundness, gram: grammaticality, lik: likability, per: personalization, qua: quality, int: interest, use: usefulness, harm: harmlessness, acc: accuracy, fac: factuality, ada: adaptability, sur: surprise, cha: character, end: ending, fea: feasibility, cre: creativity, tho: thoroughness, str: structure, lay: layout, cla: clarity, com: comprehensiveness, fpr: formality, adh: adherence, dep: topic depth, und: understandability, fle: flexibility, inq: inquisitiveness, spe: specificity, cor: correctness, sem: semantic appropriateness. none means that the method does not specify any aspects and gives an overall evaluation. the detailed explanation of most evaluation aspect can be found in @xcite . ## comparing traditional evaluators qualitative comparison traditional evaluation metrics (e.g., bleu @xcite and rouge) focus on exacting n-gram matches, which penalizes semantically correct but lexically different hypotheses. these methods are simple and fast but not robust to paraphrasing. bertscore @xcite ) measures quality through semantic similarity based on bert contextual embeddings, effectively handling paraphrases and synonyms. however, such matching-based evaluators depend on the quality of the pre-trained embeddings, may struggle with very fine-grained semantic distinctions, and neglect the overall semantics of the hypotheses and references. additionally, neither metric accounts for fluency or readability during evaluation and both still rely on reference texts. in contrast, llms have a strong capability for language understanding and generation, which supports evaluating quality without needing references. they can adapt to various domains and languages, making them suitable for a wide range of nlg tasks without requiring task-specific feature engineering. llms also provide more nuanced evaluation criteria beyond traditional metrics, such as semantic coherence, fluency and possible explanations. however, llm-based methods are computationally more intensive due to their vast architec-tures. additionally, prompting llms for nlg evaluation requires careful crafting of prompts. variations in these prompts can lead to substantial differences in evaluation outcomes, as indicated in @xcite . section 5 summarizes more open problems of llm-based metrics. performance comparison efficiency comparison ## open problems despite significant efforts and achievements in various benchmarks, several challenges persist for llm-based evaluators. ## conclusion in this paper, we have comprehensively surveyed the role of llms in the evaluation of nlg. our taxonomy categorizes the works into direct promptbased evaluation methods and tuning-based evaluation methods, each employing distinct scoring protocols. we delved into various llm-based approaches, analyzing their strengths and comparing their differences. through our paper, we highlight unresolved issues, including bias, robustness, and the need for domain-specific and unified evaluation within llm-based evaluators. we anticipate that addressing these challenges will pave the way for more reliable, general, and effective llm-based nlg evaluation techniques.