LLM -Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Models (2023)

Abstract: We propose LLM-Eval, a unified multi-dimensional automatic evaluation method for open-domain conversations with large language models (LLMs). Existing evaluation methods often rely on human annotations, ground-truth responses, or multiple LLM prompts, which can be expensive and time-consuming. To address these issues, we design a single prompt-based evaluation method that leverages a unified evaluation schema to cover multiple dimensions of conversation quality in a single model call. We extensively evaluate the performance of LLM-Eval on various benchmark datasets, demonstrating its effectiveness, efficiency, and adaptability compared to state-of-the-art evaluation methods. Our analysis also highlights the importance of choosing suitable LLMs and decoding strategies for accurate evaluation results. LLM-Eval offers a versatile and robust solution for evaluating open-domain conversation systems, streamlining the evaluation process and providing consistent performance across diverse scenarios.

## introduction effective evaluation of open-domain conversation systems is a critical yet challenging problem in natural language processing research @xcite . accurate and consistent evaluation methods are essential for understanding and improving the performance of dialogue systems. traditional automatic evaluation metrics, such as bleu @xcite and rouge @xcite , are insufficient for capturing the nuances of natural language conversations @xcite , leading to the development of various advanced metrics @xcite @xcite @xcite @xcite @xcite . however, most existing methods require annotation data , human references, or multiple prompts, which could be expensive, timeconsuming, or prone to errors. in this paper, we address the problem of evaluating open-domain conversation systems with a focus on large language models (llms) (figure 1 ). our goal is to develop an efficient and accurate evaluation method that covers multiple dimensions of conversation quality, such as content, grammar, relevance, and appropriateness, without requiring human references or multiple prompts. we build upon recent advances in llms @xcite @xcite , and propose a unified multi-dimensional evaluation method called llm-eval. existing evaluation methods have demonstrated promising results in various aspects of dialogue evaluation. however, they often rely on human annotations @xcite , ground-truth responses @xcite , or multiple llm inferences @xcite , limiting their efficiency and adaptability in practical scenarios. we aim to bridge this gap by proposing llm-eval, a single-prompt-based evaluation method that leverages a unified evaluation schema to cover multiple dimensions of conversation quality in a single model call. in llm-eval, we design a natural language instruction that defines the evaluation task and desired criteria, as well as a format instruction that specifies the structure and range of scores for each dimension. the single prompt is created by concatenating the dialogue context, reference (if available), and generated response, and then fed to a large language model, which outputs scores for each dimension based on the defined schema. we extensively evaluate the performance of llm-eval on a variety of benchmark datasets, covering diverse dialogue systems and evaluation dimensions. our experiments demonstrate that llm-eval consistently outperforms most baselines and state-of-the-art evaluation methods in terms of correlation with human judgments. the proposed method is also robust and versatile, adapting to different scoring ranges and evaluation scenarios. in summary, our main contributions are 3-fold: multi-dimensional metrics multi-dimensional evaluation metrics have been proposed to assess various aspects of dialogue quality, such as content, grammar, relevance, and appropriateness. examples include usr @xcite , which trains multiple models to measure qualities like fluency, relevance, and knowledge conditioning, and grade @xcite , which models topic transition dynamics in dialogue history using a graph representation. flowscore @xcite leverages dynamic information flow in dialog history to measure dialogue quality. unlike these approaches, llm-eval employs a single prompt-based evaluation method that leverages a unified evaluation schema, streamlining the evaluation process and providing a more efficient and adaptable solution. unsupervised metrics unsupervised evaluation metrics aim to assess the quality of dialogue responses without requiring human annotations. notable unsupervised methods include deb @xcite , which fine-tunes bert with an nsp objective on a dataset with relevant and adversarial irrelevant responses, and fed @xcite , an unsupervised method that measures dialogue quality using features derived from response embeddings and language model probabilities. in contrast, llm-eval leverages the power of large language models to provide a unified multi-dimensional evaluation, achieving better performance and adaptability compared to existing unsupervised methods. large language models for evaluation recent works have explored using large language models for dialogue evaluation. gptscore @xcite employs models like gpt-3 to assign higher probabilities to quality content, using multiple prompts for a multi-dimensional assessment. @xcite explores using chatgpt and instructgpt to evaluate text quality without references, and compares different paradigms of using llms, including generating explicit scores, using model confidence to determine implicit scores, and directly comparing pairs of texts. g-eval @xcite , a framework that leverages llms with chain-of-thoughts (cot) @xcite and a form-filling paradigm. g-eval with gpt-4 as the backbone model achieves a high correlation with human judgments on a summarization task. however, both gptscore and g-eval require multiple prompts or complex scoring functions that use probabilities of output tokens and their weighted summation as the final score, which can be inefficient or time-consuming. llm-eval addresses these issues by using a single prompt and a unified evaluation schema, offering a more efficient and adaptable evaluation method for opendomain conversations. additionally, llm-eval provides multi-dimensional evaluation scores in a single model call, further streamlining the evaluation process. ## conclusion in this paper, we introduced llm-eval, a unified multi-dimensional automatic evaluation method for open-domain conversations with large language models. the proposed method employs a single prompt along with a unified evaluation schema that covers multiple dimensions of evaluation, such as content, grammar, relevance, and appropriateness. this approach streamlines the evaluation process and eliminates the need for multiple prompts. experiments on various datasets demonstrated the effectiveness and efficiency of llm-eval, consistently outperforming most baselines and state-ofthe-art evaluation methods. as future work, we plan to explore reinforcement learning from llms feedback and investigate llm-in-the-loop evaluation strategies as an alternative to human-in-the-loop methods. this will further enhance the applicability and performance of the llm-eval method in various dialogue system evaluation scenarios.