Fusion-Eval: Integrating Assistant Evaluators with LLM s (2024)

Abstract: Evaluating natural language generation (NLG) systems automatically poses significant challenges.Recent studies have employed large language models (LLMs) as reference-free metrics for NLG evaluation, enhancing adaptability to new tasks tasks. However, these methods still show lower correspondence with human judgments compared to specialized neural evaluators.In this paper, we introduce “Fusion-Eval”, an innovative approach that leverages LLMs to integrate insights from various assistant evaluators. The LLM is given the example to evaluate along with scores from the assistant evaluators. Each of these evaluators specializes in assessing distinct aspects of responses.Fusion-Eval achieves a 0.962 system-level Kendall-Tau correlation with humans on SummEval and a 0.744 turn-level Spearman correlation on TopicalChat, which is significantly higher than baseline methods. These results highlight Fusion-Eval’s significant potential in the realm of natural language system evaluation.

## introduction evaluating the performance of natural language generation (nlg) models has significant challenges @xcite , particularly in terms of evaluation benchmarks and evaluation paradigms @xcite . this study focuses on the latter one. typically, the evaluation paradigms fall into three categories: human-based, automatic-metrics-based and model-based evaluations. among these, human evaluations are regarded as the most reliable, yet they come with high costs and issues of scalability. automatic metrics such as bleu @xcite and rouge @xcite are prevalent in evaluations, relying on comparisons with a 'gold' standard reference. however, the creation of these gold references is a labor-intensive process. ## method fusion-eval is an evaluation framework leveraging an llm to fuse assistant evaluators, to improve scoring quality. the framework's goal is to evaluate an nlg system along one or more criteria in a manner highly correlated with human judgment. the test examples are what fusion-eval will evaluate. for example in the summeval dataset, a test example is a news article and a summary. in this cause, fusion-eval will evaluate the quality of the summary given the news article. each assistant evaluator receives a test example and returns a score. the fusion-eval framework then takes evaluation task descriptions, test examples, and assistant evaluator scores as inputs. we propose two fusion-eval solutions: you are an evaluation agent. i will give you one summary written for a news article. please evaluate the quality of the summary. detailed descriptions of these metrics are as follows: coherence(1-5, any floating value):the collective quality of all sentences. <...> three assistant evaluators are provided. once we obtain the plan, we insert it into the prompt described in the fe-noplan section. this forms the complete prompt for deriving the fusiononce the plan is obtained, it is concatenated with the template. the template placeholders are filled in for each test example along with its specific assistant evaluators' scores. this complete prompt is then used to obtain the fusion-eval evaluation score from the llm. a more detailed description of this workflow, including the prompt used, is provided in appendix a.1. eval final score, depicted in the third textbox in figure1. to adapt fusion-eval to a different evaluation task, one needs to update the criteria and assistant evaluator descriptions and regenerate the plan. additionally, collecting new assistant evaluator scores for the task is necessary. full fusion-eval templates are available in appendix a.2 for summeval and a.3 for topicalchat. our framework is compatible with many possible plans, as long as they describe a valid way to incorporate the assistant evaluators. finding the optimal plan is outside the scope of our work. prompt execution in both solutions, the prepared evaluation prompt template is used with each test example. this template is filled with the inputs, responses, and assistant evaluator scores for each test example. the executing llm then processes this filled prompt, yielding fusion-eval's final evaluation scores as shown in the rightmost textbox in figure 1 . we found that the llm generated evaluation scores in the correct format, so we did not need to do anything else to control the outputs. the executing llm processes the complete prompt and generates a numerical score for each evaluation dimension. the llms are configured to produce 8 predictions with temperatures of 0.5 for palm2 and 0.1 for gpt-4. the final fusion-eval scores are the average of 8 predictions. we do this because we can't obtain log probabilities from the gpt api. ## experiment we conduct a meta-evaluation of fusion-eval, utilizing the summeval @xcite and top-icalchat @xcite benchmarks. we chose summeval and topicalchat as benchmarks for meta-evaluation because unieval @xcite and g-eval @xcite also use only those benchmarks. this facilitates effective comparison with their results. these benchmarks are widely recognized and offer a comprehensive range of evaluation metrics. we intentionally excluded datasets that rely on single-rater annotations @xcite or are limited to a singular metric @xcite . ## conclusion the paper presents fusion-eval, an innovative aggregator using large language models (llms) for diverse evaluation tasks. it effectively integrates assistant evaluators according to specific criteria. empirical results show fusion-eval achieves higher correlations with human judgments than baselines. llms are very powerful, so it's interesting that augmenting llms with scores from simpler methods can improve performance in this case.