X -Eval: Generalizable Multi-aspect Text Evaluation via Augmented Instruction Tuning with Auxiliary Evaluation Aspects (2024)

Abstract: Natural Language Generation (NLG) typically involves evaluating the generated text in various aspects (e.g., consistency and naturalness) to obtain a comprehensive assessment. However, multi-aspect evaluation remains challenging as it may require the evaluator to generalize to any given evaluation aspect even if it’s absent during training. In this paper, we introduce X-Eval, a two-stage instruction tuning framework to evaluate text in both seen and unseen aspects customized by end users. X-Eval consists of two learning stages: the vanilla instruction tuning stage that improves the model’s ability to follow evaluation instructions, and an enhanced instruction tuning stage that exploits the connections between fine-grained evaluation aspects to better assess text quality. To support the training of X-Eval, we collect AspectInstruct, the first instruction tuning dataset tailored for multi-aspect NLG evaluation spanning 27 diverse evaluation aspects with 65 tasks. To enhance task diversity, we devise an augmentation strategy that converts human rating annotations into diverse forms of NLG evaluation tasks, including scoring, comparison, ranking, and Boolean question answering. Extensive experiments across three essential categories of NLG tasks: dialogue generation, summarization, and data-to-text coupled with 21 aspects in meta-evaluation, demonstrate that X-Eval enables even a lightweight language model to achieve a comparable if not higher correlation with human judgments compared to the state-of-the-art NLG evaluators like GPT-4.

## introduction recent advancements of pre-training @xcite , prompting @xcite @xcite @xcite , and instruction the unseen aspect (i.e., interestingness) is highlighted in italics. the text to be evaluated is highlighted with underline. in this example, each evaluation score is from 0 to 1. the higher score indicates better quality. to obtain a more comprehensive assessment of text quality, multi-aspect evaluation @xcite has been proposed to evaluate the generated text from multiple fine-grained evaluation aspects, such as fluency and consistency. while most existing studies @xcite @xcite consider a closed set of aspects, in many realistic scenarios, the users may need to evaluate the text with their customized aspects and specifications, calling for building an evaluator that can be flexibly extended to any unseen aspects without the need of training data. recent studies @xcite propose to leverage large language models (llms) such as @xcite as nlg evaluators, yielding promising zero-shot performance on unseen aspects. however, such evaluations, especially with proprietary llms, are cost-intensive, time-consuming, and pose concerns about data privacy and reproducibility. in this work, we propose x-eval, an automatic evaluation framework that can conduct fine-grained evaluation on both seen and unseen aspects across various nlg tasks with a single model, as illustrated in figure 1 . x-eval follows a two-stage training paradigm: we first instruction-finetune an open-source language model to equip it with the capability of following human-written instructions for evaluation. then, motivated by the observation that evaluation aspects usually exhibit interconnections @xcite and thus their evaluations can benefit each other, we introduce an additional training stage to finetune the model on the instruction-tuning tasks enriched with the evaluations of a set of auxiliary aspects, which are expected to provide clues for evaluating the target aspect and encourage consistent evaluations across multiple aspects. during training, for each target aspect, we take all the remaining aspects defined in the corresponding dataset as auxiliary aspects and incorporate their gold evaluations into the instructions for the second-stage tuning. during inference, given the target aspect, we first select a set of auxiliary aspects based on the similarity of the aspect definitions and predict the evaluation result for each auxiliary aspect using the trained model. we then re-perform the evaluation for each target aspect by incorporating the results of auxiliary aspects. to support our proposed two-stage training of x-eval, we construct aspectinstruct, the first multi-aspect evaluation instruction tuning dataset spanning 27 diverse evaluation aspects over 65 tasks. this dataset is anchored around three core categories of nlg tasks: dialogue, summarization, and data-to-text. in light of insights from previous studies in instruction tuning @xcite , which emphasize the advantage of task diversity in enhancing zero-short generalization, we further augment the dataset by converting the original human rating data into diverse forms of nlg evaluation tasks, including scoring, comparison, ranking and boolean question answering. in addition, to incorporate auxiliary aspects, we manually create templates that convert the numerical evaluation scores of each aspect into descriptions in natural language. the main advantages of our approach are highlighted as follows: (1) generalization ability: we introduce x-eval that can be flexibly generalized to evaluate the unseen nlg tasks or the aspects customized by user instructions in a zero-shot manner with a single model; (2) strong performance with high efficiency: with significantly less amount of model parameters (780m), x-eval achieves strong performance compared to the stateof-the-art llm-based evaluators (including gpt-4) demonstrated through comprehensive experiments; ## main results we report the main results of dialogue evaluation in ## discussions ablation study of instruction tuning tasks ## conclusion in this work, we present x-eval, a novel twostage instruction-tuning framework for text evaluation across both seen and unseen aspects. to facilitate training, we collect aspectinstruct, the first instruction-tuning dataset for multi-aspect evaluation. extensive experiments on meta-evaluation benchmarks demonstrate that with significantly fewer parameters, x-eval achieves a comparable if not higher correlation with human judgments compared to the state-of-the-art nlg evaluators.