Towards a Unified Multi-Dimensional Evaluator for Text Generation (2022)

Abstract: Multi-dimensional evaluation is the dominant paradigm for human evaluation in Natural Language Generation (NLG), i.e., evaluating the generated text from multiple explainable dimensions, such as coherence and fluency. However, automatic evaluation in NLG is still dominated by similarity-based metrics, and we lack a reliable framework for a more comprehensive evaluation of advanced models. In this paper, we propose a unified multi-dimensional evaluator UniEval for NLG. We re-frame NLG evaluation as a Boolean Question Answering (QA) task, and by guiding the model with different questions, we can use one evaluator to evaluate from multiple dimensions. Furthermore, thanks to the unified Boolean QA format, we are able to introduce an intermediate learning phase that enables UniEval to incorporate external knowledge from multiple related tasks and gain further improvement. Experiments on three typical NLG tasks show that UniEval correlates substantially better with human judgments than existing metrics. Specifically, compared to the top-performing unified evaluators, UniEval achieves a 23% higher correlation on text summarization, and over 43% on dialogue response generation. Also, UniEval demonstrates a strong zero-shot learning ability for unseen evaluation dimensions and tasks. Source code, data, and all pre-trained evaluators are available at https://github.com/maszhongming/UniEval.

## introduction the rapid development of natural language generation (nlg) tasks with the support of pre-trained language models @xcite @xcite ) calls for a higher quality evaluation of generated texts. however, the evaluation process is still dominated by traditional similarity-based metrics @xcite , exemplified by rouge @xcite and bleu @xcite that compute n-gram overlap between the model output and the reference text. these metrics are potentially misleading as nlg models have advanced to the point where discrepancies between them are unlikely to be detected based on surface-level features @xcite . although using pre-trained models to obtain embedding-based similarity may alleviate this issue @xcite , these metrics still naturally lead to the question: does similarity to reference text indicate the overall quality of model output? @xcite referred to this similarity as "human-likeness" and pointed out that the ability to output human-like text may be completely unrelated to the final performance on generation tasks. realizing that creating a one-size-fits-all score is infeasible, subsequent research has focused on a more comprehensive multi-dimensional evaluation for nlg tasks. it aims to evaluate the model output from multiple explainable dimensions and has been the dominant paradigm in human evaluation @xcite . for example, text summarization typically uses four dimensions for evaluation: coherence, consistency, fluency, and relevance (see in this paper, we propose a unified multidimensional evaluator unieval for text generation tasks. unieval unifies all evaluation dimensions into a boolean question answering (qa) specifically, a unified framework can bring the following benefits: 1) ease of use. one model is sufficient, without the effort of picking multiple appropriate singledimensional evaluators for all the dimensions. 2) internal complementarity. different dimensions in the same nlg task can be closely related to each other, so it is useful to perform joint training for these dimensions to share knowledge. 3) external knowledge incorporation. the unified boolean qa format makes it possible to enhance the pre-trained language model by multi-task learning on diverse and relevant intermediate tasks before being trained on evaluation tasks. 4) extensibility and transferability. a unified evaluator can achieve better extensibility and transferability with continual learning @xcite or prompting @xcite , as it can accommodate more evaluation dimensions by modifying the input question. experimentally, unieval surpasses advanced evaluators by a large margin when evaluating three typical nlg tasks. concretely, compared to the best unified evaluators @xcite , unieval improves the correlation with human judgment by 23% on text summarization, and the improvement exceeds 43% on dialogue response generation. ablation studies verify the effectiveness of our intermediate tasks. we also conduct transfer experiments and show that unieval achieves better performance compared with strong baseline metrics on unseen dimensions and nlg tasks in a zero-shot setting. ## method in this section, we first introduce how to formulate multi-dimensional evaluation as a unified boolean qa problem, and then describe in detail the training paradigm for unieval. ## experiments following @xcite , we classify nlg tasks into three types: compression, creation, and transduction, and select typical tasks from each category to conduct experiments. for compression and creation, we choose summarization and dialogue response generation to measure the performance of unieval, as well as the ability to zero-shot to unseen dimensions. for transduction, we select data-to-text to test whether unieval has the ability to transfer to a new nlg task. ## conclusion in this paper, we emphasize the necessity of multidimensional evaluation in advancing the field of nlg. to promote this comprehensive and finegrained evaluation approach, we propose a unified multi-dimensional evaluator unieval for various nlg tasks. unieval correlates well with human judgment on three typical generation tasks and exhibits excellent transfer performance.