D ecomp E val: Evaluating Generated Texts as Unsupervised Decomposed Question Answering (2023)

Abstract: Existing evaluation metrics for natural language generation (NLG) tasks face the challenges on generalization ability and interpretability. Specifically, most of the well-performed metrics are required to train on evaluation datasets of specific NLG tasks and evaluation dimensions, which may cause over-fitting to task-specific datasets. Furthermore, existing metrics only provide an evaluation score for each dimension without revealing the evidence to interpret how this score is obtained. To deal with these challenges, we propose a simple yet effective metric called DecompEval. This metric formulates NLG evaluation as an instruction-style question answering task and utilizes instruction-tuned pre-trained language models (PLMs) without training on evaluation datasets, aiming to enhance the generalization ability. To make the evaluation process more interpretable, we decompose our devised instruction-style question about the quality of generated texts into the subquestions that measure the quality of each sentence. The subquestions with their answers generated by PLMs are then recomposed as evidence to obtain the evaluation result. Experimental results show that DecompEval achieves state-of-the-art performance in untrained metrics for evaluating text summarization and dialogue generation, which also exhibits strong dimension-level / task-level generalization ability and interpretability.

## introduction recently, pre-trained language models (plms) such as gpt @xcite , bart @xcite , and t5 @xcite have achieved promising performance in natural language generation (nlg) tasks, such as text summarization @xcite and dialogue generation @xcite . as the quality of gen-erated texts gradually approaches that of humanwritten texts, there is an increasing demand for automatic evaluation metrics of generated texts. however, existing evaluation metrics are still struggling to measure the quality of generated texts accurately. traditional metrics such as bleu @xcite , meteor @xcite , and rouge @xcite rely on n-gram overlap between generated texts and reference texts, which fail to detect the issues in the content of generated texts @xcite . recent works resort to model-based evaluation metrics to compute the similarity between generated texts and reference texts based on contextual representations from pre-trained models @xcite or adopt the score of language modeling @xcite / masked language modeling @xcite for evaluation. other works choose to train evaluation models on the evaluation datasets to fit human scores @xcite or distinguish human-written texts from negative samples @xcite , aiming to obtain higher correlations with human judgments in various evaluation dimensions (such as coherence and consistency) of specific datasets. we argue that there are two main challenges in building an evaluation metric for text generation: 1) generalization ability: most of the existing metrics that have high correlations with human judgments on evaluation datasets are directly trained on the corresponding datasets @xcite @xcite . this may result in over-fitting to task-specific data and harm their generalization ability to other nlg tasks and dimensions @xcite . 2) interpretability: although recently proposed evaluation metrics can measure the quality of generated texts from multiple dimensions, they only provide an evaluation score for each dimension without giving evidence to interpret how they predict this score @xcite . to deal with these challenges, we propose a simple yet effective evaluation metric called de-compeval. firstly, to improve the generalization ability, we formulate nlg evaluation as an instruction-style question answering (qa) task, and utilize instruction-tuned pre-trained language models @xcite to solve this task without training on task-specific data. the instruction-style question consists of an instruction, the input of nlg evaluation, and a yes/no question, e.g., "answer the following yes/no question ... is this a coherent response given the dialogue history?" for the evaluation of coherence in dialogue generation, where the specific evaluation input is omitted. secondly, we propose a question decomposition strategy to make the evaluation process more interpretable, instead of directly making instructiontuned plms answer the original question. this strategy decomposes the question into the subquestions which sequentially evaluate the corresponding dimension of each sentence in the generated texts. then, we recompose these subquestions with their answers generated by the plm as evidence to make the plm answer the original question, which is used to compute the final evaluation result. the evidence can promote the understanding of the evaluation process by indicating the potential problematic sentences that affect the evaluation score. our main contributions are as follows: ## discussion applicability in non-english languages: although the benchmark datasets in the experiment are mainly in english, our method can be also applied to non-english languages. since our base model flan-t5 has some multilingual ability @xcite , we can design instructionstyle questions / subquestions and answer words in the target language to apply decompeval to non-english evaluation tasks. decompeval can also adapt to stronger instruction-tuned multilingual plms for better applicability in non-english languages. we will further investigate the extensibility of our method to non-english evaluation tasks in the future work. ## conclusion we present an untrained evaluation metric called decompeval, which formulates nlg evaluation as an instruction-style qa task, and utilizes instruction-tuned plms to solve this task via question decomposition. experimental results show that decompeval achieves state-of-the-art performance in untrained metrics, which also exhibits better dimension-level / task-level generalization ability than trained metrics and improves the interpretability.