CTRLE val: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation (2022)

Abstract: Existing reference-free metrics have obvious limitations for evaluating controlled text generation models. Unsupervised metrics can only provide a task-agnostic evaluation result which correlates weakly with human judgments, whereas supervised ones may overfit task-specific data with poor generalization ability to other datasets. In this paper, we propose an unsupervised reference-free metric called CTRLEval, which evaluates controlled text generation from different aspects by formulating each aspect into multiple text infilling tasks. On top of these tasks, the metric assembles the generation probabilities from a pre-trained language model without any model training. Experimental results show that our metric has higher correlations with human judgments than other baselines, while obtaining better generalization of evaluating generated texts from different models and with different qualities.

## introduction controlled text generation aims to generate texts under some control variables, including prespecified content prefixes and attribute labels (such as sentiments and topics). controlled text generation has been significantly advanced by large-scale pre-trained models with respect to generation quality and various control variables @xcite @xcite @xcite . despite the great success of these generation models, it becomes critical to evaluate the quality of generated texts accurately. most of the existing studies adopt unsupervised and supervised metrics to measure the quality of generated texts under different combinations of control variables @xcite . the evaluation is commonly conducted in a reference-free setting because it is challenging to collect sufficient high-quality references for each input of control variables in this open-ended text generation task @xcite . however, both unsupervised and supervised metrics have shown limitations in the evaluation of controlled text generation: 1) unsupervised metrics such as perplexity @xcite can only provide task-agnostic evaluation regarding the overall quality of generated texts. however, controlled text generation tasks typically involve multiple evaluation aspects @xcite , including the quality of generated texts themselves and the relationship between generated texts and control variables. it is thus not surprising that existing unsupervised metrics without multi-aspect interpretability have low correlations with human judgments @xcite . 2) supervised metrics are commonly trained on the datasets of specific tasks to measure the corresponding aspects of generated texts (e.g., evaluating whether a generated text is accordant with the sentiment label) @xcite . this may cause over-fitting to task-specific data and degrade the generalization ability of metrics @xcite , thereby giving unstable evaluation of generated texts from different models or with different qualities @xcite . to deal with the above issues, we propose an unsupervised reference-free metric called ctrleval for evaluating controlled text generation models. this metric performs evaluation from different aspects without any training on task-specific data. specifically, we formulate the evaluation of each aspect into "fill-in-the-blank" tasks whose input and output patterns can be designed based on the definition of the aspect. then, we utilize a pretrained model whose pre-training task is text in-filling (such as pegasus @xcite ) as our base model, and fuse the generation probabilities from these "fill-in-the-blank" tasks as the evaluation result. to alleviate the potential bias caused by the task design @xcite , we devise multiple text infilling tasks for each aspect and use the weighted sum of all the results as the final score. in this paper, we consider three aspects which are commonly used to measure the performance of controlled text generation models, including coherence @xcite , consistency @xcite , and attribute relevance @xcite . these evaluation aspects cover both the quality of generated texts and the relationship between generated texts and different control variables, which can provide a comprehensive evaluation result for controlled text generation. experimental results show that our metric can maintain the generalization ability and achieve stable performance faced with model drift and quality drift. our main contributions are as follows: ## discussion extension to more control variables: in this paper, we evaluate the relationship between generated texts and two control variables (including content prefixes and attribute labels) via consistency and attribute relevance, respectively. we can also extend our metric to other control variables by designing additional pattern evaluators to measure the relationship between generated texts and each variable, respectively. we will further investigate the extensibility of our metric in the future work. design of pattern evaluators: with the rapid development of prompt tuning, recent works have proposed new methods on the design of prompts and verbalizers @xcite , which provide alternatives to our metric in attribute relevance. also, the weight factor of each evaluator can be set as diversity metrics @xcite besides nisf in coherence and consistency. we will leave the exploration of more settings on pattern evaluators as the future work. ## conclusion we present an unsupervised reference-free metric called ctrleval for evaluating controlled text generation. this metric formulates the evaluation of different aspects into multiple text infilling tasks, and utilizes the ensemble of generation probabilities from a pre-trained model in different tasks as the evaluation result. experimental results indicate that ctrleval obtains higher correlations with human judgments and shows better generalization ability for addressing model drift and quality drift.