Automatic Evaluation of Generative Models with Instruction Tuning (2023)

Abstract: Automatic evaluation of natural language generation has long been an elusive goal in NLP. A recent paradigm fine-tunes pre-trained language models to emulate human judgements for a particular task and evaluation criterion. Inspired by the generalization ability of instruction-tuned models, we propose a learned metric based on instruction tuning. To test our approach, we collected HEAP, a dataset of human judgements across various NLG tasks and evaluation criteria. Our findings demonstrate that instruction tuning language models on HEAP yields good performance on many evaluation tasks, though some criteria are less trivial to learn than others. Further, jointly training on multiple tasks can yield additional performance improvements, which can be beneficial for future tasks with little to no human annotated data.

## introduction natural language generation (nlg) has made significant leaps forward in recent years thanks to large language models (llms; @xcite . yet, to date, there is no standard evaluation protocol for nlg systems. human evaluation provides the most accurate assessment, but its costly and time-consuming nature makes it less practical for large-scale evaluations, and it's rarely conducted as part of the system development cycle. for this reason, automatic evaluation metrics have been widely adopted. the majority of automatic metrics compare the system outputs against a set of reference texts, measuring either lexical overlap (e.g., @xcite or semantic similarity (e.g., @xcite . reference-based metrics suffer from many drawbacks. first, system outputs that are different from the references are scored low, even if they are correct. second, multiple studies have noted poor correlation with human judgements @xcite @xcite figure 1 : example from heap, originally taken the tellmewhy dataset @xcite , here focusing on the question answerability (qa) criteria. @xcite . third, methods that were designed with one task in mind, such as bleu @xcite for machine translation and rouge for summarization @xcite , don't necessarily transfer well to other tasks @xcite . finally, by producing a single score based on similarity to the references, some important but more nuanced dimensions might be missed, such as faithfulness, answerability, and more. a recent alternative approach is learned metrics. such metrics leverage a pre-trained language model and fine-tune it to emulate human judgements (e.g.; @xcite . learned metrics are typically tailored to specific tasks (e.g., machine translation) and criteria (e.g., similarity to the references), and they can be reference-based or reference-less. in this work, we propose to train reference-less learned metrics using instruction tuning. instruction tuning involves presenting the model with natural language instructions in addition to the task inputs. including the instructions as part of the input enables models to generalize better, perform well in zero-shot and few-shot settings @xcite , and better align with human values @xcite . to train our metric, we collected the human evaluations of answer pairs dataset (heap). heap was composed from the human evaluation results for 8 datasets, along 22 diverse evaluation criteria, such as comprehensibility, appropriateness, grammaticality, and informativeness, as detailed in we used heap with instruction tuning in singletask, multi-task, and cross-task setups. we find that most criteria are learnable, though more nuanced or complex ones (e.g., answer validity) are more difficult to learn than others (e.g., grammaticality). we also show that fine-tuning on the task is essential, and that multi-tasking can help with the more difficult tasks. finally, the cross-task setup is less successful, but can be improved by training only on a subset of similar tasks to the target task. we hope that our findings will guide future research on automatic evaluation for nlg systems. 2 2 related work automatic evaluation of generative tasks. numerous automatic methods exist for evaluating generative models. the majority of metrics involve assessing the similarity between a generated output and a reference text. commonly used metrics include bleu @xcite , rouge @xcite , and meteor @xcite , which focus on measuring lexical overlap between generated outputs and a reference. more recent methods, such as bertscore @xcite , go beyond lexical overlap by embedding both the generations and the references into a shared space and computing cosine similarity between the embeddings. all these metrics operate at the surface level, predominantly focusing on lexical similarity. some metrics have been proposed which are trained to emulate human judgements. bleurt @xcite ) is a bert-based metric which is first trained to estimate the scores from existing automatic metrics for a large number of synthetic sentence pairs, and then trained to emulate human judgements for a machine translation task. similarly, @xcite proposed a robertabased metric for dialogue evaluation which is first trained on a large number of sentence pairs with a next sentence prediction objective, and then trained on a small number of human annotations for the task. learned metrics are tailored to specific tasks. they can take on different forms: reference-based, where the metric is trained to compare the system's output to a reference text, like in bluert; reference-less, where the metric scores the output along some criterion without the use of references @xcite ; or a combination of both, as seen in ghazarian et al. (2019) instruction tuning. instruction tuning is a finetuning technique that involves training a model on a variety of tasks, leveraging natural language instructions to guide the model towards producing the correct answers. recent studies have showcased the effectiveness of this technique in improving llms' ability to generalize in a zero-shot and fewshot setting @xcite @xcite . most pertinent to our work, @xcite applied instruction tuning to 48 dialogue-related tasks, including dialogue evaluation. they showed that their instruction-tuned dialogue evaluation metric achieves improved correlation with human judgements, even in a crosstask setup when training on other dialogue tasks. in this work, we use instruction-tuning to train automatic evaluation metrics for a diverse set of tasks and criteria. the use of instructions allows for more transferability between different tasks and criteria, and could be beneficial when data for a particular task is sparse. ## dataset we introduce the human evaluations of answer pairs (heap) dataset. heap is designed to train and evaluate automatic methods for the evaluation of generative tasks. it is derived from existing human evaluations that were performed on 8 generative tasks detailed in the original human evaluations for some of the datasets included comparative evaluation (i.e., task #examples dataset advice helpfulness [ah] 1,200 turingadvice: advice generation (zellers et al., 2021) answer grammaticality [ag] 598 tellmewhy: answering why-questions (lal et al., 2021) answer validity [av] 598 question answerability [qa] 1,917 hellaswag [hs] 149,841 hellaswag: commonsense nli (zellers et al., 2019) commonsense reasoning [cr] 1,079 commongen: commonsense reasoning (lin et al., 2020) best counter narrative [bcn] 1,000 counter narratives against hate speech (tekiroglu et al., 2022) choose-or-not [ccn] 884 grammaticality [cng] 863 specificity [cnsp] 1,139 suitability [cnsu] 1,471 counter narrative informativeness [cni] 783 chasm: countering online hate speech and microaggressions (ashida and komachi, 2022) counter narrative offensiveness [cno] 685 counter narrative stance [cnst] 724 hate speech offensiveness [hso] 29,970 story rewriting counterfactual [src] 4,400 timetravel: counterfactual story rewriting (qin et al., 2019) story rewriting ending [sre] 4,400 story rewriting plot [srpl] 4,400 story rewriting premise [srpr] 4,400 story rewriting second [srs] 4,400 attenuator effectiveness [dia] 7,176 defeasible inference (rudinger et al., 2020) intensifier effectiveness [dii] 7,176 ## method we propose to fine-tune pre-trained language models to predict a scalar score for text outputs along various criteria. we train the models using natu-ral language instructions (sec 4.1) and investigate the extent that this setup allows for out-of-domain generalization for new tasks (sec 4.2). ## experimental setup baselines. other than the single-task, multi-task, cross-task, and cross-cluster setups described in sec 4.2, we also included the base setup, in which we used bart off-the-shelf without fine-tuning it. hyper-parameter tuning. we performed hyperparameter tuning on the validation set to select values for the following: learning rate (2e -5, 2e -4, 3e -4), gradient accumulation (4, 8, 16, 32, 64, 128), number of epochs (1 -20), truncation, instructions, and labelling elements of the input. the selected values are available in appendix a. evaluation metrics. we evaluated the performance of our models using two metrics. the first metric is accuracy on the respective test set. that is, we obtained scores r good and r bad for the respective answers, and counted the percent of instances for which r good was greater than r bad . the second metric is spearman rank-order correlation between the scores outputted by the model and the original human evaluation scores. this metric shows the extent to which the model's preferences align with human preferences. ## results fine-tuning is essential. the base model has an average accuracy of 50.58% across tasks, which is akin to a random baseline. the single task setup substantially improves upon the base model with an average of 63.24%. most criteria are learnable, as evident by the 12.66% difference in accuracy between the base and the single-task models. however, for a few tasks, even the best performance remains relatively low: src, sre, srpr, and srs. these tasks all come from the timetravel dataset of counterfactual story rewriting @xcite and they are inherently difficult, as they require comparing two almost identical stories along various dimensions. multi-tasking is beneficial. on average, the multi-task setup achieves 68.82% accuracy, 5.58% higher than the single-task setup. this indicates that there is transfer learning among the different tasks. perhaps trivially, multi-tasking is especially beneficial when the single-task accuracy is low. the performance of tasks such as cnst, cng, cr, and cni that already achieve good performance in the single-task setup either decreases or increases very slightly. conversely, multi-tasking is the most beneficial for tasks that achieve low single-task performance, such as av and bcn. success in the cross-task setup varies. the cross-task setup performs substantially worse than the multi-task setup (54.85% compared to 68.82% on average), which is expected since the target task training data is excluded. compared to the singletask setup, the cross-task setup is beneficial for ccn, cnsu, sre, bcn, and av, but even in those cases, it is less beneficial than the multi-task setup. for cno, the cross-setup performed substantially worse even than the baseline, but we couldn't find a reasonable explanation for this behavior. the number of per-task examples is not the most important factor. notably, the number of examples available for each task had very weak correlations with the single-task performance (pearson ρ = 0.15), the gain from multi-tasking (ρ = -0.25), and the gain from the cross-task setup (ρ = 0.16). we conclude that among the important indicators for good performance are the ease of the task, i.e., "easy" tasks such as verifying grammaticality can already achieve good performance without training on additional tasks. choosing the right tasks for transfer matters. ## conclusion we proposed to use instruction tuning to learn automatic evaluation metrics. to test the effectiveness of this approach, we introduced heap, a collection of human judgements along diverse dimensions for various generative tasks. our experiments confirm the importance of fine-tuning for developing metrics that align with human judgements. further, we showed the advantage of fine-tuning on multiple tasks, and that a cross-task (zero-shot) setup yields positive results when trained on selected tasks. collectively, our experiments reveal the value of instruction tuning in the domain of automatic evaluation of generative tasks. we hope that our findings will serve as a catalyst for inspiring future research on this topic.