Evaluation of Response Generation Models: Shouldn’t It Be Shareable and Replicable? (2022)

Abstract: Human Evaluation (HE) of automatically generated responses is necessary for the advancement of human-machine dialogue research. Current automatic evaluation measures are poor surrogates, at best. There are no agreed-upon HE protocols and it is difficult to develop them. As a result, researchers either perform non-replicable, non-transparent and inconsistent procedures or, worse, limit themselves to automated metrics. We propose to standardize the human evaluation of response generation models by publicly sharing a detailed protocol. The proposal includes the task design, annotators recruitment, task execution, and annotation reporting. Such protocol and process can be used as-is, as-a-whole, in-part, or modified and extended by the research community. We validate the protocol by evaluating two conversationally fine-tuned state-of-the-art models (GPT-2 and T5) for the complex task of personalized response generation. We invite the community to use this protocol - or its future community amended versions - as a transparent, replicable, and comparable approach to HE of generated responses.

## introduction early attempts to evaluate automatic natural language generation (nlg) models using human judges dates back to before the appearance of endto-end models @xcite @xcite . however, due to the expensive requirements such as training skilled annotators and the time-consuming nature of this evaluation, automatic metrics became the common evaluation criteria in several nlg tasks. metrics such as bleu @xcite , meteor @xcite and rouge @xcite have been used to evaluate the model performance in machine translation and automatic summarization tasks as inexpensive and rapid evaluations. after 1 link to the protocol and materials repository observing the reliability of these metrics for the task they are designed for (if applied correctly), they have been used to evaluate the models in other tasks such as response generation. however, several studies have shown that currently available automatic metrics can not be good candidates for evaluating a generated response @xcite ; these criteria co-relate poorly with human judgement and are inadequate since the generation is subject to trivial factors such as coherency, fluency and grammatical structure, as well as non-trivial factors such as appropriateness, engagement, and user acceptance. human evaluation (he) is still the necessary approach to evaluate the generated responses @xcite . with the development of crowdsourcing annotation platforms, conducting an he task is less expensive and more feasible than early methodologies. nonetheless, little attention has been given to the assessment of the design of he task. due to the lack of an agreed upon and standard protocol, he tasks have been performed while suffering from nontransparent procedures, nonreplicable and incomparable results, and unclear resource allocations. in this work, we propose to standardize the experimental methodology for human evaluation for response generation models. we present a detailed protocol to the community for this task, in order to increase the comparability, replicability, and interpretability of such evaluations among works and domains. all the required steps and materials to conduct a he in a transparent and extendable way (including task design, annotator recruitment, task execution, and annotation reporting) are described and shared with the community. the proposed protocol is domain-agnostic, language-independent, and open to be extended to different versions and standards. we invite the community to not only utilize this protocol but also to improve and extend it into referable and version-able standards for the he task. in order to validate the proposed protocol, we evaluate two conversationally fine-tuned stateof-the-art models for the italian language, based on gpt-2 @xcite and t5 @xcite , for the task of response generation in personal dialogues using knowledge grounding. ## literature review earlier attempts to evaluate dialogue systems by human judges considered user satisfaction as the evaluation criterion @xcite . despite the introduction of automatic metrics and a research direction aiming to better the metrics used for the evaluation of dialogue models @xcite @xcite , human evaluation (he) is still the gold standard for assessing the qualities of a generated response and a generative model. while the importance of the proper evaluation of a dialogue model using human judges is wellestablished in the community, how to perform such evaluation is still an unsolved question @xcite . as an outcome, countless he tasks have been presented and conducted in this domain, resulting in non-comparable and non-replicable results. dialogue systems have been evaluated with different granularity (turn-level vs dialoguelevel), different evaluation policies (single-model vs pairwise-model, candidate-ranking vs. winnerselection) and in different modalities (interactive vs static) @xcite . the ambiguities in he tasks conducted so far have also been studied by @xcite , where the authors focused on disentangling the characteristics of already conducted he tasks to increase the interpretability and comparability of the evaluations and results. further inconsistency in the evaluations includes the ambiguity in the criterion name, i.e. two criteria with the same name assess two different qualities in different works, whereas the same quality has been named with various terms among works @xcite . in addition to the aforementioned works, this naming inconsistency can also be found in the grounded generation literature @xcite @xcite @xcite where a criterion with the same name refers to two different qualities and presents different definitions among works. an important factor for reproducing any crowdsourcing experiment is reporting the details related to that experiment and its settings. this issue has been studied by @xcite , where the authors identify the properties that researchers have to provide to facilitate the reproducibility of any crowd-sourcing experiments. the same problem has been studied specifically for he experiments by @xcite where the authors identify the lack of reporting crucial details, and other issues such as high levels of variation among the evaluation procedures. @xcite further stress the need for a standard and coherent experimental design and terminology for the task of he in the community. ## proposed human evaluation protocol we propose to standardize the human evaluation (he) experiments through a referable and replicable protocol to address the problems of noncomparability and inconsistency in the literature. considering the complexity of designing and executing such evaluations, we unfold the task into four main steps in order to study and analyze the crucial aspects at each step. we aim to maximize the reliability and replicability of the evaluation while minimizing the task difficulty and complexity. our proposed protocol consists of four executive steps, i.e. 1) task design; 2) annotator recruiting; 3) task execution; and 4) reporting. ## validation of the protocol we validate the proposed protocol by evaluating two response generation models for the task of personal and grounded response generation. for this purpose, we fine-tuned two of the state-ofthe-art pre-trained language models (plms) for the italian language. the first model fine-tuned is it5 @xcite , which has the same architecture as t5 plm @xcite , pretrained on a large italian corpus. we used it5-base which consists of 12 layers per stack (encoder or decoder) with 220m parameters. the second model fine-tuned in this work is geppetto (de mattei et al., 2020) which is a decoder-only autoregressive plm based on gpt-2 small @xcite , for the italian language. the model consists of 12 layers of decoder and byte-pair encoding, with 117m parameters. the fine-tuning of the two models was done using the dataset of follow-up dialogues collected by @xcite . this dataset is a collection of dyadic conversations about personal events and emotions the narrator has experienced while the listener tends to respond with personalized and helpful suggestions. the dialogues in this data set are based on a personal narrative about the same event and participant that the narrator has shared prior to the dialogue. we fine-tuned the models with and without grounding the generation on the corresponding narrative for each dialogue via the same approach used by @xcite . in our setting the knowledge selection module is not required since the correct narrative for each dialogue is deterministic. ## conclusion while human evaluation is the necessary methodological step in the assessment of response generation models, there is a lack of a standard. this deficiency has resulted in often ambiguous, incomparable and non-replicable published experiments. in this work, we aim at addressing this problem by sharing a complete methodology for evaluating generated responses using human judges. we publish the first version of the protocol and all its materials to the community. the expectation is to engage them to utilize, extend, and complement this protocol into further versions and a transparent resource that can be publicly accessed. the ultimate goal is to engage the community to consider he as an important topic of research. the complete protocol and supporting materials can be found in a public repository (including the guidelines, task design, the ui, and the analysis scripts).