Human Evaluation of Creative NLG Systems: An Interdisciplinary Survey on Recent Papers (2021)

Abstract: We survey human evaluation in papers presenting work on creative natural language generation that have been published in INLG 2020 and ICCC 2020. The most typical human evaluation method is a scaled survey, typically on a 5 point scale, while many other less common methods exist. The most commonly evaluated parameters are meaning, syntactic correctness, novelty, relevance and emotional value, among many others. Our guidelines for future evaluation include clearly defining the goal of the generative system, asking questions as concrete as possible, testing the evaluation setup, using multiple different evaluation setups, reporting the entire evaluation process and potential biases clearly, and finally analyzing the evaluation results in a more profound way than merely reporting the most typical statistics.

## introduction human evaluation in natural language generation (nlg) has become a hot topic lately, with the emergence of several survey papers on the topic that study how human evaluation has been conducted in the past in the field of nlg in general @xcite . this has led to several recent evaluation frameworks for evaluating the output of nlg systems @xcite . however, not all natural language generation tasks are of the nature that they are designed to convey factual information. some of the nlg tasks deal with producing text of aesthetic nature such as poetry, stories, humor and so on. we call these creative nlg tasks. these types of tasks are simultaneously researched in two distinct fields of science: natural language processing (nlp) and computational creativity (cc). existing survey papers have only focused on nlp research and they have not made a distinction between creative and non-creative nlg. nlp and cc fields conduct work from very different starting points @xcite . nlp is often state-of-the-art driven whereas cc presents more of exploratory research without pursuing scores that outperform a baseline. in this paper, we want to study how human evaluation of creative nlg systems is conducted in the world of nlp and in the world of cc, what similarities there are and whether the two fields can learn something from each other. we base our research on a literature review on the papers dealing with human evaluated creative nlg published in the 2020 editions of the international conference on computational creativity (iccc) and of the international conference on natural language generation @xcite . we picked these conferences as iccc is the most important venue for cc research, and inlg the most important nlp focused venue for nlg research. our results show that there is no consensus at the moment on how evaluation should be conducted despite the many different efforts of establishing guidelines for evaluating computationally creative output @xcite @xcite . we reflect on the results of our survey and propose a road-map for more sound future evaluation practices. with purely factual data-to-text generation tasks. in the iccc 2020, there were 12 papers that presented human evaluated work on creative nlg, and in the inlg 2020, there were 11 such papers. we selected these papers for our survey. fortunately, both of the venues had relatively the same amount of papers. when surveying the papers, we only focused on human evaluation and we wanted to know what the nlg task was, what parameters were being evaluated (usually reflected by the evaluation questions), how these parameters (questions) were motivated and how the actual evaluation was conducted methodologically. we also paid attention to the evaluation setup: the number of evaluators and samples used and whether the evaluators were experts or laymen. finally, we looked into the discussions and conclusions presented in the papers to see what role the human evaluation had there, especially in relation to concrete future directions in improving the system based on the evaluation results. ## discussion there are currently many different creative nlg tasks people work with, and it is understandable that each task calls for slightly different evaluation methods. however, even work on closely related topics prefers to use their own evaluation methods that are not based on any existing research. and most alarmingly, if the evaluation is based on existing research, the evaluation questions are not motivated in the earlier research either. this type of evaluation has become to be known as a symptom of the great misalignment problem @xcite . when the evaluation is not targeted towards evaluating exactly what has been modelled, any type of evaluation that seems remotely related to the task becomes seemingly valid. however, when the evaluated parameters have only little to do with what was modelled, it is only evident that none of the surveyed papers was able to clearly identify the short-comings of their systems in such a way that they could propose some clear paths to follow for any future research. in fact, if you evaluate your system based on relatedness and funniness while neither is explicitly modelled, how can you know how to make your system more funny or produce more related output? the scores might have well been achieved by mere serendipity (the annotators happened to like the humor that happened to be in the small sample) (c.f. gervás 2017) or by data the model was trained on. apart from the evaluation questions not aligning with the model, a much larger problem related to evaluation questions can be identified. firstly, most of the papers were not clear about the actual evaluation questions used, instead they listed the evaluated parameters as though human evaluation was like an automated one where one can just score abstract notions such as typicality or fluency accurately on a 5 point scale. in other fields, it is known that even small changes in survey questions can lead to different survey results @xcite @xcite . not revealing the actual questions only makes the situation worse. another problem that rises from abstract evaluation questions is that it becomes less clear why the annotators gave certain answers. furthermore, people have a tendency on reading more into computer generated output than what the intention of the system was @xcite . if you train a generative neural model on jokes, it will surely learn to output jokes, while it does not necessarily have any internal representation of humor. in such a case, the humor is purely in the eyes of the beholder and in the data the model was trained on, not in the method itself foot_2 . for instance, @xcite has shown that generated headlines were perceived more offensive by human annotators, while offensiveness was never modelled in the system. while mostly every paper we surveyed opts for coming up with their own evaluation metrics, it is astonishing that these newly created evaluation settings are used as such. there are other fields dealing with human surveys that emphasize the need for conducting tests on your survey before conducting it in a larger scale to discover potential issues in your questionnaire @xcite @xcite . none of the paper we surveyed discusses evaluation of evaluation. in-stead, it is believed that any new evaluation metric the authors came up with just for a given paper will magically work as such and will yield scientifically valid results that will pass a peer review. all this while many of the papers ask questions using ambiguous terms such as fluency (is something grammatical fluent? is something that seems to make sense semantically fluent? is something that is close to the annotator's own idiolect more fluent than something further away from it? is text generated in american english more fluent to americans than text generated in british english? and so on) and coherence (is something that repeats the same words coherent? can a complex figure of language be coherent if the annotator does not have time to think about it for more than a couple of seconds? does coherence have something to do with grammaticality as well? is a story that follows the same beliefs as the annotator seen as more coherent? and so on) that are reduced into a compact 1-5 scale that is later neatly averaged over all the annotators' opinions on all the samples. what does the average of 3.5 on a question all annotators might have interpreted differently even mean? in other fields conducting online surveys, there are a lot of worries about selection bias of the human subjects @xcite . this is hardly discussed in the fields of nlp and cc. many of the papers we surveyed conducted their evaluation on a crowd-sourcing platform such as amazon mechanical turk. none of the papers presented statistics on the demographics of the annotators. this might be a source of bias in the results. what makes such a bias even more problematic is the relatively small number of annotators that are usually recruited per individual output. fields with more established human survey practices would not consider the typical 3-5 annotators of nlp and cc enough even for a qualitative survey, which requires 5-25 participants @xcite or at least 6 participants @xcite . however, human evaluation is usually conducted quantitatively, which means that the number of annotators depends heavily on multiple parameters and requires planning and justification on its own right @xcite @xcite . it is also very well known that people do not perceive things in a vacuum but rather as a continuum of stimuli where previously perceived stimuli affect to the next one. this effect is called priming (see @xcite . to reduce the effect of priming or to have it consistent one should either shuffle the order in which the output is presented to the annotators or keep it always the same. priming is especially in play in cases where annotators are to evaluate outputs produced by different systems. in such a case, output of a mediocre system might get greatly boosted when presented together with output by a bad systems. nearly none of the papers we surveyed discussed this aspect of their evaluation setting. both cc and nlp have still a long way to go in order to reach to more sound human evaluation practices. however, inlg is still a step closer to scientific rigor as automated evaluation metrics were commonly used together with human evaluation, and sometimes as the only evaluation metric (such as @xcite , whereas iccc had several papers presenting work on creative nlg without any evaluation at all (such as @xcite @xcite . the use of experts in evaluation is something that should be taken under rigorous inspection in the future. currently, there are contradicting studies on the topic indicating that consulting expert does have an effect in machine translation @xcite but not in poem generation @xcite . however, this is a question that is very likely to depend on the output that is to be evaluated and also on how the evaluation is conducted. human computer interaction research has some more established methodologies for conducting human studies (see @xcite such as cognitive walk-through (see @xcite , human performance evaluation support system @xcite and user studies (see @xcite . these established methodologies could be taken into account when conducting evaluation of such an nlg system that calls for user interaction. ## advices for future evaluation in this section, we outline how human evaluation of creative nlg systems should be conducted. we are not going to give an exact silver bullet framework to solve the problem, as the two fields are not at the state yet where enough would be known about human evaluation to state exactly how the evaluation needs to be conducted. furthermore, we do not believe that a single fixed framework is enough to capture everything necessary in a topic as broad as creative text generation. ## conclusions in this paper, we have surveyed papers presenting work on creative natural language generation that have been published in inlg 2020 and iccc 2020. there have been many different evaluation methods including some unconventional ones such as critics' reviews and user testing. the most typical human evaluation method has been using a scaled survey, typically on a 5 point scale. while most of the papers surveyed had come up with their own evaluation metrics, the most common parameters that have been evaluated were meaning, syntactic correctness, novelty, relevance and emotional value. although, the terms used to refer to these notions have not been the same. most of the papers did not justify why they had evaluated certain parameters. instead, the parameters were usually just stated as though they were an inarguable fact. it was more often than not the case that the actual evaluation questions were not revealed. there was a lot of variation in the number of samples taken from the system output and how many annotators were used to conduct the evaluation. typically the numbers were rather small. there was no discussion about the demographics of the annotators nor about what type of a bias it might have introduced. evaluation setups were never tested out beforehand, even though other fields dealing with human surveys recommend testing your questionnaires. this means that it is impossible to tell what the annotators really understood by the evaluation questions. we established some advices for future evaluation, which include clearly defining the goal of the generative system, asking questions as concrete as possible, testing the evaluation setup, using multiple different evaluation setups, reporting the entire evaluation process and potential biases clearly, and finally analyzing the evaluation results in a more profound way than merely reporting the most typical statistics. all in all, our fields, cc and nlp, have a lot to learn from other fields with longer traditions with human questionnaires in terms of conducting human evaluation. at the current stage, none of the papers we surveyed quite reached the same level of scientific rigor in their human evaluation as it is to be expected in other fields of science. however, this is not to say that the work of the authors of the papers we surveyed is inherently bad. this is just to highlight the fact that more attention needs to be paid in how human evaluation is conducted. quite often with creative text generation, human judgment is the only viable metric to measure the performance of a system. human evaluation of generated text has been conducted in the field of nlp already as early as in the 1960s @xcite it is a pity it has not caught up with the rest of the development in the field.