Estimating Subjective Crowd-Evaluations as an Additional Objective to Improve Natural Language Generation (2021)

Abstract: Human ratings are one of the most prevalent methods to evaluate the performance of NLP (natural language processing) algorithms. Similarly, it is common to measure the quality of sentences generated by a natural language generation model using human raters. In this paper we argue for exploring the use of subjective evaluations within the process of training language generation models in a multi-task learning setting. As a case study, we use a crowd-authored dialogue corpus to fine-tune six different language generation models. Two of these models incorporate multi-task learning and use subjective ratings of lines as part of an explicit learning goal. A human evaluation of the generated dialogue lines reveals that utterances generated by the multi-tasking models were subjectively rated as the most typical, most moving the conversation forward, and least offensive. Based on these promising first results, we discuss future research directions for incorporating subjective human evaluations into language model training and to hence keep the human user in the loop during the development process.

## introduction creating spoken dialogue systems includes a multitude of challenges as they involve various language processing (nlp) components. one such important component concerns natural language generation (nlg). traditionally, the performance of a nlg unit has been evaluated using automatic metrics, such as bleu @xcite or meteor @xcite . human evaluations of nlg (i.e., rating autonomously generated dialogue responses) is still the most common (see @xcite ; @xcite ; @xcite ; @xcite for measuring the performance of such approaches). comparing automatic metrics with human evaluations, however, has shown little correlation between the two @xcite @xcite @xcite , which stresses the importance of using human evaluations to rate the suitability of a system or part of a system that will ultimately be used by humans again. in recent times, appreciable advances have been made in developing automated metrics showing correlation with the human ratings @xcite . these approaches, however, do not provide a method for measuring the affect and emotional aspects of the generated content which is central to our approach. despite human evaluations becoming increasingly prevalent and affordable, they are usually only seen as the final stage of the system design process. evaluations are hence performed after concluding the implementation work and used to compare the new approach to previous models or techniques. the resulting feedback from the users is then discarded unless used for future comparisons. in this paper, we argue for keeping the human user in the loop by including human evaluations in subsequent natural language generation processes. to keep the development of such a system at a fast pace and low overhead cost, human evaluations can not rely on a few experts but need to utilize online crowd-workers. while crowd-sourcing platforms allow us to gather ratings of several hundred dialogue lines within a few minutes, such evaluations cannot rely on sophisticated metrics requiring a high skill or long training process of the raters but need to utilize subjective ratings of lines instead. to the best of our knowledge, this paper is a first proof-of-concept exploration to include subjective utterance ratings from crowd-workers collected at a low cost and in a short time during the training of a system which is generating responses for a dialogue agent. as a domain, we use the geography-themed cooperative guessing game rdg-map in which a human and an embodied conversational agent try to identify countries on a world map @xcite . to enhance the social component of the dialogue, the human-robot team has a brief chat before and after each game. ultimately, we aim to increase people's engagement playing with the agent by adapting its behavior to the human player. depending on the learning and playing style of a person, the agent should maximize the team's performance by either encouraging or challenging the team mate during play. as a first step, the agent was given two affect states based on @xcite which influence its dialogue behavior: in addition to an indifferent behavior, utterances could be excited and encouraging or impatient and provocative. depending on the team's performance in the game and the human responses to the agent, the affect state of the agent gradually changes over time. initially, we used a crowd-authoring technique to gather potential responses for our dialogue agent @xcite . it has previously been shown that such crowd-authored content helps achieve variety in a dialogue system's responses @xcite @xcite . to design the system described in this paper, we first gathered subjective evaluations by a different set of crowd-workers, rating the dialogue utterances on the dimensions typicality, offensiveness, and affect. we then used these utterances for training models for neural language generation. we trained six model variations to generate responses for different combinations of game scenario descriptions and affective state. two models were trained using multi-task learning goals, making the estimation of the subjective affect rating of the utterance a secondary goal of the model. the main contribution of this paper is the performance analysis of the multi-task models trained on crowd-sourced evaluations compared to the models solely tasked with generating dialogue lines. to compare the different models, they were used to generate utterances for scenarios both seen and unseen during training and resulting dialogue lines were then fed back into the evaluation system, acquiring the same human evaluations obtained for the original crowd-authored lines. in addition to analyzing differences in subjective ratings of the dialogue lines, we compare the human evaluations to the bleu score as an example of a traditional automatic metric for evaluating language generation models. we conclude the pa-per by discussing advantages and challenges of our human-in-the-loop language generation pipeline and suggest future work to improve upon and further evaluate the suitability of our proposal. ## related work the role of crowd-workers in the development of nlg models can be two-folded: sentences provided by crowd-authors can be utilized as a source of the surface form of the sentences that the dialogue system needs to generate or as feedback about the performance of the nlg model. methods for crowd-sourcing content include: (i) requesting the users to generate a sentence given a context @xcite , (ii) asking users to generate surface forms using templates @xcite , and (iii) showing the dialogue to crowd-workers and asking them to paraphrase a given dialogue @xcite . utterances collected using these approaches have been shown to be diverse and have been used to train neural nlg models, some of which have achieved impressive results in recent times. another method to utilize crowd-sourcing is to request crowd-workers to rate the generated sentences on various performance metrics @xcite . recent works have studied utilizing human evaluations to train neural models directly @xcite . human judgments were shown to be particularly useful for machine learning tasks where the loss function for the intended learning goal is difficult to express with the data alone. the related work, however, did not focus on dialogue generation but on other tasks that are difficult to quantify objectively, like summarization. while recent prominent neural nlg models have been able to generate human-like sentences, they are not only very large (in terms of the number of parameters), but also trained on enormous data sets (in terms of the number of training samples) @xcite @xcite @xcite . such models can respond well even in challenging dialogue tasks @xcite @xcite . due to the hardware and data requirements of such models, fine-tuning pre-trained models is a popular approach for obtaining well-performing language generation models @xcite @xcite . lack of consistency is one of the major issues in neural dialogue generation, which has been tackled by methods such as including persona or situation description to improve the consistency between generated sentences across multiple turns of dialogue. @xcite @xcite . in a similar fashion, the question of how to incorporate information that enables the consistent generation of affective, empathetic, or emotional dialogue has been extensively studied @xcite @xcite @xcite @xcite . in this work, we extend the literature by exploring an approach for developing an nlg pipeline using crowd content and subjective evaluations for a limited corpus of in-domain data. following pruksachatkun et al. (2020 ), we leverage the empathet-icdialogues (ed) corpus by rashkin et al. (2019) as an intermediate training step before training on the domain-specific data. we apply models by radford et al. (2019) and zhang et al. (2020) on the crowd-sourced content and human evaluations to generate utterances for the given domain. like in the works of wolf et al. (2019b), zandie and mahoor (2020) and ziegler et al. ( 2019 ), we use pretrained models to reduce the amount of hardware and crowd-sourced data needed. however, we do not use human judgments for reinforcement learning, like @xcite or @xcite , but for supervised learning. ## a crowd-sourced human evaluation pipeline our pipeline to collect crowd-sourced ratings of dialogue lines follows the approach described by @xcite with few alterations. in the first evaluation stage, a set of untrained crowd-workers are asked to judge how typical and ordinary a sentence is given a situational description and how offensive it is on a five-point likert scale. they are also asked if the utterance is nonsensical, in which case the relevancy and offensiveness questions are skipped. the second evaluation stage focuses on the affect of utterances, and workers are asked to judge whether a sentence is excited, frustrated or indifferent. in case they perceived the sentence as excited or frustrated, they need to mark the strength of the affect on a scale from 1 (slightly) to 4 (extremely). for easier computation going forward, the affect rating is combined into a single scale ranging from -4 to +4, with negative values indi-cating frustration, 0 indicating indifference, and positive values indicating excitement. the pipeline runs fully automatically, given a set of input utterances. each new task that is created and uploaded to amazon mechanical turk consists of five utterances and is rated by five different crowd-workers. crowd-workers are allowed to take multiple tasks in a row, which results in a varying level of familiarity with the task of individual raters. once evaluations for the first and second stage have been performed by five people, their scores are automatically aggregated into a single average rating per line. figure 1 shows a sample evaluation of a line written by a human crowdworker and three language generation models for a given scene. crowd-workers were required to be based in the us and have an approval rate of at least 80% to take our hits. they received $ 0.15 usd per task they completed. participation was fully anonymous and no personal data was collected. people who responded randomly to our task (see section 7 for a discussion) were manually flagged as unreliable. their ratings were consequently removed from the result aggregation, and a respective number of replacement tasks were uploaded. ## analysis the performance analysis of the two models utilizing multi-task learning in comparison to the four models trained with the sole task of generating dialogue lines was based both on automated metrics as well as a second round of human evaluations. to get a first estimate of how well the models predict the rdg-map data, the average per-token perplexities of the models on the test set were recorded. we also calculated the average bleu score for utterances generated from scenarios in the test set. for each generated utterance, all corresponding lines written by humans for that specific combination of scenario and affect were used as references. for the human evaluation of the different models, a set of utterances to be evaluated was generated. all models used top-k decoding with @xmath0. six scenarios (three seen and three unseen during training) were used for testing, with three affect categories each (excited, indifferent, and impatient). each model generated five utterances for each of the six scenarios with the three affects. each model thus had 90 utterances evaluated, for a total of 540 utterances across all models. the evaluation pipeline described in section 3 was used to gather human ratings of the utterances generated by the language models. one additional question was added to the first stage, asking crowdworkers to rate how much the given utterance moves the conversation forward. 258 workers participated in the evaluation. each worker participated in 4 different tasks on average, with a standard deviation of 10 tasks. ## discussion & future work we trained six variations of neural language generators on crowd-sourced content and evaluations. our results suggest that dialogpt (mt), the model additionally tasked with predicting the subjective evaluations by crowd-workers, produced utterances that were perceived as the most typical, least offensive, and most capable of moving the conversation forward. it also generated dialogue lines covering the widest range of affects, which meets an important goal for the spoken dialogue system of the rdg-map domain. utterances generated by dialogpt (mt) reach scores comparable to those given to human-authored lines in the dimensions relevance and affect for scenarios both seen and unseen during training; in real-time and at a lower cost than the crowd-sourced approach. based on these results, we consider the multitask learning approach a success. utilization of subjective ratings while our results are promising when it comes to the success of using subjective ratings as a secondary goal in multi-task learning to generate affective dialogue lines, further research is necessary to understand the exact influence of this particular training objective. in this work, we added two additional training goals in order to further utilize the collected data: multiple choice and emotion classification. hence, it may be possible that the multiple-choice task was more influential for the success of the dialogpt (mt) model. however, in observing the training process, it was noted that the training loss for the multiple choice task decreased significantly faster automatic human subjective evaluation model data bleu forwardness offensive typical m sd m sd m sd m sd dialogpt (mt) @xmath1.41 0.3 3.6 0.5 2.5 0.6 3.2 0.6 dialogpt (mt) rdg 0.37 0.3 3.5 0.5 2.4 0.6 3.3 0.6 human ----1.9 0.7 3.4 0.7 ## conclusion the usage of human-annotated data for training machine learning models is an established practice. in this work, we propose and evaluate the utilization of subjective human evaluations for model training that would otherwise be used merely for evaluation. our results suggest that by using not only crowdsourced content, but also crowd-sourced evaluations, we can increase the performance of our models. we hence argue that future work should explore the inclusion of further subjective ratings and the possibility to make model generation and evaluation an iterative process and hence keep the human in the loop during the development process.