  Human ratings are one of the most prevalent methods to evaluate the
performance of natural language processing algorithms. Similarly, it is common
to measure the quality of sentences generated by a natural language generation
model using human raters. In this paper, we argue for exploring the use of
subjective evaluations within the process of training language generation
models in a multi-task learning setting. As a case study, we use a
crowd-authored dialogue corpus to fine-tune six different language generation
models. Two of these models incorporate multi-task learning and use subjective
ratings of lines as part of an explicit learning goal. A human evaluation of
the generated dialogue lines reveals that utterances generated by the
multi-tasking models were subjectively rated as the most typical, most moving
the conversation forward, and least offensive. Based on these promising first
results, we discuss future research directions for incorporating subjective
human evaluations into language model training and to hence keep the human user
in the loop during the development process.
