Automating Human Evaluation of Dialogue Systems (2022)

Abstract: Automated metrics to evaluate dialogue systems like BLEU, METEOR, etc., weakly correlate with human judgments. Thus, human evaluation is often used to supplement these metrics for system evaluation. However, human evaluation is time-consuming as well as expensive. This paper provides an alternative approach to human evaluation with respect to three aspects: naturalness, informativeness, and quality in dialogue systems. I propose an approach based on fine-tuning the BERT model with three prediction heads, to predict whether the system-generated output is natural, fluent, and informative. I observe that the proposed model achieves an average accuracy of around 77% over these 3 labels. I also design a baseline approach that uses three different BERT models to make the predictions. Based on experimental analysis, I find that using a shared model to compute the three labels performs better than three separate models.

## introduction the evaluation of natural language generation (nlg) systems has generally been carried out by using automatic metrics such as bleu @xcite , meteor @xcite , rouge @xcite , etc. however, previous work @xcite demonstrated that these metrics only weakly reflect human judgments of these nlg systems' output, and some form of human evaluation is required to better measure the quality of such nlg systems. human annotators are generally asked three questions to evaluate whether a system-generated reference is acceptable or not . these questions, along with the corresponding aspects, are: since human evaluations can be expensive and time-consuming, an automated approach to flag such instances could make it easier for system designers to garner insights into the kind of instances the system is failing to generate good text for. in this paper, i propose a bert-based model trained to predict answers to questions pertaining to the three aspects: naturalness, quality, and informativeness, with a "yes" (@xmath0. the proposed model automatically flags systemgenerated references that are not up to a predefined standard. to the best of my knowledge, this is the first attempt to develop an automated model for predicting scores pertaining to multiple aspects of a system-generated reference. the major contributions of this work can be summarized as follows : first, i propose a binarization scheme to binarize the human judgment scores in the dataset as these scores tend to be very subjective. a threshold is set, and all scores above the threshold are assigned a label and the scores below the threshold are assigned another label. second, the bert-based model is fine-tuned to predict three labels, answering the questions corresponding to the three aspects of the system-generated reference. i also perform an ablation study where three separate bert-models are trained independently, each of which predicts a label. the remainder of this paper is structured as follows: section 2 talks about the recent works in the same domain. section 3 discusses about the bertmodel that is used for the experiments. in section 4, i discuss about the dataset, pre-processing required, hyper-parameters as well as the baseline model's design. in section 5, i discuss about the performance of the proposed apporach in comparison with the baseline model. finally in 6, i draw conclusions and outline future works. ## related work several works have been proposed in recent years which focus on fine-tuning bert @xcite and its variants to evaluate the quality of a system-generated text. these approaches tend to correlate much better with human assessments. bertscore @xcite compares the similarity of each token in the system generated reference with each token in the original reference using contextual embeddings rather than exact matching. this metric was observed to relate very closely to human judgments for image captioning systems. moverscore @xcite is another metric that combines contextualized representations with distance measures. this metric was observed to generalize well across various tasks like summarization, machine translation, image captioning, and data-to-text generation. bleurt @xcite ) is a bert-based model that was pretrained on a large amount of synthetic data. this model can then be fine-tuned on a relatively small number of human judgments. it was observed to be very effective when the training data is scarce and imbalanced. comet @xcite ) is another neural framework that is used for training multi-lingual machine translation quality evaluation models. all these works evaluate only the quality of the system-generated reference. while quality is correlated with the other aspects of the utterance, it might not be sufficient to capture all insights about an incorrectly generated text with just a single aspect. a grammatically correct text could lack some vital information that was present in the original reference (informativeness) or may not capture the natural speech patterns of a native speaker (naturalness). @xcite proposed an automatic method for evaluating the naturalness of generated text in dialogue systems by fine-tuning a bert-based model. the proposed model predicts a score between 1 and 6, indicating how natural the systemgenerated utterance is. however, this work does not consider that human judgments tend to be subjective. the data being fed to the model is therefore ambiguous in nature. in addition, the best model proposed in this paper uses human judgments on other related aspects like quality and informativeness by leveraging the positive correlation between these three aspects. however, this paper proposes a solution that eliminates human evaluation at in-ference time. human annotations are used only for training the model. after training, the model can mimic/replace human annotators. given the success of bert-based models for system evaluation, i also use pre-trained bert in my approach. ## method bert stands for bidirectional encoder representation transformer. the architecture of bert was based on the encoder part of transformers @xcite . bert uses attention mechanism @xcite to convert the input representation into a better representation that takes context into account @xcite . bert makes use of fine-tuning to leverage the knowledge gained from pre-training. this means that bert is pretrained on a relatively generic task, and the same architecture is fine-tuned on similar downstream tasks. in this paper, i use the uncased bert-base model. that consists of 12 layers, 768 hidden states, and 12 attention heads. we will be leveraging the pre-training knowledge gained from nsp more than mlm. a @xcite token is added to the systemgenerated reference's beginning. a [sep] token is then added to the system-generated reference, followed by the original human-written reference. this is again followed by a [sep] token. the tokens fed as input are tokenized using wordpiece embeddings. sequence embeddings are also passed as input which stores information about which sentence the token belongs to. positional embeddings from the transfomer model are added to the input word embeddings along with sequence embeddings. so the model takes in two sentences as the input and predicts whether the second sentence follows the first sentence or not. the encoded representation of the [cls] token contains information about the representation of the entire sequence. this is called pooled output. the pooled output is passed through a linear layer which is then followed by the output layer with 3 nodes having sigmoid activation. the final output is three values indicating the probability that the system reference is natural, fluent, and informative, respectively (see fig 1 ). dataset in this paper. the dataset contains textual dialogue response from rnnlg foot_0 , tgen foot_1 and lols foot_2 . these are data-driven natural language generation systems that were applied on 3 different but closely related domains-sf hotel, sf restaurant @xcite and bagel @xcite respectively. sf hotel and sf restaurant are based on information regarding hotels and restaurants in san francisco, while bagel has information about restaurants in cambridge. for every nlg systemgenerated reference, there is also a human-written reference in the dataset. the dataset also contains scores from 3 different human annotators for the system-generated reference's naturalness, quality, and informativeness. these scores were provided on the 6-point likert-scale with the lowest score being one and the highest being six. hence, to remove this ambiguity in the dataset, i decided to binarize the dataset by defining a fixed threshold. @xcite classify all the ratings with scores greater than or equal to 5 as good ratings. hence, i chose 5 as the threshold. all the instances with median scores below five are assigned a label of '0' and are considered bad utterances. all instances with median scores greater than or equal to 5 are assigned a label of '1' and considered good utterances. ## results and discussion also, such a model can generalize well on new aspects that can be added in the future. the results suggest that both models can model the data well despite the class imbalance. this can be attributed to the balanced cross-entropy loss function. i use the anova test @xcite to test the statistical significance of the difference in the f-1 scores and accuracies between both approaches. i set the significance level to 0.05. i observe that the results are statistically significant for naturalness and informativeness, which clearly demonstrates that the shared bert model outperforms the 3-bert model on these two aspects. for the aspect of quality, 3-bert shows better performance. however, the gain in performance is not statistically significant. further, in terms of model complexity, shared-bert has only 2304 (768x3) learnable parameters more than a single bert model, and the 3-bert approach has three times the number of learnable parameters compared to a single bert model. hence, shared-bert is a more efficient model in terms of memory occupied and computational complexity. qualitative example: i consider the example instance from ## conclusion and future work in this paper, i proposed an automated approach to evaluate three aspects of a system-generated sentence : naturalness, quality, and informativeness. i experiment with two bert-based model approaches. experimental validation suggests that the proposed approach that uses a single bert model with three prediction heads is more efficient than three different bert models with a single prediction head each. the goal of this paper is to reduce the load on human annotators and automate the evaluation of dialogue systems. i hope that this work will motivate researchers to realize that this process can be automated and be made more reliable with the collection of additional relevant data. further, aspects other than the three considered in this paper can yield some more insights into the performance of a dialogue system. as an extension of this work, i will verify the performance of my approach on other nlg systems like image captioning, question answering, machine translation, etc.