Language Model Transformers as Evaluators for Open-domain Dialogues (2020)

Abstract: Computer-based systems for communication with humans are a cornerstone of AI research since the 1950s. So far, the most effective way to assess the quality of the dialogues produced by these systems is to use resource-intensive manual labor instead of automated means. In this work, we investigate whether language models (LM) based on transformer neural networks can indicate the quality of a conversation. In a general sense, language models are methods that learn to predict one or more words based on an already given context. Due to their unsupervised nature, they are candidates for efficient, automatic indication of dialogue quality. We demonstrate that human evaluators have a positive correlation between the output of the language models and scores. We also provide some insights into their behavior and inner-working in a conversational context.

## introduction lately, deep learning conversational systems have seen increasing interest from industry and academia alike @xcite . these systems find usage in various contexts, starting from personal speech assistants like google assistant through the "chatbots" on instant messaging platforms like facebook messenger, and finally, conversational services like luis 1 . many of these applications serve the objective of completing a specific function like purchasing a product or booking services (e.g., hotels, flights). nonetheless, these applications can still profit from open-domain dialogue skills like chit-chatting, which would provide a more human-like interaction with users. presently, scientists and engineers working on computer-based conversational systems need humanbased evaluation to assess the quality and usability of their work @xcite @xcite . these evaluations are costly in terms of resources. thus, the field of dialogue systems could take advantage of an automated method for assessing conversations. seminal works in text summarization and machine translation have already proposed their fieldspecific metrics for automated assessments -for the former rouge @xcite , and, for the latter, bleu @xcite and meteor @xcite . dialogue system research @xcite @xcite constantly uses these metrics. however, @xcite show that these metrics based on word-overlap between prediction and references are not reliable for evaluating the usefulness of dialogue systems @xcite . hence, the field should use more sophisticated methods that consider the previous utterances of a conversation and its semantic meaning. when human annotators evaluate a dialogue, they do not use an explicit reference or necessarily seek word overlap between context and response (or the lack of it). their assessment bases itself on experience with the language and the implicit knowledge they have about it. the core principle of statistical language models (lm) is to capture and reproduce these properties. lm have proven themselves invaluable in state-of-the-art approaches in natural language processing, and natural language understanding @xcite @xcite . thus, the main aim of this work foot_1 is to investigate their usability as means for evaluating dialogues since they do not need a reference or supervision. we demonstrate that there is a significant positive correlation between the predictions of language models and human evaluation scores. furthermore, we provide insights into the inner-workings and behavior of language models in the dialogue context. ## related work in this section, we present earlier work that focuses on dialogue evaluation. furthermore, we provide a concise introduction to language model transformers and recent advances in this particular set of approaches. ## methodology in this section, we report on the used datasets for assessing the usability of transformer language models for evaluating dialogue quality, introduce the used approaches in greater detail and describe their relevance to the task at hand. ## evaluation in this part of the paper, we will conduct a correlation analysis between the calculated probabilities from the lm and the scores given to dialogues by human evaluators. we provide a closer look at some auxiliary model outputs as well. ## conclusion in this study, we investigated whether transformer-based language models can evaluate dialogues in terms of coherency and fluency. overall, pearson's and spearman's correlation coefficients demonstrate that bert, gpt2, and xlnet can indicate a conversation's quality without any additional supervision or reference. while, in their core, the three use the same approach, transformers, they have further structural modifications that set them apart when considered for the current problem domain. gpt2 performs worst due to its standard language modeling approach that incorporates the least structural information about a sequence. xlnet achieves an inmprovement in terms of its correlation score by taking advantage of additional positional information when predicting a target token. finally, bert's next sentence prediction approach delivered the highest performance thanks to its structured approach in regards to separate utterances. while lm-based dialogue evaluators cannot yet replace human annotators, they have additional value when compared to word-over metrics like blue or ones that use word-embeddings. although they cannot completely replace human evaluators, they can support as weak indicators for quality. additionally, we have shown that they can perform better than competing approaches like the unreferenced component of ruber. furthermore, the autoregressive language models, gpt2 and xlnet, demonstrate an excellent initial aptitude for conducting dialogues. they can provide alternative responses that are also coherent with the context of a discussion. the lm-based method in this work considers dialogues as a series paired up utterances or questionanswers rather than one whole sequence. as future work, we will investigate how to extend the procedure so that it is more adept at capturing long-term information over the entire conversation.