Proxy Indicators for the Quality of Open-domain Dialogues (2021)

Abstract: The automatic evaluation of open-domain dialogues remains a largely unsolved challenge. Despite the abundance of work done in the field, human judges have to evaluate dialogues’ quality. As a consequence, performing such evaluations at scale is usually expensive. This work investigates using a deep-learning model trained on the General Language Understanding Evaluation (GLUE) benchmark to serve as a quality indication of open-domain dialogues. The aim is to use the various GLUE tasks as different perspectives on judging the quality of conversation, thus reducing the need for additional training data or responses that serve as quality references. Due to this nature, the method can infer various quality metrics and can derive a component-based overall score. We achieve statistically significant correlation coefficients of up to 0.7.

## introduction recently, dialogue systems powered by machine learning have gathered much attention from industry and academia alike @xcite . these systems have various applications, such as personal speech assistants, customer service, technical support, and training and education. in most cases, these systems are task-specific and help with tasks like booking a restaurant. nevertheless, they can still benefit from open-domain conversational skills, such as the ability to chit-chat to enable natural dialogues, rather than repeating the input utterance like a parrot. nowadays, people working in this field have to use human annotators to evaluate the quality of a conversation @xcite @xcite , which can be very costly in terms of resources. thus, these systems' research and development could benefit significantly from an automated approach to evaluate conversations. research in the related fields of text summarization and machine translation has developed automated measures for evaluation. some notable examples are, for the former, rouge @xcite and, for the latter, bleu @xcite and meteor @xcite . these are also adopted by works researching dialogue systems @xcite @xcite . however, @xcite demonstrated that these metrics are not suitable for replacing human evaluators. @xcite reported that even using multiple instead of single references and an overlap-based approach still underperforms. thus, more advanced techniques are needed that consider the context and semantics of a dialogue. human annotators can distinguish bad from good quality dialogues from an intuitive perspective, not because they necessarily have been taught to do so. instead, people have a notion of a fluent text or when a response is relevant (or not) to a previous utterance. thus, this work's primary goal is to investigate a similar approach that mimics component-wise human intuition foot_0 . we investigate whether natural language processing (nlp) tasks can serve as proxy indicators for a conversation's quality. for that purpose, we use a fine-tuned bert @xcite model trained on the glue benchmark @xcite . glue provides a comprehensive evaluation of general language understanding. we demonstrate that a few of the tasks exhibit a limited potential of serving as proxy indicators. the rest shows negative results. more recently, @xcite propose a framework that uses self-play and two nlp tasks as an additional source of knowledge to evaluate dialogues in a multi-turn mode scenario. they perform an ablation study using sentiment and natural language inference as proxy supervision to see whether their system can better approximate human judgment. their work shows that dialogue systems can benefit from using them. also, @xcite frame the dialogue consistency issue as a natural language inference problem and propose the dialoguenli dataset. its purpose is to benchmark a model's ability to select relevant utterances relative to a given context. finally, @xcite offered to treat dialogue evaluation as an anomaly detection problem. their results were negative and suggested that the approach suffers from insufficient training data. until very lately, @xcite , sai et al (2020), and @xcite propose the usage of language models as indicator of quality. all of their approaches require no references or supervision. however, their proposed methods do not separate the different quality aspects and only indicate a dialogue's overall quality. ## general language understanding evaluation this section briefly introduces the general language understanding evaluation benchmark @xcite , its sub-tasks, and their relevance to this work. glue has two categories of tasks -single-and pairwise-sentence tasks. they provide annotated data for training models to solve various natural language understanding problems. the section also discusses how these nlp tasks could be related to dialogue evaluation since they are initially irrelevant to this paper's core topic. the presentation of each of the tasks follows: to evaluate dialogues, cola can provide fluency measures that show how grammatically sound a conversation is. stanford sentiment treebank (sst-2) @xcite ) contains text excerpts from the movie reviews that have their sentiments annotated by humans as positive (one) or as negative (zero). common sense would suggest that attitude provides no apparent relation to dialogue quality. nonetheless, @xcite perform an ablation study as part of their work to see if knowledge distillation based on sentiment offers any benefits to evaluating a conversation. their research shows that there can be an improvement depending on the neural network model and the target dataset. so, we investigate how it relates to annotator scoring on dialogue evaluation. ## methodology 4.1 dialogue datasets to evaluate the ability of a deep-learning model trained on glue to indicate the quality of dialogues, we use the english datasets (topicalchat, personachat) provided by @xcite . they train a few different dialogue system models and use different sequence generation techniques to generate responses for certain dialogue contexts. the researchers then evaluate 660, in total, dialogue contexts and responses according to six criteria: understandable, natural, maintains context, interesting, uses knowledge, and overall quality. for a complete description of the metrics mentioned above and further details about the dataset, we forward the reader to the original work of @xcite . ## conclusion this work considered a model trained on glue as a proxy indicator for the quality of knowledgegrounded dialogues offering different perspectives on dialogue quality criteria. it does not need any references or supervision and can outperform other competing approaches like usr @xcite ). pearson's and spearman's correlation coefficients suggest that single proxy indicators and their various combinations via linear regression can infer dialogue quality either on specific criteria or in general. this composable nature can be used to tune the approach to focus more on particular criteria than others. while one might be concerned that using the approach might offer an advantage to dialogue systems incorporating bert, we think it poses little to no risk. bert is an encoder approach and is considered uncommon for sequence generation applications. hence, the risk of bias is reasonably low. in addition, one could also use any other base model architecture for training glue predictors. the model has no training or fine-tuning that is specifically geared towards dialogues. however, we showed that lack of exposure to conversational data could be problematic for metrics like maintains context. hence, we set as future work to investigate additional pre-training on dialogue data similarly as @xcite , but also considering other proxy indicators like dia-loguenli @xcite , which frame the natural language inference task in a conversational setting. finally, while we used separately trained instances of bert for each of the glue tasks, one could also consider using a multi-tasking method. for example, @xcite present multi-task deep neural networks (mt-dnn) that employ a single instance of bert for all glue tasks. we believe using multi-tasking and bert together would make its application in a productive environment much more effortless, since model weights are to a greater extent shared between the tasks.