E n D ex: Evaluation of Dialogue Engagingness at Scale (2022)

Abstract: We propose EnDex, the first human-reaction based model to evaluate dialogue engagingness. EnDex is trained on 80k Reddit-based Engagement Dataset (RED) curated using a novel distant-supervision framework. Engagingness is a key measure that captures high-level quality of AI dialogue systems and closely reflects actual user experience. However, data shortage, plus the abstract and extensive definition of engagingness makes it challenging to develop an automatic metric. Our work departs from mainstream approaches that use synthetic negative examples to train binary classifiers, and instead, proposes a solution using distant-supervision from human-reaction feedback. To support the soundness of our EnDex metric, we offer a theoretical foundation for engagement, an extensive ablation study, and empirical evidence of high correlation on five engagingness related datasets. We will release code, off-the-shelf EnDex model, and a large-scale dataset upon paper publication to facilitate future research.

## introduction many modern generative language models are trained to maximize a likelihood objective, but this paradigm tends to assign high probability to generic responses @xcite , such as "i don't know.". prior research has established that people prefer to converse with interesting, creative, and informative agents @xcite , all concepts broadly related to the notion of engagingness. furthermore, engagingness is recognized as a key evaluation metric for the quality of dialogue systems @xcite . for example, fair's parlai @xcite incorporated engagingness as the default testing metric in the blenderbot system @xcite ; dialogue data challenges, like convai2 @xcite , amazon alexa prize foot_1 , and ensemble metrics like fed (mehri and eskenazi, 2020), all measure engagingness to benchmark dialogue quality. however, the current evaluation of engagingness still primarily relies on expensive human annotation rather than off-the-shelf automatic tools, due to several theoretical and technical challenges: firstly, unlike more well-characterized properties such as fluency, the definition of engagingness is significantly more abstract and multi-dimensional @xcite , requiring well-tuned quality metrics for each sub-dimension to aggregate a final score. secondly, what qualifies as engaging is openended and many different answers may embody the concept @xcite . therefore, reference-based metrics requiring unique ground truth, such as bleurt @xcite and bertscore @xcite , cannot apply. thirdly, there's an acute shortage of large-scale, high-quality data annotated for engagingness. ghazarian et al. (2020) jump-started efforts to automatically measure dialogue engagement, where they fine-tuned a bert-based model @xcite on the convai2 and dialydialog datasets @xcite to predict an engagingness score. however, finetuning on small size supervised data could easily lead to overfitting and generalization problems. another high performing metric on engagingness usl-h @xcite assumes a positive set and generates synthetic negative samples to train model. however, credible positive samples are not always available, and synthetic negative samples may not be challenging enough to further advance classifier performance. in light of the above challenges, we propose endex, a novel metric trained with distantly supervised data to predict turn-level dialogue engagingness (figure 1 ). endex requires neither human annotations nor direct disentanglement of engagingness. instead, we leverage observed user reactions to posts as distant signals to model engagingness, which marks a departure from mainstream approach to train on synthetic negative samples @xcite @xcite . endex trains on real conversations sourced from reddit, that are automatically annotated as positive and negative examples with our framework. the novoel dataset is named red (reddit engagement dataset) with over 80k labelled samples. endex framework derives its theoretical underpinning from relevant hci works, and has shown superior performance on five benchmark datasets. ## endex metric engagingness is not only a linguistic concept useful for dialogue systems, but also manifests itself in multi-modalities and is extensively leveraged to benchmark gaming and online learning experiences @xcite @xcite . our work is inspired by hci study of human engagement @xcite , which decomposes engagingness into three major dimensions including attentional engagement (e.g., clicks and scrolls), behavior engagement (e.g., facial expressions), and emotional engagement (e.g., heart rate). endex metric follows the same intuition: we can infer engagingness of a text by analyzing human reactions to it, for which there is abundant data in social media. endex metric learns from our distant-supervised red dataset, which measures dialogue engagement along four dimensions as shown in figure 1 ; three-dimensions correspond to the original human engagement definition, and one distinct reply engagement dimension for the dialogue specific task. engaging non-engaging # of samples 40,162 40,162 emotional .605 ± .273 .152 ± .120 attentional .759 ± .127 .203 ± .100 behavioral .659 ± .274 .318 ± .285 reply .718 ± .154 .354 ± .980 endex .709 ± .048 .259 ± .033 ## conclusion this paper proposes the first human reaction based model, endex, to evaluate dialogue engagingness, and curates an 80k reddit engagement dataset (red) using a novel distant-supervision framework. the success of endex demonstrates the validity of training automatic metrics with human reaction signals, offering a strong complement to a synthetic negative sampling approach. we also release an offthe-shelf endex model, and a large scale dataset to facilitate future research.