HERALD : An Annotation Efficient Method to Detect User Disengagement in Social Conversations (2021)

Abstract: Open-domain dialog systems have a user-centric goal: to provide humans with an engaging conversation experience. User engagement is one of the most important metrics for evaluating open-domain dialog systems, and could also be used as real-time feedback to benefit dialog policy learning. Existing work on detecting user disengagement typically requires hand-labeling many dialog samples. We propose HERALD, an efficient annotation framework that reframes the training data annotation process as a denoising problem. Specifically, instead of manually labeling training samples, we first use a set of labeling heuristics to label training samples automatically. We then denoise the weakly labeled data using the Shapley algorithm. Finally, we use the denoised data to train a user engagement detector. Our experiments show that HERALD improves annotation efficiency significantly and achieves 86% user disengagement detection accuracy in two dialog corpora.

## introduction evaluation metrics heavily influence a field's research direction. the ultimate goal of open-domain dialog systems is to provide an enjoyable experience to users. previous research mainly focuses on optimizing automatic dialog evaluation metrics such as bleu, which models the distance between the system responses and a limited number of references available. however, it has been shown that these metrics correlate poorly with human judgments @xcite . open-domain dialog system evaluation has long been one of the most difficult challenges in the dialog community for several reasons: (1) the goal of 1 equal contribution. dialog evaluation should be to evaluate users' conversational experience. existing automatic evaluation metrics such as bleu are mostly constrained to a static corpus, and do not capture the user experience in a realistic interactive setting. (2) currently, self-reported user ratings are widely used to evaluate open-domain dialogs. however, self-reported ratings suffer from bias and variance among different users @xcite . although we could tell which dialog system is better by running statistical tests on a large number of noisy ratings, it is challenging to locate dialogs with bad performance reliably. only by identifying these bad dialogs effectively can we correct errors in these samples to improve dialog system quality. user engagement has been recognized as one of the essential metrics for open-domain dialog evaluation @xcite . previous research also confirms that incorporating user engagement as real-time feedback benefits dialog policy learning @xcite . one of the most costly bottlenecks of learning to detect user disengagement is to annotate many turn-level user engagement labels @xcite . in addition, the data annotation process becomes more expensive and challenging for privacy-sensitive dialog corpora, due to the privacy concerns in crowdsourcing @xcite . to improve annotation efficiency, we reframe the training data annotation process as a denoising problem. specifically, instead of manually labeling each training datum, we automatically label the training samples with a set of labeling heuristics. the heuristic functions primarily consist of regular expressions (regexes) and incorporate open-sourced natural language understanding (nlu) services. since the automatically generated labels might contain noise, we then denoise the labeled data using the shapley algorithm @xcite . we use the shapley algorithm to quantify the contribution of each training datum, so that we can identify the noisy data points with negative contribution and then correct their labels. our experiments show that herald achieves 86% accuracy in user disengagement detection in two dialog corpora. our proposed framework herald is conceptually simple and suitable for a wide range of application scenarios: first, since our model could detect user engagement in real-time (i.e., after each user utterance), our model could be plugged into existing dialog systems as a real-time user experience monitor module. in this way, dialog systems could detect and react to user's disengagement in both open-domain dialogs @xcite and taskoriented dialogs @xcite . during training, our model could also be used as real-time feedback to benefit dialog policy learning @xcite . second, herald could quantify user engagement and be used as an automatic dialog evaluation metric. it could locate dialogs with poor user experience reliably to improve dialog system quality @xcite . third, user engagement is an essential objective of dialog systems, but few dialog datasets with user engagement ratings are available. our heuristic functions, combined with the proposed workflow, can be readily deployed to annotate new dialog datasets. ## problem formulation we defined engagement as the degree to which users are willing to continue conversing with the dialog system @xcite . we focus on identifying the dialog turns with "disengaged" user response, since they usually indicate poor conversation experience. we formulate the user engagement prediction as a binary classification problem: our goal is to learn a parameterized user engagement predictor m θ that, given a dialog turn (along with its dialog context) @xmath0, predicts the turn-level user engagement label @xmath1, 1}, where label @xmath2. we start from an unlabeled train set d @xmath3. the test set d @xmath4, y i )} n test 1 contains the ground-truth label y i . the development set d dev has a similar structure as the test set d test but the development set can be much smaller than a train set (i.e., n dev n train ), making it economical to obtain. following the general architecture of neural classifiers, we formulate our model m θ = m(φ, f ) = f (φ(x)): here bert @xcite -based φ is a text encoder that maps each dialog turn x to a feature space φ(x) ∈ r d . f is the final linear layer with softmax activation. ## data to ensure our framework is generalized to various corpora, we investigate multiple open-domain dialog datasets ranging from asr-based (gunrock @xcite ) to text-based (con-vai2 @xcite , blender @xcite , and meena @xcite ) dialog systems. gunrock movie dataset gunrock movie dataset consists of dialog data collected from gunrock, an asr-based open-domain social chatbot originally designed for amazon alexa prize @xcite . the gunrock dataset comes from a user study where in-lab users were recruited to carry on conversations. we have consent to use the data and we also removed any sensitive information in the conversation. two dialog experts (co-authors of this paper) randomly annotated 134 dialogs and split them evenly into the test set and development set. in total, the experts labeled 519 turn-level disengaging user responses and 2,312 engaging user responses. they reached a high inter-annotator agreement score @xcite ) with kappa κ = 0.78. the training set contains 276 unlabeled dialogs, with 5644 dialog turns. in addition, we ensure that the data annotation is independent of the labeling heuristics collection, so there is no data leakage problem. a full example dialog can be found in appendix a.4. sational intelligence (convai) challenge @xcite . we select dialogs from the main eight participated chatbots (bot 1, 2, 3, 4, 6, 9, 11) and exclude dialogs that are one-sided or shorter than three turns. the dialog experts annotated 207 dialogs in total. the dialogs are evenly distributed over all the eight bots to ensure system diversity, and are randomly sampled within each bot. the annotated data consist of 209 disengaging turns and 1684 non-disengaging turns. they reached a high inter-annotator agreement score @xcite with kappa κ = 0.76. we split the annotated dialogs evenly into the test set and develop set. the training set contains 2,226 dialogs, with 18,306 dialog turns. ## method our goal is to train a user engagement detector with minimum data annotation efforts. traditional supervised learning paradigms require annotating many training samples. in addition, it requires additional data annotation to extend the model to a new dialog corpus. to reduce annotation work, we propose herald, a two-stage pipeline that annotates large-scale training data efficiently and accurately (figure 1 ). instead of hand-labeling training data points, we use heuristic functions to label each training datum automatically. the heuristic functions are built upon a set of user disengagement heuristics rules. since the training data are automatically labeled, their labels would be noisy. we then clean the noisy training data with shapley algorithm @xcite to improve the labeling accuracy. the shapley algorithm denoises training data by identifying data with wrong labels and flip their labels. finally, as we received clean training data, we use them to fine-tune a bertbased model and obtain the final user disengagement detection model. ## conclusion the ultimate chatbot evaluation metric should be user-centric, as chatbots are there to provide humans with enjoyable experiences. previously detecting user disengagement typically requires annotating many dialog samples for each individual dataset. we propose a two-stage pipeline her-ald to automatically label and denoise training data and, at the same time, build a user disengagement detector. our experiment shows that her-ald significantly reduces the annotation cost of a new corpus. herald's disengagement detection results highly correlate with expert judgments on user disengagement in both datasets (86.17% bacc in gunrock movie, 86.22% in convai2).