Understanding and predicting user dissatisfaction in a neural generative chatbot (2021)

Abstract: Neural generative dialogue agents have shown an increasing ability to hold short chitchat conversations, when evaluated by crowdworkers in controlled settings. However, their performance in real-life deployment – talking to intrinsically-motivated users in noisy environments – is less well-explored. In this paper, we perform a detailed case study of a neural generative model deployed as part of Chirpy Cardinal, an Alexa Prize socialbot. We find that unclear user utterances are a major source of generative errors such as ignoring, hallucination, unclearness and repetition. However, even in unambiguous contexts the model frequently makes reasoning errors. Though users express dissatisfaction in correlation with these errors, certain dissatisfaction types (such as offensiveness and privacy objections) depend on additional factors – such as the user’s personal attitudes, and prior unaddressed dissatisfaction in the conversation. Finally, we show that dissatisfied user utterances can be used as a semi-supervised learning signal to improve the dialogue system. We train a model to predict next-turn dissatisfaction, and show through human evaluation that as a ranking function, it selects higher-quality neural-generated utterances.

## introduction neural generative dialogue agents have become sufficiently mature to make contact with real users through programs such as the alexa prize @xcite . though these models have known problems with factual correctness @xcite , using dialogue history @xcite , and bias @xcite , they have nevertheless produced good written conversations when evaluated by crowdworkers or volunteers in carefullycontrolled scenarios @xcite @xcite . how are you doing today? ok but my cat threw up on the couch oh no! did you get a new cat? what oh no! did you clean it up? yeah @xmath0, we train a model to predict dissatisfaction before it occurs. the model is used to reduce the likelihood of poor-quality bot utterances. by contrast, real-life settings such as the alexa prize, in which intrinsically-motivated users speak to open-domain chatbots in noisy environments, offer unique challenges. unlike crowdworkers, users have their own expectations that may differ from those of the chatbot or its designers, and they may express dissatisfaction if those expectations are not met. it is not yet well-understood how neural generative models perform in these settings, nor the types and causes of dissatisfaction they encounter. by studying a neural generative model deployed in chirpy cardinal, an alexa prize chatbot, we seek to provide the first in-depth analysis of a neural generative model in large-scale real-life deployment, focusing on understanding the root causes of user dissatisfaction. real-life settings such as the alexa prize also offer unique opportunities. dialogue systems can be difficult to build due to a lack of sufficient publiclyavailable data in the appropriate domain; meanwhile synthetic crowdsourced dialogue datasets can contain unnatural patterns or behaviors that are then replicated by a model trained on them. we use our chatbot's real-life conversations as a source of natural in-domain data. in particular, we train a model that can predict authentic user dissatisfaction before it occurs, thus helping us to avoid it. ## chirpy cardinal chirpy cardinal, aka chirpy @xcite 1 is an open-domain socialbot developed for the third alexa prize @xcite . during the competition (december 2019 to june 2020), us alexa customers could say alexa, let's chat to connect to a random socialbot. users would chat to the bot in english for as long as desired, then provide a 1-5 rating. at the end of the competition, chirpy had an average rating of 3.6/5.0 and a median conversation duration of 2 minutes 16 seconds. like most alexa prize bots @xcite , chirpy is modular in design, combining a mix of rule-based, retrieval-based, knowledge-based and neural generative components specializing in different topics. however, this paper focuses solely on the neural chat module, which uses neural gen-1 https://stanfordnlp.github.io/chirpycardinal ## detecting user dissatisfaction we consider a user utterance to express dissatisfaction if it meets any of the definitions in regex classifiers in chirpy, we manually designed regex classifiers to identify each of the dissatisfaction types in human-labelled set to help us develop higher recall dissatisfaction classifiers, one expert annotator foot_1 gathered a set of 3240 user utterances. for each utterance u and dissatisfaction type d, they provided a label humlabel d (u) ∈ {0, 1}. the utterances are drawn from several sources, including most common utterances, utterances drawn from 1-rated conversations, and utterances which scored highly for the clarifying, closing and complaint dialogue acts in chirpy's dialogue act classifier (paranjape et al., 2020). 7 nearest neighbors classifiers to represent a user utterance u, we take a dialogpt-large model @xcite that was finetuned on chirpy conversations (appendix c), input u, and average the top-layer hidden states across the sequence. using this embedding for each utterance, we build a faiss @xcite index of the humanlabelled set. to compute a new utterance u's score problem definition % in ctrl set % when no user prob. user already dissatisfied the user has already expressed dissatisfaction in c. 12.0% 0.0% user unclear the main gist of the user's latest utterance in c is unclear or obscured. 22.0% 0.0% bot repetitive the primary content of b was already said/asked by the bot earlier in c. 6.0% 4.3% bot redundant question b is asking for information that the user has already provided earlier in c. 12.0% 15.9% bot unclear it's hard to find an interpretation of b that makes sense. 12.0% 7.2% bot hallucination b refers to something that hasn't been mentioned, acts like the user said something they didn't, confuses self with user, or seems to be responding to own utterance. 17.0% 10.1% bot ignore b ignores or fails to acknowledge the user's latest utterance, doesn't answer a question, doesn't adequately respond to a request, or switches to an unrelated topic. 20.0% 14.5% bot logical error b is generally on-topic, but makes an assumption or association that's incorrect, unfounded or strange. 15.0% 17.4% bot insulting b says or implies something insulting about the user, or about others in a way that might offend the user. 1.0% 1.4% any bot error true iff any of the above bot errors are true. 53.0% 46.4% for dissatisfaction type d (including any), we find its k nearest neighbors u 1 , ..., u k in the humanlabelled set (w.r.t. cosine distance), then compute p knn (d|u) ∈ [0, 1] as follows: that is, we first check if u has a human label or is a positive match for d's regex; if not we compute the proportion of u's neighbors that are labelled d. for each d, we evaluate the k-nn classifier on the human-labelled set for @xmath5, . . . , 30 via leaveone-out cross-validation. ## neuralchatturns dataset over the period that chirpy was online, we collect examples of the form (c, b, u) where b is a purely neural-generated bot utterance, c is the neural chat context that preceded b, and u is the user response to b. the neuralchatturns dataset has 393,841 examples in total, which we split into 315,072 train, 39,384 validation, and 39,385 test. due to user privacy constraints, we are not permitted to publicly release the neuralchatturns dataset. ## what causes user dissatisfaction? to understand dissatisfaction, we annotate errors in the generative model's conversations. ## learning to predict user dissatisfaction in this section we build a system to predict, and thus reduce the likelihood of, dissatisfaction. ## ranking neural generations to minimize dissatisfaction in this section we use the predictor to select betterquality bot utterances. ## related work previous work has used a variety of user signals to improve dialogue agents. when learning from a variable-quality human-human dataset such as reddit, @xcite showed that engagement measures like upvotes and replies are more effective than perplexity to train a ranking model. for oneon-one empathetic conversations like ours, @xcite trained a neural generative model with reinforcement learning to improve next-turn user sentiment (as simulated by a user response model, rather than human responses). though we considered taking a sentiment-based approach in chirpy, we found that user sentiment doesn't always align with good user experience: first, expressing negative emotions is sometimes unavoidable, and second, sentiment classifiers tend not to distinguish between sentiment about the conversation and sentiment about other issues. we find next-turn user dissatisfaction to be a comparatively more precise, well-aligned learning signal. dialogue systems that learn from their own interactions with humans are relatively rare. @xcite also use user satisfaction to identify high-quality bot utterances; these become additional training examples for the neural generative model. however, this work uses paid crowdworkers; research involving intrinsically-motivated, unpaid users is rarer still. in symmetric settings such as the role-playing game light @xcite , the user utterances themselves can be used to retrain the dialogue agent. in the asymmetric alexa prize setting, @xcite show that conversation-level metrics like rating and length can also be used to train an effective ranker. ## limitations our findings on user behavior are particular to the demographics of the us alexa customers who spoke to chirpy in 2019-2020. while users in other locations or time periods may differ, our analysis gives a valuable snapshot of the current attitudes and expectations of us users interacting with a voice-based socialbot or virtual assistant. second, our results are dependent on the alexa prize conversational context and the technical details of our generative model. in particular, due to latency and cost constraints, our gpt-2-medium generative model is orders of magnitude smaller than the current largest generative models, and trained on a fraction of the data @xcite . given that very large models have shown generative abilities that are absent at smaller scale, it is likely that if we had built our dialogue agent with such a model, its errors and interactions with users would have been very different. nonetheless, we believe our analysis gives useful insight into the performance of neural generative models of more accessible scale, in particular highlighting issues occurring in real-life scenarios that might not occur in crowdsourced conversations. ## conclusion in this study of an open-domain neural generative dialogue agent in real-life deployment, we found that poor-quality bot turns are common. the noisy environment -in which user utterances are often unclear -plays a large part in the bot's more basic errors (repetition, ignoring, and nonsensical utterances). however, even in clear examples where the generative model could succeed, it still makes many unforced errors; these are more likely to involve faults in reasoning or social abilities. this highlights the importance of improving neural generative dialogue models' state-tracking, commonsense abilities and use of conversational history. despite the frequency of errors, users are generally polite; most don't express overt dissatisfaction even after an error. however, unaddressed dissatisfaction escalates: it makes users more critical, offensive, and likely to quit when encountering an error, and more offensive even if there are no further errors. we find that dissatisfaction correlates with bot errors, however, it can arise unpredictably for other reasons -e.g., as a result of privacy boundaries, which are variable and personal to each user. dissatisfaction is relatively easy to automatically detect, and thus feasible as a scalable semisupervised learning signal that could be used for online learning. however, it is difficult to predict; this makes it a challenging learning signal. indeed, we find that our predictor conditions more strongly on easier-to-recognize factors such as prior user dissatisfaction, than on harder-to-recognize factors such as bot errors. nonetheless, we find that when used as a ranking function to choose between alternative bot utterances, the predictor chooses better than random selection.