Towards Continuous Estimation of Dissatisfaction in Spoken Dialog (2021)

Abstract: We collected a corpus of human-human task-oriented dialogs rich in dissatisfaction and built a model that used prosodic features to predict when the user was likely dissatisfied. For utterances this attained a F.25 score of 0.62,against a baseline of 0.39. Based on qualitative observations and failure analysis, we discuss likely ways to improve this result to make it have practical utility.

## motivation accurate models of dialog quality are needed for many purposes, including closed-loop improvement of dialog systems @xcite @xcite @xcite @xcite . spoken dialog includes much information that can be used to predict quality judgments, and successful prediction has been shown for many genres, and in particular in call-center analytics @xcite @xcite @xcite @xcite @xcite @xcite @xcite . while most work on dialog quality has focused on the quality of entire interactions, finer-grained quality estimates are more useful for many purposes. casual observation suggests that in conversation people are often not shy about indicating, moment by moment, how they feel about things, both in terms of making progress towards their goal and in terms of how happy they are with the contributions and behavior of their interlocutor. to date, however, predictive modeling of quality at the level of turns has been rarely attempted, and has focused mostly on interaction quality and conver-sational proficiency, and in only a few dialog genres, both for human-machine and human-human dialogs @xcite @xcite @xcite @xcite @xcite . in this work we attempt turn-level quality estimation in human-human dialogs in a new genre: short calls to an unknown merchant to make an appointment or arrange a simple transaction. this paper presents the first publicly available corpus of (mock) customer-service calls, describes observations on how dissatisfaction occurs in conversations gone wrong, discusses prosodic and turntaking indications, presents a simple model giving modest performance on the tasks of detecting dissatisfaction moment by moment and at the utterance level, and discusses what more is needed. ## scenario and data among the many possible contexts in which to study aspects dialog quality, we chose to examine what happens when a person is trying to get something done and expects that it can be easily accomplished, but finds that it is not possible. we would have liked to study real commercial dialogs, where customers or users often have a goal that the agent or system may be unable or unwilling to satisfy, but there appear to be no datasets in this genre available for study. we therefore did our own data collection, with the details chosen to align with the goals of our sponsor, google. in some markets, google enables users to find merchants by voice search, leading to the presentation of phone numbers to call. this is especially useful for illiterate users. unfortunately, the ecosystem includes bad actors, who purchase adwords to entice callers, but then do not offer the expected service, offer it at an excessive price, or otherwise disappoint or trick callers. google would like better ways to flag such abusive merchants, ideally from automatic analysis of behavior in the call itself. unlike most conversations addressed in call analytics, there is no large reference corpus of good behavior in the domain, these callers have no previous relationship with the business, and, conveniently for our purposes, many confounds and complexities are reduced @xcite and the causes of any negative feelings will be largely dialog-internal. we accordingly collected a new corpus of telephone calls. each participant was given rough instructions, for example, in the customer role, to call to arrange to get a flat tire patched for no more than @xmath060 tire care package or accepted an additional @xmath15 and they were incentivized with a $1 bonus for each call where they successfully made arrangements with a merchant within budget, but were told that this would not always be possible. the merchant-role participants were six trained confederates. the calls were in english, with the confederates mostly native speakers of american english and the customer actors, it turned out, mostly non-native speakers from european countries, with poland and portugal overrepresented. in total we collected 191 calls. most of the calls were, in our judgment, quite realistic, with each side trying hard to achieve their assigned goals. indeed, some callers were able to get our confederates to deviate from instructions and agree to provide the requested service at the requested price; conversely, the confederates were sometimes able to wear down callers into agreeing to a price that violated their instructions. excluding the latter category and other special cases, we had 52 "doomed" (bad-actor) calls and 62 fully satisfactory calls. calls were recorded in stereo. they were typically 1 to 4 minutes in length. full documentation is available @xcite , and the corpus itself is freely downloadable @xcite . ## subjective observations and annotation of dissatisfaction callers in the doomed-to-fail dialogs reacted diversely. often they showed surprise at the first indication that the merchant was not going to behave according to expectation. often they attempted repair, usually by restating their goals, generally more assertively than the first time. often they expressed annoyance or other negative assessment, although always politely, never with raw emotion. occasionally callers engaged in other behaviors, including negotiating, pleading, and even displaying anger. across these specific behaviors, there was often an underlying feeling of growing dissatisfaction. doomed conversations also generally lasted longer @xcite and lacked the warm and appreciative/grateful closings that were common in the control dialogs. while most call analytics systems rely on speech recognition @xcite , this makes sense mostly for high quality audio, for languages where good speech recognizers exist, and for focusing on how to improve agents' behavior; none of these are the case in our sponsor's scenario. in particular, the bad actors strive to be indistinguishable from good actors, so we chose to focus on acoustic-prosodic features of the caller. there are two lines of work that we might have built on: first, work identifying the prosodic correlates of specific dialog acts, including some relevant here @xcite , but the variety of behaviors across speakers and calls would make it difficult to leverage this work; and second work on the prosodic correlates of emotion, but the behaviors observed here were more social and linguistic than visceral or paralinguistic, so we again decided not to attempt to leverage such findings. instead, we chose to approach the problem as one of modeling undifferentiated dissatisfaction. we hoped that this would be generally, if weakly, detectable, using the same features across all contexts. although dissatisfaction was often subtle to the point that we were unsure exactly when it was present, prosodic models are often able to exploit indications below conscious awareness, and we hoped that would also be the case here. focusing on general dissatisfaction also aligns with our broader goal of better automatic quality judgments. we accordingly labeled each utterance with d for those with indications of dissatisfaction, defined broadly, to include disappointment, annoyance, sadness, disengagement and so on, n for non-dissatisfied or "neutral" utterances, and ? for those that were inaudible or otherwise impossible to classify @xcite . initially 18 dialogs were annotated, each by four people, and, for frames within utterance spans labeled by all four, the fleiss kappa was 0.57. the weak agreement, illustrated in the appendix, seemed to be mostly due to varying preferences for classifying borderline utterances as d versus ? or n, rather than substantive differences in perception. accordingly the rest of the corpus was labeled by only one annotator, and the results below are reported for these annotations. ## experiment set-up we set ourselves two tasks: 1) utterance-level prediction: distinguishing dissatisfied utterances from neutral utterances, and 2) frame-level prediction: distinguishing moments within dissatisfied utterances from moments within non-dissatisfied utterances. for both tasks, the input was only those frames (or utterances) which had been given a d or n utterance; silent regions and ambiguous regions were thus excluded. for the utterance-level and frame-level models, there are many more negative samples, as there are fewer dissatisfied dialogs and even in those many utterances are not dissatisfied. there are many more neutral utterances, since not all utterances in the dissatisfied dialogs are dissatisfied. the number of n and d utterances in the training, dev, and test sets are 46 and 24, 52 and 23, and 256 and 82. the average labeled utterance being about 2 seconds long, for the test set the frame counts were 54543 neutral and 20893 disappointed. as our primary goal is detecting dissatisfaction, the baseline is to always predict dissatisfaction, and high precision is our primary goal. however recall also has some importance, so we also report f .25 results. ## initial feature set most research in this area uses utterance-aligned features, but we wanted to avoid the travails of defining or performing segmentation, so we simply computed prosodic features everywhere. specifically, we compute features for timepoints sampled every 10 milliseconds (a 10 ms stride), using features that span about 3 seconds on either side of the point being classified. much research on paralinguistic prosody assumes that affective states directly affect the prosody in stable ways for a second or more, and accordingly uses global averages or simple functionals, but work on the prosodic correlates of stance and dialog acts suggests that here we need the ability to represent temporal configurations of prosodic features @xcite . accordingly, we used a feature set that includes time-offset features which together tile a local span. specifically we based this on a feature inventory included in the midlevel prosodic features toolkit @xcite , mono.fss. this includes measures of intensity, of pitch height (high or low), of pitch range (narrow or wide), of speaking rate (using energy flux as a proxy), and of creakiness, as this set worked well for detecting various stances @xcite . to this we added features for the cepstral peak prominence (smoothed) (cpps) across two windows, based on our observation that breathy voice was saliently present in many dissatisfied utterances. cpps is an effective measure for breathiness in clinical applications @xcite , although seldom yet used in studies of dialog. ## analysis to understand how each feature was contributing, we looked at correlations and also histograms, since the relationships were seldom simply linear. dissatisfied utterances tended to include more silent or very quiet frames, with neutral utterances richer in relatively loud frames. a clearer picture emerges when we examine the coefficients in the model for the features at specific temporal offsets, as seen in figure 1 . (the actual values are available at the companion website: http://www.cs.utep.edu/nigel/disappointment/. ) low intensity features over about 3 seconds around the frame being predicted had positive weights, with the more distant intensity features having negative weights; thus intensity that is low relative to the local context is the informative pattern. both the wide pitch and narrow pitch features were indicative of disappointment, marking departures from a normal moderate pitch range. this fact aligns with the literature about the prosodic constructions used in complaining @xcite . creaky voice was also indicative of disappointment, which may relate to its reported role in marking disengagement @xcite . so did a couple hundred milliseconds of high cpps, contrary to expectation. low creakiness and high volume also correlated with a lack of dissatisfaction, which may reflect a general tendency for people when pleased to use clear and "pleasant" voices, with strong periodicity and harmonicity. in general the prosodic indications are not local to single syllables or words, but are present distributed across wider spans. seeking further understanding, we listened to a sampling of successes. although our simplistic model could only learn one pattern, that pattern matched diverse ways of expressing dissatisfaction. this included a complaint, i think this is still too much, with narrow pitch on the first words and stress with high cpps on the word still, and a quiet, annoyed no thank you (audio for these examples are at http://www.cs.utep.edu/nigel/disappointment/ ). inversely, an example of a successful nondissatisfied prediction was for a warm, fairly loud, slightly harmonic, moderately high-pitched, closing thank you. we also listened to a sampling of failures. misses included many frames from one dialog where excessive record gain had caused constant clipping, and some frames near a loud beep in the background. our feature computations are not robust to such noise. we also examined false alarms. many were in frames near regions of silence, such as at the start of an utterances or in the vicinity of a disfluent pause, even for pauses that, to our ears, did not seem perplexed or emphatic. some false alarms occurred during the customer's explanation of their need, for example in the word flat in my front left tire that is flat because of a nail. while these did not express dissatisfaction with the merchant's behaviors, and so were not annotated as dissatisfaction, they certainly did express a negative assessment. while this could suggest tweaking the annotation guidelines, the more important lesson is that accurately predicting dissatisfaction requires modeling the stage of the dialog, not just the local context. this analysis suggested that our model has explanatory value and validity, and thus may be likely to generalize well. ## revised feature set and models based on the above analysis, we augmented the prosodic feature set with a time-into-dialog feature, for a total of 91 features. (we also did some small experiments with alternative feature sets based on opensmile's egemaps configuration @xcite , but obtained no benefit.) we continued to use the simple linear regression model for our basic task, of predicting dissatisfaction at the framelevel. (small experiments with logistic regression and k-nearest neighbors provided no benefit.) for utterance-level predictions we simply averaged the predictions for every frame within the utterance. ## results tables 1 and 2 show the performance of our framelevel and utterance-level models, on the test data. while the choice of threshold ultimately depends on the use scenario, here for each model we report performance at the value which maximizes f .25 . for the frame-level detections, the performance was modest. as an indication of the scope for improvement, our model's agreement with the annotator, in terms of cohen's kappa, was .32, far below that of our secondary human annotators, whose agreements ranged from .57 to .71. nevertheless, the frame-level model was good enough to support reasonable performance for the utterance-level discriminations. future work should address the weaknesses noted above, perhaps in part by adding features to capture cross-participant behaviors @xcite and timings. better models are another priority topic. to consider the stage of the dialog and other factors, models that represent wider context should be tried @xcite . to support such advances, code for our existing, simple models is freely available @xcite . we also should try these methods on dialogs from different genres and exhibiting quality issues of other kinds. we also need to do ablation studies to better identify the sources of performance and to evaluate our model in comparison to others. such comparisons have been rare in this research area, due to a lack of shared datasets, but our new corpus will enable other researchers to report directly comparable results. finally, since we see some level of performance across speakers with different native languages, we should investigate the possibility of universal, language-independent detection of dissatisfaction.