Comparing Pre-Trained Embeddings and Domain-Independent Features for Regression-Based Evaluation of Task-Oriented Dialogue Systems (2024)

Abstract: We use Gaussian Process Regression to predict different types of ratings provided by users after interacting with various task-oriented dialogue systems. We compare the performance of domain-independent dialogue features (e.g., duration, number of filled slots, number of confirmed slots, word error rate) with pre-trained dialogue embeddings. These pre-trained dialogue embeddings are computed by averaging over sentence embeddings in a dialogue. Sentence embeddings are created using various models based on sentence transformers (appearing on the Hugging Face Massive Text Embedding Benchmark leaderboard) or by averaging over BERT word embeddings (varying the BERT layers used). We also compare pre-trained embeddings extracted from human transcriptions with pre-trained embeddings extracted from speech recognition outputs, to determine the robustness of these models to errors. Our results show that overall, for most types of user satisfaction ratings and advanced/recent (or sometimes less advanced/recent) pre-trained embedding models, using only pre-trained embeddings outperforms using only domain-independent features. However, this pattern varies depending on the type of rating and the embedding model used. Also, pre-trained embeddings are found to be robust to speech recognition errors, more advanced/recent embedding models do not always perform better than less advanced/recent ones, and larger models do not necessarily outperform smaller ones. The best prediction performance is achieved by combining pre-trained embeddings with domain-independent features.

## introduction the quality of a human-machine dialogue interaction can be influenced by various factors, such as the domain/genre of dialogue, the dialogue system capabilities, and the user expertise and expectations. this makes it very difficult to define what a successful dialogue should look like, and evaluate system performance and predict user satisfaction. thus, despite many years of research, dialogue evaluation still remains an unsolved problem. in this paper, our focus is on task-oriented dialogue, and specifically on predicting user satisfaction after their interaction with the dialogue system. we use the communicator corpus @xcite containing the logs of user interactions with 8 spoken dialogue systems. the user's task is to book a flight and in some cases also make hotel or car-rental arrangements. each dialogue log is accompanied by user ratings after their interaction with the system. an example dialogue excerpt is shown in figure 1 in the appendix. the original communicator corpus contains system and user utterances (both human transcriptions and speech recognition outputs), timing information, and speech act and task annotations for the system's side of the conversation. an extended version of this corpus was developed by @xcite via automatic annotation. @xcite added speech act and task annotations for the user's side of the conversation, and dialogue context annotations, e.g., filled slots, filled slots values, grounded slots, speech acts history. in this paper, we use gaussian process regression for predicting user satisfaction ratings, because in our recent work @xcite it was shown to perform better than other regression methods, for this task and corpus. in our previous work @xcite , we considered only domainindependent features (e.g., duration, number of filled slots, number of confirmed slots, word error rate). these features were domain-independent because they were just based on counts, and no lexical, semantic, or specific to the task information was used. here, in addition to these domainindependent features, we also use pre-trained dialogue embeddings extracted from system and user utterances. our pre-trained dialogue embeddings are computed by averaging over sentence embeddings for each dialogue. sentence embeddings are created using various models based on sentence transformers @xcite (appearing on the hugging face massive text embedding benchmark leaderboard), or by averaging over bert word embeddings @xcite (varying the bert layers used). by definition, these embeddings are domaindependent because they encode lexical and semantic information about the domain. also, we compare pre-trained embeddings extracted from human transcriptions versus pre-trained embeddings extracted from automatic speech recognition (asr) outputs, to determine the robustness of these models to errors, which is an understudied research question @xcite . we investigate what level of performance can be achieved just by relying on the words of the system and user utterances from which we compute pre-trained dialogue embeddings, whether using only embeddings outperforms using only domain-independent features, and whether combining embeddings and domain-independent features can result in performance gains. we also examine the impact on performance of different feature combinations. to our knowledge, our work is one of a few studies (if not the first) to compare such a large variety of pre-trained embeddings (including the most recent embedding models by openai) under the same conditions, and the first study to do so for predicting user ratings in task-oriented dialogue. this is also the first work to compare all these different types of pre-trained embeddings with various domain-independent features for user ratings' prediction in task-oriented dialogue. last, but not least, this is one of a very limited number of studies comparing the performance of pre-trained embeddings on human transcriptions versus asr outputs, and the first study to do so for user ratings' prediction. ## related work despite many years of research, dialogue evaluation still remains an unsolved problem @xcite @xcite . for task-oriented dialogue there are subjective evaluation metrics, such as user satisfaction, computed using information from surveys @xcite , and objective metrics, such as task completion and dialogue length, com-puted using information from interaction logs. paradise @xcite is the most well-known framework for automatic evaluation of task-oriented dialogue. the goal of paradise is to optimize user satisfaction (or another desired quality) by formulating it as a linear combination of various factors, such as task success and dialogue cost (e.g., dialogue length, asr errors). weights calculated via linear regression determine the contribution of each factor. paradise can be used to predict user satisfaction at the end of the dialogue, but can also be applied to any point in the dialogue prior to completion. generally it is useful to be able to evaluate on the fly how the dialogue is unfolding, so that appropriate measures can be taken (e.g., transfer to a human operator), if a dialogue is problematic. based on this idea, much work has been done on estimating user satisfaction at the system-user exchange level rather than rating the whole dialogue @xcite @xcite . for chatbots and other non-task-oriented dialogue systems it is not clear what success means, and it is common to use subjective evaluations of system responses (e.g., coherence, engagingness) given some context, or use word-overlap similarity metrics (e.g., bleu, rouge) even though such metrics do not correlate well with human judgments of dialogue quality @xcite . recently, new evaluation metrics have been proposed for open-domain dialogue leveraging pre-trained language models such as bert and dialogpt @xcite , and commonsense knowledge bases @xcite . in this paper, we focus on predicting user satisfaction ratings for the whole dialogue. we use gaussian process regression (gpr) for predicting user satisfaction ratings, because in our recent work @xcite it was shown to perform better than other regression methods, for this task and corpus. in our previous work @xcite , we only used domain-independent features, but here we also use pre-trained dialogue embeddings extracted from system and user utterances. linear regression has been used before for dialogue evaluation @xcite @xcite @xcite . also, support vector regression has been used before for dialogue evaluation @xcite . we use gpr for our experiments because modern regression methods are a natural evolution of the paradise framework. furthermore, we do not have many data points for data-hungry methods such as neural networks. as we will see in section 3, we only have 500 data points in the training data and 506 data points in the test data. ## data and features we use the communicator corpus @xcite because it has been used before for this task, but also because it is one of a few taskoriented dialogue corpora that include user ratings. other popular corpora, such as multiwoz @xcite , do not include user ratings or asr outputs. the original communicator corpus contains system and user utterances (both human transcriptions and asr outputs), timing information, and speech act and task annotations for the system's side of the conversation based on the date scheme @xcite ). an extended version of this corpus was developed by @xcite via automatic annotation. based on the asr outputs, speech act and task annotations for the user's side of the conversation were added, as well as dialogue context annotations, e.g., filled slots, filled slots values. basically these extended annotations are the kind of information one would get by deploying a dialogue system, but because the original corpus did not include such information, @xcite reconstructed it. @xcite verified the validity and reliability of these automatic annotations by evaluating them with respect to the task completion metrics of the original corpus and in comparison to manually annotated data. the utility of these extended annotations has been demonstrated by their use by various researchers for different purposes, such as learning dialogue policies @xcite @xcite and building simulated users @xcite @xcite . in the appendix, figure 1 shows an example dialogue excerpt including speech act and task annotations, and figure 2 depicts an example dialogue state. these extended dialogue context annotations are divided into two broad categories: logs of the current status of the slots ('filledslotsstatus', 'filled-slotsvaluesstatus', 'groundedslotsstatus'), and logs containing information about how the status of the slots has changed over time through the dialogue ('filledslotshist', 'filledslotsvalueshist', 'groundedslotshist'). the former inform us about the current status of the slots, and may only contain one instance per slot. the latter provide information about the order in which slots have been filled or confirmed, and may contain several instances of the same slot. the annotations also include the history of speech acts and tasks. for our experiments we use the 2001 collection, which consists of 1,683 dialogues between human users and 8 dialogue systems. these systems vary in their dialogue policies, e.g., some of them request multiple pieces of information at the same time, others request explicit confirmation, others request implicit confirmation, etc. overall there are 78,718 turns (39,419 system turns and 39,299 user turns). similarly to @xcite , for our experiments we only used dialogues for which all user ratings were available: att (157 dialogues), bbn (137 dialogues), cmu (69 dialogues), colorado (157 dialogues), ibm (77 dialogues), lucent (140 dialogues), mit (166 dialogues), and sri (103 dialogues). the first half of the dialogues from each system is used for training (500 dialogues in total) and the rest for testing (506 dialogues in total). so our task is to predict the following user satisfaction ratings on a likert scale (1-5, higher is better): ease of the tasks the user had to accomplish ('task-ease'), whether it was easy or not to understand the system ('system-comprehend-ease'), the user's expertise ('user-expertise'), whether the system behaved as expected ('system-behaved-as-expected'), and if the user would use the system again in the future ('system-future-use'). we use the same domain-independent features as georgila (2022), with the addition of the number of times the user requested a 'start-over'. our 17 domainindependent features are divided into 4 categories: all these features are automatically extracted from the data. feature values are replaced with z-scores by subtracting from each feature value the mean for that feature and then dividing by the standard deviation for that feature. for each feature, the mean and standard deviation are calculated on the training data. we use 4 variations of these feature combinations: 'orig-man' (original corpus with features from manual annotations such as human transcriptions of speech plus fully automatic annotations), 'orig-auto' (original corpus with fully automatic annotations), 'ext-man' (extended corpus with features from manual annotations plus fully automatic annotations), and 'ext-auto' (extended corpus with fully automatic annotations). so 'ext-man' is a super set of 'orig-man', and 'ext-auto' is a super set of 'orig-auto', because the extended corpus contains all the annotations of the original corpus plus new annotations (note that, as mentioned above, these new annotations are automatically generated). also, 'orig-man' and 'ext-man' include both manual and automatic annotations, whereas 'orig-auto' and 'ext-auto' include only automatic annotations. for duration, the number of user speech acts is only used in 'ext-man' and 'ext-auto', because (as discussed above) the original corpus did not include annotations of the user's side of the conversation. likewise, slots-related features are only part of the extended corpus ('ext-man' and 'extauto'). information about wer is only part of the manual annotations because it can be computed only when human transcriptions are available. for clarity, we also compute pre-trained dialogue embeddings by averaging over sentence embeddings for each dialogue. sentence embeddings are created using various models based on sentence transformers (appearing on the hugging face mteb leaderboard), or by averaging over bert word embeddings (varying the bert layers used). we do not calculate z-scores for the embeddings. we use the following types of embeddings from hugging face and openai, and in parentheses we can see the sizes of the vectors they produce: the latest models of openai have a new feature that allows selecting the size of the generated vector. according to openai, this compressed vector retains its concept-representing properties. for 'textembedding-3-small' we experimented with 3 vector sizes (50, 256, 1536) and for 'text-embedding-3-large' with 3 vector sizes @xcite @xcite . because we only have 500 data points in the training data and 506 data points in the test data, and large vector sizes, we also applied principal component analysis (pca) for dimensionality reduction, with "whitening" to ensure that the resulting features are less correlated with each other. @xcite and @xcite have found that "whitening" can enhance the isotropy of sentence embeddings, with the additional advantage of reducing their dimensionality. we generated results with different numbers of pca components, and we show results with a value of 50 which performed well for all models (better e.g., than 75 or 100). of course, when we generated vectors of size 50 from openai, we did not apply pca. note that we apply pca only to the embedding vectors (the domain-independent features are not affected by pca). ## experiments and results in our previous work @xcite , we compared several state-of-the-art regression methods, and showed that gpr with an exponential kernel or a rational quadratic kernel performed the best. thus, here we use gpr with an exponential kernel. also, by performing more experiments, we verified again that gpr outperforms other regression methods, and that using an exponential kernel produces competitive results for different types of embeddings. for all gpr experiments we vary the length scale, and we report results for length scale equal to 1 (higher length scale values indicate smoother learned functions). varying the length scale did not produce significant differences. gpr is considered as the state-of-the-art for regression, and has been used before in the nlp community for machine translation quality estimation @xcite and emotion prediction @xcite . for all our experiments we use the gpy library 1 , and gpr is applied after pca. to evaluate our models, for each of the 5 ratings, we calculate the root mean square error (rmse). rmse measures the average error be-1 https://gpy.readthedocs.io/en/deploy/ tween the model predictions and the ground truth (the ratings in the test data). its value varies from 0 to 4, given that user ratings are on a scale from 1 to 5. lower rmse values are better. ## conclusions and discussion we used gpr for predicting user satisfaction ratings. we used both domain-independent features and pre-trained dialogue embeddings extracted from system and user utterances. our pre-trained dialogue embeddings were computed by averaging over sentence embeddings for each dialogue. sentence embeddings were created using various models based on sentence transformers (appearing on the hugging face mteb leaderboard) or by averaging over bert word embeddings (varying the bert layers used). our results showed that overall, for most types of user satisfaction ratings and advanced/recent pre-trained embedding models, using only pretrained dialogue embeddings outperforms using only domain-independent features. this is very interesting, because it shows that we can do quite well relying only on information from words and system and user utterances, without any additional features. combining embeddings and domainindependent features performed the best. this is also very interesting and could potentially revive interest in using domain-independent features. although overall extracting domain-independent features from the extended annotations helped, performance was similar to using domain-independent features from the original annotations. interestingly, some simpler models (e.g., 'bert-l1') performed better than more complex and more recent models. also, larger models did not necessarily outperform smaller ones. because differences between corresponding 'man' and 'auto' models were small, we conclude that pre-trained embeddings are quite robust to asr errors for our task. overall, rmse values ranged roughly from 1.1 to 1.4 depending on the model and feature combination. our overall contributions are as follows: throughout our experiments, to construct dialogue embeddings we used averaging @xcite , but the problem with averaging is that it can result in loss of important conversational information @xcite . for example, not all parts of a dialogue are of equal importance, and by trying to encode everything we may end up compressing too much information from parts that really matter. very little work has been done on constructing dialogue embeddings using techniques different from averaging. a notable recent attempt to construct dialogue embeddings is dial2vec @xcite . dial2vec uses self-guided contrastive learning (leveraging both positive and negative examples) and considers a dialogue as an information exchange process between two interlocutors. it learns embeddings for both interlocutors with the help of each other, and then the dialogue embedding is obtained by an aggregation of embeddings of the interlocutors. dial2vec was used to construct dialogue embeddings for the tasks of domain categorization, semantic relatedness, and dialogue retrieval. based on the idea of dial2vec, an interesting future research direction would be to learn dialogue embeddings for the interlocutors (system and user) participating in successful versus unsuccessful dialogues, and by aggregating the embeddings of the interlocutors learn in turn dialogue embeddings for successful versus unsuccessful dialogues.