Similarity Scoring for Dialogue Behaviour Comparison (2020)

Abstract: The differences in decision making between behavioural models of voice interfaces are hard to capture using existing measures for the absolute performance of such models. For instance, two models may have a similar task success rate, but very different ways of getting there. In this paper, we propose a general methodology to compute the similarity of two dialogue behaviour models and investigate different ways of computing scores on both the semantic and the textual level. Complementing absolute measures of performance, we test our scores on three different tasks and show the practical usability of the measures.

## introduction and related work interacting with technical systems through voice is prevalent in our every day lives and in the focus of industry and research alike. for evaluating the behaviour of voice interfaces, interaction-based or corpus-based methods have been employed, both aiming at producing absolute measures like dialogue success. while this is clearly an important aspect of dialogue behaviour evaluation, it remains superficial and does not touch on the actual difference of two behaviour models. the goal of this paper is to propose a method to quantify the similarity of two behaviour modelsthe learned or hand-crafted dialogue system decision-by means of a similarity score. the core idea is to use well-defined dialogue contextsmoments within a dialogue where the system needs to make a decision of how to respond-and compare the resulting system response of each behaviour model. we propose different similarity measures and demonstrate their usefulness in different scenarios. being able to compare behaviour models on a deeper level opens the door to a deeper understand-ing of the learned behaviour. it aims to answer questions like: these questions are of high relevance in cases where not only the average absolute performance is of interest but also the actual learned behaviour. on an application level, the answers to those question can help to decide which behaviour model to apply for a concrete live application, as they can support decision such as when to stop learning, or reveal the properties of different random seeds. from a more scientific point of view, the questions contribute to the overall problem of what we can learn about the interaction characteristics from the learned models. the core task of a voice interface, also called spoken dialogue systems (sds), is the decision of how to respond to a given user input and a dialogue context. this task is either modelled explicitly or implicitly. an explicit behaviour model usually comprises a distinct dialogue system module called dialogue policy taking in a dialogue state-a combined and dense representation of the current user input interpretation and the dialogue context-and producing an abstract system response. in a subsequent step, this abstract system response is then transferred into text by a natural language generator. an implicit behaviour model uses a neural network to learn a text response directly based on text input thus combining user input interpretation, dialogue context integration, and dialogue response selection in one model. absolute measures to evaluate the performance of these behaviour models through the interaction with real or simulated users are, for example, task success or dialogue length @xcite @xcite @xcite @xcite . other measures are user satisfaction @xcite @xcite @xcite or quality of interaction @xcite . all are often acquired through interaction-based studies 1 . others have employed corpus-based evaluation by comparing textual system responses with transcriptions of actual interaction as absolute evaluation criterion where the response in the corpus is treated as ground truth @xcite @xcite . text comparison metrics like bleu @xcite have been adopted from machine translation to evaluate how well the system response matches the one in the database, e.g., @xcite . this way of evaluating has been criticised widely as a system response that is different from the one in the data base can still be a valid system response simply leading to a different subsequent dialogue. furthermore, the bleu score evaluation hardly correlates with human judgements @xcite . dismissing text similarity measures as not useful for dialogue evaluation, however, is overhasty and shortsighted. while those measures may not help with absolute comparison of policies, they may be valuable to compare two policies with each other. in other words, they can help to reveal the similarity between two models without explicitly judging their performance. in this work, we propose a framework to com-1 for a good overview over absolute metrics including a taxonomy, please refer to @xcite . the remainder of this paper is structured as follows: in section 2, we introduce the general approach for quantifying the similarity of behaviour models. we then investigate the usability of several different ways of computing a similarity score in sec. 3, considering scores on the semantic and the textual level. in sec. 4 and 5, we describe our experimental setup, test our scores on three different tasks, and show their correlation confirming their practical usability. ## scoring framework to compare two dialogue behaviour models, this paper explores the usage of similarity measures instead of relying on absolute performance measures. the main idea is-in addition to knowing about the absolute performance-to learn about how similar or different two behaviour models are. for this, a defined set of dialogue contexts is applied to each behaviour model to generate corresponding system responses. these responses are then compared to learn about the overall similarity of the models. the general approach illustrated in figure 1 is as simple as effective: aside from finding suitable similarity measures, one of the key challenges is to find good dialogue contexts that may be used as basis for comparison. here, a dialogue context is a sub-dialogue either represented by a set of system utterances and user utterances (which is necessary, e.g., for end-toend dialogue generators) or directly by the dense representation of a dialogue state following the markovian idea (often only available in modular dialogue systems). the proposed framework relies on a well-defined set of dialogue contexts, avoiding the evaluation of unrealistic situations which would directly influence the similarity scores. in this work, the focus lies on modular dialogue systems where dialogue states are available to represent the dialogue context. thus, there are two natural options of finding a set of dialogue contexts: collecting dialogues with corresponding dialogue states from actual real dialogues noted as c col or generating a set of dialogue states noted as c gen . ## similarity scores a similarity score is computed for the comparison of two behavioural models π and π . depending on the nature of the behavioural model, for each context c @xmath0, each may produce an abstract system response actions a i , and an text response p i . each abstract system action a @xmath1, . . . , s i @xmath2, representing the communicative function like inform or request, and a set s i of j slot-value-pairs s @xmath3, v i 1 ), . . . , (s i j , v i j )} representing the concepts and their respective values foot_0 . to compute each similarity score, |c| action/text response pairs are compared using the following similarity score measures. total match rate the total match rate (tmr) is based on a binary score that regards two actions a, a as equal only if they completely match, i.e., δ a,@xmath4, else 0. the tmr is then defined by dialogue act match rate the dialogue act match rate (dmr) is based on a binary score comparing the actions a, a where both match if the corresponding dialogue acts are the same: δ act,@xmath5, else 0. the dmr is defined by concept error rate the concept error rate is a measure usually used for evaluating natural language understanding systems that translate text input to a semantic representation. the concept error rate then is computed by comparing the resulting semantic representation with a ground truth. instead of comparing a semantic representation a with a ground truth, it can also be used to compare it to another representation a produced by a different behaviour model. similar to the word error rate, it is based on the levensthein distance of two dialogue actions having one as hypothesis h and one as reference r: concept match rate as an alternative to the asymmetric cer, we propose the symmetric concept match rate. instead of basing it on an error comparing a hypothesis with a reference, it counts concepts γ that are present in both dialogue actions where m(a, a , γ) defines if a match occurred: m(a, a , γ) = 1, if γ ∈ s and γ ∈ s 0, otherwise . the concept match cm takes into account the dialogue acts and the unified set of concepts @xmath6: c m (a, a ) = δ act,act + ∈ s m(a, a , γ) (8) a concept match of two dialogue actions a and a is thus defined by cosine similarity and angular similarity the universal sentence encoder (use) @xcite is a generic sentence encoder which employs two measures for the computation of the distances between encoded sentences, namely cosine similarity and angular similarity: bleu the bleu score @xcite is a measure used for the evaluation of machine translation systems. it is based on an n-gram precision ϕ n , computed as the number of common n-grams between reference p and candidate phrase p (and vice versa) divided by the number of n-grams of the candidate phrase. the score of a corpus is the geometric mean of modified precision scores multiplied with a brevity penalty υ: f bert has been selected as a symmetric similarity score that also represents a reasonable balance between r bert and p bert . examples scores are shown in appendix a. ## application scenarios of similarity score evaluation we present three different scenarios addressing the following questions: when does the behaviour of a reinforcement learning policy converge? which effect do modifications of the random seeds have on the resulting learned policies? which effect do modifications of the reward models have on the resulting learned policies? ## correlation of scores to analyse how complementary the different scores are, all system behaviour pairs of all experiments have been used to compute correlation and mean squared error for each score pair. the results are shown in figure 6 . an interesting finding is that the cer and cmr have a very high correlation and a very low error. thus, both seem to capture the same similarities. in contrast to that, the dmr has a very low correlation with other scores and thus does provide additional information. bleu-4 also does not have a high correlation with other metrics but does also not provide a huge variety, as shown in the example in figure 3 . comparing semantic-based similarity scores with text-based similarity scores shows that cer and use-based cosine distance have a quite high correlation and a relatively small mean squared error. thus, the similarity of two systems that provide semantic output and the similarity of two other systems that only provide text output can be comparably quantified with the cer and the use-based cosine distance. the overall total match rate of the samples used for calculating the correlation and mean squared error is 63.3%. thus, the matches govern the correlation scores. computing the correlation only on the samples that do not match reveals slightly different numbers that still match the overall impression. the main difference is that the correlation between cer and cmr drops down to 0.466. ## conclusion this work proposes a first step towards a more detailed analysis of dialogue behaviour models by proposing a framework to compute similarity scores. a similarity score is meant to quantify how similar the decisions made by one dialogue behaviour model are compared to a second dialogue behaviour model. using a fixed set of dialogue contexts, each model is evaluated and the resulting system responses-as semantic representations and/or as text utterances-are captured and used for the similarity score. we proposed eight similarity scores and applied them to three different scenarios. by doing that, we were able to validate supposed certainties about reinforcement-based policy learning. we could observe that in the used set-ups, all policy models converged towards a fixed behaviour while still showing minor behavioural changes even after a very large number of training iterations. modifications of the random seeds, however, already result in a noticeable differences in the converged behaviour in the applied evaluation setup. the quantified differences are even similar in magnitude to a modification of the reward model, i.e., changing a random seed has a similar effect on the learned policy as switching from task success to interaction quality as the principal reward component. out of the eight proposed similarity scores, many seem to capture different aspects of similarity, so it remains to the application to decide which score is more useful. only text-based scores coming from the language translation field like bleu and bertscore seem not to be too useful. one reason for this might be the dependency of the absolute score on the prompt length: quantifying textual difference in this way does not necessarily capture the relevant semantic differences. moreover, the set of dialogue contexts has a high impact on the absolute score for all similarity metrics but not on the trend when comparing two dialog behaviour models with each other. for future work, the analysis must be more finegrained, e.g., by sub-dividing the set of dialogue contexts into meaningful sub-sets. furthermore, the proposed evaluation method is also suitable for directly looking at the actual behaviour of models by identifying crucial dialogue contexts and comparing the actual system reaction.