Speaker Sensitive Response Evaluation Model (2020)

Abstract: Automatic evaluation of open-domain dialogue response generation is very challenging because there are many appropriate responses for a given context. Existing evaluation models merely compare the generated response with the ground truth response and rate many of the appropriate responses as inappropriate if they deviate from the ground truth. One approach to resolve this problem is to consider the similarity of the generated response with the conversational context. In this paper, we propose an automatic evaluation model based on that idea and learn the model parameters from an unlabeled conversation corpus. Our approach considers the speakers in defining the different levels of similar context. We use a Twitter conversation corpus that contains many speakers and conversations to test our evaluation model. Experiments show that our model outperforms the other existing evaluation metrics in terms of high correlation with human annotation scores. We also show that our model trained on Twitter can be applied to movie dialogues without any additional training. We provide our code and the learned parameters so that they can be used for automatic evaluation of dialogue response generation models.

## introduction evaluating the system generated responses for open-domain dialogue is a difficult task. there are many possible appropriate responses given a dialogue context, and automatic metrics such as bleu @xcite or rouge @xcite rate the responses that deviate from the ground truth as inappropriate. still, it is important to develop and use an automatic metric because human annotation is very costly. in addition to bleu and rouge, there is a widely-used evaluation metric based on the distributed word representation @xcite , but this metric shows low correlations with human judgments. one reason for the difficulty in developing an automatic metric that correlates well with human judgements is that the range of appropriate responses for a given context is very wide. some recent metrics such as adem @xcite and ruber @xcite compute the similarity between a context and a generated response. however, adem requires humanannotated scores to train and thus cannot be applied to new datasets and domains. ruber overcomes this limitation by using the idea that a random response should be used as a "negative sample", but it is not able to distinguish the responses in the example in in this paper, we propose speaker sensitive responses evaluation model (ssrem) that analyzes the appropriateness of the responses. we use speaker sensitive responses that are generated by one speaker to train the model. we test ssrem in comparison with other evaluation metrics. first, we make annotated human scores for responses in twitter conversation data. the evaluation scores of ssrem shows a higher correlation with human scores than other evaluation metrics. and ssrem outperforms other metrics in terms of identifying the ground truth responses given a context. we show the additional advantage of ssrem: it can be applied to evaluate a new corpus in a different domain. we train ssrem on twitter corpus and test it on a corpus of movie reviews, and we show that ssrem outperforms other metrics in terms of the correlation with human scores and the task of identifying the ground truth response. our contributions in this paper include the following. ## related work in this section, we describe existing automatic evaluation metrics for dialogue response generation and discuss their limitations. for task-oriented dialogue models such as airline travel information system @xcite , completing the given task is most important, and the evaluation metrics reflect that @xcite . but open-domain conversation models do not have specific assigned tasks; the main goal of an open-domain conversation model 1 https://github.com/nosyu/ssrem is generating appropriate responses given a conversation about any topic. existing automatic evaluation metrics compare a generated response and the ground truth response. the most widely-used metric are bleu @xcite and rouge @xcite based on the overlap of words between the two responses. a limitation of these word overlap-based metrics is that they cannot identify the synonyms, and to overcome this limitation, the embedding-based metrics use distributed word vector representations @xcite . however, these metrics have poor correlation with human judgments @xcite @xcite because they still only look at the similarity between the generated response and the ground truth. ssrem is a model with the awareness that a response can be different from the ground truth response but still appropriate for the conversation context. the responses for a casual conversation can be varied. for example, there are four appropriate responses including ground truth response for a given context in ## speaker sensitive response evaluation model this section describes our speaker sensitive response evaluation model (ssrem) that trains with speaker sensitive utterance samples. ssrem looks at a given context and its ground truth response together to evaluate a generated response. we describe the motivation of ssrem with empirical observations in section 3.1. we present the structure of ssrem in section 3.2. with the motivation, we present a training method of ssrem with speaker sensitive utterance samples in section 3.3. ## annotating human scores to measure the correlation ssrem with human judgments, we first gather human judgments of responses given a conversation context. we use amazon mechanical turk (mturk) to annotate the scores of the responses. we select 300 conversations from a dataset of twitter conversations. and we generate responses for annotation using three conversation models and the ground truth response for each conversation. bm25 retrieval model @xcite that uses tf-idf vector space. then we ask two questions to the mturkers. (1) how appropriate is the response overall? (2) how on-topic is the response? these questions are used in @xcite . the authors show that these questions have high inter-annotator agreement among workers. they suggest using the first question to annotate the human score, and so we follow the suggestion. but we ask the second question to workers to filter out workers who submit random answers. each worker answers these questions on a five-point likert scale. we annotate 1,200 responses in total. one worker answers ten conversations, four responses per conversation for a total of 40 responses. each response is tagged by five workers for a total of 287 workers of which we retain the responses from 150 workers who passed all the tests. we tag the most selected score as the human score for each response. the inter-annotator fleiss' kappa @xcite is κ = 0.61 which is consistent with the results in @xcite . ## experiment 1 -comparing with human scores this section describes the experiment that looks at the correlation between the model scores and the human scores for given contexts and responses. ## experiment 2 -identifying true and false responses the second experiment presents the performance of f function in ssrem by comparing it with baselines. ruber, rsrem, and ssrem compute the score from the context of the conversation and generated responses. to investigate the performance of the score, we set up the task that identifies the true and false responses for a given context. the true responses are ground-truth responses, and false ones are four negative samples that are described in section 3.3. ## experiment 3 -applying new corpus in this section, we investigate the applicability of ssrem to a new conversation corpus. ssrem takes the speaker sensitive samples from twitter. but there are many open-domain conversation corpora such as movie scripts @xcite . @xcite run a similar experiment with ruber, but they use the similar domain of data, chinese online forum (training from douban and testing on baidu tieba). we choose the movie scripts corpus because it is written by the script writers whereas twitter is personal causal online conversations. we present the performance of ssrem on the new corpus. ## conclusion and future work in this paper, we presented ssrem, an automatic evaluation model for conversational response generation. ssrem looks at the context of the conversation and the ground-truth response together. we proposed negative sampling with speaker sensitive samples to train ssrem. we showed that ssrem outperforms the other metrics including rsrem that uses random negative samples only. we also showed that ssrem is effective in evaluating a movie conversation corpus even when it is trained with twitter conversations. there are several future directions to improve ssrem. first, we can make ssrem more robust on adversarial attacks. @xcite shows limitations of adem on adversarial attacks such as removing stopwords and replacing words with synonyms. we investigated another type of the adversarial attack named copy mechanism that copies one of the utterances in the context as the generated response. all existing automatic evaluation methods including ruber that compare the context and the response can be cheated by the copy mechanism. ssrem is also susceptible. however, ssrem is fooled less than other existing models because ssrem learns with negative samples from the set of utterances in the same conversation. ss-rem learns to differentiate among utterances in the same context. we show this empirically with an experiment to identify true and false responses (sec 6.2). when we look at the mean score for the context utterances that shows this copy mechanism compared to the mean score of the ground-truth response (gt), the mean score of context utterances is 0.07 higher by ruber, but only 0.01 higher by ssrem. ssrem does not give lower scores for the context utterances than gt, but it is not as bad as ruber. we will make ssrem more robust on the attacks. second, we can improve ssrem for a higher correlation with human judgement. we chose to approach ssrem with a classification loss because it is simple and widely used to estimate the models using negative sampling. although the classification loss is simple, ssrem outperforms all existing automatic evaluation models. however, as third, we can extend using ssrem to various conversation corpora such as task-oriented dialogues. we trained and tested ssrem on open-figure 4 : scatter plot showing model against human scores with movie corpus. we add gaussian noise drawn from n (0, 0.3) to the human scores to better visualize the density of points which is similar to @xcite . domain conversation corpora. however, contextual coherence between the input context and the generated text is important in multi-turn conversations. we will apply ssrem to various conversation tasks for evaluating the generated text automatically. we will explore these directions in our future work.