Contrastive Response Pairs for Automatic Evaluation of Non-task-oriented Neural Conversational Models (2021)

Abstract: Responses generated by neural conversational models (NCMs) for non-task-oriented systems are difficult to evaluate. We propose contrastive response pairs (CRPs) for automatically evaluating responses from non-task-oriented NCMs. We conducted an error analysis on responses generated by an encoder-decoder recurrent neural network (RNN) type NCM and created three types of CRPs corresponding to the three most frequent errors found in the analysis. Three NCMs of different response quality were objectively evaluated with the CRPs and compared to a subjective assessment. The correctness obtained by the three types of CRPs were consistent with the results of the subjective assessment.

## introduction non-task-oriented dialogue systems must generate responses based on dialogue contexts although possible responses are not limited to a few correct answers. neural conversational models (ncms), such as an encoder-decoder rnn with an attention mechanism @xcite @xcite and transformer @xcite , generate fluent responses; however, an automatic evaluation of response quality in non-task-oriented ncms has not been established yet. reference-based evaluation indices such as bleu have a low correlation with subjective scores because of the diversity of possible responses. to address this problem, there have been various proposals such as an index referencing a model response and taking into account the previous utterance of the interlocutor @xcite , an index integrating subjective and statistical evaluations @xcite , and an interactive evaluation method assuming that the quality can only be evaluated through interaction @xcite . on the other hand, neural machine translation (nmt) has improved its quality at the sentence level, and context awareness (i.e., consistency between translated sentences when processing a text or series of sentences) still remains a challenge. @xcite proposed contrastive discourse sets to evaluate how well nmt models handle anaphoric pronouns, and coherence and cohesion for context-aware nmt @xcite , by extending his proposed contrastive translation pairs (ctps) @xcite . a ctp consists of a correct translation and an incorrect one in which a minimal number of words is substituted with wrong ones. the model quality is measured on correctness, i.e., the ratio of the number of pairs in which the correct translation received a higher score in forced decoding than the incorrect one to the total number of pairs. voita et al. further analyzed errors in context-aware english-russian nmt to extract frequent error patterns and proposed a set of ctps to evaluate the accuracy of an nmt in terms of the frequent error patterns @xcite . in this paper, we propose contrastive response pairs (crps) for automatically evaluating the quality of ncm responses with reference to the ctps for evaluating context-aware nmt. we first conducted an error analysis on responses generated by ncms trained on a large-scale conversation corpus. then, we created a set of crps corresponding to three frequent error patterns. finally, we examined whether the crps correctly reflected the difference in ncm response quality by comparing the correctness of the crps and the results of a subjective assessment on three ncms with varying levels of quality. specifically, we proceeded in the following steps. terms of naturalness in the dialogue context. then, we further classified the responses that were judged unnatural into 10 error classes manually and counted their frequencies. ## error analysis of responses generated by neural conversational models we simulated conversation between women using ncms. we used a large-scale fictive conversation corpus between two japanese ladies "miss yoshida" and "miss sasaki" for training and evaluating the ncms. the corpus consists of 1.68 million fictive conversations compiled by 200 crowdworkers. the characters were kept consistent by specifying detailed personas across 80 items, which were shared among crowd-workers. we extracted 1.1m, 64k and 64k of yoshida's utterances with preceding dialogue contexts for training, validation, and evaluation of yoshida model. we trained a gru-based encoder-decoder rnn model with an attention mechanism, the network architecture of which is shown in figure 1 . the model received yoshida's and sasaki's previous utterances with two encoders, and output yoshida's response. we refer to this model as the "double attention model." the model was trained by teacher forcing with the cross-entropy loss function. the double attention model generated responses on the basis of the maximum mutual information criterion @xcite . we randomly sampled 3,000 responses from the validation set. three of the authors manually analyzed errors in the 3,000 responses. first, they rated each response as natural or unnatural in its dialogue context. if it was unnatural, they determined the reason for unnaturalness using their own criteria. then they negotiated with each other to unify the error classes and criteria. after the unity, they determined the reason for unnaturalness with the unified criteria for responses deemed unnatural by more than one rater. on average, 41.9% of the responses were classified as unnatural. cohen's kappa coefficients between all the pairs were 0.61. the unnatural responses were broken down into the distribution shown in figure 2 . the most frequent errors were caused by contextually-inappropriate content words (icw, 28.9%), followed by inappropriate function words (fnc, 9.8%), inappropriate endof-sentence expressions (ese, 8.9%) and not answering the previous question (na, 8.0%), not including others (etc, 15.0%). we created crps to evaluate the performance of the ncm on the three 3 creation of contrastive response pairs ## conclusion we proposed contrastive response pairs (crps) for automatically evaluating neural conversational models for non-task-oriented dialogue systems. three types of crps were created on the basis of an error analysis of responses generated by ncms, and their capability of measuring ncm performance was examined using three ncms of varying quality. the correctness given by automatic evaluation with the crps was mostly consistent with the results of a subjective assessment. in future work, we will increase the size of crps and create crps automatically.