Generating Negative Samples by Manipulating Golden Responses for Unsupervised Learning of a Response Evaluation Model (2021)

Abstract: Evaluating the quality of responses generated by open-domain conversation systems is a challenging task. This is partly because there can be multiple appropriate responses to a given dialogue history. Reference-based metrics that rely on comparisons to a set of known correct responses often fail to account for this variety, and consequently correlate poorly with human judgment. To address this problem, researchers have investigated the possibility of assessing response quality without using a set of known correct responses. RUBER demonstrated that an automatic response evaluation model could be made using unsupervised learning for the next-utterance prediction (NUP) task. For the unsupervised learning of such model, we propose a method of manipulating a golden response to create a new negative response that is designed to be inappropriate within the context while maintaining high similarity with the original golden response. We find, from our experiments on English datasets, that using the negative samples generated by our method alongside random negative samples can increase the model’s correlation with human evaluations. The process of generating such negative samples is automated and does not rely on human annotation.

## introduction automatic evaluation of responses can be difficult because multiple answers could be suitable for a single context. well-known metrics often used in machine translation or text summarization, such as bleu @xcite , meteor @xcite , or rouge @xcite , are based on measuring n-gram overlap with a set of humanannotated golden answers. compared to machine figure 1 : example of the three different types of responses for a given dialogue history. our method manipulates the original response "what's wrong with heading out with mark for vacation?" to generate the negative sample "what's wrong? go out with mark for dinner." to solve this problem, @xcite proposed a supervised regression model that makes predictions independent of correct answer candidates. although they were able to achieve better correlation with human evaluations, their method depends on procuring a human-annotated dataset to learn from. @xcite used the next-utterance prediction (nup) task to learn for automatic response evaluation. their model, which is unsupervised, learned to distinguish an appropriate response from random negative samples (responses randomly taken from the training corpus). the model can evaluate the response quality by estimating the probability that the response occurs directly after the dialogue history. they also demonstrated that the probability-based evaluations highly correlated with human evaluations of response quality. in this paper, we propose a method to create a negative sample by manipulating a golden response. the manipulation is carried out in three steps: (1) scoring each word, (2) selecting words to replace, and (3) replacing the selected words. in the first step, each word is assigned a score designed to determine how dependent the word is on the context. in the second step, we select all the words with a score above a threshold value, where higher scores indicate higher dependency to the dialogue history. in the third step, all previously selected words are masked and replaced with words predicted in their place by a pretrained language model (lm). figure 1 shows an example of a negative sample generated by our method. when "what's wrong with heading out with mark for vacation?" is the golden response, the tokens "with", "heading", "vacation", and "?" were selected and replaced with "?", "go", "dinner", and ".", in that order. we find that the model trained with our negative samples alongside random negative samples shows a higher correlation with human evaluations than the models trained only on random negative samples, in experiments using two datasets @xcite . we also find evidence that automatic evaluation systems trained with the negative samples generated by our proposed method can make decisions closer to human judgment than those without. the contributions of this paper are as follows: ## method in this section, we describe our method to generate negative samples. the proposed method creates a negative sample by selecting and replacing specific word(s) in a golden response. the word selection is based on the difference between (a) the estimated probability that a word would appear in the response considering the dialogue history and (b) the estimated probability that the word would appear in the response when the dialogue history is not considered. an lm that can perform mlm can be used to estimate these probabilities. words that have large differences in probability are selected and replaced with other words. when replacing a word with another word, an lm that can perform mlm can be used to predict the word that is most likely to appear in the position of the original word when the dialogue history is not given. ## conclusion in this paper, we proposed an automatic method for generating negative samples that can be used to train an unsupervised and unreferenced response evaluation model. we performed experiments to demonstrate that the proposed method can boost the unsupervised training of a response evaluation model. we analyzed the experiment results quantitatively, and examined some examples that show the distinct characteristics of our proposed method.