Open-Domain Dialog Evaluation Using Follow-Ups Likelihood (2022)

Abstract: Automatic evaluation of open-domain dialogs remains an unsolved problem. Existing methods do not correlate strongly with human annotations. In this paper, we present a new automated evaluation method based on the use of follow-ups. We measure the probability that a language model will continue the conversation with a fixed set of follow-ups (e.g. not really relevant here, what are you trying to say?). When compared against twelve existing methods, our new evaluation achieves the highest correlation with human evaluations.

## introduction despite the recent progress in natural language processing, the automatic evaluation of opendomain conversations remains an unsolved problem. it is difficult to establish criteria to measure the quality of a system. task-oriented dialog systems use metrics such as task success or dialog efficiency. however, these do not apply to opendomain conversational agents @xcite . currently, there are two options for open-domain dialog evaluation: human evaluation and automated evaluation. thanks to their understanding of natural language, humans are able to digest the entire dialog context in order to meaningfully evaluate a response @xcite . human evaluation also has its shortcomings: inconsistency in ratings (the same annotator may give two different scores depending on the mood), lack of reproducibility, and cost @xcite . the second option is to use automated evaluation metrics. methods inherited from sequence-tosequence machine translation such as bleu @xcite evaluate the generated utterance by comparing it to the ground-truth. by doing so, these methods miss the one-to-many characteristic of conversation: a conversation may evolve in more than one valid direction. figure 1 : illustration of our method. we measure the probability (log-likelihood) that a language model will continue the conversation with a set of predefined follow-ups. this paper shows that the sum of the individual log-likelihoods correlates strongly with human evaluations. to tackle this problem, researchers came up with reference-free evaluation metrics: the generated utterance is not compared to a ground truth but evaluated on its own. fed @xcite ) is an unsupervised reference-free evaluation metric. it uses the idea that one can use the next utterance in a conversation to rate the turn before it. when users speak to a system, their response to a given system may implicitly provide feedback for the system. fed uses a set of predefined follow-ups and the log-likelihood from a language model to measure 18 fine-grained attributes in a conversation. inspired by the fed metric, we propose a new evaluation method called full (follow-up log-likelihood). we start by explaining our method and how it departs from the original fed metric. next, we explain our choice of language model and follow-ups. finally, we demonstrate that our new method achieves the highest correlation with human evaluations compared to 12 automated metrics. we open-source our evaluation code foot_0 and publish full as a python package foot_1 for easy usage. ## related work this section reviews the existing literature on evaluation metrics for open-domain conversations. in the interest of space, we limit ourselves to studying reference-free methods (methods that do not require a ground truth). the interested reader is encouraged to read @xcite for a full review. grade @xcite and dynaeval @xcite use a graph-based structure to model the dialog-level interaction between a user and a system. dynaeval distinguishes between well-formed dialogs from carefully constructed negative samples. maude @xcite is also trained to distinguish a correct response from a randomly sampled negative response using a contrastive loss. flowscore @xcite evaluates the quality of a dialog using the dynamic information flow in the dialog history. usr @xcite trains several models to measure different qualities of dialogs. a masked language modeling head measures the fluency of the conversation, a retrieval model determines the relevance of a response, and a fact-toresponse model checks whether a response conditions on knowledge. usl-h @xcite also has three internal models, although they measure different attributes: grammatical correctness, sensibleness, and the likelihood of a given response. other notable evaluation methods include @xcite ; @xcite ; @xcite fed @xcite and holisticeval @xcite both use gpt-like @xcite models to evaluate conversation on several attributes. fed computes the likelihood of manually designed follow-up utterances to measure multiple dialog qualities without supervision. holisticeval uses a gpt-2 model to measure coherence, fluency, diversity, and consistency. ## method our metric full (follow-up log-likelihood) is a reference-free evaluation method for dialogs inspired by fed @xcite . figure 1 provides an overview. ## experimental settings this section explains our choices of follow-ups, language models, and conversational data. our goal is to find the combination of language models and follow-ups correlating the most with human evaluations. ## results our objective is to find the best combination of language models and follow-ups. we start by analyzing which language model correlates the most with human evaluation. in the second step, we look for the best set of follow-ups. ## conclusion this short paper introduces a new automated evaluation method (full) for open-domain conversations. full measures the quality of a conversation by computing the probability that a language model will continue the conversation with a set of followups (e.g., not really relevant here, what are you trying to say?). full achieves the highest correlation with human evaluations compared to twelve other existing methods. our experiments revealed that negative followups (e.g., not really relevant here) have a higher correlation with human evaluations than positive follow-ups (e.g., wow, interesting to know). it is easier for the model to evaluate a conversation from its bad angles rather than its good ones. future work is needed to know which finegrained attribute can be measured using the same technique. using ever-large models such as @xcite or opt @xcite could be a direction for future research, although the resulting model will likely need to be distilled to be of practical use. as language model because of its high correlation with human evaluations. for space reasons, blender is abbreviated as bld and dialogpt as dgpt. follow-up category level type level dialog x not really relevant here. specific turn neg 0.48 0.65 x you're really confusing. error recovery dialog neg 0.46 0.67 i don't understand what you're saying. correct turn neg 0.46 0.58 that's not really relevant here. specific turn neg 0.45 0.70 you are so confusing. coherent dialog neg 0.45 0.64 x you're really boring. informative dialog neg 0.44 0.65 that's not very interesting. interesting turn neg 0.44 0.60 that was a really boring response. interesting turn neg 0.43 0.63 x you don't seem interested. inquisitive dialog neg 0.43 0.61 i am so confused right now. error recovery dialog neg 0.43 0.60 i'm so confused! understandable turn neg 0.43 0.59 i don't really care. that's pretty boring. engaging turn neg 0.43 0.61 i want to talk about something else. engaging turn neg 0.43 0.65 that's not even related to what i said. relevant turn neg 0.42 0.58 x what are you trying to say? understanding dialog neg 0.42 0.68 i am so confused right now! correct turn neg 0.42 0.57 that makes no sense! semantically appropriate turn neg 0.42 0.56 i don't understand at all! understandable turn neg 0.41 0.54 that's really boring. interesting turn neg 0.41 0.54 i don't like you. likeable dialog neg 0.40 0.58 i'm so confused right now! fluent turn neg 0.40 0.56 don't change the topic! relevant turn neg 0.40 0.58 you're not understanding me! correct turn neg 0.40 0.62 that's a very generic response. specific turn neg 0.39 0.50 you don't really know much. informative dialog neg 0.39 0.52 you're not very nice. likeable dialog neg 0.38 0.56 you're not very fun to talk to. likeable dialog neg 0.37 0.55 is that real english? fluent turn neg 0.37 0.49 that's a lot of questions! inquisitive dialog pos 0.36 0.52 why are you repeating yourself? diverse dialog neg 0.35 0.50 you're making no sense at all. coherent dialog neg 0.35 0.43 you ask a lot of questions! inquisitive dialog pos 0.35 0.54 let's change the topic. engaging turn neg 0.35 0.45 you don't ask many questions. inquisitive dialog neg 0.35 0.54 why are you changing the topic? relevant turn neg 0.34 0.51 stop saying the same thing repeatedly. diverse dialog neg 0.34 0.50 do you know how to talk about something else? flexible dialog neg 0.33 0.49 you're changing the topic so much! coherent dialog neg 0.33 0.47 you know a lot of facts! informative dialog pos 0.32 0.48 tell me more! engaging turn pos 0.32 0.34 i like you! likeable dialog pos 0.31 0.43 wow that's a lot of information. informative dialog pos 0.31 0.38 stop changing the topic so much. depth dialog neg 0.31 0.44 what does that even mean? understandable turn neg 0.30 0.35 i don't want to talk about that! flexible dialog neg 0.29 0.50