Probing the Robustness of Trained Metrics for Conversational Dialogue Systems (2022)

Abstract: This paper introduces an adversarial method to stress-test trained metrics for the evaluation of conversational dialogue systems. The method leverages Reinforcement Learning to find response strategies that elicit optimal scores from the trained metrics. We apply our method to test recently proposed trained metrics. We find that they all are susceptible to giving high scores to responses generated by rather simple and obviously flawed strategies that our method converges on. For instance, simply copying parts of the conversation context to form a response yields competitive scores or even outperforms responses written by humans.

## introduction one major issue in developing conversational dialogue systems is the significant efforts required for evaluation. this hinders rapid developments in this field because frequent evaluations are not possible or very expensive. the goal is to create automated methods for evaluating to increase efficiency. unfortunately, methods such as bleu @xcite have been shown to not be applicable to conversational dialogue systems @xcite . following this observation, in recent years, the trend towards training methods for evaluating dialogue systems emerged @xcite @xcite . the models are trained to take as input a pair of context and candidate response, and output a numerical score that rates the candidate for the given context. these systems achieve high correlations to human judgments, which is very promising. unfortunately, these systems have been shown to suffer from instabilities. @xcite showed that small perturbations to the candidate response already confuse the trained metric. this work goes one step further: we propose a method that automatically finds strategies that elicit very high scores from the trained metric while being of obvious low quality. our method can be applied to automatically test the robustness of trained metrics against adversarial strategies that exploit certain weaknesses of the trained metric. dialogue policy trained metric context response rl reward context: are you married to anyone ? no , i have trouble speaking my mind , so i am shy rl-response: i love to play video games! what kind of things do you love to do with your time? i love music. trained metric score: ## related work trained metrics. in recent years the field of trained metrics gained traction after word-overlap methods have been shown to be unreliable @xcite . the first of these metrics is adem @xcite , which takes as input a context, a reference, and the candidate response and returns a score. the main issue with adem is the reliance on references and annotated data (i.e., human ratings of responses), which are costly to obtain, and need to be redone for each domain. ruber @xcite extended adem by removing the reliance on annotated data for training. however, it still relies on a reference during inference. autojudge @xcite removed the reliance on references, which allows the evaluation of multi-turn behavior of the dialogue system. however, autojudge still leverages annotated data for training. usr @xcite ) is a trained metric that does not rely on either annotated data or any reference. it is trained in a completely unsupervised manner while still highly correlated to human judgment (0.4 spearman correlation). similarly, maude @xcite is trained as an unreferenced metric built to handle the online evaluation of dialogue systems. ## method our method applies a trained metric as a reward signal r(c, r) to update a dialogue system π(c) in a reinforcement learning setting, where c denotes the context and r the response. the dialogue system is trained by generating a response for a context, which is then scored by the automated metric. the dialogue system is then updated using the score as the reward. this process is repeated for different contexts. we use the actor-critic framework to optimize the policy @xcite . see algorithm 1 for an overview. the policy gradient is defined as j rl (θ) = θ log π θ (r|c) * a(r, c), where π θ (r|c) defines the probability of the generated response for the given context, and a(c, r) the advantage function. the learned policy depends on the reward function, i.e., the automated metric. if the reward function is susceptible to adversarial attacks, the policy will likely generate an objectively suboptimal solution, which is rated highly by the automated metric. conversely, we expect the policy to improve the dialogue systems' responses if the automated metric is robust against adversarial examples. ## results the policies typically converge towards one of the following three degenerate strategies. parrot. here, the policy simply copies parts of the context into the response. sometimes, it applies slight changes. for instance, it changes the pronouns from "you" to "i". ## conclusion trained metrics for automatic evaluation of conversational dialogue systems are an attractive remedy for the costly and time-consuming manual evaluation. while high correlation with human judgments seems to validate the metrics regarding their ability to mimic human judging behavior, our analysis shows that they are susceptible to rather simple adversarial strategies that humans easily identify. in fact, all metrics that we used failed to recognize degenerate responses. our approach is easily adaptable to any newly developed trained metric that takes as input a pair of context and response. there are no known remedies for this problem. thus, the next open challenge is to find methods that improve the robustness. yi-ting yeh, maxine eskénazi, and shikib mehri. 2021. a comprehensive assessment of dialog evaluation metrics. arxiv, abs/2106.03706. saizheng zhang, emily dinan, jack urbanek, arthur szlam, douwe kiela, and jason weston. 2018. personalizing dialogue agents: i have a dog, do you have pets too? in proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers), pages 2204-2213, melbourne, australia. association for computational linguistics.