S a F e RD ialogues: Taking Feedback Gracefully after Conversational Safety Failures (2022)

Abstract: Current open-domain conversational models can easily be made to talk in inadequate ways. Online learning from conversational feedback given by the conversation partner is a promising avenue for a model to improve and adapt, so as to generate fewer of these safety failures. However, current state-of-the-art models tend to react to feedback with defensive or oblivious responses. This makes for an unpleasant experience and may discourage conversation partners from giving feedback in the future. This work proposes SaFeRDialogues, a task and dataset of graceful responses to conversational feedback about safety failures. We collect a dataset of 8k dialogues demonstrating safety failures, feedback signaling them, and a response acknowledging the feedback. We show how fine-tuning on this dataset results in conversations that human raters deem considerably more likely to lead to a civil conversation, without sacrificing engagingness or general conversational ability.

## introduction large neural generative dialogue models trained to mimic human english-language open-domain conversations have become engaging @xcite , but are still prone to uttering problematic language, e.g., displaying toxicity or bias, or agreeing with offensive statements @xcite . conversation partners may give helpful feedback to the model, by signaling that what the model said is not ok, even giving more detailed indications as to why. this could in turn be precious training signal for on-going improvement of models through online learning @xcite . in particular, the boundaries of what constitutes ok figure 1 : types of bot responses when responding to feedback about problematic inputs from the bad dataset @xcite . existing models (four bars on the left) respond by attacking or ignoring the feedback. recovery models fine-tuned on the dataset of gracious responses to feedback proposed in this work apologize without defensiveness (two bars on the right). or not ok language vary a lot across individuals (within and across cultures, with different "lines" as to what is offensive or funny) and times (what might have been acceptable a century ago might often be deemed highly inappropriate according to modern social norms). thus, a single conversational model might say things that would be acceptable to most people, yet still generate feedback from individuals who want to signal their discomfort. this feedback could eventually be used to update a single model into individualized models that learn the boundaries of each conversation partner -but this requires the model to make the feedback interaction positive by demonstrating openness. instead, current conversational models typically respond to feedback in a way that discourages the partner from giving more in the future: models often double down on their controversial position, or ignore the feedback altogether (see figure 1 and this work improves the response of end-to-end conversational models to feedback about safety ## recovering from safety failures in a conversation constructive feedback is an important tool in human learning @xcite . unfortunately, feedback can often be perceived as self-threat (i.e., challenge to a positive view of oneself), leading to various defensive responses that impede learning @xcite , such as resistance to changing beliefs, or even adoption of more extreme beliefs @xcite . these common human psychological self-defense responses widely appear in large-scale human corpora used to train neural generative conversational models, such as pushshift.io reddit @xcite . accordingly, conversational models frequently exhibit defensive or oblivious responses, rejecting the feedback instead of reflecting on it (figure 1 ). this work attempts to remedy this by collecting a crowdsourced dataset where workers are specifically instructed to acknowledge feedback in a way that would lead to a civil interaction. conversational models fine-tuned on that data would then be expected to display that target quality of graceful acceptance of feedback. this overall strategy is similar to previous work endowing models with more empathy or knowledge, by fine-tuning on data collected with the goal of exhibiting the desired quality @xcite . before providing a more detailed description of our approach, we briefly review related work. ## related work as reviewed in @xcite , neural end-toend conversational models can display a host of safety issues, e.g. generating inappropriate content @xcite , or responding inappropriately to sensitive content uttered by the conversation partner @xcite . efforts to train models on adversarially collected datasets have resulted in safer models @xcite , which can however still be goaded into uttering offensive statements @xcite . feedback from the conversation partner is likely to become an important source of information for improving deployed models, as argued in @xcite , and is particularly important for making models more robust to evolving values and social norms @xcite . in this work, we do not attempt to improve the safety of conversational models, and instead focus on improving how they respond to feedback given by the conversation partner within the conversation. several works have examined response strategies to unsafe utterances. @xcite ; @xcite look at how different response strategies (disengaging, apologizing, or counter-attacking) can change how conversational models are rated and how many negative responses they elicit. curry and @xcite show that different strategies are deemed appropriate according to the type of unsafe input. @xcite look at re-offense rates after various response types. more recent work has focused on generating counterspeech and teaching interventions @xcite @xcite . by contrast, this work looks at the other side of the conversation, where the model itself has said something unsafe and the human partner has given feedback that signals it. this set-up corresponds to a learner bot, rather than a moderator bot such as in de los riscos and d'haro (2021). ## training a recovery model in this section, we introduce a new task and dataset named saferdialogues foot_0 (sd) for training models that can recover from safety failures. ## conclusion in this work, we proposed saferdialogues, a novel task and dataset of dialogues, where a conversation participant who uttered something unsafe is given feedback that signals it, and responds in a way that acknowledges that feedback and is more likely to lead to a more civil conversation down the line. we showed that fine-tuning dialogue models on this data, while carefully multi-tasking on a more general open-domain chitchat dataset, results in conversational models that are still rated as engaging and capable of normal conversation, yet are deemed significantly more likely to produce more civil conversations. we verified that the models do not unduly apologize in normal conversation, while very reliably producing graceful apologies when confronted with feedback about some not ok utterance. in future work, we will examine how to automatically detect signaling feedback and learn from it in an online learning set up, as well as examine what happens to the trajectory of natural conversations, depending on the type of feedback given, and the type of response given to that feedback.