S afety K it: First Aid for Measuring Safety in Open-domain Conversational Systems (2022)

Abstract: The social impact of natural language processing and its applications has received increasing attention. In this position paper, we focus on the problem of safety for end-to-end conversational AI. We survey the problem landscape therein, introducing a taxonomy of three observed phenomena: the Instigator, Yea-Sayer, and Impostor effects. We then empirically assess the extent to which current tools can measure these effects and current systems display them. We release these tools as part of a “first aid kit” (SafetyKit) to quickly assess apparent safety concerns. Our results show that, while current tools are able to provide an estimate of the relative safety of systems in various settings, they still have several shortcomings. We suggest several future directions and discuss ethical considerations.

## introduction several recent studies discuss the potential harms and benefits of large language models (llms), e.g., @xcite ; @xcite ; @xcite . here, we turn our attention to neural conversational response generation models that are trained "end-to-end" on open-domain dialog data (e2e convai). examples include di-alogpt @xcite , meena bot @xcite , and blenderbot @xcite . in contrast to general generative or autoregressive llms, these specialized models are typically deployed in an interactive setting, i.e., conversing with a user. they are trained on large amounts of conversational data, for example twitter, pushshift.io reddit @xcite , or opensubtitles dataset. large neural models in general, and convai models in particular, have been shown to replicate and even amplify negative, stereotypical, and derogatory associations in the data @xcite . in addition, neural lm generation is hard to control, although there are some first steps in this direction @xcite . these two facts taken together can result in situations where convai systems generate inappropriate content @xcite , or respond inappropriately to offensive content @xcite . furthermore, recent research suggests that the anthropomorphic design of these systems (c.f. @xcite correlates with increased instances of bullying behavior @xcite . this change in interaction style and the attribution of agency @xcite results in safety scenarios that are qualitatively different from llms: here, an inappropriate response might result in severe, or even life-threatening, consequences for the user @xcite . we summarize these issues resulting in potential harm under the term "safety." in particular, we consider harmful system behavior that can lead to negative short-term impact, e.g., the user feeling insulted, and long-term harm, e.g., negative societal stereotypes being reinforced. we consider three safety-sensitive phenomena for conversational systems, which we refer to as: the instigator, yea-sayer, and impostor effects (see §2). we provide an in-depth discussion of the potential impact of these three scenarios and define them in the context of related work. we then empirically evaluate currently available tools for assessing the impact of e2e conversational ai models with respect to these phenomena. we perform detailed experiments and analyses of the tools therein using five popular conversational ai agents, release them in a open-source toolkit (safetykit), and make recommendations for future use. ## problem landscape we introduce a taxonomy of three safety-sensitive situations for e2e convai models, summarized name issue examples source instigator effect the system generates harmful content. tay bot example from mihalcik (2016) "santa claus is definitely dead." (user group specific example: inappropriate content for children.) alana v.1 @xcite reading out part of a news article @xcite . ## safety first aid kit in the following, we investigate to what extent existing tools are suitable to support researchers in making more informed decisions about building and releasing their models. we assemble these tools in a safetykit, an open-source toolkit/repository to be extended as more (suitable) tools become available. similar to a first aid kit, safetykit is meant to detect apparent/ pronounced safety concerns, however, we recommend a more thorough examination through, for example, a stakeholder-focused study in order to fully assess potential harms. in order to discourage hill-climbing on a benchmark and the negative effects which can stem from it @xcite , we do not provide an aggregate score, but discuss possible uses of the tools under investigation and their advantage and disadvantages. like a first aid kit, safetykit is quick and easily to access: it can be run with only simple api access to a model, i.e., a "black box" evaluation which only requires text output from the model, and consists of both unit tests ( §3.1), i.e., automatic metrics, and integration tests ( §3.2), i.e., human evaluations. the limitations of safetykit are discussed in depth in §5. all tools are open-sourced at https://parl.ai/projects/safety_bench/ . ## conclusion we identify three safety-sensitive situations for e2e convai systems: the instigator, yea-sayer, and impostor effects -where the latter two are unique to interactive, conversational settings. we then empirically assess the extent to which current tools can measure these effects and current systems display them. we release these tools as part of a "first aid kit" (safetykit) to quickly assess safety concerns. our results show that, while current tools are able to provide an estimate of the relative safety of systems in various settings, they still have several shortcomings -especially for utterances which are contextually unsafe. we thus encourage further contributions to safe-tykit, e.g., research into more comprehensive automatic measures, as well as into human evaluation and iterative, value-based frameworks to assess potential harms, e.g., @xcite .