Guiding the Release of Safer E 2 E Conversational AI through Value Sensitive Design (2022)

Abstract: Over the last several years, end-to-end neural conversational agents have vastly improved their ability to carry unrestricted, open-domain conversations with humans. However, these models are often trained on large datasets from the Internet and, as a result, may learn undesirable behaviours from this data, such as toxic or otherwise harmful language. Thus, researchers must wrestle with how and when to release these models. In this paper, we survey recent and related work to highlight tensions between values, potential positive impact, and potential harms. We also provide a framework to support practitioners in deciding whether and how to release these models, following the tenets of value-sensitive design.

## introduction the social impact of natural language processing and its applications has received increasing attention within the nlp community (e.g. @xcite with large language models (llms) as one of the recent primary targets (e.g. @xcite @xcite . this paper examines what considerations are salient when designing and releasing conversational ai (convai) models. we focus on neural conversational response generation models that are trained on open-domain dialog data and lack a domain-specific task formulation, but instead are designed to freely and engagingly converse about a wide variety of topics. these models are typically trained in the popular encoder-decoder paradigm, which was first introduced for this task by @xcite ; @xcite ; @xcite . we call conversational models trained in this paradigm end-to-end (e2e) systems because they learn a hidden mapping between input and output without an interim semantic representation. an important benefit of e2e convai models trained in this paradigm is that they can be adapted to new domains or taught new skills just by fine-tuning a pre-trained model on datasets of interest (e.g. @xcite @xcite . releasing these pre-trained models thus allows different groups of researchers to build on the work of others, which can increase reproducibility and progress. unfortunately, releasing a model can also have harmful impacts. we discuss a subset of ethical challenges related to the release and deployment of these models, which we summarise under the term "safety," and highlight tensions between potential harms and benefits resulting from such releases. this is particularly salient in light of recently proposed ai regulation in the european union @xcite . while several recent efforts have been made to describe and mitigate unsafe behaviour of conversational models (e.g. @xcite @xcite @xcite , this work aims to provide a framework to help practitioners think through the conflicts and tensions that arise when designing a conversational model and deciding whether or not to release it, and how. releasing models "safely" is particularly challenging for the research community. the concept of "safe language" varies from culture to culture and person to person. it may shift over time as language evolves and significant cultural or personal events provide new context for the usage of that language. in addition, the downstream consequences may not be fully known a priori, and may not even be felt for years to come. this is particularly true for large interactive e2e models, where the space of possible generated replies is both extremely vast and highly dependent on context, and can therefore not be exhaustively explored before release. researchers are then left with the task of trying to arbitrate between uncertain, changing, and conflicting values when making decisions about creating and releasing these models. we propose ways to conceptualise the interaction of values at play in conversational models (section 3). based on that understanding, we present a conceptual analytical framework to guide researchers and practitioners towards making betterinformed decisions about model release (section 4). we aim to move away from a notion of safety that is based on "the absence of risk" to a more resilience-based notion of safety that is focused on the ability of sociotechnical systems (i.e., users, developers, and technology combined) to anticipate new threats and value changes. ## safety problems and mitigations in e2e conversational ai models we first illustrate some possible sources of safety concerns for convai models through concrete examples grounded in references to existing workpointing out similarities and differences in issues shared with llms. we mainly distinguish convai and generative llms by their usage: we refer to convai models if they are used interactively and take an active role as the interlocutor in a dialogue, whereas we refer to llms if models are mainly used to generate text, e.g., via text completion or via prompting. ## tensions between values, potential positive impact, and potential harm after highlighting some existing barriers to the creation of safe convai (as well as possible mitigations), we lay out some important tensions between values, positive impact and potential harm. these considerations establish a foundational understanding of the system, after which we can consider release decisions (discussed in section 4). there is a growing understanding that computing systems encode values, and will do so whether or not the parties involved in designing and releasing the system are explicitly aware of those values @xcite . reflecting more deliberately on values throughout model development can help surface potential problems and opportunities early on, identify what informa-tion might be important to communicate as part of a model release, and allow practitioners and downstream users to make better-informed decisions. we use the broad definition of values employed in @xcite : "what a person or group of people consider important in life." with this definition, values extend beyond the use of the term akin to moral tenets, to the more general things of value. examples relevant to conversational agents could be: getting or providing education, companionship, or comfort, preserving privacy, widening access to more populations through automationor trust, friendship, accessibility, and universality. throughout this section, we employ the scenario of a hypothetical companion: a potential chatbot that leverages the constant availability and scalability of automated systems to provide companionship to people who feel lonely. however, it could raise privacy and consent concerns, e.g., if the conversations are recorded for subsequent improvement of the model without informing the user. deeper concerns would be that the system might displace human companionship in a way that creates an unhealthy reliance on a bot, a decreased motivation to engage with humans, and a lower tolerance to the limited availability and patience of humans. ## a framework for researchers to deliberate model release the topic of when and how to release llms designed by research groups has been of increasing interest to the community (e.g. @xcite @xcite ; partnership on ai, 2020; partnership on ai , 2021; @xcite . the case is similar for conversational models, with safety issues in particular posited as a reason for withholding the release of such models. for example, in a blog post about the convai model meena @xcite the authors cite safety challenges as a reason for not releasing the model. 2 within the broader context of value-sensitive design, and absent responsible release norms in the field @xcite , we propose the following elements of a framework to aid researchers in deliberating safer release, and guidance to support learning during and after release. we ground our discussion in two relevant, theoretical case studies: the researchers from case 1 would additionally like to release a small scale demo of their model through a chat interface on a website. creating such a demo would allow non-expert stakeholders to interact with the model and gain a better sense of its abilities and limitations. ## conclusion besides the overall challenges posed by large language models, conversational models present specific issues. they are inherently dependent on the unknown inputs of the users who will be repeatedly interacting with the systems and steering them in combinatorially unpredictable directions. the costs and benefits of releasing a model can thus be hard to determine, especially when they only appear after cascades of uncertain consequences at different time scales. reckoning with these issues requires weighing conflicting, uncertain, and changing values. to aid in this challenging process, we provided a framework to support preparing for and learning from model release, following principles of value-sensitive design. we illustrate each of our proposed steps with concrete, hypothetical scenarios to help practitioners in their reflection. while this is a theoretical paper, informed by an interdisciplinary collaboration, we believe in the value of publishing it through an applied conference since this will maximise the chances of reaching our target audience.