Using In-Context Learning to Improve Dialogue Safety (2023)

Abstract: While large neural-based conversational models have become increasingly proficient dialogue agents, recent work has highlighted safety issues with these systems. For example, these systems can be goaded into generating toxic content, often perpetuating social biases or stereotypes. We investigate a retrieval-based approach for reducing bias and toxicity in responses from chatbots. It uses in-context learning to steer a model towards safer generations. Concretely, to generate a response to an unsafe dialogue context, we retrieve demonstrations of safe responses to similar dialogue contexts. We find our method performs competitively with existing approaches to dialogue safety without requiring training. We also show, using automatic and human evaluation, that reductions in toxicity obtained using our approach are not at the cost engagingness or coherency. Finally, we note our method can be used in compliment to existing dialogue safety approaches, such as RLHF.

## introduction large neural-based language models are becoming increasingly proficient dialogue agents @xcite @xcite . while these models are capable of engaging in interesting and coherent dialogue, recent work has shown these systems are prone to generating unsafe content @xcite @xcite ; inter alia). for example, these models often exhibit social biases @xcite and inappropriately align themselves with offensive statements during conversation @xcite . as these models are used interactively, ensuring they generate safe and sensible responses is critical. women are terrible. all they are good for is cleaning! ## methodology we investigate a retrieval-based approach for safe response generation from decoder-only transformer @xcite models. concretely, a conversation between two persons. person 1: {utterance 1} . . . person 2: {utterance n} a conversation between two persons. person 1: {utterance 1} . . . person 2: k × demonstration target context we experiment with different sized opt @xcite , llama @xcite , and vicuna (chiang et al., 2023) models. we experiment primarily with opt models as the model code and weights are openly available however, we also highlight relevant llama and vicuna results (see appendix e for complete results) throughout our work. henceforth, we refer to the dialogue context we want to generate a response to as the target context and the demonstrations of safe model behaviour as safety demonstrations. at a high-level, our approach consists of two steps: 1) we retrieve safety demonstrations based upon their similarity to the target context; and 2) we use the retrieved safety demonstrations in-context to condition generation. we describe these steps in detail below. 1) retrieving safety demonstrations. we investigate three methods for selecting safety demonstrations for a target context: 1) randomly selecting demonstrations; 2) using bm25 @xcite to select demonstrations; and 3) using a sentencetransformer @xcite ). 2 for each retriever, we use the target context as the query to select demonstrations. these safety demonstrations are entire conversations consisting of unsafe utterances and prosocial responses. throughout our work, we refer to our sentencetransformer retriever as a "dense" retriever. ## experimental setup below, we describe the dialogue datasets used in this work. in addition, we discuss how we evaluate response safeness and relevance (i.e., quality). ## do in-context safety demonstrations improve response safeness? we first investigate if using in-context safety demonstrations can reduce toxicity from dialogue systems (q1). we also evaluate the impact of using safety demonstrations on response quality. importantly, we want to ensure safety improvements are not at the cost of interestingness, engagingness, or coherency. for example, while a dialogue system that apologizes constantly may be safe, it is not particularly interesting or engaging. this is usually dubbed as the harmless vs. helpful tradeoff @xcite . to evaluate our method, we generate responses to prosocialdialog, diasafety, and commonsense-5 https://platform.openai.com/docs/models/ gpt-3-5 dialogues. we discuss our results below. ## discussion below, we summarize our findings for each research question investigated in this work. q1: do in-context safety demonstrations improve response safeness? we find in-context learning can be used to increase dialogue system safety. our results suggest that in-context safety demonstrations are most useful when they have high similarity with the target context, evident by performance improvements with better retrievers. however, we also observed that substantial reductions in toxicity can still be obtained providing any safety demonstrations. finally, our human evaluation shows these safety improvements are not at the cost of other generation qualities. ## conclusion to the best of our knowledge, we perform the first large-scale evaluation of in-context learning for dialogue safety. we use in-context learning to reduce toxicity in three models: opt, llama, and vicuna. our results suggest that in-context learning performs competitively with traditional trainingbased approaches to dialogue safety. furthermore, our proposed method can be used in compliment with popular dialogue safety approaches, such as rlhf. we hope our work spurs future research investigating the role of retrieval in dialogue safety.