Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contexts (2021)

Abstract: Dialogue models trained on human conversations inadvertently learn to generate toxic responses. In addition to producing explicitly offensive utterances, these models can also implicitly insult a group or individual by aligning themselves with an offensive statement. To better understand the dynamics of contextually offensive language, we investigate the stance of dialogue model responses in offensive Reddit conversations. Specifically, we create ToxiChat, a crowd-annotated dataset of 2,000 Reddit threads and model responses labeled with offensive language and stance. Our analysis reveals that 42% of human responses agree with toxic comments, whereas only 13% agree with safe comments. This undesirable behavior is learned by neural dialogue models, such as DialoGPT, which we show are two times more likely to agree with offensive comments. To enable automatic detection of offensive language, we fine-tuned transformer-based classifiers on ToxiChat that achieve 0.71 F1 for offensive labels and 0.53 Macro-F1 for stance labels. Finally, we quantify the effectiveness of controllable text generation (CTG) methods to mitigate the tendency of neural dialogue models to agree with offensive comments. Compared to the baseline, our best CTG model achieves a 19% reduction in agreement with offensive comments and produces 29% fewer offensive replies. Our work highlights the need for further efforts to characterize and analyze inappropriate behavior in dialogue models, in order to help make them safer.

## introduction despite significant progress toward data-driven conversational agents @xcite , dialogue models still suffer from issues surrounding safety and offensive language. previous 1 our code and corpus are available at https:// github.com/abaheti95/toxichat because religious people who think they have license from god to do whatever they want are f***ing psychotics thank you for saying what i was thinking! user: 🤖bot 1: offensive to religious folks disagree agree reddit comment from r/atheism stance labels that's wrong, no one said they can do whatever they want. everyone in the world's the same research has shown that dialogue models can produce utterances that are gender and racially biased @xcite @xcite . for example, openai's gpt-3 @xcite , a 175 billion parameter neural network, has been shown to generate dangerous advice, such as recommending a hypothetical patient to kill themselves. 2 presenting users with content generated by a neural network presents new risks, as it is difficult to predict when the model might say something toxic, or otherwise harmful. a key challenge for conversational ai is that toxic language is often context-dependent @xcite , making it notoriously difficult to detect; text that seems innocuous in isolation may be offensive when considered in the broader context of a conversation. for example, neural chatbots will often agree with offensive statements, which is undesirable (see examples in figure 1 ). the solution employed by current systems, such as gpt-3 or facebook's blender chatbot @xcite , is to stop producing output when offensive inputs are detected @xcite . this is problematic, because today's toxic language classifiers are far from perfect, often generating false positive predictions. rather than completely shutting down, for some applications, it may be preferable to simply avoid agreeing with offensive statements. however, we are most excited about the future potential for models that can gracefully respond with non-toxic counter-speech @xcite , helping to diffuse toxic situations. to better understand stance usage in offensive contexts, we recruited crowd-workers on amazon mechanical turk to annotate toxichat, a corpus of reddit conversations that include automatically generated responses from dialogpt @xcite and gpt-3 @xcite . posts and comments are annotated for targeted-offensiveness toward a particular person or group @xcite . we also annotate stance toward each of the previous comments in the thread. using our annotated corpus, we show that 42% of human responses in offensive contexts exhibit agreement stance, whereas only 13% agree with safe comments. analysis of 5 million reddit comment threads across six months, similarly finds users are three times more likely to agree with offensive comments. furthermore, we find that neural chatbots learn to mimic this behavior -dialogpt, gpt-3, and facebook's blender chatbot are all more likely to agree with offensive comments. finally, we present initial experiments with two controllable text generation (ctg) methods that aim to control the stance of automatically generated replies. our experiments suggest that domain adaptive pretraining @xcite reduces the number of contextually offensive responses, although this does not completely eliminate the problem, suggesting the need for further research on controllable stance in neural text generation. our main contributions include: (1) we release toxichat, a corpus of 2,000 reddit conversations that are augmented with automatic responses from dialogpt and gpt-3, and annotated with targeted offensive language and stance. (2) we present an analysis of stance in offensive and safe contexts using toxichat, demonstrating that neural dialogue models are significantly more likely to agree with offensive comments. (3) we show toxichat supports training and evaluating machine learning classifiers for stance in toxic conversations. (4) we conduct preliminary experiments on controlling the stance of neural responses to prevent models from agreeing with offensive statements. ## creating the toxichat corpus addressing problematic responses in neural conversation requires both understanding whether a response is offensive and whether it agrees with previous offensive utterances. we develop an interface to annotate these two concepts in conversations that are enriched with dialogue model responses. formally, a thread consists of k @xmath0, u 2 , ..., u k }, where the last comment, u k , is generated by a dialogue model. for each u i , we collect annotations of: 1) offensiveness -we consider u i offensive if it is intentionally or unintentionally toxic, rude or disrespectful towards a group or individual following @xcite . this is a binary choice, where u i is either offensive or safe. 3 for offensive comments, we further annotate target groups from a predefined list comprising identity-based groups of people (e.g., people of various sexuality/sexualorientation/gender, people with disabilities, people from a specific race, political ideologies, etc.) and specific individuals e.g., (public figures, reddit users, etc.) we present the list of selected target groups in figure 7 in the appendix. 2) stance -we annotate the stance of u i towards each previous comment, u j , ∀@xmath1. stance is viewed as a linguistically articulated form of social action, in the context of the entire thread and sociocultural setting @xcite . stance alignment between a pair of utterances is annotated as agree, disagree or neutral. our primary interest is in analyzing the stance taken towards offensive statements. we assume that a user or a chatbot can become offensive by aligning themselves with an offensive statement made by another user (see figure 1 ). 4 additionally, for dialogue model responses u k , we also annotate their grammatical and contextual plausibility given the context. a screenshot of our annotation interface is shown in figure 8 in the appendix. ## data collection our annotated dataset contains labeled reddit conversations extended with dialogue model responses ( §3.1). we gather reddit posts and comments @xcite foot_3 that were written between may and october, 2019. from this, we construct threads, each of which comprise a title, post and subsequent comment sequence. we extract threads from two sources: (1) any subreddits: threads from all subreddits, (2) offensive sub-reddits: threads from toxic subreddits identified in previous studies @xcite and reddit community-reports. foot_4 (appendix b). we are most interested in responses generated by dialogue models in offensive contexts. however, offensive language is rare in a random sample @xcite . hence, we implement a two-stage sampling strategy: (1) random sample -from both sources, randomly sample 500 threads (total 1000). (2) offensive sample -from remaining threads in both sources, sample additional 500 threads (total 1000), whose last comment is predicted as offensive by a classifier. specifically, we used high-precision predictions (@xmath2.7) from a bert-based offensive comment classifier @xcite that was fine-tuned on the social bias inference corpus @xcite . this classifier achieves ≈ 85.4 offend label f1 on the sbic dev set. ## offensive language and stance classification we now investigate the predictability of offensive language (offensive) and stance (stance) in conversations that include generated responses. given a thread, @xmath3, u 2 , ..., u k ), we predict offensive labels o @xmath4, 1} for each utterance, u i , @xmath5, agree, disagree} for every pair of utterances (u i , u j ), @xmath6. ## mitigating offensive behavior our data analysis confirms that dialogue models can generate some contextually offensive language. to steer the generation away from offensive content, we experiment with some preliminary strategies using controlled text generation (ctg). we consider the following three control attributes: (1) offensive -to control safe or offensive response generation, (2) stance -to control agreeing or neutral response generation towards its immediately preceding comment, foot_9 and (3) both offensive and stance -to control response generation with both control types. to train ctg models, we need conversations with their last response labeled with control attributes. therefore, we extract 5 million comment threads, similar to §3, and retrieve offensiveness and stance predictions using our best dgpt modelbased offensive and stance classifiers ( §5.4). to minimize classification errors, we use high precision predictions by selecting appropriate thresholds for different classification probabilities. 12 for each thread, we retain offensive prediction of the last utterance and stance prediction between the last two utterances. for all 3 proposed control experiments, we first create samples of l ≈ 250, 000 highprecision classifier labeled threads in the format {(x i , ct i , y i )} l @xmath7. here x i is the thread without the last utterance, ct i is the classifier labeled control token and y i is the last utterance or response to x i . we discard 'disagree' stance responses, as we only found about 10, 000 high-precision disagreeing responses. our final sample contains about 100, 000 offensive responses and 75, 000 agreeing responses. we further divide into each control dataset of size l into a 95-5 ratio to get train and dev split. ## conclusion to better understand the contextual nature of offensive language, we study the stance of human and model responses in offensive conversations. we create toxichat, a corpus of 2,000 reddit conversations augmented with responses generated by two dialogue models and crowd-annotated with targeted-offensive language and stance attributes. classifiers trained on our corpus are capable of automatically evaluating conversations with contextually offensive language. our analyses consistently find that reddit users agree much more with offensive contexts. this trend could be explained by the tendency of social-media users to form echo-chambers @xcite . consequently, dialogue models learn to mimic this behavior and agree more frequently in offensive contexts. however, fine-tuning dialogue models on cleaner training data with desirable conversational properties (safe and neutral responses with dapt) can mitigate this issue to some extent. to further strengthen dialogue safety, future research on detection of offensive context @xcite and subsequent generation of nonprovocative counter-speech @xcite is crucial.