Improving Dialog Safety using Socially Aware Contrastive Learning (2024)

Abstract: State-of-the-art conversational AI systems raise concerns due to their potential risks of generating unsafe, toxic, unethical, or dangerous content. Previous works have developed datasets to teach conversational agents the appropriate social paradigms to respond effectively to specifically designed hazardous content. However, models trained on these adversarial datasets still struggle to recognize subtle unsafe situations that appear naturally in conversations or introduce an inappropriate response in a casual context. To understand the extent of this problem, we study prosociality in both adversarial and casual dialog contexts and audit the response quality of general-purpose language models in terms of propensity to produce unsafe content. We propose a dual-step fine-tuning process to address these issues using a socially aware n-pair contrastive loss. Subsequently, we train a base model that integrates prosocial behavior by leveraging datasets like Moral Integrity Corpus (MIC) and ProsocialDialog. Experimental results on several dialog datasets demonstrate the effectiveness of our approach in generating socially appropriate responses.

## introduction there is growing concern regarding the potential risks @xcite @xcite of state-of-the-art conversational ai systems. often relying on extensive knowledge @xcite and data-driven approaches, these systems can generate or endorse unsafe, toxic, unethical, rude, or even dangerous content @xcite . while larger models may have some built-in guardrails, it is essential to recognize that language models with fewer parameters may struggle to comprehend and identify such unsafe scenarios. consequently, their ability to respond appropriately and 1 https://github.com/souvikdgp16/contrastive_ dialog_safety mitigate these concerns might be limited. the concern stems from the lack of comprehensive training data and knowledge that can hinder the understanding @xcite and contextual interpretation of potentially unsafe content by smaller pre-trained language models. while these models still possess conversational capabilities @xcite , their limited exposure to a wide range of information may make them less proficient in recognizing and appropriately responding to unsafe statements or scenarios. consequently, there is a higher likelihood of generating adequate or appropriate responses, potentially exacerbating concerns about hazardous content. recently, there have been efforts to develop datasets to teach conversational agents the appropriate social paradigms to respond effectively to unsafe content while maintaining the flow of conversation @xcite @xcite . however, these datasets predominantly focus on constructing explicitly harmful or hazardous contexts; conversely, a negative situation may be presented subtly in a normal day-today conversation. as evident from figure 1 , a model trained on these adversarial datasets produces appropriate responses to obvious negative scenarios, as depicted in case 1 . however, in some hostile instances in which some intervention is required, it might fail to understand the situation and come up with a trivial response, as depicted in case 2 . also, it can exhibit inappropriate behavior in casual contexts by over-generalizing negative patterns(case 3 ) learned in the adversarial data. lastly, the model can fail to comprehend specific scenarios and generate hazardous responses(case 4 ). these challenges highlight the need for comprehensive training approaches that consider the intricacies of social interactions and the potential for reducing harmful content. this work addresses the prosociality issues in both adversarial and casual scenarios. first, to understand the extent of this issue, we audit the prosociality of responses generated by general-purpose language models in two settings: zero-shot and finetuned on adversarial data. in the next step, to circumvent the previously stated concerns, this paper proposes a dual-step fine-tuning process that utilizes adversarial datasets(mic @xcite , prosocialdialog @xcite ) to train a base model and ultimately fine-tune on target casual datasets augmented with rule of thumb(rot). we build on the work of @xcite @xcite to introduce socially-aware aware n-pair contrastive loss used in each finetuning step, which reranks each candidate based on the prosociality level. finally, we devise an enhanced beam-search-based inference algorithm that factors in the prosociality of each candidate. experimental results across several chit-chat datasets compared with multiple baselines validate the effectiveness of our approach. to summarize, we propose the following contributions: figure 2 : model audit results: the chart shows that even when a conversation happens in a casual setting, the chances of producing unsocial content by a language model are significant. ## model generated data audit we fine-tuned foot_0 several general-purpose language models like blenderbot(2.7b), flan-t5-xl(3b), gpt2-xl(1.5b) and , llama2(7b) on prosocialdialog dataset and subsequently on empathetic dialogs dataset. to make the task more challenging, we only considered one previous turn to generate responses during fine-tuning and inference foot_1 . after that, we compared the prosociality levels of 500 responses generated from each model using three settings: (1) zero-shot with casual prompts, (2) fine-tuned with adversarial prompts foot_2 and (3) fine-tuned with casual prompts. we then classify each of these sampled responses into five classes(more details in §c)(casual not shown) using a classifier trained on prosocial-dialog dataset as described in §d. based on the figure 2 , we made the following observations: • as expected, large language models fail to produce socially acceptable responses across many instances in zero-shot settings when prompted with casual prompts. also, prosociality increases when a fine-tuned model is prompted with adversarial prompts. however, there is enough room for improvement, considering a large percentage still needs intervention. ## experimentation we conducted experiments on two fronts. first, we focused on improving prosociality on the base dataset(which contains more negative cases) @xcite using our proposed base fine-tuning process. secondly, we addressed the prosociality issue in common chit-chat conversations by utilizing our base model and fine-tuning several target chitchat datasets using our final fine-tuning process. the details of the datasets are shown in §a. ## conclusion in this work, we study the propensity of generating unsocial content in certain classes of language models. our study aligns with our hypothesis. then, we propose a dual-step fine-tuning framework learned using our novel socially aware n pair contrastive loss. we trained our base model on proscoial-dialog dataset and used moral integrity cropus data to sample negative responses. finally, we train our final models and obtain results for several chit-chat dialog datasets. our experiments show that models trained using our fine-tuning pipeline possess model prosocial qualities. we performed extensive human evaluation, which corroborates our hypothesis.