Adversarial DPO : Harnessing Harmful Data for Reducing Toxicity with Minimal Impact on Coherence and Evasiveness in Dialogue Agents (2024)

Abstract: Recent advancements in open-domain dialogue systems have been propelled by the emergence of high-quality large language models (LLMs) and various effective training methodologies. Nevertheless, the presence of toxicity within these models presents a significant challenge that can potentially diminish the user experience. In this study, we introduce an innovative training algorithm, an improvement upon direct preference optimization (DPO), called adversarial DPO (ADPO). The ADPO algorithm is designed to train models to assign higher probability distributions to preferred responses and lower distributions to unsafe responses, which are self-generated using the toxic control token. We demonstrate that ADPO enhances the model’s resilience against harmful conversations while minimizing performance degradation. Furthermore, we illustrate that ADPO offers a more stable training procedure compared to the traditional DPO. To the best of our knowledge, this is the first adaptation of the DPO algorithm that directly incorporates harmful data into the generative model, thereby reducing the need to artificially create safe dialogue data.

## introduction the enhancement of large language models (llms) has significantly improved the overall performance of major nlp systems @xcite . furthermore, increasing the size of these models not only enhances performance but also enables new capabilities previously unattainable, such as code generation @xcite and applications in medical science @xcite . opendomain dialogue systems have particularly benefited from advancements in llms, with several researchers demonstrating substantial improvements in human preference gained through reinforcement learning from human feedback (rlhf) @xcite . to further enhance the performance of llms, scaling up the model and pre-training dataset size is essential. however, this creates a trade-off between performance and the potential increase in harmful content due to the growth in the size of toxic data within the collected datasets @xcite . numerous studies have demonstrated that many llms possess a non-trivial propensity to generate toxic responses @xcite @xcite , posing significant risks in downstream tasks, especially in dialogue systems. a direct solution to mitigate this issue is using filtered datasets @xcite . however, this approach incurs considerable computational costs and becomes increasingly challenging with larger pre-training datasets. an alternative solution is employing rlhf, which aligns the model with human preferences. nonetheless, @xcite found that rlhf alone does not effectively reduce toxicity. in this research, we introduce an advanced training methodology adversarial dpo (adpo), which builds upon the principles of direct preference optimization (dpo) as proposed by @xcite . the primary aim of adpo is to mitigate the generation of harmful responses by the model, while preserving overall performance. this approach is a progression from the conventional dpo, an algorithm offering stability and competitive performance as an alternative to rlhf. the novelty of adpo lies in its targeted optimization to reduce the generation of toxic responses. we hypothesize that training the model with potential toxic responses within its capability range is more effective than using out-of-scope responses. to achieve this, we fine-tune the model using a dataset of toxic dialogues derived from the bad dataset @xcite , augmented with a toxic control token "[toxic]". this process empowers the model to autonomously generate toxic responses when prompted by the "[toxic]" token. furthermore, we employ an inner toxic model configuration to demonstrate the efficacy of adpo. our results, benchmarked against the baseline model llama2 @xcite , highlight the comparative performance of adpo against standard dpo. these findings underscore the potential of adpo in reducing undesirable outputs in language models while maintaining robust performance metrics. ## related work mitigating toxicity remains a significant challenge in deploying ai for safe and effective human interaction. one prevalent strategy involves filtering inappropriate data, which can be achieved through heuristic rule-based methods or safety detectors such as offensive detection model @xcite . however, as emphasized by @xcite , this filtration process comes with a performance trade-off, highlighting the need to balance filtration levels. achieving this balance can be challenging and often relies on empirical determination. an alternative approach is to append instructions to pre-training data to signal the presence of toxicity @xcite . while these methods can be effective, they entail substantial data processing costs and depend on classifier performance, potentially limiting optimal outcomes. another promising approach involves optimizing the training process, such as rlhf. rlhf has been successfully implemented in models like instructgpt @xcite and sparrow @xcite , aiming to optimize human preferences. this is achieved by replacing actual human rewards with a reward model and aligning ai with human values, a goal that traditional crossentropy loss cannot fully accomplish. however, this approach has limitations, including the extensive human effort required for labeling model responses and the instability and sensitivity to initialization inherent in the proximal policy optimization (ppo) algorithm @xcite . as an advancement or alternative, reinforcement learning from ai feedback (rlaif) has reduced costs by replacing human annotators with llms while maintaining competitive performance compared to rlhf @xcite . dpo has recently emerged (rafailov et al., 2023), transforming rl optimization into supervised training, significantly enhancing stability and reducing computational demands. several llms using dpo have demonstrated impressive results, surpassing some models trained with rlhf. in this paper, we combine these advancements to address the vulnerabilities of rlhf and introduce an additional loss function specifically designed to mitigate inherent toxicity in ai models. ## conclusion in this paper we have concentrated on training opendomain dialogue models while mitigating inherent toxicity. our study introduces adpo, an advanced algorithm of the dpo method, which effectively reduces toxicity levels without compromising dialogue performance. adpo utilizes an internal toxic model, using harmful datasets to enhance safety. this approach enables the model to assimilate both contextual information and safety criteria derived from toxic data. moreover, compared to models trained using dpo, adpo exhibits higher stability during training across a range of hyperparameters, enhancing optimization based on human preferences while penalizing the generation of unsafe responses. to the best of our knowledge, this research represents the first adaptation of the dpo algorithm, uniquely employing unsafe data in generative models to incorporate criteria for harmlessness. in the future, we believe exploring various methodologies for effectively utilizing unsafe data presents a promising avenue for research. although toxic, it contains rich contextual information and can be instrumental in instructing dialogue agents on behaviors to avoid. further advancements in improving both helpfulness and harmfulness is also encouraging. helpfulness and harmfulness sometimes conflict each other @xcite where aiding user may inadvertently result in harmful outcomes. this suggests that models should be trained to discern when to appropriately decline a request based on the context, rather than being constantly positive.