Negating Negatives: Alignment with Human Negative Samples via Distributional Dispreference Optimization (2024)

Abstract: Large language models (LLMs) have revolutionized the role of AI, yet pose potential social risks. To steer LLMs towards human preference, alignment technologies have been introduced and gained increasing attention. Nevertheless, existing methods heavily rely on high-quality positive-negative training pairs, suffering from noisy positive responses that are barely distinguishable from negative ones. Given recent LLMs’ proficiency in generating helpful responses, this work pivots towards a new research question: **can we achieve alignment using solely human-annotated negative samples, preserving helpfulness while reducing harmfulness?** For this purpose, we propose Distributional Dispreference Optimization (D ^2 O), which maximizes the discrepancy between dispreferred responses and the generated non-negative ones. In this way, D ^2 O effectively eschews harmful information without incorporating noisy positive samples, while avoiding collapse using self-generated responses as anchors. We demonstrate that D ^2 O can be regarded as learning a distributional preference model reflecting human dispreference against negative responses, which is theoretically an upper bound of the instance-level DPO. Extensive experiments manifest that our method achieves comparable generation quality and surpasses the latest strong baselines in producing less harmful and more informative responses with better training stability and faster convergence.

## introduction the past two years have witnessed the meteoric rise of large language models (llms) @xcite @xcite , showcasing surprising capabilities of empowering diverse real-world applications. nevertheless, llms' deep integration into human society also brings potential risks, e.g., the propagation of * corresponding authors: p. zhang and x. yi harmlessness of positive and negative samples in the hh dataset scored by the rewarder in @xcite . (b) win rate of positive samples relative to negatives judged by gpt-4. (c) proportion of toxic samples in the positive ones evaluated by the classifier in @xcite . more analyses are in appendix. a. despite the significant progress, alignment methods heavily rely on carefully curated human preference data @xcite , formed as positive (preferred) and negative (dispreferred) response pairs. however, it's challenging to construct high-quality positive samples due to desiderata ambiguity @xcite , resulting in noisy preference labels @xcite . as shown in fig. 1 , we observe a marginal difference in harmlessness between positive and negative samples, and alarmingly non-negligible toxicity among the preferred responses in datasets. numerous studies have also highlighted relatively low annotator agreement rates in labeled data @xcite @xcite . this impedes alignment optimization would reinforce harmful content. given that the negative sam-ples can be identified and collected more readily by humans @xcite , and the helpfulness can be achieved more easily @xcite , we highlight a new research focus: alignment with solely human-labeled negative samples, aiming to maintain helpfulness while reducing harmfulness. nonetheless, simply minimizing the generation probability of negatives leads to severe catastrophic unlearning, rendering llms useless @xcite . to address these problems, we propose a novel distributional dispreference optimization (d 2 o). unlike previous methods that optimize an instancelevel preference model, d 2 o maximizes the discrepancy between self-generated responses progressively sampled during training and the negative ones, without any external reward or label signals. this approach can be considered as optimizing a distribution-level bradley-terry preference model over the llm policy and the distribution of negative samples, effectively reflecting human dispreference. furthermore, we demonstrate that d 2 o's preference model theoretically upper bounds that of dpo @xcite , which better constrains the deviation from the original policy and encourages exploration. in this process, llms simultaneously learn to circumvent negative samples (forgetting) and to fit the previously learned policy (exploitation), with self-generated samples as anchors, thereby minimizing the harmfulness of generated responses while alleviating catastrophic unlearning. besides, the distributional optimization effectively dilutes noises and stabilizes training. in summary, our contributions are as follows: llm unlearning relevant to our work, machine unlearning @xcite aims to remove undesired information from learned ai models, which previously concentrates on the computer vision (cv) area to handle privacy, copyright and safety issues @xcite . with the flourishing of llms that also encompass sensitive information @xcite , efforts have been made to erase privacy, biased and toxic content internalized from training data @xcite @xcite @xcite . more recently, unlearning has also been directly adopted for llm alignment to avoid generating detrimental content @xcite . however, the challenge lies in that the performance of unlearned models will exponentially degrade with more erased data @xcite . positive-unlabelled learning (pul) another relevant area is pul, a semi-supervised learning (ssl) paradigm for classification tasks with only positive samples and the unlabelled ones @xcite @xcite , which is necessary when negative labels are expensive or unavailable, e.g., in outlier detection and medical diagnosis. pul usually uses heuristic strategies to identify reliable negative data or learns generative models to generate them @xcite . similar to pul, as a special technique of contrastive learning @xcite , negative sampling (ns) selects informative negatives or generates synthetic ones for a given positive sample to learn better representation and reduce computational cost @xcite , which has been applied to various domains including information retrieval (ir) @xcite , computer vision @xcite , natural language processing (nlp) @xcite , and graph learning @xcite . in spite of certain relevance, pul and ns cannot be directly applied to llm alignment for three reasons. (1) the lack of heuristic strategies, e.g., popularity-biased sampling @xcite and external supervisory signals, e.g., propensity scores @xcite , making it infeasible to identify positive responses. (2) unlike pul and ns, alignment centers on generative rather than discriminative tasks, notably lacking positive instead of negative samples. (3) due to the ambiguity of human values @xcite and the difficulty in desire expressing @xcite , it's hard to recognize positive samples heuristically. therefore, to handle these challenges, we propose a novel d 2 o method for llm alignment. differing from aif-based methods or pul, our method operates without external reward signals for identifying or ranking positive samples, but directly contrasts the llm distribution with negative samples to eliminate harmful information. ## conclusion and future work in this work, we highlight an innovative focus of llm alignment: alignment with solely humanlabeled negatives to handle label noise. we propose d 2 o as a viable solution that theoretically learns a distributional preference model that effectively negates negatives. empirical evidence manifests d 2 o's superiority in reducing harmfulness, maintaining helpfulness, and achieving better stability and faster convergence. in the future, we plan to extend our method to explicit reward modelling and rlhf and further reduce alignment tax.