B ias DPO : Mitigating Bias in Language Models through Direct Preference Optimization (2024)

Abstract: Large Language Models (LLMs) have become pivotal in advancing natural language processing, yet their potential to perpetuate biases poses significant concerns. This paper introduces a new framework employing Direct Preference Optimization (DPO) to mitigate gender, racial, and religious biases in LLM-generated English text. By developing a loss function that favors less biased over biased completions, our approach cultivates a preference for respectful and non-discriminatory language in LLMs. We also contribute a manually designed dataset for training LLMs to recognize and correct biases. This dataset encompasses a diverse range of prompts paired with both biased and unbiased completions. Implementing this approach on the Microsoft Phi-2 model, we demonstrate substantial reductions in biased outputs as our model outperforms the baseline model on almost all bias benchmarks. Our model also achieves better performance compared to other open-source models on most benchmarks. By reducing biases in the language generated by the model, our study marks a significant step towards developing more ethical and socially responsible LLMs. We publicly release BiasDPO dataset on HuggingFace.

## introduction even though large language models (llms) have shown remarkable capabilities in complex language tasks, they are not without their flaws. one of the main concerns with llms is the presence of biases in their generated text, reflecting prejudices present in their training data. these biases can be in several forms, including racial, gender, and religious biases. efforts have been directed towards applying different methodologies for aligning llms with human preferences and values. one of the most popular approaches used is reinforcement learning from human feedback (rlhf), which trains llms to generate responses that are more likely to be rated highly by human evaluators @xcite . however, rlhf faces several challenges, such as mode collapse, training instability, as well as requiring a separate reward model which adds complexity to the training process @xcite . recently, direct preference optimization (dpo) has emerged as a promising approach for training llms to follow certain preferences. dpo works by training the model to maximize the log probability of preferred tokens and minimize the log probability of dispreferred tokens given a certain prompt from the dataset @xcite . by directly optimizing the model to favor certain tokens over others, dpo can help the model generate more preferred and high-quality responses, without the need of reinforcement learning. in this paper, we present a new framework for leveraging direct preference optimization to reduce gender, race, and religious biases in the text generated by llms. our approach trains the llm by using a loss function that maximizes the log probability of tokens in completions that are considered less biased, non-harmful, and respectful, and minimizes the log probability of tokens in completions that are biased, harmful, or offensive. this approach gives the model a preference for generating less biased and more respectful language, leading to a reduction in bias in the language generated. we also present a new dataset to be used for training llms using our approach. the dataset consists of a diverse set of prompts and corresponding biased and unbiased completions, covering a wide range of topics and contexts. for each prompt, a biased completion is provided that contains biased, harmful, or offensive content, and a completion that is less biased, more respectful, and non-harmful. by applying our training approach using our dataset to the recently released microsoft phi-2 model, results indicate that our approach reduces bias in the language generated by the llm when tested both quantitatively and qualitatively. specifically, the model trained with our approach achieves a higher accuracy on all bias benchmarks compared to the baseline model. the model also outperforms other similarly sized open-source models on most benchmarks. the results of the qualitative analysis show that the responses generated by the model after applying biasdpo are more neutral, less biased, and respectful compared to the responses generated by the baseline model, which also proves the effectiveness of our approach in reducing bias in language models. ## experiments 4.1 experimental design to apply and test our approach, we use microsoft phi-2 as the base model to be trained. phi-2 is a recently released 2.7b parameter open-source llm that demonstrates state-of-the-art performance on a wide range of language tasks compared to other models in its size range. phi-2 is trained following the "textbooks are all you need" approach @xcite , which allows it to achieve high performance on tasks such as common sense, language understanding, and logical reasoning. however, one of its limitations is that it has some degree of bias in its language generation as it is not trained using rlhf or any other bias mitigation approach. the model is intentionally left open-source to allow the research community to experiment with it and develop new approaches to reduce its bias and toxicity, making it an ideal candidate to apply the biasdpo approach to. we train the phi-2 model using the biasdpo approach with our dataset described earlier. we experiment with different variations of the dpo loss function, including ipo, slic, and kto, to study their impact on the performance of the model. we also experiment with different values of the hyperparameter β. the model is trained for 5 epochs using the adam optimizer with a learning rate of 1e-6, and a batch size of 4 on an 8 v100 gpus server. ## conclusion in this paper, we introduced biasdpo, a new approach designed to reduce bias in language models through direct preference optimization. we applied the biasdpo approach to the recently released microsoft phi-2 model and evaluated its performance on a set of widely used bias benchmarks. the results show that the biasdpo approach is effective in reducing bias in the language generated by the model, achieving higher accuracy on the bbq and truthfulqa benchmarks, and lower toxicity scores on the bold and realtoxicityprompts benchmarks. the qualitative analysis further con-firms the effectiveness of the biasdpo approach in reducing bias in the language generated by the model, resulting in more fair, respectful, and less biased responses. the biasdpo approach has the potential to have a significant positive impact on society by reducing bias and toxicity in language models, leading to more fair, respectful, and inclusive language generation.