Aligning as Debiasing: Causality-Aware Alignment via Reinforcement Learning with Interventional Feedback (2024)

Abstract: Large language models (LLMs) often generate biased outputs containing offensive, toxic, or stereotypical text. Existing LLM alignment methods such as reinforcement learning from human feedback (RLHF) alleviate biases primarily based on reward signals from current model outputs without considering the source of biases. In this work, to explore how biases are formed, we revisit LLMs’ text generation from a causal perspective. We identify pretraining data and input prompts, which contain semantic correlations of textual phrases, as two confounders between LLMs and model outputs causing biases. Inspired by our causal view, we leverage the reward model in RL alignment as an instrumental variable to perform causal intervention on LLMs. Utilizing the reward difference between an initial LLM and intervened LLM as interventional feedback to guide RL finetuning, we propose Causality-Aware Alignment (CAA) for LLM debiasing. Experiments on two text generation tasks with three different alignment objectives demonstrate the advantages of our method in aligning LLMs to generate less biased and safer outputs.

## introduction large language models (llms) @xcite @xcite have demonstrated remarkable proficiency in generating fluent texts while also reflecting biases @xcite . recent studies on reducing llms' biased outputs, e.g., offensive, toxic, and stereotypical text generations @xcite , have incorporated human feedback into the finetuning process to align llms with human values @xcite @xcite . many adopt the reinforcement learning from human feedback (rlhf) @xcite framework, where a reward model trained from human preferences is used to provide training signals for the rl finetuning of llms. for example, given a toxic model output, e.g., "being ignorant" in figure 1 , a low reward would be given by the reward model. then, the parameters of the llm are updated by rl such that it generates outputs that could receive a higher reward, which are expected to be safer and more friendly @xcite . while optimizing the model parameters based on rewards from current model outputs has shown promising results in debiasing llms, existing rl alignment methods tend to overlook how such biases could be formed. as shown in figure 1 , llms' toxic output can result from the semantic correlations of text phrases in pretraining data @xcite . specifically, when phrases indicating high certainty, e.g., "pretty sure" often co-occur with toxic phrases, e.g., "ill-advised" in pretraining data, the high frequency of co-occurrence forms a semantic correlation between these two kinds of phrases, which is learned by the llm during pretraining. for rest of the paper, we use pretraining and training interchangeably for simplicity. then, when prompted with a semantically similar phrase indicating high certainty, e.g., "99 percent sure" as shown in figure 1 , the llm would tend to generate toxic outputs, e.g., "ignorant", even though the input prompt may not contain any toxic content. without considering how such biases are formed, existing alignment methods such as rlhf are limited in exploring better policies for llms that lead to less biased and safer text generations @xcite . in this work, to better understand such biases, we revisit llms' text generation from a causal perspective. with our designed structural causal model, we identify training data and input prompt as two confounders between llms and model outputs. specifically, when training data or input prompt exhibit biased semantic correlations between certain text phrases, e.g., high certainty and high toxicity phrases, such correlations influence both the llm and model output during training and inference, resulting in confounding effects. as a result, even though the llm may not intend to generate toxic text given a non-toxic input prompt as shown in figure 1 , the likelihood of observing toxic model outputs increases because of the confounding effects in addition to the original causal effects between the llm and model outputs. inspired by our causal view, we propose to align llms for debiasing through causal intervention, aiming to alleviate confounding effects. while existing rlhf methods may be considered as a form of interventions on llms @xcite through parameter updates, these methods are not designed with the awareness of causal relations in text generation and thus may not perform effective interventions that alleviate biases. therefore, to further improve existing alignment methods for llm debiasing, we leverage the reward model in rl finetuning as an instrumental variable providing interventional feedback, which is based on the reward difference between initial llm outputs and intervened llm outputs. by adaptively weighting biased samples based on interventional feedback during rl finetuning, we propose causality-aware alignment (caa) for llm debiasing. experiments on two text generation tasks, including positive-sentiment and detoxified text continuation, and debiased text summarization, demonstrate the advantages of our method in aligning llms to generated less biased outputs. in summary, we make the following contributions: ## methodology in this section, we first revisit llms' text generation from a causal perspective to understand potential formation of llms' biases in section 3.1. inspired by our causal view, we further propose causality-aware alignment in section 3.2, leveraging reward model for causal intervention. we instantiate causality-aware alignment via rl with interventional feedback in section 3.3 ## conclusion in this work, we revisit llm text generation from a causal perspective, identifying training data and input prompts as two confounders causing biases. inspired by our causal view and to align llms to generate less biased outputs, we leverage the reward model in rl alignment as an instrumental variable to provide interventional feedback. by adaptively weighting biased samples, we propose causality-aware alignment via rl with interventional feedback. extensive experiments across multiple text generation tasks demonstrate the effectiveness of our method in debiasing llms.