A Grounded Preference Model for LLM Alignment (2024)

Abstract: Despite LLMs’ recent advancements, they still suffer from factual inconsistency and hallucination. An often-opted remedy is retrieval-augmented generation – however, there is no guarantee that the model will strictly adhere to retrieved grounding. Fundamentally, LLMs need to be aligned to be more faithful to grounding, which will require high-quality preference annotations. This paper investigates whether we can create high-quality grounded preference data for model alignment without using annotations from humans or large proprietary models. We experimented with existing entailment data and proposed approaches to generate synthetic grounded preference data, with which we train a Grounded Preference Model(GPM). We demonstrate through Proximal Policy Optimization(PPO) training of Mistral-7B-Instruct that our GPM model can successfully align powerful LLMs to generate much better grounded responses as judged by GPT4. Moreover, we show that our GPM is also a great faithfulness classifier, achieving SoTA in dialogue sub-tasks of the TRUE faithfulness Benchmark. We will release our GPM under the Apache 2.0 license.

## introduction large language models (llms) have seen rapid advancements, yet they continue to suffer hallucinations in both open-domain and grounded generations @xcite . this undermines the usability of llms for high-stake applications. to address these challenges, we explore model alignment with reinforcement learning (rl) to emulate human preferences in model outputs @xcite @xcite . more specifically, we propose grounded preference model (gpm) -a model trained to assess the overall quality of grounded responses, and leverage it to align llms toward more faithful generation. training a preference model traditionally demands substantial human annotation, which is expensive and labor-intensive. moreover, there are very few publicly available preference datasets for content-grounded dialogues. our proposed method simplifies this process by utilizing existing entailment datasets combined with synthetically generated preference data for model alignment. specifically, we explore ways to reliably generate/curate preference pairs for queries from existing contentgrounded datasets. we demonstrate the effectiveness of gpm for model alignment on a leading llm, mistral-7b-instruct-v0.1 foot_1 @xcite . the correctness and helpfulness of mistral responses, as judged by gpt4, improves after ppo training for all variants of gpm. moreover, we evaluate gpm as a faithfulness metric on true benchmark. our model performs comparably to the sota model on average, with clear gains in the dialogue sub-taskestablishing a new sota for the task. ## gpm for llm alignment we use the standard rlhf procedure @xcite for model alignment that optimizes: ## gpm as faithfulness metric while gpm combines preference and groundedness, it should preserve its ability to discern unfaithful responses and remain a reliable faithfulness metric. we evaluate gpm 's performance on the true benchmark @xcite designed to comprehensively evaluate faithfulness across 11 datasets from 4 tasks. we report performance on dialogue sub-task as well as overall scores. we use roc-auc as our evaluation metric @xcite . we compare variants of gpm against two top-performing models: t5-11b fine-tuned on anli @xcite , and wecheck @xcite . ## related work various approaches have been proposed to make llms more reliable. prompting-based methods prompt with counterfactual demonstrations @xcite or employ chain-of-thought selfverification @xcite . prefix tuning @xcite ; tunes model's system message on a synthetic task where hallucinations are easy to identify and then transfer them to abstractive summarization tasks. on the other hand, there are rl-based approaches that use automatic metrics to reward faithful generation. du and ji (2023) employ sacrebleu and bertscore as reward signals, while roit et al. (2023) use log-probabilities from the anli-classifier (nie et al., 2020). ; unlike previous works that rely on the existing automatic metrics of faithfulnes, our major contribution is a preference model trained specifically for the grounded generation tasks. a significant body of work models faithfulness as a classification task: nie et al. (2020), wu et al. (2023) and gekhman @xcite . our work is set apart in its modeling approach as well as in its ability to leverage relative preferences, obviating the need for hard labels. ## conclusions this paper investigates the recipe for grounded preference alignment. we find that entailment data can be repurposed to train good grounded preference models, which align base policy towards faithful generation. moreover, by adding synthetic preference data, we are able to train gpm that not only achieves new sota in faithful, but also serves as a reward model for llm alignment. our approach is simple and has no reliance on proprietary ais.