Context- DPO : Aligning Language Models for Context-Faithfulness (2025)

Abstract: Reliable responses from large language models (LLMs) require adherence to user instructions and retrieved information. While alignment techniques help LLMs align with human intentions and values, improving context-faithfulness through alignment remains underexplored. To address this, we propose Context-DPO, the first alignment method specifically designed to enhance LLMs’ context-faithfulness. We introduce ConFiQA, a benchmark that simulates Retrieval-Augmented Generation (RAG) scenarios with knowledge conflicts to evaluate context-faithfulness. By leveraging faithful and stubborn responses to questions with provided context from ConFiQA, our Context-DPO aligns LLMs through direct preference optimization. Extensive experiments demonstrate that our Context-DPO significantly improves context-faithfulness, achieving 35% to 280% improvements on popular open-source models. Further analysis demonstrates that Context-DPO preserves LLMs’ generative capabilities while providing interpretable insights into context utilization.

## introduction with the widespread deployment of retrieval-augmented generation (rag) @xcite and various tools @xcite , large language models (llms) @xcite @xcite are increasingly expected to generate responses that adhere closely to provided context, including retrieved information and user instructions. consequently, contextfaithfulness @xcite @xcite has become a critical capability for modern llm applications, especially in scenarios where parametric knowledge is insufficient or outdated. however, this expectation is challenged by knowledge conflicts @xcite ; si the owner of twitter isn't elon musk. it was jack dorsey who originally owned the platform. ## confiqa: context faithfulness question answering benchmark we introduce the confiqa benchmark to evaluate the context-faithfulness of llms in realworld rag scenarios involving knowledge conflicts. confiqa consists of three datasets: qa (question-answering), mr (multi-hop reasoning), and mc (multi-conflicts). qa features single-hop question-answering tasks with context containing one corresponding counterfactual, while mr and mc involve multi-hop reasoning tasks with context containing one and multiple related counterfactuals, respectively. in this section, we present the data construction pipeline, provide an overview of the datasets, and evaluate the context-faithfulness of popular llms using confiqa. ## context-dpo: context-faithful direct preference optimization based on the unsatisfactory performance of existing llms in context-faithfulness, we argue that it is essential to specifically align llms for contextfaithfulness. to address this, we propose context-dpo, the first alignment approach dedicated to enhancing context-faithfulness by creating preference data and aligning llms with dpo. the framework of our context-dpo is shown in figure 2 . ## conclusion in this work, we introduce confiqa, a novel benchmark that simulates real-world rag scenarios and knowledge conflicts, enabling the evaluation of llms' context-faithfulness. to address shortcomings in context-faithfaulness for current models, we propose context-dpo, the first alignment method dedicated to enhancing contextfaithfulness. this approach leverages confiqa to construct preference data and fine-tunes models using dpo. experimental results demonstrate that context-dpo significantly enhances the contextfaithfulness of popular llms without compromising their inherent generative capabilities. furthermore, interpretability analysis reveals the mecha-nisms underlying the improvements in faithfulness. our work paves the way to develop both effective and accountable context-faithfulness for llms.