F act A lign: Long-form Factuality Alignment of Large Language Models (2024)

Abstract: Large language models have demonstrated significant potential as the next-generation information access engines. However, their reliability is hindered by issues of hallucination and generating non-factual content. This is particularly problematic in long-form responses, where assessing and ensuring factual accuracy is complex. In this paper, we address this gap by proposing FactAlign, a novel alignment framework designed to enhance the factuality of LLMs’ long-form responses while maintaining their helpfulness. We introduce fKTO, a fine-grained, sentence-level alignment algorithm that extends the Kahneman-Tversky Optimization (KTO) alignment method. Leveraging recent advances in automatic factuality evaluation, FactAlign utilizes fine-grained factuality assessments to guide the alignment process. Our experiments on open-domain prompts and information-seeking questions demonstrate that FactAlign significantly improves the factual accuracy of LLM responses while also improving their helpfulness. Further analyses identify that FactAlign is capable of training LLMs to provide more information without losing factual precision, thus improving the factual F1 score. Our source code, datasets, and trained models are publicly available at https://github.com/MiuLab/FactAlign

## introduction generating natural language provides a natural interface for humans to communicate with artificial intelligence. with the emergence of large language models (llm) @xcite , they immediately demonstrate the potential to become the next-generation engine for information access due to their ability to generate long-form natural language response to human queries. given the largescale pre-training on web-scale datasets, llms demonstrate impressive capabilities of answering diverse questions, showcasing the vast amount of 1 our source code, datasets, and trained models are publicly available at https://github.com/miulab/factalign . the eiffel tower is a tower in paris. it opened in the 20 th century. the tower is 300 meters tall. ## preliminaries in this paper, we aim to improve the long-form factuality of llms by factuality alignment. in this section, we introduce an overview of the task of long-form factuality and alignment algorithms. ## factalign: aligning language models for long-form factuality in this section, we introduce our proposed framework factalign. an overview of our framework is illustrated in figure 2 . ## experimental stetup we conduct experiments to validate the effectiveness of our proposed framework factalign. furthermore, we perform analyses to discuss the effectiveness of each component in the framework. ## results we present the main results in with a detailed examination of the metrics, it is evident that factalign primarily improves factual recall, increasing the output of factual claims from 66.8 to 135.1, while slightly improving fac- ## conclusion in this paper, we address the issue of long-form actuality in llms by proposing a novel alignment framework, factalign. our approach, which incorporates a proposed data construction process alongside the fine-grained alignment algorithm fkto, significantly enhances the factuality of llms over long-form responses, while also boosting their helpfulness. our analysis demonstrates that factalign enables detailed control over the desired level of factual precision and recall scores. we believe that the insights and methodologies presented in our work can motivate further advancements in the factuality alignment of llms.