Improving Attributed Text Generation of Large Language Models via Preference Learning (2024)

Abstract: Large language models have been widely adopted in natural language processing, yet they face the challenge of generating unreliable content. Recent works aim to reduce misinformation and hallucinations by resorting to attribution as a means to provide evidence (i.e., citations). However, current attribution methods usually focus on the retrieval stage and automatic evaluation that neglect mirroring the citation mechanisms in human scholarly writing to bolster credibility. In this paper, we address these challenges by modelling the attribution task as preference learning and introducing an Automatic Preference Optimization (APO) framework. First, we create a curated collection for post-training with 6,330 examples by collecting and filtering from existing datasets. Second, considering the high cost of labelling preference data, we further propose an automatic method to synthesize attribution preference data resulting in 95,263 pairs. Moreover, inspired by the human citation process, we further propose a progressive preference optimization method by leveraging fine-grained information. Extensive experiments on three datasets (i.e., ASQA, StrategyQA, and ELI5) demonstrate that APO achieves state-of-the-art citation F1 with higher answer quality.

## introduction large language models (llms) have demonstrated emergent abilities and have gained widespread application in natural language processing (nlp) @xcite @xcite . for example, llms have shown remarkable in-context learning capabilities across a variety of domains and tasks @xcite . although llms have been widely adopted, a prominent issue is that they produce hallucinations in certain situations @xcite . in other words, they generate information that sounds plausible but is nonfactual, thereby limiting their applicability in the real world. to mitigate hallucinations, researchers have resorted to grounding statements in responses generated by llms to supported evidence, either by providing rationales or by adding citations to the statements @xcite . recent works have utilized external knowledge sources such as retrieved documents and knowledge graphs for attribution @xcite . generally, these works are divided into two types: 1) the model generates an answer with citations based on the retrieved documents @xcite ; 2) an answer is first generated, then modified again to add attribution references by retrieving with query and initial answer @xcite . however, these works focus mainly on the retrieval stage @xcite and the evaluation process @xcite . considering the selection of the model's desired responses and behavior from its very broad knowledge and capabilities, it is more necessary to optimize the generation process, not only reducing the hallucination of the original answer but also avoiding the hallucination of the attribution process. on the other hand, fine-tuning llms after pre-training can also significantly improve performance for users' downstream tasks. first, given positive examples of correct behavior, supervised fine-tuning can be performed using standard likelihood-based training. secondly, given positive and negative examples (binary feedback or pairwise feedback), methods such as unlikelihood training on negative examples @xcite or rlhf-ppo @xcite can be used for learning. however, these methods usually suffer from expensive data collection process, reward model training, sparse reward and text degeneration problems, making them difficult to use in practical applications @xcite . in this paper, inspired by the citation mechanisms in human scholarly writing @xcite , we address these challenges by conceptualizing the attribution task for llms as preference learning and proposing an automatic preference optimization (apo) framework, as shown in figure 1 . initially, we assemble a curated dataset comprising 6,330 examples sourced and refined from existing datasets for posttraining. this step makes the llms know the basic format and requirements of attribution. considering the substantial cost and extremely timeconsuming of preference pair annotations, we thus introduce an automated approach to generate attribution preference data, yielding 95,263 pairs. furthermore, drawing inspiration from the human process of citation and direct preference optimization @xcite , we propose a progressive preference optimization method with experience replay bypassing the need for explicit reward modeling or reinforcement learning. we conduct the extensive experiment on three datasets (i.e., asqa, strategyqa, and eli5). the experiment results demonstrate that apo surpasses compared baselines across all datasets with improved citation f1 along with higher response quality. our contributions are summarized as follows: the "2_chosen" means it's the second statement in the answer that be selected to construct preference pairs. we show apo in more detail in figure 2 . bohnet et al. (2022) adapt the ais framework for qa scenarios. further, gao et al. (2023b) extrapolate ais to evaluate generated text of llms with citations. additionally, several works focus on building and using automated ais evaluations @xcite @xcite . for a comprehensive overview, please refer to @xcite . in contrast to existing approaches, our work broadens the scope of attribution beyond just verifiable text generation and devises a methodology to enhance these attributions which frames it as a preference learning problem. ## preliminary the main pipeline of preference learning usually consists of: 1) pretraining and supervised fine-tuning (sft), where sft is not a must; 2) preference data collection; 3) preference optimization. pretraining and sft phase preference learning typically starts with a pretrained llms or llms fine-tuned on high-quality data using maximum likelihood estimation. the final policy π ref after this phase is represented as preference data collection phase after pretraining and sft phase, π ref is prompted by context x, and generate two responses y w , y l ∼ π ref (•|x). then x, y w , y l is labeled by humans to judge which response is preferred and denote y w ≻ y l |x if y w is preferred, and y l ≻ y w |x if y l is preferred. we define a new symbol @xmath0, and @xmath1, y w , y l , @xmath2: ⟨x, y w , y l , i⟩ ∼ d p . preference optimization phase in the final phase, the prevailing method uses reinforcement learning algorithm to learn an explicit or implicit reward from the preference data, and then using on-policy or off-policy policy gradient algorithm to maximize the reward. recently, some methods have derived the optimal policy using reward maximization under kl-regularization and also derive a loss with optimal policy as its solution, then learn the optimal policy by minimizing the derived loss on empirical dataset. ## conclusion this paper introduces the apo framework for attributed text generation. we treat attribution as a preference learning task, utilizing curated posttraining collections and an automated synthesis algorithm to reduce manual labeling costs. experiments on three datasets demonstrate the effectiveness of apo which achieves leading citation f1 and improved response quality. future work can explore extending apo to real-world applications.