Training Language Models to Generate Text with Citations via Fine-grained Rewards (2024)

Abstract: While recent Large Language Models (LLMs) have proven useful in answering user queries, they are prone to hallucination, and their responses often lack credibility due to missing references to reliable sources. An intuitive solution to these issues would be to include in-text citations referring to external documents as evidence. While previous works have directly prompted LLMs to generate in-text citations, their performances are far from satisfactory, especially when it comes to smaller LLMs. In this work, we propose an effective training framework using fine-grained rewards to teach LLMs to generate highly supportive and relevant citations, while ensuring the correctness of their responses. We also conduct a systematic analysis of applying these fine-grained rewards to common LLM training strategies, demonstrating its advantage over conventional practices. We conduct extensive experiments on Question Answering (QA) datasets taken from the ALCE benchmark and validate the model’s generalizability using EXPERTQA. On LLaMA-2-7B, the incorporation of fine-grained rewards achieves the best performance among the baselines, even surpassing that of GPT-3.5-turbo.

## introduction despite the recent success of large language models (llms), their outputs often contain incorrect or unverifiable claims @xcite . this reduces the usefulness of their generated content, and more importantly, hurts the user trust in llms. previous work @xcite @xcite attempts to address this issue with retrieval-augmented generation (i.e., rag), which makes the llm condition its generation on information retrieved from an external knowledge source for the task input. however, rag cannot fully solve the issue because there is no guarantee 1 our code and data is available at https://github.com/ hcy123902/atg-w-fg-rw . ## problem definition and methods we follow @xcite to define attributable text generation as: given a query x and a passage corpus d, generate a textual output y that consists of n sentences s 1 , s 2 , . . . , s n and citations (such as @xcite , @xcite , etc.) that refer to passages c i ⊆ d to provide evidence for each sentence s i . in this section, we introduce how to train an lm with fine-grained rewards to perform the task. as the corpus d usually contains millions of passages, we first retrieve a small set (e.g., top-5) of relevant passages as the first step before training an lm to generate the output, conditioning on the query and retrieved passages. the details of the retriever for each dataset are in appendix b.2. figure 2 illustrates our training process. as supervision for in-text citations is hard to obtain, we first initialize an lm with distillation from a proprietary lm such as chatgpt, as described in § 2.1. we then introduce how fine-grained rewards are computed and used to train the lm with rl or rs in § 2.2. finally, we describe how to train the lm with holistic rewards for comparison purposes in § 2.3. @xcite observe that the performance of open-sourced lms like llama-2 for attributable generation is far from satisfactory. in particular, we notice that even with in-context learning, a base llama-2-7b can only generate citations for a very small portion of sentences in its responses. as previous studies @xcite often perform rs or rl training on top of a reasonably good initial lm as a starting point, we first learn a better lm capable of generating in-text citations by distilling knowledge from a powerful proprietary model such as chatgpt foot_0 . for each question in the training set, we prompt chat-gpt with in-context demonstrations @xcite to generate answers that contain citations (see appendix i). these generated answers serve as weak supervisions to finetune our lm. we use ## results and analysis we first compare all the systems ( § 4.1), and then conduct ablation study on the training effect of each reward type ( § 4.2). we also analyze the effect of retrieval on the final model performance ( § 4.3), and the citation errors that we observe ( § 4.4). lastly, we test our models' generalization capability on expertqa that is unseen during training ( § 4.5). ## conclusion and future directions in this work, we systematically analyze common llm training methods for attributable text generation. we show that training with fine-grained rewards gives the strongest performance boost and it also enables small llms to outperform chatgpt. we propose a number of directions to explore. firstly, our retrieval analysis shows room to further improve llms' correctness recall on qa datasets, and more work can be done to enhance llms' reading comprehension and synthesis ability given the retrieved passages. secondly, according to touvron et al. (2023), iteratively applying imitation learning and reinforcement learning for multiple rounds can continuously increase the model's performance. while this is not the main focus of our work, it will be intriguing to see how far a smaller llm can go in our task setting using this approach.