Advancing Large Language Model Attribution through Self-Improving (2024)

Abstract: Teaching large language models (LLMs) to generate text with citations to evidence sources can mitigate hallucinations and enhance verifiability in information-seeking systems. However, improving this capability requires high-quality attribution data, which is costly and labor-intensive. Inspired by recent advances in self-improvement that enhance LLMs without manual annotation, we present START, a Self-Taught AttRibuTion framework for iteratively improving the attribution capability of LLMs. First, to prevent models from stagnating due to initially insufficient supervision signals, START leverages the model to self-construct synthetic training data for warming up. To further self-improve the model’s attribution ability, START iteratively utilizes fine-grained preference supervision signals constructed from its sampled responses to encourage robust, comprehensive, and attributable generation. Experiments on three open-domain question-answering datasets, covering long-form QA and multi-step reasoning, demonstrate significant performance gains of 25.13% on average without relying on human annotations and more advanced models. Further analysis reveals that START excels in aggregating information across multiple sources.

## introduction the rapid development of large language models (llms) @xcite has led to their prosperity as indispensable tools for information seeking. despite their remarkable capability to generate fluent and informative responses to user queries, llms also struggle with hallucinations @xcite . to facilitate factuality verification, recent research @xcite has explored attributed text generation, a paradigm that enables llms to generate responses with citations. by attributing models' output to verifiable sources, it can improve the explainability and credibility of llm-generated content @xcite . while beneficial, the ability to attribute contextual sources is not inherent in llms. most work induces llms to generate text with citations via in-context learning @xcite , which is far from satisfactory @xcite . the current winning recipe for accurate attribution involves fine-tuning on high-quality attribution responses foot_0 @xcite . however, acquiring such data typically requires either manual curation @xcite , or distilled from the most advanced llms @xcite , both of which are costly and not scalable, thus limiting the growth of models' attribution capability. one promising solution is self-improvement @xcite , which has demonstrated the potential to boost model performance by learning from self-generated high-quality samples. inspired by this, we aim to explore the potential of self-improvement in bootstrapping the attribution ability of llms. however, achieving this goal presents several challenges. one significant challenge lies in the risk of model stagnation during the self-improvement process, primarily due to the insufficient supervision signals obtained in the early stage. concretely, considering the inferior performance of llms in handling the attribution task @xcite , generating sufficient high-quality attribution responses solely through sampling proves difficult. this scarcity of high-quality samples limits the opportunities for llms to self-improve effectively. another challenge stems from the limitation of weak supervision signals. current self-improvement approaches @xcite primarily involve supervised fine-tuning on high-quality samples while discarding low-quality ones. when applied to llm attribu-tion, these high-quality samples provide only weak supervision signals, mainly teaching llms on the surface form of attribution (e.g., proper citation format) @xcite . such practice may neglect the potential of exploring fine-grained signals from low-quality samples to learn what constitutes a desirable attribution response. to address these challenges, we present start, a self-taught attribution framework designed to bootstrap the attribution capabilities of llms. to prevent models from stagnating early due to insufficient supervision signals, we first leverage the model to self-construct high-quality synthetic attribution data ( §3.1). the data synthesis process follows reverse attribution thinking: the model initially generates a response to a given query, then breaks it into atomic claims, and finally randomly combines them to create synthetic documents. this process not only simulates multisource information-seeking scenarios but also ensures precise attribution, as each document can be directly traced back to the specific claim it originated from. these high-quality synthetic data are then utilized for warming up, providing a good starting point for llms to self-improve. furthermore, to better explore fine-grained supervision signals for llm attribution, we introduce an iterative self-improving recipe ( §3.2). specifically, the framework meticulously designs fine-grained rewards tailored for llm attribution, covering robustness, comprehensiveness, and attributability. by scoring multiple candidates through sampling and selecting those with the highest holistic rewards for supervised fine-tuning, the framework subsequently utilizes low-quality samples to construct fine-grained preference pairs with diverse optimization rewards for preference optimization. this iterative process further fosters the self-improvement of attribution capabilities. we conduct extensive experiments across three open-domain question-answering datasets, covering long-form qa and multi-step reasoning. results indicate that start achieves significant performance gains of 25.13% on average in citation quality. moreover, start successfully achieves self-improvement in llm attribution, showing progressive improvements across iterations. ablation studies confirm that each component significantly contributes to the improvement. further analysis shows that start not only excels in generating superior attributable responses but also in effectively aggregating information across multiple sources. ## problem formulation and methodology we follow a formulation of attributed text generation as described in @xcite . this task involves processing a user query q for informationseeking, given a corpus of retrieved documents d, to generate a response s with in-line citations. we assume the response s as consisting of n statements, such that @xmath0, s 2 , . . . , s n }. each statement s @xmath1, c i2 , . . .}, where c @xmath2. citations are presented in the form of [1][2], which represent the attribution to specific documents in d. next, we present an overview of start, a training framework designed to teach llms to selfimprove their attribution ability, as illustrated in figure 2 . start consists of two essential stages: synthetic data warm-up ( §3.1) and self-improving for llm attribution ( §3.2). ## human evaluation human evaluation results, detailed in ## conclusion we propose start, a self-improvement framework to push the frontier of llm attribution. we iden-tify two key limitations for llm attribution selfimprovement. to address these, start first leverages self-constructed synthetic data for warming up, aiming to prevent models from early stagnation due to insufficient supervision signals. to explore more fine-grained supervision signals, start constructs fine-grained preference supervision signals from low-quality samples for preference optimization. both automatic and human evaluations demonstrate significant improvement in attribution without relying on human annotations and more advanced llms.