Evaluating Design Choices in Verifiable Generation with Open-source Models (2025)

Abstract: Verifiable generation is introduced to improve the transparency and trustworthiness of outputs produced by large language models (LLMs). Recent studies observe that open-source models struggle to include accurate citations to supporting documents in their generation with in-context learning, in contrast to the strong performance demonstrated by proprietary models. Our work aims to reveal the critical design choices that can benefit open-source models, including generation pipelines, fine-tuning methods, and inference-time compute techniques.We consider three generation pipelines, producing the outputs directly or decomposing the generation into subtasks.These generation pipelines are fine-tuned using supervised fine-tuning and preference-based optimization including further fine-tuning with rejection sampling data and direct preference optimization (DPO).The construction of preference data with varying content and citation diversity is also investigated.Additionally, we examine the benefit of an additional reranking step. With four open-source models, our experiments show that directly generating the outputs achieves the best performance. Compared to other fine-tuning methods, DPO that computes training signals from contrastive pairs consistently yields better performance, and it reaches the peak performance when the contrastive pairs are constructed with sufficient content diversity.We also find that reranking can further boost the performance of verifiable generation systems, but the marginal improvement might not justify the additional cost.

## introduction verifiable generation, a generation paradigm where large language models (llms) are required to produce outputs along with citations to supporting documents, has gained increased attention for its potential to enhance user trust in the model responses @xcite . by allowing users to verify the generated content against cited sources, this approach not only enhances reliability but also facilitates access to additional relevant information. the paradigm has been incorporated into online services like google and bing chat that are powered by proprietary models such as gemini @xcite and @xcite . ## verifiable generation in this section, we first introduce the candidate pipelines for verifiable generation ( §3.1). following the introduction of these pipelines, we discuss the strategies for fine-tuning models to enhance their performance and the methods for collecting training samples ( §3.2). lastly, we investigate the techniques that leverage inference-time compute ( §3.3). task formulation. we adhere to the task formulation outlined by @xcite . specifically, a system is given a query q and a set of candidate cited sources @xmath0, . . . , d m }, where m denotes the total number of candidate cited sources. each cited source d i can be either a text passage or an entire document, depending on the dataset. to process the lengthy aggregation of d, we provide each system with summarized versions of the documents. we leave the exploration of long-context processing techniques to future research, as using summarized documents achieves comparable performance to enabling truncation or more sophisticated methods such as interactive lookup of full documents @xcite . typically, verifiable generation systems indicate citations in their outputs with square brackets that enclose indices of cited sources (e.g., @xcite ). we denote the system output as @xmath1, . . . , y l ] and define this output format by treating y i as a tuple comprising a text token and a set of indices c @xmath2,1 , . . . }, which point to the supporting documents. l represents the total number of text ## conclusions in the development of verifiable generation systems, including generation pipelines and optimization methods. three generation pipelines are investigated: direct generation that outputs responses with citations in one pass; decomposed generation that connects a content generator with a citation generator to produce outputs in two steps; joint generation that combines the aforementioned pipelines. we conduct supervised fine-tuning for these generation pipelines and additionally apply preferencebased optimization including further supervised fine-tuning with rejection sampling data and direct preference optimization (dpo). moreover, we examine the effect of content and citation diversity on fine-tuned model performance. besides trainingtime techniques, we also study an inference-time technique-over-generation and reranking. our experiments find that (1) direct generation yields the best overall quality; (2) dpo is the best fine-tuning method; (3) maintaining sufficient content diversity is crucial for preference-based optimization; (3) reranking of over-generated samples can benefit all verifiable generation systems but cost-effectiveness might be low; (4) llms' ability to cite supporting sources is robust across datasets. we hope our findings can guide further development of verifiable generation systems with open-source llms. science ph.d. fellowship. we thank arr reviewers for their feedback. 7 limitations and potential risks limitations. our work conducts a wide range of experiments, but there remain design choices that are not investigated, due to the complexity of verifiable generation systems. for example, the process of handling the pool of candidate cited sources could benefit from more sophisticated strategies, which might include multi-turn processing of cited sources or the construction of dense representations. the datasets employed in our experiments provide a fixed set of candidate sources with wellformatted content. however, in real-world scenarios, candidate sources are dynamically retrieved from online search engines. the use of online search engines can introduce a greater diversity of candidate sources, resulting in domain and style shifts that could impact model behavior and task performance unpredictably. potential risks. echoing the limitations mentioned, our results are based on a pool of trustworthy sources, such as wikipedia articles. however, when verifiable generation systems are deployed in practical settings, they may encounter sources with varying degrees of reliability. this creates a risk of propagating misinformation if the system inadvertently relies on less credible sources. furthermore, dynamically retrieved data could include biased or malicious content, potentially leading to harmful consequences. therefore, our study reveals best practices of verifiable generation systems in controlled conditions, the robustness of them in uncontrolled environments requires further investigation. developers should equip their systems with additional content filters to ensure healthy outputs. tianyu gao, howard yen, jiatong yu, and danqi chen. aaron grattafiori, abhimanyu dubey, abhinav jauhri, abhinav pandey, abhishek kadian, ahmad al-dahle, aiesha letman, akhil mathur, alan schelten, alex vaughan, amy yang, angela fan, anirudh goyal, anthony hartshorn, aobo yang, archi mitra, archie sravankumar, artem korenev, arthur hinsvark, arun rao, aston zhang, aurelien rodriguez, austen gregerson, ava spataru, baptiste how d s i support s i as the citation quality measure. we take the probability of the nli model predicting "entail" as the score. as the length of d s i might exceed the maximum input length of the nli model and the nli model is trained with shorter sequences, following @xcite , we split the document into chunks of 256 tokens {d 1 s i , . . . , d m s i } and take the maximum entailment score between s i and chunks of d s i as the entailment score between s i and d s i :