Verifiable by Design: Aligning Language Models to Quote from Pre-Training Data (2025)

Abstract: To trust the fluent generations of large language models (LLMs), humans must be able to _verify_ their correctness against trusted, external sources. Recent efforts, such as providing citations via retrieved documents or post-hoc provenance, enhance verifiability but provide no guarantees on their correctness. To address these limitations, we tackle the verifiability goal with a different philosophy: _trivializing the verification process by developing models that quote verbatim statements from trusted sources in their pre-training data._We propose Quote-Tuning, which demonstrates the feasibility of aligning models to quote. The core of Quote-Tuning is a fast membership inference function that efficiently verifies text against trusted corpora. We leverage this tool to design a reward function to quantify quotes in model responses, and curate datasets for preference learning. Experiments show that Quote-Tuning significantly increases verbatim quotes from high-quality documents by up to 130% relative to base models while maintaining response quality. Quote-Tuning is applicable in different tasks, generalizes to out-of-domain data and diverse model families, and provides additional benefits to truthfulness. Our method not only serves as a hassle-free method to increase quoting but also opens up avenues for improving LLM trustworthiness through better verifiability.

## introduction recent developments have enabled large language models (llms) to generate fluent text and follow instructions @xcite @xcite . however, llms are known to produce seemingly plausible but erroneous outputs, often referred to as hallucinations @xcite . this poses significant risks to downstream users due to the difficulty of fact-checking seemingly convincing generations from llms @xcite @xcite . one of the important desiderata for llms is thus verifiability, the ability to ground responses to supporting evidence and render the produced claims easy to verify for humans. verifiability allows users to uncover the competency of llms and calibrate user trust, a crucial aspect of building trustworthy human-machine relationships @xcite . recent work increases verifiability through external artifacts such as producing citations @xcite , retrieving documents @xcite , or post-hoc attribution methods @xcite . although helpful, these approaches provide no guarantee of relevance or usefulness. models generations can be unfaithful to the retrieved documents in the context @xcite , generative search engines often produce citations that are irrelevant or inaccurate @xcite , and explanations alone do not lead to verifiability @xcite . we overcome the windingness of previous approaches through a verifiable-by-design approach: generating verbatim quotes from high-quality sources such as wikipedia. by determining verbatim quotes from large-scale and high-quality corpora with efficient membership testing tools @xcite , those generations provide a natural method for attributing and verifying the correctness of generated claims. llms' potential to quote is driven by the observation that they are pre-trained on internet scale data-a subset of which contains high quality, reliable information-and that they have memorized a wide range of content from the pre-training stage @xcite @xcite response: the four corner squares on a monopoly board are baltic avenue, mediterranean avenue, oriental avenue, and pacific avenue. these squares are located at the corners of the board and are considered some of the most valuable properties in the game due to their strategic location. response: the four corner squares on a monopoly board are baltic avenue, mediterranean avenue, oriental avenue, and pacific avenue. these squares are located at the corners of the board and are considered some of the most valuable properties in the game due to their strategic location. prompt: which is older jeopardy or wheel of fortune? response: jeopardy! was created by merv griffin and first aired in 1964, while wheel of fortune was also created by merv griffin and first aired in 1975. therefore, jeopardy! is older than wheel of fortune. prompt: which is older jeopardy or wheel of fortune? chosen response: jeopardy! was created by merv griffin and first aired in 1964, while wheel of fortune was also created by merv griffin and first aired in 1975. therefore, jeopardy! is older than wheel of fortune. rejected response: jeopardy! was created in 1964 by merv griffin, while wheel of fortune was created in 1975 by merv griffin and roy leonard. therefore, jeopardy! is older than wheel of fortune. pre-trained llm quote-tuned llm high-quality subset of pre-training corpus measure quoting via efficient membership testing step 1: sample multiple responses step 2: constructing preference data via rank-by-quoting we show this is indeed feasible with quote-tuning, our proposed method that aligns llms to quote through preference optimization and automatic feedback, without the need for human annotation. quote-tuning first generates responses from a pre-trained llm, and then synthesizes a preference dataset for quoting by ranking responses by how much they quote from a desired high-quality corpus, e.g., wikipedia. finally, quote-tuning aligns the model to quote from trusted sources by applying preference optimization algorithms (e.g., direct preference optimization @xcite ) on the synthesized reference dataset. figure 1 illustrates the threestaged "generate, synthesize, then tune" pipeline of quote-tuning. experiment results on long-form qa and openended text completion show that quote-tuning significantly increases quoting by up to 130% relative to base models while maintaining or improving downstream performance ( §4). moreover, quote-tuning generalizes to other domains and diverse model families, and enhances the truthfulness as measured by truthfulqa @xcite ( §5.1). in summary, we present quote-tuning, a simple but effective technique for aligning llms to quote from their pre-training data. the quoted responses are verifiable-by-design by inducing better verifiability without the need for human annotation and external knowledge bases (only leveraging parametric knowledge). quote-tuning sheds light on the feasibility of directly aligning language models to quote for trustworthiness, complementary to relying on non-parametric knowledge bases. ## preliminaries quantifying quoting we define a text string x as quoted from a corpus c if a verbatim copy of x is contained in c.this design allows us to use data portrait @xcite , a membership testing tool based on bloom filters @xcite , to efficiently check whether text n-grams have appeared in the corpus. specifically, we use quoted information precision score (quip-score) metric proposed by @xcite : ## aligning llms to quote with quote-tuning the design of quote-tuning is inspired by the observation that preference datasets can be used to elicit desired behaviors in lms using preference alignment frameworks @xcite @xcite . the prior work, for example, has used this approrach to address factuality @xcite , honesty @xcite , harmlessness @xcite , and relevance @xcite . we investigate whether automatic measures of quoting can be used to align llms to quote from their pretraining data. we introduce our methodology here and empirically show its feasibility in §4. illustrated in alg. 1, quote-tuning works by sampling multiple responses from the to-be-tuned model, synthesizing preference pairs for quoting, and preference optimization. we now detail each step. first, given a pre-trained llm policy π ref , for each prompt x (i) in a prompt dataset d prompt , we sample t responses y 1 , . . . , y t ∼ π ref (•|x (i) ) from the policy. next, we construct pairwise preference data (x (i) , y w , y l ) by selecting a pair of response (y w , y l ) (where y w is more preferred) from y 1 , . . . , y t that satisfies two constraints: constraint 1: quoting. quip c (y w ) -quip c (y l ) > δ quip , where δ @xmath0. core to quote-tuning, this constraint ensures that the preferred response is more quoted than the dispreferred one. constraint 2: length. if multiple pairs of responses satisfy the con-question: who won the most mvp awards in the nba? ## experiments in this section, we provide empirical evidence on how quote-tuning can provide better verifiability to llm-generated responses, while maintaining generation quality. we conduct quote-tuning on the long-form qa ( §4.1) and openended text completion ( §4.2) tasks. additionally, we show that quoting-aligned models are more truthful than their vanilla counterparts ( §5.1), and maintain downstream performance ( §5.2). ## related work improving verifiability hallucination in llms @xcite @xcite has motivated approaches that improve the verifiability of llm generations. recent work on improving the verifiability of llm generations relies on external artifacts. one emerging trend is training llms to produce citations that support generated claims @xcite @xcite . while citations improve attribution, llm can still hallucinate incorrect or irrelevant citations @xcite , which is non-trivial to verify. @xcite introduce intrinsic source citation to enable more faithful attribution to parametric knowledge, on a synthetic setting. quote-tuning is an approach complementary to citations that enhances verifiability. @xcite finds that explicit symbolic references to structured conditioning data, such as json tables, lead to faster human verification, further motivating our approach of increasing verifiability through verbatim quotes. retrieval-augmented generation @xcite @xcite allows fact-checking generation with the retrieved documents as supporting evidence. @xcite used retrieved tokens directly as generation, but is limited to the maskedfilling setting with short spans of text. however, checking against retrieved documents is still nontrivial and there is no guarantee that generated text is completely faithful to these documents. on the other hand, our framework for quoting, based on @xcite ; @xcite , makes the verification of quoted segments from fact bases trivial, given that the target model is capable of producing rich quotations after quote-tuning. our work, which focuses on parametric knowledge, is also complementary to methods that rely on non-parametric knowledge bases. ## discussions and future work in §4, we provide rich empirical evidence that quote-tuning can significantly promote parametric quoting across diverse tasks, domains, and model families. since quoted texts are autoregressively generated as spans of the model responses, and our evaluation quantitatively shows that these responses are of high adequacy and fluency (tables 2, 3, 4), generated quotes are therefore constrained to be highly adequate, fluent, and contextrelevant text. these findings demonstrate that it is not only possible but easily feasible to leverage parametric knowledge to generate more verifiable outputs. we thus argue current llms have abundant underutilized potential in improving their own verifiability, and call for future work that develop more attributable, verifiable approaches through our proposed "trivializing verification through quoting" framework. as an early exploration of the quoting framework, this work focuses on investigating the feasibility of unlocking parametric quoting through finetuning, and the generalizability of enhanced quoting across different domains, tasks, and model families. we have provided positive empirical evidence supporting both aspects. therefore, our setup focuses on measuring and improving the overall rate of quoting, as evaluated by quip-score, leaving room for future work to enhance quoting reward signals by considering other desiderata of quoted texts, such as quote completeness and usefulness under the current context. we provide an extended discussion of limitations in the subsequent section. quoting has implications on privacy, security, and copyright. we focus on enhancing quoting from wikipedia, a high-quality and low-risk corpus, for verifiability. as pre-training data contains diverse mixtures of data with varying risk levels @xcite , we argue that the grounding corpus for quote-tuning must be carefully selected and limited to trusted, public sources such as wikipedia to prevent privacy violations or copyright infringement. in conclusion, our approach presents a promising direction for leveraging the parametric knowledge of llms to facilitate easier verification of model generation and improve the calibration of humanmachine trust.