“According to . . . ”: Prompting Language Models Improves Quoting from Pre-Training Data (2024)

Abstract: Large Language Models (LLMs) may hallucinate and generate fake information, despite pre-training on factual data. Inspired by the journalistic device of “according to sources”, we propose according-to prompting: directing LLMs to ground responses against previously observed text. To quantify this grounding, we propose a novel evaluation metric (QUIP-Score) that measures the extent to which model-produced answers are directly found in underlying text corpora. We illustrate with experiments on three corpora (Wikipedia, PubMed, and the U.S. legal tax code) that these prompts improve grounding under our metrics, with the additional benefit of often improving end-task performance. Furthermore, prompts that ask the model to decrease grounding (or to ground to other corpora) indeed decrease QUIP-Score, indicating the ability of LLMs to increase or decrease grounded generations on request.

## introduction as the deployment of large language models (llms) in real-world applications continues to grow, their tendency to generate false content @xcite poses significant risks to downstream users. recent work has attempted to address this issue by augmenting them with retrieval @xcite @xcite ; however, these models still struggle with hallucination problems in practice @xcite . this work explores the intriguing possibility of steering llms by prompting them to quote more of the curated sources of information they have memorized during pre-training, thereby reducing their tendency to generate false information. as illustrated in figure 1 , we explore whether adding phrases such as "according to wikipedia" can guide llms to quote from wikipedia, which is ## methodology defining grounding there are many definitions of grounding in the community @xcite . while acknowledging the broad scope of the term, we adopt a narrow definition: we call generated text grounded with respect to a corpus if it is an exact quotation from the corpus. this is more stringent than some definitions because it does not count semantic grounding, e.g. when lexical forms do not match; however, quotation is one form of grounding that is intuitive and simple to measure. 2 hence, we use quoting and grounded interchangeably. 2 we leave it to future work to expand our metric to the semantic grounding case, as semantic grounding (e.g. finding paraphrases) while matching the generations over an entire corpus is non-trivial; using retrieval systems biases the model towards lexical match (even for dense retrieval, c.f. macavaney et al. ( 2022 )) and existing work in attribution/grounding does not scale to allow grounding to numerous (2+) passages. ## grounding via according-to prompting the previous results show 1) that we can efficiently measure quotation rate and 2) that more quotations correlate with fewer hallucinations. next, we seek to improve knowledge grounding by causing llms to quote directly from trusted resources seen during training. 8 we hope to access helpful memorized content: strings copied from high-quality or trusted documents. we induce this behavior by taking a normal task prompt (e.g. an odqa question) and appending an instructional phrase that encourages grounding such as "respond by using information from wikipedia in your response". 9 we call this strategy according-to prompting. our experiments measure the change in quip-score of generations from a according-to prompt vs one without the extra instruction (i.e. a null prompt). to verify that prompts can both increase and decrease grounding, we also include prompts that are anti-grounding (e.g. "respond by using information from [another source] in your response" or "respond without using any information from wikipedia.") this allows us to test the hypothesis that models can ground (or not ground) to a given corpus when asked because of the semantic meaning of the prompt, rather than the length of the prompt. as prompting is notoriously brittle (e.g. changing the phrasing can affect the results) we provide a number of grounding and anti-grounding prompts to test whether these prompts provide consistent gains or are merely prompting artifacts (see ## results we first analyze a wide range of according-to prompts on chatgpt. we then test the null prompt and the best performing according-to prompt on a variety of other models for further analysis. surprisingly, we find that according-to prompts also perform similarly, and sometimes even better than, the null prompt on end task performance (e.g. up to a 6% improvement on nq, 2.5% on hotpotqa). this is not the case for rouge-l on eli5, as that metric measures lexical similarity to reddit, rather than similarity to wikipedia. we use these results on chatgpt to inform our next experiments, using the null prompt and the best grounding prompt ("respond to this question using only information that can be attributed to wikipedia") in our future experiments due to cost. ## grounding to other corpora in the previous experiments we've shown that llms can ground to wikipedia upon request. however, are according-to prompts limited to the wikipedia domain only? to answer this question we build two more data portraits, one on pubmed articles and one with the u.s. legal tax code applying to tax year 2022. we then evaluate on datasets where having such background knowledge would be helpful and evaluate both the downstream task performance and quip-score, similar to the previous wikipedia experiments. datasets. we use datasets in the multimedqa benchmark suite @xcite why is it that every time we put something down our throat toothbrush, finger etc. we start to gag but when we but food and water down our throat we don't gag at all?, null prompt (chatgpt) this is because our gag reflex is triggered by objects that touch the back of our throat and activate sensory receptors called the pharyngeal receptors. when we eat or drink, the muscles in our throat automatically contract and move food and liquid down to the stomach, bypassing the pharyngeal receptors and avoiding the gag reflex. however, when we put non-food objects like a toothbrush or finger in our throat, they can activate these receptors and trigger the gag reflex. grounded prompt (chatgpt) the gag reflex is a natural protective mechanism of the body that can be triggered by stimulation of the soft palate, the back of the tongue or the back of the throat. it is a reflex contraction of the muscles of the back of the throat that can cause vomiting or choking. the reflex can be more easily triggered by foreign objects, such as toothbrushes or fingers, that are not natural to the body, compared to food and water which are familiar substances that we regularly ingest without triggering the reflex. additionally, the shape and texture of the foreign object may also contribute to triggering the gag reflex. ## discussion and future implications our results strongly suggest that llms can be steered via prompting to increase the amount by which they quote human-authored sources in their training data. this finding has strong implications not just for our considered tasks, but also for a wide array of other task spaces in which provenance grounding is important. we note that our according-to prompting strategy is orthogonal to other directions in llm grounding, including using retrieval augmentation, and as according-to prompting is simple and generally increases both grounding and task performance we would encourage future research to try our approach in tandem. ## conclusion large language models struggle with hallucination, or generating incorrect information, despite the large amount of factual pre-training data they were trained on. to help alleviate this problem, we proposed according-to prompts, asking language models to ground their output to their pre-training corpus. to quantify the extent to which models achieve this goal, we introduced a new metric, quip-score, that efficiently and quickly measures the percent of the model's generation that exists as exact quotes in the pre-training corpus. we showed that prompting models with according-to prompts greatly improves the quip-score while anti-grounding prompts reduces the quip-score, across a variety of domains and corpora. our analysis also shows that quip-score increases with the popularity of the entity in the question and the model size. we hope that this work brings more attention to the positive aspects of llm memorization and encourages more work into understanding llm grounding to their pre-training data.