RARR : Researching and Revising What Language Models Say, Using Language Models (2023)

Abstract: Language models (LMs) now excel at many tasks such as question answering, reasoning, and dialog. However, they sometimes generate unsupported or misleading content. A user cannot easily determine whether their outputs are trustworthy or not, because most LMs do not have any built-in mechanism for attribution to external evidence. To enable attribution while still preserving all the powerful advantages of recent generation models, we propose RARR (Retrofit Attribution using Research and Revision), a system that 1) automatically finds attribution for the output of any text generation model, and 2) post-edits the output to fix unsupported content while preserving the original output as much as possible. When applied to the output of several state-of-the-art LMs on a diverse set of generation tasks, we find that RARR significantly improves attribution while otherwise preserving the original input to a much greater degree than previously explored edit models. Furthermore, the implementation of RARR requires only a handful of training examples, a large language model, and standard web search.

## introduction generative language models (lms) and other text generation models are now the backbone of many ai systems. for example, large language models can perform multi-step reasoning @xcite , generate plans @xcite , use tools and apis @xcite , and answer open-domain questions @xcite . despite these incredible advances, state-of-theart lms still frequently produce biased, misleading, or unsupported content, colloquially called "hallucinations" @xcite . to make lms more trustworthy, we want to justify each generation by an attribution report @xcite that contains supporting evidence from trusted sources (e.g., encyclopedia or articles) where appropriate. most existing lms, such as those based on sequence-to-sequence architectures, lack a builtin mechanism for attribution. even retrievalaugmented models @xcite , which retrieve relevant documents and then condition on them to generate text, still do not guarantee attribution. prior work has shown that retrieval-augmented models generate text that either includes additional information outside the retrieved documents @xcite , ignores the documents altogether @xcite , or even contradicts the documents @xcite . in fact, occasionally ignoring the retrievals can make the models more robust to bad retrievals @xcite , illustrating that end-task performance and attribution are not always aligned. instead of constraining lms to generate attributed text, we propose a model-agnostic approach to improve the attribution of any existing lm: retrofit attribution using research and revision (rarr). the approach is inspired by works on fact-checking 2 where simple research-and-revise workflows are effective at attributing or correcting unattributed claims made by humans @xcite @xcite . as shown in figure 1 , after generating text with the lm, rarr does research to retrieve relevant evidence, and then revises the text to make it consistent with the evidence while preserving qualities like style or structure, enabling the revised text to be seamlessly used in place of the original. rarr can be viewed as a retrievalaugmented model where retrieval happens after generation rather than before. this allows rarr to stand on the shoulders of giant lms without having to modify them to support attribution. in our effort to expand the scope of research & revision models to handle the output of arbitrary lms, we make the following contributions. first, we formalize the editing for attribution task and propose new metrics that evaluate revision models not just on their ability to produce well-attributed revisions, but also on their ability to otherwise preserve original properties of the text. second, we use these metrics to benchmark how existing revision models perform on various types of lm outputs such as knowledge-intensive statements, reasoning chains, and dialog responses. finally, we find that existing revision models do not always generalize across many tasks (and were not originally intended to), and therefore propose a new research-and-revise model that leverages the power of few-shot prompting in large language models to robustly generalize across domains. query generation when did millie inbetween premiere? what channel was millie inbetween on? retrieval retrieval [fandom.com] … the first series premiered on 1 october 2014. [comedy.co.uk] millie inbetween. cbbc sitcom about a young … agreement agreement output attribution report @xmath0, .., e m } edit combined fi edit skipped [fandom.com] … the first series premiered on 1 october 2014. [comedy.co.uk] millie inbetween. cbbc sitcom about a young … q 1 q n [comedy.co.uk] millie inbetween. cbbc sitcom about a young … [fandom.com] … the first series premiered on 1 october 2014. ## task formulation we propose the task of editing for attribution as follows. as figure 1 shows, the input to the system is a text passage x produced by a generation model. the output is a revised text passage y along with an attribution report a, which contains evidence snippets e 1 , . . . , e m that support the content in y. optionally, the attribution report can contain additional information such as the alignment between evidence snippets and relevant parts in y. we propose to measure the quality of the revised text y and attribution report a along two dimensions: (1) attribution: how much of the revised text y can be attributed to the evidence in a, and (2) preservation: how much the revised text y preserves aspects of the original text x. ## approach we now present retrofit attribution using research and revision (rarr), a simple method for solving the editing for attribution task. as illustrated in figure 2 , given an input passage x, the research stage first generates a set of queries {q 1 , ..., q n }, each investigating one aspect of x that potentially most components for rarr are implemented using few-shot prompting @xcite . we use palm @xcite as our language model. figure 3 shows some few-shot examples we use, while appendix d lists the full prompts. ## analysis 6.1 qualitative analysis human oracle to understand the remaining headroom in our task, we ask: what is the minimal amount of editing needed to make a text passage fully attributed? the answer would depend on the quality of the lm that generated the text as well as the task difficulty. as an approximation, we manually edited 30 examples in our nq benchmark until we judged them to be 100% attributable. we achieved a preservation score of 88%, which (when combined with 100% attribution) translates to 93.6 f1 ap , indicating a significant headroom. analyzing the baselines as exemplified in figure 6 , efec frequently attempts to summarize the entire passage into one sentence, or drops later sentences. this is likely due to efec's training data, which was limited to single sentences. this behavior generally increases the attribution score, because it is usually easier to make one sentence fully attributable than many sentences. however, in datasets where the claim contains multiple sentences (nq and sqa), such a behavior yields low preservation scores, and also results in outputs that are less informative. we expect that efec could perform much better if its training data were augmented to include multiple sentences. lamda research achieves similar attribution scores to rarr. but as mentioned in section 5.2, the intent and linguistic style of the output tend to deviate from the input, resulting in lower preservation scores (figure 6 ). we emphasize that this is not a purely apples-to-apples comparison since lamda was not optimized for preservation. overall, these experiments are mainly meant to illustrate that prior models were simply not designed for the task of editing for attribution, rather than to mark rarr as the best method. analyzing rarr for the research stage, the question generation model had comprehensive coverage: a manual inspection of 40 examples @xmath1. the retriever was strongest at re- ## conclusion language models have developed increasingly good "procedural" knowledge of what should be discussed and how it should be presented, but often struggle to memorize "factoid" knowledge and produce unsubstantiated claims. we proposed rarr, a framework for revising such claims to make them attributable to the researched evidence. from experiments on text passages generated by different models on various domains, we showed that rarr can revise the passages to improve attribution while preserving other desirable properties such as writing style or structure. furthermore, rarr sits on top of existing generation models without needing to re-design or re-train lms. major headroom still remains, as discussed in section 6 and the limitations section. we hope our analysis of rarr would help with developing new approaches for integrating attribution to lms.