From RAG to Riches: Retrieval Interlaced with Sequence Generation (2024)

Abstract: We present RICHES, a novel approach that interleaves retrieval with sequence generation tasks. RICHES offers an alternative to conventional RAG systems by eliminating the need for separate retriever and generator. It retrieves documents by directly decoding their contents, constrained on the corpus. Unifying retrieval with generation allows us to adapt to diverse new tasks via prompting alone. RICHES can work with any Instruction-tuned model, without additional training. It provides attributed evidence, supports multi-hop retrievals and interleaves thoughts to plan on what to retrieve next, all within a single decoding pass of the LLM. We demonstrate the strong performance of RICHES across ODQA tasks including attributed and multi-hop QA.

## introduction large language models (llms) have increasingly become the backbone for much of natural language processing and there has been a push to formulate a wide range of tasks as sequence to sequence transduction. however, when llms need to interact with non-parametric knowledge in the form of an external evidence corpus, the typical approaches chain llm generations with calls to a separate retrieval model as part of a multi-system pipeline. in this paper we introduce a new approach, riches (retrieval interlaced with sequence generation) which can natively interleave text generations with retrievals from an evidence corpus using a single llm and decoding process. riches builds on previous work that demonstrated the application of constrained decoding to retrieval over a corpus @xcite but extends this work to support multiple retrievals, entwined in a standard text generation procedure. in this approach, we retrieve documents by directly decoding their contents or related natural language retrieval keys that point to figure 1 : example riches outputs for multi-hop queries with a single llm and decoding pass. the green quoted text is "retrieved" or generated verbatim from the retrieval corpus. riches generation natively interleaves thoughts and multiple retrieval evidences. the documents they were generated from. for example, figure 1 illustrates a solution from riches to multi-hop question answering @xcite , where evidence must be retrieved from multiple separate documents, by iteratively generating an unconstrained 'thought' about what needs to be retrieved and then generating a supporting proposition derived from an evidence corpus and tied to an original piece of supporting text. riches executes this task in a single decoder pass. for this example task, which is evaluated alongside others in section 6, we have built on recent advances in chain-of-thought reasoning via prompting alone @xcite but have directly integrated the retrieval step without needing to account for any interaction with an external retrieval system. the observations we build this work on are: there is an another advantage of using language models as search agents. of the two core operations in retrieval, indexing and search, indexing is constrained by corpus size, while search typically depends only on the index structure. using large language models for indexing billion-token corpora is highly expensive, but search does not face the same bottle-neck. this enables us to unlock the knowledge stored in very large models for retrieval. this work overlaps with a variety of related work focusing on retrieval, retrieval augmented generation @xcite , reasoning in language models, and open domain question answering. we discuss their connections to riches in section 2, then introduce the key components of the generalizable riches approach in section 3. while riches is applicable to any task that can be reduced to an interleaved generation of unconstrained text and pre-defined retrieval keys, we validate the approach with tasks in open domain question answering and show how it natively supports single-hop question answering, including the case where attribution to a source text is required; multihop question answering; and interleaving retrieval with 'planning steps' that enhance the retrieval performance. results are presented in section 6.2 along with qualitative examples and analysis in section 6.3 to help motivate the approach. generative retrieval @xcite techniques shifting the onus of search from nonparametric nearest neighbor scan to language models. differentiable search index @xcite memorizes a mapping of query to opaque document identifiers, however memorization struggles to generalize to unseen corpus @xcite ). an alternative approach is to use natural language keys as document identifiers, where keys are constrained decoded to lie in the corpus @xcite . these systems still need an external model to generate answers. 1-pager @xcite unifies evidence and answer generation, by generating a sequence of keywords that map to a document. however, isolated keywords limit context understanding and suffer similar pitfalls as lexical matching. recitation separate from retrieval augmentation, language models have been shown to recite entire passages from memory @xcite . but these passages are prone to hallucination. our aim is to intersect contextual passage generation with corpus grounding. @xcite predicts next token from a constrained corpus, however retrieval itself relies on nearest neighbor embedding match. gophercite @xcite , closest to our approach, generates quotes verbatim from a small set of documents using constrained decoding. riches aims to scale this to a billiontoken corpus. iterative reasoning and search in recent times, there have been several efforts to improve multihop question answering by better reasoning : decomposing a problem into sub-queries @xcite , interleaving cot with retrieval @xcite @xcite and iterative planning @xcite . language models have also been ap-plied to the task of search to explore alternative paths @xcite . our work builds on these advances in reasoning while integrating search within generation. ## retrieving while generating we present a method of interleaving unconstrained text generation with the generation of retrieval keys that point into a retrieval corpus. for example, figure 1 shows generations that interleave unconstrained 'thoughts' with evidence sentences drawn from a predefined corpus for a multi-hop question answering task. later in this section we'll introduce a number of different choices of retrieval key as well as a variety of tasks that benefit from interleaved generation and retrieval. however, for now we simply define a retrieval key as a sequence of tokens that exists in a pre-defined finite set of sequences k where every entry is associated with one or more documents in an underlying corpus c. formally, we focus on the sequence to sequence transduction task where we predict an output sequence @xmath0, . . . , y n ] conditioned on an input sequence @xmath1, . . . , x m ] and we mark the start and end of a retrieval key in y with special markers « and ». if we let q(y) be a function that returns all retrieval key spans from y (i.e. (i, j) ∈ q([y 0 , . . . , «, y i , . . . , y j », . . . , y n ])) then we can update the standard autoregressive language modeling probability to include the indicator function 1 k (q) that maps elements of k onto one and otherwise to zero. ## interleaving retrieval and generation we have presented a method of interleaving unconstrained text generation with constrained generation of retrieval keys. in this section we introduce a handful of tasks that make use of this interleaving either as a core task requirement, or as a means to an end by interleaving 'thoughts' with retrieval actions to help guide search. ## results in the following sections, we investigate the key building blocks of riches: i) indexing strategies (section 3.4) amenable to auto-regressive decoding ii) effect of beam decoding (section 3.1) iii) suitable mechanisms to interleave thoughts and retrieval keys (section 3.3). finally, we compare riches against conventional retrieval systems. we also draw a detailed analysis of wins and losses to fathom the strengths and pitfalls of the system. 3 shows that among the retrieval keys explored, the propositional index is best aligned with our decoding search strategy, perhaps its compact nature is most suited for autoregressive decoding. an in-depth analysis of retrieval keys is provided in appendix a.5. in the following experiments, we use proposition as our retrieval key. ## limitations first we note the limitations in our experimental setup. all our experiments are based on wikipedia, a corpus heavily seen during pre-training. this work does not analyze how riches fares on corpora unseen during pre-training. furthermore, we only examine a handful of factoid questionanswering tasks due to the lack of objective evaluations. performance on tasks such as long-form qa is deferred for future work. there are also certain inherent limitations with riches. it forces verbatim emission of corpus text, which might be an overkill for tasks where a similarity-based metric is sufficient. riches lacks the ability to retrieve dozens of documents, a necessity for certain summarization tasks. for long documents with diffused information, rewriting into propositions adds complexity and can be cumbersome. lastly, while riches's search operation is independent of corpus size, the use of beam search and communication between the fm-index and transformer model can slow down inference. ## ethical considerations all artifacts used in this paper, including models, datasets, and baselines, are under permissive licenses and publicly available. we have attempted to provide detailed information to facilitate the reproduction of our results. our findings are based on english-language data from wikipedia, and we have not tested the generalizability of our claims to other languages or domains. lastly, the datasets used in this work are not expected to contain any offensive content. however, it is important to note that large language models (llms) can exhibit biases related to gender, race, and region, and are also prone to hallucination. although riches aims to ground its generation in an external corpus, some biases may still be present.