Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions (2023)

Abstract: Prompting-based large language models (LLMs) are surprisingly powerful at generating natural language reasoning steps or Chains-of-Thoughts (CoT) for multi-step question answering (QA). They struggle, however, when the necessary knowledge is either unavailable to the LLM or not up-to-date within its parameters. While using the question to retrieve relevant text from an external knowledge source helps LLMs, we observe that this one-step retrieve-and-read approach is insufficient for multi-step QA. Here, what to retrieve depends on what has already been derived , which in turn may depend on what was previously retrieved . To address this, we propose IRCoT, a new approach for multi-step QA that interleaves retrieval with steps (sentences) in a CoT, guiding the retrieval with CoT and in turn using retrieved results to improve CoT. Using IRCoT with GPT3 substantially improves retrieval (up to 21 points) as well as downstream QA (up to 15 points) on four datasets: HotpotQA, 2WikiMultihopQA, MuSiQue, and IIRC. We observe similar substantial gains in out-of-distribution (OOD) settings as well as with much smaller models such as Flan-T5-large without additional training. IRCoT reduces model hallucination, resulting in factually more accurate CoT reasoning.

## introduction large language models are capable of answering complex questions by generating step-bystep natural language reasoning steps-so called chains of thoughts (cot)-when prompted appropriately @xcite . this approach has been successful when all information needed to answer the question is either provided as context (e.g., algebra questions) or assumed to be present in the model's parameters (e.g., commonsense reasoning). 1 code, data, and prompts are available at https:// github.com/stonybrooknlp @xcite in what country was lost gravity manufactured? the lost gravity was manufactured by mack rides. mack rides is a company from germany. the answer is germany. however, for many open-domain questions, all required knowledge is not always available or up-todate in models' parameters and it's beneficial to retrieve knowledge from external sources @xcite . how can we augment chain-of-thought prompting for open-domain, knowledge-intensive tasks that require complex, multi-step reasoning? while a one-shot retrieval from a knowledge source based solely on the question can successfully augment lms with relevant knowledge for many factoid-based tasks @xcite @xcite , this strategy has clear limitations for more complex multi-step reasoning questions. for such questions, one often must retrieve partial knowledge, perform partial reasoning, retrieve additional information based on the outcome of the partial reasoning done so far, and iterate. as an example, consider the question illustrated in fig. 1 , " in what country was lost gravity manufactured?". the wikipedia document retrieved using the question (in particular, the roller coaster lost gravity) as the query does not mention where lost gravity was manufactured. instead, one must first infer that it was manufactured by a company called mack rides, and then perform further retrieval, guided by the inferred company name, to obtain evidence pointing to the manufacturing country. thus, the retrieval and reasoning steps must inform each other. without retrieval, a model is likely to generate an incorrect reasoning step due to hallucination. additionally, without generating the first reasoning step, the text supporting the second step can't be identified easily given the lack of lexical or even semantic overlap with the question. in other words, we need retrieved facts in order to generate factually correct reasoning steps and the reasoning steps to retrieve relevant facts. based on this intuition, we propose an interleaving approach to this problem, where the idea is to use retrieval to guide the chain-of-thought (cot) reasoning steps and use cot reasoning to guide the retrieval. fig. 1 shows an overview of our retrieval method, which we call ircot. 2 we begin by retrieving a base set of paragraphs using the question as a query. subsequently, we alternate between the following two steps: (i) extend cot: use the question, the paragraphs collected thus far, and the cot sentences generated thus far to generate the next cot sentence; (ii) expand retrieved information: use the last cot sentence as a query to retrieve additional paragraphs to add to the collected set. we repeat these steps till the cot reports an answer or we reach the maximum allowed number of reasoning steps. upon termination, all collected paragraphs are returned as the retrieval outcome. finally, we use these as the context for answering the question via direct qa prompting @xcite or cot prompting @xcite . we evaluate the efficacy of our system on 4 multi-step reasoning datasets under an open-domain setting: hotpotqa @xcite , 2wikimultihopqa @xcite , musique @xcite , and iirc @xcite . our experiments using openai gpt3 (code-davinci-002) @xcite @xcite demon-2 interleaved retrieval guided by chain-of-thought. in summary, our main contribution is a novel retrieval method, ircot, that leverages lms' chainof-thought generation capabilities to guide retrieval and uses retrieval in turn to improve cot reasoning. we demonstrate that ircot: ## related work prompting for open-domain qa. llms can learn various tasks by simply using a few examples as prompts @xcite . they've also been shown to answer complex questions by producing step-by-step reasoning (chain-ofthoughts, or cot) when prompted with a few or zero demonstrations @xcite . prompting has been applied to open-domain qa @xcite @xcite but its value in improving retrieval and qa for multi-step open-domain questions remains relatively underexplored. recently three approaches have been proposed for multi-step open-domain qa. selfask @xcite prompts llms to decompose a question into subquestions and answers subquestions by a call to google search api. decomp @xcite is a general framework that decomposes a task and delegates sub-tasks to appropriate submodels. they also decompose questions but delegate retrieval to a bm25-based retriever. both of these approaches are not developed for cot reasoning, do not focus on the retrieval problem, and require a single-hop qa model to answer the decomposed questions. recently proposed react @xcite system frames the problem as generating a sequence of reasoning and action steps. these steps are much more complex, rely on much larger models (palm-540b), and require fine-tuning to outperform cot for multi-step odqa. furthermore, none of these works have been shown to be effective for smaller models without any training. while a direct comparison with these approaches is not straightforward (difference in knowledge corpus, llms, examples), we find that our odqa performance is much higher than all their reported numbers where available ( §5). ## experimental setup we evaluate our method on 4 multi-step qa datasets in the open-domain setting: hotpotqa @xcite , 2wikimul-tihopqa @xcite , answerable subset of musique @xcite , and answerable subset of iirc @xcite . for hotpotqa, we use the wikipedia corpus that comes with it for the open-domain setting. for each of the other three datasets, which originally come in a reading comprehension or mixed setting, we used the associated contexts to construct a corpus for our open-domain setting (see app. a for details). for each dataset, we use 100 randomly sampled questions from the original development set for tuning hyperparameters, and 500 other randomly sampled questions as our test set. ## results ircot retrieval is better than one-step. flan-t5-xxl and gpt3 lms. for both models, ircot significantly outperforms one-step retrieval across all datasets. for flan-t5-xxl, ircot improves our recall metric relative to one-step retrieval, on hotpotqa by 7.9, on 2wikimultihopqa by 14.3, on musique by 3.5, and on iirc by 10.2 points. for gpt3, this improvement is by 11.3, 22.6, 12.5, and 21.2 points, respectively. ircot qa outperforms nor and oner qa. fig. 4 compares odqa performance using nor, oner and ircot retriever made from flan-t5-xxl and gpt3 lms. for flan-t5-xxl, ircot qa outperforms oner qa on hotpotqa by 9.4, on 2wikimultihopqa by 15.3, on musique by 5.0 and iirc by 2.5 f1 points. for gpt3, the corresponding numbers (except for iirc) are 7.1, 13.2, and 7.1 f1 points. for gpt3, ircot doesn't improve the qa score on iirc, despite significantly improved retrieval (21 points as shown in fig. 3 ). this is likely because iirc relevant knowledge may already be present in gpt3, as also evidenced by its nor qa score being similar. for other datasets and model combinations, nor qa is much worse than ircot qa, indicating the limits of the models' parametric knowledge. ircot is effective in ood setting. since cot may not always be easy to write for new datasets, we evaluate nor, oner, and ircot on generalization to new datasets, i.e. ood setting. to do so, we use prompt demonstrations from one dataset to evaluate on another dataset. 9 for all pairs of the datasets foot_7 and for both flan-t5-xxl and gpt3, we find the same trend as in the iid setting: ircot retrieval outperforms oner (fig. 5 ), and ircot qa outperforms both oner qa and nor qa (fig. 6 ).