IAG : Induction-Augmented Generation Framework for Answering Reasoning Questions (2023)

Abstract: Retrieval-Augmented Generation (RAG), by incorporating external knowledge with parametric memory of language models, has become the state-of-the-art architecture for open-domain QA tasks. However, common knowledge bases are inherently constrained by limited coverage and noisy information, making retrieval-based approaches inadequate to answer implicit reasoning questions. In this paper, we propose an Induction-Augmented Generation (IAG) framework that utilizes inductive knowledge along with the retrieved documents for implicit reasoning. We leverage large language models (LLMs) for deriving such knowledge via a novel prompting method based on inductive reasoning patterns. On top of this, we implement two versions of IAG named IAG-GPT and IAG-Student, respectively. IAG-GPT directly utilizes the knowledge generated by GPT-3 for answer prediction, while IAG-Student gets rid of dependencies on GPT service at inference time by incorporating a student inductor model. The inductor is firstly trained via knowledge distillation and further optimized by back-propagating the generator feedback via differentiable beam scores. Experimental results show that IAG outperforms RAG baselines as well as ChatGPT on two Open-Domain QA tasks. Notably, our best models have won the first place in the official leaderboards of CSQA2.0 (since Nov 1, 2022) and StrategyQA (since Jan 8, 2023).

## introduction open-domain question answering (odqa) @xcite has attracted increasing research attention. compared with the closed-domain setting, techniques developed upon odqa models can empower search engines with the ability to respond to a wider range of user queries. as a typical knowledge-intensive task, odqa has been extensively studied within the scope of information retrieval @xcite @xcite , where access to external knowledge sources such as web pages or knowledge bases is required. another line of research exploits large language models (llms) such as gpt-3 and palm as the knowledge source @xcite and develops various prompting methods to elicit knowledge from llms that is implicitly encoded in the parameters. however, to answer reasoning questions, i.e., questions that demand some degree of reasoning ability, either retrieval-based or prompting-based approaches suffer from their intrinsic limitations. on the one hand, although @xcite has become the sota architecture for odqa, documents retrieved from common knowledge bases generally suffer from the limitations of constrained coverage and noisy information, especially for implicit reasoning questions whose answers are not well documented. for example, it's trivial for an average child to answer questions such as "can you catch a jellyfish in the dumpster?". however, it's unlikely to find the answer directly from the web or books. as shown in figure 1 , the retrieved documents can hardly answer the question in such cases. hence, relying entirely on information retrieval is insufficient to solve implicit reasoning questions. on the other hand, prompting-based methods can exploit the considerable amount of knowledge encoded in the parameters of llms for qa tasks @xcite @xcite . but the problem of hallucination (i.e., generating natural language statements that are factually incorrect) imposes limitations on llms in terms of factuality and credibility. to better control the knowledge elicited from llms, various prompting methods such as chainof-thought (cot) @xcite have been proposed by constructing intermediate reasoning steps until arriving at the conclusion. however, capability of the an llm is intrinsically constrained by its parameter size, making it unable to respond correctly to domain-specific questions beyond the scope of its training corpus. in view of the above challenges, it requires a new paradigm for building models applicable to reasoning qa. to this end, we combine the advantages of these two kinds of methods and propose iag, an induction-augmented generation framework for answering implicit reasoning questions. iag enhances the conventional rag with an inductor that generates inductive knowledge w.r.t each question. to derive such knowledge, we propose a novel prompting method, which is intuitively inspired by the cognitive functions of inductive reasoning, to elicit inductive knowledge from an llm (i.e., gpt-3). our first implementation of iag, which is termed iag-gpt, directly leverages the knowledge statements sampled from gpt-3 as evidence alongside the retrieved documents to feed into the answer generator. we show that iag-gpt improves over sota models on multiple odqa datasets and has won the first place in the official leaderboards of csqa2.0 and strategyqa. we also implement iag-student, the second variant that gets rid of dependencies on gpt service during inference, by training a student inductor model following a twostep optimization scheme. specifically, the model is firstly warmed up through distillation by using the gpt-3 statements as pseudo labels, and then further optimized with a novel tailback approach: gradient back-propagation differentiable beam scores feedback. tailback implements a differentiable beam search algorithm for the inductor and allows the feedback from the generator to be back-propagated to the inductor via beam scores. we verify that iag-student improves over rag baselines on a smaller architecture. the contributions of this paper include: 1) an inductive prompting method which improves the factuality of knowledge elicited from llms by constructing a reasoning path via inductive reasoning; 2) an iag-gpt implementation that improves over strong baseline models and chatgpt by leveraging the knowledge elicited from gpt-3 as auxiliary supporting evidence for the generator; 3) a tailback optimization algorithm to train the inductor which allows iag-student to outperform rag baselines under a small model size. prompting-based reasoning. few-shot prompting is a mainstream approach to eliciting knowledge from llms @xcite @xcite ). to answer implicit reasoning questions, various prompting methods have been proposed among which cot @xcite has attracted the most research interest in the earlier research. it allows llms to answer complex questions in a step-bystep manner. follow-up studies include enhancing the consistency of cot @xcite and improving the correctness of the reasoning steps. recently, adaptive prompting methods have been shown to achieve better performance on reasoning tasks such as babi @xcite and proofwriter @xcite . for example, the selection-inference (si) framework @xcite alternates between selection and inference to generate a series of interpretable reasoning steps. lambada @xcite proposes the idea of backward chaining and employs four llm-based sub-modules for reasoning. however, si and lambda rely on heuristic search algorithms performed in a limited context space, which cannot be easily adapted to open-domain reasoning tasks. the inductive prompting proposed in this paper can be used for general-purpose knowledge elicitation. ## conclusion to tackle the problem that retrieval-based methods cannot provide sufficient knowledge for the generator to answer implicit reasoning questions, we propose a novel iag framework that augments rag with inductive knowledge elicited from language models. we first design an inductive prompting method that enhances the factuality of knowledge elicited from gpt-3. we further propose a sophisticated optimization scheme that trains a student inductor model via distillation and tailback. our results suggest that iag outperforms rag in answering implicit reasoning questions.