Teaching Small Language Models to Reason for Knowledge-Intensive Multi-Hop Question Answering (2024)

Abstract: Large Language Models (LLMs) can teach small language models (SLMs) to solve complex reasoning tasks (e.g., mathematical question answering) by Chain-of-thought Distillation (CoTD). Specifically, CoTD fine-tunes SLMs by utilizing rationales generated from LLMs such as ChatGPT. However, CoTD has certain limitations that make it unsuitable for knowledge-intensive multi-hop question answering: 1) SLMs have a very limited capacity in memorizing required knowledge compared to LLMs. 2) SLMs do not possess the same powerful integrated abilities in question understanding and knowledge reasoning as LLMs. To address the above limitations, we introduce Decompose-and-Response Distillation (D&R Distillation), which distills two student models, namely Decomposer and Responser separately. The two models solve a knowledge-intensive multi-hop question through an interactive process of asking and answering subquestions. Our method offers two advantages: 1) SLMs have the capability to access external knowledge to address subquestions, which provides more comprehensive knowledge for multi-hop questions. 2) By employing simpler subquestions instead of complex CoT reasoning, SLMs effectively mitigate task complexity and decrease data prerequisites. Experimental results on three knowledge-intensive multi-hop question answering datasets demonstrate that D&R Distillation can surpass previous CoTD methods, even with much less training data.

## introduction large language models are capable of answering complex questions (e.g., mathematical questions) figure 1 : a comparison of d&r distillation (ours) and cotd @xcite . cotd teaches one slm to output all intermediate reasoning steps and the final answer at once, struggling on knowledge-intensive multi-hop questions. d&r distillation teaches two slms to interact by asking and answering subquestions, leading them to collectively reach the final answer. by generating step-by-step natural language reasoning paths, namely chains-of-thoughts (cots) @xcite . however, the ability to solve complex reasoning tasks through cot prompting is considered an emergence that appears in very large models with at least tens of billions of parameters @xcite , such as palm of 540b @xcite , gpt-3 of 175b @xcite , and llama of 70b @xcite . recent works have proposed to transfer the rea-soning ability of large models to small language models (slms) through chain-of-thought distillation (cotd) @xcite @xcite . specifically, as shown in the upper part of figure 1 , they leverage the llm (e.g., chat-gpt) to generate high-quality rationales and finetune a slm with rationale-augmented questionanswer pairs. cotd has successfully enhanced slms' reasoning ability on many reasoning tasks, such as arithmetic reasoning @xcite , commonsense reasoning @xcite , and symbolic reasoning @xcite . however, previous cotd works did not effectively address knowledge-intensive reasoning tasks such as multi-hop question answering @xcite . unlike arithmetic reasoning and commonsense reasoning, knowledge-intensive reasoning tasks pose greater challenges due to their requirement for both background knowledge and the ability to perform multistep reasoning. cotd has two limitations that render it unsuitable for teaching slm to reason over knowledge-intensive multi-hop question answering. to address the aforementioned limitations, motivated by question decomposition for answering complex questions @xcite , we propose a novel method to teach slms to reason for knowledge-intensive multi-hop questions, namely deompose-and-response distillation (d&r distillation, as shown in figure 1 ). specifically, we propose to prompt llm in a self-ask-self-ans strategy by iteratively asking subquestions and responding with intermediate answers. then we separately distill two student models, namely decomposer and responser. the deomposer is responsible for asking subquestions and determining the final answer based on current interaction history. the responser is responsible for answering subquestions by leveraging relevant background knowledge obtained from an external knowledge base. by formatting the reasoning process as a sequence of generating subquestions and intermediate answers, these two student models effectively address knowledge-intensive multi-hop questions within an interactive framework. compared with previous chain-of-thought distillation methods, our method offers two notable advantages: 1) by reasoning in an interactive manner, our method allows student models to utilize external knowledge with each retrieval focusing on a subquestion. compared to previous works relying solely on parameter knowledge or one-step retrieval augmentation @xcite , our method provides a more comprehensive collection of relevant knowledge required to answer multi-hop questions. 2) we transform the process of solving a reasoning question into two interrelated and decoupled subtasks: decomposing the complex question and solving a series of simpler subquestions. d&r distillation effectively reduces the overall task difficulty while significantly reducing the amount of data required for distillation. we evaluate the effectiveness of our method on three knowledge-intensive multi-hop question answering datasets: hotpotqa, strategyqa, and 2wikimultihopqa. experimental results demonstrate that d&r distillation significantly improves the knowledge-intensive reasoning ability of slms with approximately 1/10 of the full training data. notably, our method with two 220m slms (t5base) outperforms chain-of-thought prompting with an 11b (50 times larger) llm (flan-t5-xxl) on hotpotqa and 2wikimultihopqa. chain-of-thought distillation have been proposed to distill the cot reasoning ability of llms into slms @xcite @xcite , because the cot reasoning ability is considered as an emergent ability which enables llm to generate intermediate reasoning steps with cot prompting @xcite (e.g. let's think step by step). to augment chain-of-thought distillation (cotd) with external knowledge, @xcite augment slms with documents retrieved by a one-step retriever from the external knowledge base. however, cotd is less effective for knowledge-intensive multi-hop question answering tasks @xcite , where both factual knowledge and multi-hop reasoning are important to generate accurate rationale. in this paper, we propose to distill two student models and solve a knowledge-intensive multi-hop question by facilitating an interactive process of asking and answering subquestions between the two student models. question decomposition @xcite has long been a crucial technique for understanding and solving complex questions. recent works also utilize question decomposition to improve the reasoning ability of llms. @xcite enhances the cot reasoning ability of llms by decomposing questions into subquestions and sequentially solving subquestions. @xcite explicitly asks llm itself follow-up subquestions before answering the original question and answers subquestions with an external search engine. @xcite learns a semantic decomposition of the original question into a sequence of subquestions and uses it to train two models designated for question decomposition and resolution. unlike the aforementioned works, we focus on teaching small language models to reasoning for knowledge-intensive multi-hop questions with llm generations. we achieve this by distilling two student models to interactively ask and answer subquestions. ## method in this section, we provide a detailed description of our method. as illustrated in figure 2 , d&r distillation can be divided into three stages: ## conclusion in this paper, we proposed decompose-and-response distillation (d&r distillation) which enhances the reasoning capabilities of small language models (slms) on knowledge-intensive multi-hop question answering. our approach involves dis-tilling two student models separately, with one student model focusing on decomposing subquestions and another student model focusing on answering subquestions with retrieved background knowledge. through extensive experiments, we showed that d&r distillation outperforms previous chain-of-thought distillation approaches with much less training data.