B eam A gg R : Beam Aggregation Reasoning over Multi-source Knowledge for Multi-hop Question Answering (2024)

Abstract: Large language models (LLMs) have demonstrated strong reasoning capabilities.Nevertheless, they still suffer from factual errors when tackling knowledge-intensive tasks.Retrieval-augmented reasoning represents a promising approach.However, significant challenges still persist, including inaccurate and insufficient retrieval for complex questions, as well as difficulty in integrating multi-source knowledge.To address this, we propose Beam Aggregation Reasoning (BeamAggR), a reasoning framework for knowledge-intensive multi-hop QA.BeamAggR explores and prioritizes promising answers at each hop of question.Concretely, we parse the complex questions into trees, which include atom and composite questions, followed by bottom-up reasoning.For atomic questions, the LLM conducts reasoning on multi-source knowledge to get answer candidates.For composite questions, the LLM combines beam candidates, explores multiple reasoning paths through probabilistic aggregation, and prioritizes the most promising trajectory.Extensive experiments on four open-domain multi-hop reasoning datasets show that our method significantly outperforms SOTA methods by 8.5%.Furthermore, our analysis reveals that BeamAggR elicits better knowledge collaboration and answer aggregation.

## introduction large language models have showcased impressive performance across various nlp tasks @xcite . furthermore, chain-ofthought (cot) prompting further enhances the reasoning capabilities of llms @xcite @xcite . however, when the question surpasses the knowledge boundaries of llms, it leads to factual errors, also known as hallucination. retrieval-augmented generation (rag) assists llms by retrieving supporting knowledge * corresponding authors: ming liu, qianglong chen to alleviate factual errors, thereby drawing significant attention within the research community @xcite . early attempts adopt a retrieve-then-read framework for one-time retrieval @xcite @xcite . they use the original complex question as the retrieval query and achieve satisfactory performance in single-hop questions @xcite . nonetheless, when confronted with complex multi-hop questions, one-time retrieval suffers from the issues of inaccurate and irrelevant retrieval, which greatly impair their performance in multi-hop reasoning. recent work introduces iterative multi-round retrieval @xcite @xcite @xcite . they use the content generated by llms for retrieval and, in turn, use the newly retrieved content for reasoning. through the iterative alternation between retrieval-augmented reasoning and reasoning-augmented retrieval, the retrieval is substantially improved. meanwhile, a portion of research decomposes complex questions into simple sub-questions, employing sub-question retrieval to obtain more precise information @xcite @xcite . however, there are still significant issues with these methods. iterative retrieval is tough to achieve precise retrieval aligned with the model's reasoning. sub-question retrieval grapples with the challenge of accurately aggregating answers, which causes cascading errors. besides, when it comes to the open-domain setting, relying solely on knowledge from a single source proves inadequate for complex questions. introducing multisource knowledge may encounter knowledge conflicts, rendering efficient collaboration challenging, thus impeding its applications. to address the aforementioned challenges, our research focuses on the following question: how can models adaptively select and integrate knowledge from different sources during the reasoning, while reducing cascading errors in subquestions aggregation? building upon this motivation, we propose beam aggregation reasoning (beamaggr) framework for knowledge-intensive multi-hop reasoning. concretely, our method consists of three modules. (i) question decomposition: we begin by leveraging llms to decompose complex questions and convert them into question trees. the root node contains the original complex question, leaf nodes contain atomic sub-questions, and intermediate nodes contain composite questions that require compositional (or comparative) reasoning to obtain answers. afterward, we traverse the question tree in post-order, employing bottom-up reasoning. (ii) complementary multi-source reasoning: for atomic questions, we conduct multisource reasoning, followed by fine-grained answer aggregation, thus fostering knowledge collaboration. the aggregated answers are then normalized into a probability distribution, serving as candidates for beam aggregation. (iii) beam aggregation: for composite questions, we enumerate the combinations of their dependent sub-questions and conduct reasoning. finally, the reasoning results are probabilistically aggregated, and the most promising predictions are selected. in summary, our method explores reasoning trajectories at each hop of questions and prioritizes paths with higher likelihoods, thereby bringing reasoning insight and reducing cascading errors. besides, the complementary multi-source reasoning helps alleviate knowledge omission and conflict. we evaluate our method on four open-domain multi-hop reasoning datasets: hotpotqa @xcite , 2wikimqa @xcite , musique @xcite , and bamboogle @xcite . the experiments are conducted using gpt-3.5-turbo @xcite and mistral @xcite . experimental results show that our method significantly outperforms the baselines on four datasets, with an average improvement of 8.5% compared to the previous state-of-the-art method, demonstrating its superiority. furthermore, thorough analysis reveals the superiority of our approach to knowledge collaboration and answer aggregation. our contributions can be summarized as follows: compared to their approach, our method integrates diverse knowledge via complementary multisource reasoning. besides, it explores reasoning trajectories at each hop of questions and prioritizes the most promising path, thereby eliciting better answer aggregation and reducing cascading errors. ## beam aggregation reasoning beam aggregation reasoning decomposes complex questions into trees and conducts bottom-up reasoning. throughout the bottom-up reasoning, it intricately amalgamates diverse knowledge sources, thereby mitigating the dearth of knowledge. furthermore, beam aggregation probabilistically consolidates answers and selects reasoning pathways, consequently diminishing cascading errors. firstly, we decompose complex questions, then perform bottom-up multi-source knowledge reasoning in a post-order traversal manner, and employ beam aggregation at intermediate nodes until reaching the root node to obtain the final answer. the procedure is depicted in algorithm 1, with an overview of the methodology illustrated in figure 2 . we will introduce question decomposition in § 3.1, multi-source reasoning in § 3.2 and beam aggregation in § 3.3. detailed definitions of notations in the algorithm and formulas can be found in algorithm 1 beam aggregation reasoning require: complex multi-hop questions, q require: multi-source knowledge retriever, r require: large language model, llm require: candidate size in aggregation, k 1: q decomp " llm pqq 2: for n piq in postordertraversepq decomp q do 3: ## analysis 6.1 beamaggr facilitates knowledge collaboration to investigate the impact of beam aggregation on knowledge collaboration, we conduct a prelimi22.1% 22.4% 26.3% 29.2% hotpotqa 24.0% 12.2% 25.1% 38.6% musique 13.6% 19.2% 34.0% 33.3% 2wikimqa 27.4% 32.7% 11.4% 28.6% bamboogle close book parameter wikipedia serp close book parameter wikipedia serp close book parameter wikipedia serp close book parameter wikipedia serp nary experiment. we track the model's utilization of knowledge from different sources during the reasoning process, as depicted in figure 4 . the polarity aggregation method based on log-probs excessively relies on reasoning on a single source of knowledge. in more than 2/3 of cases, it can only utilize knowledge from a single source. in contrast, our approach facilitates better knowledge collaboration in reasoning. in summary, our beam aggregation can effectively employ knowledge from multiple sources at a finer granularity. furthermore, we carry out a more detailed examination of the proportions of each type of knowledge in reasoning. we track the contribution of different knowledge sources to the final answer, as illustrated in figure 5 . disparities in reasoning contributions across various datasets are observed. 2wikimqa and musique tend to favor external reasoning, bamboogle leans towards internal reasoning, and hotpotqa strikes a balance among different types of knowledge. it is noteworthy that, without manual adjustment of the weight of knowledge, beam aggregation can adaptively adjust its proportions during the reasoning process. this indicates that our method can effectively integrate multi-source knowledge. ## conclusion this paper introduces beamaggr for knowledgeintensive multi-hop reasoning. beamaggr utilizes a divide-and-conquer strategy, breaking down complex questions into a tree structure and conducting reasoning in a bottom-up fashion. at each step of reasoning, beamaggr builds upon previous candidates to identify the most likely reasoning path. furthermore, it incorporates multi-source reasoning to enhance knowledge collaboration. overall, beamaggr facilitates multi-hop reasoning through precise sub-question retrieval, efficient knowledge collaboration, accurate answer aggregation, and broad exploration of reasoning trajectories. extensive experiments on four open-domain multi-hop reasoning datasets demonstrate its effectiveness.