Gradually Excavating External Knowledge for Implicit Complex Question Answering (2023)

Abstract: Recently, large language models (LLMs) have gained much attention for the emergence of human-comparable capabilities and huge potential. However, for open-domain implicit question-answering problems, LLMs may not be the ultimate solution due to the reasons of: 1) uncovered or out-of-date domain knowledge, 2) one-shot generation and hence restricted comprehensiveness. To this end, this work proposes a gradual knowledge excavation framework for open-domain complex question answering, where LLMs iteratively and actively acquire extrinsic information, then reason based on acquired historical knowledge. Specifically, during each step of the solving process, the model selects an action to execute, such as querying external knowledge or performing a single logical reasoning step, to gradually progress toward a final answer. Our method can effectively leverage plug-and-play external knowledge and dynamically adjust the strategy for solving complex questions. Evaluated on the StrategyQA dataset, our method achieves 78.17% accuracy with less than 6% parameters of its competitors, setting new SOTA in the ~10B LLM class.

## introduction recently, powerful llms such as chatgpt, gpt4 (openai, 2023), palm @xcite , llama and its variances @xcite , exhibiting human-alike ability in conversation. it is believed the llms memorize knowledge in their parameters from the vast pretraining data @xcite . nonetheless, they could still fail to solve open-domain implicit complex questions. in real-world applications, users might ask questions in arbitrary domain that requires specific knowledge, and expect the model to return not only syntactically fluent but also factually correct answers. beyond open-domain, the questions can also be multi-step and implicit, consisting of multiple sub-questions that cannot be directly identified from the question language, but require logical reasoning to form a problem-solving strategy. because of the above challenging characteristics, how to answer open-domain implicit complex questions remains an open question. for example, in the upper part of fig. 1 , an implicit complex question "did any citizen of san antonio vote for boris johnson" confuses the llm because there is no direct information about individual voting history. however, the strategy of searching voting history is invalid does not mean the question is unsolvable. on the top right corner of fig. 1 , the question can be decomposed into subquestions about boris johnson and san antonio, respectively, and a strategy of checking citizenship can be easily unsealed from the background knowledge (marked in red). following the strategy, the answer is straightforward because us citizens cannot vote in uk elections. now if the question is re-asked with the hint of 'citizenship contradiction', the llm can successfully recall its inner knowledge about boris johnson and san antonio, hence correctly answering the question. however, different from existing works that try to design manual prompts @xcite to serve as the hint in the above example, we want to stress that this approach is not always valid. because the key 'citizenship contradiction' is not directly linked to the question text, but relies on the gradually increased knowledge during the solving process (e.g., background about boris johnson and san antonio). another problem for solving open-domain complex questions with llms is the finite pretrained knowledge. in the bottom part of fig. 1 , we rewrite the question with two less well-known entities out of the llm's knowledge scope, and it fails again due to limited knowledge. therefore, in this work, we propose a pipeline (1) in the upper part, the llm fails to answer the question with the one-shot generation, for there is no off-the-shelf answer or evidence to this question. however, the question can be decomposed into several sub-questions and be solved once the citizenship contradiction is identified. if the hint of 'citizenship contradiction' is also given, the llm can successfully solve the question with the inner knowledge now. (2) but for the bottom case with less well-known entities, the llm fails again due to a lack of specialized knowledge about 'aisin-gioro yizhu' and hence rejects to answer. moreover, the strategy of the 'political system' is not likely to be discovered from the question text only, unless enough knowledge is provided. 'citizenship contradiction' is also a possible solution. during intermediate steps, geek can adjust the rest sub-questions based on the gradually increased external knowledge, hence forming a valid strategy like in fig. 1 . considering there usually exist multiple valid solutions for a question, we enable geek to branch out different sub-questions during the solving process, thus exploring a strategy space and improving the final accuracy. we verified geek on the challenging strate-gyqa dataset @xcite , which consists of open-domain multi-step implicit questions. geek achieves 78.17% accuracy with less than 6% parameters of its competitors, refreshing the new sota for llms under ∼300b scale. our main contributions are threefold: ## related work retrieving external information is a widely adopted method that can provide flexible knowledge to extend llms for specific-domain tasks. @xcite ) leverages retrievers (e.g., @xcite or dpr @xcite ) to collect relevant passages, and fuse them in the decoder for a final answer. similarly, @xcite proposes aiso, which performs multi-round retrieval with different retriever models of bm25, dpr, and link. it then synthesizes the retrievals for a comprehensive conclusion. hopretriever @xcite adopts a multi-hop manner, which identifies significant entities in previous retrievals, and then uses the entities as queries for next step retrieval. multi-step implicit question answering involves complex questions that consist of several singlestep questions, whose answers can be directly founded in reference context or inferred via logical deduction. the question is implicit if the decomposition strategy cannot be formulated merely from the question text. strategyqa @xcite is a dataset for multi-step implicit question answering, including human-annotated solutions in the form of decomposition questions and corresponding fact sentences from wikipedia. however, while human beings can achieve 87% accuracy @xcite , the dataset has proved to be very challenging for language models to solve. merely improving the quality of retrieval @xcite or decomposition strategy @xcite cannot effectively boost final answer accuracy. several previous studies adopt the design of iterative retrieval and reasoning to solve multi-step complex questions. irgr @xcite performs iteratively retrieval to search for suitable premises. react @xcite lets the model first generate a reasoning sentence about what next action (e.g., search via web api) to be performed, and then execute the selected action. maieutic prompting @xcite introduces a maieutic tree that recursively entails component statements, and uses the concept of 'logical integrity' to verify each step. rr @xcite combines cot @xcite with retrieval to verify the correctness of the reasoning process, and achieved an accuracy of 77.73% on the strategyqa dataset, the sota for llms below ∼300b scale at that time. nonetheless, most of the previous studies us-ing iteratively reasoning for multi-step question answering only focus on datasets such as hot-potqa @xcite , which do not involve implicit questions. instead, their method assumes a straightforward direction from one step to another. for example, ircot @xcite directly uses the 'thought' (intermediate result) from the last step as the query for next-step retrieval, which highly relies on the strong connections between the entities from each step. different from the abovelisted methods, our geek modeling the process of purposely excavating external knowledge by composing sub-questions, and formulating a complete strategy gradually during the solving process. ## problem definition in this work, we focus on open-domain implicit complex question answering. given a question q, the model is asked to derive the final answer z. specifically, q is open-domain and hence requires some certain background facts to solve, denoted as @xmath0. the facts come from an external corpus c (e.g., wikipedia), not necessarily included in the model's pretraining dataset. the question q is also multi-step, meaning that it can be decomposed into several decomposition questions @xmath1. each d i corresponds to a background fact f i . noted that d is usually implicit from q, which means some d i can only be formulated until enough facts f i ⊂ f is uncovered. under the geek scenario, we define the question state q at step t as: ## gradually excavating external knowledge in this section, we introduce geek for opendomain complex question answering. geek consists of three components, the core model, retriever and extractor model. iteratively, the core model selects actions to perform from an action space a, conditioned on the current question state q. then the selected action is executed and the question state is updated, gradually accumulating external knowledge until a final answer can be drawn. ## case study in fig. 3 we show a full inference process of geek. the inference involves in total three iterations. in order to save space, we omit the input question state for each step, but show it in the top right corner. marked in colored numbers, the question state is gradually enriched as the decomposition questions and corresponding facts are added. also, for each action selection step, we circle out the core model's choice. some repeated prompts are also abbreviated, due to the page limitation. as can be observed in the example, the geek decomposes the original question into three subquestions, and takes the strategy of 'temporal confliction' to solve the problem. for sub-questions '(1)' and '(2)', external knowledge is retrieved and the original pseudo answers are verified and corrected (e.g., '1599' → '1623'). the strategy is also dynamically adjusted as more facts are acquired (e.g., 'is 1599 before 1865?' → 'is 1865 before 1623?'). for the last logical sub-question that requires no external knowledge, the core model chooses to self-answer it without calling retriever and extractor. when there is enough information, the final answer 'no' is given with reasons. ## experiment in this section, we compare geek with previous baselines and conduct ablation studies to analyze the contributions of specific modules. ## conclusion we present geek, a pipeline to progressively excavate external knowledge for boosting llm's capability in solving open-domain multi-step implicit questions. interactively, geek decomposes the question into explicit sub-questions to retrieve external knowledge, and the accumulated knowledge enlightens the model to form a better strategy in turn. the proposed method also provides explainability by showing the full reasoning process, supported by retrieved evidence. by using se, geek can also explore the strategy space and try different approaches to solve the complex question, which increases the performance further. experiment results justify our design. with geek, 78.17% accuracy is achieved using less than 6% size of the competitors, refreshing the sota accuracy for ∼10b llms. as an alternative to the paradigm of scaling for larger models and more pretraining data, we hope this research could inspire more future works to investigate how to organically excavate external knowledge and progressively formulate strategies for solving open-domain implicit questions.