P oke MQA : Programmable knowledge editing for Multi-hop Question Answering (2024)

Abstract: Multi-hop question answering (MQA) is one of the challenging tasks to evaluate machine’s comprehension and reasoning abilities, where large language models (LLMs) have widely achieved the human-comparable performance. Due to the dynamics of knowledge facts in real world, knowledge editing has been explored to update model with the up-to-date facts while avoiding expensive re-training or fine-tuning. Starting from the edited fact, the updated model needs to provide cascading changes in the chain of MQA. The previous art simply adopts a mix-up prompt to instruct LLMs conducting multiple reasoning tasks sequentially, including question decomposition, answer generation, and conflict checking via comparing with edited facts. However, the coupling of these functionally-diverse reasoning tasks inhibits LLMs’ advantages in comprehending and answering questions while disturbing them with the unskilled task of conflict checking. We thus propose a framework, Programmable knowledge editing for Multi-hop Question Answering (PokeMQA), to decouple the jobs. Specifically, we prompt LLMs to decompose knowledge-augmented multi-hop question, while interacting with a detached trainable scope detector to modulate LLMs behavior depending on external conflict signal. The experiments on three LLM backbones and two benchmark datasets validate our superiority in knowledge editing of MQA, outperforming all competitors by a large margin in almost all settings and consistently producing reliable reasoning process.

## introduction multi-hop question answering (mqa) requires a sequence of interacted knowledge facts to reach the final answer. for instance, considering the two-hop question in figure 1 , it is necessary to deduce the intermediate answer inter miami through the fact "messi plays for inter miami", and then deduce the final answer na through another fact "inter miami is located in north america"). mqa poses a great challenge to reasoning abilities of question answering systems @xcite @xcite . thanks to the natural language comprehending and reasoning brought by largescale pre-training, large language models (llms) have proven its indispensable utility in mqa tasks @xcite @xcite . however, the knowledge within llms may be factually wrong or become invalid over time. to ensure the correctness of llms without necessitating expensive retraining @xcite , technique of knowledge editing has been carried out to provide efficient and targeted updates on model behaviors @xcite @xcite . there are two popular approaches: parameter-modification based editing and memorybased editing. the former one modifies the internal model weights according to edited facts through meta-learning, fine-tuning, or knowledge locating @xcite @xcite . the latter approach leverages an external memory to explicitly store the edited facts (or termed as edits) and reason over them, while leaving llms parameters unchanged @xcite . memory-based model editing is generally adopted due to its simpleness and agnostic to backbone llms. in the context of mqa, mello @xcite is first proposed by designing a multipurpose prompt to instruct llms conducting the reasoning tasks of question decomposition and knowledge editing sequentially. in particular, after decomposing the multi-hop questions, llms generate a tentative answer for each subquestion and then detect whether there exists factual conflict between tentative answer and edited facts in memory (e.g., statements of "the current british prime minister is rishi sunak" and "the current prime minister of the uk is liz truss" are factually incompatible with each other). by repeatedly prompting llms, mello reaches the answer of multi-hop question. however, the coupling of question decomposition and knowledge editing imposes considerable demands on llms to precisely perform reasoning as demonstrations in context. first, the knowledge editing requires llms to fully understand the semantics of two candidate facts and then make conflict detection based on the factual compatibility between them. in the few-shot prompting, llms are prone to underfit this editing logic due to inadequate supervision signals, especially when embedded in a more complex task @xcite , i.e. question decomposition. second, within a unified prompt, the incorporation of knowledge editing instruction introduces noise to question decomposition in the similar way. such superposed noise prevents llms from fully focusing on parsing the syntactic structure of multi-hop questions to precisely identify the subquestions. thus, we propose programmable knowledge editing for multi-hop question answering (pokemqa), where we decouple the two essential tasks, i.e. question decomposition and knowledge editing, to alleviate burdens on llms while introducing auxiliary knowledge prompt to assist question decomposition. specifically, we offload the conflict detection in knowledge editing with a programmable scope detector, which is used to detect whether a subquestion lies within the scope affected by any edited facts in semantic space (challenge #1). a two-stage scope detector is designed: in pre-detection stage, we efficiently filter out a substantial number of irrelevant edits; in conflict-disambiguation stage, we perform precise retrieval on the remaining few candidate edits. our two-stage framework provides both computational efficiency and expressiveness given the high volume of edited facts in real scenarios. the retrieved edits are used to calibrate llms behavior. moreover, we propose a knowledge prompt to augment parse analysis in the process of question decomposition (challenge #2). the knowledge prompt recognizes key entity from input question and retrieves its external information from a knowledge source to trig the correct decomposition. additionally, we observe that the multi-hop question answering process may use the outdated or non-existent facts, but occasionally ends up with the right answer. we refer to this situation as unreliable reasoning (as shown in figure 1 ). in order to faithfully evaluate models' reasoning ability, we propose a new metric called hop-wise answering accuracy (hop-acc), measuring the extent how llms follow demonstrations, conduct question decomposition step by step, and generate desired answer to each step towards solving the multi-hop question. ## multi-hop question answering under knowledge editing notations. following previous work @xcite , we denote a fact as a triplet (s, r, o), consisting of the subject s, object o, and relation r between them, such as (messi, play for, inter miami). an edited fact (i.e., edit) is the knowledge fact that we want to update and is represented in the same form (s, r, o), such as (messi, play for, boca juniors). we consider a multi-hop question q, where answering q requires sequentially querying and retrieving multiple facts. these facts are presented in the order they were queried, forming a chain of facts ⟨(s 1 , r 1 , o 1 ) , . . . , (s n , r n , o n )⟩, where s i+1 = o i and o n is the final answer, which uniquely represents an inter-entity path @xmath0, o 1 , . . . , o n ⟩. it should be noted that except for s 1 , all the other entities o 1 , . . . , o n in p do not appear in q and need to be deduced either explicitly or implicitly through factual reasoning (like inter miami and north america in the multi-hop question in figure 1 ). if we replace the invalid fact (s i , r i , o i ) with edit @xmath1, r i , o * i ) in a multi-hop question, due to the cascading effect caused by the edited fact, the chain of facts accordingly changes to ## experimental setup we conduct experiments on mquake, a knowledge editing benchmark. it includes mquake-cf-3k, which is based on counterfactual edits, and mquake-t, which involves temporal knowledge updates. these two datasets consist of several k-hop questions (@xmath2, 3, 4}), each associated with at least one edit. more statistics can be found in appendix c. ## conclusion in this work, we propose a novel programmable knowledge editing method (pokemqa) to improve mqa performance and address unreliable reasoning. pokemqa leverages a scope detector to align llms' behavior with edited facts and incorporates auxiliary knowledge prompt to enrich contextual information. extensive experiments across three llms indicate that pokemqa helps llms answer multi-hop questions in a precise and reliable way.