Best of Both Worlds: Towards Improving Temporal Knowledge Base Question Answering via Targeted Fact Extraction (2023)

Abstract: Temporal question answering (QA) is a special category of complex question answering task that requires reasoning over facts asserting time intervals of events. Previous works have predominately relied on Knowledge Base Question Answering (KBQA) for temporal QA. One of the major challenges faced by these systems is their inability to retrieve all relevant facts due to factors such as incomplete KB and entity/relation linking errors. A failure to fetch even a single fact will block KBQA from computing the answer. Such cases of KB incompleteness are even more profound in the temporal context. To address this issue, we explore an interesting direction where a targeted temporal fact extraction technique is used to assist KBQA whenever it fails to retrieve temporal facts from the KB. We model the extraction problem as an open-domain question answering task using off-the-shelf language models. This way, we target to extract from textual resources those facts that failed to get retrieved from the KB. Experimental results on two temporal QA benchmarks show promising ~30% & ~10% relative improvements in answer accuracies without any additional training cost.

## introduction complex question answering involves the integration of multiple facts identified and extracted from disjoint pieces of information @xcite @xcite . two critical components in systems trying to achieve this are: 1) knowledge source -to retrieve relevant facts, and 2) reasoning -to reason over relevant facts to derive the final answer. both § this work was done when nithish kannen was at ibm research, india as a summer intern. at that time he was also an (integrated bachelor's+master's) electrical engineering student at iit kharagpur, india. in this paper, we present a novel combination of successful elements from past approaches, i.e., kb-based and text-corpus based, for complex qa, in a way to overcome their individual limitations. our combination strategy is to use a targeted extraction technique that would assist kbqa whenever it fails. kbqa failure is when it fails to retrieve relevant facts needed to answer the question from the kb because of reasons such as incomplete kb and inaccurate entity/relation linking. identification of these specific points of kbqa failure is a critical step in our approach, which is enabled by decomposition of the question's logical representation obtained by semantic parsing. the targeted extraction technique compensates for those kb failures by extracting facts from textual resources in an open-domain qa fashion. concretely, we make effective utilization of the kb (reliable but not exhaustive) and the textual resources (vast but noisy), essentially combining the best of both worlds. in this work, we focus on temporal questions. they additionally involve reasoning over temporal facts, i.e., assertions on points and intervals in time of events 1 . for example, to answer who was the president of the united states during world war 2?, we need to know the temporal facts of both world war 2 and the list of all us presidents. the main contributions of our work are: 1) a novel combination of kbqa and textual-extraction for temporal question answering. 2) λ-calculus based semantic representation to identify kbqa gaps. 3) an open-domain qa style approach for targeted extraction of temporal facts from textual resources using off-the-shelf models. we show that it is possible to achieve significant improvements on two temporal qa benchmarks even without any task-specific training and believe the promising initial results will foster research in this direction. related work: graft-net @xcite and @xcite utilize both kb and textual resources but do not address temporal reasoning. unlike approaches using end-to-end neural models, we adopt a modular approach as in @xcite because of the flexibility it offers to combine textual extraction with the kbqa, with additional benefits such as interpretability and easy domain-adaptation. another main difference in our textual extraction approach is the usage of lms off-the-shelves, hence bypassing the need for domain-specific training and datasets that are hard to obtain. a detailed literature review is presented in a.6. 1 in this paper, we consider entities and facts (represented as triples in kbs) with associated time intervals as events. for example, {world war 2, (start time:1939, end time:1945)}. ## our approach figure 2 shows a block diagram of our proposed approach, with two groups of modules: 1) upper line of kbqa pipeline modules, and 2) lower line of extraction pipeline modules for targeted fact extraction. our strategy is to bank on the kbqa pipeline, owing to its reliability, and trigger extraction pipeline only to aid kbqa when it fails. ## experimental setup we used wikidata as kb and wikipedia as textual resource, and evaluate on two aspects: 1) temporal qa performance of the overall system and 2) targeted temporal fact extraction performance. we experiment with 2 datasets: 1) tempqa-wd (neelam et al., 2022b) with 839 questions and 2) time-questions @xcite . baselines: 1) only-kb -answers only from kbqa, 2) kb+temporaltext -extracting temporal facts only from text and the remaining facts from kb, and 3) open-domain-qa -a state-of-the-art opendomain-qa system called rag @xcite ) that answers purely from text. rag consists of a retriever based on dense passage retrieval (dpr) and a generator based on bart jointly trained. we refer the readers to the original paper for more details on this baseline. ## results and discussion due to discrepancies discussed in a.4, we could not use our kbqa pipeline to evaluate on time-questions dataset. alternatively, we used the end ## conclusion in this paper, we propose an approach to combine the knowledge resources of kb (structured) and text (unstructured) for temporal qa by using textual resources to aid wherever kbqa fails. in our approach, a semantic representation of the question as λ-expression is used to represent the set of facts and reasoning required to answer a question. those facts that failed to get fetched from kbqa are extracted from the textual resources using rcqa style fact extraction. this way, we perform targeted extraction of temporal facts to compensate for kbqa failures. the results of experimental evaluation on two temporal qa benchmark show the effectiveness of our approach even without any additional training cost. we additionally conduct error analysis to highlight the scope of improvement in order to guide future work.