From Parse-Execute to Parse-Execute-Refine: Improving Semantic Parser for Complex Question Answering over Knowledge Base (2023)

Abstract: Parsing questions into executable logical forms has showed impressive results for knowledge-base question answering (KBQA). However, complex KBQA is a more challenging task that requires to perform complex multi-step reasoning. Recently, a new semantic parser called KoPL has been proposed to explicitly model the reasoning processes, which achieved the state-of-the-art on complex KBQA. In this paper, we further explore how to unlock the reasoning ability of semantic parsers by a simple proposed parse-execute-refine paradigm. We refine and improve the KoPL parser by demonstrating the executed intermediate reasoning steps to the KBQA model. We show that such simple strategy can significantly improve the ability of complex reasoning. Specifically, we propose three components: a parsing stage, an execution stage and a refinement stage, to enhance the ability of complex reasoning. The parser uses the KoPL to generate the transparent logical forms. Then, the execution stage aligns and executes the logical forms over knowledge base to obtain intermediate reasoning processes. Finally, the intermediate step-by-step reasoning processes are demonstrated to the KBQA model in the refinement stage. With the explicit reasoning processes, it is much easier to answer the complex questions. Experiments on benchmark dataset shows that the proposed PER-KBQA performs significantly better than the stage-of-the-art baselines on the complex KBQA.

## introduction knowledge-base question answering (kbqa) @xcite is a task that answers natural language questions over a given knowledge graph. kbqa has become a hot research topic since it can lead to a large number of applications, e.g., intelligent assistants. * *corresponding author many algorithms @xcite @xcite @xcite have been proposed for kbqa. a notable research line for kbqa is semantic parsing @xcite . it encodes the questions into formal meaning representations/logical forms @xcite , then the logical forms are executed over the knowledge base to give the final answers, which is called parse-execute paradigm. kafsp @xcite is a knowledge-aware fuzzy semantic parsing framework for kbqa. rng-kbqa @xcite is a rank-and-generate approach for solving generalization problem in kbqa. several benchmarks kbqa datasets have also been proposed to promote research in this field. for example, metaqa @xcite , webqsp @xcite and lc-quad @xcite are the datasets for kbqa. recently, due to the complexities of users' information requirements, the more challenging task called complex kbqa @xcite has been studied, which requires multi-step/compositional reasoning. for complex kbqa, it is important that the qa models can be learnt to explicitly model the reasoning processes, which can provide better interpretation of compositional reasoning. hence, @xcite had proposed a benchmark dataset kqa pro and a knowledge-oriented programming language (kopl). kqa pro is a large-scale dataset for complex kbqa and provides explicit reasoing process for each qa pair. kopl @xcite is a compositional and interpretable programming language for complex reasoning. the kopl parser provides human understandable descriptions of logical forms. based on the large-scale pretrained language model bart @xcite , the bart-based kopl has shown impressive results on kqa pro dataset. although kopl can explicitly describe the reasoning processes, the logical forms generated from semantic parser are the intermediate results but not the final answer, which may result in sub-optimized solutions. for example, the slight generation deviation @xcite may cause the logical forms to run incorrectly. thus a natural question arises: can we use these intermediate results to further improve the complex reasoning ability? in this paper, we further explore how to unlock kopl and kbqa model for improving its capability of complex reasoning. we find that the complex reasoning ability can benefit from demonstrating the kopl's intermediate results to the kbqa model. that is, with the intermediate step-by-step reasoning processes from kopl, we can elicit the multi-step reasoning for complex kbqa and achieve better performances compared to kopl. observed from that, we propose a novel parse-execute-refine paradigm for complex kbqa. figure 1 shows an overview of the proposed approach called per-kbqa, which includes a parsing stage, an execution stage, and a refinement stage for complex kbqa. given a question, the parser first utilizes a sequence-to-sequence model to generate the logical forms. since kopl can explicitly describe the reasoning process, we use kopl as the logical forms in the parsing stage. then, the execution stage aligns and executes the generated logical forms over the knowledge base. with that, we can obtain the intermediate step-bystep reasoning processes. finally, in the refinement stage, the intermediate reasoning results are used as execution contexts. we combine the question and the execution contexts as inputs to the kbqa model. the main idea of this paradigm is that the answer may benefit from the intermediate step-bystep reasoning processes. we evaluate the per-kbqa on kqa pro dataset and the extensive experimental results show that 1) the proposed per-kbqa performs significantly better than the existing state-of-the-art baselines, and 2) per-kbqa is a simple method that can further unlock the kbqa model for improving its complex reasoning ability. besides, per-kbqa provides insight into a novel way to combine the intermediate results into the final answer. recently, the kopl parser is proposed from kqa pro @xcite , and it is transparent about the execution process and can provide luxuriant information of the intermediate process, which makes it possible for our proposed method to combine both of the mainstream methods to make full use of their strengths. odqa. open domain question answering aims to answer factoid questions based on a large collection of documents. the mainstream solution for open domain qa is that firstly use a retriever to select a small subset of passages where some of them might contain the answer to the question, and then utilize an extractive or generative reader to identify the answer. the retriever can be sparse vector space models such as tf-idf @xcite and bm25 @xcite , or trainable dense vector space models such as dpr @xcite and mdr @xcite . the extractive reader predicts a span from the retrieved context to answer an open domain question in previous works such as drqa @xcite , hgn @xcite and dpr @xcite . recently, the generative reader achieved further improvements in multiple open domain qa benchmarks, such as fid @xcite , kg-fid @xcite and path-fid @xcite . in our work, we can view the proposed refinement stage as the generative reader, and the proposed parsing and execution question 1: what country is associated with quincy, known as gem city, where mary astor was born? find quincy find mary astor filterstr official name gem city queryrelationqualifier location of birth country quincy nick name: gem city … mary astor data of birth: 1906/5/3 … place of birth country: united states of america knowledge base: alignment predicted kopl: execution context: @xmath0: bart encoder bart decoder stages as the retrieval in odqa. ## per-kbqa in this section, we formulate the complex kbqa task and describe the implementation of three stages of per-kbqa. in this paper, we consider the kopl language that explicitly describes the reasoning process as our logical form. since only the kqa pro @xcite ) dataset provides the kopl, we follow the kqa pro setting. that is, each natural language question is paired with executable logical forms (i.e., kopl). the logical forms can be executed against the kb to obtain the answer. ## experiments in this section, we conduct the experiments to compare our proposed approach with various baselines, and provide detailed analysis and ablation study on the results. dataset: since kopl is a recently proposed parser that explicitly describes the reasoning process and only the kqa pro @xcite dataset provides kopl, we use kqa pro, a challenging dataset for complex question answering over knowledge base for evaluation. in kqa pro training set, each instance consists of the textual question, the corresponding kopl, ten candidate choices, and the golden answer. and there are two settings: open-ended setting and multiple-choice setting. we conduct our experiments in the open-ended setting where the ten candidate choices are not used. besides, it provides a high-quality knowledge base which has rich literal knowledge (attributes and relations about entities) as well as qualifier knowledge (qualifier about attributes and relations). we can execute kopl over the knowledge base to obtain the answer. metrics: we evaluate the qa models based on the accuracy of the test set in kqa pro dataset. following the kqa pro @xcite , we provide in-depth analysis of models' fine-grained reasoning abilities for complex kbqa including multihop, qualifier, comparison, logical, count, verify, and zero-shot, where multi-hop denotes multi-hop questions, qualifier and comparison measure the ability to query the literal knowledge or qualifier knowledge of the relations or attributes, count and verify rely on the intermediate results of entities list, zero-shot requires the qa models to answer the questions related to unseen relations. baselines and implementation details: we reproduce the strong baseline bart based kopl parser proposed in kqa pro @xcite . based on this baseline, we implement three stages of our approach. we also report the published methods in the leaderboard including kvmnet @xcite , srn @xcite , embedkgqa @xcite , rgcn @xcite , graphq ir @xcite bart kopl @xcite , we utilize two independent bart-base models to instantiate the parsing and refinement stages respectively. we set the learning rate for bart parameters as 3e-5, the learning rate for other parameters as 1e-3, and the weight decay as 1e-5. we used the optimizer adam (kingma and ba, 2014) for all models. please note that our proposed execution stage is simple, efficient and requires no learning. ## conclusion in this work, we propose a novel approach called per-kbqa for complex question answering over the knowledge base. our method consists of three stages: parsing, execution and refinement. the parsing stage generates logical forms conditioned on the question. the execution stage alleviates the problem of slight generation deviation via the alignment to the knowledge base and obtains the intermediate results. the refinement stage combines the logical forms and corresponding intermediate results as the reasoning process and regards it as execution context to generate the answer. experiments on kqa pro benchmark show that our method can significantly improve the performance in terms of most evaluated measures.