KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Base (2022)

Abstract: Complex question answering over knowledge base (Complex KBQA) is challenging because it requires various compositional reasoning capabilities, such as multi-hop inference, attribute comparison, set operation, etc. Existing benchmarks have some shortcomings that limit the development of Complex KBQA: 1) they only provide QA pairs without explicit reasoning processes; 2) questions are poor in diversity or scale. To this end, we introduce KQA Pro, a dataset for Complex KBQA including around 120K diverse natural language questions. We introduce a compositional and interpretable programming language KoPL to represent the reasoning process of complex questions. For each question, we provide the corresponding KoPL program and SPARQL query, so that KQA Pro can serve for both KBQA and semantic parsing tasks. Experimental results show that state-of-the-art KBQA methods cannot achieve promising results on KQA Pro as on current datasets, which suggests that KQA Pro is challenging and Complex KBQA requires further research efforts. We also treat KQA Pro as a diagnostic dataset for testing multiple reasoning skills, conduct a thorough evaluation of existing models and discuss further directions for Complex KBQA. Our codes and datasets can be obtained from https://github.com/shijx12/KQAPro_Baselines .

## introduction thanks to the recent advances in deep models, especially large-scale unsupervised representation learning @xcite , question answering of simple questions over knowledge base (simple kbqa), i.e., single-relation factoid questions @xcite , begins to saturate @xcite @xcite . however, tackling complex questions (complex kbqa) is still an ongoing challenge, due to the unsatisfied capability of compositional reasoning. as shown in to address these problems, we create kqa pro, a large-scale benchmark for complex kbqa. in kqa pro, we define a knowledge-oriented programming language (kopl) to explicitly describe question 1: sparql: kopl: when did cleveland cavaliers pick up lebron james select distinct ?qpv where { ?@xmath0:@xmath1. ?@xmath2:@xmath3. ?@xmath4. [ <pred:@xmath5; <pred:@xmath6; <pred:@xmath7. } queryrelationqualifier drafted by point in time find cleveland cavaliers find lebron james question 2: who is taller, lebron james jr. or his father? sparql: select ?e where { { ?@xmath8." . } union { ?@xmath9." . ?@xmath10. } ?@xmath11. } order by desc @xcite limit 1 kopl: relate father selectbetween height greater find lebron james jr. find lebron james jr. knowledge base drafted by point in time: 26 june 2003 entity with literal knowledge relational knowledge between entities qualifier knowledge about facts cleveland cavaliers incept: 1970; social media followers: 3,242,471 point in time:6 january 2021 lebron james jr. height: 188 centimetre; mass: 80 kilogram; date of birth: 6 october 2004; father child lebron james height: 206 centimetre; mass: 113 kilogram; work period (start): 2003 population: 199,110 point in time:2010 akron p la c e o f b ir th figure 1 : example of our kb and questions. our kb is a dense subset of wikidata @xcite , including multiple types of knowledge. our questions are paired with executable kopl programs and sparql queries. the reasoning process for solving complex questions (see fig. 1 ). a program is composed of symbolic functions, which define the basic, atomic operations on kbs. the composition of functions well captures the language compositionality (baroni, 2019). besides kopl, following previous works @xcite , we also provide the corresponding sparql for each question, which solves a complex question by parsing it into a query graph. compared with sparql, kopl 1) provides a more explicit reasoning process. it divides the question into multiple steps, making human understanding easier and the intermediate results more transparent; 2) allows humans to control the model behavior better, potentially supporting human-in-the-loop. when the system gives a wrong answer, users can quickly locate the error by checking the outputs of intermediate functions. we believe the compositionality of kopl and the graph structure of sparql are two complementary directions for complex kbqa. to ensure the diversity and scale of kqa pro, we follow the synthesizing and paraphrasing pipeline in the literature @xcite , first synthesize large-scale (canonical question, kopl, sparql) triples, and then paraphrase the canonical questions to natural language questions (nlqs) via crowdsourcing. we combine the following two factors to achieve diversity in questions: (1) to increase structural variety, we leverage a varied set of templates to cover all the possible queries through random sampling and recursive composing; (2) to increase linguistic variety, we filter the paraphases based on their edit distance with the canonical utterance. finally, kqa pro consists of 117,970 diverse questions that involve varied reasoning skills (e.g., multi-hop reasoning, value comparisons, set operations, etc.). besides a qa dataset, it also serves as a semantic parsing dataset. to the best of our knowledge, kqa pro is currently the largest corpus for nlq-to-sparql and nlq-to-program tasks. we reproduce the state-of-the-art kbqa models and thoroughly evaluate them on kqa pro. from the experimental results, we observe significant performance drops of these models compared with on existing kbqa benchmarks. it indicates that complex kbqa is still challenging, and kqa pro could support further explorations. we also treat kqa pro as a diagnostic dataset for analyzing a model's capability of multiple reasoning skills, and discover weaknesses that are not widely known, e.g., current models struggle on comparisonal reasoning for lacking of literal knowledge (i.e., (le-bron james, height, 206 centimetre)), or perform poorly on questions whose answers are not obverseved in the training set. we hope all contents of kqa pro could encourage the community to make further breakthroughs. ## related work complex kbqa aims at answering complex questions over kbs, which requires multiple reasoning capabilities such as multi-hop inference, quantitative comparison, and set operation @xcite . current methods for complex kbqa can be grouped into two categories: 1) semantic parsing based methods @xcite @xcite , which parses a question to a symbolic logic form (e.g., λcalculus @xcite , λ-dcs @xcite @xcite , sql @xcite , amr @xcite , sparql @xcite , and etc.) and then executes it against the kb and obtains the final answers; 2) information retrieval based methods @xcite et al., 2020; @xcite @xcite @xcite , which constructs a question-specific graph extracted from the kb and ranks all the entities in the extracted graph based on their relevance to the question. compared with information retrieval based methods, semantic parsing based methods provides better interpretability by generating expressive logic forms, which represents the intermediate reasoning process. however, manually annotating logic forms is expensive and labor-intensive, and it is challenging to train a semantic parsing model with weak supervision signals (i.e., question-answer pairs). lacking logic form annotations turns out to be one of the main bottlenecks of semantic parsing. ## kqa pro construction to build kqa pro dataset, first, we extract a knowledge base with multiple kinds of knowledge (section 4.1). then, we generate canonical questions, corresponding kopl programs and sparql queries with novel compositional strategies (section 4.2). in this stage, we aim to cover all the possible queries through random sampling and recursive composing. finally, we rewrite canonical questions into natural language via crowdsourcing (section 4.3). to further increase linguistic variety, 2 the complete function instructions are in appendix a. 3 the grammar rules of kopl are in appendix b. ## experiments the primary goal of our experiments is to show the challenges of kqa pro and promising complex kbqa directions. first, we compare the performance of state-of-the-art kbqa models on current datasets and kqa pro, to show whether kqa pro is challenging. then, we treat kqa pro as a diagnostic dataset to investigate fine-grained reasoning abilities of models, discuss current weakness and promising directions. we further conduct an experiment to explore the generation ability of our proposed model. last, we provide a case study to show the interpretablity of kopl. ## conclusion and future work in this work, we introduce a large-scale dataset with explicit compositional programs for complex kbqa. for each question, we provide the corresponding kopl program and sparql query so that kqa pro can serve for both kbqa and semantic parsing tasks. we conduct a thorough evaluation of various models, discover weaknesses of current models and discuss future directions. among these models, the kopl parser shows great interpretability. as shown in fig. 4 , when the model predicts the answer, it will also give a reasoning process and a confidence score (which is ommited in the figure for simplicity). when the parser makes mistakes, humans can easily locate the error through reading the human-like reasoning process or checking the outputs of intermediate functions. in addition, using human correction data, the parser can be incrementally trained to improve the performance continuously. we will leave this as our future work. yushi wang, jonathan berant, and percy liang. 2015b. building a semantic parser overnight. in proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 1: long papers), pages 1332-1342, beijing, china. association for computational linguistics. peng wu, shujian huang, rongxiang weng, zaixiang zheng, jianbing zhang, xiaohui yan, and jiajun chen. 2019. learning representation mapping for relation detection in knowledge base question answering. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 6130-6139, florence, italy. association for computational linguistics. shan wu, bo chen, chunlei xin, xianpei han, le sun, weipeng zhang, jiansong chen, fan yang, and xunliang cai. 2021. from paraphrasing to semantic parsing: unsupervised semantic parsing via synchronous semantic decoding. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 5110-5121, online. association for computational linguistics. wen-tau yih, ming-wei chang, xiaodong he, and jianfeng gao. 2015. semantic parsing via staged query graph generation: question answering with knowledge base. in proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 1: long papers), pages 1321-1331, beijing, china. association for computational linguistics. wen-tau yih, matthew richardson, chris meek, ming-wei chang, and jina suh. 2016. the value of semantic parse labeling for knowledge base question answering. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 2: short papers), pages 201-206, berlin, germany. association for computational linguistics. yuyu zhang, hanjun dai, zornitsa kozareva, alexander j. smola, and le song. 2018. variational reasoning for question answering with knowledge graph. in proceedings of the thirty-second aaai conference on artificial intelligence, (aaai-18), the 30th innovative applications of artificial intelligence (iaai-18), and the 8th aaai symposium on educational advances in artificial intelligence (eaai-18), new orleans, louisiana, usa, february 2-7, 2018, pages 6069-6076. aaai press. victor zhong, caiming xiong, and richard socher. 2017. seq2sql: generating structured queries from natural language using reinforcement learning. arxiv preprint, abs/1709.00103. mantong zhou, minlie huang, and xiaoyan zhu. 2018. an interpretable reasoning network for multi-relation question answering. in proceedings of the 27th international conference on computational linguistics, pages 2010-2022, santa fe, new mexico, usa. association for computational linguistics.