Complex question answering over knowledge base (Complex KBQA) is challenging because it requires the compositional reasoning capability . Existing benchmarks have three shortcomings that limit the development of Complex KBQA: 1) they only provide QA pairs without explicit reasoning processes; 2) questions are either generated by templates, leading to poor diversity, or on a small scale ; and 3) they mostly only consider the relations among entities but not attributes . To this end, we introduce KQA Pro, a large-scale dataset for Complex KBQA. We  generate questions,  SPARQLs, and functional programs with recursive templates and then paraphrase the questions by crowdsourcing, giving rise to around 120K diverse instances. The SPARQLs and programs depict the reasoning processes in various manners , which can benefit a large spectrum of QA methods. We contribute a unified codebase and conduct extensive evaluations for baselines and state-of-the-arts: a blind GRU obtains 31.58 \\%DIF < , the best model achieves only 35.15\\%, and humans top at 97.5\\%, which offers great research potential to fill the gap.abstract %DIF > answering accuracy, which is even close to the human expert performance...
Complex question answering over knowledge base (Complex KBQA) is challenging because it requires various compositional reasoning capabilities, such as multi-hop inference, attribute comparison, set operation, and etc . Existing benchmarks have some shortcomings that limit the development of Complex KBQA: 1) they only provide QA pairs without explicit reasoning processes; 2) questions are either generated by templates, leading to poor diversity, or on a small scale . To this end, we introduce KQA Pro, a large-scale dataset for Complex KBQA. We define a compositional and highly-interpretable formal format, named Program, to represent the reasoning process of complex questions. We propose compositional strategies to generate questions, corresponding SPARQLs, and Programs with a small number of templates, and then paraphrase the generated questions to natural language questions (NLQ) by crowdsourcing, giving rise to around 120K diverse instances. SPARQL and Program depict two complementary solutions to answer complex questions , which can benefit a large spectrum of QA methods. Besides the QA task, KQA Pro can also serves for the semantic parsing task. As far as we know, it is currently the largest corpus of NLQ-to-SPARQL and NLQ-to-Program. We conduct extensive experiments to evaluate whether machines can learn to answer our complex questions in different cases, that is, with only QA supervision or with intermediate SPARQL/Program supervision. We find that state-of-the-art KBQA methods learnt from only QA pairs perform very poor on our dataset, implying our questions are more challenging than previous datasets. However, pretrained models learnt from our NLQ-to-SPARQL and NLQ-to-Program annotations surprisingly achieve about 90 \\%DIF < , the best model achieves only 35.15\\%, and humans top at 97.5\\%, which offers great research potential to fill the gap.abstract %DIF > answering accuracy, which is even close to the human expert performance...
R
the compositional reasoning capability
various compositional reasoning capabilities, such as multi-hop inference, attribute comparison, set operation, and etc
R
three
some
D
; and 3) they mostly only consider the relations among entities but not attributes
A
define a compositional and highly-interpretable formal format, named Program, to represent the reasoning process of complex questions. We propose compositional strategies to
A
corresponding
R
functional programs with recursive templates
Programs with a small number of templates,
R
questions
generated questions to natural language questions (NLQ)
R
The SPARQLs and programs depict the reasoning processes in various manners
SPARQL and Program depict two complementary solutions to answer complex questions
R
We contribute a unified codebase and conduct extensive evaluations for baselines and state-of-the-arts: a blind GRU obtains 31.58
Besides the QA task, KQA Pro can also serves for the semantic parsing task. As far as we know, it is currently the largest corpus of NLQ-to-SPARQL and NLQ-to-Program. We conduct extensive experiments to evaluate whether machines can learn to answer our complex questions in different cases, that is, with only QA supervision or with intermediate SPARQL/Program supervision. We find that state-of-the-art KBQA methods learnt from only QA pairs perform very poor on our dataset, implying our questions are more challenging than previous datasets. However, pretrained models learnt from our NLQ-to-SPARQL and NLQ-to-Program annotations surprisingly achieve about 90