Logical Form Generation via Multi-task Learning for Complex Question Answering over Knowledge Bases (2022)

Abstract: Question answering over knowledge bases (KBQA) for complex questions is a challenging task in natural language processing. Recently, generation-based methods that translate natural language questions to executable logical forms have achieved promising performance. These methods use auxiliary information to augment the logical form generation of questions with unseen KB items or novel combinations, but the noise introduced can also leads to more incorrect results. In this work, we propose GMT-KBQA, a Generation-based KBQA method via Multi-Task learning, to better retrieve and utilize auxiliary information. GMT-KBQA first obtains candidate entities and relations through dense retrieval, and then introduces a multi-task model which jointly learns entity disambiguation, relation classification, and logical form generation. Experimental results show that GMT-KBQA achieves state-of-the-art results on both ComplexWebQuestions and WebQuestionsSP datasets. Furthermore, the detailed evaluation demonstrates that GMT-KBQA benefits from the auxiliary tasks and has a strong generalization capability.

## introduction question answering over knowledge bases (kbqa) is the task of answering natural language questions based on the facts stored in knowledge bases (kbs). in recent years, an increasing number of kbqa methods arise with the emergence of large-scale kbs, such as freebase @xcite , dbpedia @xcite and wikidata @xcite . a mainstream paradigm of kbqa methods is semantic parsing (sp) @xcite , where natural language questions are parsed into logical forms such as λ-dcs @xcite , sparql @xcite , s-expression @xcite , etc. however, complex questions that involve reasoning over multiple 1 our code is available at https://github.com/ hxx97/gmt-kbqa . with the success of applying natural language generation to various tasks @xcite , recent kbqa methods @xcite @xcite cast semantic parsing to a logical form generation task in a sequence-to-sequence (seq2seq) manner, finetuning pre-trained encoder-decoder models to generate logical forms from natural language questions. however, it is impractical for a simple seq2seq model to generate unseen entities and relations that never appear in the training set. to alleviate such cases, more generation-based methods @xcite @xcite augment logical form generation with auxiliary information such as linked entities @xcite , similar question-query pairs @xcite , candidate logical forms @xcite , etc. auxiliary information enhances the generalization capability of the generation models, but can also lead to incorrect results due to the noisy information introduced. we find that a seq2seq model can generate exact logical forms for around 92% questions in complexwebques-tions dataset if provided with golden entities and relations. however, the proportion drops drastically to 51% if the linking results are from practical linkers. this shows that seq2seq models can construct correct logical forms augmented with auxiliary information, and the quality of the auxiliary information has a great impact on the generated results. inspired by this discovery, we propose a generation-based kbqa method gmt-kbqa (generation via multi-task learning) that learns to question: what is the most predominant religion in the country benjamin netanyahu holds a governmental position? candidate relation retrieval candidate entity retrieval candidate entities m.0fm2h (benjamin netanyahu) m.05p3vsk (benzion netanyahu) m.06bvp (religion) • • • candidate relations location.religion_percentage.religion location.statistical_region.religions base.popstra.religion_choice.religion • • • classified relations location.religion_percentage.religion location.statistical_region.religions • • • relation classification entity disambiguation disambugated entities m.0fm2h (benjamin netanyahu) logical form generation target logial form: (join (r location.religion_percentage.religion) (argmax (join (r location.statistical_region.religions) (join government.governmental_jurisdiction.governing_officials (join government.government_position_held.appointed_by m.0fm2h))) location.religion_percentage.percentage)) shared encoder figure 1: overview of gmt-kbqa. the t5 encoder is learned with multiple tasks including the entity/relation classification and the logical form generation. the main contributions of this work are as follows: ## methodology this section details the gmt-kbqa method. given a natural language question, we first retrieve auxiliary information including candidate entities and relations by dense retrieval. then we refine the retrieved auxiliary information and generate the target logical form via multi-task learning. as shown in figure 1 , the three tasks share a common encoder, and each task has an individual layer on top of the encoder. details of our method will be given in the following subsections. ## related work existing kbqa methods can be mainly divided into two categories: information retrieval-based methods (ir-based methods) and semantic parsingbased methods (sp-based methods). ir-based methods @xcite @xcite follow a retrieval-and-rank paradigm. they first retrieve a question-specific graph from the kb and then rank entities in the graph by their relevance to the question. for complex questions, recent irbased methods turn their attention to graph retrieval @xcite and multihop reasoning over graphs @xcite @xcite . generally, ir-based methods fit into end-to-end training, but they lack interpretability because of the black-box reasoning process. sp-based methods, closely relevant to our method, are more transparent compared with irbased methods. they answer questions by parsing them into logical forms executable against kbs, including λ-dcs @xcite , sparql @xcite , query graph @xcite @xcite , and s-expression @xcite . past sp-based methods parse questions with a bottom-up semantic parser @xcite or iteratively generate and rank candidate query graphs @xcite . however, previous semantic parsers have limited coverage for diverse complex queries @xcite , and query graph generation methods suffer from the high computational cost of expanding the graphs @xcite . recent methods @xcite @xcite take advantage of language generation models to directly generate executable logical forms from questions. as vanilla generation models do not generalize well to questions on kb items with novel combinations and unseen ones, @xcite generate complex logical forms conditioned on retrieved similar questions along with their logical forms, but they need to add human-labeled cases to cover absent relations in the case memory. @xcite first rank a pool of candidate logical forms obtained by enumerating relation paths over the kbs, and then generate the final logical form based on the question combined with topranked candidates. the enumeration of candidate logical forms involves a large search space on the kb, which is time-consuming and computationally expensive. ## conclusion we present gmt-kbqa to improve the generalization capability of generation-based methods utilizing auxiliary information. gmt-kbqa first retrieves candidate entities and relations in dense space. then, its multi-task learning framework learns to refine the auxiliary information along with generating target logical forms at the same time. experimental results on two datasets, cwq and webqsp, show that our method sets new stateof-the-art by a large margin. the further analysis illustrates that our logical form generation task and auxiliary tasks benefit from each other and gmt-kbqa achieves strong performance for unseen kb items. in general, gmt-kbqa gives an insight into generating more accurate logical forms through auxiliary information retrieval and multi-task learning. currently, answering questions that require in-depth understanding such as logical or commonsense reasoning remains a challenging task, and we will strengthen gmt-kbqa for such scenarios in the future.