Structured Context and High-Coverage Grammar for Conversational Question Answering over Knowledge Graphs (2021)

Abstract: We tackle the problem of weakly-supervised conversational Question Answering over large Knowledge Graphs using a neural semantic parsing approach. We introduce a new Logical Form (LF) grammar that can model a wide range of queries on the graph while remaining sufficiently simple to generate supervision data efficiently. Our Transformer-based model takes a JSON-like structure as input, allowing us to easily incorporate both Knowledge Graph and conversational contexts. This structured input is transformed to lists of embeddings and then fed to standard attention layers. We validate our approach, both in terms of grammar coverage and LF execution accuracy, on two publicly available datasets, CSQA and ConvQuestions, both grounded in Wikidata. On CSQA, our approach increases the coverage from 80% to 96.2%, and the LF execution accuracy from 70.6% to 75.6%, with respect to previous state-of-the-art results. On ConvQuestions, we achieve competitive results with respect to the state-of-the-art.

## introduction graphs are a common abstraction of real-world data. large-scale knowledge bases can be represented as directed labeled graphs, where entities correspond to nodes and subject-predicate-object triplets are encoded by labeled edges. these socalled knowledge graphs (kgs) are used both in open knowledge projects (yago, wikidata) and in the industry @xcite . a prominent task on kgs is factual conversational question answering (conversational kg-qa) and it has spurred interest recently, in particular due to the development of ai-driven personal assistants. the conversational kg-qa task involves difficulties of different nature: entity disambiguation, long tails of predicates @xcite , conversational nature of the interaction. the topology of the underlying graph is also problematic. not only can kgs be huge (up to several billion entities), they also exhibit hub entities with a large number of neighbors. a recent prominent approach has been to cast the problem as neural semantic parsing @xcite @xcite . in this setting, a semantic parsing model learns to map a natural language question to a logical form (lf), i.e. a tree of operators over the kg. these operators belong to some grammar, either standard like sparql or ad-hoc. the logical form is then evaluated over the kg to produce the candidate answer. in the weak supervision training setup, the true logical form is not available, but only the answer utterance is (as well as annotated entities in some cases, see section 5.4). hence the training data is not given but it is instead generated, in the format of (question, logical form) pairs. we refer to this data as silver data or silver lfs, as opposed to unknown gold ground truth. however, this approach has two main issues. first, the silver data generation step is a complex and often resource-intensive task. the standard procedure employs a breadth-first search (bfs) exploration @xcite , but this simple strategy is prone to failure, especially when naively implemented, for questions that are mapped to nested lfs. this reduces the coverage, i.e. the percentage of training questions associated to a logical form. @xcite proposes to add a neural component for picking the best operator, in order to reduce the computational cost of this task, however complicating the model. @xcite proposes a two-step semantic parser: the question is first paraphrased into a "canonical utterance", which is then mapped to a lf. this approach simplifies the lf generation by separating it from the language understanding task. second, most of the semantic parsing models do not leverage much of the underlying kg structure to predict the lf, as in @xcite ; @xcite . yet, this contextual graph information is rich @xcite , and graph-based models leveraging this information yield promising results for kg-qa tasks @xcite . however these alternative approaches to semantic parsing, that rely on node classification, have their inherent limitations, as they handle less naturally certain queries (see appendix c.3) and their output is less interpretable. this motivates the desire for semantic parsing models that can make use of the kg context. ## related work neural semantic parsing our work falls within the neural semantic parsing approaches for knowledge-based qa @xcite @xcite @xcite . the more specific task of conversational kg-qa has been the focus of recent work. @xcite introduces d2a, a neural symbolic model with memory augmentation. this model has been extended by s2a+maml @xcite with a metalearning strategy to account for context, and by d2a+es @xcite with a neural component to improve bfs. @xcite proposes a reinforcement learning model to benefit from denser supervision signals. finally, @xcite introduces masp, a multi-task model that performs both entity linking and semantic parsing, with the hope of reducing erroneous entity linking (see appendix c.2 for a comparison with our setup). @xcite extended the latter in carton. they first predict the lf using a transformer architecture, then specify the kg items using pointer networks. this work was further extended in @xcite , which includes a graph attention network to exploit correlations between entities and predicates. another related contemporaneous work is @xcite , where the decoder is informed with entity embeddings coming from kg random walks. learning on knowledge graphs classical graph learning techniques can be applied to the specific case of kgs. in convex @xcite , at each turn, a subgraph is expanded by matching the utterance with neighboring entities. then a candidate answer is found by a node classifier. other methods include unsupervised message passing @xcite . however, these approaches lack strong nlp components. other directions include learning differentiable operators over a kg @xcite , or applying graph neural networks @xcite (gnns) to the kg, which has been done for entity classification and link prediction tasks @xcite . gnns have also been used to model relationships between utterances and entities @xcite . structured input for neural models our approach of using json-like input falls in the line of computing neural embeddings out of structured inputs. @xcite introduced tree-lstm for computing tree embeddings bottom-up. it has then been applied for many tasks, including computer program translation @xcite , semantic tree structure learning (such as json or xml) (woof and chen, 2020) and supervised kg-qa tasks @xcite @xcite . in the latter context, tree-lstm is used to model the syntactic structure of the question. other related approaches include tree transformer @xcite and tree attention @xcite . syntactic structures were also modeled as graphs @xcite . specific positional embeddings can also be used to encode structures @xcite . ## a grammar for kg exploration several previous kg-qa works were based on the grammar from d2a @xcite . we also take inspiration from their grammar, but redesign it to model a wider range of queries. by defining more generic operators, we achieve this without increasing the number of operators nor the average depth of lfs. section 3.4 presents a comparison. ## experiments additional comments about the datasets, setups, and additional results can be found in the appendix. ## conclusion for the problem of weakly-supervised conversational kg-qa, we proposed object-aware transformer, a model capable of processing structured input in a json-like format. this allows to flexibly provide the model with structured kg contextual information. we also introduced a kg grammar with increased coverage, which can hence be used to model a wider range of queries. these two contributions are fairly independent: on the one hand, since the model predicts lfs as a list of tokens, it is grammar agnostic, and thus it could be used with another grammar. on the other hand, the grammar is not tied to the model, and can be used to generate training data for other model architectures. experiments on two datasets validate our approach. we plan to extend our model with richer kg context, as we believe there is headroom for improvements. systems 32: annual conference on neural information processing systems 2019, neurips 2019, december 8-14, 2019, vancouver, bc, canada, pages 12058-12068. kai sheng tai, richard socher, and christopher d. manning. 2015. improved semantic representations from tree-structured long short-term memory networks. in proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 1: long papers), pages 1556-1566, beijing, china. association for computational linguistics. raghuveer thirukovalluru, mukund sridhar, dung thai, shruti chanumolu, nicholas monath, sankaranarayanan ananthakrishnan, and andrew mccallum. 2021. knowledge informed semantic parsing for conversational question answering. in proceedings of the 6th workshop on representation learning for nlp (repl4nlp-2021), pages 231-240, online. association for computational linguistics. peihao tong, qifan zhang, and junjie yao. 2019. leveraging domain context for question answering over knowledge graph. data science and engineering, 4(4):323-335. svitlana vakulenko, javier david fernandez garcia, axel polleres, maarten de rijke, and michael cochez. 2019. message passing for complex question answering over knowledge graphs. in proceedings of the 28th acm international conference on information and knowledge management, cikm 2019, beijing, china, november 3-7, 2019, pages 1431-1440. acm. ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n. gomez, lukasz kaiser, and illia polosukhin. 2017. attention is all you need. in advances in neural information processing systems 30: annual conference on neural information processing systems 2017, december 4-9, 2017, long beach, ca, usa, pages 5998-6008. william woof and ke chen. 2020. a framework for end-to-end learning on semantic tree-structured data. computing research repository, arxiv:2002.05707. kun xu, lingfei wu, zhiguo wang, mo yu, liwei chen, and vadim sheinin. 2018. exploiting rich syntactic information for semantic parsing with graph-to-sequence model. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 918-924, brussels, belgium. association for computational linguistics. hamid zafar, giulio napolitano, and jens lehmann. 2019. deep query ranking for question answering over knowledge bases. in machine learning and knowledge discovery in databases, proceedings of the european conference, ecml pkdd 2018, lecture notes in computer science, pages 635-638, dublin, ireland. springer international publishing. jiawei zhang, haopeng zhang, congying xia, and li sun. 2020. graph-bert: only attention is needed for learning graph representations. computing research repository, arxiv:2001.05140. version 2. a clarification questions in csqa take the following dialog as example: t1 can you tell me which cities border verderio inferiore? cornate d'adda, bernareggio, robbiate t2 and which cities flank that one? did you mean robbiate? t3 no, i meant cornate d'adda. bottanuco, busnago, trezzo sull'adda the second turn is a "clarification" question: the system asks the user for disambiguation. the disambiguation question usually takes the form "did you mean", followed by an entity chosen among the previous turn answers. this choice appears to be entirely random. for this reason, we found that it would not be very interesting to try to predict this entity, as baselines propose. hence we only ask the model to predict that the question is a clarification (via a special clarification operator). we report in