Conversational Question Answering over Knowledge Graphs with Transformer and Graph Attention Networks (2021)

Abstract: This paper addresses the task of (complex) conversational question answering over a knowledge graph. For this task, we propose LASAGNE (muLti-task semAntic parSing with trAnsformer and Graph atteNtion nEworks). It is the first approach, which employs a transformer architecture extended with Graph Attention Networks for multi-task neural semantic parsing. LASAGNE uses a transformer model for generating the base logical forms, while the Graph Attention model is used to exploit correlations between (entity) types and predicates to produce node representations. LASAGNE also includes a novel entity recognition module which detects, links, and ranks all relevant entities in the question context. We evaluate LASAGNE on a standard dataset for complex sequential question answering, on which it outperforms existing baselines averaged on all question types. Specifically, we show that LASAGNE improves the F1-score on eight out of ten question types; in some cases, the increase is more than 20% compared to state of the art (SotA).

## introduction since their inception in the late 2000s, publicly available knowledge graphs (e.g., dbpedia @xcite and yago @xcite ) have been widely used as a source of knowledge in several natural language processing (nlp) tasks such as entity linking, relation extraction, fact-checking, and question answering. question answering (qa), in particular, is an essential task that maps a user natural language question figure 1: conversational question answering task with examples similar to csqa dataset @xcite . to a query over a knowledge graph (kg) to retrieve the correct answer @xcite . with the increasing popularity of intelligent personal assistants (e.g., alexa, siri), the research focus has been shifted to conversational question answering that involves multi-turn dialogues, incorporating the phenomenon of anaphora and ellipses @xcite (c.f. figure 1 ). conversational qa is often realised by using semantic parsing approaches, mapping an utterance to a logic form for extracting answers from a kg @xcite . the state of the art for semantic parsing approaches decomposes the semantic parsing process into two stages @xcite . first, a logical form is generated based on low-level features and then the missing details are filled by considering both the question and the template. other approaches @xcite @xcite first employ an entity linking model to identify entities in the question and subsequently use another model to map the question to a logical form. @xcite point out that the modular approaches suffer from the common issue of error propagation along the qa pipeline, resulting in accumulated errors. to mitigate these errors, @xcite proposed a multi-task framework, where a pointer-equipped semantic parsing model was designed to resolve coreference in conversations and empower joint learning with a type-aware entity detection model. furthermore, the authors used simple classifiers to predict the required (entity) types and predicates for the generated logical forms. in this paper, we argue that @xcite model (the current sota) has the following shortcomings: 1) the (entity) type and predicate classifiers share no common information, except for the supervision signal propagated to them. 2) hence, due to missing common information, the model can produce ambiguous results, since the classifiers can predict entities and predicates that do not correlate with each other. ## related work we point to the survey by @xcite ) that provides a holistic overview of neural approaches in conversational ai. in this paper, we stick to our closely related work, i.e., semantic parsing-based approaches in conversations. @xcite introduce a neural symbolic machine (nsm) extended with a key-value memory network, where keys and values are the output of the sequence model in different encoding or decoding steps. the nsm model is trained using the reinforce algorithm with weak supervision and evaluated on the webquestionssp dataset @xcite . @xcite propose a hybrid model of the hred model @xcite and the keyvalue memory network model @xcite . the model consists of three components. the first one is the hierarchical encoder, which computes a representation for each utterance. the next module is a higher-level encoder that computes a representation for the context. the second component is the key-value memory network. it stores each of the candidate tuples as a key-value pair where the key contains the concatenated embedding of the relation and the subject. in contrast, the value contains the embedding of the object. the last component is the decoder used to create an end-to-end solution and produce multiple types of answers. @xcite present a model that converts an utterance in conversation to a logical form. the model follows a flexible grammar, in which the generation of a logical form is equivalent to predicting a sequence of actions. a dialogue memory management is proposed and integrated into the model, so that historical entities, predicates, and action sub-sequences can selectively be replicated. @xcite proposed the first multi-task learning framework that learns type-aware entity detection and pointer-equipped logical form generation simultaneously. the multi-task learning framework takes advantage of the supervision from the subtasks. ## lasagne in a conversation, the input data consists of utterances u and their answers a, extracted from the knowledge graph. our framework lasagne employs a multi-task semantic parsing approach. in particular, it maps the utterance u to a logical form z, depending on the conversation context. figure 2 shows the architecture of lasagne. ## conclusions in this article, we focus on complex question answering over a large-scale knowledge graph containing conversational context. we provide a transformer-based framework to handle the task in a multi-task semantic parsing manner. at the same time, we propose a named entity recognition module for entity detection, filtering, and permutation. furthermore, we also introduce a graph attentionbased module, which exploits correlations between (entity) types and predicates for identifying the gold ones for each particular context. we empirically show that our model achieves the best results for numerous question types and also overall. our ablation study demonstrates the effectiveness of the multi-task learning and of our graph-based module. we also present an error analysis on a random sample of "wrong examples" to discuss our model's weaknesses. for future work, we believe that reinforcement learning is a viable alternative to explore complex conversational question answering without gold annotations.