Complex Question Answering on knowledge graphs using machine translation and multi-task learning (2021)

Abstract: Question answering (QA) over a knowledge graph (KG) is a task of answering a natural language (NL) query using the information stored in KG. In a real-world industrial setting, this involves addressing multiple challenges including entity linking, multi-hop reasoning over KG, etc. Traditional approaches handle these challenges in a modularized sequential manner where errors in one module lead to the accumulation of errors in downstream modules. Often these challenges are inter-related and the solutions to them can reinforce each other when handled simultaneously in an end-to-end learning setup. To this end, we propose a multi-task BERT based Neural Machine Translation (NMT) model to address these challenges. Through experimental analysis, we demonstrate the efficacy of our proposed approach on one publicly available and one proprietary dataset.

## introduction question answering on knowledge graphs (kgqa) has mainly been attempted on publicly available kgs such as freebase @xcite , db-pedia @xcite , yago @xcite , etc. there is also a demand for questions answering on proprietary kgs created by large enterprises. for example, kgqa, on a) a kg that contains information related to retail products, can help the customers choose the right product for their needs, or b) a kg containing document catalogs (best practices, white papers, research papers) can help a knowledge worker find a specific piece of information, or c) a kg that stores profiles of various companies can be used to do preliminary analysis before giving them a loan, etc. our motivating use-case comes from an enterprise system (referred to as loca) that is expected to answer users' questions about the r&d division of an enterprise. sample questions from loca dataset are shown in figure 1 . the schema of the corresponding kg is shown in figure 2 . answering such questions often requires a traversal of kg along multiple relations which may not form a directed chain graph, and may follow more complex topology as shown for question 5, 7 and 8 in figure 1 . it can also be observed that most often words of the natural language question (nlq) and corresponding relations have a weak correlation. most of the proposed approaches on the kgqa task @xcite parse the nlq and convert it into a structured query and then execute the structured query on the kg to retrieve the factoid answers. such conversion involves multiple sub-tasks: a) linking the mentioned entity with corresponding entity-node in the kg @xcite al. (2015 paper 1. title 2. url 3. date of publication 4. conference 5. track/ workshop 6. type 7. pdf available sub area 1. name 2. desc researcher 1. name 2. emp id 3. e-mail id 4. role area_keys sub_area used_in uses published explored_in has_asset has_program explored_in has_asset key_person key_contributor_sra heads_rp supervisor owner_of owner sub_area belongs_to head 1. name 2. emp id 3. e-mail id 4. role research area 1. name 2. high level focus evangelist 1. name 2. emp id 3. e-mail id 4. role asset_evangelist evangelist head_of head ra_evangelist area_evangelist rp_evangelist key_contributor_prog prog_evangelist rp_key_person in accumulation of errors @xcite . further, most of the kgqa tasks are not as complex as loca. for example, a) all questions of sim-pleqa @xcite can be answered using single triple, b) nlqs for most of the datasets (e.g., simpleqa, meta qa) contain only one metioned entity, and c) even if multiple relations are required for answer entity retrieval, they are organized in a sequence, i.e., chain. our motivating example contains specific types of questions that pose many challenges with respect to each of the aforementioned tasks. moreover, some of the questions can only be answered via a model that attempts more than one sub-tasks together. for example, the first two questions of figure 1 mention the same words, i.e., "deep learning" but they get associated with two different entity nodes of the kg. additionally, the prior work could detect the set of relations when the schema sub-graph follows a specific topology, however, in our example, most of the questions follow a different topology. we demonstrate in section 5 that most of the prior art approaches fail to solve such challenges. we provide a summary of such challenges in section 2. in this paper, we propose cqa-nmt, a novel transformer-based, nmt (neural machine translation) model to solve the aforementioned challenges by performing four tasks jointly using a single model, i.e., i) detection of mentioned entities, ii) prediction of entity types of answer nodes, iii) prediction of topology and relations involved, and iv) question type classification such as 'factoid', 'count', etc. cqa-nmt not only performs the four sub-tasks but also helps downstream tasks of mentioned entity disambiguation and subsequent answer retrieval from the kg. the key contributions of this paper are: as applicable to our dataset and helps us solve most of the real-world industrial challenges. ## kgqa problem and challenges for answering natural language questions (nlq), we assume that the background knowledge is stored in a knowledge graph g, comprising of a set of nodes v (g), and edges e(g). here, nodes represent entities, and edges represent the relationship between a pair of entities or connect an entity to one of its properties. an nlq (q) is a sequence of words w i of a natural language (e.g., english), i.e., @xmath0, w 2 , ..., w n }. we also assume that the nlqs can mention zero, one, or more entities present in g and enquire about another entity of g, which is connected with the mentioned entity(s). we pose the kgqa problem as a supervised learning problem and next, describe the labels assumed to be available for every question in the training data and that need to be predicted for every question in the test data. entity linking annotation some of the ngrams (η i ) in an nlq refer to entity(s) of kg. such n-grams have been underlined in figure 1 . the entity-id (as shown in the third column of figure 1 ) of the mentioned entity is also assumed to be available as part of label annotation for every question. answer entity type annotation (aet), τ : we assume that every nlq has an entity type (t i ) for the answer entities. these are shown in the middle column of figure 1 . we refer τ as a set of all entity types in the knowledge graph g. relation sequence and topology annotation (path) sequence of relations connecting the linked entities to the answer entities can be considered paths (path i ), each of which can contain one or more relations. these paths are connected to form a topology, as shown in figure 1 . this topology of the paths and relations are also assumed to be available for an nlq in training data. these paths need not be the shortest paths between the linked entities and the answer entities. for example, the last three columns of figure 1 indicate a) the set of paths separated by a semicolon (;), b) whether this is the shortest path, and c) topology of the paths connecting the linked entities to the answer entities. question type annotation (q type ) some nlqs can be answered by a single triple of the knowledge graph('simple'), while some of them require traversal along with more complex topology as indicated earlier('factoid'), some questions require an aggregate operation such as count ('count', see question 6, in figure 1 ), and finally, some questions perform existence check ('boolean', see question 8, in figure 1 ). such information is also assumed to be available for every nlq in training data. we now describe the challenges that need to be addressed while performing the kgqa task. to the best of our efforts we could not find any prior work that covers all these challenges together. ## related work in this section, we present a view of prior work, on the kgqa problem as an nlp task, and then on set of techniques used for this task. 2019 ) rely on the jointly learning multiple sub-tasks tasks of kgqa problem. however, all these approaches focus on singlehop relations only, and therefore we cannot take such approaches as a baseline for our model. in a more complex setting, @xcite proposed a joint learning task for entity linking, path prediction (chains topology only), and question type. however, their model does not predict answer entity type. we do not compare our approach with @xcite because they focus on the implicit mention of the entities in previous sentences of dialogue, and also because they do not attempt to predict non-chain topology or the answer entity type. non-chain multi-hop relations @xcite proposed an embedding based approach to predict non-chain multi-hop relation prediction (for a fixed and small set of topologies). they perform only one task of relationship prediction. ## proposed architecture question type prediction who starred films for the screenwriter of rear window? shared encoder entity mention detection path prediction answer entity type prediction entity id path knowledge base relevant subgraph question type answer entity type entity linking multi-task model rear window 1954 grace kelly thelma ritter classic thriller alfred hitchcock cornell woolrich john michael hayes wri tten _by wri tten _by re le a se _ ye a r ta g s sta rre d_a cto r st a rr e d _ a ct o r d ir e ct e d _ b y has_genre ... ... ... ... ... ... peyton place missisipi mermaid has _wr itten has _wr itten ... ... lee philips star red _ac tor catherine deneuve ... ... st ar re d_ ac to r ... ... ... factoid na rear window cornell woolrich john michael hayes wri tten _by wri tten _by peyton place missisipi mermaid has _wr itten has _wr itten ... ... lee philips star red _ac tor catherine deneuve ... st ar re d_ ac to r ... ... lee philips, catherine deneuve, ... output rear_window input written_by, has_written, starred_actor rear window algo (seq2seq) (seq2seq) (classification) (classification) joint model for kgqa in this paper, we extend bert to generate path (or inference chains), perform sequence labeling and classification jointly. details of each module are described next. ## experiments and system details in this section, we first introduce the datasets used for our experiments. we pre-process all nlqs (of all datasets) by downcasing and tokenizing. ## conclusion we presented a complex version of the kgqa problem, which involves mention of multiple entities in the question. multiple sequence of relationships combined in complex topologies, are required to answer such questions. it is evident that such questions, while required to be answered in real world industrial setting, cannot be answered using prior approaches. we propose a novel cqa-nmt model to answer such questions and have performed a detailed comparison of our approach with prior art on metaqa and loca datasets. we have shown that cqa-nmt not only solves more complex task, but also performs better on metaqa dataset as compared to baseline approaches.