Semantic Parsing for Conversational Question Answering over Knowledge Graphs (2023)

Abstract: In this paper, we are interested in developing semantic parsers which understand natural language questions embedded in a conversation with a user and ground them to formal queries over definitions in a general purpose knowledge graph (KG) with very large vocabularies (covering thousands of concept names and relations, and millions of entities). To this end, we develop a dataset where user questions are annotated with Sparql parses and system answers correspond to execution results thereof. We present two different semantic parsing approaches and highlight the challenges of the task: dealing with large vocabularies, modelling conversation context, predicting queries with multiple entities, and generalising to new questions at test time. We hope our dataset will serve as useful testbed for the development of conversational semantic parsers. Our dataset and models are released at https://github.com/EdinburghNLP/SPICE .

## introduction conversational information seeking is the process of acquiring information through conversations @xcite . recent years have seen an increasing number of applications aiming to build conversational interfaces based on information retrieval @xcite and user recommendation @xcite . the popularity of intelligent voice assistants such as amazon's alexa or apple's siri has further stimulated research on question answering over general purpose knowledge graphs (e.g., wikidata). key to question answering in this context is the ability to ground natural language onto concepts, entities, and relations in order to produce an executable query (e.g., sparql) which will retrieve an answer or denotation from the knowledge graph (kg). this grounding process, known as semantic parsing has been studied in the context of one or few domain-specific databases @xcite ; jain 1 our dataset and models are released at spice. and lapata, 2021; @xcite or without taking the conversational nature of the task into account @xcite @xcite . however, due to the complexities of the semantic parsing task, there are no large scale datasets consisting of information seeking conversations with executable queries against a kg. conversational semantic parsing over kgs requires handling very large vocabularies covering thousands of concept names and relations, and millions of entities rather than specialized terms consisting of hundreds of tables and column names. moreover, information seeking conversations are by nature incremental involving interrelated rather than isolated questions. in this work, we create spice, a semantic parsing dataset for conversational question answering over wikidata. spice consists of userassistant interactions where natural language questions are paired with sparql parses and answers provided by the system correspond to sparql execution results. we derive this dataset from csqa @xcite , an existing benchmark originally proposed for retrieval-based conversational question answering @xcite . although csqa does not have executable queries, it contains a large number of natural language questions and their corresponding answers, highlighting a range of conversational phenomena such as coreference, ellipsis, and topic change as well as different types of questions exemplifying varying intents. for our semantic parsing task, we establish two strong baseline models which tackle the large vocabulary problem and the prediction of logical forms in different ways. the first approach @xcite uses dynamic vocabularies derived from kg subgraphs for each question and a simple sequence-to-sequence architecture to predict complete sparql queries. the other approach @xcite predicts sparql query templates and then fills in entity, relation, and type slots by means of an entity and ontology classifier. our experiments reveal several shortcomings in both approaches, such as not being able to encode large sets of kg elements and generate the same entity several times. both approaches struggle with ellipsis, they cannot resolve coreference when the referent appears in the conversation context beyond the previous turn, have reduced performance on questions with multiple entities, and struggle with unseen question intents. we discuss these challenges and outline research directions for conversational semantic parsing. ## the spice dataset the csqa dataset @xcite aims to facilitate the development of qa systems that handle complex and inter-related questions over a knowledge graph. in contrast to simple factual questions that can be answered with a single kg triple (i.e., {subject, relation, object}), complex questions require manipulating sets of triples and reasoning over these. in questions and answers in this dataset were elicited from human experts playing user and system roles as well as from crowd-workers. in a second stage, templates derived from the humanauthored qa pairs were used to automatically augment the dataset. human experts also suggested complex reasoning questions and derived templates thereof. conversations were built as sequences of nb. instances 197k nb. entities 12.8m nb. relations 2738 nb. types 3064 avg. turn length 9.5 avg. entities per conversation 7.6 avg. types per conversations 6.5 avg. neighbourhood per turn 181.4 triples logical reasoning, quantitative reasoning, comparative reasoning, quantitative reasoning count, comparative reasoning count, verification, simple question clarification, coreference, ellipsis qa pairs exploring paths in the kg. by construction, the qa pairs in a conversation are connected through one or several entities in the kg. questions fall into two coarse categories, simple and reasoning-based, and the way qa pairs are organised in a sequence introduces various conversational phenomena which we summarize below. simple questions are factoid questions, seeking information related to an entity (e.g., which tournament did detroit tigers participate in? in reasoning questions are complex questions which require the application of numerical and logical operators over sets of entities. for instance, to answer the question how many sports teams participated in that tournament? requires finding the set of sports teams that participated in a given tournament (e.g., 1909 world series) and taking its count. questions in this category also involve general entities (ge) such as tournament, in addition to named entities (ne), and multiple entities (both ne and ge) in a single question (e.g., which tournaments have less number of participating sports teams than 1909 world series? ). some question types also combine multiple reasoning operators. conversations contain sequences of mixedinitiative interactions where the system requests clarification on ambiguous questions. conversations also include discourse phenomena such as coreference (e.g., which sports team was the champion of that tournament? in there are 10 question types and 47 question subtypes. in ## the semantic parsing task we propose two approaches for this semantic parsing task which establish strong baseline performance and highlight various challenges. these differ in the way they handle large kg vocabularies and how they generate logical forms. figure 1 provides a sketch of the two models discussed below. ## results we examine how the two models just described fare on different question types and subtypes. we report results on spice i.i.d train/valid/test splits (containing 152,391/16,813/27,797 conversations, respectively) but also create new splits that assess out-of-distribution generalisation. in all cases, following previous work @xcite , we use execution-based automatic metrics. micro f1-score evaluates question parses that return a set of entities, while accuracy is used for question parses that evaluate to true/false or return a numerical value. in addition, we report exact match (em) against the gold sparql parse. ## related work much previous work on semantic parsing focuses on mapping stand-alone utterances to logical forms. relatively few datasets have been constructed for conversational semantic-parsing @xcite @xcite partly due to the difficulty of eliciting annotations in an interactive context. as a result, existing benchmarks are either single-domain or small-scale (see the comparison in large kgs, like wikidata @xcite , are becoming an increasing valuable source of information. various question-answering datasets have been recently released @xcite @xcite , which are either not conversational or contain sequences of dialogue turns but the questions are not annotated with executable queries like sparql. the csqa dataset introduced in @xcite is conversational and covers a wide range of linguistic phenomena (e.g., ellipsis, coreference) but frames the qa task as an information retrieval problem. follow-on work @xcite @xcite has used hand-crafted grammars to automatically obtain semantic annotations which are are not executable with a real kg engine (e.g., blazegraph), and cumbersome to extend to new question intents. ## conclusion in this work we introduce spice, a conversational semantic parsing dataset over knowledge graphs. our dataset contains sparql annotations which are executable on a real kg engine and requires handling complex questions, type, relation, and entity linking on a large scale. moreover, it showcases multiple linguistic phenomena such as coreference and ellipsis. we establish two strong baselines for the semantic parsing task and present detailed analysis stratifying performance by question type and linguistic phenomena. we also study generalisation to unseen intents and create multiple dataset splits with different query patterns. to move forward conversational semantic parsing over large scale kgs would need to improve entity linking, modelling of conversation context, and generalisation capabilities. we hope our dataset will serve as a useful testbed for the development of conversational semantic parsers.