A Methodology for Creating Question Answering Corpora Using Inverse Data Annotation (2020)

Abstract: In this paper, we introduce a novel methodology to efficiently construct a corpus for question answering over structured data. For this, we introduce an intermediate representation that is based on the logical query plan in a database, called Operation Trees (OT). This representation allows us to invert the annotation process without loosing flexibility in the types of queries that we generate. Furthermore, it allows for fine-grained alignment of the tokens to the operations. Thus, we randomly generate OTs from a context free grammar and annotators just have to write the appropriate question and assign the tokens. We compare our corpus OTTA (Operation Trees and Token Assignment), a large semantic parsing corpus for evaluating natural language interfaces to databases, to Spider and LC-QuaD 2.0 and show that our methodology more than triples the annotation speed while maintaining the complexity of the queries. Finally, we train a state-of-the-art semantic parsing model on our data and show that our dataset is a challenging dataset and that the token alignment can be leveraged to significantly increase the performance.

## introduction question answering (qa) over structured data, also called natural language interfaces to databases (nli2db) or text-to-sql, is a key task in natural language processing and the semantic web. it is usually approached by mapping a natural language question (nl question) into executable queries in formal representations such as logical forms, sparql or sql. the state-of-the-art in this problem uses machine learning techniques to learn the mapping. unfortunately, the construction of labeled corpora to train and evaluate nli2db systems is time-and costintensive, which is slowing down progress in this area. in particular, it usually requires recruiting sql or sparql experts to write queries for natural language questions. for instance, in spider @xcite , the authors recruited students to write sql queries. they worked 500 person-hours to generate 5,600 queries, which corresponds to more than 5 minutes per question. as a more cost-effective alternative to writing formal queries manually, some authors propose to use templates to generate them automatically. for instance, lc-quad 2.0 @xcite used 22 templates based on the structure of the target knowledge graph. constructing templates is also time-consuming, and the expressiveness of the automatically produced queries is limited. apart from the high cost of generating queries, the natural language questions in current datasets do not necessarily cover the whole range of data present in the database. in spider, the coverage is limited by the creativity of the students, and in lc-quad 2.0 by the templates. in this paper, we propose a new procedure to increase the speed of the annotation process. for this, we first introduce an intermediate representation of the structured queries, which we call operation trees (ots, see figure 1 ). our ots follow a context-free grammar and are based on logical query plans that can easily be mapped to sparql or sql, making our system more versatile. in addition, it has been shown that working on abstract tree representations instead of sequences yields better results @xcite . recent work by @xcite shows the successful use of tree-like abstractions as an intermediate representation to parse text into semantic representations, reinforcing our choice of operation trees as the main representation language. our annotation process works as follows. first, we use the context-free grammar to sample random ots for a given database. we then let annotators in a first round write the corresponding nl questions for the sampled ots. in a second, optional, round the annotators perform an assignment of tokens from the nl question to the operations in the ot. this additional annotation enriches the information in the dataset, and, as we will show below, allows for performance gains, especially in low data regimes. our approach to producing datasets has the following advantages with respect to the methodology used in previous work: 1) it reduces the time needed for an annotation (less than 2 minutes, compared to more than 5 in spider). 2) it allows us to cover the whole range of data present in the database structure and not to focus on the most prominent examples. 3) our annotation procedure provides alignments between operations in the formal language and words in the question, which are an additional source of supervision when training. we applied our approach foot_0 to five datasets, yielding a large corpus called otta foot_1 which consists of 3,792 complex nl questions plus their corresponding ots as well as the token assignment for one of our domains. besides, we have adapted a stateof-the-art system @xcite to work on to operation trees, and included a mechanism to profit from token alignment annotations when training. the system yields better results with up to 7 point increase when trained on aligned ots. ## related work in this section, we first review the related work in the area of natural language interfaces to databases (nli2db). afterwards, we focus on the data resources that are currently available to evaluate these systems. ## operation trees in our setting, the goal is to generate an operation tree (ot) that finds the correct answer for a given question in natural language. an ot is a binary tree that is closely related to a logical query plan in sql database engines. an ot is composed of a sequence of operations that can be mapped to a database query language such as sql or sparql to retrieve the proper result. example. assume that we have a database about movies that we want to query in natural language. in figure 1 , an example of an ot is depicted for the question "who starred in 'the notebook'?". in order to answer this question, the tables person and movie are selected, then the table movie is filtered by movie title the notebook. in the next step, the tables are joined via the bridge-table cast. finally, the person.name column is extracted. we enhance these ots by associating a reasonable subset of tokens from the nl question to each operation in the tree. for instance, the token "starred" could be associated to the join operation, as this operation implies that an actor starred in a movie, whereas the tokens "how many" could be associated to the count operation. this mapping between tokens and operations will help later on to train machine learning algorithms to generate ots automatically from natural language questions with better quality. definition. more formally, the ots follow a predefined context-free grammar. in the current state, the set of operations includes major operations from the relational algebra with specific extensions. the full grammar is shown in figure 2 . the ots can be used to represent queries for any entity-relationship data paradigm. for instance, in sql databases the entity types are the tables, the attributes are the columns, and the relationships are represented as tables as well. similar mapping is possible for other paradigms. ## corpus construction the evident way to construct a corpus with nl questions and their corresponding ot queries would consist of two main parts: first, collect a set of nl questions, and then create the corresponding ot queries to these questions. however, this approach is very time-consuming and has a major issue. in essence, questions tend to be very narrow in scope, i.e., they do not necessarily cover the whole range of entity types, attributes and relationships that are present in the database. moreover, writing the corresponding ot queries for the nl questions requires sufficient sql skills as well as a mechanism to verify that the ot statements actually correspond to the question. thus, we decided to invert the process. that is, we first randomly sample an ot using the abovedefined context-free grammar, and then annotators write a corresponding question in natural language. in the last step, annotators manually map tokens of the question to the operations. there are several advantages to this procedure: 1) it allows for controlling the characteristics of the ots, i.e., we can control the question type, the response type, the constraints, and the entity type. 2) it allows them to create more complex questions that better cover the variety of the underlying data. 3) the annotation process is less time consuming, as the annotators do not have to build the trees or write queries. rather they can focus on writing the question and assigning tokens. we now describe the process of automatic sampling and manual annotation in more detail. ## corpus otta we applied our corpus construction procedure to a set of five databases and produced a new corpus with nl questions and corresponding ots, called otta. in order to compare our results with previous work, we used four databases from the spider corpus (chinook, college, driv-ing school, and formula i), which we extended with a dump from imdb foot_2 that we refer to as moviedata. for the annotations, we employed 22 engineers with basic knowledge in sqldatabases. ## conclusion in this paper, we introduced a fast annotation procedure to create nl queries and corresponding database queries (in our case, operation trees). our procedure more than triples the velocity of annotation in comparison to previous methods, while ensuring a larger variety of different types of queries and covering a larger part of the underlying databases. furthermore, our procedure allows a fine-grained alignment of tokens to operations. we then used our new method to generate otta, a novel corpus for semantic parsing based on operation trees in combination with token assignments. generating this corpus was more time-and costefficient than with previous approaches. our statistical analysis showed that the corpus yields a higher coverage of attributes in the databases and more complex natural language questions than other existing methods. furthermore, we implemented a baseline system for automatically generating ots from nl queries. this baseline achieves scores of up to 48% precision, which are already reasonable while also leaving large potential for improvement in future research. finally, we showed that the inclusion of the token alignment results in an increase of precision of up to 7%. based on these results, we will explore ways to leverage the token assignment to domain adaption and few-shot learning. we also plan to enhance the annotation process by automatically generating proposals for the nl questions and token assignments and letting the annotators only perform corrections. we hope that this increases annotation efficiency even more.