Mention Extraction and Linking for SQL Query Generation (2020)

Abstract: On the WikiSQL benchmark, state-of-the-art text-to-SQL systems typically take a slot- filling approach by building several dedicated models for each type of slots. Such modularized systems are not only complex but also of limited capacity for capturing inter-dependencies among SQL clauses. To solve these problems, this paper proposes a novel extraction-linking approach, where a unified extractor recognizes all types of slot mentions appearing in the question sentence before a linker maps the recognized columns to the table schema to generate executable SQL queries. Trained with automatically generated annotations, the proposed method achieves the first place on the WikiSQL benchmark.

## introduction text-to-sql systems generate sql queries according to given natural language questions. textto-sql technology is very useful as it can empower humans to naturally interact with relational databases, which serve as foundations for the digital world today. as a subarea of semantic parsing @xcite ), text-to-sql is known to be difficult due to the flexibility in natural language. recently, by the development of deep learning, significant advances have been made in text-to-sql. on the wikisql @xcite benchmark for multi-domain, single table text-to-sql, state-of-the-art systems @xcite can predict more than 80% of entire sql queries correctly. most of such systems take a sketch-based approach @xcite that builds several specialized modules, each of which is dedicated to predicting a particular type of slots, such as the column in select, or the filter value in where. such dedicated modules are complex and often fall short of capturing inter-dependencies among sql sub-clauses, as each type of slots is modeled separately. to deal with these drawbacks, this paper formulates text-to-sql as mention extraction and linking problems in a sequence labeling manner (section 2). in this new formulation, the key to synthesizing sql is to extract the mentions of sql slots and the relations between them. consider the question and its corresponding sql query in example (1), with the headers in the schema being {lane, name, nationality, split (50m), time}. we can see that many sql elements, or slots, such as column names of split (50m) and lane, values like "josefin lillhage" and 8, as well as @xmath0. moreover, the relations between the slot mentions, such @xmath1, above, 8> forming a filter condition, are represented by proximity in linear order or other linguistic cues. thus, the recognition of the mentions and their relations would mostly reconstruct the intended sql query from natural language question. to this end, we leverage one unified bert-based @xcite extractor (section 2.1) to recognize the slot mentions as well as their relations, from the natural language questions. the output of the extractor can be deterministically translated into pseudo sqls, before a bert-based linker (section 2.2) maps the column mentions to the table headers to get executable sql queries. a major challenge to the proposed method is the absence of manual annotation of mentions and relations. thus we propose an automatic annotation method (section 2.4) based on aligning tokens in a sql with corresponding question. also, preliminary results show that the prediction of aggregation function (agg) restricts model performance, which induces us to put forward agg prediction enhancement (ae) method inspired by @xcite . trained with such annotations and applied ae method, the proposed method can already achieves the first place on the wikisql benchmark. the main contribution of this paper is the mention and relation extraction-based approach to textto-sql task. to the best of our knowledge, this is the first work that formulates the task as sequence labeling-based extraction plus linking, which enjoys the advantage of structural simplicity and interdependency awareness. in addition, we also propose an automatic method to generate annotations. such annotations can be useful for developing novel methods for text-to-sql, such as question decomposition-based approaches. ## related work semantic parsing @xcite is to map natural language utterances to machine-interpretable representations, such as logic forms @xcite , program codes @xcite , and sql queries @xcite . text-to-sql is a sub-area of semantic parsing, which is widely studied in recent years. earlier work @xcite @xcite @xcite follow a neural sequence-to-sequence paradigm @xcite with attention mechanism @xcite . pointer networks @xcite are also commonly adopted. these sequence-to-sequence approaches often suffer the "ordering issue" since they are designed to fit an ordered sequence, while the conditions in whereclause are unordered in nature. sqlnet @xcite introduces sketchbased method, which decomposes the sql synthesis into several independent classification sub-tasks, including select-aggregation/column and wherenumber/column/operator/value. except wherevalue, which is usually predicted by a pointer network, all the other sub-tasks use their own dedicated classifiers to make predictions. these sketch-based models raise challenges in training, deployment and maintenance. moreover, each submodule solves its own classification problem, without considering the dependencies with sql elements modeled by other sub-modules. recent advances @xcite @xcite follow this approach and achieve comparative results on wik-isql, mostly by using pre-trained language models as the encoder. while our sequence labeling method is also based on pre-trained language model, it differs from state-of-the-art methods in that it explicitly ex-tracts mentions from the questions and can benefit from inter-dependency modeling between extracted mentions. the mentions for values, operators and corresponding columns often appear in proximity in the question, thus the sequence labeling model can better capture their dependencies and benefits the recognition for all of them, as experiment results suggest. furthermore, our extractor-linker architecture is also much simpler than sketch-based methods. recent trend @xcite @xcite in academia starts to shift to multi-table and complex queries setting of text-to-sql, as in the spider task @xcite . state-of-the art methods on spider typically fall into two categories: grammarbased approach @xcite , and sketch-based approach, such as ryan-sql @xcite and recparser @xcite . the latter ones have slot prediction modules similar to sqlnet for the wikisql, while recursion modules are introduced to handle the generation of complex sql sketches, a characteristic in spider but absent in wikisql. at a high level, our method is along the same line of sqlnet-ryansql, yet differs with them, as our method extracts slots in a unified way rather than using dedicated modules to predict each slot type. we can extend our method to the spider task by following existing sketch construction methods as in ryansql, while replacing their slot classification modules with our extractor-linker methods. ## conclusion and future work thanks to the simple, unified model for mention and relation extraction and its capacity for capturing inter mention dependencies, the proposed method proves to be a promising approach to textto-sql task. equipped with automatic-generated labels and agg enhancement method, our model achieves state-of-the-art results on the wikisql benchmark. since the current automatic-generated annotations are still noisy, it is useful to further improve the automatic annotation procedure. we also plan to extend our approach to cope with multitable text-to-sql task spider.