Structure-Grounded Pretraining for Text-to- SQL (2021)

Abstract: Learning to capture text-table alignment is essential for tasks like text-to-SQL. A model needs to correctly recognize natural language references to columns and values and to ground them in the given database schema. In this paper, we present a novel weakly supervised Structure-Grounded pretraining framework (STRUG) for text-to-SQL that can effectively learn to capture text-table alignment based on a parallel text-table corpus. We identify a set of novel pretraining tasks: column grounding, value grounding and column-value mapping, and leverage them to pretrain a text-table encoder. Additionally, to evaluate different methods under more realistic text-table alignment settings, we create a new evaluation set Spider-Realistic based on Spider dev set with explicit mentions of column names removed, and adopt eight existing text-to-SQL datasets for cross-database evaluation. STRUG brings significant improvement over BERTLARGE in all settings. Compared with existing pretraining methods such as GRAPPA, STRUG achieves similar performance on Spider, and outperforms all baselines on more realistic sets. All the code and data used in this work will be open-sourced to facilitate future research.

## introduction semantic parsing is the task of mapping a natural language (nl) utterance to a machineunderstandable representation such as lambda calculus, abstract meaning representation, or a structured query language (e.g., sql). in this paper, we focus on the task of translating nl questions to executable sql queries @xcite . this is a fundamental task for building natural language interfaces for databases, which can enable non-expert users to effortlessly query databases @xcite . in both examples, the associations between tokens in the nl utterance and columns in the table are indicated. in this paper, we aim to leverage the text-table alignment knowledge in the parallel text-table corpus to help text-to-sql. one of the key challenges in text-to-sql is text-table alignment, that is, to correctly recognize natural language references to columns and values and to ground them in the given database schema. consider the example in the top half of fig. 1 . a model needs to first identify the column mentions total credits, department, and value mention history, and then ground them to the given schema. this is challenging for three reasons. first, the model needs to jointly understand the nl utterance and the database schema, as the user may refer to a column using various expressions which usually differ from the original column name. second, the model needs to be able to generalize to new database schemas and referential language that is not seen in training. finally, in the case that accessing cell values is not possible, the model still needs to identify potential value mentions and link them to the correct columns without exhaustively searching and matching over the database. on the other hand, text-table alignment naturally exists in parallel text-table corpora, e.g., web tables with context @xcite , tableto-text generation datasets @xcite , table-based question answering datasets @xcite . such datasets can be collected from web pages, documents, etc., and requires much less human effort to create compared with text-to-sql datasets. the bottom half of fig. 1 gives an example of such an alignment dataset. there are three value mentions 11417, pune junction and nagpur jnction, which can be grounded to the train number, departure station and arrival station columns respectively. such alignment information can be easily obtained by leveraging the table contents or using some human annotation. in this work, we aim to incorporate the text-table alignment knowledge contained in a parallel corpus via pretraining and use it to help the downstream text-to-sql task. we present a novel weakly supervised structuregrounded pretraining framework (strug) for textto-sql. we design a set of prediction tasks and optimize them leveraging a parallel corpus containing both nl sentences and tabular data to encourage the encoded representation to capture information required to support tasks that require table grounding. more specifically, we identify three critical tasks for aligning text with table: column grounding, value grounding and column-value mapping (examples shown in fig. 2 ). we re-purpose an existing large-scale table-to-text generation dataset totto @xcite for pretraining and gain labels for the three tasks via weak supervision. we experiment under two settings, with or without human assistance: (1) human assisted setting, using totto's revised descriptions and cell annotations; as pointed out by @xcite , existing text-to-sql benchmarks like spider @xcite render the text-table alignment challenge easier than expected by explicitly mentioning exact column names in the nl utterances. contrast this to more realistic settings where users may refer to the columns using a variety of expressions. @xcite propose a new cross-database setting that uses spider for training and includes eight other single-domain text-to-sql datasets for evaluation. in addition to adopting their setting, we create a new evaluation set called spider-realistic from the original spider dev set, by removing explicit mentions of column names from an utterance. we pretrain strug using 120k text-table pairs from totto. experiments show that our structuregrounded pretraining objectives are very efficient and usually converge with around 5 epochs in less than 4 hours. this dramatically reduces the pretraining cost compared to previous pretraining methods @xcite . we adopt the same model architecture as bert @xcite , with simple classification layers on top for pretraining. for downstream tasks, strug can be used as a text-table encoder and easily integrated with any existing state-of-the-art model. we conduct extensive experiments and show that: ## related work cross-database text-to-sql. remarkable progress has been made in text-to-sql over the past few years. with sufficient in-domain training data, existing models already achieve over 80% exact matching accuracy @xcite on single-domain benchmarks like atis @xcite and geoquery @xcite . however, annotating nl questions with sql queries is expensive making it cost-prohibitive to collect training examples for all possible databases. a model that can generalize across domains and databases is desired. in light of this, @xcite present spider, a cross-database text-to-sql benchmark that trains and evaluates a system using different databases. more recently, @xcite provide a holistic analysis of the challenges introduced in cross-database text-to-sql and propose to include single-domain datasets in evaluation. their study uncovers the limitations of current text-to-sql models, and demonstrates the need for models that can better handle the generalization challenges. pretraining for text-table data . inspired by the success of pretrained language models, some recent work has tried to apply similar pretraining objectives to text-table data. tabert @xcite and tapas @xcite jointly learn texttable representations by leveraging a large amount of web tables and their textual context. they flatten the tables and use special embeddings to model the structure information. a masked language model (mlm) objective is then used to predict the masked tokens in the text-table data. mlm is good at modeling the contextualized semantic representations of a token, but is weak at capturing the alignment between a pair of sequences (e.g., text-table). more recently, grappa @xcite explores a different direction for pretraining which shares some similarity with existing work on data augmentation for semantic parsing. grappa first constructs synthetic question-sql pairs using templates (a synchronous context free grammar) induced from existing text-to-sql datasets, a sql semantic prediction objective is then used to learn compositional inductive bias from the synthetic data. however, as the synthetic data is generated using templates, and the column names and values are directly filled in the questions, it has the same problem as existing text-to-sql datasets that eases the text-table alignment challenge. in constrast, strug aims to directly learn the text-table alignment knowledge from parallel text-table corpora via structure-grounded pretraining objectives. we also note that existing pretraining methods and strug can be complementary and combined together in the future. structure grounding in text-to-sql. structure grounding has been proven to be crucial for textto-sql, where a model needs to correctly identify column and value mentions in an nl utterance and link them to the given database schema @xcite @xcite . most existing text-to-sql systems have specially designed components for structure grounding, which is also referred to as schema linking. for example, @xcite ; @xcite explore using simple heuristics like string matching for schema linking, and use the linking results as direct hints to their systems. however, such heuristics may not generalize well in real world scenarios where there are varied ways to refer to a column, which usually differ from the original column name. more recently, @xcite and @xcite take a step forward and manually annotate wikitablequestions @xcite and spider with fine-grained alignment labels for supervised training (together with the text-to-sql objective), which brings significant improvements. the main drawback of these models is that they are limited to learn the alignment knowledge from a relatively small training corpus, and cannot generalize well in a cross-domain setting. moreover, sql annotations and fine-grained alignment labels are both expensive to get manually. in contrast, this paper aims to re-purpose an existing parallel text-table corpus for pretraining models to learn structure grounding, where we generate alignment labels at large scale with low or no cost. ## creating a more realistic evaluation set as one of the first datasets to study cross-database text-to-sql, spider has been a widely used benchmark in assessing a model's ability to generalize to unseen programs and databases. however, as pointed out by @xcite , spider eases the task by using utterances that closely match their paired sql queries, for example by explicitly mentioning the column names in the question, while in practice nl references to columns usually differ from the original column name. to alleviate this problem, @xcite propose to train the model with cross-domain dataset like spider, and add another eight single-domain datasets like atis @xcite and geo-query (zelle and mooney, 1996) for evaluation. however, some of the datasets differ a lot from spider, introducing many novel query structures and dataset conventions. 2 as we can see from for the base model, we use sqlova @xcite without execution-guided decoding. follow- ## conclusion in this paper, we propose a novel and effective structure-grounded pretraining technique for textto-sql. our approach to pretraining leverages a set of novel prediction tasks using a parallel texttable corpus to help solve the text-table alignment challenge in text-to-sql. we design two settings to obtain pretraining labels without requiring complex sql query annotation: using human labeled cell association, or leveraging the table contents. in both settings, strug significantly outperforms bert large in all the evaluation sets. meanwhile, although strug is surprisingly effective (using only 120k text-table pairs for pretraining) and performs on par with models like tabert (using 26m tables and their english contexts) and grappa (using 475k synthetic examples and 391.5k examples from existing text-table datasets) on spider, we believe it is complementary with these existing text-table pretraining methods. in the future, we plan to further increase the size of the pretraining corpus, and explore how to incorporate mlm and synthetic data.