SeaD: End-to-end Text-to-SQL Generation with Schema-aware Denoising
Abstract
On the WikiSQL 1 benchmark, most methods tackle the challenge of text-to-SQL with predeﬁned sketch slots and build sophisticated sub-tasks to ﬁll these slots.
Though achieving promising results, these methods suffer from over-complex model structure.
In this paper, we present a simple yet effective approach that enables auto-regressive sequenceto-sequence model to robust text-to-SQL generation.
Instead of formulating the task of text-to-SQL as slot-ﬁlling, we propose to train sequence-to-sequence model with S chemaa ware Denoising (SeaD), which consists of two denoising objectives that train model to either recover input or predict output from two novel erosion and shufﬂe noises.
These modelagnostic denoising objectives act as the auxiliary tasks for structural data modeling during sequence-to-sequence generation.
In addition, we propose a clause-sensitive execution guided (EG) decoding strategy to overcome the limitation of EG decoding for generative model.
The experiments show that the proposed method improves the performance of sequence-to-sequence model in both schema linking and grammar correctness and establishes new state-of-the-art on WikiSQL benchmark.
Our work indicates that the capacity of sequence-to-sequence model for text-to-SQL may have been under-estimated and could be enhanced by specialized denoising task.
Introduction
Text-to-SQL aims at translating natural language into valid SQL query.
It enables layman to explore structural database information with semantic question instead of dealing with the complex grammar required by logical -form query.
On the WikiSQL benchmark, most models adopt a sketch-based slot ﬁlling approach.
It decomposes the task of convert query to SQL into several sub-tasks that are relatively easy to handle, e.g., the ‘SELECT‘ column
[Table]
mentioned or the query span corresponding to a condition value.
The entire SQL can be recovered from the results of the sub-tasks deterministically.
Though being a typical sequence-to-sequence (seq2seq) task, auto-regressive models (LSTM,
Transformer, etc.), however, fail to achieve state-ofthe-art results for text-to-SQL task.
Previous works attribute the sub-optimal results of seq2seq models to three major limitations.
First, SQL queries with different clause order may have exact same semantic meaning and return same results by execution.
The token interchangeability may confusion model that based on seq2seq generation.
Second, the grammar constraint induced by structural logical form is ignored during auto-regressive decoding, therefore the model may predict SQL with invalid logical form.
Third, schema linking, which has been suggested to be the crux of text-to-SQL task, is not specially addressed by vanilla seq2seq model.
In this paper, we present a simple yet effective method to boost the performance of seq2seq model for text-to-SQL task.
Instead of building extra sub- module or putting constraint on model output, we propose two novel schema-awared denoising objectives trained along with the original seq2seq generation task.
These denoising objectives deal with the intrinsic attribute of logical form and could facilitate schema linking required for text-to-SQL task.
The inductive schema-awared noises can be categorized into two types: erosion and shufﬂe .
Erosion acts on schema input by randomly permute, drop and add columns into the current schema set.
The related schema entity in target SQL query will be jointly modiﬁed according to the erosion result.
Shufﬂe is applied via randomly re-ordering the mentioned entity and values in natural language (NL) or SQL with respect to the schema columns.
During training procedure, shufﬂe is performed during monolingual self-supervision that trains model to recover original text given the noised one.
Erosion is applied to seq2seq task that trains model to generate corrupted SQL sequence, given NL and eroded schema as input.
These proposed denoising objectives are combined along with the origin seq2seq task to train a SeaD model.
In addition, to deal with the limitation of execution-guided (EG) decoding, we propose a clause-sensitive EG strategy that decide beam size with respect to the clause token that is predicted.
The proposed method establish new state-of-the-art on the WikiSQL benchmark.
The main contribution of this work is the schemaaware denoising objectives that are designed for text-to-SQL task.
The denoising objectives are model-agnostic and could apply to any seq2seq model that are trained in auto-regressive manner.
In addition, we also propose a clause-sensitive EG decoding strategy, which can improve the searching efﬁciency of EG during seq2seq generation.
The results of the work demonstrate the effectiveness of the schema-aware denoising approach and shad lights on the importance of task-oriented denoising objective.
Related Work
Semantic Parsing
The problem of mapping natural language to meaningful executable programs has been widely studied in natural language processing research.
Logic forms (Zettlemoyer and
Collins, 2012; Artzi and Zettlemoyer, 2011, 2013;
Cai and Yates, 2013; Reddy et al., 2014; Liang et al., 2013; Quirk et al., 2015; Chen et al., 2016) can be considered as a special instance to the more generic semantic parsing problem.
As a sub-task of semantic parsing, the text-to-SQL problem has been studied for decades.
(Warren and Pereira, 1982; Popescu et al., 2003; Li et al., 2006; Giordani and Moschitti, 2012; Wang et al., 2017).
Slotﬁlling model (Hwang et al., 2019;
He et al., 2019a;
Lyu et al., 2020) translates the clauses of SQL into subtasks, (Ma et al., 2020) treat this task as a twostage sequence labeling model.
However, the convergence rate between subtasks is inconsistent or the interaction between multiple subtasks may lead to the model may not converge well.
Like lots of previous work (Dong and Lapata, 2016; Lin et al., 2018;
Zhong et al., 2017; Suhr et al., 2020; Raffel et al., 2019), we treat text-to-SQL as a translation problem, and taking both the natural language question and the DB as input.
Hybrid Pointer Networks Proposed by (Vinyals et al., 2015), copying mechanism (CM) uses attention as a pointer to copy several discrete tokens from input sequence as the output and have been successfully used in machine reading comprehension (Wang and Jiang, 2016;
Trischler et al., 2016;
Kadlec et al., 2016; Xiong et al., 2016), interactive conversation (Gu et al., 2016; Yu and Joty, 2020;
He et al., 2019b), geometric problems (Vinyals et al., 2015) and program generation (Zhong et al., 2017; Xu et al., 2017; Dong and Lapata, 2016; Yu et al., 2018; McCann et al., 2018; Hwang et al., 2019).
In text-to-SQL, CM can not only facilitate the condition value extraction from source input, but also help to protect the privacy of the database.
In this paper, we use a Hybrid Pointer Generator
Network which is similar to (Jia and Liang, 2016;
Rongali et al., 2020) to generate next step token.
Denoising Self-training Language model pretraining (Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019; Lan et al., 2019) has been shown to improve the downstream performance on many NLP tasks and brought signiﬁcant gains.
(Radford et al., 2018;
Peters et al., 2018; Song et al., 2019) are beneﬁcial to seq2seq task, while they are problematic for some tasks.
While (Lewis et al., 2019) is a denoising seq2seq pre-training model, which is effective for both generative and discriminative tasks, reduces the mismatch between pre-training and generation tasks.
Inspired by this, we propose a denosing self-training architecture in training to learn mapping corrupted documents to the original. Erosion
[Figure]
Methodology
Given natural language question Q and a schema S , our goal is to obtain the corresponding SQL query
Y .
Here the natural question Q = { q 1 , ..., q | Q | } denotes a word sequence, the schema S = { c 1 , ..., c | S | } is composed of a set of columns, where each column c i
= { c 1 , ..., c | c i | } is a sequence of words.
Y =
y 1 , ..., y | Y | denotes the token-wise raw SQL sequence.
We approach this task with directly auto-regressive generation, i.e., predicting the SQL sequence token by token.
We choose Transformer as our base architecture, which is a widely adopted in seq2seq translation and generation tasks.
In this section, we ﬁrst present the sample formulation that transform text-to-SQL into typical seq2seq task, followed by a brief introduce of the Transformer architecture with pointer generator.
Then we describe the proposed schema-aware denoising method and clause-sensitive EG decoding strategy.
3. Sample Formulation
Given training samples { X i , Y i } , i = 1 , ..., N ,
X = { Q, S } , where Q denotes the NL sequence and S denotes the schema set, Y is the SQL sequence.
Sample formulation is a function (cid:101) X, (cid:101)
Y = format ( X, Y ) S that transforms heterogeneous data into pairwise token sequence.
It is performed by ﬁlling template that acts as a prompt to guide seq2seq model to generate different types of token with respected to various contexts.
For schema formulation, each column name is preﬁxed with a separate special token <coli> , where i denotes the ith column in the schema set.
The column type of each column is also append to the name sequence to form the template for a schema column <coli> col name : col type .
All columns in schema is formulated and concatenated together to compose the input sequence for schema.
The schema sequence is further concatenated with the NL sequence for model input.
We explicitly introduce schema-mention alignment to NL sequence by surrounding schema names that are mentioned in NL sequence with bracket tokens [] , in order to improve the learning of schema linking,
For SQL sequence, we initialize it with raw SQL query and perform several modiﬁcations on it: 1) surrounding entities and values in SQL query with a " ‘ " token, and dropping other surroundings if exist; 2) replacing col entities with their corresponding separate token in schema; 3) inserting spaces between punctuation and words.
The formulated SQL sequence is illustrated in Figure 1.
The formatting procedure improves consistency between tokenized sequences of source and target, and contributes to the identiﬁcation and linking of schema entities.
3. Transformer with Pointer
Following the previous works on seq2seq semantic parsing, we use Transformer (Vaswani et al., 2017) as the backbone of our model.
The vanilla Transformer generate tokens with a feed-forward layer that computes the unnormalized score over the target vocabulary.
In text-to-SQL task, however, most schema and value mentions can be extracted from the input sequence.
Therefore, we adopt a Hybrid
Pointer Generator Network (Jia and Liang, 2016) in our architecture to generate tokens from the target vocabulary V or copy from the input context.
During inference, input sequence X is ﬁrst encoded into a sequence of hidden states H enc .
Then, the decoder produces the hidden states h dec for step t based on previously generated sequence and encoded output.
The unnormalized scores scores v = { s 1 , ..., s | V | } over V can be obtained from h dec through a feed-forward layer.
V =
{ V q , V c , V s } is the target vocabulary, where V q denotes corpora token vocabulary, V c denotes column token set and V s denotes avaliable SQL keywords, e.g.
SELECT, MAX, MIN , etc.
The decoder output h dec is also used to compute the unnormalized attention scores score s
= { i 1 , ..., i | X | } over the input sequence tokens, where | X | is the sequence length.
We concatenate scores v and score s to get the hybrid score score hybrid = { s 1 , ..., s | V | , i 1 , ..., i | X | } , where the ﬁrst | V | elements represent the output distribution of the target vocabulary V and the remained | X | are pointers tokens referred to corresponding input tokens.
The ﬁnal probability distribution is computed by P = softmax ( score hybrid ) , to determine the next token during generation.
3. Schema-aware Denoising
Similar to masked language modeling and other denoising task, we propose two schema-aware objectives, erosion and shufﬂe, that train model to either reconstruct the origin sequence from noising input or predict corrupted output otherwise.
The denoising procedure is illustrated in Figure 2.
3.3. Erosion
Given input sample { Q, S, Y } , erosion corrupts the schema sequence S with a serial compositions of three noising operations:
Permutation Re-order the concatenation sequence of schema columns during schema formulation.
Removal For each column, remove it with a dropping probability p drop .
Addition With a addition probability p add , extract a column from another schema that exists in the training database and insert it into current schema set.
During all operations above, the order of separating special tokens remains unchanged, therefore the
Input : training corpus
X = { ( Q i , S i , Y i ) } , i ∈ 1 , ... |X | ,
S2S Transformer Θ foreach ( Q i , S i , Y i ) ∈
X do
T src , T tgt ← Q i , Y i ;
T tgt , S i ← Erosion( T tgt , S i ) with P shuffle do with P swap do
T src , T tgt ← T tgt , T src ; end
T src ← Shuffle ( T tgt ) end
T type ← SeqType( T tgt ) if T type = SQL then
T prefix ← <2sql> ; else
T prefix ← <2nl> ; end
T src ← T prefix + T src + S i ;
TrainOneSample( T src , T tgt , Θ ) end corresponding anonymous entities in SQL query should be updated along with the erosion operations in schema sequence.
In particular, if a column entity mentioned in SQL query is removed during erosion, we substitute the corresponding column token in SQL with a masking token <unk> to cope with the absence of the schema information.
With such joint modiﬁcation for schema and
SQL sequence, the model is required to identify the schema entities that are truly related to the NL question and learns to raise an unknown exception whenever the schema information is insufﬁcient to compose the target SQL.
3.3. Shufﬂe
Given input sequence X (cid:48)
= {Q , S } , where Q = { Q, Y } , the shufﬂe noise reorders the mentioning sequence of entities in the source query while the schema sequence S is ﬁxed.
The denoising objective trains model to reconstruct the query sequence
Q with entities in correct order.
The objective of recovering shufﬂed entity orders trains model to capture the inner relation between different entities and therefore contributes to the schema linking performance.
It is also notable that, as a selfsupervision objective, both Q and Y are engaged in this denoising task and get trained separately.
Though we dependent on the SQL query to identify the value entities in NL query, order shufﬂing with only column entities is sufﬁcient to obtain promising performance.
Since no parallel data is required, additional corpus with monolingual data for both
SQL and NL could help with the re-order task and will be one of the further direction of this work.
3.3. Training Procedure
Inspired by previous works on denoising selftraining (Song et al., 2019; Lewis et al., 2019), we propose to train the schema-aware denoising objectives along with the primary seq2seq task.
During training, for each training sample, we apply a nosing pipeline to it before feeding it into the model.
The noises with different type are applied to the sample individually.
Through the control of activate probability, they could share the same weights in the overall objective.
Such continual noising pipeline generates random-wise corrupted samples during training.
It prevents the model from fast over-ﬁtting and could yield results with better generalization (Siddhant et al., 2020).
In practice, such simple combination noising strategy could perform better comparing to model-based curriculum method.
The whole procedure is summarized in Algorithm 1.
3. Clause-sensitive EG Decoding
During the inference of text-to-SQL task, the predicted SQL may contain errors related to inappropriate schema linking or grammar.
EG decoding (Wang et al., 2018) is proposed to amend these errors through an executor-in-loop iteration.
It is performed by feeding SQL queries in the candidate list into the executor in sequence and discarding those queries that fail to execute or return empty result.
Such decoding strategy, while effective, suggests that the major disagreement in the candidate list focuses on schema linking or grammar.
Directly perform EG to the candidates generated with beam search leads to trivial improvement, as the candidates consist of redundant variations focuses on selection or schema naming, etc.
This problem can be addressed by setting the beam length of most of the predicted tokens to 1 and releasing those tokens related to schema linking (e.g.,
WHERE ).
We also notice that there are cases that combine incorrect schema linking with some aggregation in SELECT clause, which return some trivial results such as 0 , thus suppress the EG ﬁlter. [Table]
To mitigate the issue, we suggest to drop aggregate operator in SELECT during EG to maximize the effectiveness of it.
Note that with such strategy, the condition with inequation in WHERE clause should be dropped together to ensure the validity of the ground-truth SQL results.
Experiment
To demonstrate the effectiveness of the proposed method, we evaluate the proposed model on WikiSQL benchmark and compare it to other state-ofthe-art methods.
4. Dataset
As the largest human-annotated dataset of textto-SQL, WikiSQL consists of 56 , 355 , 8 , 421 and 15 , 878 NL-SQL pairs for training, validation and inference respectively.
All ground-truth SQL queries are guaranteed with at least one query result.
Each SQL contains SELECT clause with at most one aggregation operator and WHERE clause with at most 4 conditions that connected by AND .
Each SQL is associated with a schema in database.
4. Implementation details
We implement our method using AllenNLP (Gardner et al.) and Pytorch (Paszke et al.).
For the model architecture, we use Transformer with layers in each of the encoder and decoder with a hidden size of 1024 .
We initialize the model weight with bart-large pretrained model provided by
Huggingface community (Wolf et al.)
and ﬁnetune it on training dataset for 20 epochs.
The batch size during training is set to 8 with a gradient accumulation step of 2 .
We choose Adam (Kingma and Ba) as the optimizer and set the learning rate to 7 e − 5 with a warm-up step ratio of 1% .
We searched for the best learning rate for our model out of [1e-4, 7e-5, 1e-5, 5e-6, 5e-7].
The weight decay for regulation is set to 0 .
We set the activation probability P swap = 0 .
5 and P shuffle = 0 .
3 , which lets the self-supervision and seq2seq objectives share equal weight during training process.
P drop for column removal in erosion is set to 0 .
The early stop patience is set to 5 with respect to the BLUE metric (Papineni et al.) on validation set.
The overall training procedure spend aroundhours on an Ubuntu server with 8 NVIDIA V GPUs.
4. Competitors
We compare the proposed method to the following models:
(1) SQLNet (Xu et al., 2017) is a sketch-based method; (2) SQLova (Hwang et al., 2019) is a sketch-based method which leverage the pre-trained language model for representation; (3)
X-SQL (He et al., 2019a) enhances the structural schema representation with contextual embedding;
(4) HydraNet (Lyu et al., 2020) transforms schema linking into column-wise matching and ranking
; (5) IESQL (Ma et al., 2020) treats text-to-SQL as a sequence labeling task;
(6) BRIDGE (Lin et al., 2020) is a sequential architecture for modeling dependencies between natural language question and related schema; (7) SDSQL (Hui et al., 2021) is a multi-task model with explicitly schema dependency guided module.
4. Comparison with State-of-the-art Models
The comparison results are summarized in Table 1.
Models sufﬁxed with ♣ leverage additional annotation of the dataset.
Models sufﬁxed with ♦ utilize database content during training procedure.
Without using EG, SeaD signiﬁcantly outperforms all models without the auxiliary of table content or schema linking annotation.
When combined with
EG decoding, SeaD achieve best performance even compared to those models that utilize additional training information.
It indicates the effectiveness of the proposed denoising objectives on model- [Table] [Table] ing text-to-SQL through vanilla seq2seq.
Notably, the annotation noise makes aggregation prediction a major challenge for WikiSQL.
Previous works suggested to improve AGG prediction via rulebased annotation amendment.
As shown in Table 2, we argue that the proposed aggregation dropping strategy for EG achieves comparable enhancement, while less human effort is involved.
Combined with the AGG dropped clause-sensitive EG, the SeaD model establishes new state-of-the-art on WikiSQL benchmark.
To analysis the detailed improvement for SeaD on text-to-SQL task, in Table 3 we report the accuracy on WikiSQL test set with respect to several SQL components with and without EG decoding.
SeaD shows promising results on column selection, aggregation, where column and where value prediction.
It outperforms all method except
SDSQL, which leverages rule-based annotation of schema linking.
After applying EG decoding, SeaD [Table] achieves best performance on four out of ﬁve components among all competitors.
4. Ablation Study
To evaluate the contribution of each proposed objective, we perform ablation study to SeaD (Table 4) with WikiSQL dataset.
We start from the Bart model and add components to it in sequence.
The pointer net contributes to 1 . 3% absolute improvement of Acc lf on test set.
Combine text inﬁlling, an effective denoising objective utilized by Bart, into training procedure brings 0 .
2 absolute Acc lf improvement.
On the other hand, erosion and shufﬂe objectives contribute to 1 .
5% and 0 .
5% absolute Acc lf improvement for SeaD on test set respectively.
It demonstrates the effectiveness of the schema-aware denoising objective for improving seq2seq generation in text-to-SQL task.
Conclusions
In this paper, we proposed to train model with novel schema-aware denoising objectives, which could improve performance of seq2seq generation for text-to-SQL task.
These objectives are applied individually with respective to their activate probabilities, which are ﬁxed during the training procedure.
A noise re-weighting model will be considered for future work.
Combined with the proposed clausesensitive EG decoding strategy, our model achieves state-of-the-art on the WikiSQL benchmark.
The success of the SeaD highlights the potential of utilizing task-oriented denoising objective for seq2seq model enhancement.