S ea D : End-to-end Text-to- SQL Generation with Schema-aware Denoising (2022)

Abstract: On the WikiSQL benchmark, most methods tackle the challenge of text-to-SQL with predefined sketch slots and build sophisticated sub-tasks to fill these slots. Though achieving promising results, these methods suffer from over-complex model structure. In this paper, we present a simple yet effective approach that enables auto-regressive sequence-to-sequence model to robust text-to-SQL generation. Instead of formulating the task of text-to-SQL as slot-filling, we propose to train sequence-to-sequence model with Schema-aware Denoising (SeaD), which consists of two denoising objectives that train model to either recover input or predict output from two novel erosion and shuffle noises. These model-agnostic denoising objectives act as the auxiliary tasks for structural data modeling during sequence-to-sequence generation. In addition, we propose a clause-sensitive execution guided (EG) decoding strategy to overcome the limitation of EG decoding for generative model. The experiments show that the proposed method improves the performance of sequence-to-sequence model in both schema linking and grammar correctness and establishes new state-of-the-art on WikiSQL benchmark. Our work indicates that the capacity of sequence-to-sequence model for text-to-SQL may have been under-estimated and could be enhanced by specialized denoising task.

## introduction text-to-sql aims at translating natural language into valid sql query. it enables layman to explore structural database information with semantic question instead of dealing with the complex grammar required by logical -form query. on the wikisql benchmark, most models adopt a sketch-based slot filling approach. it decomposes the task of convert query to sql into several sub-tasks that are relatively easy to handle, e.g., the 'select' column mentioned or the query span corresponding to a condition value. the entire sql can be recovered from the results of the sub-tasks deterministically. though being a typical sequence-to-sequence (seq2seq) task, auto-regressive models (lstm, transformer, etc.), however, fail to achieve state-ofthe-art results for text-to-sql task. previous works attribute the sub-optimal results of seq2seq models to three major limitations. first, sql queries with different clause order may have exact same semantic meaning and return same results by execution. the token interchangeability may confusion model that based on seq2seq generation. second, the grammar constraint induced by structural logical form is ignored during auto-regressive decoding, therefore the model may predict sql with invalid logical form. third, schema linking, which has been suggested to be the crux of text-to-sql task, is not specially addressed by vanilla seq2seq model. in this paper, we present a simple yet effective method to boost the performance of seq2seq model for text-to-sql task. instead of building extra sub-module or putting constraint on model output, we propose two novel schema-awared denoising objectives trained along with the original seq2seq generation task. these denoising objectives deal with the intrinsic attribute of logical form and could facilitate schema linking required for text-to-sql task. the inductive schema-awared noises can be categorized into two types: erosion and shuffle. erosion acts on schema input by randomly permute, drop and add columns into the current schema set. the related schema entity in target sql query will be jointly modified according to the erosion result. shuffle is applied via randomly re-ordering the mentioned entity and values in natural language (nl) or sql with respect to the schema columns. during training procedure, shuffle is performed during monolingual self-supervision that trains model to recover original text given the noised one. erosion is applied to seq2seq task that trains model to generate corrupted sql sequence, given nl and eroded schema as input. these proposed denoising objectives are combined along with the origin seq2seq task to train a sead model. in addition, to deal with the limitation of execution-guided (eg) decoding, we propose a clause-sensitive eg strategy that decide beam size with respect to the clause token that is predicted. the proposed method establish new state-of-the-art on the wikisql benchmark. the main contribution of this work is the schemaaware denoising objectives that are designed for text-to-sql task. the denoising objectives are model-agnostic and could apply to any seq2seq model that are trained in auto-regressive manner. in addition, we also propose a clause-sensitive eg decoding strategy, which can improve the searching efficiency of eg during seq2seq generation. the results of the work demonstrate the effectiveness of the schema-aware denoising approach and shad lights on the importance of task-oriented denoising objective. hybrid pointer networks proposed by @xcite , copying mechanism (cm) uses attention as a pointer to copy several discrete tokens from input sequence as the output and have been successfully used in machine reading comprehension @xcite @xcite , interactive conversation @xcite @xcite , geometric problems @xcite and program generation @xcite @xcite @xcite . in text-to-sql, cm can not only facilitate the condition value extraction from source input, but also help to protect the privacy of the database. in this paper, we use a hybrid pointer generator network which is similar to @xcite to generate next step token. denoising self-training language model pretraining @xcite @xcite has been shown to improve the downstream performance on many nlp tasks and brought significant gains. @xcite @xcite are beneficial to seq2seq task, while they are problematic for some tasks. while @xcite is a denoising seq2seq pre-training model, which is effective for both generative and discriminative tasks, reduces the mismatch between pre-training and generation tasks. inspired by this, we propose a denosing self-training architecture in training to learn mapping corrupted documents to the original. erosion sead select `<@xmath0,677 ẁhich week had an attendance of 53,677 <@xmath1,677 <@xmath2,677 ` from `table ` where `<@xmath3,677 had an week of attendance which week had an attendance of 53,677 ## methodology given natural language question q and a schema s, our goal is to obtain the corresponding sql query y . here the natural question ## experiment to demonstrate the effectiveness of the proposed method, we evaluate the proposed model on wik-isql benchmark and compare it to other state-ofthe-art methods. ## conclusions in this paper, we proposed to train model with novel schema-aware denoising objectives, which could improve performance of seq2seq generation for text-to-sql task. these objectives are applied individually with respective to their activate probabilities, which are fixed during the training procedure. a noise re-weighting model will be considered for future work. combined with the proposed clausesensitive eg decoding strategy, our model achieves state-of-the-art on the wikisql benchmark. the success of the sead highlights the potential of utilizing task-oriented denoising objective for seq2seq model enhancement.