RASAT : Integrating Relational Structures into Pretrained S eq2 S eq Model for Text-to- SQL (2022)

Abstract: Relational structures such as schema linking and schema encoding have been validated as a key component to qualitatively translating natural language into SQL queries. However, introducing these structural relations comes with prices: they often result in a specialized model structure, which largely prohibits using large pretrained models in text-to-SQL. To address this problem, we propose RASAT: a Transformer seq2seq architecture augmented with relation-aware self-attention that could leverage a variety of relational structures while inheriting the pretrained parameters from the T5 model effectively. Our model can incorporate almost all types of existing relations in the literature, and in addition, we propose introducing co-reference relations for the multi-turn scenario. Experimental results on three widely used text-to-SQL datasets, covering both single-turn and multi-turn scenarios, have shown that RASAT could achieve competitive results in all three benchmarks, achieving state-of-the-art execution accuracy (75.5% EX on Spider, 52.6% IEX on SParC, and 37.4% IEX on CoSQL).

## introduction text-to-sql is the task that aims at translating natural language questions into sql queries. since it could significantly break down barriers for nonexpert users to interact with databases, it is among the most important semantic parsing tasks that are of practical importance @xcite . various types of relations have been introduced for this task since @xcite collected the first large-scale text-to-sql dataset, which has resulted in significant boosts in the performance through recent years. for example, @xcite introduced schema encoding to represent the schema structure of the database, and the resulting augmented lstm encoder-decoder architecture was able to generalize better towards unseen database schema. @xcite introduced relations between the entity mentioned in the question and the matched entries in the database to utilize database content effectively. their bert-based encoder is followed by an lstm-based pointer network as the decoder, which generalizes better between natural language variations and captures corresponding schema columns more precisely. rat-sql @xcite introduced schema linking, which aligns mentions of entity names in the question to the corresponding schema columns or tables. their augmented transformer encoder is coupled with a specific tree-decoder. sadga @xcite introduced the dependency structure of the natural language question and designed a graph neural network-based encoder with a tree-decoder. on the other hand, a tree-decoder that can generate grammatically correct sql queries is usually needed to better decode the encoder output, among which @xcite is one of the most widely used. although integrating various relational structures as well as using a tree-decoder have been shown to be vital to generating qualitative sql queries and generalizing better towards unseen database schema, the dev of various specifically designed model architectures significantly deviate from the general sequential form, which has made it hard if one considers leveraging large pre-trained models for this task. existing methods either use bert output as the input embedding of the specifically designed model @xcite @xcite , or stack a specific decoder on top of @xcite . in another thread, pretrained seq2seq models just have unveiled their powerful potential for this task. recent attempts by @xcite show that directly fine-tuning a t5 model @xcite on this task without presenting any relational structures could achieve satisfying results. moreover, picard @xcite presents a way to prune invalid beam search results during inference time, thus drastically improving the grammatical correctness of the sql queries generated by the autoregressive decoder that comes with t5. in this work, different from the more common approach of fine-tuning the original pretrained model or using prompt tuning, we propose to augment the self-attention modules in the encoder and introduce new parameters to the model while still being able to leverage the pre-trained weights. we call the proposed model rasat 2 . our model can incorporate almost all existing types of relations in the literature, including schema encoding, schema linking, syntactic dependency of the question, etc., into a unified relation representation. in addition to that, we also introduce coreference relations to our model for multi-turn text-to-sql tasks. experimental results show that rasat could effectively leverage the advantage of t5. it achieves the stateof-art performance in question execution accuracy (ex/iex) on both multi-turn (sparc and cosql) and single-turn (spider) text-to-sql benchmarks. on sparc, rasat surpasses all previous methods in interaction execution accuracy (iex) and improves state-of-the-art performance from 21.6% to 52.6%, 31% absolute improvements. on cosql, we improve state-of-the-art iex performance from 8.4% to 37.4%, achieving 29% absolute improvements. moreover, on spider, we improve state-ofthe-art execution accuracy from 75.1% to 75.5%, achieving 0.4% absolute improvements. ## related work early works usually exploit a sketch-based slotfilling method that uses different modules to predict the corresponding part of sql. these methods decompose the sql generation task into several independent sketches and use different classifiers to predict corresponding part, such as sqlnet @xcite , sqlova @xcite , x-sql @xcite , ryansql @xcite , et.al,. however, most of these methods only 2 rasat: relation-aware self-attention-augmented t5 handle simple queries while failing to generate correct sql in a complex setting such as on spider. faced with the multi-table and complex sql setting, using graph structures to encode various complex relationships is a major trend in the text-to-sql task. for example, global-gnn @xcite represents the complex database schema as a graph, rat-sql @xcite introduces schema encoding and linking and assigns every two input items a relation, lgesql @xcite further distinguishes local and non-local relations by exploiting a line graph enhanced hidden module, sadga @xcite uses contextual structure and dependency structure to encode question-graph while database schema relations are used in schema graph, s 2 sql @xcite adds syntactic dependency information in relational graph attention network (rgat) @xcite . for the conversational context-dependent textto-sql task that includes multiple turns of interactions, such as sparc and cosql, the key challenge is how to take advantage of historical interaction context. edit-sql @xcite edits the last turn's predicted sql to generate the newly predicted sql at the token level. igsql @xcite uses cross-turn and intra-turn schema graph layers to model database schema items in a conversational scenario. tree-sql @xcite uses a tree-structured intermediate representation and assigns a probability to reuse subtree of historical tree-sqls. ist-sql @xcite proposes an interaction state tracking method to predict the sql query. rat-sql-tc @xcite adds two auxiliary training tasks to explicitly model the semantic changes in both turn grain and conversation grain. r 2 sql @xcite and hie-sql @xcite introduce a dynamic schema-linking graph by adding the current utterance, interaction history utterances, database schema, and the last predicted sql query. recently, @xcite showed that finetuning a pre-trained t5-3b model could yield results competitive to the then-state-of-the-art. based on this discovery, @xcite proposed to constrain the autoregressive decoder through incremental parsing during inference time, effectively filtering out grammatically incorrect sequences on the fly during beam search, which significantly improved the qualities of the generated sql. ## experiments in this section, we will show our model's performance on three common text-to-sql datasets: spider @xcite , sparc @xcite and cosql @xcite . besides, we experiment on a more realistic setting of the spider dataset: spider-realistic @xcite to test the gen-eralizability of our model. the statistics of these datasets are shown in ## conclusion in this work, we propose rasat, a relation-aware self-attention-augmented t5 model for the textto-sql generation. compared with previous work, rasat can introduce various structural relations into the sequential t5 model. different from the more common approach of fine-tuning the origi-nal model or using prompt tuning, we propose to augment the self-attention modules in the encoder and introduce new parameters to the model while still being able to leverage the pre-trained weights. rasat had achieved state-of-the-art performances, especially on execution accuracy, in the three most common text-to-sql benchmarks.