D uo RAT : Towards Simpler Text-to- SQL Models (2021)

Abstract: Recent neural text-to-SQL models can effectively translate natural language questions to corresponding SQL queries on unseen databases. Working mostly on the Spider dataset, researchers have proposed increasingly sophisticated solutions to the problem. Contrary to this trend, in this paper we focus on simplifications. We begin by building DuoRAT, a re-implementation of the state-of-the-art RAT-SQL model that unlike RAT-SQL is using only relation-aware or vanilla transformers as the building blocks. We perform several ablation experiments using DuoRAT as the baseline model. Our experiments confirm the usefulness of some techniques and point out the redundancy of others, including structural SQL features and features that link the question with the schema.

## introduction language user interfaces to databases allow nonspecialists to retrieve and process information that might otherwise not be easily available to them. much of the recent research in this area has focused on neural models that can generalize to new relational databases without any human intervention. given a relational database schema (and often also content), such models translate the user's question directly into an sql query @xcite @xcite . such crossdatabase text-to-sql research was spurred by the introduction of large datasets such as wikisql @xcite and spider @xcite that feature utterance-query pairs for hundreds or even thousands of databases. state-of-the-art text-to-sql models employ many sophisticated techniques. these include, but are not limited to, grammar-constrained models and recurrent neural networks with parent feeding @xcite , intermediate meaning representations @xcite , relation-aware attention @xcite , schema linking and table joining heuristics @xcite , slot filling @xcite and re-ranking @xcite models. the high complexity of these models raises the barrier of entry and can slow down text-to-sql research. in this work, we attempt to distill the essence of high-performing text-to-sql systems. we start with a transformer-only reimplementation of the state-of-the-art rat-sql model @xcite . importantly, our resulting duorat model trains three times faster than rat-sql. we then systematically study how duorat can be simplified without losing performance. our ablation study confirms the usefulness of many but not all techniques employed in rat-sql. for example, we show that the benefits of explicit matching of question spans with the column or table names (name-based schema linking, nbsl) become marginal when a pretrained transformer @xcite is used to jointly encode the question and the schema. by contrast, we confirm the benefit of using a grammar to constrain the inference to only produce well-formed queries. these and other findings of our work bring much-needed insight of what enables higher performance in modern text-to-sql models. ## methods our base model, duorat, is a reimplementation of rat-sql @xcite . it is an encoderdecoder model with attention and a pointer-network copy mechanism, fig. 1 . contrary to rat-sql, both the encoder and the decoder are relation-aware transformers @xcite . the input is modelled as a labelled directed graph, where the nodes are the input tokens and the edges are the so-called relations, see below. cls which singers are from france … cls number singer_id cls text country … cls number concert id cls number singer id … cls singer cls singer in concert apply[sql] apply[select] ... apply[from] apply[table_unit] table[singer] ... apply[val] str["] str[france] str["] ... bert large table match value match table-column foreign key relation-aware transformer decoder apply[select] ... apply[table_unit] table[singer] ... str["] str[france] str["] .. relation-aware transformer encoder attention figure 1: the duorat architecture. the encoder consists of a stack of bert layers and several relation-aware self-attention layers, while the decoder is a relation-aware transformer with encoder cross-attention. the inputs to bert are, from left to right, the question (red), each column type and name (blue), and each table name (yellow). column and table representations are pooled (blue and yellow boxes), the question's representation is not (red boxes). relation-aware attention compared to vanilla self-attention, relation-aware self-attention takes two additional tensor inputs, key relations r (k) i j and value relations r (v ) i j , that amplify or diminish contributions in the scaled dot-product attention for each of the h attention heads: the relations r i j are shared across all layers. let r be the total number of relational edge labels. if the relation @xmath0, . . . , r} exists between the i-th and j-th input, then we assign the s-th learned embedding r (s) i j to r i j . otherwise, we use padding. encoder the duorat encoder is divided into two stages, a pretrained relation-unaware transformer stage followed by a relation-aware transformer stage that is trained from scratch. the first stage is initialized with bert weights @xcite and is fed embeddings of the question tokens, the table name tokens, the column name tokens, and one token for each column data type. we add [cls] tokens between segments, cf. fig. 1 for the input layout. the second stage has two inputs: an input sequence and the input relations corresponding to graph nodes and labelled edges, respectively. the input sequence is comprised of the bert outputs for all question token positions and the [cls] token position outputs for each table and each column. we use relational edge labels similar to those introduced by rat-sql. the labels are divided into three groups; (i) schemalinking relations, (ii) table-column relations, and (iii) foreign-key relations. (i) schema-linking relations provide explicit alignment between the question and the schema. we distinguish between namebased schema linking (nbsl) and content-based schema linking (cbsl), where the former uses the names of tables and columns only and the latter uses the database content. an example for nbsl is when the question references a table by name, like "singer" in fig. 1 . cbsl identifies when the question references a value in a database column, e.g. the word "france" in fig. 1 . we use a common schema-linking heuristic where question n-grams are compared at the character level with names of tables and columns for nbsl and with the contents of column cells for cbsl. (ii) the table-column relations describe which columns belong to which tables, and which columns occur in the same table. finally, (iii) the foreign-key relations indicate the foreign key constraints between columns. see appendix a for a complete list of the encoder relations. decoder we have extended the tranx framework for grammar-constrained sequence prediction @xcite to relation-aware transformers. like in the original framework, the decoder is restricted to generate only those sequences of grammar actions that encode a valid sql abstract syntax tree (ast), see appendix b. we consider two output grammars, one for complete sql and another for sql with underspecified from clause (sql uf ) @xcite . in a sql uf query, the from clause is replaced by the uf clause that contains only the tables that were not mentioned in other clauses of the original sql query. after decoding a sql uf query, we recover the from clause by adding tables from other clauses and joining them using the foreign-key relations. rat-sql and the tranx framework use a custom parent-feeding lstm decoder where the lstm is fed also its own state from a previous step at which the constructor of the current action's parent ast node was generated. by contrast, in duorat's relation-aware transformer decoder, we experiment with relations that are derived from the structure of the sql program code, see appendix a for a list. the relations can bias the transformer decoder towards attending ast parent or sibling nodes, allowing for the model to get a sense of the ast's structure. however, it is unclear whether or not this is necessary in a model with self-attention. the decoder is coupled to the encoder via a relation-aware memory attention mechanism. here we use relations to indicate which tokens from the input were copied to the output, that is, either question tokens, tables, or columns, depending on the type of the literal that was produced. ## experiments for most of our experiments we use the spider dataset @xcite and evaluate the predicted sql with the exact-match (em) accuracy from the official spider evaluation script. the spider training set contains 8,659 questions for 146 databases. the spider development set contains 1,034 questions for 20 databases. we exclude the baseball1 questions from the training data because the schema of this database is too large. to compare duorat to the models in the literature we evaluate it on the original development set as released on january 8, 2019. in all other experiments we use the corrected development set that was released on june 7, 2020. we also test our spider-trained models on several system em (dev.) ryansql @xcite 70.6 rat-sql @xcite 69.7 irnet @xcite 65.5 ## conclusion our investigations have revealed several possible simplifications of relation-aware text-to-sql transformer models. in particular, we have shown that a transformer decoder with vanilla self-and memoryattention is sufficient, and that heuristic schema linking based on table and/or column names brings only a marginal benefit. to the contrary, we confirm the importance of grammar-constrained decoding, relational schema representations, contentbased schema linking. looking forward, we believe that content-based schema-linking will remain important, while the impact of name-based schema linking will further decrease as the language models get bigger and absorb more data. this prediction is based on the fact that the mapping from an entity to the entity type that namebased schema linking effectively performs can be highly domain-and schema-specific. last but not least, we have shown that predicting a more compact sql version with an underspecified from clause improves the model's out-of-distribution performance, despite bearing little influence on the model's performance on spider. in future work, we will combine the successful simplifications from this paper to build the simplest yet high-performing text-to-sql model. one promising direction for further simplification is to use a pretrained encoder-decoder pair as proposed in @xcite and @xcite .