RAT-SQL : Relation-Aware Schema Encoding and Linking for Text-to- SQL Parsers (2020)

Abstract: When translating natural language questions into SQL queries to answer questions from a database, contemporary semantic parsing models struggle to generalize to unseen database schemas. The generalization challenge lies in (a) encoding the database relations in an accessible way for the semantic parser, and (b) modeling alignment between database columns and their mentions in a given query. We present a unified framework, based on the relation-aware self-attention mechanism, to address schema encoding, schema linking, and feature representation within a text-to-SQL encoder. On the challenging Spider dataset this framework boosts the exact match accuracy to 57.2%, surpassing its best counterparts by 8.7% absolute improvement. Further augmented with BERT, it achieves the new state-of-the-art performance of 65.6% on the Spider leaderboard. In addition, we observe qualitative improvements in the model’s understanding of schema linking and alignment. Our implementation will be open-sourced at https://github.com/Microsoft/rat-sql .

## introduction the ability to effectively query databases with natural language (nl) unlocks the power of large datasets to the vast majority of users who are not proficient in query languages. as such, a large body of research has focused on the task of translating nl questions into sql queries that existing database software can execute. the development of large annotated datasets of questions and the corresponding sql queries has catalyzed progress in the field. in contrast to prior semantic parsing datasets @xcite , new tasks such as wikisql @xcite and spider @xcite pose the reallife challenge of generalization to unseen database schemas. every query is conditioned on a multitable database schema, and the databases do not overlap between the train and test sets. schema generalization is challenging for three interconnected reasons. first, any text-to-sql parsing model must encode the schema into representations suitable for decoding a sql query that might involve the given columns or tables. second, these representations should encode all the information about the schema such as its column types, foreign key relations, and primary keys used for database joins. finally, the model must recognize nl used to refer to columns and tables, which might differ from the referential language seen in training. the latter challenge is known as schema linking -aligning entity references in the question to the intended schema columns or tables. while the question of schema encoding has been studied in recent literature @xcite , schema linking has been relatively less explored. consider the example in figure 1 . it illustrates the challenge of ambiguity in linking: while "model" in the question refers to car_names.model rather than model_list.model, "cars" actually refers to both cars_data and car_names (but not car_makers) for the purpose of table joining. to resolve the column/table references properly, the semantic parser must take into account both the known schema relations (e.g. foreign keys) and the question context. prior work @xcite addressed the schema representation problem by encoding the directed graph of foreign key relations in the schema with a graph neural network (gnn). while effective, this approach has two important shortcomings. first, it does not contextualize schema encoding with the question, thus making reasoning about schema linking difficult after both the column representations and question word representations are built. second, it limits information propagation during schema encoding to the predefined graph of foreign key relations. the advent of self-attentional mechanisms in nlp @xcite shows that global reasoning is crucial to effective representations of relational structures. however, we would like any global reasoning to still take into account the aforementioned schema relations. in this work, we present a unified framework, called rat-sql, 1 for encoding relational structure in the database schema and a given question. it uses relation-aware self-attention to combine global reasoning over the schema entities and question words with structured reasoning over predefined schema relations. we then apply rat-sql to the problems of schema encoding and schema linking. as a result, we obtain 57.2% exact match accuracy on the spider test set. at the time of writing, this result is the state of the art among models unaugmented with pretrained bert embeddings -and further reaches to the overall state of the art (65.6%) when rat-sql is augmented with bert. in addition, we experimentally demonstrate that rat-sql enables the model to build more accurate internal representations of the question's true alignment with schema columns and tables. ## related work semantic parsing of nl to sql recently surged in popularity thanks to the creation of two new multi-table datasets with the challenge of schema generalization -wikisql @xcite and spider @xcite . schema encoding is not as challenging in wikisql as in spider because it lacks multi-table relations. schema linking is relevant for both tasks but also more challenging in spider due to the richer nl expressiveness and less restricted sql grammar observed in it. the state of the art semantic parser on wikisql (he et al., 1 relation-aware transformer. the recent state-of-the-art models evaluated on spider use various attentional architectures for question/schema encoding and ast-based structural architectures for query decoding. irnet @xcite encodes the question and schema separately with lstm and self-attention respectively, augmenting them with custom type vectors for schema linking. they further use the ast-based decoder of @xcite to decode a query in an intermediate representation (ir) that exhibits higher-level abstractions than sql. @xcite encode the schema with a gnn and a similar grammar-based decoder. both works emphasize schema encoding and schema linking, but design separate featurization techniques to augment word vectors (as opposed to relations between words and columns) to resolve it. in contrast, the rat-sql framework provides a unified way to encode arbitrary relational information among inputs. concurrently with this work, @xcite published global-gnn, a different approach to schema linking for spider, which applies global reasoning between question words and schema columns/tables. global reasoning is implemented by gating the gnn that encodes the schema using the question token representations. this differs from rat-sql in two important ways: (a) question word representations influence the schema representations but not vice versa, and (b) like in other gnn-based encoders, message propagation is limited to the schema-induced edges such as foreign key relations. in contrast, our relation-aware transformer mechanism allows encoding arbitrary relations between question words and schema elements explicitly, and these representations are computed jointly over all inputs using self-attention. we use the same formulation of relation-aware self-attention as @xcite . however, they only apply it to sequences of words in the context of machine translation, and as such, their relation types only encode the relative distance between two words. we extend their work and show that relationaware self-attention can effectively encode more complex relationships within an unordered set of elements (in our case, columns and tables within a database schema as well as relations between the schema and the question). to the best of our knowledge, this is the first application of relation-aware self-attention to joint representation learning with both predefined and softly induced relations in the input structure. hellendoorn et al. (2020) develop a similar model concurrently with this work, where they use relation-aware self-attention to encode data flow structure in source code embeddings. @xcite use a heterogeneous graph of kb facts and relevant documents for open-domain question answering. the nodes of their graph are analogous to the database schema nodes in rat-sql, but rat-sql also incorporates the question in the same formalism to enable joint representation learning between the question and the schema. ## relation-aware self-attention first, we introduce relation-aware self-attention, a model for embedding semi-structured input sequences in a way that jointly encodes pre-existing relational structure in the input as well as induced "soft" relations between sequence elements in the same embedding. our solutions to schema embedding and linking naturally arise as features implemented in this framework. consider a set of inputs @xmath0. in general, we consider it an unordered set, although x i may be imbued with positional embeddings to add an explicit ordering relation. a self-attention encoder, or transformer, introduced by @xcite , is a stack of self-attention layers where each layer (consisting of h heads) transforms each x i into y @xmath1: @xmath2. one interpretation of the embeddings computed by a transformer is that each head of each layer computes a learned relation between all the input elements x i , and the strength of this relation is encoded in the attention weights α (h) ij . however, in many applications (including text-to-sql parsing) we are aware of some preexisting relational features between the inputs, and would like to bias our encoder model toward them. this is straightforward for non-relational features (represented directly in each x i ). we could limit the attention computation only to the "hard" edges where the preexisting relations are known to hold. this would make the model similar to a graph attention network @xcite , and would also impede the transformer's ability to learn new relations. instead, rat provides a way to communicate known relations to the encoder by adding their representations to the attention mechanism. @xcite describe a way to represent relative position information in a self-attention layer by changing equation (1) as follows: here the r ij terms encode the known relationship between the two elements x i and x j in the input. while shaw et al. used it exclusively for relative position representation, we show how to use the same framework to effectively bias the transformer toward arbitrary relational information. consider r relational features, each a binary relation r ## rat-sql we now describe the rat-sql framework and its application to the problems of schema encoding and linking. first, we formally define the text-to-sql semantic parsing problem and its components. in the rest of the section, we present our implementation of schema linking in the rat framework. ## experiments we implemented rat-sql in pytorch @xcite . during preprocessing, the input of questions, column names and table names are tokenized and lemmatized with the standfordnlp toolkit @xcite . within the encoder, we use glove @xcite word embeddings, held fixed in training except for the 50 most common words in the training set. for rat-sql bert, we use the wordpiece tokenization. all word embeddings have dimension 300. the bidirectional lstms have hidden size 128 per direction, and use the recurrent dropout method of @xcite with rate 0.2. we stack 8 relation-aware self-attention layers on top of the bidirectional lstms. within them, we set d @xmath3, @xmath4, and use dropout with rate 0.1. the position-wise feed-forward network has inner layer dimension 1024. inside the decoder, we use rule embeddings of size 128, node type embeddings of size 64, and a hidden size of 512 inside the lstm with dropout of 0.21. ## conclusion despite active research in text-to-sql parsing, many contemporary models struggle to learn good representations for a given database schema as well as to properly link column/table references in the question. these problems are related: to encode & use columns/tables from the schema, the model must reason about their role in the context of the question. in this work, we present a unified framework for addressing the schema encoding and linking challenges. thanks to relation-aware self-attention, it jointly learns schema and question representations based on their alignment with each other and schema relations. empirically, the rat framework allows us to gain significant state of the art improvement on text-to-sql parsing. qualitatively, it provides a way to combine predefined hard schema relations and inferred soft self-attended relations in the same encoder architecture. this representation learning will be beneficial in tasks beyond text-to-sql, as long as the input has some predefined structure.