Bridging the Generalization Gap in Text-to- SQL Parsing with Schema Expansion (2022)

Abstract: Text-to-SQL parsers map natural language questions to programs that are executable over tables to generate answers, and are typically evaluated on large-scale datasets like Spider (Yu et al., 2018). We argue that existing benchmarks fail to capture a certain out-of-domain generalization problem that is of significant practical importance: matching domain specific phrases to composite operation over columns. To study this problem, we first propose a synthetic dataset along with a re-purposed train/test split of the Squall dataset (Shi et al., 2020) as new benchmarks to quantify domain generalization over column operations, and find existing state-of-the-art parsers struggle in these benchmarks. We propose to address this problem by incorporating prior domain knowledge by preprocessing table schemas, and design a method that consists of two components: schema expansion and schema pruning. This method can be easily applied to multiple existing base parsers, and we show that it significantly outperforms baseline parsers on this domain generalization problem, boosting the underlying parsers’ overall performance by up to 13.8% relative accuracy gain (5.1% absolute) on the new Squall data split.

## introduction text-to-sql parsing is the task of translating natural language questions over provided tables to sql queries which can be executed to produce answers. in recent years, with the availability of large-scale datasets (e.g., @xcite , neural semantic parsers have witnessed significant success on this task. however, recent work @xcite has suggested that these state-of-the-art parsers are far from successful in terms of out-of-domain generalization in real scenarios, where users may ask questions related to potentially very large tables with the goal of improving their productivity (e.g., while they are viewing or editing a large excel spreadsheet). ## background task. semantic parsing has been widely studied in the context of multiple other tasks like instruction following @xcite , code generation @xcite , knowledge graph question answering @xcite , etc. we focus on using tables as the context in which semantic parsing is performed, where the goal is to translate pairs of natural language questions and tables to executable sql queries, also known as text-to-sql parsing @xcite . note that, while we focus on questions in the english language, there exists prior work on multilingual semantic parsing as well @xcite and the contributions of our work also apply there. formally, our goal is to map a pair (q, t ), where q is a natural language question and t is a table, to an executable program π that, when executed against table t , will produce the answer α to question q. we focus on the fully-supervised setting where the target executable program π * is provided as supervision for training our parser. out-of-domain generalization. generalization in machine learning is often defined as the ability to do well on a test set after learning from a training set, where all examples in both sets are drawn independently from the same distribution (i.i.d. generalization). however, as @xcite argue, in real-world applications such as semantic parsing, the test data may involve new compositional structures (compositional generalization), or new domains (domain generalization) that are not encountered during training. existing work in compositional generalization for semantic parsing has focused on using synthetic datasets (e.g., @xcite , or repartitioning existing text-to-sql datasets into new train and test splits (e.g., @xcite . both approaches have generally shown that compositional generalization remains an important challenge (e.g., @xcite . we focus on the arguably even more challenging domain generalization problem, also known as domain adaptation, where entire domains may never be encountered during training or may only be encountered a small number of times @xcite . even though this problem has been studied extensively in the context of classification @xcite , machine translation @xcite , and question answering (talmor and @xcite , it remains underexplored for semantic parsing. to be applicable in real scenarios, semantic parsers must be able to generalize to new domains since collecting domain-specific labeled data is often prohibitively expensive. recent approaches have focused on data synthesis @xcite , meta-learning @xcite , relation-aware schema encoding @xcite , and encoder pretraining @xcite @xcite . in this paper, we hone in on one aspect of domain generalization that we shall broadly refer to as column operations and which was introduced in §1 and illustrated in figure 1 . evaluation benchmarks. text-to-sql parsing became popular after the introduction of large-scale datasets and evaluation benchmarks. @xcite first introduced wikisql, which contains wikipedia tables paired with questions and annotated with sql queries, albeit the queries are generated from a limited set of templates. spider was introduced by @xcite the following year. it contains more complex questions and sql queries and focuses on generalizing to previously unseen database schemas, but the dataset has the artifact from its annotation design that the references columns are often mentioned verbatim in the natural language questions. @xcite attempt to address this limitation by repartitioning spider to produce a more realistic benchmark, and @xcite propose a challenging test set from kaggle for evaluat-ing parsers trained on spider dataset. however, as @xcite point out, spider also uses a simplified setting which excludes examples that involve multiple columns (e.g., adding two columns together), as well as ones that require background knowledge. these benchmarks are thus limited in their usefulness for evaluating parsers in real-world settings where they may encounter complex questions that require mapping specific phrases to expressions over table columns, rather than to a single column. furthermore, while both wikisql and spider assume "simple" tables with only string-or number-valued columns, in practice we may encounter tables where the columns themselves may have structured types (e.g., timespan). for example, consider the table shown on the top left of figure 2 . in this case, the "term" column is of type timespan and consists of two numbers that represent the beginning and the end of the timespan. in this case, users may ask questions that require constructing expressions to access nested elements from the "term" column (e.g., "how long was pier's term?"). recently, @xcite introduced squall, a dataset that annotates wikitableques-tions @xcite with sql queries and refined column types like date, score, (t1, t2), and list[t]. however, squall distributes tables evenly between the train and test splits, thus not allowing us to evaluate the kind of out-of-domain generalization we are interested in. therefore as we will show in the following section, we aim to address this limitation by repartitioning squall into new train and test splits. neural text-to-sql parsers. neural encoder-decoder models have recently gained popularity for text-to-sql parsing (e.g., @xcite . we focus on two models that represent the current state-of-the-art for squall and spider, respectively: seq2seq of shi et al. (2020) foot_0 and smbop of @xcite . both models concatenate the question with a textual representation of the table schema, separated by a special @xcite token, and feed the combined sequence to a pre-trained instance of the bert @xcite language model. the activations of the last layer represent the encoded representations of the question and the table schema. seq2seq then uses a autoregressive decoder, which represents programs as token sequences and at each decoding step it: (1) predicts the next token type (i.e., whether the next token is a sql keyword, a column name, or a literal value), and (2) predicts the token conditioned on its type. smbop, on the other hand, uses bottom-up decoding, which represents programs as abstract syntax trees and constructs these trees in a bottom-up fashion (i.e., it starts by predicting the leaf nodes and then recursively composes generated sub-trees into new trees and ranks them, in a way that resembles beam search), until it reaches the tree root. we refer the reader to the aforementioned papers for details. ## proposed evaluation benchmarks our goal is to design an evaluation benchmark that has the following out-of-domain generalization properties: (i) the training data involves a different set of domains from the test data, (ii) the questions and tables that appear in the train and test data are non-overlapping, not only in terms of the domains they belong to, but also in terms of the program fragments that they contain, and (iii) to simulate the more challenging setting that is often encountered in real applications, the test data is biased to contain more examples that involve both nested column access operations, like getting the start of a "term" in figure 2 , as well as composite column expressions, like getting the duration of a "term". to this end, we propose a new synthetic dataset and a repartitioning of the squall dataset into new train/test splits. ## proposed method in this section we propose a simple approach for tackling this specific out-of-domain generalization problem that ought to serve as evidence that it is a real problem and that it is solvable, as well as a reference point for evaluating future approaches. our approach consists of two new components that can be used in combination with any existing text-to-sql parser: schema expansion and schema pruning. these components interact with the parser by preprocessing the table that is fed to it as input. this is illustrated in figure 2 . as discussed in §1, there are two kinds of challenges related to out-of-domain generalization in text-to-sql parsing, column matching and column operations, with the latter being more challenging. the goal of schema expansion is to reduce column operation challenges to column matching by adding synthetic columns to the table schema. these synthetic columns correspond to expressions or accessors over existing columns (e.g., a column that represents the sum of two columns). rather than learning (or memorizing) the ways in which different types of columns can be composed together, we propose to inject prior knowledge as to what kind of symbolic operations are possible based solely on the column types in a schema. this reduces column operations to column matching by effectively bringing the target programs closer to their surface form in the natural language question. for example, "income" can now map to a synthetic column that corresponds to the sum of "salary" and "stock" instead of having the parser produce the sum expression directly. since our expansion is based on column types, we argue that it is reasonable to assume that all schemas are typed and our expansion could be applied to any new domain. it is also worth noting that even though our templates may not cover all cases, foot_2 when applying our method to new domains, developers can declare a few templates of their interest and apply schema expansion on these templates to create parser-friendly schemas. this would be more cost-effective compared to collecting large in-domain training data for training the parser. naturally, having a component that expands the table schema means that we may end up with large schemas that the parser has to deal with, which will often involve a lot of irrelevant columns (partially because the schema expansion component does not peek at the question). this can result in increased latency which is not desirable in real-world systems. to this end, we introduce a schema pruning component that looks at both the expanded table schema and the question and decides which columns to prune before invoking the parser. it can be argued that this pruning is as hard as parsing itself, but there is evidence from other areas that it can indeed be helpful (e.g., vocabulary selection; @xcite . as we shall show schema pruning can actually provide an additional boost in accuracy, depending on architecture of the underlying parser. ## experiments we performed experiments on the two proposed benchmarks (as well as the existing version of the squall benchmark), using the two current state-of-the-art parser architectures presented in §2 in combination with our proposed schema expansion and pruning components. ## conclusion in this paper, we introduced and focused on column operations, an important challenge related to out-ofdomain generalization for text-to-sql parsing. we proposed two new evaluation benchmarks-one based on a new synthetic dataset and one based on a repartitioning of the squall dataset-and showed that current stateof-the-art parsers significantly underperform when it comes to this form of generalization. we then proposed a simple way to incorporate prior domain knowledge to the parser via a new component called schema expansion that allows us to reduce the column operations challenge to column matching; an arguably easier challenge. we also introduced a schema pruning component allowing us to scale schema expansion, and showed that when paired together, these two components can boost the performance of existing text-to-sql parsers by a significant amount (up to 13.8% relative accuracy gain / 5.1% absolute in our experiments). through column expansion, we created a new table schema that is more friendly to downstream parsers. our work uses heuristics based schema expansion and works well when limited to columns that have specified types (e.g., scores or timespans), but our synthetic experiments suggest much larger potential on this problem. we hope this work could motivate future research on creating a parserfriendly table ontology. future work could explore learning approaches that use models to automatically expand any table schema, for example, by showing appropriate prompts to ask pre-trained language models to tackle it @xcite .