Exploring Unexplored Generalization Challenges for Cross-Database Semantic Parsing (2020)

Abstract: We study the task of cross-database semantic parsing (XSP), where a system that maps natural language utterances to executable SQL queries is evaluated on databases unseen during training. Recently, several datasets, including Spider, were proposed to support development of XSP systems. We propose a challenging evaluation setup for cross-database semantic parsing, focusing on variation across database schemas and in-domain language use. We re-purpose eight semantic parsing datasets that have been well-studied in the setting where in-domain training data is available, and instead use them as additional evaluation data for XSP systems instead. We build a system that performs well on Spider, and find that it struggles to generalize to our re-purposed set. Our setup uncovers several generalization challenges for cross-database semantic parsing, demonstrating the need to use and develop diverse training and evaluation datasets.

## introduction semantic parsing is the task of mapping natural language utterances to formal meaning representations, and has been studied in tasks including instruction following, evaluating sentence meaning, and building interfaces to knowledge bases. in this paper, we focus on the task of mapping from natural language utterances to sql queries executable in a database. most prior work in mapping from natural language to sql queries train and test the system on a single database. we refer to this setup as single-database semantic parsing (ssp). well-studied datasets used in the ssp setting include geoquery @xcite and atis @xcite . however, semantic parsing systems should be able to generalize to new domains and databases, figure 1 : examples of generalization challenges revealed in the cross-database semantic parsing (xsp) setting. the top three examples are from datasets originally studied in the single-database (ssp) setting. without in-domain training data, generalization is more difficult, requiring identifying entities, mapping unfamiliar phrases and entities to the database, and using new and complex database schemas. in contrast, existing xsp evaluation data such as spider simplifies some of these challenges, for example by including utterances that closely match their paired sql query. as it is often cost-prohibitive to collect a sufficient number of training examples for all possible databases. several datasets, including spider @xcite , were proposed to evaluate this dimension of generalization. these datasets include examples grounded in multiple databases, distinguishing between training databases and evaluation databases. we refer to this setup as cross-database semantic parsing (xsp). while these datasets have been valuable in understanding and addressing some of the additional generalization challenges introduced by xsp, current evaluation of xsp systems has been limited to datasets designed for xsp. this limits the types of generalization challenges studied to those introduced by these datasets. existing xsp evaluation data such as spider simplifies some of these challenges, for example by including utterances that closely match their paired sql query, as shown in last row of figure 1 . this setup misses an important opportunity for studying cross-database semantic parsing: evaluating on challenging datasets designed for singledatabase semantic parsing, like geoquery and atis. while the in-domain challenges of these datasets are relatively well-understood, generalization challenges introduced by studying these datasets in an xsp context have not been addressed. in this paper, we propose a more holistic analysis and evaluation setup for xsp. we propose to evaluate a semantic parsing system not only on evaluation data designed for xsp, but also on datasets that have only been studied in the ssp setting. our repurposed evaluation set includes eight well-studied datasets like atis, but in a completely new setting. instead of training on the original training data for these datasets, we train a single model on training data designed for the xsp setting, and evaluate the trained model on each evaluation dataset. these datasets were collected at different times, by different researchers, and with different motivations. this results in a wide variety of language usage, database structures, and sql styles across datasets, further stressing a system's ability to adapt to unseen datasets. these variations pose many new generalization challenges for cross-database semantic parsing models, where in-domain examples are not available at training time. our proposed xsp evaluation setup addresses several evaluation challenges posed by these dataset variations. with our proposed setup, we are able to analyze the potential limitations of current crossdatabase semantic parsing models. we uncover and attempt to address several new forms of generalization in cross-dataset semantic parsing. we develop a neural semantic parsing model is com-petitive all public systems on the spider development set, and evaluate its ability to generalize to the evaluation datasets. first, we observe that the datasets originally designed for ssp become much more difficult under the xsp setting, with a notable drop in performance from both the spider development results. second, we experiment with several techniques that improve generalization to the eight evaluation datasets. finally, we provide in-depth qualitative analysis on our results. our results and analysis demonstrate a need for diverse training and evaluation datasets for xsp. our code and experimental setup is available at https://github.com/google-research/ language/tree/master/language/xsp . ## background and related work we focus on the task of semantic parsing for databases. a natural language utterance u is a sequence u 1 , . . . , u |u| , where each u i is a natural language token. the task is to map u to an executable formal query @xmath0, . . . , y |y| executable in a database d, where each y i is a sql query token. ## evaluating on re-purposed data we propose to study the task of xsp by training on datasets designed for xsp, and evaluating on datasets originally designed for ssp. in our full model, we use both the spider foot_0 @xcite and wikisql @xcite datasets for training. for evaluation, in addition to the spider development set, foot_1 we use eight english-language ssp datasets curated by @xcite covering a variety of domains, for example flights, geography, and movies. 3 for each dataset, we evaluate on as much data as possible, excluding test sets. developing evaluation metrics for these repurposed evaluation sets is challenging because of the diversity of sql styles across different databases. @xcite 's proposed evaluation metric compares components of the predicted and correct query, allowing for variation in the exact form of the query, for example using different table aliases. however, it does not capture all possible sql syntax, and fails to cover some of the gold queries in our evaluation datasets. for example, it does not handle assigning an alias to the results of an intermediate select statement. moreover, it does not measure equivalence of values, meaning original task dataset splits # examples (all/filtered) % col. mentioned ssp atis @xcite dev 486/289 0.2 geoquery @xcite train/dev 598/532 32.4 restaurants @xcite splits 0-9 378/ 27 0.0 academic @xcite splits 0-9 196/180 8.2 imdb @xcite splits 0-9 131/107 4.6 yelp @xcite splits 0-9 128/ 54 7.0 scholar @xcite train/dev 599/394 1.0 advising @xcite train/dev 2858/309 0.5 xsp spider @xcite ) dev 1034/ -72. 4 we propose to use variation of execution accuracy as our main metric. execution accuracy over an evaluation set is the proportion of predicted queries which, when executed against the database, result in a table equivalent to the correct query's result. if the correct query requires ordering on the final table, we require the tables be exactly the same; if it does not, we consider result tables equivalent if they contain the same set of rows. we supplement the results with additional baselines and data filtering to address the problem of over-crediting spurious predictions. we report the empty-table prior for each dataset, demonstrating how well a model could perform if predicting incorrect queries that result in empty tables. we create a filtered subset where relative performance of systems is more meaningful, including attempting to remove examples that are impossible to solve without in-domain training data. our heuristics include removing examples with correct queries that result in empty tables, and where the correct query contains a value token that is not copiable from the input. 4 for example, in restaurants, the phrase good restaurant always requires constraining the sql query to restaurants with a rating greater than 2.5, even when the rating is not explicitly mentioned. ## generalization challenges single-database semantic parsing requires recognizing unseen, in-domain entities, understanding new compositional structures, and generating executable representations. cross-database semantic 4 details are included in appendix a. ## model and learning our model takes as input an utterance x and a database schema s. similar to @xcite , we serialize s into a sequence of wordpieces tab is a token noting the beginning of a table schema serialization. t i is the tokenization of table t i 's name. each c i,j is a serialization of a column c i,j , where c i,@xmath1,j + c i,j . ct i,j is a token denoting the type of the column's contents as provided by the database schema, for example numerical or text. c i,j is the tokenization of the column's name. the ordering of table schemas in s and table columns in each t i is arbitrary. 7 the input to the encoder is the concatenation of the query wordpieces and the serialized schema, represented as the sequence of tokens @xmath2. the inputs to the encoder are embedded and passed to a pretrained transformer encoder such as bert @xcite . the decoder is an autoregressive transformer decoder @xcite that attends over the outputs of the encoder and the generated prefix. we use a training set {x (l) , y (l) , s (l) } n @xmath3, gold sql queries, and database schemas. we train the encoder and decoder end-to-end, minimizing the token-level cross-entropy loss of the gold query y (l) . we update the parameters of the pre-trained encoder during training. for training data we use training sets developed for xsp. importantly, to ensure we are evaluating the cross-database setting, our training data does not include examples from the evaluation databases. during inference, we use beam search and execute the highest-probability, syntactically correct prediction. we impose a maximum execution time of 45 seconds for predictions. more details on the model, learning, and evaluation setup are available in appendix b. ## experiments comparison to existing xsp systems our best model performs well on the spider development set. we find that all three techniques for addressing generalization challenges are effective. first, including wikisql in the training data results in better performance than only using spider training data. we hypothesize that this is due to the addi-8 like semql, this conversion is not possible if foreign key relations between predicted tables are not provided or if a given table is referenced more than once in a from clause. this can also result in a lossy or ambiguous conversion if there are multiple foreign key relations between a pair of tables. ours tional domains in wikisql, as well as the larger proportion of examples that require mapping from entities in the utterance to column names @xcite . using sql uf also improves performance, as it produces queries coherent with respect to the schema, for example only selecting columns from tables where the column exists. finally, using value placeholders significantly reduces execution accuracy in all datasets. while masking values decreases exact set match on spider by 10.9%, its effect on execution accuracy can be devastating both for spider and the eight evaluation datasets. this demonstrates the need to consider execution results when evaluating semantic parsing systems. error analysis for each evaluation dataset, we analyze twenty random predictions from the filtered subset. examples of the most common error types are shown in figure 2 , along with the proportion of analyzed predictions in the eight target datasets that contain the error type. appendix d discusses the complete results of error analysis. 40% of errors are caused by comparing an entity to the wrong column, for example searching for 'james bond' in the director.name column when it actually refers to a movie.title. this usually requires using domain knowledge identify to columns that are likely to contain the mentioned entity (section 4.1). 31.1% of errors are caused by missing constraints specified in the utterance, for example by failing to use a relevant entity in the predicted query. 28.8% of errors are also caused by incorrectly identifying entity spans, for example by treating fin 340 as a single entity rather than two separate entities in the database (section 4.1). another common error is predicting the wrong final column. while choosing what to return is difficult for the model due to understanding domain-specific phrases such as how many (20.0% of errors; section 4.1), sometimes the errors are due to dataset conventions (26.9% of errors; section 4.3). for example, the paperid column should be selected instead of the title column in scholar. such dataset conventions could be learned through fewshot learning, where a small number of in-domain training examples are available. our system is required to generalize to larger databases than it was trained on, including more complex compositions of tables (section 4.2). for ## discussion we study the task of cross-database semantic parsing (xsp), where a system that maps natural language utterances to executable sql queries is evaluated on databases unseen at training time. while this task has been studied through datasets developed specifically for xsp, we propose a more holistic evaluation for xsp, where we also evaluate on datasets originally studied in a setting where in-domain training data is available. we identify several new generalization challenges that arise when evaluating in our proposed setup, including identifying entities, mapping entities and domainspecific phrases to a database schema, and generalizing to more complex database schemas. using a model that performs well on evaluation data designed for xsp, we are able to move towards addressing some of the generalization challenges on these additional evaluation sets without any indomain training data. our results and analysis demonstrate the need for developing more holistic evaluation of cross-database semantic parsing using a more diverse set of language and databases. several significant generalization challenges remaining, including improving commonsense and in-domain reasoning and table schema understanding capabilities. some examples in our filtered evaluation set still require reasoning about dataset conventions that are difficult to acquire without indomain training examples. future work could also make the stronger assumption that a small number of in-domain training examples are available, and train and evaluate in a few-shot setting.