Searching for Better Database Queries in the Outputs of Semantic Parsers (2023)

Abstract: The task of generating a database query from a question in natural language suffers from ambiguity and insufficiently precise description of the goal. The problem is amplified when the system needs to generalize to databases unseen at training. In this paper, we consider the case when, at the test time, the system has access to an external criterion that evaluates the generated queries. The criterion can vary from checking that a query executes without errors to verifying the query on a set of tests. In this setting, we augment neural autoregressive models with a search algorithm that looks for a query satisfying the criterion. We apply our approach to the state-of-the-art semantic parsers and report that it allows us to find many queries passing all the tests on different datasets.

## introduction generating a database query from a naturallanguage description of the user's intent is a longstanding and important task. in the recent years, most of the community focus was on the spider dataset @xcite , which poses the task in the zero-shot regime, meaning that a method has to generalize to databases unseen at training. the spider dataset contains english questions and sql queries. the progress has been remarkable, and the accuracy has moved from below 30% to above 70%. a part of this success can be attributed to the adoption of pre-trained transformer models like bert @xcite into most of the pipelines. given such progress, it is natural to ask whether we are getting closer to solving the problem. several recent studies have noted that the task might be harder than it looks. @xcite found that many single-database datasets had identical queries in both train and test sets and showed that using such splits effectively reduced the problem to classifying the queries from the train set. @xcite continued the study in the multidatabase setting and showed that the compositional generalization was hard to achieve, and even to measure it, one should be very careful with splits. in a different line of thought, @xcite examined how the models trained on spider generalize to other datasets and reported that generalization was challenging. even within one dataset, many questions have several interpretations leading to different queries, and annotation policies do not cover these ambiguities or cover them differently. acknowledging that the zero-shot setting might be too difficult to tackle as is, we aim to better define and simplify the problem to achieve better results in terms of the number of correctly generated queries. in this task, most modern models produce a distribution over all possible outputs, which can guide the search at the test time. we observe that if the search algorithm has access to a criterion that can evaluate the output by treating it as a database query, the overall method can produce much better results. we consider the following criteria ordered by their "strength": a query is executed without errors, a query produces output from correct columns, a query produces a correct result on one test database, and a query produces correct results on a set of test databases. we experiment with different search methods and report that the complete anytime beam search @xcite outperforms sampling-based alternatives. many practical cases arise when the user is willing to trade off some of their time to improve the output query. our approach allows the user to obtain a better query by interactively guiding the search via providing the target output columns or the answer on one or more test databases. the user is expected to supply this information without the gold query. notably, the execution criterion does not require extra user input but relies on executing generated queries. in addition, the test time database can be out of domain w.r.t. the training set. one can annotate gold queries for fine-tuning on new databases to improve out-of-domain performance, but this can be prohibitively expensive. our approach gives a way to improve out-of-domain results without extra annotated training data. we view our approach as a way to control the trained model leading to a more reliable and responsible query synthesis. for our studies, we used three state-of-the-art models in terms of execution accuracy (with publicly available implementations): t5-3b @xcite fine-tuned to spider by @xcite , bridge @xcite and @xcite . in this paper, we make several observations. the complete anytime beam search works with different search criteria and with all the three models. reasonable results can be obtained with the maximum beam size of 100, which fits on a single modern gpu. searching with the execution criterion can significantly improve the quality of the decoders that generate output as an unconstrained token sequence, e.g., t5, and using such criterion does not require extra user input. searching for the queries that return a correct output on one database allows finding many queries that provide a correct output on that database. therefore, such searching w.r.t. one database can result in false positives, and it is important to evaluate the queries on a set of databases. based on the method of @xcite , we built the test suite of databases for the evaluation. with these test suites, we show that there are multiple false positives among the queries that pass one test, while searching for the queries that pass the test suite produces the outputs of higher quality. finally, we experiment with the generalization of the models trained on spider to the geo-query (zelle and mooney, 1996), imdb, yelp @xcite and academic @xcite datasets. we show that searching w.r.t. different criteria still works under this distribution shift, and searching w.r.t. the criteria with tests is often comparable to fine-tuning the network to a test dataset directly. this paper is organized as follows. in section 2, we review our setting. in section 3, we provide the details of our method. section 4 provides the details of the test-suite construction procedure, section 5 describes the experimental setup. in section 6, we provide the experimental results and discussion. we review some related works in section 7 and conclude in section 8. ## preliminaries we consider the problem of generating queries to databases given the description of the user's intent in natural language in the cross-database setting where the train, validation and test splits contain different databases. models trained in this setting, in theory, can be evaluated on any database. a typical model for the cross-database setting is an encoder-decoder neural network. encoders typically consist of a pre-trained bert-like transformer followed by a specialized encoder that can incorporate the database structure in some form @xcite @xcite . sometimes the bert part is further fine-tuned on database-related objectives @xcite . the encoder input is a concatenation of the tokenized question and a sentence representation of the database schema separated by the special delimiter token. the representation of the database schema consists of the tokenized table and column names and values related to the question. these values are commonly extracted by string matching with question tokens @xcite . decoders are typically autoregressive based on lstm or transformers. some decoders do not check the syntactic correctness of the output and its consistency with the database. some provide output w.r.t. a grammar (yin and neubig, 2017); some use post-hoc checks with parsers. in this paper, we experiment with three models: t5-3b fine-tuned on the spider dataset by @xcite , bridge @xcite and @xcite . we provide a detailed description of these models in section 5.3. ## search with models we now describe our approach to searching for queries on top of a learned model. we first generate full query candidates using a search method and then select the first one that passes the selected search criterion. we show possible search criteria in section 3.1 and search methods in section 3.2. ## test suite construction testing on one database is generally not enough to ensure the semantic correctness of the generated query, but running the query on too many databases can be computationally inefficient. the inefficiency problem is especially acute in our task due to the large number of query candidates that should be tested and several rounds of the searching process. we build our test suites by modifying the method of @xcite , which relies on generating the so-called neighbor queries from a given set of gold queries and randomly sampling databases to distinguish gold queries from as many neighbors as possible. we observed two key drawbacks of the test databases generated by @xcite . first, the test suites contained many databases, and some were unnecessarily large, which resulted in very long testing on them. second, outputs of many queries were often empty (or zero for queries with aggregators) on these test databases. if the output of the gold query is empty on all the elements of the test suite, it cannot be distinguished from trivial dummy queries. this effect is more salient if the gold query returns empty output on the original database. we alleviate these issues by independently generating test databases for each gold query, explicitly limiting the number of rows in each table and putting extra effort into generating at least one database where the gold query returns a non-empty output. the details of our procedure are provided in appendix a. we compare our test suites with the original ones of @xcite on the five considered datasets described in section 5.1. for a fair comparison, we generate the independent sets of neighbor queries for each gold query. these neighbor queries are different from the neighbors generated in the process of creating the test suites. in ## conclusion we studied the search over the outputs of the neural autoregressive models for better database query generation. we considered three state-of-the-art models: t5-3b, bridge and sq-qdmr. we observed that the search algorithms work with multiple criteria for selecting the output query. we also compared the search-augmented methods with the fine-tuned models on the geoquery, imdb, yelp and academic datasets (under the distribution shift) and observed that the method with search can sometimes work even better than fine-tuning. com-pared to fine-tuning, the search-based method does not require additional training data but relies on additional information on each test example. with such properties, our search based methods might be helpful for use cases like interactive query generation or annotating new datasets.