Exploring Underexplored Limitations of Cross-Domain Text-to- SQL Generalization (2021)

Abstract: Recently, there has been significant progress in studying neural networks for translating text descriptions into SQL queries under the zero-shot cross-domain setting. Despite achieving good performance on some public benchmarks, we observe that existing text-to-SQL models do not generalize when facing domain knowledge that does not frequently appear in the training data, which may render the worse prediction performance for unseen domains. In this work, we investigate the robustness of text-to-SQL models when the questions require rarely observed domain knowledge. In particular, we define five types of domain knowledge and introduce Spider-DK (DK is the abbreviation of domain knowledge), a human-curated dataset based on the Spider benchmark for text-to-SQL translation. NL questions in Spider-DK are selected from Spider, and we modify some samples by adding domain knowledge that reflects real-world question paraphrases. We demonstrate that the prediction accuracy dramatically drops on samples that require such domain knowledge, even if the domain knowledge appears in the training set, and the model provides the correct predictions for related training samples.

## introduction research on cross-domain text-to-sql benchmarks has led to numerous advances. recent works @xcite @xcite have achieved over 70% accuracy on spider benchmark @xcite and over 90% accuracy on wikisql benchmark @xcite , which seems to suggest that existing models already solved most problems in this field. however, the follow-up studies from @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite show that the generalization performance is much worse in more challenging scenarios. for example, @xcite investigate the cases when the explicit mentions of database columns are removed from the question. similarly, @xcite observe that the model accuracy dramatically drops by replacing schema-related words with some synonyms. on the other hand, @xcite find that the generalization to other databases is much worse, due to the distribution shift of both questions and sql queries. these papers introduce important challenges for improving the generalization performance, i.e., the model trained on a cross-domain text-to-sql dataset (e.g., spider @xcite ) does not generalize to a new external database. however, the performance degradation is somehow expected for the following reasons. first, removing the explicit mentions breaks the assumptions that make the schema linking effective. second, sql queries in other databases could come from a different distribution; e.g., according to the hardness criteria defined by spider benchmark, over 40% spider sql queries are medium hardness, but there are less than 10% medium sql queries in the geoquery dataset @xcite . in this work, we demonstrate that the generalization performance could be poor even when both the nl questions and sql queries follow the similar distribution to the training set. specifically, we constructed spider-dk, a challenging variant of the spider development set, with the focus of evaluating the model understanding of domain knowledge. a domain means a certain type of application scenarios; for example, the spider benchmark includes various distinct domains such as geography and university. cross-domain text-to-sql research aims to build a text-to-sql model that can generate correct sql queries and generalize to different domains. therefore, one main challenge of crossdomain text-to-sql generalization is to understand different knowledge required by different domains. ## conclusion we introduce spider-dk, a human-curated dataset based on the spider benchmark for evaluating the generalization of text-to-sql models, with the focus of understanding the domain knowledge. we demonstrate that the performance of existing textto-sql models drops dramatically on spider-dk, even if the domain knowledge appears in the training set. our evaluation indicates that the models do not always understand the underlying domain knowledge for prediction, thus we consider improving the model understanding of domain knowledge as an important direction to achieve the crossdomain generalization.