Shifted PAUQ : Distribution shift in text-to- SQL (2023)

Abstract: Semantic parsing plays a pivotal role in advancing the accessibility of human-computer interaction on a large scale. Spider, a widely recognized dataset for text2SQL, contains a wide range of natural language (NL) questions in English and corresponding SQL queries. Original splits of Spider and its adapted to Russian language and improved version, PAUQ, assume independence and identical distribution of training and testing data (i.i.d split). In this work, we propose a target length split and multilingual i.i.d split to measure compositionality and cross-language generalization. We present experimental results of popular text2SQL models on original, multilingual, and target length splits. We also construct a context-free grammar for the evaluation of compositionality in text2SQL in an out-of-distribution setting. We make the splits publicly available on HuggingFace hub via https://huggingface.co/datasets/composite/pauq

## introduction in this paper, we focus on a subtask of semantic parsing called text-to-sql, which involves mapping natural language (nl) questions to structured query language (sql). sequence-to-sequence (seq-to-seq) models such as rat-sql @xcite , @xcite , and resdsql @xcite have been widely employed for the text-to-sql task. however, recent studies have highlighted the limitations of seq-to-seq models in out-of-distribution settings @xcite @xcite . the evaluation process for text-to-sql models is currently a topic of active research. @xcite determined the difficulty of requests based on the number of sql components, resulting in four distinct categories: "easy", "medium", "hard", and "extra hard". @xcite demonstrated that the division of the popular text-to-sql dataset into training and test sets is inadequate for assessing the model's generalization abilities. @xcite highlighted that it is important to access performance on functional test sets and out-of-distribution examples, as random train and test splits can overestimate real-world performance and miss important error cases. @xcite proposed a text-to-sql specific perturbation benchmark . this benchmark encompasses 17 categories for evaluation on text-to-sql models robustness. @xcite proposed new train and test splits of non-synthetic datasets for modeling out-of-distribution (ood) setting in order to measure compositionality in semantic parsing. in this work, we propose three splits of the existing pauq dataset @xcite , an improved version of spider @xcite , for measuring across-language generalization and compositional generalization. the first split evaluates how models benefit from training in multiple languages of the same task. the split is constructed by joining original splits of russian and english versions of pauq (mpauq). when samples in the train set are longer than samples in the test set named trl. the second split is when samples in the test are longer than samples in the train named tsl. trl examines the ability of the model to generalize to simpler sql without directly learning to construct them. as stated in @xcite , trl evaluates model systematicity, while tsl assesses its productivity. ## proposed split we propose a target length-based split to mimic full shift @xcite in the text-to-sql task for measuring the compositional generalization ability of nlp models. we design split in the following way: we make sure that there is no template intersection in train/test for both trl and tsl splits. tsl is the most complicated split because the model has to recombine known tokens to form new complex and long queries. trl split requires the model to also recombine new tokens but to generate queries of shorter length. for measuring cross-language generalization, we merge english and russian train sets and evaluate english and russian test sets separately. our motivation for the multilingual split is whether training in two languages on the same task can benefit one model. ## baselines and experiments we focus our dataset on two types for generalization evaluation -compositional generalization and generalization across languages (see appendix b for the genbench card @xcite ). we utilized four popular spider models: resdsql decouples schema linking and query generation tasks into two stages. the first stage selects the most relevant schema items for the question. during the second stage, the model learns to decode the sql template of the actual query concatenated with the original actual query. that way, the model can condition generating sql skeleton before the full original query. resdsql uses roberta-large @xcite large for the schema linking and t5-base for query generation. for rupauq and mpauq, we replace t5-base model with mt0-base @xcite . rat-sql is an encoder-decoder model that uses a relation-aware transformer within the encoder to model alignments between database schema and content and question tokens. the decoder of the model is tree-structured and generates an abstract syntax tree in the context-free sql grammar. bridge, in turn, utilizes database schema and content as input to the model. it has an encoderdecoder architecture with the pointer-generator network using beam-search. the model generates queries in execution-guided order. both rat-sql and bridge use the bert-base @xcite language model as an encoder. the details on hyperparameters are presented in the appx. a. ## overall results we evaluate 4 models on our splits -t5-base, res-dsql (with t5-base or mt0-base), rat-sql, and bridge. we trained each model on a split train set and evaluated it on a test set with 3 random seeds and averaged predictions. target length split is our evaluation towards compositional generalization measurements ( ## in-depth compound analysis we wanted to explore the ability of compositional generalization for proposed target length splits. we have developed context-free grammar (cfg) in order to parse queries. in order to analyze, we utilize the concept of compound. for example, we have dataset tokens select, count, sum, goals, teams (goals and teams are table attributes). composition of these tokens such select count goals is called compounds. cfg covers compounds for request, join, condition, group, order, limit. for evaluation, we apply the same masking technique used for split generation for generated sql. then we parse it with cfg to extract compounds (see figure 1 ). for in-depth compound analysis, we propose and evaluate three accuracy metrics derived from our compounds: we calculate the proposed metrics independently for each sample and then average over all test set predictions. in tab. 5, we see that a more advanced resdsql overall model increases only syntax accuracy metric compared to the t5-base on os and trl splits, while other two metrics are relatively close to each other for all splits. ood compound accuracy is significantly lower then compound accuracy (e.g., 0.66 vs 0.94 for resdsql on the trl split, respectively). the trl split compound metrics show that both models are able to generate unseen compounds in approximately 1.5 times better than in the tsl split. ## discussion and conclusion in this work, we have explored two types of generalization -compositional generalization (trl and tsl splits) and across language generalization (mpauq). we have evaluated 4 text-to-sql models on these splits. for an in-depth analysis on compositional generalization, we have developed cfg and proposed two metrics for compound evaluation. our results show that tsl split is the most challenging split for the model. trl split shows that the models are able to generalize to unseen short sql. multilingual split shows that some models can benefit from learning on the translated task to gain performance on individual language. in future work, we plan to perform more experiments to make more compositional generalization splits (e.g., template or target maximum compound divergence splits @xcite ). we also plan to explore different neural architectures and training strategies to enhance the model's ability to handle complex queries and measure compositional generalization and generalization across languages. we hope to prepare the pauq leaderboard and encourage further research in this area.