Exploring the Compositional Generalization in Context Dependent Text-to- SQL Parsing (2023)

Abstract: In the context-dependent Text-to-SQL task, the generated SQL statements are refined iteratively based on the user input utterance from each interaction. The input text from each interaction can be viewed as component modifications to the previous SQL statements, which could be further extracted as the modification patterns. Since these modification patterns could also be combined with other SQL statements, the models are supposed to have the compositional generalization to these novel combinations. This work is the first exploration of compositional generalization in context-dependent Text-to-SQL scenarios. To facilitate related studies, we constructed two challenging benchmarks named CoSQL-CG and SParC-CG by recombining the modification patterns and existing SQL statements. The following experiments show that almost all current models struggle on our proposed benchmarks. Furthermore, we found that better aligning the previous SQL statements with the input utterance could give models better combinatorial generalization ability. Based on these observations, we propose a method name p-align to improve the combinatorial generalization of Text-to-SQL models. Further experiments validate the effectiveness of our model.

## introduction recently, the poor generalization of semantic parsing models to out-of-distribution samples is under increasing attention @xcite . these examples are usually obtained by recombining existing structures. for example, in the scan dataset @xcite , models may fail to parse "jump twice and walk" even though "jump twice" and "walk" could be parsed successfully. ## compositional generalization in context-dependent text-to-sql to facilitate the understanding of the following sections, we provide a more detailed explanation of compositional generalization in context-dependent text-to-sql parsing in this section. the template split is a typical compositional generalization setting, where the structure templates in the training and test set are completely separated. our compositional generalization scenario can be viewed as an extension of the template split, where the combination of basic sql templates and modification templates in the training and test set are separated. note that basic sql and modification templates in the test set all appear in the training set individually. for instance, in figure 1 , in the inference phrase, although all the templates are seen during training, their combinations are novel. from another point of view, our compositional generalization scenario could also be viewed as a special case of tmcd split @xcite , where the sql templates and modification templates could be seen as atoms and their combination results are the compounds. note the utterance to the sql templates (first atom) are provided during training, which could be further utilized to improve the compositional generalization (section 5). ## benchmark construction since there are few data satisfying the compositional generalization setting in the origin sparc and cosql development set. we first construct new benchmarks to facilitate the related research. as illustrated in figure 2 , the benchmark construction process can be divided into four steps. the first step is to filter out context-independent examples; next, modification patterns are extracted from the remaining examples; after that, these modification patterns are combined with other sql statements, and finally, corresponding utterances are generated. ## methods after constructing the sparc-cg and cosql-cg, we further explore how to improve the compositional generalization in context-dependent textto-sql parsing. according to the previous works @xcite , the key to improving the compositional generalization is to construct better component alignment between inputs and outputs. in the contextdependent text-to-sql settings, the utterancequery pair of previous interactions could be utilized to align input utterances and output queries. based on this motivation, we propose p-align to improve the compositional generalization of existing text-to-sql models. note that our method follows the common practice of most competitive text-to-sql models which take the concatenation of all utterances as input. specifically, given the input utterances @xmath0, x 2 , ..., x n ] at the n-th interaction, where x @xmath1, ....x j ] is an utterance with j words, the encoder aims to generate embeddings for each word such that @xmath2. in the origin decoding process, the result query y could be represented as an action sequence [a 1 , ...a t ] and the whole decoding process could be represented as the product of question1 show all visitors question2 order them by their memership question3 just show their names ## experiment in this section, we first perform more detailed statistics on our constructed sparc-cg and cosql-cg. then we further analyze our benchmarks with current competitive text-to-sql models. finally, several experiments are conducted to verify the effectiveness of our p-align method.