Measuring and Improving Compositional Generalization in Text-to- SQL via Component Alignment (2022)

Abstract: In text-to-SQL tasks — as in much of NLP — compositional generalization is a major challenge: neural networks struggle with compositional generalization where training and test distributions differ. However, most recent attempts to improve this are based on word-level synthetic data or specific dataset splits to generate compositional biases. In this work, we propose a clause-level compositional example generation method. We first split the sentences in the Spider text-to-SQL dataset into sub-sentences, annotating each sub-sentence with its corresponding SQL clause, resulting in a new dataset Spider-SS. We then construct a further dataset, Spider-CG, by composing Spider-SS sub-sentences in different combinations, to test the ability of models to generalize compositionally. Experiments show that existing models suffer significant performance degradation when evaluated on Spider-CG, even though every sub-sentence is seen during training. To deal with this problem, we modify a number of state-of-the-art models to train on the segmented data of Spider-SS, and we show that this method improves the generalization performance.

## introduction neural models in supervised learning settings show good performance on data drawn from the training distribution. however, generalization performance can be poor on out-of-distribution (ood) samples @xcite @xcite . this might be the case even when the new samples are composed of known constituents; e.g., on the scan dataset @xcite , many models give incorrect predictions for the input "jump twice and walk", even when "jump twice", "walk", and "walk twice" are seen during training. this 1 our code and dataset are available at https://github.com/ygan/spiderss-spidercg (often lacking) ability to generalize to novel combinations of elements observed during training is referred to as compositional generalization. previous work on compositional generalization in text-to-sql focuses on query split. for example, @xcite propose tmcd split based on sql atoms and compounds analysis and question split based on length. @xcite proposes a query template-based split with word substitution that was much more challenging than the question split. however, these splits are limited by the dataset content, making it difficult to construct a challenging benchmark while ensuring that every question phrase (sub-sentence) appears in the training set. previous works @xcite @xcite improve generalization by enhancing the model's component awareness. similarly, @xcite and @xcite propose span-based semantic parsers that predict a sub-program over an utterance span. however, these works are based on datasets where component alignment is relatively easy to achieve; but for more complex text-to-sql, their methods cannot be used directly. for example, as shown in the lower part of figure 1 , to align the sub-sentence with the sub-sql, the algorithm needs to know that 'youngest' corresponds to 'age', and 'weigh' corresponds to 'weight'. for small or single-domain settings, such an alignment algorithm can be built by establishing rules; however, there is currently no simple and feasible alignment method for large complex cross-domain text-to-sql, as in e.g. the spider benchmark @xcite . in this work, we first introduce a new dataset, spider-ss (ss stands for sub-sentence), derived from spider @xcite ; figure 1 compares the two. to build spider-ss, we first design a sentence split algorithm to split every spider sentence into several sub-sentences until indivisible. next, we annotate every sub-sentence with its corresponding sql clause, reducing the difficulty of this task by using the intermediate representation language natsql @xcite , which is simpler and syntactically aligns better with natural language (nl). spider-ss thus provides a new resource for designing models with better generalization capabilities without designing a complex alignment algorithm. furthermore, it can also be used as a benchmark for evaluating future alignment algorithms. to our knowledge, this is the first sub-sentence-based text-to-sql dataset. our annotated spider-ss provides us with subsentences paired with natsql clauses, which serve as our elements. based on spider-ss, we then construct a further dataset spider-cg (cg stands for compositional generalization), by substituting sub-sentences with those from other samples, or composing two sub-sentences to form a more complicated sample. spider-cg contains two subsets; figure 2 shows one example for each. the first subset contains 23,569 examples generated by substituting sub-sentences; we consider most data in this subset as in-distribution. the second subset contains 22,030 examples generated by appending sub-sentences, increasing the length and complexity of the sentence and the sql query compared to the original samples; we consider this subset as ood. we demonstrate that when models are trained only on the original spider dataset, they suffer a significant performance drop on the second ood subset of spider-cg, even though the domain appears in the training set. to improve the generalization performance of text-to-sql models, we modify several previous state-of-the-art models so that they can be applied what is the name and nation of the singer subsentence: spider-ss : select singer.name select singer.country natsql: what are the names of the singers subsentence: select singer.name natsql: who have a song having 'hey' in its name? subsentence: where concert.song_name like '%hey%' natsql: example-1: who performed in a concert in 2014? subsentence: where concert.@xmath0: example-2: what is the name and nation of the singer who performed in a concert in 2014? sentence: spider-cg : select singer.name, singer.country where concert.@xmath1: subset-1: sub-sentence substitution in example 1 and 2 2 what is the name and nation of the singer who have a song having 'hey' in its name and who performed in a concert in 2014? sentence: select singer.name, singer.country where concert.song_name like '%hey%' and concert.@xmath2: subset-2: example-1 append a sub-sentence from example-2 figure 2: two spider-cg samples generated by: (1) substituting the sub-sentence with one from another example; or (2) composing sub-sentences from 2 examples in spider-ss. to the spider-ss dataset, with the model trained sub-sentence by sub-sentence. this modification obtains more than 7.8% accuracy improvement on the ood subset of spider-cg. in short, we make the following contributions: ## model existing text-to-sql models input a sentence and output the corresponding sql query. so the easiest way to think of using the spider-ss dataset is to train the model where inputting sub-sentence and outputting the corresponding natsql clauses. however, this method is not workable because it will lose some essential schema information. for example, if you only look at the third sub-sentence in figure 1 , you do not know whether it enquires about the weight of pets or people. in order to take into account the context and the sub-sentence data of spider-ss, we propose that a seq2seq model can encode the whole sentence but decode only the sub-sentence. figure 6 presents the workflow of encoding the whole sentence but only decoding the sub-sentence of 'who is older than ten' and outputting the corresponding natsql clause. based on this modification, a seq2seq text-to-sql model can be adapted to the spider-ss. although previous span-based semantic parsers @xcite can work with aligned annotations based on the spider-ss dataset, none of them are designed for complex text-to-sql problems. our modification idea is similar in principle to the span-based semantic parsers, but we did not change the existing model according to the span-based because our modification idea has a smaller workload. in general, we can make the seq2seq-based textto-sql models adapt to the spider-ss in three steps. (1) data preprocess. split the spider-ss examples by sub-sentence. for example, the example in figure 6 is split to two examples shown in figure 7 . (2) model modification. after data preprocessing, there are two input data for a model. the first input is an entire question that directly goes to the encoder. the second input is the sub-sentence indexes, which are used to select the encoder output, as shown in figure 6 . (3) output combination. since the model output may be only a clause, not a complete natsql query, we generate the final nat-sql query after the model outputting all natsql clauses. ## limitation of this work the spider-ss and spider-cg are based on spider, an english large-scale text-to-sql dataset, and we did not extend the experiment to other language and text-to-sql datasets. therefore, we did not verify whether these methods work well in other languages and datasets. besides, since this work is based on natsql, there will be around 5% of natsql that can not be converted to the correct sql. compositional generalization for semantic parsing. compositional generalization for semantic parsing has captured wide attention recently @xcite @xcite . most prior works on text-to-sql tasks focus on the crossdomain generalization, which mainly assess how the models generalize the domain knowledge to although both the tmcd split and our spider-cg can be used to evaluate the text-to-sql compositional generalization ability, their problem setting is different. tmcd split is based on sql syntax structure, while spider-cg is based on the natural language syntax, which leads to different requirements for compositional generalization ability. for example, tmcd splits requires model learning "give me the name of students who is the oldest" can predict the "give me the name of the oldest student" since their sql is the same. spider-cg does not expect the model to do so because the syntax of questions is different, i.e., "give me the name of students who is the oldest" contains two sub-sentences, and none of them is close to the "give me the name of the oldest student". in other words, spider-cg requires the model learning "list the id of the oldest dog" can predict the "give me the name of the oldest student". our model is inspired by prior works on neural parsers constructed to capture granular information from a whole. @xcite describe a span-level supervised attention loss that improves compositional generalization in semantic parsers. @xcite propose spanbasedsp, a parser that predicts a span tree over an input utterance, and dramatically improves performance on splits that require compositional generalization. @xcite propose the neural-symbolic stack machine (ness), which integrates a symbolic stack machine into a seq2seq generation framework, and learns a neural network as the controller to operate the machine. however, these works are based on datasets where component alignment is relatively easy to achieve; but for more complex text-to-sql, their methods cannot be used directly. our proposed spider-ss can be used to replace or evaluate the alignment algorithm. ## conclusion we introduce spider-ss and spider-cg for measuring compositional generalization of text-to-sql models. specifically, spider-ss is a human-curated sub-sentence-based text-to-sql dataset built upon the spider benchmark. spider-cg is a synthetic text-to-sql dataset constructed by substituting and appending sub-sentences of different samples, so that the training and test sets consist of different compositions of sub-sentences. we found that the performance of previous text-to-sql models drop dramatically on the spider-cg ood subset, while modifying the models to fit the segmented data of spider-ss improves compositional generalization performance.