Simple and effective data augmentation for compositional generalization (2024)

Abstract: Compositional generalization, the ability to predict complex meanings from training on simpler sentences, poses challenges for powerful pretrained seq2seq models. In this paper, we show that data augmentation methods that sample MRs and backtranslate them can be effective for compositional generalization, but only if we sample from the right distribution. Remarkably, sampling from a uniform distribution performs almost as well as sampling from the test distribution, and greatly outperforms earlier methods that sampled from the training distribution.We further conduct experiments to investigate the reason why this happens and where the benefit of such data augmentation methods come from.

## introduction compositional generalization is the ability of a system to correctly predict the meaning of complex sentences when trained only on simpler sentences @xcite . it has been studied in particular detail in the context of semantic parsing, the task of mapping sentences to symbolic meaning representations. recent findings suggest that even powerful pretrained seq2seq models such as bart @xcite and t5 @xcite , which excel at broad-coverage semantic parsing @xcite , perform very poorly on compositional generalization @xcite . one promising method for compositional generalization is data augmentation @xcite @xcite . the idea is to generate additional training data by sampling from an augmentation distribution, in the hope that a model trained on the augmented data will generalize better to the out-of-distribution test data. data augmentation for semantic parsing is complicated by the fact that it needs to recombine matching pieces of the sentence and of the meaning representation, but this matching is not made explicit in the training data. many approaches therefore use somewhat complex methods to e.g. induce synchronous grammars @xcite . as a simpler alternative, @xcite proposed to learn only a grammar for generating meaning representations, and then to use backtranslation to map the sampled meaning representations into sentence-mr pairs. the effectiveness of a data augmentation regime depends on the distribution from which the augmented data is sampled. @xcite sample from the training distribution and find that this improves semantic parsing accuracy on out-ofdistribution text-to-sql tasks. however, it is not clear that augmenting from the training distribution is universally helpful, especially on compositional generalization tasks where the test instances are deliberately designed to be unlikely under the training distribution. in this paper, we investigate the impact that the choice of augmentation distribution has on the ability of a semantic parser to generalize compositionally. we compare wang et al.'s approach (fit a grammar for meaning representations to the training data) to an approach where we fit the mr grammar to the test data (as an upper bound). finally, we look at an mr grammar with uniform rule weights. figure 1 shows the difference between these three methods. in an evaluation across four compositional generalization datasets (cogs, cfq, geo-query, scan), we find that augmentation based on the test data strongly outperforms augmentation based on the training data; but surprisingly, augmentation with the uniform grammar is almost as effective as augmentation from the test data. this can be partially explained by the ability of the uniform grammar to contribute unseen local structures @xcite and assign low perplexity to the test mrs. our findings point to a remarkably simple method for effective data augmentation for compositional generalization: obtain a grammar for the meaning representations (a formal language), set uniform rule weights, sample, and backtranslate. we will release our code online 1 . similar to our method, @xcite adopt iterative back-translation for compositional semantic parsing, but they directly use a subset of meaning representations from development or test set as augmented meaning representations. our work instead shows that meaning representations generated from a probabilistic grammar still work. closest in spirit to this paper is the work of @xcite , who also sample only meaning representations and generate input sentences through backtranslation. figure 2 illustrates their method. the key difference to our work is that we explore the impact of augmentation distributions. ## methodology our method consists of two steps: sample meaning representations and then backtranslate them into natural language sentences. it exploits the fact that in many realistic use cases of a semantic parser, one can generate arbitrary amounts of symbolic meaning representations from a grammar: these are from a formal language, and the developer of a semantic parser either has access to a grammar for this formal language or can easily write one. ## discussion the surprising finding so far is that across all four compositional generalization datasets, augmenting from p unif orm performs on par with p test . this seems counterintuitive: the uniform augmentation strategy has no knowledge of the test data's distribution, and one would expect that augmentation data sampled from a grammar-based approximation to the test distribution should perform much better. we therefore investigate this finding in detail. augmentation data statistics we present statistics of the generated augmentation data in the bigram coverage is determined by dividing the number of observed bigrams in the test set that also exist in the training set by the total count of possible bigrams in the test set. instance coverage is calculated analogously. as expected, p test always yields the highest coverage values on the meaning representations, suggesting that the mr grammar approximates the test distribution effectively. on the other hand, instancelevel coverage on the english side does not grow very high for any dataset. this indicates that the backtranslation model, which is trained on the original in-distribution data, still struggles to produce novel recombinations of the english sentences. p unif orm is on par with p test on many measures and datasets, and considerably outperforms p train . this suggests that novel structural combinations are judged unlikely based on the training distribution, or are simply assigned a probability of zero because structures were entirely unobserved. it is remarkable that p test and p unif orm produce meaning representations of similar length on cogs and could therefore be capable of generating augmentation data of similar structural complexity. at the same time, p test achieves a significantly higher parsing accuracy on the pp and cp recursion generalization types. a plot of the distribu4 6 8 10 12 pp recursion depth 0 200 400 count p_train p_test p_uniform gold 4 6 8 10 12 cp recursion depth 0 200 400 count p_train p_test p_uniform gold figure 5: depth distribution of train set for cogs. ## conclusion we investigated the impact of the choice of augmentation distribution on compositional generalization. we found that a pcfg for the meaning representations with uniform rule weights supports much more effective data augmentation than one that is trained on the training data, and almost on part with one that is trained on the test data. a detailed analysis revealed that this is because the uniform grammar both achieves low perplexity on the test meaning representations and greatly improves structural coverage. thus, sampling meaning representations from a uniform pcfg and backtranslating them into natural-language sentences can serve as a simple and efficient data augmentation strategy for compositional generalization. it would be interesting to investigate the space of augmentation distributions in more detail in future work to see, for instance, how the generation of structurally even more di-verse augmentation instances can be encouraged. our findings also suggest that compositional generalization in mr-to-text generation tasks @xcite , which is still an underexplored area, is also an interesting direction for future research.