Revisiting the Compositional Generalization Abilities of Neural Sequence Models (2022)

Abstract: Compositional generalization is a fundamental trait in humans, allowing us to effortlessly combine known phrases to form novel sentences. Recent works have claimed that standard seq-to-seq models severely lack the ability to compositionally generalize. In this paper, we focus on one-shot primitive generalization as introduced by the popular SCAN benchmark. We demonstrate that modifying the training distribution in simple and intuitive ways enables standard seq-to-seq models to achieve near-perfect generalization performance, thereby showing that their compositional generalization abilities were previously underestimated. We perform detailed empirical analysis of this phenomenon. Our results indicate that the generalization performance of models is highly sensitive to the characteristics of the training data which should be carefully considered while designing such benchmarks in future.

## introduction according to the principle of compositionality, the meaning of a complex expression (e.g., a sentence) is determined by the meaning of its individual constituents and how they are combined. humans can effectively recombine known parts to form new sentences that they have never encountered before. despite the unprecedented achievements of standard seq-to-seq networks such as lstms and transformers in nlp tasks, previous work has suggested that they are severely limited in their ability to generalize compositionally (lake and baroni, 2018; furrer et al., 2020). problem statement. our work relates to a central challenge posed by compositional generalization datasets such as scan (lake and baroni, 2018) and colors (lake et al., 2019), which we refer to as one-shot primitive generalization: the dataset consists of input-output sentence pairs (e.g. 'walk @xmath0; input sentences swim clap move more primitives scan train set neural sequence models jump opposite right rturn rturn jump jump thrice jump jump jump jump left lturn jump jump opposite right rturn rturn jump jump thrice jump jump jump jump left lturn jump scan generalization set failure success move move left move twice move move lturn move move move move left move twice move move lturn move move move move left move twice move move lturn move move clap clap right clap thrice clap clap clap rturn clap clap clap clap right clap thrice clap clap clap rturn clap clap clap clap right clap thrice clap clap clap rturn clap clap swim swim left swim thrice swim swim swim lturn swim swim swim swim left swim thrice swim swim swim lturn swim swim swim swim left swim thrice swim swim swim lturn swim swim same-sized train set with more primitives and their use cases jump jump walk walk opposite left lturn lturn walk walk look look thrice look look look look run run twice run run run neural sequence models isolated primitive exactly three example primitives isolated primitive a lot more than three example primitives jump jump figure 1: overview of the scan generalization task (left) and our approach (right) that enables standard neural sequence models to generalize compositionally. a model with the right inductive bias should generalize on the test data after having seen compositional expressions with other primitives during training. the need for such inductive bias is justified via psychological experiments (lake et al., 2019) indicating that humans do have the ability to generalize on such tasks. previous works have suggested that seq-to-seq models lack the appropriate inductive bias necessary to generalize on this task since they achieve near-zero accuracies on both scan and colors benchmarks. this has led to the development of many specialized architectures (li et al., 2019; gordon et al., 2020; chen et al., 2020; akyurek and andreas, 2021), learning procedures (lake, 2019; conklin et al., 2021) and data augmentation methods (andreas, 2020; guo et al., 2020) to solve the task. contributions. the primary claim of our paper is that, contrary to prior belief, neural sequence models such as transformers and rnns do have an inductive bias foot_1 to generalize compositionally which can be enabled using the right supervision. (i) we show that by making simple and intuitive changes to the training data distribution, standard seq-to-seq models can achieve high generalization performance even with a training set of size less than 20% of the original training set. in particular, if we incorporated examples with more novel primitives in the training set without necessarily increasing the size of the training set (see right part of fig. 1 ), then the generalization performance of standard seq-to-seq models improves and reaches near-perfect score after a certain point. our results also exemplify the importance of the training distribution apart from architectural changes and demonstrate that providing the right supervision can significantly improve the generalization abilities of the models. (ii) we investigate the potential cause behind the improvement in generalization performance and observe that the embedding of the isolated primitive becomes more similar to other primitives when the training set has higher number of primitives and their use cases. (iii) to understand the phenomenon better, we characterize the effect of different training distributions and model capacities. our results show that the parameters of the experimental setting play a crucial role while evaluating the generalization abilities of models. ## enabling generalization by providing the right supervision setup. we focus on the scan and colors datasets. @xcite both these datasets have exactly one isolated primitive. we refer to all other primitives (i.e., those that are also composed with other words to form sentences in the training set) as example primitives. both the scan and colors training sets have exactly three example primitives. the training set of scan has 13.2k examples while the test set has 7.7k examples. colors has just 14 training examples and 8 test examples. more details on implementation and datasets can be found in appendix a & b. our source code is available at https://github.com/arkilpatel/compositionalgeneralization-seq2seq . ## conclusion while it is essential to make progress in building architectures with better compositional generalization abilities, we showed that the generalization performance of standard seq-to-seq models (often used as baselines) is underestimated. a broader implication of our experiments is that although systematicity must be preserved when designing such benchmarks, it is imperative to carefully explore different parameters associated with the experimental setup to draw robust conclusions about a model's generalization abilities.