The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers (2021)

Abstract: Recently, many datasets have been proposed to test the systematic generalization ability of neural networks. The companion baseline Transformers, typically trained with default hyper-parameters from standard tasks, are shown to fail dramatically. Here we demonstrate that by revisiting model configurations as basic as scaling of embeddings, early stopping, relative positional embedding, and Universal Transformer variants, we can drastically improve the performance of Transformers on systematic generalization. We report improvements on five popular datasets: SCAN, CFQ, PCFG, COGS, and Mathematics dataset. Our models improve accuracy from 50% to 85% on the PCFG productivity split, and from 35% to 81% on COGS. On SCAN, relative positional embedding largely mitigates the EOS decision problem (Newman et al., 2020), yielding 100% accuracy on the length split with a cutoff at 26. Importantly, performance differences between these models are typically invisible on the IID data split. This calls for proper generalization validation sets for developing neural networks that generalize systematically. We publicly release the code to reproduce our results.

## introduction systematic generalization @xcite ) is a desired property for neural networks to extrapolate compositional rules seen during training beyond training distribution: for example, performing different combinations of known rules or applying them to longer problems. despite the progress of artificial neural networks in recent years, the problem of systematic generalization still remains unsolved @xcite @xcite @xcite . while there has been much progress in the past years @xcite @xcite @xcite , in particular on the popular scan dataset (lake and @xcite where some methods even achieve 100% accuracy by introducing some non-trivial symbolic components into the system @xcite , the flexibility of such solutions is questionable. in fact, the existing scan-inspired solutions have limited performance gains on other datasets @xcite . it is thus not enough to solely focus on the scan dataset to progress research on systematic generalization. recently, many datasets have been proposed for testing systematic generalization, including pcfg @xcite and cogs @xcite . the baseline transformer models which are released together with the dataset are typically shown to dramatically fail at the task. however, the configurations of these baseline models are questionable. in most cases, some standard practices from machine translation are applied without modification. also, some existing techniques such as relative positional embedding @xcite , which are relevant for the problem, are not part of the baseline. in order to develop and evaluate methods to improve systematic generalization, it is necessary to have not only good datasets but also strong baselines to correctly evaluate the limits of existing architectures and to avoid false sense of progress over bad baselines. in this work, we demonstrate that the capability of transformers @xcite and in particular its universal variants @xcite on these tasks are largely underestimated. we show that careful designs of model and training configurations are particularly important for these reasoning tasks testing systematic generalization. by revisiting configurations such as basic scaling of word and positional embeddings, early stopping strategy, and relative positional em-bedding, we dramatically improve the performance of the baseline transformers. we conduct experiments on five datasets: scan (lake and baroni, 2018), cfq @xcite , pcfg @xcite , cogs @xcite , and mathematic dataset @xcite . in particular, our new models improve the accuracy on the pcfg productivity split from 50% to 85%, on the systematicity split from 72% to 96%, and on cogs from 35% to 81% over the existing baselines. on the scan dataset, we show that our models with relative positional embedding largely mitigates the so-called end-of-sentence (eos) decision problem @xcite , achieving 100% accuracy on the length split with a cutoff at 26. also importantly, we show that despite these dramatic performance gaps, all these models perform equally well on iid validation datasets. the consequence of this observation is the need for proper generalization validation sets for developing neural networks for systematic generalization. we thoroughly discuss guidelines that empirically yield good performance across various datasets, and we will publicly release the code to make our results reproducible. ## datasets and model architectures for systematic generalization here we describe the five datasets, and specify the transformer model variants we use in our experiments. the selected datasets include both already popular ones and recently proposed ones. statistics of the datasets can be found in ## improving transformers on systematic generalization in this section, we present methods which greatly improve transformers on systematic generalization tasks, while they could be considered as details in standard tasks. for each method, we provide experimental evidences on a few representative datasets. in section 4, we apply these findings to all datasets. ## results across different datasets in this section, we apply the methods we illustrated in the previous section across different datasets. ## conclusion in this work we showed that the performance of transformer architectures on many recently proposed datasets for systematic generalization can be greatly improved by revisiting basic model and training configurations. model variants with relative positional embedding often outperform the ones with absolute positional embedding. they also mitigate the eos decision problem, an important problem previously found by @xcite when considering the length generalization of neural networks. this allows us to focus on the problem of compositions in the future, which is the remaining problem for the length generalization. we also demonstrated that reconsidering early stopping and embedding scaling can greatly improve baseline transformers, in particular on the cogs and pcfg datasets. these results shed light on the discrepancy between the model performance on the iid validation set and the test accuracy on the systematically different generalization split. as consequence, currently common practice of validating models on the iid dataset is problematic. we conclude that the community should discuss proper ways to develop models for systematic generalization. in particular, we hope that our work clearly demonstrated the necessity of a validation set for systematic generalization in order to establish strong baselines and to avoid a false sense of progress.