Making Transformers Solve Compositional Tasks (2022)

Abstract: Several studies have reported the inability of Transformer models to generalize compositionally, a key type of generalization in many NLP tasks such as semantic parsing. In this paper we explore the design space of Transformer models showing that the inductive biases given to the model by several design decisions significantly impact compositional generalization. We identified Transformer configurations that generalize compositionally significantly better than previously reported in the literature in many compositional tasks. We achieve state-of-the-art results in a semantic parsing compositional generalization benchmark (COGS), and a string edit operation composition benchmark (PCFG).

## introduction although modern neural network architectures reach state-of-the-art performance in many challenging natural language tasks, they seem to exhibit a low amount of "compositional generalization", i.e., the ability to learn a set of basic primitives and combine them in more complex ways than those seen during training @xcite . for example, suppose a system has learned the meaning of "jump" and that "jump twice" means that the action "jump" has to be repeated two times. upon learning the meaning of the action "jax", it should be able to infer what "jax twice" means. compositional generalization is a key aspect of natural language and many other tasks we might want machine learning models to learn. while both humans and classical ai techniques (such as grammars or search-based systems) can handle compositional tasks with relative ease, it seems that modern deep learning techniques do not possess this ability. a key question is thus: can we build deep learning architectures that can also solve compositional tasks? in this paper we focus on transformers @xcite , which have been shown in the literature to exhibit poor compositional generalization (see section 2). through an empirical study, we show that this can be improved. with the goal of creating general models that generalize compositionally in a large range of tasks, we show that several design decisions, such as position encodings, decoder type, weight sharing, model hyper-parameters, and formulation of the target task result in different inductive biases, with significant impact for compositional generalization foot_0 . we use a collection of twelve datasets designed to measure compositional generalization. in addition to six standard datasets commonly used in the literature (such as scan @xcite , pcfg @xcite , cfq @xcite and cogs @xcite )), we also use a set of basic algorithmic tasks (such as addition, duplication, or set intersection) that although not directly involving natural language, are useful to obtain insights into what can and cannot be learned with different transformer models. we also include tasks where we do not see significant improvements, to understand what types of compositional generalization are improved with our proposed modifications, and which are not. the main contributions of this paper are: (1) a study of the transformer design space, showing which design choices result in compositional learning biases across a variety of tasks. (2) state-of-theart results in cogs, where we report a classification accuracy of 0.784 using an intermediate representation based on sequence tagging (compared to 0.35 for the best previously reported model @xcite ), and the productivity and systematicity splits of pcfg @xcite . the rest of this paper is organized as follows. section 2 provides some background on compositional generalization and transformers. in sec-tion 3, we present the datasets used in our empirical evaluation, which is presented in section 4. the paper closes with a discussion on the implications of our results, and directions for future work. ## background this section briefly provides background on compositional generalization and transformer models. ## evaluation datasets we use a collection of 12 datasets that require different types of compositional generalization. six of those dataset consist of "algorithmic" tasks (addition, reversing lists, etc.), and six of them are standard datasets used to evaluate compositional generalization (most involving natural language). we note that our algorithmic tasks mostly require productivity-style compositional generalization, while other datasets also require systematicity or synonimity @xcite . specifically, we used the following datasets (see appendix e for details, and figure 1 for examples): addition (add): a synthetic addition task, where the input contains the digits of two integers, and the output should be the digits of their sum. the training set contains numbers with up to 8 digits, and the test set contains numbers with 9 or 10 digits. numbers are padded to reach a length of 12. additionnegatives (addneg): the same as the previous one, but 25% of the numbers are negative (preceded with thesymbol). reversing (reverse): where the output is expected to be the input sequence in reverse order. training contains sequences of up to 16 digits, and the test set contains lengths between 17 to 24. duplication (dup): the input is a sequence of digits and the output should be the same sequence, repeated twice. training contains sequences up to 16 digits, and test from 17 to 24. cartesian (cart): the input contains two sequences of symbols, and the output should be their cartesian product. training contains sequences of up to 6 symbols (7 or 8 for testing). intersection (inters): given two sequences of symbols, the output should be whether they have a non-empty intersection. training contains sets with size 1 to 16, and testing 17 to 24. scan-length (scan-l): the length split of the scan dataset @xcite . scan-add-jump (scan-aj): the add primitive jump split of the scan dataset @xcite . pcfg-productivity (pcfg-p): the productivity split of the pcfg dataset @xcite pcfg-sytematicity (pcfg-s: the systematicity split of the pcfg dataset @xcite . cogs: the generalization split of the cogs semantic parsing dataset @xcite . cfq-mcd1 (cfq): the mcd1 split of the cfq dataset @xcite . enc2enc attention feed forward input embedding + add & norm absolute position encodings inputs dec2dec attention dec2enc attention add & norm x l output embedding outputs + feed forward add & norm add & norm linear + softmax add & norm output probabilities (without copy decoder) linear + relu linear d f d v k q + ✕ scale mask + relative position encodings softmax ✕ bias embedding + intermediate representation to final output x l multi-head attention x h feed forward encoder decoder 5 3 1 4 copy decoder attention softmax q c one-hot linear + sigmoid ✕ copy decoder 2 ✕ p 1 p 2 output probabilities (with copy decoder) 1-w w the training and test sets come from the same distribution, and most transformer models achieve near 100% accuracy (except a few hard tasks like the cartesian product or set intersection). hence, splitting train and test data in a way that requires compositional generalization is key (e.g., having examples with larger sequences in the test set than in the training set). we want to make sure models do not just learn shortcuts @xcite that do not generalize to out-of-distribution data. ## empirical results in this section we present an evaluation of the compositional generalization abilities of transformers with different architectural configurations. specifically we evaluated: (1) the type of position encodings, (2) the use of copy decoders, (3) model size, (4) weight sharing, and (5) the use of intermediate representations for prediction (see figure 2 ). for this systematic experimentation, we used small transformer models, without pre-training (all models are trained from scratch). even if previous work has reported benefits of pre-training in some compositional tasks (e.g., in cfq @xcite )), we aim at disentangling the effects of each architecture decision in and of itself, in the search for compositional inductive biases. our results show that, while these decisions do not affect certain types of compositional generalization tasks, we see significant gains in others. we report the average of at least 3 training runs (for algorithmic tasks, we use at least 5 training runs, and 10 for set intersection since they have a higher variance; see appendix b). we use sequence-level accuracy as the evaluation metric: an output sequence with even just a single wrong token is considered wrong. ## discussion an overall trend is that algorithmic tasks seem to be greatly affected by the different architecture design decisions we explored. in all datasets, except for cartesian product, there is at least one combination in our experiments that achieved high performance (close to 0.8 accuracy or higher). cartesian products remain an open challenge for future work, where one of the big obstacles is learning to produce much longer outputs than seen during training (output is quadratic with respect to input size). there are some datasets, such as scan-aj, where we did not see large improvements in performance. the main obstacle is learning to handle a symbol ("jump") having seen it very few times (or even just once) during training (this also happens in some types of generalization in cogs). none of the variations we experimented with were enough to handle this type of compositionality either. in conclusion, we observed: ## conclusions this paper presented an empirical study of the design space of transformer models, evaluated in a collection of benchmarks for compositional generalization in language and algorithmic tasks. our results show that, compared to a baseline transformer, significant gains in compositional generalization can be achieved. specifically, the baseline transformer achieved an average sequence-level accuracy of 0.137, while we showed this can increase to up to 0.527 with some design changes. accuracy levels of up to 0.493 can be achieved without increasing the parameter count of our baseline model (see appendix c for parameter counts). moreover, we achieved state-of-the-art results in cogs (at the time of submission), showing 0.784 accuracy on the generalization set, and two pcfg splits (0.634 and 0.828 respectively). this shows that a key factor in training models that generalize compositionally is to provide the right inductive biases. as part of our future work, we want to explore more dimensions, such as pre-training and optimizer parameters, and study the implications of our results in compositional generalization in large models on real world tasks.