Inducing Transformer’s Compositional Generalization Ability via Auxiliary Sequence Prediction Tasks (2021)

Abstract: Systematic compositionality is an essential mechanism in human language, allowing the recombination of known parts to create novel expressions. However, existing neural models have been shown to lack this basic ability in learning symbolic structures. Motivated by the failure of a Transformer model on the SCAN compositionality challenge (Lake and Baroni, 2018), which requires parsing a command into actions, we propose two auxiliary sequence prediction tasks as additional training supervision. These automatically-generated sequences are more representative of the underlying compositional symbolic structures of the input data. During inference, the model jointly predicts the next action and the next tokens in the auxiliary sequences at each step. Experiments on the SCAN dataset show that our method encourages the Transformer to understand compositional structures of the command, improving its accuracy on multiple challenging splits from ≤ 10% to 100%. With only 418 (5%) training instances, our approach still achieves 97.8% accuracy on the MCD1 split. Therefore, we argue that compositionality can be induced in Transformers given minimal but proper guidance. We also show that a better result is achieved using less contextualized vectors as the attention’s query, providing insights into architecture choices in achieving systematic compositionality. Finally, we show positive generalization results on the grounded-SCAN task (Ruis et al., 2020).

## introduction human intelligence, including natural languages, demonstrates systematic compositionality, the algebraic capacity to understand and produce a potentially infinite number of novel combinations of known components @xcite . for example, we know the usage of words "walk," "twice," "and"; once we learn a new verb "dax", we can immediately understand or produce utterances like "dax twice and walk." this type of compositionality is central to the human ability of making strong generalizations from limited data @xcite . however, there have been arguments that neural networks are associative devices that cannot capture systematic compositionality @xcite @xcite @xcite . supporting this view, it has been shown that general neural models, like rnns and transformers @xcite , generalize poorly to the development set's unseen combination of components seen in training set @xcite . however, recent works have equipped recurrent neural networks (rnns) with separate primitive and functional embeddings of the input tokens @xcite . on the scan dataset @xcite ) that requires parsing a command into actions, these models can effectively parse "jump thrice" when only trained on how to "walk thrice", "walk", and "jump". in this work, we first show that this dual embedding method from cgps-rnn @xcite can be transferred to the transformer architecture to achieve nearly perfect results in substituting new primitives. our cgps-transformer encoder maintains a functional/syntactic embedding and a primitive/semantic embedding for every word in the vocabulary. this separation of syntax and semantics is crucial in achieving compositionality: during the training, the model successfully learns the syntactic similarity between "jump" and other primitives through training examples like "jump-→jump" and "walk-→walk". the semantic difference between primitives (e.g., "jump" should be translated into "jump" rather than "walk") is encoded into the semantic embeddings, which do not participate in on the three mcd splits, the cgps-transformer that predicts the auxiliary sequences achieves a perfect test-set performance of 100% accuracy, as compared to 7.66%, 3.25%, and 6.12% from the same model but without the auxiliary sequence prediction tasks. our approach also generalizes to longer action sequences with 100% accuracy as compared to 0% from the baseline without the auxiliary sequences. between the two previous works that achieved significant progress on the mcd splits, lane @xcite explicitly models the procedure of recognizing a symbolic function (e.g., "x twice") and applying the function via two separate models. these two models each make their discrete predictions step-by-step and are jointly trained with hierarchical reinforcement learning. @xcite proposed to use monolingual dev/test data for semi-supervised learning and their results are worse than ours. our approach differs from these two works in that it builds upon the general seq2seq architecture and does not require a peek into novel dev-set commands. we further demonstrate that the cgps-transformer model only needs a small number of supervision with auxiliary sequences to develop the compositionality, as it achieves 97.8% accuracy on mcd1 split with 418 (5% of all) training examples. this suggests that systematic compositionality does not require a ton of training examples. instead, a small number of well-designed demonstrations that exhibit the compositional structures of the data can better induce a generalizable model. our ablation study shows that both the auxiliary tasks are necessary in promoting the compositional generalization behavior of the model. we further conduct experiments to show that it is important to use the output/intermediate vectors of the decoder's first layer as the queries in the task-specific attention for predicting the first auxiliary sequence (e.g., [2, 2, 1, 1, 0, 0] for "jump left thrice"). if we instead use the decoder's highly contextualized final outputs as the queries, the model would fail to predict the correct auxiliary sequence and the target actions. we make three arguments from this ablation study: finally, we also show that our auxiliary sequence prediction method can be transferred to grounded-scan @xcite , a newer multi-modal compositional challenge that requires new recombinations of seen phrases. overall, we hope our method and findings can provide an insightful view of the compositional generalization in deep neural models and inspire future works in this direction. ## method first, we briefly introduce the baseline model where we apply our method (sec. 3.1). we then motivate and introduce the auxiliary sequences we create to improve the model's compositional generalization ability (sec. 3.2). finally, we explain how a seq2seq model can jointly predicts its target sequence and the auxiliary sequences in the training and inference (sec. 3.3). ## discussion in this section, we discuss what our experiments reveal regarding the compositionality of transform-ers as well as the limitation of our method in terms of its applicability to other datasets. first, we develop our method for scan @xcite , which is a synthetic dataset and its language commands are produced from a limited set of rules. thus, it is unclear how the findings on the simplified scan setting can be transferred to large-scale, natural datasets. however, the community believes that scan is a valuable benchmark and useful analysis tool for studying language compositionality because, first, its inputs are still realistic english, i.e., they use the same set of functional words that people use in natural language ("and", "after", etc.) and each of these words has an symbolic function that influences the structure of the output sequence. second, it has been shown that even large pre-trained language models cannot achieve strong performance on scan, indicating that exposure to more texts and linguistic structures do not naturally induce compositionality in neural models. therefore, our method's effectiveness and simplicity should still provide some key insights into the nature of the neural model's acquisition of compositionality. second, it is worth noting that the synthetic language input in scan can be written as a contextfree grammar; as a result, we can design an automatic procedure to generate both auxiliary sequences based on the underlying grammar. applying this method to a dataset with natural language requires designing a heuristic to approximate the underlying grammar. however, as the community is still trying to establish a basic understanding of whether/how a neural network can recognize the compositionality in language, an important first step could be done under a simpler setting (e.g., scan) with a controlled grammar. furthermore, predicting the auxiliary sequences, although seemingly simple, is not a trivial task and is highly correlated with the compositional structure of symbolic functions. the fact that transformer can predict the two auxiliary sequences perfectly suggests that it can model the compositional structure without extra information at test time if given the proper training supervision. therefore, we believe that some of our observations are promising and exciting to the community. last but not least, our method achieves strong few-shot generalization (97.8% on mcd1 with only 418 training instances) and perfect length generalization. this opens up the possibility of using a small number of human-annotated auxiliary sequences to improve the models' performance on large-scale, natural datasets where automatically generating auxiliary sequences is infeasible. ## conclusion in this work, we propose two auxiliary sequence prediction tasks to induce the compositional generalization ability in a transformer model. on the challenging length and mcd splits of the scan dataset, our method achieves the perfect 100% accuracy, a huge improvement from @xmath0. we further show that our method works well in low-resource settings as it reaches 97.8% accuracy with only needs 418 training examples. ablation analysis shows that the model achieves better compositionality using the decoder's less contextualized vectors to compute the next token in auxiliary sequences.