Compositional Generalization by Factorizing Alignment and Translation (2020)

Abstract: Standard methods in deep learning for natural language processing fail to capture the compositional structure of human language that allows for systematic generalization outside of the training distribution. However, human learners readily generalize in this way, e.g. by applying known grammatical rules to novel words. Inspired by work in cognitive science suggesting a functional distinction between systems for syntactic and semantic processing, we implement a modification to an existing approach in neural machine translation, imposing an analogous separation between alignment and translation. The resulting architecture substantially outperforms standard recurrent networks on the SCAN dataset, a compositional generalization task, without any additional supervision. Our work suggests that learning to align and to translate in separate modules may be a useful heuristic for capturing compositional structure.

## introduction a crucial property underlying the expressive power of human language is its systematicity @xcite : syntactic or grammatical rules allow arbitrary elements to be combined in novel ways, making the number of sentences possible in a language to be exponential in the number of its basic elements. recent work has shown that standard deep learning methods in natural language processing fail to capture this important property: when tested on unseen combinations of known elements, standard models fail to generalize (lake and @xcite @xcite . it has been suggested that this failure represents a major deficiency of current deep learning models, especially when they are compared to human learners @xcite @xcite . from a statistical-learning perspective, this failure is quite natural. the neural networks trained on compositional generalization tasks fail to generalize because they have memorized biases that do indeed exist in the training set. these tasks require networks to make an out-of-domain (o.o.d.) extrapolation @xcite , rather than merely interpolate according to the assumption that training and testing data are independent and identically distributed (i.i.d.). to the extent that humans can perform well on certain kinds of o.o.d. tests, they must be utilizing inductive biases that are lacking in current deep learning models @xcite . it has long been suggested that the human capacity for systematic generalization is linked to mechanisms for processing syntax, and their functional separation from the meanings of individual words @xcite . in this work, we take inspiration from this idea and explore operationalizing it as an inductive bias in an existing neural network architecture. first, we notice a connection between syntactic structure and the correct alignment of words in the source sequence to meanings in the target. in our model, alignment is accomplished with an attention mechanism @xcite that determines the relevance of each word in the source to the translation of the next word in the target. this process must take into account the syntactic structure of both sequences (e.g. if a verb was just translated, it would be important to know whether there is in the source sequence an adverb that modifies it). we reasoned that if alignment was separated from direct translation (analogous to a separation of syntax and the meanings of individual words), the difficult o.o.d. problem of composing known elements into a novel combination would be reduced to two easier i.i.d. problems, because the distributions of correct alignments and translations would be similar in training and testing data (see figure 1 ). we implemented this intuition by modifying an existing attention mechanism @xcite , and call the resultant architecture syntactic attention to reflect the intuition that the attention mechanism used for alignment should operate primarily on syntactic information, which should be separated from the information relevant to translating individual words. we show that this modification achieves substantially improved compositional generalization performance over the original architecture on the scan dataset. ## syntactic attention the syntactic attention model improves the compositional generalization capability of an existing attention mechanism @xcite by separating two streams of information processing for alignment and translation (see figure 2 ). we describe the mechanisms of this separation and the other details of the model below. ## related work we were inspired by work in cognitive science emphasizing the relationship between systematicity and syntax @xcite . others have explored similar ideas in different natural language tasks @xcite @xcite @xcite . this work supports the suggestion that intuitions from cognitive science can aid architecture design in deep learning. ## conclusion in this work we attempt to operationalize an intuition from cognitive science, implementing it as inductive bias in the form of a factorization between alignment and translation in the seq2seq setting. we showed that this can improve compositional generalization performance on the scan task, and that it doesn't degrade performance on a small mt task. we believe this factorization prevents the model from memorizing spurious correlations in the data, and note that similar ideas may be useful in other natural language tasks. we would also like to thank all of the members of the computational cognitive neuroscience lab at uc davis for helpful ongoing discussion on these topics. this work was supported by onr n00014-19-1-2684 / n00014-18-1-2116, onr n00014-14-1-0670 / n00014-16-1-2128, and onr n00014-18-c-2067. kept as they were in the original dataset, but following @xcite , we used early stopping by validating on a 20% held out sample of the training set. all reported results are from runs of 200,000 iterations with a batch size of 1. except for the additional batch of 25 runs for the add-jump split, each architecture was trained 5 times with different random seeds for initialization, to measure variability in results. all experiments were implemented in pytorch. initial experimentation included different implementations of the assumption that syntactic information be separated from semantic information. after the architecture described in the main text showed promising results, a hyperparameter search was conducted to determine optimization (stochastic gradient descent vs. adam), rnn-type (gru vs. lstm), regularizers (dropout, weight decay), and number of layers (1 vs. 2 layers for encoder and decoder rnns). we found that the adam optimizer (kingma and ba, 2015) with a learning rate of 0.001, two layers in the encoder rnn and 1 layer in the decoder rnn, and dropout worked the best, so all further experiments used these specifications. then, a grid-search was conducted to find the number of hidden units and dropout rate. we tried hidden dimensions ranging from 50 to 400, and dropout rates ranging from 0.0 to 0.5. the best model used an lstm with 2 layers and 200 hidden units in the encoder, and an lstm with 1 layer and 400 hidden units in the decoder, and used 120-dimensional m j vectors, and a dropout rate of 0.5. the results for this model are reported in the main text. all additional experiments were done with models derived from this one, with the same hyperparameter settings. all evaluation runs are reported in the main text: for each evaluation except for the add-jump split, models were trained 5 times with different random seeds, and performance was measured with means and standard deviations of accuracy. for the add-jump split, we included 25 runs to get a more accurate assessment of performance. this revealed a strong skew in the distribution of results, so we included the median as the main measure of performance. occasionally, the model did not train at all due to an unknown error (possibly very poor random initialization, high learning rate or numerical error). for this reason, we excluded runs in which training accuracy did not get above 10%. no other runs were excluded.