Can Transformers Jump Around Right in Natural Language? Assessing Performance Transfer from SCAN (2021)

Abstract: Despite their failure to solve the compositional SCAN dataset, seq2seq architectures still achieve astonishing success on more practical tasks. This observation pushes us to question the usefulness of SCAN-style compositional generalization in realistic NLP tasks. In this work, we study the benefit that such compositionality brings about to several machine translation tasks. We present several focused modifications of Transformer that greatly improve generalization capabilities on SCAN and select one that remains on par with a vanilla Transformer on a standard machine translation (MT) task. Next, we study its performance in low-resource settings and on a newly introduced distribution-shifted English-French translation task. Overall, we find that improvements of a SCAN-capable model do not directly transfer to the resource-rich MT setup. In contrast, in the low-resource setup, general modifications lead to an improvement of up to 13.1% BLEU score w.r.t. a vanilla Transformer. Similarly, an improvement of 14% in an accuracy-based metric is achieved in the introduced compositional English-French translation task. This provides experimental evidence that the compositional generalization assessed in SCAN is particularly useful in resource-starved and domain-shifted scenarios.

## introduction while sequence-to-sequence (seq2seq) models achieve remarkable performance in many tasks @xcite @xcite , they often fail to generalize in a systematic way @xcite @xcite @xcite . these shortcomings are particularly obvious in the experiments on the scan domain @xcite @xcite . in scan, inputs are instructions that describe trajectories and outputs define sequences of actions to follow them (see this hypothesis stumbles into a perplexing situation: despite failing at compositional generalization, considered as a core requirement for language understanding, seq2seq models have tremendous success in practice. is the type of compositional generalization, that scan probes for, useful for nlp tasks? if so, in what scenarios? in this work, we aim to answer this question. firstly, we introduce focused modifications to transformer that greatly improve accuracy performance on scan. to build such modifications, we exploit two observations: (i) transformer's architecture is very similar to convolution-based seq2seq models (convs2s) @xcite , (ii) convs2s performs well on scan @xcite . this capability (ii) is hypothesized to be due to explicitly localized representation of the sequences, where only deep layers of the model can access more distant tokens @xcite . such a capability should also benefit to natural language processing as human languages are proved to favor local syntactic constructions @xcite . motivated by these observations, we focus on the major differ-ences between convs2s and transformer, namely the localized receptive field span and the gating mechanism, to inject into transformer inductive biases useful for scan. as a testbed, we use the machine translation (mt) domain as one of the most popular applications for seq2seq models. we consider both resource-rich (iwslt'14 @xmath0. finally, to evaluate scan-style capabilities in natural language tasks, we build a dataset that probes whether models can systematically generalize w.r.t. noun-adjective ordering while translating from english to french. we construct this dataset by filtering the europarl corpus (a part of wmt'14). our results indicate that combining two convs2s-inspired changes improves accuracy on one scan split (scan-jump) from 3.4% to 43.0%, while maintaining a high accuracy on the other splits (scan-simple and scan-aroundright). as expected, given that scan is an artificial diagnostic dataset, not all modifications lead to equal improvements on an mt task. we select one of the considered modifications that performs on par with the vanilla transformer on iwslt'14. testing the selected modification on lowresource data, we observe that it provides between 3.6% and 13.1% bleu improvements over transformer. on the noun-adjective ordering dataset, we find that our modification results into gains in generalization of 14%. this leads to the following picture: the localized attention, augmented by a gating mechanism, provides a useful inductive bias that proves to be beneficial for scan-style generalization. additionally, it turns out useful in low-resource and distributionshifted settings. thus, testing seq2seq models on scan while controlling for a non-degraded performance leads to improvement in domains where syntactic compositionality is crucial for a task success. ## transformers and convs2s architecture overview both transformer and convs2s are encoder-decoder architectures @xcite , where the decoder has an attention mechanism to peek into the encoder's representation @xcite . these representations are obtained by embedding the inputs, adding a positional embedding, and passing them through a sequence of layers. in transformer's encoder, the output representations are the result of a sequential application of two (sub)layer types: self-attention and fully connected layers. the input representation can "skip" any sublayer via a residual connection. the output of the sublayer is passed through a dropout mechanism and added to the residual. this sum is then layer-normalized. any relation between input tokens is modeled solely by self-attention modules. in convs2s, the encoder is also a sequence of identical blocks. the inter-term dependencies are modeled by 1d convolutions with glu activation functions @xcite . 1 in contrast to self-attention, convolutions have a finite kernel size, thus effectively capping the maximal distance of intermediate dependencies that can be modeled. the glu activation function serves as a gate, allowing convs2s to control the balance between residuals and the output of the convolution. after the glu operation, the intermediate representation is added to the residual and scaled. the output of the final convolution is then passed into a fully connected layer. in convs2s and transformer, decoders have similar structures to those of encoders, with an additional @xmath1, respectively. despite the similarities, there are numerous low-level differences between the two architectures: normalization (layer norm @xcite vs. weight normalization @xcite ), optimization (adam (kingma and ba, 2014) with a ramp-up vs. nag @xcite ), etc. a priori, any of those can affect models' inductive biases. however, we concentrate on some of the most obvious architectural differences: the limited convolution span and glu activations. we believe these features can greatly affect models' performance on scan. indeed, scan has only local dependencies between tokens, thus the ability to avoid spurious correlations with more distant tokens can be useful. similarly, the ability to weight contributions from the token interactions into the intermediate representation is intuitively prerequisite to build compositional representation. glu @xcite given a vector input x, glu splits it in two equally sized halves x 1 and x 2 ; one is passed through a sigmoid (σ(x) = (1 + e -x ) -1 ). then both parts are pointwise multiplied: jump ⇒ jump jump around right ⇒ rturn jump rturn jump rturn jump rturn jump turn left twice ⇒ lturn lturn jump opposite left after walk around left ⇒ lturn walk lturn walk lturn walk lturn walk lturn lturn jump this allows a network to implement a gating mechanism, where one half of its output gates the signal from the second. self-attention instead of convolutions, transformer uses multi-headed self-attention to model interactions between tokens. given n embeddings, x 1 , x 2 , ...x n of dimensionality d, the self-attention layer transforms them in the following way. firstly, each embedding x i is projected by three matrices q, k, and v to get query q i , key k i , and value v i representations, respectively: next, a scaled dot-product between query q i and key k j is calculated as follow: this dot-product defines the attention weights w @xmath2: o @xmath3. this process is done in parallel for multiple heads, acting on independent slices of the input embeddings; their outputs are concatenated and passed through a fully connected layer. ## transformer modifications self-attention gate (sag) the simplest way to imitate the effect of glu activation (eq. 1) is to weight (gate) the output of self-attention by a learned scalar parameter. to ensure that it is nonnegative and is scaled in [0, 1], we parameterize it as a sigmoid of a real-valued learned parameter β. algorithm 1 illustrates the introduced change. in comparison to transformer, sag adds one scalar parameter for each encoder and decoder layer. we treat β 0 , the value β is initialized with before training, as a hyperparameter. in the preliminary experiments, we found that after training, encoder layers often have small negative β values (-2.. -0.5), while decoder layers have positive values (0.2..4.5) that grow monotonically for higher layers. a similar modification was considered in an effort to stabilize transformer training in the reinforcement learning domain @xcite . convolution as self-attention in the limit case, we can entirely replace self-attention with convolutions. this modification introduces one hyperparameter (kernel size). however, convolutional layers have fewer parameters than the self-attention mechanism. one might consider this not to be a transformer variant due to the lack of self-attention, but as self-attention generalizes convolutions (cordonnier et al., 2020), we consider this as an extreme form of regularization. fixed-span self-attention a less extreme modification would be to use the regular multi-head self-attention mechanism, but without allowing attention to peek beyond some distance. this mimics the limited kernel size of convolutions in convs2s. we achieve this by adding a fixed bias term b ij to the self-attention logits (eq. 2): setting b ij to -∞ when the difference |i -j| exceeds some fixed value s and to 0 otherwise prevents the self-attention to look beyond distance s. fixed-span self-attention with a span parameter s has the same "receptive field" as 1d convolution with kernel size 2s + 1. this modification adds one hyperparameter (span size), but does not introduce new learned parameters. t5 attention further relaxing constraints on self-attention, we consider the case where we allow transformer to learn how to (soft-)limit its self-attention. we introduce the bias term b ij that is learned as a function of a (signed) difference i -j, capping it to [-s, +s] (e.g., positions with difference above s would have the same bias b s ). this modification is similar to one introduced by @xcite in t5, with the only exception that we allow each head to have its own bias. again, the span size is a new hyperparameter. in a model with n h heads and n l layers, this modification requires (2s + 1) × n l × n h new parameters, which is negligible in comparison with the sizes of fully connected layers. examples of the learned b ij parameters are shown in supplementary when training on scan. algorithm 1: self-attention gate (sag). the only introduced change is on line 4. β is a learned per-layer scalar parameter. implementation details we used the fairseq @xcite implementation of transformer seq2seq as a foundation, with its initialization and default parameters. t5 and fixed-span attentions are implemented by providing additive attention masks to transformer. ## scan experiments in our preliminary experiments, we found that our modifications of the self-attention mechanism do not lead to improvements over the standard transformer when they are not combined with the selfattention gate (sag). hence, we focus our experiments on architectures that include sag. we report our results in transformer sag + conv s.-a. + fix. span + t5 34.64 ±0.03 34.28 ±0.08 33.44 ±0.04 34.32 ±0.01 34.66 ±0.04 adding sag increases accuracy on jump 5fold (17.2%) at the expense of a small drop in around-right scores (not stat. sig.). further, we observe that changes of the selfattention mechanism (replacing it with convs, limiting its span, and adding a relative positiondependent bias), can further increase the performance on jump. apart from sag+conv as s.-a, the self-attention modifications do not significantly alter the performance on around-right. we see that the architectural changes that we proposed improve the compositional capabilities of the transformer models. as expected, the introduced hybrid architectures reach significantly better performance on jump (up to 12x improvements for sag+t5) while keeping high performance on the around-right & simple tasks. ## machine translation experiments iwslt'14 de-en in flores, si/ne ↔ en we report results on english↔nepali and english↔sinhala translation in noun-adjective ordering bleu scores on the test set are reported in discussion upon analyzing our experiments on scan and machine translation tasks, we see the following picture. indeed the hybrid models that we described in section 2 have considerably higher accuracy on scan-jump w.r.t. transformer and a comparable performance on the other scan splits. hence, our results suggest the importance of both gating and (the ability of) limiting the attention span for scan generalization. as expected, the improvement on scan do not consistently entail improvements on the resourcerich dataset, and only the combination of sag and t5 showed a tiny improvement. this emphasizes the importance of testing models on realistic setups to model from being too scan-tailored. finally, we test sag + t5 on low-resource and compositional tasks. the hybrid architecture shows consistent improvements on flores for all translation directions, with at up to 13.1% relative improvement, and on the the natural language compositional task with 14% relative improvement on comp. our qualitative analysis also showed that sag + t5 correctly handles noun-adjective order-8 related work compositionally-biased models several approaches were proposed to build scan-capable architectures. they span from meta-learning @xcite , disentangling syntax and semantics @xcite , learning equivariant @xcite and disentangled representations @xcite or combining neural & symbolic computations @xcite . in contrast, we do not build new models that are specialized to scan. instead, we show that a standard model can be incrementally modified so that performs well on scan and still performs well on a standard mt task. having such incrementally improved models allows us to step back and wonder if scan (or similar artificial tasks) should be used as a guidance when developing new models. @xcite raised concerns due to scan being too artificial by showing that even degenerate architectures can perform well on some scan tasks. our results echo their findings: by developing architectures tailored for scan, one can easily come up with models that perform worse on general tasks. however, we find that if one avoids this "scan overfitting" and endows a model with capabilities that scan probes for without harming its general performance, they can gain in lowresource scenarios and better handle relevant phenomena in language. changing attention mechanisms self-and cross-attention mechanisms were tweaked in earlier work in order to inject useful biases, e.g., by adding information of relative positions of tokens @xcite or accounting for the locality bias in cross-attention @xcite . @xcite demonstrated that having a short attention span on the lower layers of transformer models is enough for good language modeling performance. ## conclusion in this work, we primarily focused on whether and in which scenarios the inductive bias for compositional generalization, that scan looks for, can be useful in natural language tasks. we ran study in two steps. as the first step, by exploiting convs2s/transformer similarities, we came up with a modification of the transformer next, we tested this modification in low-resource and distribution-shifted setups. in the low-resource mt setup (flores si/ne↔en), we found that our considered architecture improves by up to 13.1% in bleu score over the vanilla transformer. then, we introduced a new dataset that probes specifically for compositional reasoning in natural language. unlike scan, our compositional dataset is built by filtering an existing natural language corpus (europarl en-fr) to probe how models perform noun-adjective ordering under a (minimal) distribution shift. thus, we are largely closer to testing the compositional generalization required by humans compared to scan, and succeeding on the test set requires both compositional reasoning and good language model performance (see examples in our findings indicate the following. firstly, as hypothesized before @xcite , the limited attention span provides a useful inductive bias that allows models to perform better on compositional generalization induction, that scan probes for. further, endowing a model with scan-style generalization capabilities can lead to improvements in low-resource and distribution-shifted scenarios as long as we ensure that we do not overfit to scan. we believe that the contribution of diagnostic datasets like scan is of great value. as performance grows on tasks such as mt, identifying gaps where a model's performance lags will become fundamental and will guide us to develop architectures that cover genuine new linguistic grounds and not just overfit to peculiarities of standard datasets.