Solving SCAN Tasks with Data Augmentation and Input Embeddings (2021)

Abstract: We address the compositionality challenge presented by the SCAN benchmark. Using data augmentation and a modification of the standard seq2seq architecture with attention, we achieve SOTA results on all the relevant tasks from the benchmark, showing the models can generalize to words used in unseen contexts. We propose an extension of the benchmark by a harder task, which cannot be solved by the proposed method.

## introduction compositionality describes the property of language that the syntactic and semantic aspects of complex language units are composed of the syntactic and semantic aspects of primitive units @xcite . the scan benchmark @xcite has been designed to assess the ability of current neural networks to utilize compositionality of language to deal with systematic difference between the training and test data distributions (see we use a novel combination of existing ideas (data augmentation, adding noise and predicting outputs based only on the weighted average of input embeddings) to achieve high accuracy in all the relevant tasks of the scan benchmark. an approach working well for all the tasks has not been reported yet. 1 seeing the good results, we analyze some underlying assumptions of the tasks and propose a new split of the scan data (i.e. a new task) that proves to be more difficult for our approach. in section 2, we describe the scan benchmark for testing the systematic use of compositionality in detail. then we present some approaches tested on the benchmark so far (section 3), introduce our own method of solving the tasks (section 4) and its results and analysis of some of the decisions (section 5). in section 6, we discuss the existing tasks and propose a new one. we conclude the paper in section 7. the scan dataset (lake and baroni, 2018) was designed to test this specific aspect of learning. it represents the problem of sequence transduction (sequence to sequence transformation). an agent in a grid world environment is supposed to perform a sequence of primitive actions (output) based on a sequence of commands (input). however, neither the agent nor the environment play any role in the generation of the next examples (there is no 'state' as known from reinforcement learning tasks) or in the interpretation of the commands. scan is designed to test traditional supervised learning models, namely those that translate from the "command language" into the "action language". some examples taken from the dataset are given in commands actions (look twice) and (turn right twice) (look look) (right right) (run thrice) after (look left) (left look) (run run run) (walk around right) (right walk right walk right walk right walk) jump jump (walk opposite left) after (run) (run) (left left walk) . . . . . . (walk opposite left) after (jump) (jump) (left left walk) jump twice jump jump (jump thrice) and (walk) (jump jump jump) (walk) . . . . . . left, right). for more detail, see the supplementary in @xcite . as far as the training and test data are produced randomly, standard seq2seq models achieve high accuracy of 99.8% @xcite . (the accuracy is computed over whole sequences, i.e. the models make errors in about 0.2% of the testing sequences). once the training and test data reflect a systematic distribution change, the models struggle. in this work, we address two main sets of tasks presented by scan: generalization to a new primitive (jump) and generalization to a new combination of learned concepts (around right). we discuss them in the following subsections. ## model our architecture is a straightforward extension of the baseline lstm seq2seq model with attention @xcite . we modify the attention mechanism by using input word embeddings as the attention values (as opposed to using contextualized representations produced by the encoder). this can also be seen as a simplified version of the architectures introduced by @xcite ; @xcite . unlike them, we train only one set of embeddings that are used both as the encoder input and as attention values. also, we use the traditional autoregressive decoder (the previous predicted output serves as the input for the decoder at each step), which is not the case in the two cited approaches. we added standard gaussian noise to the input embeddings during training, as suggested by @xcite , but did not use the l2 regularization. we also experimented with weight tying, known for example, from some language modelling literature @xcite . we tried to avoid using a separate output layer and instead, we computed similarity (dot product) of the attention output with the output token embeddings (the embeddings used by the decoder) and used these vectors as logits. similarly to @xcite , we experiment with adding artificial primitives to the training data. we hope this would make the encoder more robust as it should recognize the syntactic patterns (x opposite right) rather than memorize each instance (walk opposite right) using the available embedding (walk). for the jump task, we introduce artificial action commands action1, action2, action3, . . . with the corresponding actions action1, action2, ac-tion3, . . . . for all the other tasks (lturn, jump around right, primitive right, primitive opposite right, primitive around right), we introduce artificial directions: @xmath0, @xmath1, @xmath2. . . . unlike lake (2019), we are not permuting the command -action assignment during training and we do not introduce the heldout phrases to either the input or the output of the training examples. ## results and analysis we provide the test accuracy of our model on the two most widely discussed scan tasks (jump and around right) in ## discussion as seen above ( we hypothesize that current methods (including ours) manage to solve this compositionality challenge without using much compositionality. specifically, what is enough for all the tasks as they have been defined is to learn to attend to correct positions and then use a one to one mapping from the input to the output space. this "align and translate" @xcite approach exhibits some degree of compositionality. however, having learned somewhat complex information (say in the case of grammatical words such as around or opposite), this needs to be combined with just the identity of another word. this was actually the motivation of @xcite to introduce new tasks into the benchmark (e.g. around right, see section 2.2). we believe that in this respect, the new tasks suffer from the same weaknesses as the old ones (as evidenced by the possibility of easily using data augmentation). ## conclusion the scan benchmark is a very accessible dataset used for investigating compositionality. cherrypicking from previous research, we introduced the first approach that is successful in all the relevant tasks from the benchmark. analyzing the important features defining our approach, we introduced a new task, where this approach fails. we hypothesize the new task is more challenging than the existing ones since it tests for deeper aspects of compositionality.