Systematic Generalization on g SCAN : W hat is Nearly Solved and What is Next? (2021)

Abstract: We analyze the grounded SCAN (gSCAN) benchmark, which was recently proposed to study systematic generalization for grounded language understanding. First, we study which aspects of the original benchmark can be solved by commonly used methods in multi-modal research. We find that a general-purpose Transformer-based model with cross-modal attention achieves strong performance on a majority of the gSCAN splits, surprisingly outperforming more specialized approaches from prior work. Furthermore, our analysis suggests that many of the remaining errors reveal the same fundamental challenge in systematic generalization of linguistic constructs regardless of visual context. Second, inspired by this finding, we propose challenging new tasks for gSCAN by generating data to incorporate relations between objects in the visual environment. Finally, we find that current models are surprisingly data inefficient given the narrow scope of commands in gSCAN, suggesting another challenge for future work.

## introduction systematic generalization refers to the ability to understand new compositions of previously observed concepts and linguistic constructs. while humans exhibit this ability, neural networks often struggle. to study systematic generalization, several synthetic datasets have been proposed. @xcite introduced scan, a dataset composed of natural language instructions paired with action sequences, and splits in various ways to assess systematic generalization. recently, @xcite introduced the grounded scan (gscan) benchmark, which similarly pairs natural language instructions with action sequences, but further requires that instructions are interpreted within the context of a grid-based visual navigation environment. in this work, we analyze which aspects of gscan can currently be solved by general-purpose models, and propose new tasks and evaluation metrics for future research of systematic generalization on gscan. first, to understand which aspects of gscan can be addressed by a general-purpose approach, we evaluate a transformer-based model with crossmodal attention. cross-modal attention has been proven effective for other multi-modal tasks @xcite @xcite . it achieves strong performance on a majority of the splits, surprisingly outperforming several "specialist" approaches designed for gscan @xcite @xcite . we analyze the remaining errors, finding that many of the errors appear to be related to the same fundamental challenge in systematic generalization of linguistic constructs studied in datasets such as scan, regardless of the visual context. our analysis motivates the creation of an additional gscan task, which features a greater degree of complexity in how natural language instructions are grounded in the visual context. in this task, the agent needs to reason about spatial relations between objects expressed in language. we find this new task to be challenging for existing models. finally, we also assess the data efficiency of our cross-modal attention model on gscan. we find that despite the simplicity of the world state and the grammar used to generate instructions, model performance on most splits declines significantly when provided with less than ∼40% of the 360,000 original training examples. this suggests that we should consider sample complexity for future work. ## cross-modal attention solves gscan, almost experimental setup gscan has two types of generalization tasks: compositional generalization (cg) and length generalization. we focus on cg splits, which consist of a shared training set, 1 ran-split seq2seq (2020) geca ( 2020) kuo (2020) heinze (2020) gao (2020) film (2018 our model we implement a seq2seq model with 6 transformer layers for the encoder and the decoder each. the architecture is similar to vil-bert @xcite , a popular multi-modal model for visual and text information fusion. on a high-level, the encoder's text stream reads the commands while its visual stream encodes the world states. there are cross-modal attention between the two streams. the details are in appendix a. results we also compare to models with specialized designs for cg or even gscan: a seq2seq model using data augmentation @xcite , auxiliary loss @xcite , task-specific architecture @xcite ; gao prediction: l_turn l_turn l_turn l_turn walk pull pull target: l_turn l_turn l_turn l_turn walk l_turn l_turn l_turn l_turn pull l_turn l_turn l_turn pull command: pull a yellow big circle while spinning et al., 2020). the cross-modal attention model outperforms others on 5 out 8 splits. we hypothesize it is more effective as we leverage cross-modal attention to allow bi-directional interaction between language instruction and visual environment, in contrast to prior works that only have uni-directional attention from text to visual context @xcite @xcite . the additional attention from visual context to text improves grounding natural language instructions to the visual environment. however, on the "hard" splits (d, g and h), all methods struggle. analysis first, we analyze in details the "hard" splits. certain aspects of interpreting instructions are highly dependent on visual grounding. most prominently, every instruction requires resolving the location of a referred object. however, adverbs such as cautiously or while spinning have the same meaning regardless of the visual context. figure 1 shows one such example from the h split. the agent can successfully locate the target object but fails to combine the seen verb pull and adverb while spinning to generate the correct action sequence. therefore, to assess the degree to which errors are caused by incorrect visual grounding, we calculate split col match↑ row match↑ full match↑ no match↓ d 32.91 ± 3.02 45.30 ± 6.46 0.00 ± 0.00 21.78 ± 3.58 g 98.43 ± 3.40 98.82 ± 2.52 97.91 ± 4.50 0.66 ± 1.42 h 91.23 ± 8.53 92.90 ± 6.54 88.16 ± 11.40 4.04 ± 3.71 command: pull a green small square hesitantly prediction: r_turn walk stay pull stay pull stay target: l_turn l_turn walk stay l_turn walk stay the exact match between the target position and the agent's final position. the results are shown in next, we analyze the splits (a, b, c, e, and f) where the cross-modal attention model performs well. ## sample complexity one inspiration of studying systematic generalization is to develop techniques that can reduce the sample complexity for learning novel behaviors. while we observe that many models perform well at least on some splits, we ask a natural question that has not been studied before: even on those splits, are our models fulfilling the promise of learning systematic generalization? figure 4 shows that for the original gscan and the newly proposed task, performance for the cross-modal attention model starts to significantly drop when trained using less than around 40% of the training data for most splits. the model without cross-modal attention is even more data inefficient; performance starts to significantly drop when trained using less than 70% of the training data for the original compositional splits, or when reducing the training data by any amount for the spatial relation splits. this suggests exploring model architectural priors can help improve data efficiency and provide more benefits for generalization. to investigate how the number of primitives influences the data efficiency, we re-generate smaller compositional splits and spatial relation splits by reducing the number of primitives. for compositional splits, we exclude one noun (cylinder), one color adjective (blue), and one adverb (while zigzagging). for spatial relation splits, we exclude one noun (cylinder) and one location preposition (next to). this reduces the number of training examples by around 2/3 and leads to around 110,000 and 74,000 training examples for the smaller compositional splits and spatial relation splits respectively. we then perform similar experiments with our cross-modal attention model. the results are shown in figure 5 . when the number of primitives decreases, the percentage of training data required to attain satisfactory performance is increased, with more than 80% of training data needed for both splits. one possible explanation is that the total number of training examples is reduced when the number of primitives decreases. therefore, the model sees fewer combinations during training, thus needs a higher percentage of data to properly learn visual grounding and compositionality. this provides further evidence that current model need extensive training data to achieve systematic generalization, demonstrating the necessity of evaluating sample complexity for future work. ## related work many datasets and tasks have been proposed for examining systematic generalization: visual question answering @xcite @xcite , visually grounded navigation instruction following @xcite @xcite . many approaches have been proposed too, though a large percentage of them uses task-specific design and only work well for the specific tasks or datasets, for example, learning neural program executions @xcite @xcite @xcite , in addition to the ones we have described previously. this work focuses on using a generic crossmodal attention model to probe the gscan dataset in the hope of understanding in what aspect the task/dataset is challenging. @xcite also presented new evidences that a neural-based model can solve similar cg tasks. our interest is to use such models to inspire new task designs that continue to challenge neural models. ## conclusion in this work, we have demonstrated that a generalpurpose cross-modal attention model can achieve strong performance on a majority of gscan splits and outperform more specialized prior work. we have proposed a challenging additional task for gscan that requires agents to reason over spatial relations between objects in the visual scene, and have highlighted data efficiency as a consideration for future work.