Systematic Generalization on g SCAN with Language Conditioned Embedding (2020)

Abstract: Systematic Generalization refers to a learning algorithm’s ability to extrapolate learned behavior to unseen situations that are distinct but semantically similar to its training data. As shown in recent work, state-of-the-art deep learning models fail dramatically even on tasks for which they are designed when the test set is systematically different from the training data. We hypothesize that explicitly modeling the relations between objects in their contexts while learning their representations will help achieve systematic generalization. Therefore, we propose a novel method that learns objects’ contextualized embeddings with dynamic message passing conditioned on the input natural language and end-to-end trainable with other downstream deep learning modules. To our knowledge, this model is the first one that significantly outperforms the provided baseline and reaches state-of-the-art performance on grounded SCAN (gSCAN), a grounded natural language navigation dataset designed to require systematic generalization in its test splits.

## introduction systematic generalization refers to a learning algorithm's ability to extrapolate learned behavior to unseen situations that are distinct but semantically similar to its training data. it has long been recognized as a key aspect of humans' cognitive capacities @xcite . specifically, humans' mastery of systematic generalization is prevalent in grounded natural language understanding. for example, humans can reason about the relations between all pairs of concepts from two domains, even if they have only seen a small subset of pairs during training. if a child observes "red squares", "green squares" and "yellow circles", he or she can recognize "red circles" at their first encounter. humans can also contextualize their reasoning about objects' attributes. for example, a city being referred to as "the larger one" within a state might be referred to as "the smaller one" nationwide. in the past decade, deep neural networks have shown tremendous success on a collection of grounded natural language processing tasks, such as visual question answering (vqa), image captioning, and vision-and-language navigation @xcite . despite all the success, recent literature shows that current deep learning approaches are exploiting statistical patterns discovered in the datasets to achieve high performance, an approach that does not achieve systematic generalization. @xcite discovered that annotation artifacts like negation words or purpose clauses in natural language inference data can be used by simple text classification categorization model to solve the given task. @xcite demonstrated that adversarial examples can fool reading comprehension systems. indeed, deep learning models often fail to achieve systematic generalizations even on tasks on which they are claimed to perform well. as shown by @xcite , state-of-the-art visual questioning answering (vqa) @xcite ) models fail dramatically even on a synthetic vqa dataset designed with systematic difference between training and test sets. in this work, we focus on approaching systematic generalization in grounded natural language understanding tasks. we experiment with a recently introduced synthetic dataset, grounded scan (gscan), that requires systematic generalization to solve @xcite . for example, after observing how to "walk hesitantly" to a target object in a grid world, the learning agent is tested with instruction that requires it to "pull hesitantly", therefore testing its ability to generalize adverbs to unseen adverb-verb combinations. when presented with a world of objects with different attributes, and natural language sentences that describe such objects, the goal of the model is to generalize its ability to understand unseen sentences describing novel combinations of observed objects, or even novel objects with observed attributes. one of the essential steps in achieving this goal is to obtain good object embeddings to which natural language can be grounded. by considering each object as a bag of its descriptive attributes, this problem is further transformed into learning good representations for those attributes based on the training data. this requires: 1) learning good representations of attributes whose actual meanings are contextualized, for example, "smaller" and "lighter", etc.; 2) learning good representations for attributes so that conceptually similar attributes, e.g., "yellow" and "red", have similar representations. we hypothesize that explicitly modeling the relations between objects in their contexts, i.e., learning contextualized object embeddings, will help achieve systematic generalization. this is intuitively helpful for learning concepts with contextualized meaning, just as learning to recognize the "smaller" object in a novel pair requires experience of comparison between semantically similar object pairs. learning contextualized object embeddings can also be helpful for obtaining good representations for semantically similar concepts when such concepts are the only difference between two contexts. inspired by @xcite , we propose a novel method that learns an object's contextualized embedding with dynamic message passing conditioned on the input natural language. at each round of message passing, our model collects relational information between each object pair, and constructs an object's contextualized embedding as a weighted combination of them. such weights are dynamically computed conditioned on the input natural sentence. this contextualized object embedding scheme is trained end-to-end with downstream deep modules for specific grounded natural language processing tasks, such as navigation. experiments show that our approach significantly outperforms a strong baseline on gscan. ## related work research on deep learning models' systematic generalization behavior has gained traction in recent years, with particular focus on natural language processing tasks. ## conclusion and future work in this paper, we proposed a language-conditioned message passing model for a grounded language navigation task that can dynamically extract contextualized embeddings based on input command sentences, and can be trained end-to-end with the downstream action-sequence decoder. we showed that obtaining such contextualized embeddings improves performance on a recently introduced challenge problem, gscan, significantly outperforming the state-of-the-art across several test splits designed to test a model's ability to represent novel concept compositions and achieve systematic generalization. nonetheless, our model's fairly poor performance on split b and f shows that challenges still remain. as explained in the discussion section, our model is falling short of estimating the effect of each action on the agent's state. an alternative view of this problem is as a reinforcement learning task with sparse reward. sample-efficient model-based reinforcement learning @xcite could then be used, and its natural ability to explicitly model environment change should improve performance on this task. it would also be beneficial to visualize the dynamically generated edge weights during message passing to have a more intuitive understanding of what contextual information is integrated during the message passing phase. currently, we consider all objects appearing on the grid, including the agent, as homogeneous nodes during message passing, and all edges in the message passing graph are modelled in the same way. however, intuitively, we should model the relation between different types of objects differently. for example, the relation between the agent and the target object of pulling might be different from the relation between two objects on the grid. inspired by @xcite , it would be interesting to try modeling different edge types explicitly with neural modules, and perform type-specific message passing to obtain better contextualized embeddings.