Syntax-Guided Transformers: Elevating Compositional Generalization and Grounding in Multimodal Environments (2023)

Abstract: Compositional generalization, the ability of intelligent models to extrapolate understanding of components to novel compositions, is a fundamental yet challenging facet in AI research, especially within multimodal environments. In this work, we address this challenge by exploiting the syntactic structure of language to boost compositional generalization. This paper elevates the importance of syntactic grounding, particularly through attention masking techniques derived from text input parsing. We introduce and evaluate the merits of using syntactic information in the multimodal grounding problem. Our results on grounded compositional generalization underscore the positive impact of dependency parsing across diverse tasks when utilized with Weight Sharing across the Transformer encoder. The results push the state-of-the-art in multimodal grounding and parameter-efficient modeling and provide insights for future research.

## introduction compositional generalization refers to the ability of an intelligent agent to generalize its understanding of the underlying structure of a problem, especially when it is faced with novel compositions of the previously seen building blocks or components @xcite . it is fundamental for models to be able to extrapolate from their training environment to novel situations, a common occurrence in real-world applications. @xcite categorizes compositional generalization capabilities into five categories, systematicity, substitutivity, localism & globalism, and overgeneralization. these abilities are crucial for models to achieve strong performance on tasks that require reasoning and understanding of hierarchical structures, such as natural language understanding, object classification, and robotics. humans understand new compositions of previously observed concepts and simpler constructs. input command: pull the small blue object that is inside of the small green box and in the same row as the red circle while zigzagging. action sequence: turn left, turn left, walk, turn right, walk, turn right, walk, pull figure 1 : this example is taken from the reascan dataset. here, an agent is provided with a command. its objective is to generate/execute a series of predefined actions to fulfill the task within the given environment. on the other hand, despite remarkable progress in the field of artificial intelligence, even state-of-theart language models demonstrate limitations in this aspect @xcite @xcite . especially, they often fail to effectively generalize in the reasoning depth, which involves handling multi-turn reasoning about entities and their properties in the world or even the co-occurrence of unseen spatial relations @xcite . these limitations indicate a crucial need for innovative approaches to address these issues. in this research, our objective is to exploit the syntactic structure of language to enhance compositional generalization. our focus is mainly on the multimodal problem setting that entangles vision and language. in this unique setting, compositional linguistic descriptions must be accurately grounded in the environment to devise coherent action plans or achieve specific goals. an illustrative example of this scenario is shown in figure 1 . the motivation behind leveraging syntax in our approach stems from the inherent structure and compositionality of natural language. syntactic parsing provides crucial structural information about how words in a sentence relate to each other. we hypothesize that syntactic structure can improve intelligent agents' ability to discern the applicable attributes and descriptions for each object in its environment and better apprehend deeper levels of reasoning. by imposing an understanding of language structure through syntactic parsing, we aim to extend the capabilities of current multimodal language models. this could potentially pave the way for more sophisticated models capable of robustly interacting with dynamic and complex vision and language environments. apart from using structure, we equipped our end-to-end model with weight sharing that has demonstrated improving the generalization capabilities in single-modality tasks. as a result, we reach state-of-the-art performance on the reascan compositional generalization benchmark, showing improvement across all test splits, especially ones requiring sentence structure comprehension. in summary, our contributions include: ## related work the machine learning research community primarily focused on understanding the error bounds and the bias-variance trade-off @xcite to understand and improve the models' generalizability. later, techniques like dropout @xcite were introduced to improve neural models' generalization. recently, studies have examined the generalizability of various neural network architectures using specialized generalization evaluation tasks @xcite @xcite . additionally, numerous datasets such as scan (lake and baroni, 2018), cfq @xcite , and cogs (kim and linzen, 2020) have been developed to assess compositional generalization capabilities. diverse strategies such as data augmentation @xcite , innovative architectural designs @xcite , and neuro-symbolic methods @xcite , have been proposed to enhance these capabilities. consequently, these advances in text-based generalization have inspired research in multimodal compositional generalization, with developments including complex benchmarks like gscan @xcite and reascan @xcite , and advanced architectures applied to multimodal grounding @xcite @xcite @xcite . furthermore, recent research highlights the significant role of syntactic information in enhancing neural models' compositional generalization capability. @xcite suggested aligning the compositional structure of networks with the problem domain, resulting in a dynamic compositional neural network. moreover, @xcite and @xcite recommended grammar induction-based data augmentation techniques to improve compositional generalization. unlike our work that focuses on input command structure, @xcite introduced the concept of using parse tree node annotations in the target sequence of sequence-to-sequence tasks for enhancing compositional generalization. meanwhile, @xcite incorporated parse tree nodes into the etc @xcite ) model. they employed attention masking specific for etc to symbolize the relations of tokens and aid this model in a simplified classification task based on the cfq dataset. we are inspired by previous research @xcite ) that employs a similar technique with manually extracted parses for compositional generalization on the single text modality. however, our model utilizes off-the-shelf parsers instead of accurate manually generated parse trees, and it is generally applicable independent of the underlying models. ## problem setting various studies on compositional generalization have presented a range of tasks and problem settings (lake and @xcite @xcite @xcite . these datasets are comprised of a training set and several test sets. to ensure rigorous evaluation, the test sets have been deliberately structured to differ from the training set in a way that requires the compositional capability to succeed. our paper focuses on grounding natural language instructions in the visual modality, where we map words to specific objects or actions in a multimodal environment that provides a framework to evaluate an intelligent agent's compositional structures and spatial reasoning capabilities. we use the most recent multimodal compositional generalization benchmarks to assess our models comprehensively. in these benchmarks, an agent receives natural language instruction to carry out an action or navigate specific environments. these datasets are inherently synthetic, and they have been carefully crafted to guarantee that the test sets are systematically different from the training sets. by placing commands within a spatial context, these benchmarks bridge the gap between abstract cognitive understanding and practical action execution. consequently, they stand as both a scholarly tool for studying compositional generalization and a valuable resource for fields like robotics that require comprehension of spatially anchored commands. among these benchmarks, our primary focus is reascan, owing to its heightened complexity and recent introduction to the academic community. an example of this dataset, depicted in figure 1 , consists of three main components: the initial state of the world, the provided input command, and the corresponding target command. tasked with this information, the agent aims to infer the target command by leveraging both the information from the input command and the initial state. structurally, the world's representation in reas-can is formulated as a 6×6×17 matrix. each matrix cell comprises a 17-dimensional vector encapsulating information pertaining to an object's attributes-namely, color, shape, and size-along with indicators of the agent's positioning and orientation. the evaluation metric for this dataset is the percentage of exact matches of the predicted action sequence. the reascan dataset includes one random test split that mirrors the training's component and compound distribution, in addition to seven compositional generalization test splits. each of these splits is designed to probe a specific facet of a model's grounding generalization capability, as detailed in our transformer is based on the grocot model @xcite . each encoder layer employs a cross-attention mechanism between modalities, followed by modality-specific self-attention. our computed input command masks are utilized in the self-attention modules of the textual modality. remarkably, encoder layer weights are consistently shared across all layers. in the end, we concatenate the encoded result of each modality and pass it to the transformer's auto-regressive decoder to generate the action sequence corresponding to the input command given the environment. ## conclusion our research demonstrated that exploiting the syntactic structure of compositional and complex linguistic and spatial expressions improved the grounding ability of the instruction-follower agent in multimodal environments. our results indicated improvements compared to the previous state-ofthe-art models. in particular, we show that our proposed model is effective for generalization on tasks and test splits that require generalization over unobserved reasoning depths, such as the c1 split in the reascan dataset. by utilizing the syntacticguided attention masking along with the weight sharing, we achieved not only more accurate but also more parameter-efficient models for grounded compositional generalization.