CLEVR Parser: A Graph Parser Library for Geometric Learning on Language Grounded Image Scenes (2020)

Abstract: The CLEVR dataset has been used extensively in language grounded visual reasoning in Machine Learning (ML) and Natural Language Processing (NLP). We present a graph parser library for CLEVR, that provides functionalities for object-centric attributes and relationships extraction, and construction of structural graph representations for dual modalities. Structural order-invariant representations enable geometric learning and can aid in downstream tasks like language grounding to vision, robotics, compositionality, interpretability, and computational grammar construction. We provide three extensible main components – parser, embedder, and visualizer that can be tailored to suit specific learning setups. We also provide out-of-the-box functionality for seamless integration with popular deep graph neural network (GNN) libraries. Additionally, we discuss downstream usage and applications of the library, and how it can accelerate research for the NLP community.

## introduction the clevr dataset @xcite is a modern 3d incarnation of historically significant shapes-based datasets like shrdlu @xcite , used for demonstrating ai efficacy on language understanding @xcite @xcite . although originally aimed at the visual question answering (vqa) problem @xcite , its versatility has seen its use in diverse ml domains, including extensions to physics simulation engines for language augmented hierarchical reinforcement learning @xcite and causal reasoning @xcite . 1 code is available at - https://github.com/ raeidsaqur/clevr-parser (a) question on image (figure 2 ): 'is the color of the metal block that is right of the yellow rubber object the same as the large metal cylinder?' parallelly, research interest in geometric learning and @xcite @xcite based techniques have seen a dramatic surge in recent deep learning zeitgeist. in this focused paper, we present a library that allows easy integration and application of geometric representation learning on clevr dataset tasks -enabling the nlp research community to apply gnn based techniques to their research (see 4). the library has three main (extensible) components: 1. parser: allows extraction of graph structured relationships among objects of the environment -both for textual questions, and semantic image scene graphs, 2. embedder: allows generation of latent embeddings using any models or desired backend of choice (like pytorch 2 ), 3. visualizer: provides tools for visualizing structural graphs and latent embeddings. ## clevr-parser here we describe each of the main library components in detail. while we provide a concrete implementation using the spacy foot_0 nlp library, any other library like the stanford parser foot_1 , or nltk foot_2 could be used in its place. the output of the parser from a question and image is depicted in figure 1 .