Mind the Context: The Impact of Contextualization in Neural Module Networks for Grounding Visual Referring Expressions (2021)

Abstract: Neural module networks (NMN) are a popular approach for grounding visual referring expressions. Prior implementations of NMN use pre-defined and fixed textual inputs in their module instantiation. This necessitates a large number of modules as they lack the ability to share weights and exploit associations between similar textual contexts (e.g. “dark cube on the left” vs. “black cube on the left”). In this work, we address these limitations and evaluate the impact of contextual clues in improving the performance of NMN models. First, we address the problem of fixed textual inputs by parameterizing the module arguments. This substantially reduce the number of modules in NMN by up to 75% without any loss in performance. Next we propose a method to contextualize our parameterized model to enhance the module’s capacity in exploiting the visiolinguistic associations. Our model outperforms the state-of-the-art NMN model on CLEVR-Ref+ dataset with +8.1% improvement in accuracy on the single-referent test set and +4.3% on the full test set. Additionally, we demonstrate that contextualization provides +11.2% and +1.7% improvements in accuracy over prior NMN models on CLOSURE and NLVR2. We further evaluate the impact of our contextualization by constructing a contrast set for CLEVR-Ref+, which we call CC-Ref+. We significantly outperform the baselines by as much as +10.4% absolute accuracy on CC-Ref+, illustrating the generalization skills of our approach.

## introduction visual referring expression recognition is the task of identifying the object in an image that is referred to by a natural language expression @xcite . it is a fundamental r 1 : the metallic object that is second one of the thing from right and to the right of [ big cubical metallic thing ] filter shape filter size filter material @xcite [large] [big cubical metallic thing] although achieving promising results, exist-ing nmn models primarily focused on designing module architectures with textual inputs directly hard-coded in the module instantiation @xcite . for example, processing the textual inputs 'red' and 'blue' require the instantiation of two different modules filter_color @xcite and filter_color @xcite . however, such a design demands a large number of learnable modules (and network parameters) and they cannot share weights for similar contextual textual inputs (e.g. 'dark cube' vs. 'black cube', 'shiny cylinder' vs. 'metallic cylinder'). lack of these contextual signals leads to poor generalization performance on unseen but known language contexts (lake and @xcite . moreover, in the prior implementations of nmn such as iep-ref @xcite , the modules in execution engine are not conditioned on the surrounding context of their textual input in the expression. this is problematic as the modules are not given the opportunity to watch the neighborhood of textual input that helps in extracting the informative visiolinguistic context from the module's visual input. for example, the module filter_color @xcite needs to pick a black colored cube or a red-colored cube depending on the neighborhood context in the expression (e.g. "the dark thing that is hardly visible" vs. "the dark thing among the red cubes") and the type of cubes available in its visual input. few implementations of nmn such as film @xcite and n2nmn @xcite parametrize the surrounding context of their textual input. however, the visiolinguistic context in these modules is rather shallow as they cannot jointly co-attend over potential objects of interest directly from the visual input and textual inputs. in this work, we address the aforementioned issues and evaluate the impact of contextual signals in improving the performance of nmn models. first, we address the problem of hard-coded language inputs by parameterizing the module arguments (figure 1 ), i.e., for example, we treat "filter_size" module as parameterized by textual input "large" instead of as a standalone function "filter_size[large]" ( §3). we show that module parametrization reduces the total number of learnable modules by 75% without affecting the performance of nmns. second, we use the ground-truth annotations in clevr-ref+ @xcite , a challenging synthetic referring expression dataset, to show the evidence that providing the relevant neighborhood context of the textual input to the neural module (see figure 1 ) is beneficial for improving the model's grounding performance ( §4.1). we next propose a contextualization method to learn to select the most relevant neighborhood context by jointly co-attending on visual and textual inputs, eliminating the need for ground-truth contextual information ( §4.2). our experimental results show that our approach is effective in capturing visiolinguistic relations and contextual dependencies, especially when the textual inputs are long, and has complex linguistic structures. we demonstrate that our proposed method significantly improves the performance of nmn ( §5.4) in grounding visual referring expressions. specifically, on clevr-ref+ benchmark, we outperform competing nmn approaches such as iep-ref, film and n2nmn by as much as +8.1% accuracy on single-referent split (s-ref) and +4.3% on full-referent split (f-ref). additionally, we also test our approach on closure @xcite and nlvr2 @xcite benchmarks. closure is a vqa benchmark consisting of clevr-like questions with emphasis on simple and complex referring expressions. nlvr2 is a language grounding task where the goal is to determine whether an expression is true based on two paired real images. our approach significantly outperforms the existing nmn approaches with +11.2% and +1.7% improvements in accuracy on closure and nlvr2 respectively. we further evaluate the impact of our contextualization by constructing a set of contrasting perturbations around clevr-ref+ test instances @xcite , and call our new dataset cc-ref+ ( §5.6). we significantly outperform the state-ofthe-art models by as much as +10.4% absolute accuracy on cc-ref+. ## related work referring expression recognition. visual referring expression recognition (ref) is the task of identifying the object in an image that is referred to by a natural language expression @xcite . datasets containing real images and expressions such as refcoco+ @xcite and ref-cocog @xcite have been proposed to evaluate the progress on this task. multi-modal transformers @xcite @xcite , using pretrain-then-transfer approach, have shown superior performance on these datasets. however, these models fail to learn robust visio-linguistic contextual representations and are shown to exploit the imbalanced distribution in the train and test splits @xcite . recently, clevr-ref+ @xcite has been introduced as a synthetic diagnostic benchmark that allows control over dataset bias. there are nearly 0.8m referring expressions of which 32% of expressions refer to only a single object (single-referent) and 68% refer to more than one object (multi-referent). in this paper, we refer to the full dataset as f-ref and the single-referent subset as s-ref. module network @xcite @xcite based architectures achieved new state-of-the-art performance on this dataset. ## module parameterization in nmn we propose parametrization as the first step to enable weight sharing and exploiting associations between similar textual contexts. specifically, we evaluate the effectiveness of parameterizing module textual inputs using iep-ref @xcite as the baseline nmn implementation. iep-ref, a nmn solution based on iep @xcite we parameterize each rcb module m as follows: (a) we feed all the words in the textual input e m into an lstm; (b) the last hidden state of lstm h t is then used to perform element-wise multiplication with the output of the first convolution layer in the rcb block to produce joint representation c m of module's textual input (e m ) and visual input (v m ), which is then passed to relu function (see appendix figure 1b ): in addition to evaluating the model performance on the full clevr-ref+ dev (f-dev) and test (f-test) splits, we also evaluate the model on single-referent (s-ref) dev (s-dev) and test (s- we extend our parameterized model by contextualizing it with the neighborhood context of textual input in the referring expression. figure 1 shows an example. we leverage the ground-truth annotations available in clevr-ref+ to provide neighborhood context for the modules as follows: let us denote the ground-truth neural modules as m 1 , m 2 , m 3 , ..., m n for a given input referring expression q. suppose the modules m j and m k are children for the parent module m i in the ground-truth execution tree. we modify the architecture of each neural module shown where we concatenate the groundtruth arguments of all the children modules m j and m k and pass it as the neighborhood context to the parent module m i (see appendix figure 1c ). we test if this contextualization helps. as an ablation, we also test the model performance where the entire expression q is provided as neighborhood context for the modules instead of the relevant neighborhood. ## conclusion neural module networks (nmns) are widely used in language and vision tasks. we show that contextualizing these modules dramatically reduces the number of modules required and improve their grounding abilities, achieving a new state-of-the-art results on the clevr-ref+ visual referring expressions task. our analysis on clevr-ref+, clo-sure, nlvr2 and a new contrast set cc-ref+ demonstrate that our proposed method enhances nmns' ability to exploit visiolinguistic relationships. a.2 neural modules in parameterized iep-ref iep-ref (liu et al., 2019), the current state-ofthe-art neural module network (nmn) model for the clevr-ref+ dataset, uses a generic design of neural module architecture adapted from iep (john @xcite , which was designed for vqa task. 4 the modules take either two visual inputs (binary modules) or one visual input (unary modules). there are total 60 distinct modules in iep-ref. after parameterization (see figure 8b ), the distinct number of modules drop to 15 without any drop in the model performance (section 2 of main paper). that is, the number of a distinct set of modules (and the total number of parameters) used in the parameterized model reduces by 75%. moreover, although the network parameters of each parameterized module slightly increase due to the additional lstm unit, since each module in iep-ref can have multiple instantiations for the same textual input, we have fewer parameters than ief-ref in total.