Comprehensive Multi-Modal Interactions for Referring Image Segmentation (2022)

Abstract: We investigate Referring Image Segmentation (RIS), which outputs a segmentation map corresponding to the natural language description. Addressing RIS efficiently requires considering the interactions happening across visual and linguistic modalities and the interactions within each modality. Existing methods are limited because they either compute different forms of interactions sequentially (leading to error propagation) or ignore intra-modal interactions. We address this limitation by performing all three interactions simultaneously through a Synchronous Multi-Modal Fusion Module (SFM). Moreover, to produce refined segmentation masks, we propose a novel Hierarchical Cross-Modal Aggregation Module (HCAM), where linguistic features facilitate the exchange of contextual information across the visual hierarchy. We present thorough ablation studies and validate our approach’s performance on four benchmark datasets, showing considerable performance gains over the existing state-of-the-art (SOTA) methods.

## introduction traditional computer vision tasks like detection and segmentation have dealt with a pre-defined set of categories, limiting their scalability and practicality. substituting the pre-defined categories with natural language expressions (nle) is a logical extension to counteract the above problems. indeed, this is how humans interact with objects in their environment; for example, the phrase "the kid running after the butterfly" requires localizing only the child running after the butterfly and not the other kids. formally, the task of localizing objects based on nle is known as visual grounding. existing works either approach the grounding problem by predicting a bounding box around the referred object or a segmentation mask corresponding to the referred object. we focus on the latter approach, as a segmentation mask can effectively pinpoint the exact location and capture the actual shape of the referred object. the task is formally known as referring image segmentation (ris). ris requires understanding both visual and linguistic modalities at an individual level, specifically word-word and region-region interactions. additionally, a mutual understanding of both modalities is required to identify the referred object from the linguistic expression and localize it in the image. for instance, to ground a sentence "whatever is on the truck", it is necessary to understand the relationship between words as grounding just the individual words will not work. similarly, regionto-region interactions in visual modality help group semantically similar regions, e.g., all regions belonging to the truck. finally, to identify the referent regions, we need to transfer the distinctive information about the referent from the linguistic modality to the visual modality; this is taken care of by the cross-modal word-region interactions. the current sota methods @xcite @xcite @xcite take a modular approach, where these interactions happen in parts, sequentially. different methods differ in how they model these interactions. @xcite first perform a region-word alignment (cross-modal interaction). the second stage takes these alignments as input to select relevant image regions corresponding to the referent. @xcite and @xcite use the dependency tree structure of the referring expression for the reasoning stage instead. @xcite select a suitable combination of words for each region, followed by selecting the relevant regions corresponding to referent based on the affinities with other regions. the performance of the initial stages bounds these approaches. furthermore, they ignore the crucial intra-modal interactions for ris. in this paper, we perform all three forms of interactions simultaneously. we propose a synchronous multi-modal fusion module (sfm) which captures the inter-modal and intra-modal interactions between visual and linguistic modalities in a single step. intra-modal interactions handle the cases for identifying the relevant set of words and semantically similar image regions. inter-modal interactions transfer contextual information across modalities. additionally, we propose a novel hierarchical cross-modal aggregation module (hcam) to exchange contextual information relevant to referent across visual hierarchies and refine the referred object's segmentation mask. we motivate the benefits of simultaneous interactions over sequential in figure 1 by presenting a failure case of the latter. for the given referring expression "anywhere, not on the people", sequential approaches fail to identify the correct word to be grounded, and the error gets propagated till the end. cmpc @xcite which predicts the referent word from the expression in the first stage, identifies "people" as the referent (middle image in figure 1 ) and completely misses "anywhere" which is the correct entity to ground. similarly, @xcite , and @xcite , which utilize dependency tree structure to govern their reasoning process, identify the referred entity "anywhere" as an adverb from the dependency tree. however, considering the expression in context with the image, the word "anywhere" should be perceived as a "pronoun". the proposed sfm module successfully addresses the aforementioned limitations. overall, our work makes the following contributions:-1. we propose sfm to reason over regions, words, and region-word features in a synchronous manner, allowing each modality to focus on relevant semantic information to identify the referred object. 2. we propose a novel hcam module, which routes hierarchical visual information through linguistic features to produce a refined segmentation mask. 3. we present thorough quantitative and qualitative experiments to demonstrate the efficacy of our approach and show notable performance gains on four ris benchmarks. referring expression comprehension: localizing a bounding box/proposals based on an nle is a task commonly referred to as referring expression comprehension (rec). the majority of methods for rec learn a joint embedding space for visual and linguistic modalities and differ in how joint space is computed and how it is used. earlier methods, @xcite @xcite used joint embedding space as a metric space to rank proposal features with linguistic features. later methods like @xcite @xcite utilized attention over the proposals to select the appropriate one. more recent methods like @xcite utilize transformer-based architecture to project multi-modal features to common semantic space. specifically, they utilize a self-attention mechanism to align proposal-level features with linguistic features. in our work, we utilize pixel-level image features which are crucial for the task of ris. additionally, compared to @xcite , we explicitly capture inter-modal and intra-modal interactions between visual and linguistic modalities. referring image segmentation: bounding boxbased methods in rec are limited in their capabilities to capture the inherent shape of the referred object, which led to the proposal of the ris task. it was first introduced in @xcite , where they generate the referent's segmentation mask by directly concatenating visual features from cnn with tiled language features from lstm. @xcite generates refined segmentation masks by incorporating multi-scale semantic information from the image. since each word in expression makes a different contribution in identifying the desired object, @xcite model visual context for each word separately using query attention. @xcite uses a self-attention mechanism to capture long-range correlations between visual and textual guy wearing green, long sleeves and blue denim pants modalities. recent works @xcite @xcite utilize cross-modal attention to model multi-modal context, @xcite use dependency tree structure and @xcite use coarse labelling for each word in the expression for selective context modelling. most of the existing works capture inter and intra modal interactions separately to model the context for referent. in this work, we concurrently model the comprehensive interactions across visual and linguistic modalities. ## method given an image and a natural language referring expression, the goal is to predict a pixel-level segmentation mask corresponding to the referred entity described by the expression. the overall architecture of the network is illustrated in figure 2 . visual features for the image are extracted using a cnn backbone, and linguistic features for the referring expression are extracted using a lstm. a synchronous multi-modal fusion module (sfm) simultaneously aligns visual regions with textual words and jointly reasons about both modalities to identify the multi-modal context relevant to the referent. sfm is applied to hierarchical visual features extracted from cnn backbone since hierarchical features are better suited for segmentation tasks @xcite @xcite . a novel hierarchical cross-modal aggregation module (hcam) is applied to effectively fuse sfm's multi-level output and produce a refined segmentation mask for the referent. we describe the feature extraction process in the next section, and both sfm and hcam modules are described in the subsequent sections. ## conclusion in this work, we tackled the task of referring image segmentation. we proposed a simple yet effective sfm to capture comprehensive interactions between modalities in a single step, allowing us to simultaneously consider the contextual information from both modalities. furthermore, we introduced a novel hcam module to aggregate multi-modal context across hierarchies. our approach achieves strong performance on ris benchmarks without any post-processing. we present thorough quantitative and qualitative experiments to demonstrate the efficacy of all the proposed components.