Referring Image Segmentation via Joint Mask Contextual Embedding Learning and Progressive Alignment Network (2023)

Abstract: Referring image segmentation is a task that aims to predict pixel-wise masks corresponding to objects in an image described by natural language expressions. Previous methods for referring image segmentation employ a cascade framework to break down complex problems into multiple stages. However, its defects also obvious: existing methods within the cascade framework may encounter challenges in both maintaining a strong focus on the most relevant information during specific stages of the referring image segmentation process and rectifying errors propagated from early stages, which can ultimately result in sub-optimal performance. To address these limitations, we propose the Joint Mask Contextual Embedding Learning Network (JMCELN). JMCELN is designed to enhance the Cascade Framework by incorporating a Learnable Contextual Embedding and a Progressive Alignment Network (PAN). The Learnable Contextual Embedding module dynamically stores and utilizes reasoning information based on the current mask prediction results, enabling the network to adaptively capture and refine pertinent information for improved mask prediction accuracy. Furthermore, the Progressive Alignment Network (PAN) is introduced as an integral part of JMCELN. PAN leverages the output from the previous layer as a filter for the current output, effectively reducing inconsistencies between predictions from different stages. By iteratively aligning the predictions, PAN guides the Learnable Contextual Embedding to incorporate more discriminative information for reasoning, leading to enhanced prediction quality and a reduction in error propagation. With these methods, we achieved state-of-the-art results on three commonly used benchmarks, especially in more intricate datasets. The code will be released.

## introduction referring image segmentation is a task that aims to predict a pixel-wise mask for objects referred to in a natural language expression. it is one of the most basic tasks in human-robot interaction @xcite . in particular, it helps robots understand commands and localize corresponding objects. unlike semantic/instance segmentation @xcite , which focuses on the image only, referring image segmentation involves two different modalities: image and text. thus, in referring image segmentation, how to achieve a good alignment between the image and the most beneficial information in text is essential for the mask prediction. the prevailing methods in the field of referring image segmentation rely on the cascade framework, where static-text are merged with feature maps produced by the current stage to generate the output of the subsequent stage. as illustrated in fig. 1(a ) and (b), the late and early fusion frameworks are the two commonly used cascade frameworks. the late fusion framework, as adopted in methods like cgan and cris @xcite , extracts text and image features separately, and then integrates them to form multi-modality features for reasoning. on the other hand, the early fusion framework, employed in methods such as lavt and efn @xcite , extracts language features and conducts the entire fusion operation in the image encoding backbone. the cascade framework is a methodology that involves breaking down complex problems into multiple stages or steps, with each step building upon the previous ones. this approach can enhance the salience or prominence of referent objects in multistage reasoning processes. despite their effectiveness, its limitations are also apparent: (a) if the static text used in each stage of the cascade framework is overly complex or does not appropriately focus on the most relevant information for that particular stage, it can hinder the effectiveness of the approach. for instance, consider the expression in fig. 1 , "the giraffe not eating grass from a pole." a human observer would inspect the "giraffe" in the image and utilize additional attributes to localize the target. (b) the mistakes made in the early stages of the cascade framework can propagate and potentially affect the later stages. since the reasoning process in the cascade framework is built upon previous stages, any errors or incorrect information introduced early on can impact subsequent stages and potentially lead to incorrect conclusions or results. to address the aforementioned issue, we propose the joint mask contextual embedding learning network (jmceln). jmceln is designed to enhance the cascade framework by incorporating a learnable contextual embedding and a progressive alignment network (pan). the primary objective is to improve the accuracy of mask prediction by dynamically storing and utilizing reasoning information for each subsequent step. the learnable contextual embedding in jmceln allows for the dynamic storage and utilization of reasoning information. it is updated based on the current mask prediction results, rather than relying on static natural language expressions. this adaptability enables the network to capture and refine pertinent information effectively, leading to improved mask prediction. additionally, we introduce the progressive alignment network (pan) as part of jmceln. pan operates by using the output from the previous layer as a filter for the current output. it addresses inconsistencies between predictions from different stages by reducing the response of the current output and guiding the contextual embedding to include more discriminative information for reasoning. this iterative alignment process enhances the quality of predictions and reduces the propagation of mistakes. by combining the learnable contextual embedding and pan, jmceln achieves dynamic reasoning and refinement of mask predictions. this integrated approach overcomes the limitations of static text and offers a powerful solution for enhancing the salience and accuracy of referent objects within multi-stage reasoning tasks. in summary, cascade network presents two key challenges: the repetition of static text accumulates noise throughout reasoning and misunderstanding reasoning at earlier stages propagate, causing inaccurate predictions in subsequent stages. to avoid introduce too much noise, we introduce learnable contextual embedding. learnable contextual embedding aims to provide each stage dynamic and the most pertinent information, while pan combats inconsistencies between stages to avoid error propagation. learnable contextual embedding and progressive alignment network ensure the successful operation of our error correction mechanism within the cascade network. to evaluate the effectiveness of the proposed methods, we conduct extensive on three commonly used benchmarks and we achieved state-of-the-art results especially in in more intricate datasets. in conclusion, our main contributions are as follows: ## related work referring image segmentation the task of referring image segmentation was first proposed by @xcite . all work in this area can be categorized into two types. the first type includes one-stage image and language fusing of different layer feature maps in the image encoding backbone followed by a multi-layer feature exchange for better results. for example, rmi @xcite and dmn(margffoy-tuay et al., 2018) utilize a memory unit @xcite to fuse individual words with visual features. cmpc @xcite first perceives all entities in the image and then suppresses unrelated ones by graph reasoning. cmsa @xcite has a cross-modal self-attention module to capture long-range dependencies between image and language. brinet @xcite utilizes vision-guided linguistic attention to get useful language information and then language-guided visual attention to get the final outputs. busnet @xcite uses a bottom-up shift that progressively locates the referent. all these methods work separately on feature maps from different layers in the image encoding backbone. then, they fuse the feature maps produced by different layers by using a memory unit @xcite @xcite or multi-level feature exchange @xcite . not so much work implements methods like those above anymore because a one-stage interaction between visual and language information is not enough to predict accurate pixel-wise masks. the more recent second type of method aims at achieving better language and visual alignment. in particular, cgan @xcite has a cascade framework in which language features can fuse many times with multi-modality features output from the previous layer. cris @xcite utilizes a similar framework as cgan @xcite , while it benefits from a pre-trained clip @xcite model. @xcite and lavt @xcite show that better cross-modal alignments can be achieved through early fusion of linguistic and visual features. vlt @xcite generates multiple sets of word attention weights to represent different understandings. all of these methods employ static text to fuse with multi-modality features, while the information required in every stage is different. dynamic query based segmentation inspired by detr @xcite , some studies have proposed to use a zero/random initial vector to hook helpful information from the image feature map in semantic/instance/panoptic segmentation. maskformer @xcite and mask2former @xcite treat segmentation as a mask classification task and use a random initial query to generate information from the image feature map for final segmentation. k-net @xcite employs semantic kernels and instance kernels as dynamic kernels to generate helpful information stage-by-stage from the image encoding backbone (i.e. the cnn and transformer) for the final mask prediction. panoptic segformer @xcite proposes to use things queries and stuff queries to store different types of information. all these studies verify that learnable embedding can catch demanding information based on the task requirements. our study is the first to introduce a learnable embedding which can be updated adaptively in accordance with the requirements of each stage for stage reasoning. ## methods the overall framework is presented below. then, we detail the image and text encoder, lsran, and jmceln. finally, the upsampling module and loss is introduced. ## visualization while lavt and cris encountered difficulties and made mistakes in these four challenging cases, our method demonstrated robust performance and successfully tackled them. figure 4 presents qualitative results obtained from the ablation study. starting from the baseline and progressing to experiment #4, the cascade framework with learnable contextual embedding is introduced. this addition results in improved performance, as observed in the case of the "blue car" where the cascade framework with learnable contextual embedding predicts a more compact mask compared to the baseline. experiment #5 introduces the progressive alignment network (pan). notably, in the cases of "robertson" and "girl front right with hair up", pan demonstrates its ability to rectify incorrect predictions during the reasoning process. this showcases the effectiveness of pan in refining and improving the accuracy of the segmentation outputs. ## limitations as shown in tab.3, our methods produce a relative improvement in the miou metric of 4.38% in testa and 0.50% in testb. moreover, in testa, there is 4.38% improvement in pr@50, while pr@90 only improves by 2.94%. the improvement in pr@50 in testb is 1.07%, while pr@90 shows no improvement. by analyzing testa and testb, we find the target in testa is the human instances whose size are larger while the target in testb is mainly objects whose size are smaller. this indicates that our methods have more ability to identify larger target objects, but for the smaller objects, the improvement is limited. and our method is not good at producing high-quality output mask. compared with lavt, our cascade stages only uses 52 × 52 feature maps for reasoning, while lavt utilizes a clear-sighted early fusion method, in which the query is fused with a larger feature map with a 104 × 104 size in the first stage. the larger feature map is good for finding smaller objects and keeping the boundary. however, because the number of layers in image encoding backbone designed for other task is fixed, it is difficult for early fusion methods to define their own reasoning stages, which limits their performance. for example, swin-transformer @xcite ) has four layers, so lavt performs image-text alignment four times. by comparison, we adopt the late fusion cascade framework which is more flexible in defining the number of reasoning stages fit for this task. ## conclusion and future work we proposed a joint mask and contextual embedding learning network that dynamically updates the contextual embedding on the basis of the reasoning output and a long-short relation-aware network to learn objects relations. experimental results demonstrate the effectiveness of our methods. in the future, we want to handle the limitation mentioned in sec.6 and improve the targeting rate for small objects and recover boundary information better.