Extending CLIP ’s Image-Text Alignment to Referring Image Segmentation (2024)

Abstract: Referring Image Segmentation (RIS) is a cross-modal task that aims to segment an instance described by a natural language expression. Recent methods leverage large-scale pretrained unimodal models as backbones along with fusion techniques for joint reasoning across modalities. However, the inherent cross-modal nature of RIS raises questions about the effectiveness of unimodal backbones. We propose RISCLIP, a novel framework that effectively leverages the cross-modal nature of CLIP for RIS. Observing CLIP’s inherent alignment between image and text features, we capitalize on this starting point and introduce simple but strong modules that enhance unimodal feature extraction and leverage rich alignment knowledge in CLIP’s image-text shared-embedding space. RISCLIP exhibits outstanding results on all three major RIS benchmarks and also outperforms previous CLIP-based methods, demonstrating the efficacy of our strategy in extending CLIP’s image-text alignment to RIS.

## introduction referring image segmentation (ris) is a multimodal task that aims to produce a pixel-wise mask of an instance referred to by a natural language expression. the task holds great potential with various applications, such as language-based image editing @xcite @xcite and human-robot interaction @xcite . ris poses a formidable challenge, demanding a nuanced understanding of both visual and linguistic modalities. thus, conventional methods @xcite @xcite @xcite leverage the profound knowledge learned by large-scale pretrained models, employing image and text encoders as backbones, such as swin-t @xcite trained on imagenet-21k @xcite and bert (de- however, the inherent cross-modal nature of ris raises questions about the effectiveness of unimodal backbones. in contrast, we posit that the cross-modal nature of clip @xcite makes it a better candidate for ris. extending on the findings of maskclip @xcite , we observe that clip possesses image-text alignment beneficial for ris-cosine similarity maps between patch-and sentence-level features @xcite produce preliminary groundings. to fully utilize this image-text alignment, we freeze clip and build upon this promising starting point. while previous work @xcite demonstrates the potential of adopting clip for ris, their performances trail behind the current state of the arts, indicating an opportunity for improvement. in our framework, risclip, we effectively adapt clip to ris, capitalizing on its image-text alignment. firstly, we enhance the unimodal feature extraction by introducing cross-modal interaction between the image and text encoders with crossmodal feature extraction (cfe) modules. these modules effectively align the image and text features at candidate regions-regions described by or related to the target text. then, we leverage the rich alignment knowledge captured in clip's image-text shared-embedding space by introducing inter-and intra-modal interactions after the feature extraction process with shared-space knowledge exploitation (ske) modules. the comprehensive interactions allow risclip to discern the target from the candidate regions. our cfe and ske modules effectively adapt clip to ris, evolving clip's preliminary image-text alignment into accurate groundings, as shown in fig. 1 . risclip exhibits outstanding performance on all three major ris benchmarks. particularly, risclip excels on the more challenging datasets, such as refcocog @xcite with complicated texts. such result indicates that adopting a cross-modal backbone like clip, which trains on varied captions including extensive expressions, is beneficial for ris. furthermore, risclip also surpasses previous clip-based methods @xcite , proving that such performance arises from both clip and our effective adaptation strategy. ## related work referring image segmentation. ris aims at predicting a pixel-wise mask of an object described by a natural language text. the pioneering work @xcite extracts image and text features with recurrent lstms and a cnn and concatenates them along the channel dimension into multi-modal features. follow-up work expands on this framework by incorporating recurrent multi-modal interactions @xcite along with more fine-grained segmentation with hierarchical visual features @xcite @xcite . another line of research focuses on attending to more important words in the referring expression @xcite @xcite and proposes effective cross-modal attention modules @xcite @xcite . recent methods adopt pretrained transformer encoders to extract image and text features @xcite , and others further leverage the encoder transformer layers for multi-modal feature extraction @xcite @xcite . moving towards real-world conditions, recent work tackles settings where expressions describe none to multiple objects @xcite , image-expression pairs only are provided without segmentation masks @xcite @xcite , and no labels are provided for training @xcite . contrastive language-image pre-training. clip @xcite is well-known for its general cross-modal capacity. acquired through extensive contrastive pretraining on large-scale image-text pairs, clip carries not only expertise knowledge in both visual and linguistic modalities but also general alignment between image and text features. various multi-modal tasks, including text-to-image generation @xcite and visual captioning @xcite , benefit from clip's rich multi-modal alignment. several works attempt to adapt clip to dense prediction tasks, such as open vocabulary object detection @xcite and semantic segmentation @xcite . in particular, maskclip @xcite exploits the alignment between patchand sentence-level features for zero-shot open vocabulary segmentation. we hypothesize that such patch-level alignment is a good starting point for ris, and, consistent with such hypothesis, observe that the alignment produces a noticeable 23.86 miou on the most challenging ris benchmark-refcocog-umd (nagaraja et al., 2016)-with a upsampling decoder attached. thus, we propose a new framework that effectively exploits such informative cross-modal alignment of clip to produce accurate ris predictions. clip for ris. although methods that adopt clip for ris exist @xcite , they either fully finetune clip and risk losing its general knowledge @xcite or do not explicitly lever-age the alignment between the image and text features learned from millions of image-text pairs @xcite . above all, their performance falls behind the state of the art, suggesting room for improvement. thus, we take a new approach of explicitly exploiting clip's rich image-text alignment by extracting preliminary grounding maps from frozen clip and enhancing them into accurate segmentations with our adaptive modules. our framework achieves compelling results, showing that we effectively extend clip's image-text alignment to ris. ## method fig. 2 illustrates the overall pipeline of our method, risclip. we exploit clip's cross-modal alignment between patch-and sentence-level features and evolve cosine similarity maps between them into precise pixel-wise groundings with our new framework. the cross-modal feature extraction (cfe) modules enhance the unimodal feature extraction of clip with cross-modal communication, aligning image and text features at candidate regions related to the text. the shared-space knowledge exploitation (ske) modules exploit clip's rich knowledge captured within the imagetext shared embedding space to discern the target referent from candidate regions, particularly those described by complicated expressions. together, cfe and ske modules adapt clip to ris, producing precise patch-level grounding maps. finally, a simple decoder refines these maps into pixel-level segmentations. the following sections detail each module-cfe, ske, and the decoder-starting with the original clip feature extraction. ## conclusion risclip effectively extends the image-text alignment of clip to ris, achieving outstanding performance on all major ris benchmarks. we effectively build upon clip's patch-level image-text alignment by introducing cross-modal communication during feature extraction and leveraging the rich cross-modal alignment within clip's imagetext shared-embedding space to successfully delineate referents described by complicated texts.