Text Augmented Spatial Aware Zero-shot Referring Image Segmentation (2023)

Abstract: In this paper, we study a challenging task of zero-shot referring image segmentation. This task aims to identify the instance mask that is most related to a referring expression without training on pixel-level annotations. Previous research takes advantage of pre-trained cross-modal models, e.g., CLIP, to align instance-level masks with referring expressions. Yet, CLIP only considers the global-level alignment of image-text pairs, neglecting fine-grained matching between the referring sentence and local image regions. To address this challenge, we introduce a Text Augmented Spatial-aware (TAS) zero-shot referring image segmentation framework that is training-free and robust to various visual encoders. TAS incorporates a mask proposal network for instance-level mask extraction, a text-augmented visual-text matching score for mining the image-text correlation, and a spatial rectifier for mask post-processing. Notably, the text-augmented visual-text matching score leverages a P -score and an N -score in addition to the typical visual-text matching score. The P -score is utilized to close the visual-text domain gap through a surrogate captioning model, where the score is computed between the surrogate model-generated texts and the referring expression. The N -score considers the fine-grained alignment of region-text pairs via negative phrase mining, encouraging the masked image to be repelled from the mined distracting phrases. Extensive experiments are conducted on various datasets, including RefCOCO, RefCOCO+, and RefCOCOg. The proposed method clearly outperforms state-of-the-art zero-shot referring image segmentation methods.

## introduction different from the traditional semantic segmentation tasks that predict masks belonging to predefined categories @xcite †corresponding author. 2021; @xcite , referring expression segmentation is a challenging task that requires identifying a specific object described by a referring expression @xcite @xcite . the task has wide application scenarios such as robot interaction, and image editing @xcite . the acquisition of precise referring expressions and dense mask annotations is labor-intensive, thereby limiting the practicality in real-world applications. moreover, the quality and precision of the obtained annotations cannot be guaranteed regarding the labor-intensive annotation process. therefore, we investigate zero-shot referring image segmentation to reduce labor costs as training on annotations is not required under this setting. recently, a zero-shot referring image segmentation framework is proposed @xcite . this framework initially extracts instance masks through an off-the-shelf mask proposal network. subsequently, the appropriate mask is selected by com-puting a global-local clip @xcite similarity between the referring expressions and the masked images. however, the method focuses on the single object in each mask proposal and does not consider other distracting objects within the image. moreover, since clip is trained on image-text pairs, directly applying it to the referring expression segmentation task that requires fine-grained region-text matching could degenerate the matching accuracy @xcite . another challenge arises from the domain gap between masked images and natural images @xcite @xcite , which affects the alignment between masked images and referring expressions. to this end, we introduce a text augmented spatial-aware (tas) zero-shot referring expression image segmentation framework composed of a mask proposal network, a text-augmented visualtext matching score, and a spatial rectifier. we utilize the off-the-shell segment anything model (sam) @xcite as the mask proposal network to obtain high-quality instance-level masks. to enhance the region-text aligning ability of clip and bridge the domain gap between the masked images and the natural images, a textaugmented visual-text matching score consisting of three components is calculated. the first score, called v -score, is the masked image text matching score used for measuring the similarity between masked images and referring expressions. the second component is the p -score. it bridges the textvisual domain gap by translating masked images into texts. specifically, a caption is generated for each masked image, followed by calculating its similarity with the referring expression. the inclusion of captions enhances the consistency between the referring expressions and the masked images. to improve fine-grained region-text matching accuracy, we further repel distracting objects in the image by calculating the n -score. the n -score is the cosine similarity between masked images and negative expressions. we mine these negative expressions by extracting noun phrases from the captions of the input images. the mask that is most related to the referring expression is selected according to a linear combination of the above scores. another challenge arises from the limitation of clip in comprehending orientation descriptions, as highlighted by @xcite . to address this issue, we propose a spatial rectifier as a post-processing module. for instance, to find out the mask corresponding to the referring expression "man to the left", we calculate the center point coordinates of all the masks and pick the mask with the highest text-augmented visual-text matching score from the left half of the masks. without modifying the clip architecture or further fine-tuning, our method facilitates clip prediction using a text-augmenting manner, boosting the zero-shot referring expression segmentation performance. we conduct experiments and ablation studies on refcoco, refcoco+, and re-fcocog. the proposed framework outperforms previous methods. ## conclusion in this paper, we propose a text augmented spatialaware (tas) framework for zero-shot referring image segmentation composed of a mask proposal network, a text-augmented visual-text matching score, and a spatial rectifier. we leverage off-theshell sam to obtain instance-level masks. then the text-augmented visual-text matching score is calculated to select the mask corresponding to the referring expression. the score uses positive text and negative text to bridge the visual-text domain gap and enhance fine-grained region-text alignment with the help of a caption model. followed by the post-processing operation in the spatial rectifier, tas is able to deal with long sentences with orientation descriptions. experiments on refcoco, refcoco+, and refcocog demonstrate the effectiveness of our method. future work may need to enhance comprehension of hard expressions over non-salient instances in the image. one potential way is to leverage the reasoning ability of large language models like gpt4.