R e CLIP : A Strong Zero-Shot Baseline for Referring Expression Comprehension (2022)

Abstract: Training a referring expression comprehension (ReC) model for a new visual domain requires collecting referring expressions, and potentially corresponding bounding boxes, for images in the domain. While large-scale pre-trained models are useful for image classification across domains, it remains unclear if they can be applied in a zero-shot manner to more complex tasks like ReC. We present ReCLIP, a simple but strong zero-shot baseline that repurposes CLIP, a state-of-the-art large-scale model, for ReC. Motivated by the close connection between ReC and CLIP’s contrastive pre-training objective, the first component of ReCLIP is a region-scoring method that isolates object proposals via cropping and blurring, and passes them to CLIP. However, through controlled experiments on a synthetic dataset, we find that CLIP is largely incapable of performing spatial reasoning off-the-shelf. We reduce the gap between zero-shot baselines from prior work and supervised models by as much as 29% on RefCOCOg, and on RefGTA (video game imagery), ReCLIP’s relative improvement over supervised ReC models trained on real images is 8%.

## introduction visual referring expression comprehension (rec)the task of localizing an object in an image given a textual referring expression-has applications in a broad range of visual domains. for example, rec is useful for guiding a robot in the real world @xcite and also for creating natural language interfaces for software applications with visuals @xcite . though the task is the same across domains, the domain shift is problematic for supervised referring expression models, as shown in figure 1 : the same simple (a) refcoco+ @xcite (b) refgta @xcite figure 1 : predictions from reclip (cyan) and uniter-large @xcite (red) for the same referring expression on images from two visual domains. uniter-large fails on the gta (video game) domain, while reclip selects the correct proposal in both cases. close-ups of the two gta boxes are shown. collecting task-specific data in each domain of interest is expensive. weakly supervised rec @xcite partially addresses this issue, since it does not require the ground-truth box for each referring expression, but it still assumes the availability of referring expressions paired with images and trains on these. given a large-scale pretrained vision and language model and a method for doing rec zero-shot-i.e. without any addi-figure 2 : overview of reclip. given object proposals, we isolate the corresponding image regions by cropping and blurring (only cropping shown here). using a parser, we extract the noun chunks of the expression. for each noun chunk, clip outputs a distribution over proposals. the relations from the parser and clip's probabilities are combined by a spatial relation resolver to select the final proposal. in this example, clip ranks b 3 highest for both noun chunks, but using the relation resolver we obtain the correct answer b 4 . pre-trained vision and language models like clip @xcite achieve strong zeroshot performance in image classification across visual domains @xcite and in object detection @xcite , but the same success has not yet been achieved in tasks requiring reasoning over vision and language. for example, @xcite show that a straightforward zero-shot approach for vqa using clip performs poorly. specific to rec, @xcite introduce a zeroshot approach via colorful prompt tuning (cpt), which colors object proposals and references the color in the text prompt to score proposals, but this has low accuracy. in both of these cases, the proposed zero-shot method is not aligned closely enough with the model's pre-training task of matching naturally occurring images and captions. in this work, we propose reclip, a simple but strong new baseline for zero-shot rec. reclip, illustrated in figure 2 , has two key components: a method for scoring object proposals using clip and a method for handling spatial relations between objects. our method for scoring region proposals, isolated proposal scoring (ips), effectively reduces rec to the contrastive pre-training task used by clip and other models. specifically, we propose to isolate individual proposals via cropping and blurring the images and to score these isolated proposals with the given expression using clip. to handle relations between objects, we first consider whether clip encodes the spatial information necessary to resolve these relations. we show through a controlled experiment on clevr images @xcite that clip and another pre-trained model albef @xcite are unable to perform its pre-training task on examples that require spatial reasoning. thus, any method that solely relies on these models is unlikely to resolve spatial relations accurately. consequently, we propose spatial heuristics for handling spatial relations in which an expression is decomposed into subqueries, clip is used to compute proposal probabilities for each subquery, and the outputs for all subqueries are combined with simple rules. on the standard refcoco/g/+ datasets @xcite , we find that reclip outperforms cpt @xcite by about 20%. compared to a stronger gradcam @xcite baseline, reclip obtains better accuracy on average and has less variance across object types. finally, in order to illustrate the practical value of zero-shot grounding, we also demonstrate that our zero-shot method surpasses the out-of-domain performance of state-of-the-art supervised rec models. we evaluate on the refgta dataset @xcite , which contains images from a video game (out of domain for models trained only on real photos). using reclip and an object detector trained outside the target domain, we outperform uniter-large @xcite (using the same proposals) and mdetr @xcite by an absolute 4.5% (relative improvement of 8%). in summary, our contributions include: (1) re-clip, a zero-shot method for referring expression comprehension, (2) showing that clip has low zero-shot spatial reasoning performance, and (3) a comparison of our zero-shot rec performance with the out-of-domain performance of state-of-the-art fully supervised rec systems. foot_0 2 background in this section, we first describe the task at hand ( §2.1) and introduce clip, the pre-trained model we primarily use ( §2.2). we then describe two existing methods for scoring region proposals using a pre-trained vision and language model: colorful prompt tuning ( §2.3) and gradcam ( §2.4). ## reclip reclip consists of two main components: (1) a region-scoring method that is different from cpt and gradcam and (2) a rule-based relation resolver. in this section, we first describe our region scoring method ( §3.1). however, using controlled experiments on a synthetic dataset, we find that clip has poor zero-shot spatial reasoning performance ( §3.2). therefore, we propose a system that uses heuristics to resolve spatial relations ( §3.3). ## related work referring expression comprehension datasets for rec span several visual domains, including photos of everyday scenes @xcite , video games @xcite , objects in robotic context @xcite , and webpages @xcite . spatial heuristics have been used in previous work @xcite . our work is also related to @xcite , which similarly decomposes the reasoning process into a parsing step and visual execution steps, but the visual execution is driven by learned binary classifiers for each predicate type. in the supervised setting, prior work shows that using an external parser, as we do, leads to lower accuracy than training a language module jointly with the remainder of the model @xcite . there is a long line of work in weakly supervised rec, where at training time, pairs of referring expressions and images are available but the ground-truth bounding boxes for each expression are not @xcite @xcite @xcite . our setting differs from the weakly supervised setting in that the model is not trained at all on the rec task. @xcite discuss a zero-shot setting different from ours in which novel objects are seen at test time, but the visual domain stays the same. pre-trained vision and language models early pre-trained vision and language models @xcite @xcite used a cross-modal transformer @xcite and pre-training tasks like masked language modeling, image-text matching, and image feature regression. by contrast, clip and similar models @xcite use a separate image and text transformer and a contrastive pre-training objective. recent hybrid approaches augment clip's architecture with a multi-modal transformer @xcite . zero-shot application of pre-trained models models pre-trained with the contrastive objective have exhibited strong zero-shot performance in image classification tasks @xcite . @xcite use clip can be to classify objects by computing scores for class labels with cropped proposals. our ips is different in that it isolates proposals by both cropping and blurring. @xcite show that a simple zero-shot application of clip to visual question answering performs almost on par with random chance. @xcite describe a zero-shot method for rec based on a pre-trained masked language model (mlm); we show that their zero-shot results and a version of their method adapted for models pre-trained to compute image-text scores (rather than mlm) are substantially worse than isolated proposal scoring and gradcam. ## conclusion we present reclip, a zero-shot method for referring expression comprehension (rec) that decomposes an expression into subqueries, uses clip to score isolated proposals against these subqueries, and combines the outputs with spatial heuristics. reclip outperforms zero-shot rec approaches from prior work and also performs well across visual domains: reclip outperforms state-of-the-art supervised rec models, trained on natural images, when evaluated on refgta. we also find that clip has low zero-shot spatial reasoning performance, suggesting the need for pre-training methods that account more for spatial reasoning.