ReCo: Retrieve and Co-segment for Zero-shot Transfer
Anonymous Author(s)
AffiliationAddress email
Abstract
Semantic segmentation has a broad range of applications, but its real-world impacthas been significantly limited by the prohibitive annotation costs necessary toenable deployment.
Segmentation methods that forgo supervision can side-stepthese costs, but exhibit the inconvenient requirement to provide labelled examplesfrom the target distribution to assign concept names to predictions.
An alternativeline of work in language-image pre-training has recently demonstrated the potential to produce models that can both assign names across large vocabularies ofconcepts and enable zero-shot transfer for classification, but do not demonstratecommensurate segmentation abilities.
In this work, we strive to achieve a synthesis of these two approaches that combinestheir strengths.
We leverage the retrieval abilities of one such language-image pretrained model, CLIP, to dynamically curate training sets from unlabelled imagesfor arbitrary collections of concept names, and leverage the robust correspondencesoffered by modern image representations to co-segment entities among the resulting collections.
The synthetic segment collections are then employed to construct asegmentation model (without requiring pixel labels) whose knowledge of conceptsis inherited from the scalable pre-training process of CLIP.
We demonstrate thatour approach, termed Re trieve and Co -segment (ReCo) performs favourably to unsupervised segmentation approaches while inheriting the convenience of nameablepredictions and zero-shot transfer.
We also demonstrate ReCo’s ability to generatespecialist segmenters for extremely rare objects.
Introduction
The objective of semantic segmentation is to partition an image into coherent regions and to assignto each region a semantic label.
This task has myriad applications across domains such as medicalimage analysis, autonomous driving, industrial process monitoring and wildlife tracking.
However,there are several key challenges that have hindered the deployment of existing semantic segmentationapproaches to date: (1) Cost : collecting manual pixel-level annotations is extraordinarily expensive (e.g. 90 minutes per image for high quality labels [14]), limiting the use of fully-supervisedapproaches; (2) Flexibility : supervised approaches have typically been trained with limited listsof pre-defined categories and lack the ability to recognise rare or novel categories (such as thosedescribed by free-form text); (3) Complexity of deployment : unsupervised segmentation methodshave dramatically reduced annotation costs, but still exhibit the inconvenience of requiring labelledexamples to assign names to predictions; (4) Data access : many existing approaches (both supervisedand unsupervised) are trained on the target data distribution , requiring both that this distribution is known at training time (limiting flexibility) and that this data is accessible, which may not be the casefor legal/ethical reasons (e.g. medical image data).
There is a rich body of semantic segmentation literature that proposes solutions to subsets of thesechallenges, but to our knowledge no existing work addresses their full combination.
To tackle all fourchallenges, we draw inspiration from two lines of recent research.
The first line of research has shownthat modern deep architectures (most notably vision transformers [17]) develop the ability to inferthe spatial extent of objects without pixel supervision [7, 57, 76, 83, 74] and moreover can establishsemantically consistent correspondences across images [13, 23].
The second line of research hasdemonstrated that large-scale visual-language pre-training [68, 34] produces models that possess botha large vocabulary and remarkable zero-shot transfer potential—the ability to recognise concepts ontarget datasets without access to the target data distribution during training.
In this work, we target a synthesis of these two approaches that draws on their respective strengthsfor the semantic segmentation task.
We first employ CLIP [68] to curate training sets from unlabelledimages for any desired list of concepts.
We then exploit the robust semantic correspondences offeredby modern vision backbones to co-segment concepts among the resulting curated collections.
Finally,we use these co-segmented concepts to construct a segmentation model for the given concepts, withoutrequiring training.
This framework, which we term Re trieve and Co -segment (ReCo), performsfavourably to existing unsupervised segmentation approaches while preserving the benefits of a widevocabulary, named predictions and zero-shot transfer exemplified by CLIP.
Our contributions are three-fold: (1) We propose the ReCo framework, enabling open vocabulary semantic segmentation, without pixel annotations or the need to provide labelled examplesfrom the target domain, thus supporting the zero-shot transfer; (2) We compare our approach toprior work on standard semantic segmentation benchmarks (COCO-Stuff [6], Cityscapes [14], and KITTI-STEP [84]), and further illustrate the ability of ReCo to segment rare concepts beyond thesebenchmarks; (3) For cases when the target distribution is available, we demonstrate that a simpleextension of our approach, ReCo+, can exploit this data access via unsupervised adaptation to bringfurther gains.
Related work
Our work is connected to several themes in the literature, which we describe next.
Unsupervised semantic segmentation.
There has been considerable recent progress towardsunsupervised semantic segmentation with deep neural networks by leveraging ideas from selfsupervised learning.
Learning objectives based on maximising mutual information between views [33,58], metric learning across proposals [89, 79], equivariance and invariance constraints [12], distillationof self-supervised feature correspondences [23] and cross modal cues (vision and LiDAR) [82] haveall shown their potential for this task.
One drawback of these approaches is a reliance on either nearestneighbour search on a held-out set with pixel level annotations or the Hungarian algorithm [44](optimally matching predictions against ground-truth semantic masks) to produce segments with names.
By contrast, ReCo is independent of labelled examples during both training and inference,simplifying deployment.
Weakly-supervised semantic segmentation.
To reduce annotation costs, a number of workshave explored weaker cues such eye tracking [60], pointing [4, 67, 11], sparse pixel labels [75],scribbles [48], web-queried samples [35], boxes [15, 41, 77], extreme clicks [61, 52], image-levellabels [91, 85, 18, 1, 8, 64, 66] and free-form text [87].
However, such approaches still requirethe weak annotation to be attached to the data used to train the segmentation model—by contrast,
ReCo can in principle train a segmenter from any unlabelled collection of images.
ReCo alsobears a conceptual similarity with webly-supervised approaches [19] for semantic segmentation [35,73].
These methods employ an image search engine such as Google to provide training samplesfor concepts.
However, their flexibility is limited (this approach cannot be applied to private orcommercially sensitive data, for example) and lack the ability to leverage the knowledge of the searchengine itself during inference (we demonstrate that integrating the vision-language model into theinference procedure brings significant gains in performance).
Zero-shot semantic segmentation with pretrained language/vision-language embeddings.
A richbody of work has explored zero-shot semantic segmentation, broadly defined as the task of segmentingcategories for which no labels were provided during training (often termed zero-label semanticsegmentation [86]).
The key idea underlying many of these works is to leverage relationships encodedin pretrained word embeddings (such as word2vec [53] or GloVe [65]) to enable generalisation tounseen categories [90, 5, 22, 86, 39, 28, 46, 62].
More recent work has sought to leverage the visionlanguage embedding space learned by CLIP [68] to improve dense prediction in various settings [92,69 , 45].
We adopt a variation of DenseCLIP [92] as a component of our framework.
Additionally,differently from the above, we pursue the formulation of zero-shot transfer popularised by CLIP [68]which evaluates performance on unseen datasets rather than unseen categories .
Consequently,unlike these works our model has no access to either labelled or unlabelled examples from thetarget data distribution.
We note one exception: in addition to their primary zero-shot evaluations, DenseCLIP [92] also report an “annotation-free” evaluation without access to the target dataset—wecompare our approach with theirs under an equivalent setting.
Large-vocabulary/rare concept segmenters.
To scale up the number of concepts that can besegmented by a model several strategies based on captions [20], grounded text descriptions [38]and annotation transfer [29, 31] have been explored.
In a differing direction, various losses andincremental learning techniques have been employed [30, 26] to better segment rare concepts.
Unlike
ReCo, however, each of the above approaches still requires costly pixel-level annotations.
Co-segmentation which aims to segment common regions among a collection of images, has beenwidely studied with classical computer vision approaches [70, 56, 80, 3, 37, 81].
The topic has beenrevisited with deep learning using shared encoder networks [9, 47, 2, 50], iterative refinement [88]and weak (class-label) supervision [27].
While ReCo can in principle make use of any unsupervisedco-segmentation algorithm, we find that a simple correlation strategy works well, and thus we adoptit for our approach.
Method
In this section, we introduce the ReCo framework (Sec. 3.1) which enables zero-shot transfer forsemantic segmentation with arbitrary categories.
We describe a language-based gating mechanism toenhance the segmentation quality (Sec. 3.2), and a pseudo-labelling scheme, ReCo+, that adapts to atarget distribution using predictions from ReCo (Sec. 3.3).
3.1 Retrieve and Co-segment (ReCo)
The inputs to ReCo are a collection of unlabelled images, and a list of text descriptions of conceptsto be segmented.
Through a combination of image retrieval and co-segmentation across the imagecollection, ReCo constructs a segmenter for the given concepts on the fly.
During inference , this segmenter is applied without fine-tuning to images from a target distribution of interest, thus supporting
Unlabelled images ⋮
A photo of a lionfish (1) Retrieval (2) Co-segmentation
F 1 ⊗ F
Language-guided co
-segmentation & context elimination
F 1 ⊗ F k
CLIP
Image encoder
Dense CLIP
F
F⋮
F k
F 1 ⊗ F 2 ⋯
F 2 ⊗ F 1 F 2 ⊗ F 2 ⋯ F 2 ⊗ F k ⋮ ⋮ ⋱
F k ⊗ F 1 F k ⊗ F 2 ⋯ F k ⊗ F k lionfish ⋯ ⋮ ⋯
A photo of a lionfish
Dense CLIP
Image encoder seeds a new lionfish image reference image embedding
F new ⊗ lionfish (3) Inference ⊙ ⋮
[Figure]
archive
⊗ and ⊙ denote inner product and Hadamard product, respectively.
See Sec.
3 for details.
the zero-shot transfer. The interaction of these three stages is illustrated in Fig. 2 and discussed inmore detail below. Curating exemplars through image retrieval.
The first hypothesis underpinning our approachis that it is possible to construct a sufficiently large and diverse unlabelled image collection thatcontains examples (or closely related examples) of any concept we wish to segment.
We considerthis hypothesis to be reasonable in light of the fact that recent research has produced a number ofdiverse datasets spanning billions of images [51, 21, 34, 72].
To make use of this hypothesis for segmentation, we employ a vision-language model to curate anarchive of images for each concept of interest.
In this work, we use CLIP [68], a model comprisingan image encoder and a text encoder that enables efficient image retrieval for free-form text queries.
In more detail, for a given text query describing a concept c the CLIP text encoder ψ T producesan embedding ψ T ( c ) ∈ R e which can be compared against the embeddings produced by the imageencoder ψ I for each image x in the unlabelled collection U : { ψ I ( x ) ∈ R e : x ∈ U} .
We thenbuild an archive from the k images that form nearest neighbours for each concept c .
While a moresophisticated strategy (for instance, adjusting the archive size according to concept difficulty orprevalence) is possible with ReCo, we found that this simple approach worked well, and so weadopt it here.
Note that, thanks to mature approximate nearest neighbour search techniques [36], thisarchive construction process can be readily applied to a collection U containing billions of images.
Co-segmentation with seed pixels.
The second hypothesis underpinning our approach is that modernvision-language models such as CLIP are capable of constructing archives of high purity (i.e. a highratio of the archive images contains the concept of interest) provided that such exemplars exist inthe underlying image collection U (our first hypothesis).
Our second hypothesis, which we validatethrough experiments in Sec. 4, enables the use of co-segmentation to obtain concept segments.
Concretely, given the k images comprising the archive for category c constructed by CLIP, we aimto segment regions corresponding to concepts that re-occur across the images.
Since we expectthe archive to be of high purity, and since modern visual backbones provide consistent semanticcorrespondences across images [23], we adopt a strategy of first identifying a seed pixel in each image that we are confident belongs to the target category, c .
Intuitively, good seed pixels are onesthat have close neighbours (i.e. strong support) across each of the archive images, since we assumethat: (i) the target concept is common to all images in the archive, (ii) our visual backbone willproduce consistent features for pixels belonging to the same concept.
We then use these seed pixelsto construct a reference embedding that can be used to classify pixels belonging to c .
In detail, we first extract dense features for each image in the archive using a pre-trained imageencoder ϕ
I :
[Equation]
where F i ∈ R d × h × w denotes the dense features for the i th image with height h , width w and dchannels.
Each such feature is L2-normalised along its channel dimension.
1 Note that any imageencoder ϕ
I can be employed here (it need not be CLIP).
We identify seed pixels in four steps: First, we construct an adjacency matrix
A khw × khw amongall features in the archive.
Here, each of the k × k submatrices, A ij ∈ R hw × hw , encodes pairwisesimilarities between features from image i and image j for i, j ∈ { 1 , ..., k } .
Second, we aim toidentify, for each pixel in the archive, the similarity of its nearest neighbour among each of the kimages.
To do so, we apply a max operator along the columns of each submatrix (reducing the hwcolums of each submatrix to 1 and reducing the overall adjacency matrix dimensions to khw × k ).
Third, we aim to identify the average support that each pixel has among the k archive images.
For this,we apply a mean operator over the columns of each submatrix such that each row of the resultingkhw × 1 matrix encodes the mean maximum similarity across k images.
Finally, we identify theseed pixel locations by applying an argmax operator to each of the k submatrices of size hw × 1 ,yielding the spatial indices of the features in each image with highest average maximum similarityacross the archive.
To construct a classifier for concept c , we simply average the embeddings of the k seed pixels fromits archive and L2-normalise the resultant vector to produce the reference embedding f c ∈ R d .
Inference.
To localise an instance of the category in a new image x new , we first compute the dotproduct between f c and the L2-normalised dense features F new from the new image, and pass theresult through a sigmoid:
[Equation]
where P c denotes an initial estimate of the probability map corresponding to category c .
To refine this probability map, we draw inspiration from recent work [92, 69] showing that densevisual CLIP features can be usefully correlated against a given CLIP text embedding.
Concretely,we employ the DenseCLIP mechanism of Zhou et al.
[92] to highlight regions of the input that aresalient for the target category c as follows.
For image x new , we extract features V new ∈ R e v × h × w(here e v denotes the value feature dimension) from the last self-attention layer values of the CLIPimage encoder, project the features into the joint space R e with the CLIP image encoder’s final linearprojection and L2-normalise the result.
We then compute the CLIP text embedding ψ T ( c ) ∈ R efor the target concept c (and L2-normalising) before producing a saliency map S c new ∈ [0 , 1] h × w byapplying the text embedding as a 1 × 1 convolution to the visual features and applying a sigmoidactivation function to the result.
Note that our use of a sigmoid activation function differs from thesoftmax used by [92], since we process each concept independently.
Our final probability map for category c is produced by the Hadamard product of these estimates:
[Equation]
In case of multiple categories predictions, we concatenate all the category prediction maps and applyargmax to the category dimension.
As a post-processing step, we also experiment with the effect ofapplying a CRF [43] as a simple post-processing step (similarly to [23]).
Pseudocode for ReCo canbe found in the supplementary material.
3.2 Language-guided co-segmentation and context elimination
Even when an archive has high purity, co-segmenting the target concept c from the collection ofimages can be challenging due to the potential presence of distractor category, ˜ c that often co-occurswith c .
For example, cars often co-occur with roads and likewise aeroplanes with sky .
To minimisethe risk of the co-segmentation algorithm anchoring on unintended categories, we introduce twomechanisms into the co-segmentation procedure, namely, language-guided co-segmentation andcontext elimination .
Language-guided co
segmentation.
Specifically, for each x i in the archive, we first compute thecorresponding saliency map S ci ∈ [0 , 1] h × w using DenseCLIP.
We then vectorise this map anduse it to filter the co-segmentation similarities by replacing each row of A j,i with the Hadamardproduct of itself with vec( S ci ) ∈ [0 , 1] 1 × hw (where vec( · ) denotes vectorisation) for each submatrixj = { 1 , ..., k } .
Intuitively, this serves to act as a gate that preserves similarities only for pixelsidentified by CLIP as salient for the target concept.
Context elimination.
To reduce the influence of widely co-occurring distractors, we select a fewcommon background categories ˜ c that frequently appear in images ( e.g., tree, sky, road ) and computetheir reference embeddings f ˜ c as described in the seeding procedure of Sec. 3.1.
We then use theresulting attention maps P ˜ c
i produced by these reference embeddings to suppress regions of commonbackground context in the subsequent co-segmentation processes for different categories.
This isdone similarly as above, by vectorising and replacing each row of A j,i with the Hadamard product ofitself with vec(1 − P ˜ ci ) ∈ [0 , 1] 1 × hw for each submatrix j = { 1 , ..., k } .
Note that if a target conceptc is identical to one of the common background categories ˜ c , we simply assign f c with f ˜ c .
The ReCo framework outlined above requires no training and no access to the target distribution.
However, it is possible to consider a simple extension to this idea when access to the target distribution is available.
Our proposed extension, ReCo+, simply trains a segmentation architecture ( e.g., DeepLabv3 + [10]) with the segmentation masks from ReCo as pseudo-masks.
Experiments
In this section, we start by describing the datasets used for our experiments (Sec. 4.1) and implementation details (Sec. 4.2).
Then, we conduct an ablation study (Sec. 4.3) and compare our model tostate-of-the-art methods for unsupervised semantic segmentation (Sec. 4.4).
Finally, we showcaseour model’s ability to segment rare-category objects (Sec. 4.5) and discuss limitations (Sec. 4.6).
4.1 Datasets
For our ablation study, we use the ImageNet1K [16] validation set to curate archive for conceptsof interest.
The dataset covers 1K classes with 50 images for each class.
To measure segmentationperformance in the zero-shot transfer setting, we use the PASCAL-Context [55] validation set forevaluation, which has 5,104 images of 59 categories excluding background class.
To compare with previous unsupervised segmentation methods, we further exploit the ImageNet1Ktraining set to construct the reference embedding for the concept we wish to segment.
The dataset has1K classes with 1.2M images.
We evaluate on standard benchmarks including the Cityscapes [14]validation split, which has 500 urban scene images of 27 categories, KITTI-STEP [84] validation set,which is composed of 2,981 urban scene images of 19 categories, and COCO-Stuff [6] validationsplit, which has 4,172 images of 171 low-level thing and stuff categories excluding backgroundclass.
Following [12, 23], we only use the 27 mid-level categories for evaluation.
For unsupervisedadaptation with ReCo+ (Sec. 3.1), we train on ReCo pseudo-labels on the Cityscapes training setwith 2,975 images, KITTI-STEP training set which contains 5,027 images, and the COCO-Stuff10Ksubset which has 9,000 images for each respective benchmark.
We emphasize that no ground-truthlabels are used for training.
Finally, to demonstrate our model’s ability to segment rare concepts, we use the LAION-5Bdataset [72] with 5 billion images as a large collection of images that we expect to satisfy our
[Figure]
first hypothesis, namely that it will have coverage of rare concepts.
To assess performance, we usethe FireNet dataset [59] which has 1,452 images spanning rare fire safety-related classes.
For ourexperiment, we select the fire extinguisher class as an example of a concept that is important butrare in vision datasets (it is not contained in ImageNet1K [71], for example) and evaluate ReCo on263 images containing at least one instance of the category.
As a further proof of concept, we alsodemonstrate co-segmentations of the Antikythera mechanism (a historical item that does not appearin WordNet [54], or any labelled vision datasets that we are aware of).
4.2 Implementation details
Here, we describe the hyperparameters used to train ReCo+, inference details and evaluation metrics.
Our implementation is based on the PyTorch library [63] and will be made publicly available.
ReCo+ Training.
While ReCo does not require training, we train ReCo+ based on the
DeepLabv3 + [10] segmentation architecture with a ResNet101 [25] backone on the predictionsfrom ReCo as described in 3.1.
All training images are resized and center-cropped to
× 320 pixelsand data augmentations such as random scaling, cropping, and horizontal flipping are applied withrandom color jittering, color dropping, and Gaussian blurring.
We use the Adam optimiser [42]with an initial learning rate of 5 × 10 − 4 and a weight decay of 2 × 10 − 4 with the Poly learning rateschedule as in [49, 10].
Training consists of 20K gradient iterations with a batch size of 8 and takesabout 5 hours on a single 24GB NVIDIA P40 GPU.
Inference.
Whenever DenseCLIP is employed, we use the ResNet50x16 model (following [92]) toconstruct a saliency map for each image.
For the COCO-Stuff and Cityscapes benchmarks, we resizeand center crop the input images to 320 × 320 pixels as in [23].
For the KITTI-STEP validation set,we use the original resolution of each image as in [40].
For the FireNet benchmark, we resize theshorter side of images to 512 pixels and predict a single class of fire extinguisher by thresholding thepredicted heatmap with probability of 0.5.
Evaluation metrics.
Following the common practice [33, 12, 23], we report pixel accuracy (Acc.)and mean intersection-over-union (mIoU).
4.3 Ablation studies
Ability of CLIP to curate archives.
We begin by assessing the validity of our second hypothesis—namely that CLIP is capable of achieving high purity archives from unlabelled images.
To this end,we evaluate the retrieval performance of different CLIP models on ImageNet1K validation set whenconstructing different archive sizes.
In detail, for each archive size, k , we compute the precision ofthe top- k retrieved images based on whether the the ground-truth image-labels match the query text.
As can be seen in Fig.
3 (left), all CLIP models achieve solid retrieval performance, suggesting theirpotential for curating high purity archives as part of ReCo.
Since ViT-L/14@336px performs best,we employ this as our retrieval model in the remaining experiments.
LGC CE CRF Acc.
mIoU ✗ ✗ [Table] ✗
[Table] ✓ [Table] ✓
[Table] ✓ ✓ [Table]
DenseCLIP
[Table]
ReCo
[Table]
✗
[Table]
✓
[Table]
Influence of archive size and visual encoder used for co-segmentation.
In Fig.
3 (right) we illustratethe effect of using different pre-trained architectures, e.g. MoCov2 [24], DINO [7], CLIP [68], DeiTSIN [57], as the archive size (and thus the number of images used for co-segmentation) changes.
They-axis depicts segmentation performance for ReCo with these configurations on the PASCAL-Contextbenchmark.
We observe that using larger archives tends to improve performance (likely due to theirhigh purity) and that features from DeiT-SIN perform best.
We therefore use DeiT-SIN with anarchive size of k =50 for the remaining experiments otherwise stated.
Influence of ReCo framework components.
We next assess the effect of the language-guidedco-segmentation and context elimination components of ReCo which seek to improve the quality ofco-segmentation achieved across each archive to boost downstream segmentation performance.
Whenapplying context elimination, we select tree, sky, building, road , and person as common backgroundconcepts appearing in natural images to be suppressed.
In Tab. 1 (left), we show the effect of both ofthese strategies, together with the effect of applying a CRF [43] as post-processing.
We observe thatboth language-guided co-segmentation and context elimination bring major gains in co-segmentationperformance, while the CRF brings a further small gain. In Tab. 1 (right), we study the influence ofintegrating DenseCLIP into the inference procedure. We observe a significant gain which we believeis driven by the notable robustness of CLIP features under zero-shot transfer [68].
We therefore useeach of these strategies (including CRF post-processing) in the remaining experiments.
4.4 Comparison to state-of-the-art unsupervised methods
We compare ReCo and ReCo+ to state-of-the-art unsupervised semantic segmentation models onstandard benchmarks, including COCO-Stuff [6], Cityscapes [14] and KITTI-STEP [84] under bothzero-shot transfer and unsupervised adaptation (training without labels on the target distribution).
For COCO-Stuff, we observe that the mid-level categories used for evaluation are somewhat abstract forretrieval (for instance, one mid-level category is “outdoor objects”, which may include many low-levelcategories beyond the target hierarchy).
To avoid introducing ambiguity to the co-segmentationprocedure, we instead directly use the low-level categories and then merge the predictions into themid-level categories.
Additionally, we rephrase two category names to be reduce ambiguity ( parkingto parking lot and vegetation to tree ) in Cityscapes and KITTI-STEP based on the descriptions foundin [14].
A detailed discussion can be found in the supplementary material.
As shown in Tab. 2, ReCo strongly outperforms prior models on all benchmarks for zero-shot transfer.
Under an unsupervised adaptation protocol, ReCo+ outperforms the state-of-the-art by a large marginon the Cityscapes and KITTI-STEP.
On COCO-Stuff, ReCo+ achieves slightly lower pixel accuracycompared to [23] but a considerably higher mIoU. In Fig.
1, we visualise the sample predictions ofour models on the COCO-Stuff.
4.5 Segmenting rare concepts
By virtue of inheriting CLIP’s diverse knowledge of nameable visual concepts, ReCo exhibits theability to segment rare categories.
We first demonstrate this ability for fire extinguisher objects,which have important fire-safety implications but seldom appear in popular semantic segmentationbenchmarks.
To assess performance, we evaluate segmentation quality on the FireNet dataset (asdescribed in Sec. 4.1) and achieve reasonable performance on pixel accuracy (93.3) and IoU (44.9)
[Table]
anism . We show selected samples from ReCo archives for each concept, together with their cosegmentations. In each cases, ReCo successfully identifies the regions associated with the concept. metrics.
In Fig.
4 (left) we visualise the co-segmentation produced by ReCo across sample imagescontainer fire extinguishers.
As an additional demonstration (Fig. 4, right), we also show cosegmentations for images containing the views of the (unique) Antikythera mechanism .
In both cases,we observe that ReCo is capable of co-segmenting the concept of interest without labelled examples.
4.6 Limitations
Our work has several limitations: (1) There may be cases for which our first hypothesis does nothold—concepts so rare that they do not appear in billion-image scale datasets.
In such cases, e.g. “apurple elephant with square orange feet wearing an inverted cowboy hat in front of the Doge’s
Palace”, ReCo will struggle.
(2) We make use of CLIP (a computationally heavy model) duringinference.
Future work could address this with pruning and hardware optimisation strategies.
We employ ImageNet as the unlabelled image collection for our primary investigations, which isknown to exhibit an object-centric bias.
While qualitative results suggest that ReCo can learn fromextremely diverse data [72], a more comprehensive empirical evaluation would strengthen this result.(4) Models like CLIP are expensive to train and are consequently updated infrequently.
If a newconcept emerges (e.g. a new product) after CLIP was trained, ReCo will be unable to build an archiveto enable co-segmentation for this concept.
Although our work is zero-shot in the sense that it isnot trained on the target distribution, we still use of a subset of the target data distribution (primarilythrough ablations on PASCAL-Context) to guide the development of our method.
As a result, ourdesign likely contains choices that subtly favour performance on the evaluation tasks.
Conclusion
In this work, we introduced the ReCo framework for semantic segmentation zero-shot transfer.
Bydrawing on the strengths of large-scale language-image pre-training and modern visual backbones,
ReCo attains the ability to segment rare concepts and to directly assign names to concepts withoutlabelled examples from the target distribution.
In addition, experimental comparisons demonstratethat ReCo strongly outperforms existing zero-shot transfer approaches that forgo pixel supervision.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 4.6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
We will
leave a discussion about potential negative societal impacts in the supp.
mat.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[No] We do
not provide code to reproduce the experiments.
However, we do provide
detailed
descriptions of experimental settings and pseudo- code.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 4.
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[No] ReCo involves no training (so we simply run inference).
For ReCo+ we experiments with a single seed.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.
If you are using existing assets (e g , code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] We will leave a discussion about this
in the supp.
mat.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We will provide a url to the code used for this work.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] We will leave a discussion about this in the supp.
mat.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] We will leave a discussion about this in the
supp.
mat.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]