ReCo: Retrieve and Co-segment for Zero-shot Transfer
Anonymous Author(s)
AffiliationAddress
Abstract
Semantic segmentation has a broad range of applications, but its real-world impacthas been significantly limited by the prohibitive annotation costs necessary toenable deployment.
Segmentation methods that forgo supervision can side-stepthese costs, but exhibit the inconvenient requirement to provide labelled examplesfrom the target distribution to assign concept names to predictions.
An alternativeline of work in language-image pre-training has recently demonstrated the potential to produce models that can both assign names across large vocabularies ofconcepts and enable zero-shot transfer for classification, but do not demonstratecommensurate segmentation abilities.
We leverage the retrieval abilities of one such language-image pre-trained model, CLIP, to dynamically curate training sets from unlabelled images for arbitrarycollections of concept names, and leverage the robust correspondences offeredby modern image representations to co-segment entities among the resulting collections.
The synthetic segment collections are then employed to construct asegmentation model (without requiring pixel labels) whose knowledge of conceptsis inherited from the scalable pre-training process of CLIP.
We demonstrate that ourapproach, termed Re trieve and Co -segment (ReCo) performs favourably to conventional unsupervised segmentation approaches while inheriting the convenience ofnameable predictions and zero-shot transfer.
We also demonstrate ReCo’s abilityto generate specialist segmenters for extremely rare objects.
Introduction
The objective of semantic segmentation is to partition an image into coherent regions and to assignto each region a semantic label.
This task has myriad applications across domains such as medicalimage analysis, autonomous driving, industrial process monitoring and wildlife tracking.
However,there are several key challenges that have hindered the deployment of existing semantic segmentationapproaches to date: (1) Cost : collecting manual pixel-level annotations is extraordinarily expensive (e.g. 90 minutes per image for high quality labels [14]), limiting the use of fully-supervisedapproaches; (2) Flexibility : supervised approaches have typically been trained with limited lists of pre- and the to recognise rare or novel categories (such as those described by -form unsupervised segmentation the inconvenience of requiring to predictions existing approaches (both supervised and unsupervised) trained is at training limiting be case
There is a rich body of semantic segmentation literature that proposes solutions to subsets of thesechallenges, but to our knowledge no existing work addresses their full combination.
To tackle allfour challenges, we draw inspiration from two lines of recent research.
The first line of research hasshown that modern deep architectures (most notably vision transformers [18]) develop the abilityto infer the spatial extent of objects without pixel supervision [7, 59, 78, 85, 76, 54] and moreovercan semantically consistent correspondences across images [13,
The second of research large-scale -language pre -training [70, 35]
In this work, we target a synthesis of these two approaches that draws on their respective strengthsfor the semantic segmentation task.
We first employ CLIP [70] to curate training sets from unlabelledimages for any desired list of concepts.
We then exploit the robust semantic correspondences offeredby modern vision backbones to co-segment concepts among the resulting curated collections.
Finally,we use these co-segmented concepts to construct a segmentation model for the given concepts, withoutrequiring training.
This framework, which we term Re trieve and Co -segment (ReCo), performsfavourably to conventional unsupervised segmentation approaches while preserving the benefits of awide vocabulary, named predictions and zero-shot transfer exemplified by CLIP.
Our contributions are three-fold: (1) We propose the ReCo framework, enabling open vocabularysemantic segmentation, without pixel annotations or the need to provide labelled examples from thetarget domain thereby enabling zero-shot transfer; (2) We compare our approach to prior work on standard semantic segmentation benchmarks (COCO-Stuff [6], Cityscapes [14], and KITTI-STEP [86]),and further illustrate the ability of ReCo to segment rare concepts beyond these benchmarks; (3) Forcases when the target image distribution is available, we demonstrate that a simple extension of ourapproach, ReCo+, can exploit this data access via unsupervised adaptation to bring further gains.
Related work
Our work is connected to several themes in the literature, which we describe next.
Unsupervised semantic segmentation.
There has been considerable recent progress towardsunsupervised semantic segmentation with deep neural networks by leveraging ideas from selfsupervised learning.
Learning objectives based on maximising mutual information between views [34,60], metric learning across proposals [92, 81], equivariance and invariance constraints [12], distillationof self-supervised feature correspondences [24] and cross modal cues (vision and LiDAR) [84] haveall shown their potential for this task.
One drawback of these approaches is a reliance on either nearest neighbour search on a held-out setwith pixel level annotations or the Hungarian algorithm [45] (optimally matching predictions againstground-truth semantic masks) to produce segments with names.
By contrast, ReCo is independent oflabelled examples during both training and inference, simplifying deployment.
Weakly-supervised semantic segmentation.
To reduce annotation costs, a number of workshave explored weaker cues such eye tracking [62], pointing [4, 69, 11], sparse pixel labels [77],scribbles [49], web-queried samples [36], boxes [15, 42, 79], extreme clicks [63, 53], image-levellabels [94, 87, 19, 1, 8, 66, 68] and free-form text [89].
However, such approaches still requirethe weak annotation to be attached to the data used to train the segmentation model—by contrast,
ReCo can in principle train a segmenter from any unlabelled collection of images.
ReCo alsobears a conceptual similarity with webly-supervised approaches [20] for semantic segmentation [36,75].
These methods employ an image search engine such as Google to provide training samplesfor concepts.
However, their flexibility is limited (this approach cannot be applied to private orcommercially sensitive data, for example) and they lack the ability to leverage the knowledge of thesearch engine itself during inference (we demonstrate that integrating the vision-language model intothe inference procedure brings significant gains in performance).
Zero-shot semantic segmentation with pretrained language/vision-language embeddings.
Adiverse body of work has explored zero-shot semantic segmentation, broadly defined as the taskof segmenting categories for which no labels were provided during training (often termed zerolabel semantic segmentation [88]).
The key idea underlying many of these works is to leveragerelationships encoded in pretrained word embeddings (such as word2vec [55] or GloVe [67]) toenable generalisation to unseen categories [93, 5, 23, 88, 40, 29, 47, 64].
More recent work hassought to leverage the vision-language embedding space learned by CLIP [70] to improve denseprediction in various settings [95, 71, 46, 90, 17].
We adopt a variation of DenseCLIP [95] as acomponent of our framework.
Additionally, differently from the above, we pursue the formulationof zero-shot transfer popularised by CLIP [70] which evaluates performance on unseen datasetsrather than unseen categories .
Consequently, unlike these works, our model has no access to eitherlabelled or unlabelled examples from the target data distribution (or pixel-level labels from the sourcedistribution, as investigated by [90]).
We note one exception: in addition to their primary zero-shotevaluations, DenseCLIP [95] also report an “annotation-free” evaluation without access to the targetdataset—we compare our approach with theirs under an equivalent setting.
Large-vocabulary/rare concept segmenters.
To scale up the number of concepts that can besegmented by a model several strategies based on captions [21], grounded text descriptions [39]and annotation transfer [30, 32] have been explored.
In a differing direction, various losses andincremental learning techniques have been employed [31, 27] to better segment rare concepts.
Unlike
ReCo, however, each of the above approaches still requires costly pixel-level annotations.
Co-segmentation which aims to segment common regions among a collection of images, has beenwidely studied with classical computer vision approaches [72, 58, 82, 3, 38, 83].
The topic has beenrevisited with deep learning using shared encoder networks [9, 48, 2, 51], iterative refinement [91]and weak (class-label) supervision [28].
While ReCo can in principle make use of any unsupervisedco-segmentation algorithm, we find that a simple correlation strategy works well, and thus we adoptit for our approach.
Method
In this section, we introduce the ReCo framework (Sec. 3.1) which enables zero-shot transfer forsemantic segmentation with arbitrary categories.
We describe a language-based gating mechanismto enhance segmentation quality (Sec. 3.2), and a pseudo-labelling scheme, ReCo+, that adapts to atarget distribution using predictions from ReCo (Sec. 3.3).
3.1 Retrieve and Co-segment (ReCo)
The inputs to ReCo are a collection of unlabelled images, and a list of text descriptions of conceptsto be segmented.
Through a combination of image retrieval and co-segmentation across the imagecollection, ReCo constructs a segmenter for the given concepts on the fly.
During inference , this segmenter is applied without fine-tuning to images from a target distribution of interest, thus supportingthe zero-shot transfer.
The interaction of these three stages is illustrated in Fig.
2 and discussed inmore detail below.
Unlabelled images ⋮
A photo of a lionfish (1) Retrieval (2) Co-segmentation
F 1 ⊗ F
Language-guided co
-segmentation & context elimination
F 1 ⊗ F k
CLIP
Image encoder
Dense CLIP
F
F⋮
F k
F 1 ⊗ F 2 ⋯
F 2 ⊗ F 1 F 2 ⊗ F 2 ⋯ F 2 ⊗ F k ⋮ ⋮ ⋱
F k ⊗ F 1 F k ⊗ F 2 ⋯ F k ⊗ F k lionfish ⋯ ⋮ ⋯
A photo of a lionfish
Dense CLIP
Image encoder seeds a new lionfish image reference image embedding
F new ⊗ lionfish (3) Inference ⊙ ⋮
[Figure]
archive
Curating exemplars through image retrieval.
The first hypothesis underpinning our approachis that it is possible to construct a sufficiently large and diverse unlabelled image collection thatcontains examples (or closely related examples) of any concept we wish to segment.
We considerthis hypothesis to be reasonable in light of the fact that recent research has produced a number ofdiverse datasets spanning billions of images [52, 22, 35, 74].
To make use of this hypothesis for segmentation, we employ a vision-language model to curate anarchive of images for each concept of interest.
In this work, we use CLIP [70], a model comprisingan image encoder and a text encoder that enables efficient image retrieval for free-form text queries.
In more detail, for a given text query describing a concept c the CLIP text encoder ψ T producesan embedding ψ T ( c ) ∈ R e which can be compared against the embeddings produced by the imageencoder ψ I for each image x in the unlabelled collection U : { ψ I ( x ) ∈ R e : x ∈ U} .
We thenbuild an archive from the k images that form nearest neighbours for each concept c .
While a moresophisticated strategy (for instance, adjusting the archive size according to concept difficulty orprevalence) is possible with ReCo, we found that this simple approach worked well, and so weadopt it here.
Note that, thanks to mature approximate nearest neighbour search techniques [37], thisarchive construction process can be readily applied to a collection U containing billions of images.
Co-segmentation with seed pixels.
The second hypothesis underpinning our approach is that modernvision-language models such as CLIP are capable of constructing archives of high purity (i.e. a highratio of the archive images contain the concept of interest) provided that such exemplars exist inthe underlying image collection U (our first hypothesis).
Our second hypothesis, which we validatethrough experiments in Sec. 4, enables the use of co-segmentation to obtain concept segments.
Concretely, given the k images comprising the archive for category c constructed by CLIP, we aimto segment regions corresponding to concepts that re-occur across the images.
Since we expectthe archive to be of high purity, and since modern visual backbones provide consistent semanticcorrespondences across images [24], we adopt a strategy of first identifying a seed pixel in eachimage that we are confident belongs to the target category, c .
Intuitively, good seed pixels are onesthat have close neighbours (i.e. strong support) across each of the archive images, since we assumethat: (i) the target concept is common to all images in the archive, (ii) our visual backbone will produce consistent features for pixels belonging to the same concept.
We then use these seed pixelsto construct a reference embedding that can be used to classify pixels belonging to c .
In detail, we first extract dense features for each image in the archive using a pre-trained imageencoder ϕ
I :
[Equation]
where F i ∈ R d × h × w denotes (spatially) dense features for the i th image with height h , width w andd channels.
Each such feature is L2-normalised along its channel dimension.
1 Note that any imageencoder ϕ
I can be employed here (it need not be CLIP).
We identify seed pixels in four steps: First, we construct an adjacency matrix
A khw × khw amongall features in the archive.
Here, each of the k × k submatrices, A ij ∈ R hw × hw , encodes pairwisesimilarities between features from image i and image j for i, j ∈ { 1 , ..., k } .
Second, we aim toidentify, for each pixel in the archive, the similarity of its nearest neighbour among each of the kimages.
To do so, we apply a max operator along the columns of each submatrix (reducing the hwcolums of each submatrix to 1 and reducing the overall adjacency matrix dimensions to khw × k ).
Third, we aim to identify the average support that each pixel has among the k archive images.
For this,we apply a mean operator over the columns of each submatrix such that each row of the resultingkhw × 1 matrix encodes the mean maximum similarity across k images.
Finally, we identify theseed pixel locations by applying an argmax operator to each of the k submatrices of size hw × 1 ,yielding the spatial indices of the features in each image with highest average maximum similarityacross the archive.
To construct a classifier for concept c , we simply average the embeddings of the k seed pixels fromits archive and L2-normalise the resultant vector to produce the reference embedding f c ∈ R d .
Inference.
To localise instances of the category in a new image x new , we first compute the dotproduct between f c and the L2-normalised dense features F new from the new image, and pass theresult through a sigmoid:
[Equation]
where P c new denotes an initial estimate of the probability map corresponding to category c .
To refine this probability map, we draw inspiration from recent work [95, 71] showing that densevisual CLIP features can be usefully correlated against a given CLIP text embedding.
Concretely,we employ the DenseCLIP mechanism of Zhou et al.
[95] to highlight regions of the input that aresalient for the target category c as follows.
For image x new , we extract features V new ∈ R e v × h × wfrom the last self-attention layer values of the CLIP image encoder (here e v denotes the value featuredimension), project the features into the joint space R e with the CLIP image encoder’s final linearprojection and L2-normalise the result.
We then compute the CLIP text embedding ψ T ( c ) ∈ R efor the target concept c and L2-normalise it before producing a saliency map S c new ∈ [0 , 1] h × w byapplying the text embedding as a 1 × 1 convolution to the visual features and applying a sigmoidactivation function to the result.
Note that our use of a sigmoid activation function differs from thesoftmax used by [95], since we process each concept independently.
Our final probability map for category c is produced by the Hadamard product of these estimates:
[Equation]
In case of multiple categories predictions, we concatenate all the category prediction maps and applyargmax to the category dimension.
As a simple post-processing step, we also experiment withthe effect of applying a CRF [44] (similarly to [24]).
Pseudocode for ReCo can be found in thesupplementary material.
3.2 Language-guided co-segmentation and context elimination
Even when an archive has high purity, co-segmenting the target concept c from the collection ofimages can be challenging due to the potential presence of distractor categories ˜ c that often co-occurwith c .
For example, cars often co-occur with roads and likewise aeroplanes with sky .
To minimisethe risk of the co-segmentation algorithm anchoring on unintended categories, we introduce twomechanisms into the co-segmentation procedure, namely, language-guided co-segmentation andcontext elimination .
Language-guided co
segmentation.
Specifically, for each x i in the archive, we first compute thecorresponding saliency map S ci ∈ [0 , 1] h × w using DenseCLIP.
We then vectorise this map anduse it to filter the co-segmentation similarities by replacing each row of A j,i with the Hadamardproduct of itself with vec( S ci ) ∈ [0 , 1] 1 × hw (where vec( · ) denotes vectorisation) for each submatrixj = { 1 , ..., k } .
Intuitively, this serves to act as a gate that preserves similarities only for pixelsidentified by CLIP as salient for the target concept.
Context elimination.
To reduce the influence of widely co-occurring distractors, we select a fewcommon background categories ˜ c that frequently appear in images ( e.g., tree, sky, road ) and computetheir reference embeddings f ˜ c as described in the seeding procedure of Sec. 3.1.
We then use theresulting attention maps P ˜ c
i produced by these reference embeddings to suppress regions of commonbackground context in the subsequent co-segmentation processes for different categories.
This isdone similarly as above, by vectorising and replacing each row of A j,i with the Hadamard productof itself with vec(1 − P ˜ ci ) ∈ [0 , 1] 1 × hw for each submatrix j = { 1 , ..., k } .
If a target concept c isidentical to one of the common background categories ˜ c , we replace f c with the corresponding f ˜ c . 3.3 ReCo
The ReCo framework outlined above requires no training and no access to the target distribution.
However, it is possible to consider a simple extension to this idea when access to the target imagedistribution is available.
Our proposed extension, ReCo+, simply trains a segmentation architecture( e.g., DeepLabv3 + [10]) with the segmentation masks from ReCo as pseudo-masks.
Experiments
In this section, we start by describing the datasets used for our experiments (Sec. 4.1) and implementation details (Sec. 4.2).
Then, we conduct an ablation study (Sec. 4.3) and compare our model tostate-of-the-art methods for unsupervised semantic segmentation with and without language-imagepre-training (Sec. 4.4).
Finally, we showcase our model’s ability to segment rare-category objects(Sec. 4.5) and discuss limitations (Sec. 4.6).
4.1 Datasets
For our ablation study, we use the ImageNet1K [16] validation set to curate archive for conceptsof interest.
The dataset covers 1K classes with 50 images for each class.
To measure segmentationperformance in the zero-shot transfer setting, we use the PASCAL-Context [57] validation set forevaluation, which has 5,104 images of 59 categories excluding the background class.
To compare with previous unsupervised segmentation methods, we use the ImageNet1K training setto construct the reference embedding for the concept we wish to segment.
The dataset has 1K classeswith 1.2M images.
We evaluate on standard benchmarks including the Cityscapes [14] validationsplit, which has 500 urban scene images of 27 categories, KITTI-STEP [86] validation set, whichis composed of 2,981 urban scene images of 19 categories, and COCO-Stuff [6] validation split,which has 4,172 images of 171 low-level thing and stuff categories excluding background class.
Following [12, 24], we use the 27 mid-level categories for evaluation.
For unsupervised adaptationwith ReCo+ (Sec. 3.1), we train on ReCo pseudo-labels on the Cityscapes training set with 2,975images, KITTI-STEP training set which contains 5,027 images, and the COCO-Stuff10K subsetwhich has 9,000 images for each respective benchmark.
We emphasize that no ground-truth labelsare used for training.
Finally, to demonstrate our model’s ability to segment rare concepts, we use the LAION-5Bdataset [74] with 5 billion images as a large collection of images that we expect to satisfy ourfirst hypothesis, namely that it will have coverage of rare concepts. To assess performance, we use
[Figure]
the FireNet dataset [61] which has 1,452 images spanning rare fire safety-related classes.
For ourexperiment, we select the fire extinguisher class as an example of a concept that is important butrare in vision datasets (it is not contained in ImageNet1K [73], for example) and evaluate ReCo on263 images containing at least one instance of the category.
As a further proof of concept, we alsodemonstrate co-segmentations of the Antikythera mechanism (a historical item that does not appearin WordNet [56], or any labelled vision datasets that we are aware of).
4.2 Implementation details
Here, we describe the hyperparameters used to train ReCo+, inference details and evaluation metrics.
Our implementation is based on the PyTorch library [65] and will be made publicly available.
ReCo+ Training.
While ReCo does not require training, we train ReCo+ based on the
DeepLabv3 + [10] segmentation architecture with a ResNet101 [26] backone on the predictionsfrom ReCo as described in 3.1.
All training images are resized and center-cropped to
× 320 pixelsand data augmentations such as random scaling, cropping, and horizontal flipping are applied withrandom color jittering and Gaussian blurring.
We use the Adam optimiser [43] with an initial learningrate of 5 × 10 − 4 and a weight decay of 2 × 10 − 4 with the Poly learning rate schedule as in [50, 10].
Training consists of 20K gradient iterations with a batch size of 8 and takes about 5 hours on a single24GB NVIDIA P40 GPU.
Inference.
For each benchmark, we pre-compute reference image embeddings for a list of categoriesfor the benchmark and store the embeddings to form a classifier.
Whenever DenseCLIP is employed,we use the ResNet50x16 model (following [95]) to construct a saliency map for each image.
Unlessotherwise stated, for the COCO-Stuff and Cityscapes benchmarks, we resize and center crop theinput images to 320 × 320 pixels as in [24].
For the KITTI-STEP validation set, we use the originalresolution of each image as in [41].
For the FireNet benchmark, we resize the shorter side of imagesto 512 pixels and predict a single class of fire extinguisher by thresholding the predicted heatmapwith probability of 0.5.
Evaluation metrics.
Following the common practice [34, 12, 24], we report pixel accuracy (Acc.)and mean intersection-over-union (mIoU).
4.3 Ablation studies
Ability of CLIP to curate archives.
We begin by assessing the validity of our second hypothesis—namely that CLIP is capable of achieving high purity archives from unlabelled images.
To this end,we evaluate the retrieval performance of different CLIP models on the ImageNet1K validation setwhen constructing different archive sizes.
In detail, for each archive size, k , we compute the precisionof the top- k retrieved images based on whether the the ground-truth image-labels match the querytext.
As can be seen in Fig.
3 (left), all CLIP models achieve solid retrieval performance, suggestingtheir potential for curating high purity archives as part of ReCo.
Since ViT-L/14@336px performsbest, we employ this as our retrieval model in the remaining experiments.
DenseCLIP LGC CE CRF Acc.
mIoU ✗ ✗ ✗ ✗ 16.
5.7 ✓ ✗ ✗ ✗ 41.
21.8 ✓ ✓ ✗ ✗ 43.
23.1 ✓ ✗ ✓ ✗ 49. 26.0 ✓ ✓ ✓ ✗ 50.
26.6 ✓ ✓ ✓ ✓ 51.
[Table]
[Table]
Influence of archive size and visual encoder used for co-segmentation.
In Fig.
3 (right) we illustratethe effect of using different pre-trained architectures, e.g. MoCov2 [25], DINO [7], CLIP [70], DeiTSIN [59], as the archive size (and thus the number of images used for co-segmentation) changes.
They-axis depicts segmentation performance for ReCo with these configurations on the PASCAL-Contextbenchmark.
We observe that using larger archives tends to improve performance (likely due to theirreasonably high purity) albeit non-monotonically, and that features from DeiT-SIN perform best.
Wetherefore use DeiT-SIN with an archive size of k =50 for the remaining experiments otherwise stated.
Influence of ReCo framework components.
We next assess the effect of employing DenseCLIPduring inference, the language-guided co-segmentation and context elimination components of ReCowhich seek to improve the quality of co-segmentation achieved across each archive to boost downstream segmentation performance.
When applying context elimination, we select tree, sky, building,road , and person as common background concepts appearing in natural images to be suppressed.
In Tab. 1, we show the effect of three of these strategies, together with the effect of applying a CRF [44] as post-processing.
We observe that integrating DenseCLIP into the inference procedurebrings a significant gain in performance which we believe is driven by the notable robustness of CLIPfeatures under zero-shot transfer [70]. In addition, language-guided co-segmentation and contextelimination further boost co-segmentation performance, while the CRF brings a small gain.
Wetherefore use each of these strategies (including CRF post-processing) in the remaining experiments.
4.4 Comparison to state-of-the-art unsupervised methods
We compare ReCo and ReCo+ to state-of-the-art unsupervised semantic segmentation modelswith and without visual-language pre-training on standard benchmarks, including COCO-Stuff [6], Cityscapes [14] and KITTI-STEP [86] under both zero-shot transfer and unsupervised adaptation(training without labels on the target distribution).
For COCO-Stuff, we observe that the mid-levelcategories used for evaluation are somewhat abstract for retrieval (for instance, one mid-level categoryis “outdoor objects”, which may include many low-level categories beyond the target hierarchy).
Toavoid introducing ambiguity to the co-segmentation procedure, we instead directly use the low-levelcategories and then merge the predictions into the mid-level categories.
Additionally, we rephrasetwo category names to reduce ambiguity ( parking to parking lot and vegetation to tree ) in Cityscapesand KITTI-STEP based on the descriptions found in [14].
A detailed discussion can be found in thesupplementary material.
As shown in Tab. 2, ReCo strongly outperforms prior models on all benchmarks for zero-shot transfer.
Under an unsupervised adaptation protocol, ReCo+ outperforms the state-of-the-art by a large marginon the Cityscapes and KITTI-STEP.
On COCO-Stuff, ReCo+ achieves slightly lower pixel accuracycompared to [24] but a considerably higher mIoU. In Fig.
1, we visualise the sample predictions ofour models on the COCO-Stuff.
4.5 Segmenting rare concepts
By virtue of inheriting CLIP’s diverse knowledge of nameable visual concepts, ReCo exhibits theability to segment rare categories.
We first demonstrate this ability for fire extinguisher objects,which have important fire-safety implications but seldom appear in popular semantic segmentationbenchmarks.
To assess performance, we evaluate segmentation quality on the FireNet dataset (asdescribed in Sec. 4.1) and achieve reasonable performance on pixel accuracy (93.3) and IoU (44.9)
[Table]
shot transfer and unsupervised adaptation protocols, respectively metrics.
In Fig.
4 (left) we visualise the co-segmentation produced by ReCo across sample imagescontainer fire extinguishers.
As an additional demonstration (Fig. 4, right), we also show cosegmentations for images containing the views of the (unique) Antikythera mechanism .
In both cases,we observe that ReCo is capable of co-segmenting the concept of interest without labelled examples.
4.6 Limitations
Our work has several limitations: (1) There may be cases for which our first hypothesis does nothold—concepts so rare that they do not appear in billion-image scale datasets.
In such cases, e.g. “apurple elephant with square orange feet wearing an inverted cowboy hat in front of the Doge’s
Palace”, ReCo will struggle.
(2) During inference, we make use of a visual encoder and a pre-trainedlanguage-image model, i.e., CLIP (a computationally heavy model).
Future work could address thiswith training a light student network which distills knowledge from both models. (3) We employ
ImageNet as the unlabelled image collection for our primary investigations, which is known to exhibitan object-centric bias.
While qualitative results suggest that ReCo can learn from extremely diversedata [74], a more comprehensive empirical evaluation would strengthen this result. (4) Models like CLIP are expensive to train and are consequently updated infrequently.
If a new concept emerges(e.g. a new product) after CLIP was trained, ReCo will be unable to build an archive to enableco-segmentation for this concept.
Although our work is zero-shot in the sense that it is not trainedon the target distribution, we still use of a subset of the target data distribution (primarily throughablations on PASCAL-Context) to guide the development of our method.
As a result, our designlikely contains choices that subtly favour performance on the evaluation tasks.
Conclusion
In this work, we introduced the ReCo framework for semantic segmentation zero-shot transfer.
Bydrawing on the strengths of large-scale language-image pre-training and modern visual backbones,
ReCo attains the ability to segment rare concepts and to directly assign names to concepts withoutlabelled examples from the target distribution.
In addition, experimental comparisons demonstratethat ReCo strongly outperforms existing zero-shot transfer approaches that forgo pixel supervision.
Checklist
For all authors...(
a) Do the main claims made in the abstract and introduction accurately
reflect
the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 4.6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
We will
leave a discussion about potential negative societal impacts in the supp.
mat.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[No] We do
not provide code to reproduce the experiments.
However, we do provide detailed
descriptions of experimental settings and pseudo-code.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[No] ReCo involves no training (so we simply run inference).
For ReCo+ we experiments with a single seed.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] We will leave a discussion about this
in the supp.
mat.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We will provide a url to the code used for this work.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] We will leave a discussion about this in the supp.
mat.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] We will leave a discussion about this in the
supp.
mat.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]