Self-Supervised Visual Representation Learning with Semantic Grouping
Abstract
In this paper, we tackle the problem of learning visual representations from unlabeled scene-centric data.
Existing works have demonstrated the potential of utilizing the underlying complex structure within scene-centric data; still, they commonly rely on hand-crafted objectness priors or specialized pretext tasks to build a learning framework, which may harm generalizability.
Instead, we propose contrastive learning from data-driven semantic slots, namely SlotCon, for joint semantic grouping and representation learning.
The semantic grouping is performed by assigning pixels to a set of learnable prototypes, which can adapt to each sample by attentive pooling over the feature and form new slots.
Based on the learned data-dependent slots, a contrastive objective is employed for representation learning, which enhances the discriminability of features, and conversely facilitates grouping semantically coherent pixels together.
Compared with previous efforts, by simultaneously optimizing the two coupled objectives of semantic grouping and contrastive learning, our approach bypasses the disadvantages of hand-crafted priors and is able to learn object/group-level representations from scene-centric images.
Experiments show our approach effectively decomposes complex scenes into semantic groups for feature learning and significantly benefits downstream tasks, including object detection, instance segmentation, and semantic segmentation.
Code is available at: https://github.com/CVMI-Lab/SlotCon .
Introduction
Existing self-supervised approaches have demonstrated that visual representations can be learned from unlabeled data by constructing pretexts such as transformation prediction [22], instance discrimination [72, 32], and masked image modeling [2, 31, 69], etc .
Among them, approaches based on instance discrimination [29, 6, 7, 11] which treat each image as a single class and employ a contrastive learning objective for training, have attained remarkable success and are beneficial to boost performance on many downstream tasks.
However, this success is largely built upon the well-curated object-centric dataset ImageNet [15], which has a large gap with the real-world data for downstream applications, such as city scenes [13] or crowd scenes [48].
Directly applying the instance discrimination pretext to these real-world data by simply treating the scene as a whole overlooks its intrinsic structures ( e.g ., multiple objects and complex layouts) and thus will limit the potential of pre-training with scene-centric data [63].
This leads to our focus: learning visual representations from unlabeled scene-centric data.
Recent efforts to address this problem can be coarsely categorized into two types of research.
One stream extends the instance discrimination task to pixel level for dense representation learning [56,
[Figure]
68, 75], which shows strong performance in downstream dense prediction tasks.
Yet, these methods still lack the ability to model object-level relationships presented in scene-centric data, which is crucial for learning representations.
Although another stream of works attempts to perform objectlevel representation learning, most of them still heavily rely on domain-specific priors to discover objects, which include saliency estimators [64, 58], unsupervised object proposal algorithms [70, 74], hand-crafted segmentation algorithms [84, 35] or unsupervised clustering [36].
However, if the representation is supervised by hand-crafted objectness priors, it will be discouraged from learning objectness from the data itself and prone to mistakes from priors.
Therefore, the capability and generalizability of the representation will be limited.
In this work, we aim at a fully learnable and data-driven approach to enable learning representations from scene-centric data for enhanced effectiveness, transferability and generalizability.
We propose contrastive learning from data-driven semantic slots, namely SlotCon, for joint semantic grouping and representation learning.
Semantic grouping is formulated as a feature-space pixel-level deep clustering problem where the cluster centers are initialized as a set of learnable semantic prototypes shared by the dataset, and grouping is achieved by assigning pixels to clusters.
The cluster centers can then adapt to each sample by softly assigning pixels into cluster centers and aggregating their features via attentive pooling to form new ones, also called slots .
Further, upon the learned slots from two random views of one image, a contrastive objective, which attempts to pull positive slots ( i.e ., slots from the same prototype and sample) together and push away negative ones, is employed for representation learning.
The optimized representations will enhance the discriminability of prototypes and slots, which conversely facilitates grouping semantically coherent pixels together.
Compared with previous efforts, by simultaneously optimizing the two coupled objectives of semantic grouping and contrastive representation learning, our method bypasses the disadvantages of hand-crafted objectness priors 1 and is able to discover object/group-level representations from scene-centric images.
We extensively assess the representation learning ability of our model by conducting transfer learning evaluation on COCO [48] object detection, instance segmentation, and semantic segmentation on
Cityscapes [13], PASCAL VOC [20], and ADE20K [86].
Our method shows strong results with both COCO pre-training and ImageNet-1K pre-training, bridging the gap between scene-centric and object-centric pre-training.
As a byproduct, our method also achieves notable performance in unsupervised segmentation, showing strong ability in semantic concept discovery.
In summary, our main contributions in this paper are: 1) We show that the decomposition of natural scenes (semantic grouping) can be done in a learnable fashion and jointly optimized with the representations from scratch.
We demonstrate that semantic grouping can bring object-centric representation learning to large-scale real-world scenarios.
Combining semantic grouping and representation learning, we unleash the potential of scene-centric pre-training, largely close its gap with object-centric pre-training and achieve state-of-the-art results in various downstream tasks.
Related work
Our work is in the domain of self-supervised visual representation learning, where the goal is to learn visual representations without human annotations.
We briefly review relevant works below.
Image-level self-supervised learning aims at learning visual representations by treating each image as one data sample.
To this end, a series of pretext tasks are designed in which the labels are readily available without human annotations.
Early explorations range from low-level pixel-wise reconstruction tasks that include denoising [66], inpainting [57], and cross-channel prediction [83] to higher-level instance discrimination [17], rotation prediction [22], context prediction [16], jigsaw puzzle [54], counting [55], and colorization [82].
Modern variants of instance discrimination [17] equipped with contrastive learning [62, 37] have shown strong potential in learning transferable visual representations [72, 8, 32, 60, 87, 85].
Other works differ in their learning objectives but still treat an image as a whole [29, 10, 81].
To further utilize the complex structure in natural images, some works exploit local crops [6, 73, 76, 63] while others either step to pixel- or object-level, detailed as follows.
Pixel-level contrastive learning extends the instance discrimination task from image-level feature vectors to feature maps [56, 49, 68, 75].
Their main differences lie in the way positive pixel-pairs are matched (spatial adjacency [56, 75], feature-space nearest-neighbor [68], sink-horn matching [49]), and the image-level baseline they build upon (MoCo v2 [56, 68], BYOL [49, 75]).
Their pixellevel objective naturally helps learn dense representations that are favorable for dense prediction downstream tasks but lacks the grasp of holistic semantics and commonly require an auxiliary image-level loss to attain stronger performance [68, 75].
Object-level contrastive learning first discovers the objects in images and applies the contrastive objective over them, achieving a good balance in fine-grained structure and holistic semantics, yielding strong empirical gains with both object-centric [35, 36] and scene-centric data [74].
The key issue lies in finding objects in an image without supervision.
Current works, however, still heavily rely on heuristic strategies that include saliency estimators [64, 58], selective-search [70, 74], hand-crafted segmentation algorithms [84, 35], or k-means clustering [36].
In contrast, our semantic grouping method is fully learnable and end to end, ensuring transferability and simplicity.
Unsupervised semantic segmentation is an emerging task that targets addressing semantic segmentation with only unlabeled images.
The first attempt of IIC [41] maximizes the mutual information of augmented image patches, and later works [39, 64] rely on saliency estimators as a prior to bootstrap semantic pixel representations.
Recently, PiCIE [40] adopt pixel-level deep clustering [5] to cluster the pixels into semantic groups, which SegDiscover [38] further improves by adopting super-pixels.
On the other hand, Leopart [88], STEGO [30], and FreeSOLO [67] exploits pre-trained networks’ attention maps for objectness.
Still, they commonly rely on a (self-supervised) pre-trained network for initialization, while our method is trained fully from scratch.
Object-centric representation learning is viewed as an essential component of data-efficient, robust and interpretable machine learning algorithms [28].
Towards unsupervised object-centric representation learning, a series of works have been proposed [27, 3, 19, 50, 25, 24, 23].
Directly extracting objects from images is challenging due to the lack of supervision and thus prior works have long been restricted to synthetic data [43, 53, 1].
Recent works try to step toward real-world videos; however, they either adopt motion [77, 42] or depth [18] as a cue for objectness to solve this problem.
Instead, our method shows that we can first explicitly optimize for desirable properties of clusters (that can describe an object) over the dataset, then retrieve the objects from an image with the learned prototypes.
More importantly, it first shows the possibility of learning object-centric representations from large-scale unlabelled scene-centric natural images.
Method 3.1 Semantic grouping with pixel-level deep clustering
Given a dataset D of unlabeled images, we aim at learning a set of prototypes S that classifies each pixel into a meaningful group, such that pixels within the same group are semantic-consistent (have
[Figure]
Intuitively, a meaningful semantic grouping should be invariant to data augmentations.
Thus, for different augmentations of the same image, we enforce the pixels that lie in the same location to have similar assignment scores w.r.t.
the same set of cluster centers (prototypes).
Besides consistent grouping, the groups should be different from each other to ensure that the learned representations are discriminative and avoid trivial solutions, e.g ., identical features.
Together with standard techniques used in self-supervised learning ( e.g ., non-linear projector and momentum teacher [29, 7], etc .)
, this leads to the following framework.
Specifically, as illustrated in Figure 2, our method contains two neural networks: the student network is parameterized by θ and include an encoder f θ , a projector g θ and a set of K learnable prototypes
S θ = (cid:2) s 1 θ , s 2 θ , . . .
, s Kθ (cid:3) ∈
R K × D ; the teacher network holds the same architecture with the student, but uses a different set of weights ξ that updates as an exponential moving average of θ .
Given an input image x , two random augmentations are applied to produce two augmented views vand v 2 .
Each augmented view v l ∈ { v 1 , v 2 } is then encoded with an encoder f into a hidden feature map h l ∈ R H × W × D , and then transformed with a multilayer perceptron (MLP) g to get z l = g (cid:16) h l (cid:17)
∈
R H
× W × D .
We then compute the assignment P lθ of the projections
z lθ with the corresponding prototypes S θ , and enforce it to match the assignment Q l ′ ξ produced with another view v l ′ by the teacher network.
More precisely, with ℓ 2 -normalized projections z
= z / ∥ z ∥ and prototypes s = s / ∥ s ∥ ,
[Equation]
with τ s , τ t > 0 temperature parameters that control the sharpness of the output distribution of the two networks.
The c can be omitted and will be explained later.
Note that the scale or layout of the two feature maps z lθ and z l ′ ξ can be inconsistent due to geometric image augmentations, e.g ., random crop, scale or flip.
So we perform the inverse augmentation process (including RoIAlign [33] and an optional flipping, details provided in Section A.1) on the predicted assignments to align their spatial locations: (cid:101) Q l ′ ξ = invaug (cid:16) Q l ′ ξ (cid:17) , (cid:101) P lθ = invaug (cid:0) P lθ (cid:1) .
The inverse augmentation is performed on the assignments Q rather than the projections z to keep the context information out of the overlapping area for slot generation, which will be detailed in Section 3.2.
Based on the aligned assignments, we apply the cross-entropy loss L CE θ,ξ (cid:0) q ξ , p
[Equation]
to enforce the consistency in assignment score between spatial-aligned pixels from different views
[Equation]
The cross-entropy loss is averaged over all spatial locations to produce the grouping loss:
[Equation]
Directly optimizing the above objective resembles an unsupervised variant of Mean Teacher [59], which collapses as shown in [29].
In order to avoid collapsing, we follow [7] to maintain a mean logit c ∈ R K and reduce it when producing the teacher assignments Q ξ , as indicated in Eq. 1.
The mean logit stores an exponential moving average of all the logits produced by the teacher network:
[Equation]
where B stands for the batch size.
Intuitively, reducing the mean logit amplifies the difference in assignment between different pixels, preventing all pixels from being assigned to the same prototype.
Besides that, the teacher temperature τ t is smaller than the student temperature τ s to produce a sharper target and avoid uniform assignments.
Both operations help avoid collapse and force the network to learn a meaningful semantic grouping.
Discussion with DINO.
The resulting solution for semantic grouping may seem like a naive extension of DINO [7].
However, this is not the whole picture.
DINO is an image-level representation learning approach that adopts a large number of prototypes ( e.g ., 65536 ), while our objective is built on pixel-level representations and is tailored to learn meaningful semantic groups.
Intuitively,
DINO captures scene semantics that can be more complex and thus requires more prototypes to represent all scene-level variations, while our method learns object-level semantics which can be composed to depict a complicated scene and thus only requires a small number of prototypes ( e.g ., 256 for COCO).
Besides, the representation learning objective of DINO is built on prototypes that are shared by the whole dataset , while our method instead adapts prototypes for each image and performs contrastive learning over groups (detailed in Section 3.2).
In summary, we present a novel view for the decoupling of pixel-level online clustering ( i.e ., semantic grouping) and object-level representation learning.
3.2 Group-level representation learning by contrasting slots
Inspired by Slot Attention [50], we then reuse the assignments computed by the semantic grouping module in Eq.
1 to perform attentive pooling over the dense projections z to produce group-level feature vectors (rephrased as slots ), as shown in Figure 2.
Intuitively, as the softmax normalization applies to the slot dimension, the attention coefficients sum to one for each individual input feature vector.
As a result, the soft assignments A of the dense projections z w.r.t.
the corresponding prototypes S can also be viewed as the attention coefficients that describe how the prototypes decompose the dense projections into non-overlapping groups.
This inspires us to decompose the dense projections with attentive pooling following [50].
Specifically, for a dense projection z lθ produced from view v l , we extract K slots:
[Equation]
where ⊙ denotes the Hadamard product, and a similar operation applies for the teacher network to produce S l ′ ξ .
Since the initial slots are shared by the whole dataset, the corresponding semantic may be missing in a specific view v l , thus producing redundant slots.
Therefore, we compute the following binary indicator 1 l to mask out the slots that fail to occupy a dominating pixel:
[Equation]
i
[Equation]
j such that
[Equation]
and 1 l ′ ξ is computed similarly.
Following the literature in object-level SSL [64, 35, 74, 36], we then apply a contrastive learning objective to discriminate the slot that holds the same semantic across views from distracting slots with the InfoNCE [62] loss:
[Equation]
This objective helps maximize the similarity between different views of the same slot, while minimizing the similarity between slots from another view with different semantics and all slots from other images.
Note that here an additional predictor q θ with the same architecture as the projector g θ is applied to the slots S θ as empirically it yields stronger performance [11, 70, 36].
And the resulting slot-level contrastive loss also follows a symmetric design like Eq. 2:
[Equation]
3.3 The overall optimization objective
We jointly optimize the semantic grouping objective (Eq. 2) and the group-level contrastive learning objective (Eq. 7), controlled with a balancing factor λ g :
[Equation]
At each training step, the student network is optimized with gradients from the overall loss function: θ ← optimizer (cid:16) θ, ∇ θ L Overall θ,ξ , η (cid:17) , where η denotes the learning rate; and the teacher network updates as an exponential moving average of the student network: ξ ← λ t ξ + (1 − λ t ) θ , with λ t denoting the momentum value.
After training, only the teacher encoder f ξ is kept for downstream tasks.
Experiments 4.1 Implementation details
[Table]
COCO+ [48] 241K
[Table]
Pre-training datasets.
We pre-train our models on COCO train2017 [48] and
ImageNet-1K [15], respectively.
COCO train2017 [48] contains ∼ 118K images of diverse scenes with objects of multiple scales, which is closer to real-world scenarios.
In contrast, ImageNet-1K is a curated object-centric dataset containing more than ∼ 1 .
28M images, which is better for evaluating a model’s potential with large-scale data.
Besides, we also explore the limit of scene-centric pre-training on COCO+, i.e ., COCO train set plus the unlabeled2017 set.
See details in Table 1.
Data augmentation.
The image augmentation setting is the same as BYOL [29]: a 224 × 224 -pixel random resized crop with a random horizontal flip, followed by a random color distortion, random grayscale conversion, random Gaussian blur, and solarization.
The crop pairs without overlap are discarded during training.
Network architecture.
We adopt ResNet-50 [34] as the default encoder for f θ and f ξ .
The projector g θ , g ξ and predictor q θ are MLPs whose architecture are identical to that in [7] with a hidden dimension of 4096 and an output dimension of 256 .
Optimization.
We adopt the LARS optimizer [79] to pre-train the model, with a batch size of 512 across eight NVIDIA
Ti GPUs.
Following [75], we utilize the cosine learning rate decay schedule [52] with a base learning rate of 1 . 0 , linearly scaled with the batch size (LearningRate = 1 .
× BatchSize / 256 ), a weight decay of 10 − 5 , and a warm-up period of 5 epochs.
Following [74, 68, 75], the model is pre-trained for 800 epochs on COCO(+) and 100 / 200 epochs on ImageNet, respectively.
Following the common practice of [74, 75], the momentum value λ t of the teacher model starts from 0 .
and is gradually increased to 1 following a cosine schedule.
Synchronized batch normalization and automatic mixed precision are also enabled during training.
Hyper-parameters.
The temperature values τ s and τ t in the student and teacher model are set to 0 .
1 and 0 .
07 , respectively.
Besides, the center momentum λ c is set to 0 .
The default number of prototypes K is set to 256 for COCO(+) and 2048 for ImageNet, according to our empirical finding that setting the number of prototypes close to the number of human-annotated categories can help downstream performance.
The temperature value τ c for the contrastive loss is set to 0 .following
[11], and the default balancing ratio λ g is set to 0 .
4.2 Evaluation protocols
Following the common practice of previous self-supervised works [68, 75, 74], we evaluate the representation ability of the pre-trained model by taking it as the backbone of downstream tasks.
Specifically, we add a newly initialized task-specific head to the pre-trained model for different downstream tasks, i.e ., object detection and instance segmentation on COCO [48], and semantic segmentation on PASCAL VOC [20], Cityscapes [13], and ADE20K [86].
Object detection and instance segmentation.
We train a Mask R-CNN [33] model with R50FPN [47] backbone implemented in Detectron2 [71].
We fine-tune all layers end-to-end on COCO train2017 split with the standard 1 × schedule and report AP, AP 50 , AP 75 on the val2017 split.
Following [68, 75, 74] we train with the standard 1 × schedule with SyncBN.
Semantic segmentation.
The evaluation details of PASCAL VOC and Cityscapes strictly follow [32].
We take our network to initialize the backbone of a fully-convolutional network [51] and fine-tune all the layers end-to-end.
For PASCAL VOC, we fine-tune the model on train_aug set for 30k iterations and report the mean intersection over union (mIoU) on the val2012 set.
For
Cityscapes, we fine-tune on the train_fine set for 90k iterations and evaluate it on the val_fine set.
For ADE20K, we follow the standard 80k iterations schedule of MMSegmentation [12].
Unsupervised semantic segmentation.
We also evaluate the model’s ability of discovering semantic groups in complex scenes, which is accomplished by performing unsupervised semantic segmentation on COCO-Stuff [4].
We follow the common practice in this field [41, 40, 38] to merge the labels into 27 categories ( 15 "stuff" categories and 12 "thing" categories), and evaluate with a subset created by [41].
We perform inference with resolution 320 and number of prototypesfollowing the standard practice.
The predicted labels are matched with the ground truth through
Hungarian matching [45], and evaluated on mIoU and pixel accuracy (pAcc) [Table]
4.3 Transfer learning results
COCO pre-training.
In Table 2, we show the main results with COCO pre-training.
There have been steady improvements in object-level pre-training with COCO, in which the top performance methods are DetCon [46] and ORL [74], which still rely on objectness priors like selective-search [61] or hand-crafted segmentation algorithms [21], yet they still fail to beat the pixel-level state-of-the-art
[Table]
PixPro ‡ [75] ✗ ✗ 40.5 60.
Object / Group-level approaches
DetCon † [35] ✗ ✓ 39.8 59.
ORL † [74] ✓ ✓ 40.3 60.
36.7 Ours (SlotCon) ✗ ✗ 41.0 61.
PixPro [75].
Our method alleviates such limitations and significantly improves over current objectlevel methods in all tasks, achieving consistent improvement over the previous approaches and even several methods that were pre-trained on the larger dataset ImageNet-1K (Table 4).
It is also notable that our method can achieve a better performance on the largest and most challenging dataset for segmentation, ADE20K, adding to the significance of this work.
COCO+ pre-training.
In Table
[Table]
we report the results with COCO+ pre-training.
The COCO+ is
COCO train2017 plus unlabeled2017 set, which roughly doubles the number of training images and greatly adds to the data diversity.
Our method further sees a notable gain in all tasks with extended
COCO+ data [Table] and even shows comparable results with our best-performing model pre-trained on
ImageNet-1K ( 5 × large of COCO+), showing the great potential of scene-centric pre-training
[Table]
[Table]
ImageNet-1K pre
training.
In Table 4, we also benchmark our method with ImageNet-1K pretraining and show its compatibility with object-centric data.
Without selective-search for object proposals and without pre-training and transferring the FPN head, our method still beats most of the current works and largely closes the gap with the detection-specialized method SoCo.
[Table]
PixPro † [75] ✗ ✗ 40.7 60.
Object / Group-level approaches
DetCon [35] ✗ ✓ 40.
SoCo ‡ [70] ✓ ✓ 41.6 61.
37.8 Ours (SlotCon) ✗ ✗ 41.4 61.
38.6 Ours (SlotCon) ✗ ✗ 41.8 62.
4.4 Unsupervised semantic segmentation results
Given our approach’s consistent improvement in representation learning, we further analysis of how well our semantic grouping component can parse scenes quantitatively and qualitatively.
It should be
[Table]
I m a g e
P
i C
I E
O u r s
Ground
Person Building
Plant Sky
Water
Rawmater.
Floor
Wall Furniture
Animal clarified that this is just to help understand the learned prototypes, and not to propose a new SOTA.
Unlike current SSL approaches that exhaustively enumerate the massive object proposals and report the best score [7, 36], we follow the common practice of unsupervised semantic segmentation [41, 40] to match the predicted results with the ground-truth using the Hungarian algorithm [45], where each ground-truth label is assigned to a prototype mutual-exclusively.
For fair comparisons, the model used for evaluation is trained with K = 27 to match the number of categories of COCO-Stuff.
As shown in Table 5, our method can surpass the previous works PiCIE [40] and SegDiscover [38] with 4 points higher mIoU.
Meanwhile, the pAcc is lower since we train the model with a lower resolution ( 7 × 7 v.s. 80 × 80 feature map).
Besides Table 5, we also depict the visualization results, in which our method distinguishes confusing objects apart (4th column) and successfully localizes small objects (5th column).
Since we only need to separate pixels with different semantics within the same image, the errors in category prediction can be ignored.
4.5 Ablation study
[Table]
Number of prototypes K .
Table 6a ablates the number of prototypes, we observe that the most suitable K for COCO detection is 256 , which is close to its real semantic class number 172 (thing + stuff) [4].
Besides, the performance on Cityscapes and PASCAL VOC have a consistent tendency to drop, while the performance on ADE20K is consistently good if K is big enough.
We hypothesize that a suitable K can encourage learning data-specific semantic features, which are only helpful when the pre-training and downstream data are alike (from COCO to COCO); increasing K produces fine-grained features that may lack discriminability in semantics but hold better transferability to
ADE20K that require fine-grained segmentation [14].
Loss balancing weight λ g .
Table 6b ablates the balancing between the semantic grouping loss and the group-level contrastive loss, where the best balance is achieved with both losses treated equally.
It is notable that when λ g = 1 . 0 , only the semantic grouping loss is applied, and the performance drops considerably, indicating the importance of our group-level contrastive loss for learning good representations.
Teacher temperature τ t .
Table 6c ablates the teacher model’s temperature parameter, indicating that a softer teacher distribution with τ t = 0 .
07 helps achieve better performance.
4.6 Probing the prototypes
Finally, we analyze whether the prototypes learn semantic meanings by visualizing their nearest neighbors in COCO val2017 split.
We first perform semantic grouping on each image to split them into non-overlapping groups (segments), then pool each group to a feature vector, and retrieve the top
-neighbor segments for each prototype according to cosine similarity.
As shown in
Figure 3, the prototypes well bind to semantic meanings that cover a wide range of scenarios and semantic granularities from animals, foods, and sports, to furniture, buildings, etc., localizing them well regardless of small object size and occlusion; and notably, without any human annotation .
Discussion on the emergence of objectness
It should be clarified that the discovered objectiveness is at the semantic level, and object instances with identical semantics can be indistinguishable.
Concerning the emergence of objectness, we impose geometric-covariance and photometric-invariance as guiding cues which force the model to decompose a large complex dataset into a small number of clusters through optimizing feature space and cluster centers.
The emergence of objects/parts that are compositional and thus occupy a reasonable number of prototypes, can be viewed as a natural consequence under such conditions.
Concerning granularity, our intuition is that the prototype number and the dataset distribution generate a bottleneck for the granularity of groups.
For example, in the COCO dataset, with 256 prototypes, the model finds that splitting animals into cats, dots, elephants, etc ., is enough and won’t further separate them.
In contrast, for humans (the most occupying category of COCO), as shown in the appendix, the model discovers not only human parts (Figure 10) but also different types of activities (Figure 11), indicating that parts are more helpful in this scenario and deserve more prototypes.
Conclusion
This work presents a unified framework for joint semantic grouping and representation learning from unlabeled scene-centric images.
The semantic grouping is performed by assigning pixels to a set of learnable prototypes, which can adapt to each sample by attentive pooling over the feature map and form new slots.
Based on the learned data-dependent slots, a contrastive objective is employed for representation learning, enhancing features’ discriminability and facilitating the grouping of semantically coherent pixels together.
By simultaneously optimizing the two coupled objectives of semantic grouping and contrastive learning, the proposed approach bypasses the disadvantages of handcrafted priors and can learn object/group-level representations from scene-centric images.
Experiments show the proposed approach effectively decomposes complex scenes into semantic groups for feature learning and significantly facilitates downstream tasks, including object detection, instance segmentation, and semantic segmentation.
Acknowledgments
This work has been supported by Hong Kong Research Grant Council - Early Career Scheme (Grant No. 27209621), HKU Startup Fund, and HKU Seed Fund for Basic Research.
The authors acknowledge SmartMore, LunarAI, and MEGVII for partial computing support.
Appendix A Additional implementation details A.1 Inverse augmentation
The inverse augmentation process aims to recover the original pixel locations and ensure the two feature maps produced from two augmented views are spatially aligned after inverse augmentation.
There are two operations in our data augmentation pipeline that changes the scale or layout of the image, i.e ., random resized crop and random horizontal flip.
Since we already know the spatial coordinates where each view is cropped from, we can map the coordinates to the corresponding feature maps and cut the rectangular part from the feature map where the two sets of coordinates intersect.
This is followed by a resize operation to recover the intersect part to the original size ( e.g ., 7 × 7 for a 224 × 224 input).
In implementation, we achieve this through RoIAlign [33].
If the horizontal flip operation is applied to produce the view, we also use a horizontal flip operation after
RoIAlign to recover the original spatial layout.
After the inverse augmentation, each pixel in the two feature maps is spatial-aligned, making applying the per-pixel cross-entropy loss easy.
A.2 Transfer learning A.2.1 Object detection and instance segmentation
We train a Mask R-CNN [33] model with R50-FPN backbone [47] implemented with the open-source project Detectron2 [71], following the same fine-tuning setup with [68, 75, 74].
Specifically, we use a batch size of 16 , and fine-tune for 90k iterations (standard 1 × schedule) with batch normalization layers synchronized.
The learning rate is initialized as 0 .
02 with a linear warm-up for 1000 iterations, and decayed by 0 .
1 at 60k and 80k iterations.
The image scale is [640 , 800] during training andat inference.
We fine-tune all layers end-to-end on COCO [48] train2017 set with the standard 1 × schedule and report AP, AP 50 , AP 75 on the val2017 set.
A.2.2 Semantic segmentation
Cityscapes and PASCAL VOC.
We strictly follow [32] for transfer learning on these two datasets.
Specifically, we use the same fully-convolutional network (FCN)-based [51] architecture as [32].
The backbone consists of the convolutional layers in ResNet-50, in which the 3 × 3 convolutions in conv blocks have dilation 2 and stride 1 .
This is followed by two extra 3 × 3 convolutions of 256 channels (dilation set to 6 ), with batch normalization and ReLU activations, and then a 1 × 1 convolution for per-pixel classification.
The total stride is 16 (FCN-16s [51]).
Training is performed with random scaling (by a ratio in [0 . 5 , 2 . 0] ), cropping, and horizontal flipping.
The crop size is 513 on PASCAL VOC [20] and 769 on Cityscapes [13], and inference is performed on the original image size.
We train with batch size 16 and weight decay 0 . 0001 .
The learning rate is 0 .
003 on VOC and is 0 .
01 on Cityscapes (multiplied by 0 .
1 at the 70 th and 90 th percentile of training).
For PASCAL VOC, we fine-tune the model on train_aug2012 set for 30k iterations and report the mean intersection over union (mIoU) on the val2012 set.
For Cityscapes, we fine-tune on the train_fine set for 90k iterations and evaluate it on the val_fine set.
ADE20K.
For ADE20K [86], we train with a FCN-8s [51] model on the train set and evaluate on the val set, and the optimization specifics follows the standard 80k iterations schedule of
MMSegmentation [12].
Specifically, we fine-tune for 80k iterations with stochastic gradient descent, with a batch size of 16 and weight decay of 0 . 0005 .
The learning rate is 0 .
01 and decays following the poly schedule with power of 0 .
9 and min_lr of 0 . 0001 .
A.3 Unsupervised semantic segmentation
Experiment setting.
We follow the common practice in this field [41, 40, 38] to use a modified version of COCO-Stuff [4], where the labels are merged into 27 categories ( 15 "stuff" categories and 12 "thing" categories).
We perform inference with resolution 320 and number of prototypesfollowing the common practice, and evaluate on mIoU and pixel accuracy (pAcc).
Inference details.
Intuitively, each prototype can be viewed as the cluster center of a semantic class.
Therefore, we simply adopt the prototypes S ∈ R K × D as a 1 × 1 convolution layer for per-pixel classification, and predict the prototypical correspondence of each pixel with the argmax operation.
[Equation]
where the resize operation denotes bi-linear interpolation on the logits to the size of the image ( 320 × 320 in this case).
To match the prototypes with the ground truth clusters, we follow the standard protocol [41, 40] of finding the best one-to-one permutation mapping using Hungarianmatching [45].
Then the pAcc and mIoU are calculated according to the common practice [40].
During inference, we only take the teacher model parameterized by ξ .
A.4 Visual concept discovery
Simply speaking, the visual concept discovery is similar to the semantic segment retrieval task [63], except that the queries are prototypes rather than segments.
Specifically, we adopt the COCO val2017 set, consisting of 5k images, and the default model trained on COCO with the number of prototypes K = 256 .
Each image is first resized to 256 pixels along the shorter side, after which a 224 × 224 center crop is applied.
We then follow Eq.
9 to assign a prototype index to each pixel; thus, each image is split into a set of groups, such that the pixels within each group hold the same prototypical assignments.
We rephrase the groups as segments , and compute the feature vector for each segment by average pooling.
Then for each prototype, we calculate the cosine similarity between it and all segments in the dataset assigned to this prototype and retrieve those with top- k high similarity scores.
A.5 Re-implementing related works
Some current works may differ in implementation details for downstream tasks ( e.g ., SoCo [70] uses different hyper-parameters for COCO object detection and instance segmentation, DetCon [35] uses different hyper-parameters for semantic segmentation, and DenseCL [68] adopts different network architectures for semantic segmentation).
For a fair comparison, we re-produced the transfer learning results with a unified setting with the official checkpoints and re-implement the pre-training with the official code if needed.
B Additional transfer learning results B.1 Longer COCO schedule
[Table]
In Table 6, we further provide the downstream results of SlotCon in COCO object detection and instance segmentation with a longer transfer learning schedule ( 2 × ).
Compared with the results with the 1 × schedule, it shows significant improvements in all metrics.
B.2 Pre-training with autonomous driving data
In Table 7, we show the results with BDD100K [80] pre-training and evaluated on Cityscapes semantic segmentation.
The model is trained on BDD100K for 800 epochs with 64 prototypes.
The result is notably weaker than its COCO counterpart, yet still surpasses MoCo v2 pre-trained on COCO.
The BDD100K dataset is indeed challenging for pre-training as its images are less discriminative, and the task of pre-training on autonomous driving data is a valuable direction for future explorations.
[Table]
C Additional ablation studies
[Table]
In Table
[Table]
we provide the results of further ablation studies in batch size, the type of group-level contrastive loss, the place to apply inverse augmentation, the influence of an image-level learning objective, and the importance of geometric augmentations.
We discuss them as follows:
Batch size.
Table 8a shows the most suitable batch size for our method is 512 .
Increasing it to 1024 does not result in better performance.
We argue that the real slot-level batch size is actually bigger than 512 , and should be multiplied by the number of pixels ( 49 ) or slots ( ∼ 8 ) per image for the grouping loss and the group-level contrastive loss, respectively.
Considering the mismatch in the batch size scale of the two loss functions, the learning rate might should be further tuned to work with larger batches [26].
Type of group-level loss.
Table 8b shows that both the BYOL [29]-style regression loss and the contrastive loss are helpful to learning transferable features, and the results with the contrastive loss are especially higher for object detection in COCO.
This may indicate that the contrastive loss, which better pushes negative samples apart, is beneficial for object detection, in which the ability to tell confusing objects apart is also critical.
Place to apply inverse augmentation.
Table 8c ablates whether to apply the inverse augmentation operation on the dense projections or the grouping assignments, and shows the latter is better.
This can keep the non-overlapping features for the group-level contrastive loss and utilize more information.
Image-level learning objective.
Table 8d ablates the effect of an auxiliary image-level learning objective.
We add a MoCo v3 [11] style image-level contrastive learning loss to SlotCon (COCOepoch pre-training setting), and the experiment results on COCO show that it depends on the batch size.
With a smaller batch size of 512 , the detection AP drops by 0 .
2 points, while with a higher batch size of 1024 , the detection AP rises by 0 .
4 points.
Our explanation is that the image-level objective is more sensitive to the batch size, and it requires a larger batch size to learn holistic representations that are complementary to the object-level objective.
Geometric augmentations.
Table 8e ablates the effect of the geometric augmentations.
Our main finding is that geometric augmentations are necessary to learn object-centric representations in our setting.
We train SlotCon on COCO for 800 epochs with two identical crops applied for each image, thus only the photometric invariance is adopted as the supervision.
We then visualized the slots the same way as Figure 3, and found that almost none of the slots can bind to a meaningful semantic.
Most of them attend to a similar-shaped region that locates at the same position across different images while holding diverging semantics.
And some of them learn textures like animal fur, cloudy sky, snowland, or leaves.
A fast evaluation on PASCAL VOC semantic segmentation shows a significant performance drop, yet the feature is still notably better than random initialization.
D Statistics about the binary indicator
How many slots are active on average for each image?
It depends on the number of categories/semantics per image.
As shown in Figure 4, seven slots are active on average for one image after convergence.
How often is one slot active over the whole dataset?
It depends on the category/semantic distribution of the dataset, as the slots are roughly bound to realworld semantic categories.
We studied the activeness of the slots over the COCO val2017 set that contains 5000 images, and found that 40 out of the 256 slots are dead, and not active to any image.
The activeness of the remaining slots follows a long-tailed distribution.
The top-5 active slots correspond to tree ( 376 ), sky ( 337 ), streetside car ( 327 ), building exterior wall ( 313 ), and indoor wall ( 307 ); and the bottom-5 active slots correspond to skateboarder ( 44 ), grassland ( 45 ), train ( 56 ), luggage ( 57 ), and airplane ( 57 ).
How many terms are excluded in Eq.
Given the slot number as 256 , if all slots are active for one image and the two crops overlap well, there should be 256 positive pairs.
We studied a well-converged model and found that, on average, only around 3 . 6 pairs are active, so around 252 .terms are excluded.
It is reasonable considering that around 7 slots are active on one image, and the overlapping area between the two crops can be small.
Considering a total batch size of 512 , the number of excluded negative samples should be around 512 × (256 − 7) = 127488 .
E Computational costs
[Table]
In Table 8, we give a direct computational cost comparison between SlotCon and two previous works.
The experiments are conducted on the same machine with 8 NVIDIA GeForce RTX 3090 GPUs.
Both PixPro and SlotCon adopt a batch size of 1024 and have amp turned on, and
DenseCL adopts a batch size of 256 by default.
The training time of DenseCL might be higher than optimal as we failed to install apex.
F Additional qualitative results
Unsupervised semantic segmentation.
In Figure 5, 6, 7, we provide the visualization of more results in COCO-Stuff unsupervised semantic segmentation.
Compared with PiCIE [40], our method’s overall successes in distinguishing confusing objects apart and localizing small objects.
Visual concept discovery on COCO.
In Figure 8, 9, we show more results of visual concepts discovered by our model from COCO, which cover a wide range of natural scenes.
We further show that the model tends to categorize person-related concepts into fine-grained clusters.
For example, in Figure 10, we show that it also groups segments according to the part of the human body; and in
Figure 11, we show that it also groups person segments by the sport they are playing.
We hypothesize that persons are too common in COCO, and the model finds that allocating more prototypes to learn person-related concepts can better help optimize the grouping loss.
Visual concept discovery on ImageNet.
In Figure 12, 13, we also provide examples of visual concepts discovered by our method from ImageNet.
Due to the scale of ImageNet, it is hard to compute the segments for all the images.
As ImageNet is basically single-object-centric, we simply treat each image as a single segment to save computation for nearest-neighbor searching.
The visualization verifies the compatibility of our method with object-centric data.
G Limitations and negative social impacts
Grouping precision.
Since we directly learn a set of semantic prototypes with a quite low-resolution feature map ( 32 × downsample) and do not have any supervision for precise object boundaries, it is hard for our model to perform detailed semantic grouping and cases are that many foreground instances are segmented with over-confidence.
Using post-processing through iterative refinements such as CRF [44] or pre-compute visual primitives (super-pixels) on the raw image [38] may improve the result, but they are out of the scope of this work.
Besides, modern object discovery techniques such as Slot Attention [50] that incorporate attention mechanism and iterative refinement may also help learn better semantic groups; we leave this for future work.
Training cost.
As all self-supervised learning methods do, our approach also needs to pre-train with multiple GPU devices for a long time, which may increase carbon emissions.
However, for one thing, the pre-training only needs to be done once and can help reduce the training time of multiple downstream tasks; for another, our method can learn relatively good representations with shorter training time, e.g ., our method pre-trained on ImageNet for 100 epochs achieves compatible performance with PixPro [75] pre-trained for 400 epochs in COCO objection detection (AP b = 41 . 4 ).
The data can be unreliable.
With reduced human priors, our method learns to discover objects/semantics form large-scale natural images.
However, totally relying on the data may lead to "bad" biases.
In our experiments on scene-centric data, we noticed that the model allocates more prototypes to human-related concepts (as humans occur most frequently in COCO), while many other kinds of animals only have one prototype (see Figure 9, 11, 10).
When pre-training on a more long-tailed and less discriminative scenario ( e.g ., autonomous driving data, detailed in Sec B.2), the data can lead to highly biased prototypes and representations, and harm downstream performance.
Injecting human priors to guide the criterion for objects/semantics could be a promising direction.
Semantic rather than object.
Our method performs semantic grouping; this means that objects with identical semantics can be indistinguishable.
Therefore, the contrastive learning objective (Eq. 7) does not contrast between objects in the same image with the same semantic ( e.g ., two elephants in one image).
However, the success of self-supervised learning has shown that discriminating instances with identical category labels can further boost representation quality.
The transition from semantic to objectness is to be explored.
H License of used datasets
All the datasets used in this paper are permitted for research use.
The terms of access to the images of
COCO [48] and ImageNet [15] allow the use for non-commercial research and educational purposes.
Besides, the annotations of COCO [48] and COCO-Stuff [4] follow the Creative Commons Attribution 4.0 License, also allowing for research purposes.
[Figure]
[Figure]
simply treat to is hard to compute
ImageNet basically simply treat as a segment to save computation for nearest