Self-Supervised Visual Representation Learning with Semantic Grouping
Abstract
In this paper, we tackle the problem of learning visual representations from unlabeled scene-centric data.
Existing works have demonstrated the potential of utilizing the underlying complex structure within scene-centric data; still, they commonly rely on hand-crafted objectness priors or specialized pretext tasks to build a learning framework, which may harm generalizability.
Instead, we propose contrastive learning from data-driven semantic slots, namely SlotCon, for joint semantic grouping and representation learning.
The semantic grouping is performed by assigning pixels to a set of learnable prototypes, which can adapt to each sample by attentive pooling over the feature and form new slots.
Based on the learned data-dependent slots, a contrastive objective is employed for representation learning, which enhances the discriminability of features, and conversely facilitates grouping semantically coherent pixels together.
Compared with previous efforts, by simultaneously optimizing the two coupled objectives of semantic grouping and contrastive learning, our approach bypasses the disadvantages of hand-crafted priors and is able to learn object/group-level representations from scene-centric images.
Experiments show our approach effectively decomposes complex scenes into semantic groups for feature learning and significantly benefits downstream tasks, including object detection, instance segmentation, and semantic segmentation.
Code is available at: https://github.com/CVMI-Lab/SlotCon .
Introduction
Existing self-supervised approaches have demonstrated that visual representations can be learned from unlabeled data by constructing pretexts such as transformation prediction [22], instance discrimination [70, 31], and masked image modeling [2, 30, 67], etc .
Among them, approaches based on instance discrimination [28, 6, 7, 11] which treat each image as a single class and employ a contrastive learning objective for training, have attained remarkable success and are beneficial to boost performance on many downstream tasks.
However, this success is largely built upon the well-curated object-centric dataset ImageNet [15], which has a large gap with the real-world data for downstream applications, such as city scenes [13] or crowd scenes [46].
Directly applying the instance discrimination pretext to these real-world data by simply treating the scene as a whole overlooks its intrinsic structures ( e.g ., multiple objects and complex layouts) and thus will limit the potential of pre-training with scene-centric data [61].
This leads to our focus: learning visual representations from unlabeled scene-centric data.
Recent efforts to address this problem can be coarsely categorized into two types of research.
One stream extends the instance discrimination task to pixel level for dense representation learning [54,
[Figure]
66, 73], which shows strong performance in downstream dense prediction tasks.
Yet, these methods still lack the ability to model object-level relationships presented in scene-centric data, which is crucial for learning representations.
Although another stream of works attempts to perform objectlevel representation learning, most of them still heavily rely on domain-specific priors to discover objects, which include saliency estimators [62, 56], unsupervised object proposal algorithms [68, 72], hand-crafted segmentation algorithms [81, 34] or unsupervised clustering [35].
However, if the representation is supervised by hand-crafted objectness priors, it will be discouraged from learning objectness from the data itself and prone to mistakes from priors.
Therefore, the capability and generalizability of the representation will be limited.
In this work, we aim at a fully learnable and data-driven approach to enable learning representations from scene-centric data for enhanced effectiveness, transferability and generalizability.
We propose contrastive learning from data-driven semantic slots, namely SlotCon, for joint semantic grouping and representation learning.
Semantic grouping is formulated as a feature-space pixel-level deep clustering problem where the cluster centers are initialized as a set of learnable semantic prototypes shared by the dataset, and grouping is achieved by assigning pixels to clusters.
The cluster centers can then adapt to each sample by softly assigning pixels into cluster centers and aggregating their features via attentive pooling to form new ones, also called slots .
Further, upon the learned slots from two random views of one image, a contrastive objective, which attempts to pull positive slots ( i.e ., slots from the same prototype and sample) together and push away negative ones, is employed for representation learning.
The optimized representations will enhance the discriminability of prototypes and slots, which conversely facilitates grouping semantically coherent pixels together.
Compared with previous efforts, by simultaneously optimizing the two coupled objectives of semantic grouping and contrastive representation learning, our method bypasses the disadvantages of hand-crafted objectness priors 1 and is able to discover object/group-level representations from scene-centric images.
We extensively assess the representation learning ability of our model by conducting transfer learning evaluation on COCO [46] object detection, instance segmentation, and semantic segmentation on
Cityscapes [13], PASCAL VOC [20], and ADE20K [83].
Our method shows strong results with both COCO pre-training and ImageNet-1K pre-training, bridging the gap between scene-centric and object-centric pre-training.
As a byproduct, our method also achieves notable performance in unsupervised segmentation, showing strong ability in semantic concept discovery.
In summary, our main contributions in this paper are: 1) We show that the decomposition of natural scenes (semantic grouping) can be done in a learnable fashion and jointly optimized with the representations from scratch.
We demonstrate that semantic grouping can bring object-centric representation learning to large-scale real-world scenarios.
Combining semantic grouping and representation learning, we unleash the potential of scene-centric pre-training, largely close its gap with object-centric pre-training and achieve state-of-the-art results in various downstream tasks.
Related work
Our work is in the domain of self-supervised visual representation learning, where the goal is to learn visual representations without human annotations.
We briefly review relevant works below.
Image-level self-supervised learning aims at learning visual representations by treating each image as one data sample.
To this end, a series of pretext tasks are designed in which the labels are readily available without human annotations.
Early explorations range from low-level pixel-wise reconstruction tasks that include denoising [64], inpainting [55], and cross-channel prediction [80] to higher-level instance discrimination [17], rotation prediction [22], context prediction [16], jigsaw puzzle [52], counting [53], and colorization [79].
Modern variants of instance discrimination [17] equipped with contrastive learning [60, 36] have shown strong potential in learning transferable visual representations [70, 8, 31, 58, 84, 82].
Other works differ in their learning objectives but still treat an image as a whole [28, 10, 78].
To further utilize the complex structure in natural images, some works exploit local crops [6, 71, 74, 61] while others either step to pixel- or object-level, detailed as follows.
Pixel-level contrastive learning extends the instance discrimination task from image-level feature vectors to feature maps [54, 47, 66, 73].
Their main differences lie in the way positive pixel-pairs are matched (spatial adjacency [54, 73], feature-space nearest-neighbor [66], sink-horn matching [47]), and the image-level baseline they build upon (MoCo v2 [54, 66], BYOL [47, 73]).
Their pixellevel objective naturally helps learn dense representations that are favorable for dense prediction downstream tasks but lacks the grasp of holistic semantics and commonly require an auxiliary image-level loss to attain stronger performance [66, 73].
Object-level contrastive learning first discovers the objects in images and applies the contrastive objective over them, achieving a good balance in fine-grained structure and holistic semantics, yielding strong empirical gains with both object-centric [34, 35] and scene-centric data [72].
The key issue lies in finding objects in an image without supervision.
Current works, however, still heavily rely on heuristic strategies that include saliency estimators [62, 56], selective-search [68, 72], hand-crafted segmentation algorithms [81, 34], or k-means clustering [35].
In contrast, our semantic grouping method is fully learnable and end to end, ensuring transferability and simplicity.
Unsupervised semantic segmentation is an emerging task that targets addressing semantic segmentation with only unlabeled images.
The first attempt of IIC [40] maximizes the mutual information of augmented image patches, and later works [38, 62] rely on saliency estimators as a prior to bootstrap semantic pixel representations.
Recently, PiCIE [39] adopt pixel-level deep clustering [5] to cluster the pixels into semantic groups, which SegDiscover [37] further improves by adopting super-pixels.
On the other hand, Leopart [85], STEGO [29], and FreeSOLO [65] exploits pre-trained networks’ attention maps for objectness.
Still, they commonly rely on a (self-supervised) pre-trained network for initialization, while our method is trained fully from scratch.
Object-centric representation learning is viewed as an essential component of data-efficient, robust and interpretable machine learning algorithms [27].
Towards unsupervised object-centric representation learning, a series of works have been proposed [26, 3, 19, 48, 25, 24, 23].
Directly extracting objects from images is challenging due to the lack of supervision and thus prior works have long been restricted to synthetic data [42, 51, 1].
Recent works try to step toward real-world videos; however, they either adopt motion [75, 41] or depth [18] as a cue for objectness to solve this problem.
Instead, our method shows that we can first explicitly optimize for desirable properties of clusters (that can describe an object) over the dataset, then retrieve the objects from an image with the learned prototypes.
More importantly, it first shows the possibility of learning object-centric representations from large-scale unlabelled scene-centric natural images.
Method 3.1 Semantic grouping with pixel-level deep clustering
Given a dataset D of unlabeled images, we aim at learning a set of prototypes S that classifies each pixel into a meaningful group, such that pixels within the same group are semantic-consistent (have
[Figure]
Intuitively, a meaningful semantic grouping should be invariant to data augmentations.
Thus, for different augmentations of the same image, we enforce the pixels that lie in the same location to have similar assignment scores w.r.t.
the same set of cluster centers (prototypes).
Besides consistent grouping, the groups should be different from each other to ensure that the learned representations are discriminative and avoid trivial solutions, e.g ., identical features.
Together with standard techniques used in self-supervised learning ( e.g ., non-linear projector and momentum teacher [28, 7], etc .)
, this leads to the following framework.
Specifically, as illustrated in Figure 2, our method contains two neural networks: the student network is parameterized by θ and include an encoder f θ , a projector g θ and a set of K learnable prototypes
S θ = (cid:2) s 1 θ , s 2 θ , . . .
, s Kθ (cid:3) ∈
R K × D ; the teacher network holds the same architecture with the student, but uses a different set of weights ξ that updates as an exponential moving average of θ .
Given an input image x , two random augmentations are applied to produce two augmented views vand v 2 .
Each augmented view v l ∈ { v 1 , v 2 } is then encoded with an encoder f into a hidden feature map h l ∈ R H × W × D , and then transformed with a multilayer perceptron (MLP) g to get z l = g (cid:16) h l (cid:17)
∈
R H
× W × D .
We then compute the assignment P lθ of the projections
z lθ with the corresponding prototypes S θ , and enforce it to match the assignment Q l ′ ξ produced with another view v l ′ by the teacher network.
More precisely, with ℓ 2 -normalized projections z
= z / ∥ z ∥ and prototypes s = s / ∥ s ∥ ,
[Equation]
with τ s , τ t > 0 temperature parameters that control the sharpness of the output distribution of the two networks.
The c can be omitted and will be explained later.
Note that the scale or layout of the two feature maps z lθ and z l ′ ξ can be inconsistent due to geometric image augmentations, e.g ., random crop, scale or flip.
So we perform the inverse augmentation process (including RoIAlign [32] and an optional flipping, details provided in Section A.1 in the appendix) on the predicted assignments to align their spatial locations: (cid:101) Q l ′ ξ = invaug (cid:16) Q l ′ ξ (cid:17) , (cid:101) P lθ = invaug (cid:0) P lθ (cid:1) .
The inverse augmentation is performed on the assignments Q rather than the projections z to keep the context information out of the overlapping area for slot generation, which will be detailed in Section 3.2.
Based on the aligned assignments, we apply the cross-entropy loss L CE θ,ξ (cid:0) q
[Equation]
to enforce the consistency in assignment score between spatial-aligned pixels from different views.
The cross-entropy loss is averaged over all spatial locations to produce the grouping loss:
[Equation]
Directly optimizing the above objective resembles an unsupervised variant of Mean Teacher [57], which collapses as shown in [28].
In order to avoid collapsing, we follow [7] to maintain a mean logit c ∈ R K and reduce it when producing the teacher assignments Q ξ , as indicated in Eq. 1.
The mean logit stores an exponential moving average of all the logits produced by the teacher network:
[Equation]
where B stands for the batch size.
Intuitively, reducing the mean logit amplifies the difference in assignment between different pixels, preventing all pixels from being assigned to the same prototype.
Besides that, the teacher temperature τ t is smaller than the student temperature τ s to produce a sharper target and avoid uniform assignments.
Both operations help avoid collapse and force the network to learn a meaningful semantic grouping.
Discussion with DINO.
The resulting solution for semantic grouping may seem like a naive extension of DINO [7].
However, this is not the whole picture.
DINO is an image-level representation learning approach that adopts a large number of prototypes ( e.g ., 65536 ), while our objective is built on pixel-level representations and is tailored to learn meaningful semantic groups.
Intuitively,
DINO captures scene semantics that can be more complex and thus requires more prototypes to represent all scene-level variations, while our method learns object-level semantics which can be composed to depict a complicated scene and thus only requires a small number of prototypes ( e.g ., 256 for COCO).
Besides, the representation learning objective of DINO is built on prototypes that are shared by the whole dataset , while our method instead adapts prototypes for each image and performs contrastive learning over groups (detailed in Section 3.2).
In summary, we present a novel view for the decoupling of pixel-level online clustering ( i.e ., semantic grouping) and object-level representation learning.
3.2 Group-level representation learning by contrasting slots
Inspired by Slot Attention [48], we then reuse the assignments computed by the semantic grouping module in Eq.
1 to perform attentive pooling over the dense projections z to produce group-level feature vectors (rephrased as slots ), as shown in Figure 2.
Intuitively, as the softmax normalization applies to the slot dimension, the attention coefficients sum to one for each individual input feature vector.
As a result, the soft assignments A of the dense projections z w.r.t.
the corresponding prototypes S can also be viewed as the attention coefficients that describe how the prototypes decompose the dense projections into non-overlapping groups.
This inspires us to decompose the dense projections with attentive pooling following [48].
Specifically, for a dense projection z lθ produced from view v l , we extract K slots:
[Equation]
where ⊙ denotes the Hadamard product, and a similar operation applies for the teacher network to produce S l ′ ξ .
Since the initial slots are shared by the whole dataset, the corresponding semantic may be missing in a specific view v l , thus producing redundant slots.
Therefore, we compute the following binary indicator 1 l to mask out the slots that fail to occupy a dominating pixel:
[Equation]
i
[Equation]
j such that
[Equation]
and 1 l ′ ξ is computed similarly.
Following the literature in object-level SSL [62, 34, 72, 35], we then apply a contrastive learning objective to discriminate the slot that holds the same semantic across views from distracting slots with the InfoNCE [60] loss:
[Equation]
This objective helps maximize the similarity between different views of the same slot, while minimizing the similarity between slots from another view with different semantics and all slots from other images.
Note that here an additional predictor q θ with the same architecture as the projector g θ is applied to the slots S θ as empirically it yields stronger performance [11, 68, 35].
And the resulting slot-level contrastive loss also follows a symmetric design like Eq. 2:
[Equation]
3.3 The overall optimization objective
We jointly optimize the semantic grouping objective (Eq. 2) and the group-level contrastive learning objective (Eq. 7), controlled with a balancing factor λ g :
[Equation]
At each training step, the student network is optimized with gradients from the overall loss function: θ ← optimizer (cid:16) θ, ∇ θ L Overall θ,ξ , η (cid:17) , where η denotes the learning rate; and the teacher network updates as an exponential moving average of the student network: ξ ← λ t ξ + (1 − λ t ) θ , with λ t denoting the momentum value.
After training, only the teacher encoder f ξ is kept for downstream tasks.
Experiments 4.1 Implementation details
[Table]
COCO+ [46] 241K
[Table]
Pre-training datasets.
We pre-train our models on COCO train2017 [46] and
ImageNet-1K [15], respectively.
COCO train2017 [46] contains ∼ 118K images of diverse scenes with objects of multiple scales, which is closer to real-world scenarios.
In contrast, ImageNet-1K is a curated object-centric dataset containing more than ∼ 1 .
28M images, which is better for evaluating a model’s potential with large-scale data.
Besides, we also explore the limit of scene-centric pre-training on COCO+, i.e ., COCO train set plus the unlabeled2017 set.
See details in Table 1.
Data augmentation.
The image augmentation setting is the same as BYOL [28]: a 224 × 224 -pixel random resized crop with a random horizontal flip, followed by a random color distortion, random grayscale conversion, random Gaussian blur, and solarization.
The crop pairs without overlap are discarded during training.
Network architecture.
We adopt ResNet-50 [33] as the default encoder for f θ and f ξ .
The projector g θ , g ξ and predictor q θ are MLPs whose architecture are identical to that in [7] with a hidden dimension of 4096 and an output dimension of 256 .
Optimization.
We adopt the LARS optimizer [77] to pre-train the model, with a batch size of 512 across eight NVIDIA
Ti GPUs.
Following [73], we utilize the cosine learning rate decay schedule [50] with a base learning rate of 1 . 0 , linearly scaled with the batch size (LearningRate = 1 .
× BatchSize / 256 ), a weight decay of 10 − 5 , and a warm-up period of 5 epochs.
Following [72, 66, 73], the model is pre-trained for 800 epochs on COCO(+) and 100 / 200 epochs on ImageNet, respectively.
Following the common practice of [72, 73], the momentum value λ t of the teacher model starts from 0 .
and is gradually increased to 1 following a cosine schedule.
Synchronized batch normalization and automatic mixed precision are also enabled during training.
Hyper-parameters.
The temperature values τ s and τ t in the student and teacher model are set to 0 .
1 and 0 .
07 , respectively.
Besides, the center momentum λ c is set to 0 .
The default number of prototypes K is set to 256 for COCO(+) and 2048 for ImageNet, according to our empirical finding that setting the number of prototypes close to the number of human-annotated categories can help downstream performance.
The temperature value τ c for the contrastive loss is set to 0 .following
[11], and the default balancing ratio λ g is set to 0 .
4.2 Evaluation protocols
Following the common practice of previous self-supervised works [66, 73, 72], we evaluate the representation ability of the pre-trained model by taking it as the backbone of downstream tasks.
Specifically, we add a newly initialized task-specific head to the pre-trained model for different downstream tasks, i.e ., object detection and instance segmentation on COCO [46], and semantic segmentation on PASCAL VOC [20], Cityscapes [13], and ADE20K [83].
Object detection and instance segmentation.
We train a Mask R-CNN [32] model with R50FPN [45] backbone implemented in Detectron2 [69].
We fine-tune all layers end-to-end on COCO train2017 split with the standard 1 × schedule and report AP, AP 50 , AP 75 on the val2017 split.
Following [66, 73, 72] we train with the standard 1 × schedule with SyncBN.
Semantic segmentation.
The evaluation details of PASCAL VOC and Cityscapes strictly follow [31].
We take our network to initialize the backbone of a fully-convolutional network [49] and fine-tune all the layers end-to-end.
For PASCAL VOC, we fine-tune the model on train_aug set for 30k iterations and report the mean intersection over union (mIoU) on the val2012 set.
For
Cityscapes, we fine-tune on the train_fine set for 90k iterations and evaluate it on the val_fine set.
For ADE20K, we follow the standard 80k iterations schedule of MMSegmentation [12].
Unsupervised semantic segmentation.
We also evaluate the model’s ability of discovering semantic groups in complex scenes, which is accomplished by performing unsupervised semantic segmentation on COCO-Stuff [4].
We follow the common practice in this field [40, 39, 37] to merge the labels into 27 categories ( 15 "stuff" categories and 12 "thing" categories), and evaluate with a subset created by [40].
We perform inference with resolution 320 and number of prototypesfollowing the standard practice.
The predicted labels are matched with the ground truth through
Hungarian matching [43], and evaluated on mIoU and pixel accuracy (pAcc)
[Table]
4.3 Transfer learning results
COCO pre-training.
In Table 2, we show the main results with COCO pre-training.
There have been steady improvements in object-level pre-training with COCO, in which the top performance methods are DetCon [44] and ORL [72], which still rely on objectness priors like selective-search [59] or hand-crafted segmentation algorithms [21], yet they still fail to beat the pixel-level state-of-the-art
[Table]
PixPro ‡ [73] ✗ ✗ 40.5 60.
Object / Group-level approaches
DetCon † [34] ✗ ✓ 39.8 59.
ORL † [72] ✓ ✓ 40.3 60.
36.7 Ours (SlotCon) ✗ ✗ 41.0 61.
PixPro [73].
Our method alleviates such limitations and significantly improves over current objectlevel methods in all tasks, achieving consistent improvement over the previous approaches and even several methods that were pre-trained on the larger dataset ImageNet-1K (Table 4).
It is also notable that our method can achieve a better performance on the largest and most challenging dataset for segmentation, ADE20K, adding to the significance of this work.
COCO+ pre-training.
In Table
[Table]
we report the results with COCO+ pre-training.
The COCO+ is
COCO train2017 plus unlabeled2017 set, which roughly doubles the number of training images and greatly adds to the data diversity.
Our method further sees a notable gain in all tasks with extended
COCO+ data
[Table]
and even shows comparable results with our best-performing model pre-trained on
ImageNet-1K ( 5 × large of COCO+), showing the great potential of scene-centric pre-training
[Table] [Table]
ImageNet-1K pre
training.
In Table 4, we also benchmark our method with ImageNet-1K pretraining and show its compatibility with object-centric data.
Without selective-search for object proposals and without pre-training and transferring the FPN head, our method still beats most of the current works and largely closes the gap with the detection-specialized method SoCo.
[Table]
PixPro † [73] ✗ ✗ 40.7 60.
Object / Group-level approaches
DetCon [34] ✗ ✓ 40.
SoCo ‡ [68] ✓ ✓ 41.6 61.
37.8 Ours (SlotCon) ✗ ✗ 41.4 61.
38.6 Ours (SlotCon) ✗ ✗ 41.8 62.
4.4 Unsupervised semantic segmentation results
Given our approach’s consistent improvement in representation learning, we further analysis of how well our semantic grouping component can parse scenes quantitatively and qualitatively.
It should be
[Table]
I m a g e
P
i C
I E
O u r s
Ground
Person Building
Plant Sky
Water
Rawmater.
Floor
Wall Furniture
Animal clarified that this is just to help understand the learned prototypes, and not to propose a new SOTA.
Unlike current SSL approaches that exhaustively enumerate the massive object proposals and report the best score [7, 35], we follow the common practice of unsupervised semantic segmentation [40, 39] to match the predicted results with the ground-truth using the Hungarian algorithm [43], where each ground-truth label is assigned to a prototype mutual-exclusively.
For fair comparisons, the model used for evaluation is trained with K = 27 to match the number of categories of COCO-Stuff.
As shown in Table 5, our method can surpass the previous works PiCIE [39] and SegDiscover [37] with 4 points higher mIoU.
Meanwhile, the pAcc is lower since we train the model with a lower resolution ( 7 × 7 v.s. 80 × 80 feature map).
Besides Table 5, we also depict the visualization results, in which our method distinguishes confusing objects apart (4th column) and successfully localizes small objects (5th column).
Since we only need to separate pixels with different semantics within the same image, the errors in category prediction can be ignored.
4.5 Ablation study
[Table]
Number of prototypes K .
Table 6a ablates the number of prototypes, we observe that the most suitable K for COCO detection is 256 , which is close to its real semantic class number 172 (thing + stuff) [4].
Besides, the performance on Cityscapes and PASCAL VOC have a consistent tendency to drop, while the performance on ADE20K is consistently good if K is big enough.
We hypothesize that a suitable K can encourage learning data-specific semantic features, which are only helpful when the pre-training and downstream data are alike (from COCO to COCO); increasing K produces fine-grained features that may lack discriminability in semantics but hold better transferability to
ADE20K that require fine-grained segmentation [14].
Loss balancing weight λ g .
Table 6b ablates the balancing between the semantic grouping loss and the group-level contrastive loss, where the best balance is achieved with both losses treated equally.
It is notable that when λ g = 1 . 0 , only the semantic grouping loss is applied, and the performance drops considerably, indicating the importance of our group-level contrastive loss for learning good representations.
Teacher temperature τ t .
Table 6c ablates the teacher model’s temperature parameter, indicating that a softer teacher distribution with τ t = 0 .
07 helps achieve better performance.
4.6 Probing the prototypes
Finally, we analyze whether the prototypes learn semantic meanings by visualizing their nearest neighbors in COCO val2017 split.
We first perform semantic grouping on each image to split them into non-overlapping groups (segments), then pool each group to a feature vector, and retrieve the top
-neighbor segments for each prototype according to cosine similarity.
As shown in
Figure 3, the prototypes well bind to semantic meanings that cover a wide range of scenarios and semantic granularities from animals, foods, and sports, to furniture, buildings, etc., localizing them well regardless of small object size and occlusion; and notably, without any human annotation .
Discussion on the emergence of objectness
It should be clarified that the discovered objectiveness is at the semantic level, and object instances with identical semantics can be indistinguishable.
Concerning the emergence of objectness, we impose geometric-covariance and photometric-invariance as guiding cues which force the model to decompose a large complex dataset into a small number of clusters through optimizing feature space and cluster centers.
The emergence of objects/parts that are compositional and thus occupy a reasonable number of prototypes, can be viewed as a natural consequence under such conditions.
Concerning granularity, our intuition is that the prototype number and the dataset distribution generate a bottleneck for the granularity of groups.
For example, in the COCO dataset, with 256 prototypes, the model finds that splitting animals into cats, dots, elephants, etc ., is enough and won’t further separate them.
In contrast, for humans (the most occupying category of COCO), as shown in the appendix, the model discovers not only human parts (Figure 7) but also different types of activities (Figure 8), indicating that parts are more helpful in this scenario and deserve more prototypes.
Conclusion
This work presents a unified framework for joint semantic grouping and representation learning from unlabeled scene-centric images.
The semantic grouping is performed by assigning pixels to a set of learnable prototypes, which can adapt to each sample by attentive pooling over the feature map and form new slots.
Based on the learned data-dependent slots, a contrastive objective is employed for representation learning, enhancing features’ discriminability and facilitating the grouping of semantically coherent pixels together.
By simultaneously optimizing the two coupled objectives of semantic grouping and contrastive learning, the proposed approach bypasses the disadvantages of handcrafted priors and can learn object/group-level representations from scene-centric images.
Experiments show the proposed approach effectively decomposes complex scenes into semantic groups for feature learning and significantly facilitates downstream tasks, including object detection, instance segmentation, and semantic segmentation.
Acknowledgments
This work has been supported by Hong Kong Research Grant Council - Early Career Scheme (Grant No. 27209621), HKU Startup Fund, and HKU Seed Fund for Basic Research.
The authors acknowledge SmartMore, LunarAI, and MEGVII for partial computing support.