B
OOTSTRAPPING S EMANTIC S EGMENTATION WITH R EGIONAL C ONTRAST
Anonymous authors
Paper under double-blind review
A BSTRACT
We present ReCo, a contrastive learning framework designed at a regional level to assist learning in semantic segmentation.
ReCo performs pixel-level contrastive learning on a sparse set of hard negative pixels, with minimal additional memory footprint.
ReCo is easy to implement, being built on top of off-the-shelf segmentation networks, and consistently improves performance, achieving more accurate segmentation boundaries and faster convergence.
The strongest effect is in semisupervised learning with very few labels.
With ReCo, we achieve high quality semantic segmentation model, requiring only 5 examples of each semantic class
I NTRODUCTION
Semantic segmentation is an essential part of applications such as scene understanding and autonomous driving, whose goal is to assign a semantic label to each pixel in an image.
Signiﬁcant progress has been achieved by use of large datasets with high quality human annotations.
However, labelling images with pixel-level accuracy is time consuming and expensive; for example, labelling a single image in CityScapes can take more than 90 minutes (Cordts et al., 2016).
When deploying semantic segmentation models in practical applications where only limited labelled data are available, high quality ground-truth annotation is a signiﬁcant bottleneck.
To reduce the need for labelled data, there is a recent surge of interest in leveraging unlabelled data for semi-supervised learning.
Previous methods include improving segmentation models via adversarial learning (Hung et al., 2019; Mittal et al., 2019) and self-training (Zou et al., 2019; 2018;
Zhu et al., 2020).
Others focus on designing advanced data augmentation strategies to generate pseudo image-annotation pairs from unlabelled images (Olsson et al., 2021; French et al., 2020).
[Figure]
In both semi-supervised and supervised learning, a segmentation model often predicts smooth label maps, because neighbouring pixels are usually of the same class, and rarer high-frequency regions are typically only found in object boundaries.
This learning bias produces blurry contours and regularly mis-labels rare objects.
After carefully examining the label predictions, we further observe that wrongly labelled pixels are typically confused with very few other classes; e.g. a pixel labelled as rider has a much higher chance of being wrongly classiﬁed as person , compared to train or bus .
By understanding this class structure, learning can be actively focused on the challenging pixels to improve overall segmentation quality.
Here we propose ReCo, a contrastive learning framework designed at a regional level.
Speciﬁcally,
ReCo is a new loss function which helps semantic segmentation not only to learn from local context (neighbouring pixels), but also from global semantic class relationships across the entire dataset.
ReCo performs contrastive learning on a pixel-level dense representation, as visualised in Fig.
For each semantic class in a mini-batch, ReCo samples a set of pixel-level representations (queries), and encourages them to be close to the class mean representation (positive keys), and simultaneously pushes them away from representations sampled from other classes (negative keys).
For pixel-level contrastive learning with high-resolution images, it is impractical to sample all pixels.
In ReCo, we actively sample a sparse set of queries and keys, consisting of less than 5% of all available pixels.
We sample negative keys from a learned distribution based on the relative distance between the mean representation of each negative key and the query class.
This distribution can be interpreted as a pairwise semantic class relationship, dynamically updated during training.
We sample queries for those having a low prediction conﬁdence.
Active sampling helps ReCo to rapidly focus on the most confusing pixels for each semantic class, and requires minimal additional memory.
ReCo enables a high-accuracy segmentation model to be trained with very few human annotations.
We evaluate ReCo in a semi-supervised setting, with two different modes: i) Partial Dataset Full
Labels — a sparse subset of training images, where each image has full ground-truth labels, and the remaining images are unlabelled; ii) Partial Labels Full Dataset — all images have some labels, but covering only a sparse subset of pixels within each image.
In both settings, we show that ReCo can consistently improve performance across all methods and datasets.
R ELATED W ORK
Semantic Segmentation One recent direction is in designing more effective deep convolutional neural networks.
Fully convolutional networks (FCNs) (Long et al., 2015) are the foundation of modern segmentation network design.
They were later improved with dilated/atrous convolutions with larger receptive ﬁelds, capturing more long range information (Chen et al., 2017; 2018).
Alternative approaches include encoder-decoder architectures (Ronneberger et al., 2015; Kirillov et al., 2019), sometimes using skip connections (Ronneberger et al., 2015) to reﬁne ﬁltered details.
A parallel direction is to improve optimisation strategies, by designing loss functions that better respect class imbalance
(Lin et al., 2017) or using point-wise rendering strategy to reﬁne uncertain pixels from high-frequency regions improving the label quality (Kirillov et al., 2020).
ReCo is built upon this line of research, a model-agnostic framework to improve segmentation by providing additional supervision on hard pixels.
Semi-supervised Classiﬁcation and Segmentation
The goal of semi-supervised learning is to improve model performance by taking advantage of a large amount of unlabelled data during training.
Here consistency regularisation and entropy minimisation are two common strategies.
The intuition is that the network’s output should be invariant to data perturbation and geometric transformation.
Based on these strategies, many semi-supervised methods have been developed for image classiﬁcation (Sohn et al., 2020; Tarvainen & Valpola, 2017; Berthelot et al., 2019; Kuo et al., 2020).
However, for segmentation, generating effective pseudo-labels and well-designed data augmentation are non-trivial.
Some solutions improved the quality of pseudo-labelling, using adversarial learning (Hung et al., 2019; Mittal et al., 2019) and class activation maps (Zou et al., 2021); or enforcing consistency from different augmented images (French et al., 2020; Olsson et al., 2021), perturbed features (Ouali et al., 2020) and different networks (Ke et al., 2020).
In this work, we show that rather than designing a more advanced pseudo-labelling strategy, we can improve the performance of current semi-supervised segmentation methods by jointly training with a suitable auxiliary task.
Contrastive Learning Contrastive learning learns a similarity function to bring views of the same data closer in representation space, whilst pushing views of different data apart.
Most recent contrastive frameworks learn similarity scores based on global representations of the views, parameterising data with a single vector (He et al., 2020; Chen et al., 2020; Khosla et al., 2020).
Dense representations , on the other hand, rely on pixel-level representations and naturally provide additional supervision, capturing ﬁne-grained pixel correspondence.
Contrastive pre-training based on dense representations has recently been explored, and shows better performance in dense prediction tasks, such as object detection and keypoint detection (Wang et al., 2021b; O. Pinheiro et al., 2020).
Contrastive Learning for Semantic Segmentation Contrastive learning has been recently studied to improve semantic segmentation, with a number of different design strategies. Zhang et al.
(2021) and Zhao et al. both perform contrastive learning via pre-training, based on the generated auxiliary labels and ground-truth labels respectively, but at the cost of huge memory consump- tion. In contrast, ours performs contrastive learning whilst requiring much less memory, via active sampling.
In concurrent work, (Wang et al., 2021a; Alonso et al., 2021) also perform contrastive learning with active sampling.
However, whilst both these methods are applied to a stored feature bank, ours focuses on sampling features on-the-ﬂy. Active sampling in Alonso et al. (2021) is further based on learnable, class-speciﬁc attention modules, whilst ours only samples features based on relation graphs and prediction conﬁdence, without introducing any additional computation overhead, which results in a simpler and much more memory-efﬁcient implementation.
R E C O – R EGIONAL C ONTRAST 3. P IXEL -L EVEL C ONTRASTIVE L EARNING
Let ( X, Y ) be a training dataset with training images x ∈ X and their corresponding C -class pixellevel segmentation labels y ∈ Y , where y can be either provided in the original dataset, or generated automatically as pseudo-labels.
A segmentation network f is then optimised to learn a mapping f θ :
X (cid:55)→ Y , parameterised by network parameters θ .
This segmentation network f can be decomposed into two parts: an encoder network: φ : X (cid:55)→ Z , and a decoder classiﬁcation head ψ c : Z (cid:55)→ Y .
To perform pixel-level contrastive learning, we additionally attach a decoder representation head ψ r on top of the encoder network φ , parallel to the classiﬁcation head, mapping the encoded feature into a higher m -dimensional dense representation with the same spatial resolution as the input image: ψ r : Z (cid:55)→ R, R ∈ R m .
This representation head is only applied during training to guide the classiﬁer using the ReCo loss as an auxiliary task, and is removed during inference.
A pixel-level contrastive loss is a function which encourages queries r q to be similar to the positive key r + k , and dissimilar to the negative keys r − k .
All queries and keys are sampled from the decoder representation head: r q , r + , − k ∈ R .
In ReCo, we use a pixel-level contrastive loss across all available semantic classes in each mini-batch, with the distance between keys and queries measured by their normalised dot product.
The general formation of the ReCo loss L reco is then deﬁned as:
[Equation]
for which C is a set containing all available classes in the current mini-batch, τ is the temperature control of the softness of the distribution, R cq represents a query set containing all representations whose labels belong to class c , R ck represents a negative key set containing all representations whose labels do not belong to class c , and r c, + k represents the positive key which is the mean representation of class c .
Suppose P is a set containing all pixel coordinates with the same resolution as R , these queries and keys are then deﬁned as:
[Equation]
3. A CTIVE H ARD S AMPLING ON Q UERIES AND K EYS
Contrastive learning on all pixels in high-resolution images would be computationally expensive.
Here, we introduce active hard sampling strategies to optimise only a sparse set of queries and keys.
Active Key Sampling When classifying a pixel, a semantic network might be uncertain only over a very small number of candidates, among all available classes.
The uncertainty from these candidates typically comes from a close spatial ( e.g. rider and bicycle ) or semantic ( e.g. horse and cow ) relationship.
To reduce this uncertainty, we propose to sample negative keys non-uniformly, based on the relative distance between each negative key class and the query class.
This involves building a pair-wise class relationship graph G , with G ∈ R |C|×|C| , computed and dynamically updated for each mini-batch.
This pair-wise relationship is measured by the normalised dot product between the mean representation from a pair of two classes and is deﬁned as:
[Equation]
We further apply SoftMax to normalise these pair-wise relationships among all negative classes j for each query class c , which produces a probabilistic distribution: exp( G [ c, i ]) / (cid:80) j ∈C ,j (cid:54) =
c exp( G [ c, j ]) .
We sample negative keys for each class i based on this distribution, to learn the corresponding query class c .
This procedure allocates more samples to hard, confusing classes chosen speciﬁcally for each query class, helping the segmentation network to learn a more accurate decision boundary.
Active Query Sampling Due to the natural class imbalance in semantic segmentation, it is easy to over-ﬁt on common classes, such as the road and building classes in the CityScapes dataset, or the background class in the Pascal VOC dataset.
These common classes contribute to the majority of pixel space in training images, and so randomly sampling queries will under-sample rare classes and provide minimal supervision to these classes.
Therefore, we instead sample hard queries — for those whose corresponding pixel prediction conﬁdence is below a deﬁned threshold.
Accordingly, ReCo’s loss would then guide the segmentation network by providing appropriate supervision on these less certain pixels.
The easy and hard queries are deﬁned as follows, and visualised in Fig.
[Equation]
where ˆ y q is the predicted conﬁdence of label c after the SoftMax operation corresponding to the same pixel location as r q , and δ s is the user-deﬁned conﬁdence threshold.
[Equation]
[Figure] [Figure]
S EMANTIC S EGMENTATION WITH R E C O
ReCo can easily be added to modern semi-supervised segmentation methods without changing the training pipeline, with no additional cost at inference time.
To incorporate ReCo, we simply add an additional representation head ψ r as described in Section 3.1, and apply the ReCo loss (in Eq. 1) to this representation using the sampling strategy introduced in Section 3.2.
We apply the Mean Teacher framework (Tarvainen & Valpola, 2017) following prior state-of-the-art semi-supervised segmentation methods (Olsson et al., 2021; Mittal et al., 2019).
Instead of using the original segmentation network f θ (which we call the student model), we instead use f θ (cid:48) (which we call the teacher model) to generate pseudo-labels from unlabelled images, where θ (cid:48) is a moving average of the previous state of θ during training optimisation: θ (
cid:48) t = λθ (cid:48)
t − 1 +(1 − λ ) θ t , with a decay parameter λ = 0 . 99 .
This teacher model can be treated as a temporal ensemble of student models across training time t , resulting in more stable predictions for unlabelled images.
The student model f θ is then used to train on the augmented unlabelled images, with pseudo-labels as the ground-truths.
For all pixels with deﬁned ground-truth labels, we apply the ReCo loss on dense representations corresponding to all valid pixels.
For all pixels without such labels, we only sample pixels whose predicted pseudo-label conﬁdence is greater than a threshold δ w .
This avoids sampling pixels which are likely to have incorrect pseudo-labels.
We apply the ReCo loss to a combined set of labelled and unlabelled pixels.
The overall training loss for semi-supervised segmentation is then the linear combination of supervised cross-entropy loss (on ground-truth labels), unsupervised cross-entropy loss (on pseudo-labels) and ReCo loss:
[Equation]
[Equation]
where η is deﬁned as the percentage of pixels whose predicted conﬁdence are greater than δ s , a scalar re-weighting the contribution for unsupervised loss, following prior methods (Olsson et al., 2021;
L
GT
L
S
L L
Labelled Images Augmented Images
Student Network
Predicted Features
Ground-
[Figure]
Labels Predicted Labels Supervised Loss
U U
S
U U
Unlabelled Images
Augmented Images
Student Network
Predicted Features
Augmented Labels Predicted Labels Unsupervised Loss
G
T
Generated Labels
Teacher Network
G
U L
GT
G
ReCo Loss
Feature Space
Masked Features
Mittal et al., 2019).
This makes sure the segmentation network would not be dominated by gradients produced by uncertain pseudo-labels, which typically occur during the early stage of training.
Fig.
3 shows a visualisation of the ReCo framework for semi-supervised segmentation.
E XPERIMENTS
Semi-Supervised Segmentation Benchmark Redesign We propose two modes of semisupervised segmentation tasks, aiming at two different applications.
i) Partial Dataset Full Labels : A small subset of the images is trained with complete ground-truth labels for each image, whilst the remaining training images are unlabelled.
This is the de-facto standard of evaluating semi-supervised segmentation in prior works.
ii) Partial Labels Full Dataset : All images are trained with partial labels, but only a small percentage of labels are provided for each class in each training image.
We create the dataset by ﬁrst randomly sampling a pixel for each class, and then continuously apply a [5 × 5] square kernel for dilation until we meet the percentage criteria.
[Figure]
The Partial Dataset Full Label setting evaluates the ability to generalise semantic classes given a few examples with perfect boundary information.
The Partial Label Full Dataset evaluates learning semantic class completion given many examples with no or minimal boundary information.
Datasets We experiment on segmentation datasets: Cityscapes (Cordts et al., 2016) and
Pascal VOC 2012 (Everingham et al., 2015) in both partial and full label setting.
We also evaluate on a more difﬁcult [Table] scene
[Table]
setting only, mainly due to the low quality annotations making it difﬁcult for fair evaluation in the partial label setting.
An example of the partially labelled Pascal VOC is shown in Fig.
Strong Baselines Prior semi-supervised segmentation methods are typically designed with different backbone architectures, and trained with different strategies, which makes it difﬁcult to compare them fairly.
In this work, we standardise the baselines and implement four strong semi-supervised segmentation methods ourselves: S4GAN (Mittal et al., 2019): an adversarial learning based semisupervised method; CutOut (French et al., 2020), CutMix (French et al., 2020), ClassMix (Olsson et al., 2021): three image augmentation strategies designed speciﬁcally for semi-supervised segmentation.
Our implementations for all baselines obtain performance on par with, and most of the time surpassing, the performance reported in each original publication, giving us a set of strong baselines.
All baselines and our method were implemented on DeepLabV3+ (Chen et al., 2018) with ResNet-101 backbone (He et al., 2016), and all with the same optimisation strategies.
Detailed hyper-parameters used for each dataset are provided in the Appendix A.
4. R ESULTS ON P ASCAL VOC, C ITY S CAPES , SUN RGB-D (F ULL L ABELS )
First, we compared our results to semi-supervised baselines in a full label setting.
We applied ReCo on top of ClassMix, which consistently outperformed other semi-supervised baselines.
Table 1 shows the mean IoU validation performance on three datasets over three individual runs (different labelled and unlabelled data splits).
The number of labelled images shown in the three columns for each dataset, are chosen such that the least-appeared classes have appeared in 5, 15 and 50 images respectively.
In the fewest-label setting for each dataset, applying ReCo with ClassMix can improve results by a signiﬁcant margin, with up to 1 .
5 − 4 .
5% absolute improvement.
[Table] [Table]
To further justify the effectiveness of ReCo, we also include results on existing benchmarks, to compare with other semi-supervised methods in Table 2.
Here, all baselines were re-implemented and reported in the PseudoSeg setting (Zou et al., 2021), where the labelled images are sampled from the original PASCAL dataset, with a total of 1.4k images.
In both benchmarks, ReCo shows state-of-the-art performance, and speciﬁcally is able to reach PseudoSeg’s performance, whilst requiring only half the labelled data .
Additional results on other data partitions are further shown in Appendix C.
In Fig.
5, we present qualitative results from the semi-supervised setup with the fewest labels:labels for CityScapes and 50 labels SUN RGB-D datasets.
The 60 labelled Pascal VOC is further shown in Appendix D. In Fig.
5, we can see the advantage of ReCo, where the edges and boundaries of small objects are clearly more pronounced such as in the person and bicycle classes in
CityScapes, and the lamp and pillow classes in SUN RGB-D. Interestingly, we found that in
SUN RGB-D, though all methods may confuse ambiguous class pairs such as table and desk or window and curtain , ReCo still produces consistently sharp and accurate object boundaries compared to the Supervised and ClassMix baselines where labels are noisy near object boundaries.
4. R ESULTS ON P ASCAL VOC AND C ITY S CAPES (P ARTIAL L ABELS )
In the partial label setting, we evaluated on the CityScapes and Pascal VOC datasets.
Table 3 compared ReCo to the two best semi-supervised baselines and a supervised baseline.
Again, we see
ReCo can improve performance in all cases when applied on top of ClassMix, with around 1 − 3%
Ground Truth
Supervised (20 Labels)
ClassMix (20 Labels)
ReCo + ClassMix (20 Labels)
Road
Sidewalk
Building
Wall
Fence
Pole
Trafﬁc Light Trafﬁc Sign Vegetation
Terrain Sky
Person
Rider
Car
Truck
Bus
Train
Motorcycle
Bicycle
Undeﬁned
Ground Truth
Supervised (50 Labels)
ClassMix (50 Labels)
ReCo + ClassMix (50 Labels)
Wall
Floor
Cabinet
Bed
Chair
Sofa
Table
Door
Window
Bookshelf Picture
Counter
Blinds
Desk
Shelves
Curtain
Dresser
Pillow
Mirror
Floor Mat Clothes
Ceiling
Books
Fridge
TV
Paper
Towel
Bath Curtain
Box
Whiteboard Person
Nightstand
Toilet
Sink
Lamp
Bathtub
Bag
Undeﬁned
[Table]
Method
[Table]
Supervised
[Table]
[Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
Method
[Table]
Supervised 44.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
absolute improvement.
We observe less relative performance improvement than in the full label setting; very sparse ground-truth annotations could confuse ReCo, resulting in inaccurate supervision.
We show qualitative results on Pascal VOC dataset trained on 1 labelled pixel per class per image in
Fig.
As in the full label setting, we see smoother and more accurate boundary predictions from
ReCo.
More visualisations from CityScapes are shown in Appendix E.
4. A BLATIVE A NALYSIS
Next we present an ablative analysis on 20 labelled CityScapes images to understand the behaviour of ReCo with respect to hyper-parameters.
We use our default experimental setting from Section 4.1, using ReCo with ClassMix.
Additional ablations are further shown in Appendix B.
Number of Queries and Keys
We ﬁrst evaluate the performance by varying the number of queries and keys used in ReCo framework, whilst ﬁxing all other hyper-parameters.
In Fig.
7a and 7b, we can observe that performance is better when sampling more queries and keys, but after a certain
[Table]
[Table]
point, the improvements become marginal.
Notably, even in our smallest option of having 32 queries per class in a mini-batch — consisting of less than 0.5% among all available pixel space — this can still improve performance by a non-trivial margin.
Compared to a concurrent work (Zhang et al., 2021) which requires 10k queries and 40k keys in each training iteration, ReCo can be optimised with × 50 more efﬁciency in terms of memory footprint.
Ratio of Unlabelled Data We evaluate how ReCo can generalise across different levels of unlabelled data.
In Fig.
7c, we show that by only training on 10% unlabelled data in the original setting, we can already surpass the ClassMix baseline.
This shows that ReCo can achieve strong generalisation not only in label efﬁciency but also in data efﬁciency.
Choice of Semi-Supervised Method Finally, we show that ReCo is robust to the choice of different semi-supervised methods.
In Fig.
7d, we can see that ReCo obtains a better performance from a variety choice of semi-supervised baselines.
Effect of Active Sampling In Fig.
7e, we see that randomly sampling queries and keys gives much less improvement compared to active sampling in our default setting.
Particularly, hard query sampling has a dominant effect on generalisation: if we instead only sample from easy queries, ReCo only marginally improves on the baseline.
This further veriﬁes that most queries are redundant.
[Figure]
V ISUALISATIONS AND I NTERPRETABILITY OF C LASS R ELATIONSHIPS
In this section, we visualise the pair-wise semantic class relation graph deﬁned in Eq. 3, additionally supported by a semantic class dendrogram using the off-the-shelf hierarchical clustering algorithm in SciPy (Virtanen et al., 2020) for better visualisation.
The features for each semantic class used in both visualisations are averaged across all available pixel embeddings in each class from the validation set.
In all visualisations, we compared features learned with ReCo built on top of supervised learning compared to a standard supervised learning method trained on all data, representing the semantic class relationships of the full dataset.
Using the same deﬁnitions in Section 3.1, we ﬁrst choose such pixel embedding to be the embedding
Z predicted from the encoder network φ in both supervised learning and with ReCo.
We also show the visualisation for embedding R which is the actual representation we used for ReCo loss and active sampling.
[Table]
[Figure]
[Table] [Table]
r i d e r c a r t r u c k b u s t r a i n m o t
o r c
y
c l e b i c y c
l e
[Table]
vegetationterrainskypersonridercartruckbustrainmotorcyclebicycle 0.
r o a d
s i d e w a l k t e rr a i n t r a ff i c l i g h t t r a ff i
c
s i g n p o l e b u i l
d i n g v e g e t a t i o n s k
[Table]
r
[Table]
r
[Table]
vegetationterrainskypersonridercartruckbustrainmotorcyclebicycle
[Table]
[Figure]
In Fig.
[Figure]
we present the semantic class relationship and dendrogram for the CityScapes dataset by embedding R and Z with and without ReCo.
We can clearly see that ReCo helps disentangle features compared to supervised learning where many pairs of semantic classes are similar.
In addition, we ﬁnd that the dendrogram generated by ReCo based on embedding Z is more structured, showing a clear and interpretable semantic tree by grouping semantically similar classes together: for example, all large transportation classes car , truck , bus and train are under the same parent branch.
In addition, we ﬁnd that nearly all classes based on embedding R are perfectly disentangled, except for bus and train , suggesting the CityScapes dataset might not have sufﬁcient bus and train examples to learn a distinctive representation for these two classes.
[Figure]
The pair-wise relation graph helps us to understand the distribution of semantic classes in each dataset, and clariﬁes the pattern of incorrect predictions from the trained semantic network.
We additionally provide a dendrogram based on embedding R for the SUN RGBD dataset, clearly showing ambiguous class pairs, such as night stand and dresser ; table and desk and floormat , consistent with our results shown in Fig.
Complete visualisations of these semantic class relationships are shown in Appendix F.
C ONCLUSION
In this work, we have presented ReCo, a new pixel-level contrastive framework with active sampling, designed speciﬁcally for semantic segmentation.
ReCo can improve performance in semantic segmentation methods with minimal additional memory footprint.
In particular, ReCo has shown its strongest effect in semi-supervised learning with very few labels, where we improved on the stateof-the-art by a large margin.
In further work, we aim to design effective contrastive frameworks for video representation learning.
R EPRODUCIBILITY
All of the information for reproducibility is shown in Appendix A.
A PPENDIX A I MPLEMENTATION D ETAILS
We trained all methods with SGD optimiser with learning rate 2 .
5 × 10 − 3 , momentum 0 .
9 , and weight decay 5 · 10 − 4 .
We adopted the polynomial annealing policy to schedule the learning rate, which is multiplied by (1 − iter total iter ) power with power = 0 . 9 , and trained for 40k iterations for all datasets.
Code is attached in the supplementary material.
For CityScapes, we ﬁrst downsampled all images in the dataset to half resolution [512 × 1024] prior to use.
We extracted [512 × 512] random crops and used a batch size of 2 during training.
For Pascal VOC, we extracted [321 × 321] random crops, applied a random scale between [0 . 5 , 1 . 5] , and used a batch size of 10 during training.
For SUN RGB-D, we ﬁrst rescaled all images to [384 × 512] resolution, extracted [321 × 321] random crops, applied a random scale between [0 . 5 , 1 . 5] , and used a batch size of 5.
We additionally re-organised the original training and validation split in SUN RGB-D dataset from 5285 and 5050 to 9860 and 475 samples respectively, to increase the amount of training data which we think is more appropriate for semi-supervised task.
All datasets were additionally augmented with Gaussian blur, colour jittering, and random horizontal ﬂip.
The pre-processing for CityScapes and Pascal VOC are consistent with the prior work (Olsson et al., 2021).
In Table 2, we extracted [513 × 513] random crops and applied a random scale between [0 . 5 , 2 . 0] , following PseudoSeg’s training setup (Zou et al., 2021).
In our ReCo framework, we sampled 256 query samples and 512 key samples and used temperature τ = 0 .
5 for each mini-batch, which we found to work well in all datasets.
The dimensionality for pixel-level representation was set to m = 256 .
The conﬁdence thresholds were set to δ w = 0 .
7 and δ s = 0 . 97 . B A DDITIONAL A BLATIVE S TUDIES [Table] ReCo Only Results Table 4 shows ReCo designed with and without data augmentation, trained on 20 and 50 labelled CityScapes dataset.
We observe that using pure semisupervised learning with additional unlabelled data will lead to worse performance compared to supervised learning without such labelled data. This shows data augmentation strategies designed for semi-supervised segmentation are the key component to make best use of the unlabelled data. Although the vanilla ReCo still performs better compared to standard semi-supervised learning, the active sampling of ReCo based on incorrect pseudo-labels leads to marginal improvement compared to a pure data augmentation method like ClassMix.
Therefore, ReCo performs better as an auxiliary framework combined with a strong semi-supervised method.
Compared to Feature Bank Methods We have experimented with ReCo with a stored feature bank framework similar to the design in the concurrent works (Alonso et al., 2021;
Wang et al., 2021a).
We found that just by replacing our batch-wise sampling method with a feature bank sampling method will achieve a similar performance (49.34 mIoU) compared to our original design (49.86 mIoU) on 20 labelled CityScapes, but with a slower training speed.
This veriﬁes our assumption that batch-wise sampling is an accurate approximation of class distribution over the entire dataset.
C R ESULTS ON S EMI - SUPERVISED S EGMENTATION B ENCHMARKS
Here, we present quantitative results for other semi-supervised semantic segmentation benchmarks in CityScapes and Pascal VOC datasets.
Note that, this benchmark is much less challenging compared to our proposed benchmark in Section 4.1, evaluated with signiﬁcantly less number of labelled images.
Since some methods applied with different backbones and training strategies, we compared each result with respect to its performance gap compared to its corresponding fully supervised result, as shown in brackets, to ensure fairness following Feng et al.
In Table
[Table]
we show results for ReCo applied on top of ClassMix, and trained with both DeepLabv (Chen et al., 2017) and DeepLabv
+ (Chen et al., 2018).
We can observe that ReCo achieved the best performances in most cases in both datasets, showing its robustness to different backbone architectures and number of labelled training images
[Table]
We list the percentage along with the number of labelled images at the top row.
The ﬁrst and second best performances in each data partition setting are coloured in red and orange respectively.
∗ trained images in doubled resolution.
All results were taken from the corresponding publications.
D V ISUALISATION ON P ASCAL VOC (T RAINED WITH 60 L ABELLED I MAGES )
In the full label setting, the baselines Supervised and ClassMix are very prone to completely misclassifying rare objects such as boat, bottle and table , while our method can predict these rare classes accurately.
[Table]
Bird
Boat
Bottle
Bus
Car
Cat
Chair
Cow
[Table]
Plant
Sheep
Sofa
Train
Monitor
Invalid
E V ISUALISATION ON C ITY S CAPES (T RAINED WITH 1% L ABELLED P IXEL )
In the partial label setting, the performance improvements are less pronounced compared to the full label setting in CityScapes dataset.
The improvements typically come from the more accurate predictions in small object boundaries such as in traffic light and traffic sign .
Learning semantics with partial labels with minimal boundary information remains an open research question and still has huge scope for improvements.
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
F V ISUALISATION ON S EMANTIC C LASS R ELATIONSHIP FROM P ASCAL VOC ( TOP ) AND SUN RGB-D ( BOTTOM )
We show features learned by ReCo are more disentangled compared to the Supervised baseline in all datasets, which helps the segmentation model to learn a better decision boundary. Brighter colour represents closer (more confused) relationship. Best viewed in zoom. [Figure]
[Table]
o u
[Table]
a e
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
wallfloorcabinetbedchairsofatabledoorwindowbookshelfpicturecounterblindsdeskshelvescurtaindresserpillowmirrorfloor
[Table]