Semi-supervised Semantic Segmentation with Prototype-based Consistency Regularization
Abstract
Semi-supervised semantic segmentation requires the model to effectively propagate the label information from limited annotated images to unlabeled ones.
A challenge for such a per-pixel prediction task is the large intra-class variation, i.e., regions belonging to the same class may exhibit a very different appearance even in the same picture.
This diversity will make the label propagation hard from pixels to pixels.
To address this problem, we propose a novel approach to regularize the distribution of within-class features to ease label propagation difficulty.
Specifically, our approach encourages the consistency between the prediction from a linear predictor and the output from a prototype-based predictor, which implicitly encourages features from the same pseudo-class to be close to at least one within-class prototype while staying far from the other between-class prototypes.
By further incorporating
CutMix operations and a carefully-designed prototype maintenance strategy, we create a semi-supervised semantic segmentation algorithm that demonstrates superior performance over the state-of-the-art methods from extensive experimental evaluation on both Pascal VOC and Cityscapes benchmarks 2 .
Introduction
Semantic segmentation is a fundamental task in computer vision and has been widely used in many vision applications [34, 2, 31].
Despite the advances, most existing successful semantic segmentation systems [27, 6, 9, 48] are supervised, which require a large amount of annotated data, a timeconsuming and costly process.
Semi-supervised semantic segmentation [51, 46, 32, 21, 8, 47, 20, 40] is a promising solution to this problem, which only requires a limited number of annotated images and aims to learn from both labeled and unlabeled data to improve the segmentation performance.
Recent studies in semi-supervised learning approaches suggest that pseudo-labeling [25, 1, 45] and consistency-based regularization [24, 3, 42] are two effective schemes to leverage the unlabeled data.
Those two schemes are often integrated into a teacher-student learning paradigm: the teacher model generates pseudo labels to train a student model that takes a perturbed input [36].
In such a scheme, and also for most pseudo-labeling-based approaches, the key to success is how to effectively propagate labels from the limited annotated images to the unlabeled ones.
A challenge for the semi-supervised semantic segmentation task is the large intra-class variation, i.e., regions belonging to the same class may exhibit a very different appearance even in the same picture.
This diversity will make the label propagation hard from pixels to pixels.
In this paper, we propose a novel approach to regularize the distribution of within-class features to ease label propagation difficulty.
Our method adopts two segmentation heads (a.k.a, predictors): a standard linear predictor and a prototype-based predictor.
The former has learnable parameters that could be updated through back-propagation, while the latter relies on a set of prototypes that are essentially local mean vectors and are calculated through running average.
Our key idea is to encourage the consistency between the prediction from a linear predictor and the output from a prototype-based predictor.
Such a scheme implicitly regularizes the feature representation: features from the same class must be close to at least one class prototype while staying far from the other class prototypes.
We further incorporate CutMix operation [44] to ensure such consistency is also preserved for perturbed (mixed) input images, which enhances the robustness of the feature representation.
This gives rise to a new semi-supervised semantic segmentation algorithm that only involves one extra consistency loss to the state-of-the-art framework and can be readily plugged into other semisupervised semantic segmentation methods.
Despite its simplicity, it has demonstrated remarkable improvement over the baseline approach and competitive results compared to the state-of-the-art approaches, as discovered in our experimental study.
Related Work
Semi-supervised Learning has made great progress in recent years due to its economic learning philosophy [50].
The success of most of the semi-supervised learning researches can attribute to the following two learning schemes: pseudo-labeling and consistency regularization.
Pseudolabeling based methods [25, 5, 1, 45] propose to train the model on unlabeled samples with pseudo labels generated from the up-to-date optimized model.
While consistency regularization based methods [24, 37, 39, 3, 42] build upon the smoothness assumption [28] and encourage the model to perform consistent on the same example with different perturbations.
The recently proposed semisupervised method FixMatch [36] successfully combine these two techniques together to produce the state-of-the-art classification performance.
Our approach draws on the successful experience of general semi-supervised learning and applies it to the semi-supervised semantic segmentation task.
Semi-supervised Semantic Segmentation benefits from the development of general semi-supervised learning and various kinds of semi-supervised semantic segmentation algorithms have been proposed.
For example, PseudoSeg method [51] utilizes the Grad-CAM [33] trick to calibrate the generated pseudo-labels for semantic segmentation network training.
While CPS [8] builds two parallel networks to generate cross pseudo labels for each each.
CutMix-Seg method [14] introduces the CutMix augmentation into semantic segmentation to construct consistency constraints on unlabeled samples.
Alternatively, CCT [32] chooses to insert perturbations into the manifold feature representation to enforce a consistent prediction.
And U 2 PL [40] proposes to make sufficient use of unreliable pseudo supervisions.
Meanwhile, considering the class-imbalance problem of semi-supervised semantic segmentation, several researches [20, 19, 15] have been published.
Our approach is inspired by the observation that large intra-class variation hinders the label information propagation from pixels to pixels in semi-supervised semantic segmentation and we propose a prototype-based consistency regularization method to alleviate this problem which is novel for related literature.
Prototype-based Learning has been well studied in the machine learning area [17].
The nearest neighbors algorithm [11] is one of the earliest works to explore the use of prototypes.
Recently, researchers have successfully used prototype-based learning to solve various problems, e.g., the prototypical networks [35] for few-shot learning and prototype-based classifier for semantic segmentation [48].
Our work further introduces the prototype-based learning into the semi-supervised problem and proves its effectiveness.
Our Approach
In this section, we first give an overview of our approach and then introduce the core concept of prototype-based consistency regularization for semi-supervised semantic segmentation.
Finally, we introduce how the prototype is constructed and maintained throughout the learning process.
3.1 Preliminary
Problem setting: Given a set of labeled training images D l = { ( I li , Y li ) } N l i =1 and a set of unlabeled images D u
= { I ui } N u i
=1 , where N u ≫ N l , semi-supervised semantic segmentation aims to learn
a
[Table]
[Figure]
of propagating label information from pixels to pixels.
This simple modification brings a significant improvement.
segmentation model from both the labeled and unlabeled images.
We use ˜ Y denote the segmentation output and ˜ Y [ a, b ] indicates the output at the ( a, b ) coordinate.
Overview: the overall structure of the proposed method is shown in Figure1, our approach is built on top of the popular student-teacher framework for semi-supervised learning [37, 36, 49, 29, 45].
During the training procedure, the teacher model prediction will be selectively used as pseudo-labels for supervising the student model.
In other words, the back-propagation is performed on the student model only.
More specifically, the parameters of the teacher network are the exponential moving average of the student network parameters [37].
Following the common practice [36], we also adopt the weak-strong augmentation paradigm by feeding the teacher model weakly-augmented images and the student strongly-augmented images.
In the context of image segmentation, we take the normal data augmentation (i.e., random crop and random horizontal flip of the input image) as the weak augmentation and CutMix [44] as the strong data augmentation.
The key difference between our method and existing methods [14, 32, 43, 8, 40] is the use of both a linear predictor (in both teacher and student models) and a prototype-based predictor (in the student model only).
As will be explained in the following section, the prediction from the teacher model’s linear predictor will be used to create pseudo labels to supervise the training of the prototype-based predictor of student model.
This process acts as a regularization that could benefit the label information propagation.
3.2 Prototype-based Predictor for Semantic Segmentation
Prototype-based classifier is a long-standing technique in machine learning [22, 4].
From its early form of the nearest neighbour classifier or the nearest mean classifier to prototypical networks in the few-shot learning literature [35], its idea of using prototypes instead of a parameterized classifier has been widely adopted in many fields.
Very recently, prototype-based variety has been introduced into the semantic segmentation task [48] and has been proved to be effective under a fully-supervised setting.
Formally, prototype-based classifier/predictors make the prediction by comparing test samples with a set of prototypes.
The prototype can be a sample feature or the average of a set of sample features of the same class.
Without loss of generality, we denote the prototype set as P = { ( p i , y i ) } , with p i indicate the prototype and y i is its associated class.
Note that the number of prototypes could be larger than the number of classes.
In other words, one class can have multiple prototypes for modelling its diversity.
More formally, with the prototype set, the classification decision can be made by using
[Equation]
where sim ( · , · ) represents the similarity metric function, e.g., cosine distance.
˜
y means the class assignment for the test data x .
The posterior probability of assigning a sample to the c -th class can also be estimated in prototype-based classifier via:
[Equation]
where T is the temperature parameter and can be empirically set.
Note that Eq.
2 essentially uses the maximal similarity between a sample and prototypes of a class as the similarity between a sample and a class.
3.3 Consistency Between Linear Predictor and Prototype-based Predictor
Although both prototype-based classifiers and linear classifiers can be used for semantic segmentation [48], they have quite different characteristics due to the nature of their decision-making process.
Specifically, linear classifiers could allocate learnable parameters 3 for each class, while prototypebased classifiers solely rely on a good feature representation such that samples from the same class will be close to at least one within-class prototypes while stay far from prototypes from other classes.
Consequently, linear classifiers could leverage the learnable parameter to focus more on discriminative dimensions of a feature representation while suppressing irrelevant feature dimensions, i.e., by assigning a higher or lower weight to different dimensions.
In contrast, prototype-based classifiers cannot leverage that and tend to require more discriminative feature representations.
The different characteristics of prototype-based and linear classifiers motivate us to design a loss to encourage the consistency of their predictions on unlabeled data to regularize the feature representation.
Our key insight is that a good feature should support either type of classifier to make correct predictions.
In addition to using two different types of classifiers, we also incorporate the CutMix [44] strategy to enhance the above consistency regularization.
CutMix augmentation is a popular ingredient in many state-of-the-art semi-supervised semantic segmentation methods [8, 26, 40].
Specially, we first perform weak augmentation, e.g., random flip and crop operations, to the input images of the teacher model and obtain the pseudo-labels from the linear classifier.
Next, we perform the CutMix operation by mixing two unlabeled images mix ( I i , I j ) and their associated prediction mix ( ˜ Y i , ˜ Y j ) .
The mixed image mix ( I i , I j ) is fed to the student model and the output from the prototype-based classifier is then enforced to fit the pseudo-labels generated from mix ( ˜ Y i , ˜ Y j ) .
Algorithm details: As a semi-supervised segmentation algorithm, we apply different loss functions for labeled images and unlabeled images.
For a batch of labeled images { ( I li , Y li ) } B l i =1 ∈ D l , we train both the linear predictor and the prototype-based predictor.
The linear classifier { w i } Ci =1 can produce a posterior probability estimation p linears ( Y [ a, b ]
= c
| I li )
[Equation]
where F li [ a, b ] =
f
( A 0 ( I li )) means the feature extracted at location ( a, b ) by first performing weak data augmentation A 0 to I li and then feed it to the feature extractor f .
Meanwhile, the posterior probability of prototype-based predictor p prototypes ( Y [ a, b ] = c
| I li ) can also be estimated via Eq.
We use cosine similarity for sim ( · , · ) and empirically set the temperature hyperparameter T to 0 .
Based on the ground truth label Y li , the student model will be optimized by the gradient back-propagated from the two predictors simultaneously
[Equation]
where
[Equation] [Equation]
Algorithm
Inputs
Outputs
Sample examples labeled set D set data the student model is updated based on the given ground truth, please refer to
Eq.(3)-(6) of main paper
For unlabeled data , weakly augmented version is fed into the teacher model to generate pseudo-labels and the student model is updated with the strongly augmented unlabeled sample based on the pseudo-labels.
Please refer to Eq.
(8)-(10) of main paper; 7: truth samples pseudo labels samples refer main paper; 8: the the student model
10: end for
For a batch of unlabeled images { I ui } B u
i =1 ∈ D u , we first use the teacher model to estimate their posterior probability p
[Equation]
[Equation] ) where { w ′ i } Ci =1 means the linear classifier weights of the teacher model and F ui [ a, b ] = f (cid:0)
A
( I ui ) (cid:1)
[ a, b ] denotes the extracted feature representation of pixel ( a, b ) from a weakly augmented unlabeled images.
Then, the class corresponding to the maximal posterior probability is the predicted class of a pixel in the given unlabeled sample, that is, ˜ Y ui
[ a, b ] = arg max c p lineart ( Y [ a, b ] =
c | I ui ) .
If p lineart ( ˜ Y [ a, b ] | I ui ) ≥ τ , where τ is a confidence threshold which is empirically set to 0 . 8 in our study, ˜ Y [ a, b ] will be used as pseudo-labels to train the student model.
Meanwhile, for the student model we perform CutMix [44] operation among weakly augmented unlabeled samples in the same batch to create an new image (essentially, the created mix-image can be considered as a strongly-augmented image), i.e., ˆ I uij = mix (cid:0)
A
( I ui ) , A
( I uj ) (cid:1) s.t., { i, j } ∈ B u , and their corresponding mixed prediction
ˆ Y uij =
mix ( ˜ Y ui , ˜ Y uj ) .
Therefore, the student model can learn from the unlabeled samples through the following training objectives
[Equation]
linearu
[Equation]
prototypeu , where
[Equation] [Equation] [Equation] where p linears ( Y [ a, b ] | ˆ I uij ) and p prototypes ( Y [ a, b ] | ˆ I uij ) are posterior probability predictions from linear classifier and prototype-based classifier of student model respectively.
Note that we use the studentteacher training for both the linear predictor and the prototype predictor, as shown in L linearu and
L prototypeu respectively.
A global view of our approach is presented in Algorithm 1.
Understand L prototypeu in Eq.
In order to better understand the proposed regularization loss term
L prototypeu , we can consider the following significantly-simplified version of our method by omitting Inputs Process: 1: supervised training: Train the semantic segmentation network on the subset of fully-labeled samples (please refer to Section 4.1 for training details); 2: feature extraction: Use the trained segmentation network to extract feature representations of labeled samples (i
e. the feature representation before feed into the classifier of DeepLabv3+ and perform interpolation on the feature representation to match the input image size).
We then sample a certain amount of pixels with their representations for each category; 3: feature clustering: Perform K-Means clustering (other clustering algorithms are also possible) on sampled pixel representations from each category.
This step creates K sub-classes for each category.
We use the feature average of samples in each subclass to obtain the initial prototypes of each category.
the CutMix operation: now let’s imagine at a certain point of the training process, the learned feature representation can successfully support the linear classifier in making a correct prediction for some pixels.
This means there are at least some discriminative feature dimensions that can distinguish classes.
Without loss of generality, let’s assume the feature vector for each pixel consists of two parts x = [ x d , x c ] , where x d is the discriminative part while x c is a less discriminative part, e.g., features shared by many classes.
Linear classifiers can assign lower weights to x c to suppress its impact, however, the impact of x c cannot be avoided by using prototype-based classifiers.
Thus from the supervision of the linear classifier, the training objective of optimizing the prototype-based classifier could further suppress the generation of x c .
Geometrically, this also encourages the features from the same class gather around a finite set of prototypes and being apart from prototypes of other classes.
In this way, the (pseudo) class label can propagate more easily from pixel to pixel, which in turn benefits the learning of the linear classifier.
3.4 Prototype Initialization and Update
Prototype initialization: The prototype-based classifier does not have learnable classifier parameters but relies on a set of good prototypes.
Thus it is vitally important to carefully devise strategies to initialize and maintain the pool of prototypes.
To initialize the prototypes, we first use the given labeled samples to train the semantic segmentation network (with a linear predictor) in a fully-supervised way for several epochs.
Then we extract pixelwise feature representation for each class with the trained segmentation network.
With the in-class pixel-wise feature representations, we propose to perform clustering on them to find out internal sub-classes, and the initial micro-prototypes will be obtained by averaging the feature representations within the same subclass.
Please find the Algorithm 2 for prototype initialization details.
Prototype update: In our approach, the prototypes are dynamically updated from the features extracted from the labeled images and those from unlabeled samples during the semi-supervised learning process.
When a labeled image is sampled, we assign each pixel to a prototype based on two conditions: (1) the assigned prototype p k should belong to the same class as the pixel.
(2) p k should be the most similar prototype among all other prototypes in the same class.
Once the assignment is done, we update p k via
[Equation]
where F [ a, b ] is the feature representation for the pixel at ( a, b ) .
α is a hyper-parameter controlling the prototype update speed.
We set α = 0 .
99 throughout our experiment.
For unlabeled images, the ground-truth class label for each pixel is unavailable, thus we use pseudolabel instead.
Recall that the pseudo-label is generated when the prediction confidence is higher than a threshold.
Thus, not every pixel will be used to update the prototype.
Also, since prototype-based classifier is only used for images after the CutMix [44] operation.
In our implementation, we use features extracted from the CutMix images to update the prototype rather than the original images.
Empirically we find this could slightly improve the performance.
Experiments 4.1 Experimental Setup
Our experiment setting follows the recently proposed state-of-the-art work U 2 PL [40] including the evaluation datasets, semantic segmentation networks and training schedules for a fair comparison 4 .
Some experimental details are listed as follows
Datasets: PASCAL VOC 2012 [13] is designed for visual object class recognition.
It contains twenty foreground object classes and one background class.
The standard partition of the dataset for training/validation/testing are 1,464/1,449/1,556 images, respectively.
In the semi-supervised semantic segmentation literature, some researches [8, 20, 43, 40] also include the augmented set [16] for model training.
This augmented set contains 9,118 images with coarse annotations.
In the literature [40], two ways of selecting the labeled data are considered: the classic and the blender setting.
The former selects labeled data from the original 1,464 candidate labeled images while the latter selects among all the 10,582 images.
We evaluate our method on both settings.
Cityscapes [10] is an urban scene understanding benchmark.
The initial 30 semantic classes are re-mapped into 19 classes for the semantic segmentation task.
The training, validation and testing set includes 2,975, 500 and 1,525 finely annotated images respectively.
For both of these two datasets, four kinds of label partitions are considered: 1/16, 1/8, 1/4 and 1/2.
In this paper, we compare all methods under the identical released label splits from U 2 PL [40] for a fair comparison.
Evaluation: We use single scale cropping for the evaluation of PASCAL VOC 2012 and slide window evaluation for Cityscapes for its high resolution.
The mean of Intersection over Union (mIoU) is adopted as the evaluation metric.
All numbers reported in this paper are measured on the validation set of these two datasets.
Methods: We compare our approach with several peer-reviewed semi-supervised segmentation algorithms [Table] Mean Teacher ( [Table]
[Table]
[Table]
[Table] (BMVC [Table] ) [14], PseudoSeg (ICLR 2020) [51], CCT (CVPR 2020) [32], GCT (ECCV 2020) [21], CPS (CVPR 2021) [8], PC 2 Seg(ICCV 2021) [47], AEL (NeurIPS 2021) [20] and U 2 PL (CVPR 2022)
Meanwhile, performance of supervised only on labeled data is also reported for a reference baseline.
To make a fair comparison, we conduct all experiments based on the same codebase released by the authors of U 2 PL [40].
Implementation Details: Following the common practice, we use ResNet-101 [18] pre-trained on
ImageNet [23] as our backbone and DeepLabv3+ [7] as the decoder.
We take the default segmentation head as the pixel-level linear classifier.
The feature representations for constructing the prototypes of our approach are extracted from the output of ASPP module [6].
Our experiments were run on 8 *
NVIDIA Tesla V100 GPUs (memory is 32G/GPU).
For both datasets, we adopt stochastic gradient descent (SGD) as the optimizer and set batch size to 16 for model optimization.
While other training details are slightly different, e.g., PASCAL VOC 2012 is trained with initial learning rate 1 .
× 10 − 3 , weight decay 1 .
× 10 − 4 and 80 training epochs; while Cityscapes is trained with initial learning rate 1 .
× 10 − 2 , weight decay 5 .
× 10 − 4 andtraining epochs.
Meanwhile, we use the polynomial policy to dynamically decay the learning rate along the whole training: lr = lr init · (1 − iter totaliter ) 0 . 8 .
4.2 Comparison with State-of-the-Arts
Results on PASCAL VOC 2012 Dataset [13]: Table 1 and Table 2 report the comparison results on PASCAL VOC 2012 validation set under different label quality settings.
First, the results in
Table 1 are obtained under the classic setting and our approach achieves consistent performance improvements over the compared methods.
Specifically, our method outperforms the Supervised
Only baseline by a large margin especially for the fewer data settings, e.g., +24.29% for 1/16 and [Table] [Table] AEL [20] 77.
U 2 PL [40] 77. Ours 78. +19.79% for 1/8 setting respectively. Meanwhile, our approach also successfully beats other semisupervised methods.
Taking the recently proposed state-of-the-art method U 2 PL [40] as an example, the performance gain of our approach reaches to +5.56% and +3.50% mIoU improvements under 1/8 and 1/4 label partitions, respectively.
Table 2 presents comparison results on the blender setting.
It is clear that our proposed method still achieves overall significant improvement over all other baselines.
For example, our method excels to the Supervised Only baseline over 10% mIoU on the 1/16 split.
Compared with previous well performed algorithms, e.g., AEL [20] and U 2 PL [40], our approach yields superior segmentation performance, e.g., +1.39% , +1.70%
and +1.48% on 1/16, 1/8 and 1/4 label partitions respectively.
Results on Cityscapes Dataset [10]: Table 3 provides comparison results of our method against several existing algorithms on Cityscapes validation set.
Compared to Supervised Only baseline, our method achieves a great performance improvement due to the make use of unlabeled data, e.g., under the 1/16 label partition, our approach surpasses Supervised Only baseline by 7.67% .
Then, compared to the simple Mean Teacher [37] baseline, our approach also performs better in all cases.
Furthermore, our approach is superior than the state-of-the-art algorithm U 2 PL [40], e.g., Ours excels to U 2 PL by 3.11% , 1.94% and 1.93% under the 1/16, 1/8 and 1/4 label partition, respectively.
Note that our method performs slightly worse than AEL [20] on the 1/16 label partition, it is because the class imbalance issue is more severe on this partition, and the AEL method, which is specially designed for handling the class imbalance problem, thus gains greater improvement.
Since the purpose of this paper is to explore the new consistency loss to alleviate intra-class variation for the semi-supervised semantic segmentation task, we do not explicitly consider measures to handle the label imbalance issue.
Theoretically, the techniques for solving label imbalance issues can also be incorporated into our method for optimizing the overall performance. [Table]
4.3 Ablation Study
To investigate how our approach works on the semi-supervised semantic segmentation task, we conduct ablation studies on the classic PASCAL VOC 2012 setting under 1/16 and 1/8 partition protocols from the following perspective views:
Effectiveness of different components : Table 4 presents ablation studies of several variants of our approach based on the usage of different components.
The variant ④ , which uses of all components, is the default setting of our approach and is presented here for a reference.
The variant ① only contains a linear predictor and the prototype-based predictor is omitted.
It is clear that the performance of this variant drops a lot compared to our approach and this proves that the prototype-based predictor plays a key role in our approach.
On the contrary, the variant ② only maintains a prototype-based predictor and dynamically updates the prototypes during the training.
The corresponding results are shown to be the worst among all the compared variants in Table 4.
We postulate the potential reason is that the prototype-based predictor itself is not good enough to generate high quality pseudo-labels without the help of the linear classifier under the limited labeled data setting and thus cannot fully leverage the large amount of unlabeled samples.
The variant ③ ablates the necessity of prototype update in our approach and the performance gap between this variant and variant ④ shows that our approach will benefit from the prototype update procedure and produce overall best performance.
Distribution of feature representation :
The core idea of introducing prototype-based predictor in our approach is to utilize the prototype-based consistency regularization for alleviating the strong intra-class variation problem in semi-supervised semantic segmentation. Therefore, we are interested [Figure] [Figure]
[Figure] [Figure] I [Figure]
[Figure] [Table] in the influence of our method on feature distribution. Figure 2 presents the feature distribution of various methods for some classes of Pascal VOC 2012.
We can find that our method tends to produce more compact feature distributions than other compared methods for every semantic class and such compact feature will ease the label information propagation from pixels to pixels and thus weaken the influence of intra-class variation.
Number of prototype : For the prototype-based classifier, the number of prototype is not restricted to be equal to the number of classes.
In our approach, we construct multiple prototypes for each semantic class to handle the intra-class variation problem of semi-supervised semantic segmentation task.
In order to explore the influence of the number of prototypes on our method, we conduct ablation studies on our approach with different number of prototypes.
As the results shown in Figure 3, the performance is tend to be saturate when the prototype number reaches to 4 for each semantic class.
Therefore, we empirically take this number as the default value of our approach.
Limitations
One underlying assumption about our approach is that we mainly consider convolutional based semantic segmentation networks.
Recently transformer-based algorithms [9, 41] are being investigated for semantic segmentation that are not explored in this paper and is left for future work.
One underlying assumption about our approach is that we mainly consider semantic segmentation networks of per-pixel prediction style.
Conclusion
Semi-supervised semantic segmentation aims to propagate label information from pixels to pixels effectively, but the large intra-class variation hinders the propagation ability.
In this paper, we introduce a prototype-based predictor into our semi-supervised semantic segmentation network and propose a novel prototype-based consistency loss to regularize the intra-class feature representation to be more compact.
Experimental results show that our method successfully achieves superior performance than other approaches.
Impacts and Ethics
This paper proposes a method for semi-supervised semantic segmentation which is a fundamental research topic in computer vision area and no potential negative societal impacts are known up to now.
In terms of ethics, we do not see immediate concerns for the models we introduce and to the best of our knowledge no datasets were used that have known ethical issues.
Acknowledgements This work is partially supported by Centre of Augmented Reasoning of the
University of Adelaide.
Meanwhile, we would like to thank the anonymous reviewers for their insightful comments.
We also gratefully acknowledge the support of MindSpore 5 , CANN (Computer
Architecture for Neural Networks) and Ascend AI Processor used for this research.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] See Section(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] See Section2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Abstract section.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Sections 4.1 and 4.3.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] All compared algorithms are experimented on the same random seed for a fair comparison in our paper.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Sections 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Sections 4.
(b) Did you mention the license of the assets?
[Yes] See Sections 4.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[No] Data used in this paper has been open source for research purpose.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No] Data used in this paper has been open source for research purpose.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
In this appendix, we first present quantitative metrics for comparing the intra-/inter-class discrimination of various methods. Next, we provide another two ablation studies to further inspect our approach. Finally, we further visualize the semantic segmentation results of our approach for better understanding.
In the main paper, the visualization of feature distribution in Figure 2 (c) has demonstrated that our approach can encourage a more compact within-class feature distribution and thus ease the large intra-class variation problem in the semi-supervised semantic segmentation. In order to have quantitative comparison, we borrow the principle of linear discriminant analysis (LDA) and calculate the intra-/inter-class variance of the feature representations for each comparing methods. As the results shown in Table 5, our approach has not only improved the intra-class variance but also the inter-class variance, and thus the overall discrimination.
classic VOC 2012(1/16 setting)
[Table]
B Ablation Studies B.1 Strong Data Augmentation
In the main paper, our approach is built upon the popular student-teacher weak-strong augmentation framework and the CutMix [44] strong data augmentation is utilized as the default setting.
In order to further investigate the effectiveness of our approach, we conduct an ablation study by varying the data augmentation approaches while keeping other modules unchanged in any comparing methods.
As results shown in Table 6, our method can still achieve overall best segmentation results with different strong data augmentations.
[Table]
B.2 Confidence Threshold
We are also interested in how our approach will be performed when various confidence thresholds are selected.
From the result shown in Table 7, we find that our approach can achieve good performance when the confidence threshold falls into a reasonable range, e.g., [0.75, 0.95].
C Semantic Segmentation Visualization
In our main paper, we have verified the effectiveness of our proposed method through extensive quantitative comparative experiments.
In the appendix, we want to provide more qualitative results to further support our conclusion.
[Table]
Figure 4 and Figure 5 present the segmentation results of comparing methods on the PASCAL
VOC 2012 validation set from the perspective of object boundary perception and object intra-class prediction consistency, respectively.
Specifically, Figure 4 illustrates that our method can produce better segments for the boundary of objects.
As the highlighted region shown in yellow dotted boxes, i.e., the lower edge of train (row 1), the body of person (row 2-3), the wing of airplane (row 4) and the bottle (row 5), the generated segments are much more precise for our method than the baseline method without prototype-based consistency regularization constraints.
Similarly, Figure 5 demonstrates that our method can achieve consistent category prediction within the objects, while the comparison method may always predict different parts of the same object into different categories (e.g., the dog at row 1, the train at row 2, the cow at row 3 and the cat at row 4) and sometimes even completely wrong prediction for the whole object (the sofa at row 5 and the cow at last row are completely mispredicted as chair and horse, respectively).
The superior semantic segmentation performance of our approach is attributed to the proposed prototype-based consistency regularization which encourages the features from the same class to be close to at least one within-class prototype while staying far away from the other between-class prototypes.
Such kind of constraints will ease the label information propagation from pixels to pixels for the semi-supervised semantic segmentation task and therefore our approach can produce more precise segments and predict consistent categories within the same segment.
[Figure]
[Figure]