Semi-Supervised Semantic Segmentation via Gentle Teaching Assistant
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Semi-Supervised Semantic Segmentation aims at training the segmentation modelwith limited labeled data and a large amount of unlabeled data.
To effectively leverage the unlabeled data, pseudo labeling, along with the teacher-student framework,is widely adopted in semi-supervised semantic segmentation.
Though proved tobe effective, this paradigm suffers from incorrect pseudo labels which inevitablyexist and are taken as auxiliary training data.
To alleviate the negative impactof incorrect pseudo labels, we delve into the current Semi-Supervised Semantic
Segmentation frameworks.
We argue that the unlabeled data with pseudo labelscan facilitate the learning of representative features in the feature extractor, butit is unreliable to supervise the mask predictor.
Motivated by this consideration,we propose a novel framework, Gentle Teaching Assistant (GTA-Seg) to disentangle the effects of pseudo labels on feature extractor and mask predictor of thestudent model.
Speciﬁcally, in addition to the original teacher-student framework,our method introduces a teaching assistant network which directly learns frompseudo labels generated by the teacher network.
The gentle teaching assistant(GTA) is coined gentle since it only transfers the beneﬁcial feature representationknowledge in the feature extractor to the student model in an Exponential Moving
Average (EMA) manner, protecting the student model from the negative inﬂuencescaused by unreliable pseudo labels in the mask predictor.
The student model isalso supervised by reliable labeled data to train an accurate mask predictor, furtherfacilitating feature representation.
Extensive experiment results on benchmarkdatasets validate that our method shows competitive performance against previousmethods.
We promise to release our code towards reproducibility.
Introduction
The rapid development in deep learning has brought signiﬁcant advances to semantic segmentation [29,5, 52] which is one of the most fundamental tasks in computer vision.
Existing methods often heavilyrely on numerous pixel-wise annotated data, which is labor-exhausting and expensive.
Towardsthis burden, great interests have been aroused in Semi-Supervised Semantic Segmentation, whichattempts to train a semantic segmentation model with limited labeled data and a large amount ofunlabeled data.
The key challenge in semi-supervised learning is to effectively leverage the abundant unlabeled data.
One widely adopted strategy is pseudo labeling [27].
As shown in Figure 1, the model assigns pseudolabels to unlabeled data based on the model predictions on-the-ﬂy.
These data with pseudo labelswill be taken as auxiliary supervision during training to boost performance.
To further facilitatesemi-supervised learning, the teacher-student framework [42, 46, 43] is incorporated.
The teachermodel, which is the Exponential Moving Average (EMA) of the student model, is responsible for
[Figure]
The model generates pseudo labels by itself and in turn, learns from them the teacher student
The teacher model is responsible for generating pseudo labels while the student model learns from the pseudo labels and the ground-truth labels simultaneously.
Knowledge Transmission is conducted between the two models via Exponential Moving Average (
) of all parameters method attaches a gentle teaching assistant (GTA) module to the teacher-student framework.
Different from the original one in (b), the gentle teaching assistant (GTA) learns from the pseudo labels while the student model only learns from ground-truth labels.
We design the representation knowledge transmission between the GTA and student to mitigate the negative inﬂuence caused by unreliable pseudo labels.
generating smoothly updated pseudo labels.
Via jointly supervised by limited data with ground-truthlabels and abundant data with pseudo labels, the student model can learn more representative features,leading to signiﬁcant performance gains.
Although shown to be effective, the pseudo labeling paradigm suffers from unreliable pseudo labels,leading to inaccurate mask predictions.
Previous research work alleviates this problem by ﬁlteringout predictions that are lower than a threshold of classiﬁcation scores [3, 39, 50].
However, thismechanism can not perfectly ﬁlter out wrong predictions, because some wrong predictions may havehigh classiﬁcation scores, named over-conﬁdence or mis-calibration [17] phenomenon.
Moreover, ahigh threshold will heavily reduce the number of generated pseudo labels, limiting the effectivenessof semi-supervised learning.
Towards the aforementioned challenge, it is necessary to propose a new pseudo labeling paradigmthat can learn representative features from unlabeled data as well as avoid negative inﬂuences causedby unreliable pseudo labels.
Delving into the semantic segmentation framework, it is composedof a feature extractor and a mask predictor.
Previous works ask the feature extractor and the maskpredictor to learn from both ground-truth labels and pseudo labels simultaneously.
As a result, theaccuracy of the model is harmed by incorrect pseudo labels.
To better leverage the unlabeled datawith pseudo labels, a viable solution is to let the feature extractor learn feature representation fromboth ground-truth labels and pseudo labels, while the mask predictor only learns from ground-truthlabels to predict accurate segmentation results.
Accordingly, we propose a novel framework, Semi-Supervised Semantic Segmentation via Gentle Teaching Assitant (GTA-Seg) , which attaches an additional gentle teaching assistant (GTA)module to the original teacher-student framework.
Figure 1 compares our method with previousframeworks.
In our method, the teacher model generates pseudo labels for unlabeled data and thegentle teaching assistant (GTA) learns from these unlabeled data.
Only knowledge of the featureextractor in the gentle teacher assistant (GTA) is conveyed to the feature extractor of the studentmodel via Exponential Moving Average (EMA).
We coin this process as representation knowledgetransmission .
Meanwhile, the student model also learns from the reliable ground-truth labels tooptimize both the feature extractor and mask predictor.
The gentle teaching assistant (GTA) is calledgentle since it not only transfers the beneﬁcial feature representation knowledge to the student model,but also protects the student model from the negative inﬂuences caused by unreliable pseudo labels inthe mask predictor.
Furthermore, a re-weighting mechanism is further adopted for pseudo labels tosuppress unreliable pixels.
Extensive experiments have validated that our method shows competitive performance on mainstreambenchmarks, proving that it can make better utilization of unlabeled data.
In addition, we can observefrom the visualization results that our method boasts clearer contour and more accurate classiﬁcationfor objects, which indicates better segmentation performance.
Related Work
Semantic Segmentation Semantic Segmentation, aiming at predicting the label of each pixel inthe image, is one of the most fundamental tasks in computer vision.
In order to obtain the densepredictions, FCN [29] replaces the original fully-connected layer in the classiﬁcation model withconvolution layers.
The famous encoder-decoder structure is borrowed to further reﬁne the pixel-leveloutputs [34, 2].
Meanwhile, intensive efforts have been made to design network components that aresuitable for semantic segmentation.
Among them, dilated convolution [48] is proposed to enhancereceptive ﬁelds, global and pyramid pooling [28, 5, 52] are shown to be effective in modeling contextinformation, and various attention modules [51, 53, 14, 21, 41] are adopted to capture the pixelrelations in images.
These works mark milestones in this important computer vision task, but theypay rare attention to the data-scarce scenarios.
Semi-Supervised Learning Mainstream methods in Semi-Supervised Learning [55] (SSL) fallinto two lines of work, self-training [16, 27] and consistency reguralization [26, 38, 33, 45, 42].
Thecore spirit of self-training is to utilize the model predictions to learn from unlabeled data.
Pseudo
Labeling [27], which converts model predictions on unlabeled data to one-hot labels, is a widely-usedtechnique [3, 39, 50] in semi-supervised learning.
Another variant of self-training, entropy minimization [37], is also proved to be effective both theoretically [44] and empirically [16].
Consistency
Regularization [38, 45] forces the model to obtain consistent predictions when perturbations areimposed on the unlabeled data.
Some recent works unveil that self-training and consistency regularization can cooperate harmoniously.
MixMatch [3] is a pioneering holistic method and boasts remarkableperformance.
On the basis of MixMatch, Fixmatch [39] further simplify the learning process while
FlexMatch [50] introduces a class-wise conﬁdence threshold to boost model performance.
Semi-Supervised Semantic Segmentation Semi-Supervised Semantic Segmentation aims at pixellevel classiﬁcation.
Borrowing the spirit of Semi-Supervised Learning, self-training and consistencyregularization gives birth to various methods.
One line of work [56, 7, 20, 43] applies pseudolabeling in self-training to acquire auxiliary supervision, while methods based on consistency [32]pursue stable outputs at both feature [25, 54] and prediction level [36].
Apart from them, Generative
Adversarial Networks (GANs) [15] or adversarial learning are often leveraged to provide additionalsupervision in relatively early methods [40, 22, 31, 24].
Various recent methods tackles this problemfrom other perspectives, such as self-correcting networks [23] and contrastive learning [1].
Amongthem, some works [49] unveil another interesting phenomenon that the most fundamental trainingparadigm, equipped with strong data augmentations, can serve as a simple yet effective baseline.
Inthis paper, we shed light on semi-supervised semantic segmentation based on pseudo labeling andstrives to alleviate the negative inﬂuence caused by noisy pseudo labels.
Method
3.1 Preliminaries
Semi-Supervised Semantic Segmentation In Semi-Supervised Semantic Segmentation, wetrain a model with limited labeled data D l = { x li , y li } N l
i
=1 and a large amount of unlabeled data
D u = { x ui } N u i =1 , where N u is often much larger than N l .
The semantic segmentation network iscomposed of the feature extractor f and the mask predictor g .
The key challenge of Semi-Supervised
Semantic Segmentation is to make good use of the numerous unlabeled data.
And one commonsolution is pseudo labeling [27, 47].
Pseudo Labeling Pseudo Labeling is a widely adopted technique for semi-supervised segmentation,which assigns pseudo labels to unlabeled data according to model predictions on-the-ﬂy.
Assuming there are K categories, considering the j th pixel on the i th image, the model prediction p uij and thecorresponding conﬁdence c uij will be
[Equation]
where k denotes the k  th category, larger c uij indicates that the model is more certain on this pixel,which is consequently, more suitable for generating pseudo labels.
Speciﬁcally, we often keep thepixels whose conﬁdence value is greater than one threshold, and generate pseudo labels as
[Equation]
where  t is the conﬁdence threshold at the t iteration.
We note that  t can be a constant or a variedvalue during training.
The j th pixel on i th image with a conﬁdence value larger than  t will beassigned with pseudo label ˆ y uij .
The unlabeled data that are assigned with pseudo labels will be takenas auxiliary training data, while the other unlabeled data will be ignored.
Teacher-Student Framework Teacher-Student [9, 42, 43] framework is a currently widely appliedparadigm in Semi-Supervised Segmentation, which consists of one teacher model and one studentmodel.
The teacher model is responsible for generating pseudo labels while the student model learnsfrom both the ground-truth labels and pseudo labels.
Therefore, the loss for the student model is
[Equation]
In Semi-Supervised Semantic Segmentation, L l and L u are the cross-entropy loss on labeled dataand unlabeled data with pseudo labels, respectively [43], and µ is a loss weight to adjust the trade-offbetween them.
The optimization of the student model can be formulated as
[Equation]
where  denotes the learning rate.
In the Teacher-Student framework, after the parameters of thestudent model are updated, the parameters of the teacher model will be updated by the studentparameters in an Exponential Moving Average (EMA) manner.
[Equation]
where ✓ teacher ( t ) and ✓ student ( t ) denote the parameters of the teacher and student model at t -thiteration, respectively.
↵ is a hyper-parameter in EMA, where ↵ 2 [0 , 1] .
3.2 Gentle Teaching Assistant
In this section, we will introduce our Gentle Teaching Assistant framework for semi-supervisedsemantic segmentation (GTA-Seg), as shown in Figure 2, which consists of the following three steps.
Step 1: Pseudo Label Generation and Re-weighting.
Similar to previous work [43], the teachermodel is responsible for generating pseudo labels.
A conﬁdence threshold is also adopted to ﬁlter outthe pseudo labels with low conﬁdence.
For the kept pixels, instead of treating all of them equally, wepropose a re-weighting mechanism according to the conﬁdence of each pixel as follows,
[Equation]
In our re-weighting strategy, the pixel with higher conﬁdence will be highlighted while the other willbe suppressed.
As a result, the negative inﬂuence caused by unreliable pseudo labels can be further
FeatureExtractor
FeatureExtractor FeatureExtractor
Predictor Predictor
Predictor
CrossEntropy
Re-weightedpseudolabel
Backward Forward
Student
Teaching Assistant Teacher
Pseudo Label
EMA EMA
EMA
Ground Truth Label
Unlabeled Data
Labeled Data
Unlabeled Data ① ② ③ ③ ①
Representation Knowledge Transmission ③ ② Gentle Teaching Assistant $ !"
#$ $ ""#$
Student $ !
#&'()# $ "%#&'()#
Teacher $ !
#($*+(, $ "#($*+(, alleviated.
We adopt Laplace Smoothing [30] to avoid over penalization where ⌧ is a predeﬁnedcoefﬁcient.
With this re-weighting mechanism, the unsupervised loss on unlabeled data becomes
[Equation]
Step 2: Representation Knowledge Transmission via Gentle Teaching Assistant (GTA).
Gentle
Teaching Assistant (GTA) plays a crucial role in our framework.
Previous works force the studentmodel to learn from both labeled and unlabeled data simultaneously.
We argue that it is dangerous totreat ground-truth labels and pseudo labels equally since the incorrect pseudo labels will mislead themask prediction.
Therefore, we want to disentangle the effects of pseudo labels on feature extractorand mask predictor of the student model.
Concretely, our solution is to introduce one additionalgentle teaching assistant, which learns from the unlabeled data and only transfers the beneﬁcialfeature representation knowledge to the student model, protecting the student model from the negativeinﬂuences caused by unreliable pseudo labels.
After optimized on unlabeled data with pseudo labels as in Eq.
8, the gentle teaching assistant modelis required to convey the learned representation knowledge in feature extractor to the student modelvia Exponential Moving Average (EMA) as in Eq.
[Equation]
[Equation] [Equation]
li , y li } N l
i =1 , u = { ui u i where ✓ gta ( t ) is the parameters of the gentle teaching assistant model at t -th iteration, ✓ student ( t ) isthe parameters of the student model at t -th iteration, and ✓ f denotes the parameters of the featureextractor.
Through our representation knowledge transmission, the unlabeled data is leveraged tofacilitate feature representation of the student model, but it will not train the mask predictor.
Step 3:
Optimize student model with ground truth labels and update teacher model.
With thegentle teaching assistant module, the student model in our framework is only required to learn fromthe labeled data,
[Equation]
Here, the whole model, including the feature extractor as well as the mask predictor, is updatedaccording to the supervised loss computed by the ground-truth labels of labeled data.
Then the teacher model is updated by taking the EMA of the student model according to the traditionalparadigm in the teacher-student framework.
[Equation]
[Equation]
Finally, the teacher model, which absorbs the knowledge of both labeled and unlabeled data from thestudent model, will be taken as the ﬁnal model for inference.
Experiment
4.1 Datasets
We evaluate our method on 1) PASCAL VOC 2012 [11]: a widely-used benchmark dataset forsemantic segmentation, with 1464 images for training and 1449 images for validation.
Someresearches [7, 47] augment the training set by incorporating the 9118 coarsely annotated imagesin SBD [18] to the original training set, obtaining 10582 labeled training images, which is calledthe augmented training set.
In our experiments, we consider both the original training set and theaugmented training set, taking 92 , 183 , 366 , 732 , and 1464 images from the 1464 labeled images inthe original training set, and 662 , 1323 and 2645 images from the 10582 labeled training images in
[Table]
[Table]
The notations and network architecture are the same as in Table 1.
* means that we reimplement with ResNet
101 backbone for a
[Table]
the augmented training set.
2) Cityscapes [8], a urban scene dataset with 2975 images for trainingand 500 images for validation.
We sample 100 , 186 , 372 , 744 images from the 2975 labeled imagesin the training set.
We take the split in [56] and report all the performances in a fair comparison.
4.2 Implementation Details
We take ResNet-101 [19] pre-trained on ImageNet [10] as the network backbone and DeepLabv3+ [6]as the decoder.
The segmentation head maps the 512-dim features into pixel-wise class predictions.
We take SGD as the optimizer, with an initial learning rate of 0.001 and a weight decay of 0.0001 for
PASCAL VOC.
The learning rate of the decoder is 10 times of the network backbone.
On Cityscapes,the initial learning rate is 0.01 and the weight decay is 0.0005.
Poly scheduling is applied to thelearning rate with lr = lr init · (1  tT ) 0 .
9 , where lr init is the initial learning rate, t is the currentiteration and T is the total iteration.
We take 4 GPUs to train the model on PASCAL VOC, and8 GPUs on Cityscapes.
Following previous works, input images are center cropped in PASCAL
VOC during evaluation, while on Cityscapes, sliding window evaluation is adopted.
The mean of
Intersection over Union (mIoU) measured on the validation set serves as the evaluation metric.
4.3 Experimental Results
PASCAL VOC
We ﬁrst evaluate our method on the original training set of PASCAL VOC2012.
The results in Table 1 validate that our method surpasses previous methods by a large margin.
Speciﬁcally, our method improves the supervised-only (SupOnly) model by 24 .
24 , 9 . 69 , 6 . 68 ,7 . 97 in mIoU
when 0 .
9% of the data is labeled, respectively.
Whencompared to the readily strong semi-supervised semantic segmentation method, our method stillsurpasses it by 13 .
32 , 6 . 32 respectively.
We note that in the original training set, theratio of labeled data is relatively low ( 0 . 9% to 13 .
Therefore, the results verify that our methodis effective in utilizing unlabeled data in semi-supervised semantic segmentation.
We further compare our method with previous methods on the augmented training set of PASCAL
VOC 2012, where the annotations are relatively low in quality since some of labeled images come
[Table] [Table]
[Table] [Table]
from SBD [18] dataset with coarse annotations.
We can observe from Table
[Table]
our method consistentlyoutperforms the previous methods in a fair comparsion.
Cityscapes For Cityscapes, as shown in Table 3, our method still shows competitive performanceamong previous methods, improving the existing state-of-the-art method by 1 . 80 , 1 . 64 , 1 . 57 , 2 . 19 inmIoU when 3 . 3% , 6 . 25% , 12 .
5% , 25 . 0% of the data is labeled.
4.4 Analyses
Component Analysis We analyze the effectiveness of different components in our method,
i
[Table]
e. , theoriginal teacher-student framework, gentle teaching assistant and re-weighted pseudo labeling as in
Table 4.
According to the results in Table 4, the carefully designed gentle teaching assistant mechanism (the third row) helps our method outperform the previous methods, pushing the performanceabout 13 .
1 higher than the original teacher-student model (the second row).
Further, the re-weightedpseudo labeling brings about 1 .
1 performance improvements.
With all of these components, ourmethod outperforms the teacher-student model by over 14 . 0 and SupOnly by over 18 .
in mIoU.
Gentle Teaching Assistant As mentioned in Table 4, our proposed gentle teaching assistantframework brings about remarkable performance gains.
Inspired by this, we delve deeper into thegentle teaching assistant model in our framework.
We ﬁrst consider the representation knowledgetransmission mechanism.
In Table 5, we compare our mechanism with other methods such as theoriginal EMA [42] that updates all of the parameters via EMA and Unbiased ST [4] that introducesan additional agent to convey representation knowledge.
We can observe that all these mechanismsboost SupOnly remarkably, while our mechanism is superior to other methods.
We next pay attention to the three models in our framework, i.e. gentle teaching assistant model,student model, and teacher model.
Table 6 reports the evaluation performance of them.
All of themshow relatively competitive performance.
For the teacher assistant model, it is inferior to the studentmodel.
This is reasonable since it is only trained on pseudo labels, while the student model inheritsthe representation knowledge of unlabeled data from the gentle teaching assistant as well as trainedon labeled data.
In addition, the teacher model performs best, which agrees with previous works [42].
Method Design In our method, we train GTA with pseudo labels and the student model withlabeled data.
It is interesting to explore the model performance of other designs.
Table 10 showsthat 1) train the student model with pseudo labels will cause signiﬁcant performance drop, which isconsistent with our statement that the student model shall not learn from the pseudo labels directly.
Incorporating labeled data in training GTA is not beneﬁcial to model performance.
We conjecturethat when we transmit the knowledge of labeled data from GTA to the student model, as well assupervised the student model with labeled data, the limited labels overwhelm the updating of the
[Figure]
student model, leading to overﬁtting and harming the student model’s performance.
And since theteacher model is purely updated by the student model via EMA, the performance of the teacher modelis also harmed.
Since the ultimate goal is a higher performance of the teacher model, we choose totrain GTA with only pseudo labels.
Visualization Besides quantitative results, we present the visualization results to further analyzeour method.
We note that the model is trained on as few as 183 labeled samples and about 10400unlabeled samples.
As shown in Figure 3, facing such limited labeled data, training the modelmerely in the supervised manner (SupOnly) appears to be vulnerable.
Under some circumstances,the model is even ignorant of the given images (the third and the fourth row).
While methods thatutilize unlabeled data (teacher-student model and our method), show stronger performance.
Further,compared with the original teacher-student model, our method shows a stronger ability in determininga clear contour of objects (the ﬁrst row) and recognizing the corresponding categories (the secondrow).
Our method is also superior to previous methods in distinguishing objects from the background(the third and fourth row).
Limitations One limitation of our method is that it brings about more training costs since itincorporates an extra gentle teaching assistant model.
Fortunately, the inference efﬁciency is notinﬂuenced since only the teacher model is taken for inference.
On the other hand, our method onlyattempts at making better use of the unlabeled data, but little attention has been paid to the labeleddata.
We consider it promising to conduct research on how to better leverage the labeled data insemi-supervised semantic segmentation.
Conclusion
In this paper, we propose a novel framework, Gentle Teaching Assistant, for semi-supervised semanticsegmentation (GTA-Seg).
Concretely, we attach an additional teaching assistant module to disentanglethe effects of pseudo labels on the feature extractor and the mask predictor.
GTA learns representationknowledge from unlabeled data and conveys it to the student model via our carefully designedrepresentation knowledge transmission.
Through this framework, the model optimizes representationwith unlabeled data, as well as prevents it from overﬁtting on limited labeled data.
A conﬁdencebased pseudo label re-weighting mechanism is applied to further boost the performance.
Extensiveexperiment results prove the effectiveness of our method.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 4.4(c)
Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] We promise to
release the code and models upon the paper accepted.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4.1(b)
Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No](d) Did you discuss whether
and how consent was obtained from people whose data you’reusing/curating?
[No]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[No]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]