Original: S EMI - SUPERVISED S EMANTIC S EGMENTATION USING A UXILIARY N ETWORK
Revised: S EMI - SUPERVISED S EMANTIC S EGMENTATION USING A UXILIARY N ETWORK

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recently, the convolutional neural networks (CNNs) have shown great success on semantic segmentation task.
Revised: Recently, the convolutional neural networks (CNNs) have shown great success on semantic segmentation task.

Original: However, for practical applications such as autonomous driving, the popular supervised learning method faces two challenges: the demand of low computational complexity and the need of huge training dataset accompanied by ground truth.
Revised: However, for practical applications such as autonomous driving, the popular supervised learning method faces two challenges: the demand of low computational complexity and the need of huge training dataset accompanied by ground truth.

Original: Our focus in this paper is semi-supervised learning.
Revised: Our focus in this paper is semi-supervised learning for semantic segmentation.

Original: We wish to use both labeled and unlabeled data in the training process.
Revised: We wish to use both labeled and unlabeled data in the training process.

Original: A highly efﬁcient semantic segmentation network is our platform, which achieves high segmentation accuracy at low model size and high inference speed.
Revised: A highly efﬁcient semantic segmentation network is our platform, which achieves high segmentation accuracy with a low model size and high inference speed.

Original: We propose a semi-supervised learning approach to improve segmentation accuracy by including extra images without labels.
Revised: We propose a semi-supervised learning scheme to improve segmentation accuracy by including extra images without labels.

Original: While most existing semi-supervised learning methods are designed based on the adversarial learning techniques, we present a new and different approach, which trains an auxiliary CNN network that validates labels (ground-truth) on the unlabeled images.
Revised: While most existing semi-supervised learning methods are designed based on the adversarial learning techniques, we present a new and different approach, which trains an auxiliary CNN network that validates labels (ground-truth) on the unlabeled images.

Original: In the supervised training phase, both the segmentation network and the auxiliary network are trained using  labeled images.
Revised: In the supervised training phase, both the segmentation network and the auxiliary network are trained using the labeled images.

Original: Then, in the unsupervised training phase, the unlabeled images are segmented and a subset of image pixels are picked up by the auxiliary network; and then they are used as ground truth to train the segmentation network.
Revised: Then, in the unsupervised training phase, the unlabeled images are segmented and a subset of image pixels are picked up by the auxiliary network; and then they are used as ground truth to train the segmentation network.

Original: Thus, at the end, all dataset images can be used for retraining the segmentation network to improve the segmentation results.
Revised: Thus, at the end, all dataset images can be used for retraining the segmentation network to improve the segmentation results.

Original: We use Cityscapes and CamVid datasets to verify the effectiveness of our semi-supervised scheme, and our experimental results show that it can improve the mean IoU for about 1.2% to 2.9% on the challenging Cityscapes dataset.
Revised: We use Cityscapes and CamVid datasets to verify the effectiveness of our semi-supervised scheme. Our experimental results show that it can improve the mean IoU for about 1.2% to 2.9% on the challenging Cityscapes dataset.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Semantic Segmentation, which identiﬁes the category label of each pixel, is an important task in computer vision.
Revised: Semantic Segmentation, which identiﬁes the category label of each pixel, is an important task in computer vision.

Original: A number of convolutional neural network (CNN) based semantic segmentation systems have been developed in recent years such as Chen et al.
Revised: A number of convolutional neural network (CNN) based semantic segmentation systems have been developed in recent years such as Chen et al.

Original: (2017a;b; 2018); Long et al. (2015);
Revised: (2017a;b; 2018); Long et al. (2015);

Original: Zhao et al.
Revised: Zhao et al.

Original: ; Ronneberger et al. (2015).
Revised: ; Ronneberger et al. (2015).

Original: For practical applications such as autonomous driving, there are a high demand for real-time processing and sufﬁcient training data.
Revised: For practical applications such as autonomous driving, there is a high demand for real-time processing and sufﬁcient training data.

Original: Hence, different research directions have been explored.
Revised: Hence, different research directions have been explored.

Original: For example, some  proposed various efﬁcient network structures (Chen et al., 2019; Paszke et al., 2016; Poudel et al., 2018;
Revised: For example, some researchers proposed various efﬁcient network structures (Chen et al., 2019; Paszke et al., 2016; Poudel et al., 2018;

Original: Lo et al., 2018; Poudel et al., 2019; 
Revised: Lo et al., 2018; Poudel et al., 2019; Yu et al., 2018;

Original: Yu et al., 2018; Zhao et al., 2018; Romera et al., 2017; Li et al., 2019), and  others focus on the weakly- or semi-supervised learning schemes (Hung et al., 2018;
Revised: Zhao et al., 2018; Romera et al., 2017; Li et al., 2019), and the others focus on the weakly- or semi-supervised learning schemes (Hung et al., 2018;

Original: Lin et al., 2016; Bearman et al., 2016;
Revised: Lin et al., 2016; Bearman et al., 2016;

Original: Qi et al., 2016; Rajchl et al., 2016; Pathak et al., 2015; Papandreou et al., 2015; Hong et al., 2015; Luc et al., 2016; Dai et al., 2015; Pathak et al., 2014).
Revised: Qi et al., 2016; Rajchl et al., 2016; Pathak et al., 2015; Papandreou et al., 2015; Hong et al., 2015; Luc et al., 2016; Dai et al., 2015; Pathak et al., 2014; Zhu et al., 2019; Mittal et al., 2019).

Original: In this study, we ﬁrst design an efﬁcient segmentation network inspired by Chen et al.
Revised: In this study, we ﬁrst design an efﬁcient segmentation network inherited from Chen et al.

Original: as our baseline model, and then propose a semi-supervised learning scheme.
Revised: as our baseline model, and then propose a semi-supervised learning scheme.

Original: There are three training stages in our system.
Revised: There are three training stages in our system.

Original: In the ﬁrst stage, the segmentation network is trained using the labeled images.
Revised: In the ﬁrst stage, the segmentation network is trained using the labeled images.

Original: Then, in the second stage, an auxiliary network is trained using the segmentation network  trained in the previous stage and the labeled data to generate the conﬁdence map.
Revised: Then, in the second stage, an auxiliary network is trained using the segmentation network model trained in the previous stage and the labeled data to generate the conﬁdence map.

Original: Inspired by Hung et al.
Revised: Inspired by Hung et al.

Original: (2018), we use the concept of conﬁdence map to assign a conﬁdence score to each segmented pixel on the unlabeled images.
Revised: (2018), we use the concept of conﬁdence map to assign a conﬁdence score to each segmented pixel on the unlabeled images.

Original: In their work (Hung et al., 2018), the conﬁdence map is the output of a  discriminator networktrained by the GAN framework (Goodfellow et al., 2014), where the discriminator network learns to distinguish between the segmentation map and the ground truth map.
Revised: In their work (Hung et al., 2018), the conﬁdence map is the output of a GAN discriminator network (Goodfellow et al., 2014), where the discriminator network learns to distinguish between the segmentation map and the ground truth map.

Original: However, we ﬁnd that the trusted (high conﬁdence) regions in their conﬁdence map are mostly located on the large target objects, and thus the effectiveness of semi-supervised learning is limited.
Revised: However, we ﬁnd that the trusted (high conﬁdence) regions in their conﬁdence map are mostly located on the large target objects, and thus the effectiveness of semi-supervised learning is limited.

Original: To obtain more reliable small object labels, we adopt a different approach in generating the conﬁdence map.
Revised: To obtain more reliable small object labels, we adopt a different approach in generating the conﬁdence map.

Original: In our approach, the conﬁdence map is generated by an auxiliary (CNN) network, which is trained using the proposed auxiliary loss function.
Revised: In our approach, the conﬁdence map is generated by an auxiliary (CNN) network, which is trained using the proposed auxiliary loss function.

Original: We carefully design the auxiliary loss function such that it leads to a reliable conﬁdence map, particularly, on the small objects.
Revised: We carefully design the auxiliary loss function such that it leads to a reliable conﬁdence map, particularly, on the small objects.

Original: In the third stage, the unlabeled images are used as inputs to the proposed system to generate the labels.
Revised: In the third stage, the unlabeled images are used as inputs to the proposed system to generate the labels.

Original: Therefore, we can use both originally labeled and unlabeled images to retrain the segmentation network  to achieve a better performance in the end.
Revised: Therefore, we can use both originally labeled and unlabeled images to retrain the segmentation network model to achieve a better performance in the end.

Original: In summary, our main contributions of this work are as follows.
Revised: In summary, our main contributions of this work are as follows.

Original: First, Based on DSNet-fast (Chen et al., 2019), we design a powerful segmentation network, which achieves a very good balance between speed and accuracy.
Revised: First, Based on DSNet-fast (Chen et al., 2019), we design a powerful segmentation network, which achieves a very good balance between speed and accuracy.

Original: It produces 73.9% mean IoU on the Cityscapes testing set with a speed of 73 FPS on a single GTX 1080Ti card.
Revised: It produces 73.9% mean IoU on the Cityscapes testing set with a speed of 73 FPS on a single GTX 1080Ti card.

Original: Second, We propose a semi-supervised learning scheme with the notion of auxiliary network , which can be used to annotate the unlabeled images.
Revised: Second, We propose a semi-supervised learning scheme with the notion of auxiliary network , which can be used to annotate the unlabeled images.

Original: Third, Our semi-supervised learning method can include  unlabeled images in training, which improves the segmentation accuracy.
Revised: Third, with the help of auxiliary network, our semi-supervised learning method can include the unlabeled images in training, which improves the segmentation accuracy.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Recently, CNNs have been widely used in many ﬁelds of computer vision.
Revised: Recently, CNNs have been widely used in many ﬁelds of computer vision.

Original: For the semantic segmentation task, FCN (Long et al., 2015) is a pioneer.
Revised: For the semantic segmentation task, FCN (Long et al., 2015) is a pioneer.

Original: It replaces the fully-connected layers of the classiﬁcation network (Krizhevsky et al., 2012) with  convolution layers, and thus it can generate dense label prediction of the same size as input image.
Revised: It replaces the fully-connected layers of the classiﬁcation network (Krizhevsky et al., 2012) with the convolution layers, and thus it can generate dense labels on the input image of the same size.

Original: After that, SegNet (Badrinarayanan et al., 2017) is subsequently proposed, which uses a symmetric encoder-decoder architecture for feature  down-sampling and up-sampling.
Revised: Then, SegNet (Badrinarayanan et al., 2017) was subsequently proposed, which uses a symmetric encoder-decoder architecture for feature map down-sampling and up-sampling.

Original:  U-Net (Ronneberger et al., 2015) introduces the concatenation operation to up-sample the features with different levels.
Revised: The U-Net (Ronneberger et al., 2015) introduces the concatenation operation to up-sample the features with different levels.

Original: PSPNet (Zhao et al., 2017) and DeepLab (Chen et al., 2017a;b; 2018) propose the atrous spatial pyramid pooling (ASPP) module to integrate multi-scale features.
Revised: PSPNet (Zhao et al., 2017) and DeepLab (Chen et al., 2017a;b; 2018) propose the atrous spatial pyramid pooling (ASPP) module to integrate multi-scale features.

Original: There are many other studies on improving the segmentation results; however, for practical applications, a high inference speed network is very desirable.
Revised: There are many other studies on improving the segmentation results; however, for practical applications, a high inference speed network is very desirable.

Original: For example, ENet (Paszke et al., 2016) is a real-time segmentation model with good segmentation results.
Revised: For example, ENet (Paszke et al., 2016) is a real-time segmentation model with good segmentation results.

Original: ICNet (Zhao et al., 2018) and BiSeNet (Yu et al., 2018) were recently proposed, and they aim at a better balance between speed and accuracy.
Revised: ICNet (Zhao et al., 2018) and BiSeNet (Yu et al., 2018) were recently proposed, and they aim at a better balance between speed and accuracy.

Original: Another challenge in training a semantic segmentation network is the need of a large amount of labeled data (ground truth).
Revised: Another challenge in training a semantic segmentation network is the need of a large amount of labeled data (ground truth).

Original: The labeling cost of pixel-level annotation is extremely expensive.
Revised: The labeling cost of pixel-level annotation is extremely expensive.

Original: Hence, in recent years, the study of the weakly- and semi-supervised learning approachesbecome popular to tackle this problemand attracted a lot of attention.
Revised: Hence, in recent years, the study of the weakly- and semi-supervised learning approaches to tackle this problem attracted a lot of attention.

Original: For the weakly supervised methods,  pixel-level annotations are no longer used to train the segmentation network, but instead, other forms of annotations that are easier to obtain are used for training.
Revised: For the weakly supervised methods, the pixel-level annotation is no longer used to train the segmentation network, but instead, other forms of annotations obtained at low costs are used for training.

Original: There are several different types of weakly-supervision have been studied, including bounding box supervision (Rajchl et al., 2016; Dai et al., 2015), scribbles supervision (Lin et al., 2016), point supervision (Bearman et al., 2016), and image-level supervision (Qi et al., 2016; Pathak et al., 2014).
Revised: Several different types of weakly-supervision have been studied, including bounding box supervision (Rajchl et al., 2016; Dai et al., 2015), scribbles supervision (Lin et al., 2016), point supervision (Bearman et al., 2016), and image-level supervision (Qi et al., 2016; Pathak et al., 2014).

Original: Particularly, the development of image-level supervision is the most popular one.
Revised: Particularly, the development of image-level supervision is the most popular one.

Original: Pathak et al.
Revised: Pathak et al.

Original: (2015) convert image-level labels to restrict the distribution of CNN output.
Revised: (2015) convert image-level labels to restrict the distribution of CNN output.

Original: Papandreou et al.
Revised: Papandreou et al.

Original: (2015) combine image-level labels with EM algorithms to train the segmentation model.
Revised: (2015) combine image-level labels using EM algorithms to train the segmentation model.

Original: Wei et al.
Revised: Wei et al.

Original: (2018) propose a generic classiﬁcation network, which adopts the convolutional blocks with different dilated rates to generate dense object localization maps.
Revised: (2018) propose a generic classiﬁcation network, which adopts the convolutional blocks with different dilated rates to generate dense object localization maps.

Original: It can produce reliable segmentation masks for training the segmentation model.
Revised: It can produce reliable segmentation masks for training the segmentation model.

Original: For semi-supervised learning, Hong et al.
Revised: For semi-supervised learning, Hong et al.

Original: (2015) propose a method to separately train the classiﬁcation and segmentation networks, and then pass the information from the classiﬁcation network to the segmentation network to reduce the search space for effective segmentation.
Revised: (2015) propose a method to separately train the classiﬁcation and segmentation networks, and then pass the information from the classiﬁcation network to the segmentation network to reduce the search space for effective segmentation.

Original: Moreover, Luc et al.
Revised: Luc et al.

Original: (2016) employ an adversarial network to enhance the segmentation quality.
Revised: (2016) employ an adversarial network to enhance the segmentation quality.

Original: Hung et al.
Revised: Recently, Zhu et al.

Original: (2018) propose a self-taught learning approach based on the adversarial network.
Revised: (2019) propose a video prediction model to generate new samples for improving segmentation accuracy.

Original: Inspired by Hung et al.
Revised: Moreover Hung et al.

Original: (2018), we propose an auxiliary network, which can determine the credibility of each pixel on the segmentation map of unlabeled images and  the trusted pixels are used as  ground truth  in training.
Revised: (2018) propose a self-taught learning approach based on the dense pixel-level probability maps produced by adversarial network. Mittal et al. (2019) describe the dual-branch method including a semi-supervised classiﬁcation branch to learn from unlabeled data. In this study, we propose an auxiliary network, which can determine the credibility of each pixel on the segmentation map of unlabeled images and then the trusted pixels can be used as additional ground truth data in training.

Revised: Different from the previous works, we focus on the quality of the generated labels.

Original:  In designing the auxiliary network, we propose an auxiliary loss function containing two vital terms to address the small objects of segmentation map; we believe that the supervision signals of small objects are more important than the bigger one.
Revised: We believe that the supervision signals of small objects are more important than the larger ones. Thus, we assign different weights to the generated tags of different categories, especially biased towards the small objects. In training the auxiliary network, we propose an auxiliary loss function containing two critical terms to achieve this objective.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: In this section, we ﬁrst describe the general framework of our semi-supervised semantic segmentation scheme, and then we describe the architecture as well as design concepts of the proposed segmentation network and auxiliary network.
Revised: In this section, we ﬁrst describe the general framework of our semi-supervised semantic segmentation scheme, and then we describe the architecture and design concepts of the proposed segmentation network and auxiliary network.

Original: 3. S EMI - SUPERVISED LEARNING USING AUXILIARY NETWORK
Revised: 3. S EMI - SUPERVISED LEARNING USING AUXILIARY NETWORK

Original: There are in total three training stages in our semi-supervised learning scheme.
Revised: There are in total three training stages in our semi-supervised learning scheme.

Original: In the ﬁrst stage, we simply train the segmentation network using  labeled images.
Revised: In the ﬁrst stage, we simply train the segmentation network using the labeled images.

Original: In this paper, we only use the typical cross entropy loss in the supervised training of segmentation network.
Revised: In this paper, we simply use the typical cross entropy loss in the supervised training of segmentation network.

Original: Then in the second stage, an auxiliary network is trained using the proposed auxiliary loss function.
Revised: Then in the second stage, an auxiliary network is trained using the proposed auxiliary loss function.

Original: Both the originally labeled images and the previously trained segmentation network are used in this stage.
Revised: Both the originally labeled images and the previously trained segmentation network are used in this stage.

Original: Our auxiliary network takes the class probability maps of size H × W × C produced by the segmentation network as input, where H and W are the height and width of the image size and C denotes the number of classes.
Revised: Our auxiliary network takes the class probability maps of size H × W × C produced by the segmentation network as input, where H and W are the height and width of the image size and C denotes the number of classes.

Original: The auxiliary network outputs a conﬁdence map of size H × W × 1 .
Revised: The auxiliary network outputs a conﬁdence map of size H × W × 1 .

Original: Each pixel of the conﬁdence map is a probability value between 0 and 1, which represents the credibility of that pixel on the segmentation map.
Revised: Each pixel of the conﬁdence map is a probability value between 0 and 1, which represents the credibility of that pixel on the segmentation map.

Original: In the third stage, we feed the unlabeled images into the segmentation network pre-trained in the ﬁrst stage to obtain the class probability map, and then pass this map through the auxiliary network to obtain the conﬁdence map.
Revised: In the third stage, we feed the unlabeled images into the segmentation network pre-trained in the ﬁrst stage to obtain the class probability map, and then pass this map through the auxiliary network to obtain the conﬁdence map.

Original: Based on the probability value of each pixel of the conﬁdence map, we can determine the segmentation reliability of each pixel on the segmentation map.
Revised: Based on the probability value of each pixel of the conﬁdence map, we can determine the segmentation reliability of each pixel on the segmentation map.

Original: Then, we select the pixels with high reliability as the annotated data points.
Revised: Then, we select the pixels with high reliability as the annotated data points.

Original: And we ignore the pixels with lower reliability in the next-phase training.
Revised: And we ignore the pixels with lower reliability in the next-phase training.

Original: We set a threshold T aux on the conﬁdence map to separate the high reliability pixels from the low reliability pixels.
Revised: We set a threshold T aux on the conﬁdence map to separate the high reliability pixels from the low reliability pixels.

Original: Hence, we can mask the error-prone pixels on the output segmentation map and produce the  annotated maps for the unlabeled images.
Revised: Hence, we can mask the error-prone pixels on the output segmentation map and produce the reliable annotated maps for the unlabeled images.

Original: Finally, since all  images now have labels, we retrain the segmentation network using all of them, and produce a more accurate segmentation model.
Revised: Finally, since all the images now have labels, we retrain the segmentation network using all of them, and produce a more accurate segmentation model.

Original: Figure 1 shows the process for generating annotated labels for  images without ground truth.
Revised: Figure 1 shows the process for generating annotated labels for the images without ground truth.

Original: Loss function for auxiliary network To train the auxiliary network, we create another auxiliary ground truth map of dimension H × W × 1 .
Revised: Loss function for auxiliary network To train the auxiliary network, we create another auxiliary ground truth map of dimension H × W × 1 .

Original: Assume that we have the original labeled ground truth class map and the estimated (segmented) class map produced by the segmentation network. The pixel value on  auxiliary ground truth map is set to 1 if the class at that pixel on the estimated segmentation map matches that on the ground truth class map; and it is set to 0, if not.
Revised: Assuming that we have the original labeled ground truth class map and the estimated (segmented) class map produced by the segmentation network, the pixel value on the auxiliary ground truth map is set to 1 if the class at that pixel on the estimated segmentation map matches that on the ground truth class map; and it is set to 0, if not.

Original: Different from the typical binary cross entropy loss,  the loss function for auxiliary network is deﬁned as:
Revised: Different from the typical binary cross entropy loss, we assign weights to the terms in the loss function for auxiliary network, which is deﬁned below.

Original: [Equation]
Revised: [Equation]

Original: where y nh,w equals to 1 if the estimate class on the segmentation map at pixel ( h, w ) is identical to the class value on the ground truth label map at the same pixel, and y n h,w equals to 0 if they are different.
Revised: where y nh,w equals to 1 if the estimate class on the segmentation map at pixel ( h, w ) is identical to the class value on the ground truth label map at the same pixel, and y nh,w equals to 0 if they are different.

Original: In equation (1), a h,w ( x n ) denotes the conﬁdence map output at pixel ( h, w ), when the auxiliary network takes the class probability map x n as input.
Revised: In equation (1), a h,w ( x n ) denotes the conﬁdence map output at pixel ( h, w ), when the auxiliary network takes the class probability map x n as input.

Original: W nh,w represents the class weighting value at pixel ( h, w ) on the conﬁdence map, and it is deﬁned as 1 / ln( c + p class ) , inspired by ENet (Paszke et al., 2016), where we set c equal to 1.04.
Revised: W nh,w represents the class weighting value at pixel ( h, w ) on the conﬁdence map, and it is deﬁned as 1 / ln( c + p class ) , inspired by ENet (Paszke et al., 2016), where we set c equal to 1.04.

Original: The class weighting at different locations of conﬁdence map is decided by the ground truth label map.
Revised: The class weighting at different locations of conﬁdence map is decided by the ground truth label map.

Original: For example, when the class at the pixel ( h, w ) on ground truth label map isthe class i , the class weight at that pixel on the conﬁdence map is the weighting value associated with class i .
Revised: For example, when the class at the pixel ( h, w ) on ground truth label map is class i , the class weight at that pixel on the conﬁdence map is the weight value associated with class i .

Original: Class weighting term is used to tackle the data imbalance issue.
Revised: Class weighting term is used to tackle the data imbalance issue.

Original: In other words, some classes having few number of pixels are often unfavored in training.
Revised: That is, some classes having few number of pixels are often unfavored in training.

Original: With class weighting, our auxiliary network performs well not only on the big objects but also on  small objects.
Revised: With class weighting, our auxiliary network performs well not only on the big objects but also on the small objects.

Original: Moreover, for different classes, theissue of imbalance  between positive and negative samples are quite different.
Revised: Moreover, for different classes, the imbalance issue between positive and negative samples are quite different.

Original: Thus, we propose a weight term γ nh,w , which is deﬁned as # negative pixel class / # positive pixel class , and it is calculated using the auxiliary network ground truth.
Revised: Thus, we propose a weight term γ n h,w , which is deﬁned as # negative pixel class / # positive pixel class , and it is calculated using the auxiliary network ground truth.

Original: Similarly, this weight term is location variant depending on the ground truth label map.
Revised: Similarly, this weight term is location variant depending on the ground truth label map.

Original: Both
Revised: Both

Original: W nh,w and γ nh,w are the crucial terms in the proposed auxiliary loss function.
Revised: W nh,w and γ nh,w are the critical terms in the proposed auxiliary loss function.

Revised: There were several versions tested and the current version gives the best result.

Original: 3. N ETWORK ARCHITECTURE 3.2. S EGMENTATION NETWORK
Revised: 3. N ETWORK ARCHITECTURE 3.2. S EGMENTATION NETWORK

Original: We adopt the DSNet-fast (Chen et al., 2019) as our backbone model, but we make some modiﬁcations.
Revised: We adopt the DSNet-fast (Chen et al., 2019) as our backbone model, but we made some modiﬁcations.

Original: From our observations, once the receptive ﬁeld of the network is large enough to cover the entire image, make good use of the shallow-layer features is more effective than increasing the network depth for the semantic segmentation task.
Revised: Based on our observations, once the receptive ﬁeld of the network is large enough to cover the entire image, making good use of the shallow-layer features is more effective than increasing the network depth for the semantic segmentation task.

Original: So, we redesign the initial block and the early block of DSNet-fast, and reduce layers in the deeper block to achieve a better accuracy and inference speed.
Revised: Thus, we redesigned the initial block and the early block of DSNet-fast, and reduced the CNN layers in the deeper block to achieve a better accuracy and inference speed.

Original: The overall architecture is shown in Figure 2.
Revised: The overall architecture is shown in Figure 2.

Original: In the encoder part, we adopt the same bottleneck unit and transition layer from DSNet-fast, and propose new initial block and non-bottleneck unit.
Revised: In the encoder part, we adopt the same bottleneck unit and transition layer in DSNet-fast, and propose new initial block and non-bottleneck units.

Original: All our core units are shown in Figure 3.
Revised: All our core units are shown in Figure 3.

Original: In  initial block, after applying  3 × 3 convolution with stride 2 on the input image, we divide it into two branches.
Revised: In the initial block, after applying the 3 × 3 convolution with stride 2 on the input image, we split it into two branches.

Original: The 1 × 1 convolution is applied in left branch to integrate features and reduce the number of channels, and on the right branch, a stack of two 3 × 3 convolutions is applied to learn the features.
Revised: The 1 × 1 convolution is applied to the left branch to integrate features and reduce the number of channels, and on the right branch, a stack of two 3 × 3 convolutions is used to learn the features.

Original: Furthermore, the feature maps of these two branches are concatenated and followed by an average pooling with stride 2 to produce the output.
Revised: Furthermore, the feature maps of these two branches are concatenated and followed by an average pooling with stride 2 to produce the output.

Original: Since  1 × 1 convolution does not take the spatial information into account, the feature maps concatenated from the two branches have different receptive ﬁelds, and thus it helps the network to extract features at different scales.
Revised: Since the 1 × 1 convolution does not take the spatial information into account, the feature map concatenated from the two branches have different receptive ﬁelds, and thus it helps the network to extract features at different scales.

Original: On the other hand, due to dimension reduction made by the 1 × 1 convolution operation, the left branch helps the right branch to produce more feature channels under the constraint of ﬁxed number of concatenated feature maps.
Revised: On the other hand, due to dimension reduction made by the 1 × 1 convolution operation, the left branch helps the right branch to produce more feature channels under the constraint of ﬁxed number of concatenated feature maps.

Original: Similarly, we use the same concept to design our non-bottleneck unit.
Revised: Similarly, we use the same concept to design our non-bottleneck unit.

Original: We ﬁnd that this design strategy is indeed helpful for producing better segmentation results at similar model complexity.
Revised: We ﬁnd that this design strategy is indeed helpful in producing better segmentation results at similar model complexity.

Original: In the decoder part, we employ 3 × 3 convolution with { 24, 32, 16, 24 } channels after the Block1-4, respectively, to reduce the computational complexity.
Revised: In the decoder part, we employ 3 × 3 convolutions with { 24, 32, 16, 24 } channels after the Blocks 1-4, respectively, to reduce the computational complexity.

Original: Furthermore, the up-sampling and concatenation operations are adopted, same as DSNet-fast, to integrate the feature maps at different  levels.
Revised: Furthermore, the up-sampling and concatenation operations are adopted, same as DSNet-fast, to integrate the feature maps at different spatial levels.

Original: Finally, the concatenated feature map is up-sampled by a factor of 4 using the deconvolution kernel to produce the estimated segmentation map.
Revised: Finally, the concatenated feature map is up-sampled by a factor of 4 using a deconvolution kernel to produce the estimated segmentation map.

Original: All the convolution layers are followed by batch normalization (Ioffe & Szegedy, 2015) and ReLU activations.
Revised: All the convolution layers are followed by batch normalization (Ioffe & Szegedy, 2015) and ReLU activations.

Original: 3.2. A UXILIARY NETWORK
Revised: 3.2. A UXILIARY NETWORK

Original: Our auxiliary network mainly consists of ﬁve convolution layers, one atrous spatial pyramid pooling (ASPP) module, and one deconvolution layer.
Revised: Our auxiliary network mainly consists of ﬁve convolution layers, one atrous spatial pyramid pooling (ASPP) module, and one deconvolution layer.

Original: The overall architecture is shown in Table 1.
Revised: The overall architecture is shown in Table 1.

Revised: A few auxiliary network architectures have been tested before we settle on the current design, which is quite simple but provides good results.

Original: [Table]
Revised: [Table]

Original: Since we ﬁnd that detailed spatial information is vital in the auxiliary network, our feature map size is only 1/4 of the original image size before the deconvolution layer.
Revised: Since we ﬁnd that detailed spatial information is vital in the auxiliary network, our feature map size is only 1/4 of the original image size before the deconvolution layer.

Original: We use the 3 × 3 convolution layers at the beginning, and then it is followed by a single atrous spatial pyramid pooling module similar to DeepLab v3+ (Chen et al., 2018) to extract multi-scale information as well as enlarging the receptive ﬁeld.
Revised: We use the 3 × 3 convolution layers at the beginning, and then it is followed by a single atrous spatial pyramid pooling module similar to DeepLab-v3+ (Chen et al., 2018) to extract multi-scale information as well as enlarging the receptive ﬁeld.

Original: For the ASPP (Atrous Spatial Pyramid Pooling) module of our auxiliary network, we use  dilated convolution with dilated rate { 1, 2, 4, 8, 16 } for branch 1-5 respectively, and do not use the image pooling branch.
Revised: For the ASPP (Atrous Spatial Pyramid Pooling) module of our auxiliary network, we use the dilated convolution with dilated rate { 1, 2, 4, 8, 16 } for branch 1-5 respectively, and do not use the image pooling branch.

Original: Each branch in our ASPP module has 32 feature channels.
Revised: Each branch in our ASPP module has 32 feature channels.

Original: Finally, the deconvolution kernel is applied after  ASPP module to rescale the output to the input size and generate  dense conﬁdence map, moreover, it is followed by  sigmoid function to limit  output between 0 and 1.
Revised: Finally, the deconvolution kernel is applied after the ASPP module to rescale the output to the input size and generate the dense conﬁdence map; moreover, it is followed by the sigmoid function to limit the output between 0 and 1.

Original: Same as  segmentation network, All the convolution layers are followed by batch normalization (Ioffe & Szegedy, 2015) and ReLU activations.
Revised: Same as the segmentation network, all the convolution layers are followed by batch normalization (Ioffe & Szegedy, 2015) and ReLU activations.

Original:  E XPERIMENTAL RESULTS
Revised:  E XPERIMENTAL RESULTS

Original: In this section, we ﬁrst introduce two datasets, Cityscapes (Cordts et al., 2016) and CamVid (Brostow et al., 2008), that are used to evaluate proposed method.
Revised: In this section, we ﬁrst introduce two datasets, Cityscapes (Cordts et al., 2016) and CamVid (Brostow et al., 2008), that are used to evaluate proposed method.

Original: Both are popular road scenesemantic segmentation datasets.
Revised: Both are popular road scene datasets.

Original: Then, we describe our training details for the segmentation and the auxiliary network.
Revised: Then, we describe our training details for the segmentation and the auxiliary networks.

Original: At the end, we show the comparisons with the state-of-the-art methods as well as some visual results.
Revised: At the end, we show the comparisons with the state-of-the-art methods as well as some visual results.

Original: 4. D ATASETS
Revised: 4. D ATASETS

Original: Cityscapes The Cityscapes dataset is a road scene dataset with the image resolution of× 2048, and it provides 19 object classes for evaluation.
Revised: Cityscapes The Cityscapes dataset is a road scene dataset with the image resolution of× 2048, and it provides 19 object classes for evaluation.

Original: There are two forms of annotation data in its database, namely, ﬁne and coarse.
Revised: There are two forms of annotation data in its database, namely, ﬁne and coarse.

Original: Only the ﬁne annotation set is used in our experiments, where it contains 2975, 500, 1525 images for training, validation, and testing respectively.
Revised: Only the ﬁne annotation set is used in our experiments, where it contains 2975, 500, 1525 images for training, validation, and testing respectively.

Original: We resize the image to 512 × 1024 for training and testing due to the hardware limitation; however, for evaluation on the testing set, we restore the resolution of segmentation results to the original size for fair comparison.
Revised: We resize the image to 512 × 1024 for training and testing due to the hardware limitation; however, for evaluation on the testing set, we restore the resolution of segmentation results to the original size for fair comparison.

Original: CamVid The CamVid dataset is also a road scene dataset, but its image resolution  is much smaller than  the Cityscapes dataset, which is about 360 × 480.
Revised: CamVid The CamVid dataset is also a road scene dataset, but its image resolution (about 360 × 480) is much smaller than that of the Cityscapes dataset.

Original: It provides 11 semantic classes, and contains 367, 101, 233 images for training, validation, and testing.
Revised: It provides 11 semantic classes, and contains 367, 101, 233 images for training, validation, and testing.

Original: 4. I MPLEMENTATION DETAILS
Revised: 4. I MPLEMENTATION DETAILS

Original: We use PyTorch framework to implement our method, and we measure the network speed on a single GTX 1080Ti GPU.
Revised: We use PyTorch framework to implement our method, and we measure the network speed on a single GTX 1080Ti GPU.

Original: The popular metric, mean IoU, is used to compute the segmentation performance.
Revised: The popular metric, mean IoU, is used to compute the segmentation performance.

Original: In addition, the encoder part of our segmentation network has been pre-trained on
Revised: In addition, the encoder part of our segmentation network has been pre-trained on

Original: ImageNet (Deng et al., 2009) to produce better initial parameters.
Revised: ImageNet (Deng et al., 2009) to produce better initial parameters.

Original: Segmentation network To train the segmentation network for the Cityscapes dataset, we use the stochastic gradient decent (SGD) optimization with momentum 0.9, weight decay 0.0001, and batch size 4.
Revised: Segmentation network To train the segmentation network for the Cityscapes dataset, we use the stochastic gradient decent (SGD) optimization with momentum 0.9, weight decay 0.0001, and batch size 4.

Original: We adopt the poly learning rate policy as described in Chen et al.
Revised: We adopt the poly learning rate policy as described in Chen et al.

Original: (2017a) with power 0.9 in all our experiments and the initial learning rate is set to 0.05 here.
Revised: (2017a) with power 0.9 in all our experiments and the initial learning rate is set to 0.05 here.

Original: We train it for 200 epochs in total.
Revised: We train it for 200 epochs in total.

Original: For the CamVid dataset, the SGD optimization is also used but with momentum 0.9, weight decay 0.0005, and batch size 8.
Revised: For the CamVid dataset, the SGD optimization is also used but with momentum 0.9, weight decay 0.0005, and batch size 8.

Original: In addition, the initial learning rate is set to 0.075, and we train it forepochs in total.
Revised: In addition, the initial learning rate is set to 0.075, and we train it forepochs in total.

Original: For both Cityscapes and CamVid datasets, we adopt the class weighting scheme same as in Paszke et al.
Revised: For both Cityscapes and CamVid datasets, we adopt the class weighting scheme same as in Paszke et al.

Original: with an additional hyper-parameter c set to 1.1, and the data augmentation strategies are employed to boost the performance.
Revised: with an additional hyper-parameter c set to 1.1, and the data augmentation strategies are employed to boost the performance.

Original: Our data augmentation strategies include random horizontal ﬂip, random pixel translation, and random scaling.
Revised: Our data augmentation strategies include random horizontal ﬂip, random pixel translation, and random scaling.

Original: For the Cityscapes dataset, the scaling factors inthe random scalingtechnique are { 0.75, 1.0, 1.25, 1.5, 1.75 } , and for the CamVid dataset, they are { 0.8, 1.0, 1.3, 1.7 } .
Revised: For the Cityscapes dataset, the scaling factors in random scaling are { 0.75, 1.0, 1.25, 1.5, 1.75 } , and for the CamVid dataset, they are { 0.8, 1.0, 1.3, 1.7 } .

Original: Auxiliary network To train the auxiliary network for Cityscapes dataset, we use the Adam optimization (Kingma & Ba, 2014) with momentum 0.9, weight decay 0.0001,and batch size 4, and the  initial learning rate is set to 0.0005.
Revised: Auxiliary network To train the auxiliary network for Cityscapes dataset, we use the Adam optimization (Kingma & Ba, 2014) with momentum 0.9, weight decay 0.0001, batch size 4, and the [Table] initial learning rate is set to 0.0005.

Original: For the CamVid dataset, different from the former, the weight decay for Adam optimization is set to 0.0005, and batch size is set to 8.
Revised: For the CamVid dataset, different from the former, the weight decay for Adam optimization is set to 0. [Table] and batch size is set to 8.

Original: Both datasets are training for 50 epochs, and we use the proposed auxiliary loss function in training the auxiliary network.
Revised: The training duration is 50 epochs for both datasets [Table] and we use the proposed auxiliary loss function in training the auxiliary network.

Original: The data augmentation strategies for training auxiliary network are same as described in the above.
Revised: The data augmentation strategies for training auxiliary network are same as described in the above.

Original: 4. P ERFORMANCE EVALUATION
Revised: 4. P ERFORMANCE EVALUATION

Original: Evaluation on Cityscapes dataset In order to evaluate the performance of the proposed segmentation network, we test it on the Cityscapes testing set, and compare the results with the state-of-theart networks.
Revised: Evaluation on Cityscapes dataset In order to evaluate the performance of the proposed segmentation network, we test it on the Cityscapes testing set, and compare the results with the state-of-theart networks.

Original: It is worth noticing that we do not adopt any testing technology such as multi-scale testing in the evaluation process.
Revised: It is worth noticing that we do not adopt any testing technology such as multi-scale testing in the evaluation process.

Original: From Table 2, we ﬁnd that our segmentation network can achieve [Table] [Table] off between speed and accuracy.
Revised: From Table 2, we ﬁnd that our segmentation network can achieve a good trade-off between speed and accuracy.

Original: It achieves 73.9% mean IoU with fewer parameters and higher inference speed comparing to the other segmentation nets.
Revised: It achieves 73.9% mean IoU with fewer parameters and higher inference speed comparing to the other segmentation nets.

Original: Moreover, to evaluate the proposed semi-supervised learning scheme,  we randomly sample half of images from the training set as the labeled data similar to that in Hung et al. andthen take the other half as the unlabeled data.
Revised: Moreover, to evaluate the proposed semi-supervised learning scheme, similar to that in work (Hung et al., 2018), we randomly sample half of images from the training set and call them the labeled data, and the other half are the unlabeled data.

Original: We conduct the experiments on  Cityscapes validation set.
Revised: We conduct the experiments on the Cityscapes validation set.

Original: First of all, we train the segmentation and auxiliary network using the labeled images as described in Section 3.1, and then we measure the conﬁdence map performance.
Revised: First of all, we train the segmentation and auxiliary network using the labeled images as described in Section 3.1, and then we measure the conﬁdence map performance.

Original: In taking the measure, we feed the validation set images to our entire network, and then mask the error-prone pixels of the segmentation map by the conﬁdence map.
Revised: In this process, we feed the validation set images to the entire network, and then mask the error-prone pixels of the segmentation map by the conﬁdence map.

Original: Then, compare the masked segmentation map (generated annotations) with its true ground truth to calculate the accuracy.
Revised: Then, compare the masked segmentation map (generated annotations) with its true ground truth to calculate the accuracy.

Original: We use three performance metricsin total for evaluation, and the experimental results are shown in Table 3, where T aux represents the threshold set for the conﬁdence map.
Revised: We use three performance metrics for evaluation, and the experimental results are shown in Table 3, where T aux represents the threshold set for the conﬁdence map.

Original: The mean IoU metric is the mIoU calculated by the generated annotations and the truth ground truth without considering the masked pixels (by the conﬁdence map).
Revised: The mean IoU metric is the mIoU calculated by the generated annotations and the true ground truth without considering the masked pixels (by the conﬁdence map).

Original: The average class semi ratio is the average value of the “class semi ratio” of all classes.
Revised: The average class semi ratio is the average value of the “class semi ratio” of all classes.

Original: To compute the class semi ratio for class i , we only consider the region of pixels that belong to class i on the ground truth label map.
Revised: To compute the class semi ratio for class i , we only consider the region of pixels that belong to class i on the ground truth label map.

Original: The denominator of class semi ratio for class i is the number of all pixels; and the numerator is the number of pixels, not masked by the conﬁdence map, on the estimated segmentation map.
Revised: The denominator of class semi ratio for class i is the number of all pixels; and the numerator is the number of pixels not masked by the conﬁdence map, on the estimated segmentation map.

Original: In calculating the selected pixels , we ﬁrst count the pixels of the estimated segmentation map that are selected by the conﬁdence map and  used as ground truth  in training.
Revised: In calculating the selected pixels , we ﬁrst count the pixels of the estimated segmentation map that are selected by the conﬁdence map and are used as ground truth (GT) in training.

Original: Then, we compute its ratio with respect to the entire image.
Revised: Then, we compute its ratio with respect to the entire image.

Original: From Table 3, we can ﬁnd that with a higher threshold, the pixels selected as GT are more accurate, but the selected pixels are  fewer; this is a trade-off.
Revised: From Table 3, we can ﬁnd that with a higher threshold, the pixels selected as GT are more accurate, but the selected pixels are [Table] [Table] fewer; this is a trade-off.

Original: In addition, observing from the average class semi ratio and the individual semi ratio for each class, our conﬁdence map can also produce good results on the small objects with a higher threshold.
Revised: In addition, as shown in Table [Table] our conﬁdence map can also produce good results on the small objects with a higher threshold.

Original: Thus, the generated annotations used for retraining segmentation network can achieve good quality.
Revised: Thus, the generated annotations used for retraining segmentation network have good quality.

Original: [Table]

Original: [Table]

Original: After generating the annotated labels for 1/2 unlabeled images
Revised: After generating the annotated labels for 1/2 unlabeled images

Original: [Table]
Revised: [Table]

Original: we retrain the segmentation network using all images.
Revised: we retrain the segmentation network using all images.

Original: The experimental results of selecting pixels based on different threshold values are shown in Table 4.
Revised: The experimental results of selecting pixels based on different threshold values are shown in Table 5.

Original: The proposed semi-supervised learning scheme can improve the performance for about 1.0% to 1.2%.
Revised: The proposed semi-supervised learning scheme can improve the performance for about 1.0% to 1.2%.

Original: If T aux is set to 0, it means that all the pixels of the estimated segmentation map are used as ground truth for unlabeled image, which is obviously unreasonable and it decreases the accuracy.
Revised: If T aux is set to 0, it means that all the pixels of the estimated segmentation map are used as ground truth for unlabeled image, which is obviously unreasonable and it decreases [Table] [Table]

Original:  Since T aux at 0.9 gives the best results, we use the same value in the following experiments.
Revised: the accuracy. Since T aux at 0.9 gives the best results, we use the same value in the following experiments.

Original: To verify the effectiveness of our approach, we conduct experiments with different amount of data, and shows the comparison with Hung et al.
Revised: To verify the effectiveness of our approach, we conduct experiments with different amount of data, and shows the comparison with Hung et al.

Original: (2018)  in Table 5.
Revised: (2018) and Mittal et al. in Table 6.

Original: For the fair comparison, we also adopt the same segmentation network used by Hung et al. (2018) as our backbone  model.
Revised: For the fair comparison, we also adopt the DeepLab-v2 framework as our backbone segmentation net model.

Original: Under the same settings that using 1/2, 1/4, and 1/8 training images as labeled data and using the rest of images as unlabeled data, our semi-supervised learning method can lead to higher performance improvements.
Revised: Under the same settings that using 1/2, 1/4, and 1/8 training images as labeled data and using the rest of images as unlabeled data, our semi-supervised learning method can lead to higher performance improvements.

Original: It can improve the mean IoU by 3.9%, 2.5%, and 1.6% for 1/8, 1/4, and 1/2 training datasets, respectively.
Revised: It can improve the mean IoU by 3.9%, 2.5%, and 1.6% for 1/8, 1/4, and 1/2 training datasets, respectively.

Original: Particularly,  if we focus on the performance differences between the schemes with and without semi-supervised learning mechanism, our semi-supervised learning approach offers quite signiﬁcant improvement.
Revised: Particularly, compared with Hung et al. (2018), if we focus on the performance differences between the schemes with and without semi-supervised learning mechanism, our semi-supervised learning approach offers quite signiﬁcant improvement.

Original: Then, if our proposed segmentation network is used as the backbone network, it increases the mean IoU by another 1.2% to 2.9%.
Revised: Furthermore, when our proposed segmentation network is used as the backbone segmentation net, it increases the mean IoU for another 1.2% to 2.9%.

Original: In the end, we show some sample  results in Figure 4.
Revised: At the end, we show some sample visual results in Figure 4.

Original: Evaluation on CamVid dataset We also evaluate our approach on the CamVid dataset.
Revised: Evaluation on CamVid dataset We also evaluate our scheme on the CamVid dataset.

Original: As the results shown in Table 6, our semi-supervised learning method can boost the performance of the segmentation network, and it can achieve 1.3% improvement in mean IoU.
Revised: As the results shown in Table 7, our semi-supervised learning method can boost the performance of the segmentation network, and it can achieve 1.3% improvement on mean IoU.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper we ﬁrst propose a highly efﬁcient segmentation network as our platform and then we design a semi-supervised learning scheme using an auxiliary network.
Revised: In this paper we ﬁrst propose a highly efﬁcient segmentation network as our platform, and then we design a semi-supervised learning scheme using an auxiliary network.

Original: The auxiliary network is used to verify the estimated segmentation map and to generate annotations on the unlabeled images.
Revised: The auxiliary network is used to verify the estimated segmentation map and to generate annotations on the unlabeled images.

Original: Equipped with the carefully designed auxiliary loss function in training, the auxiliary network performs well not only on the large objects but also on the small objects.
Revised: Equipped with the carefully designed auxiliary loss function in training, the auxiliary network performs well not only on the large objects but also on the small objects.

Original: It shows that the unlabeled images together with the generated annotations (labels) can be used to retrain the segmentation network for better segmentation quality.
Revised: It shows that the unlabeled images together with the generated annotations (labels) can be used to retrain the segmentation network for better segmentation quality.

Original: Our experimental results on the Cityscapes and CamVid datasets demonstrate the effectiveness of the proposed method
Revised: Our experimental results on the Cityscapes and CamVid datasets demonstrate the effectiveness of the proposed method