S EMI - SUPERVISED SEMANTIC SEGMENTATION NEEDS STRONG , HIGH - DIMENSIONAL PERTURBATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Consistency regularization describes a class of approaches that have yielded ground breaking results in semi-supervised classiﬁcation problems.
Prior work has established the cluster assumption — under which the data distribution consists of uniform class clusters of samples separated by low density regions — as key to its success.
We analyze the problem of semantic segmentation and ﬁnd that the data distribution does not exhibit low density regions separating classes and offer this as an explanation for why semi-supervised segmentation is a challenging problem.
We then identify the conditions that allow consistency regularization to work even without such low-density regions.
This allows us to generalize the recently proposed CutMix augmentation technique to a powerful masked variant,
CowMix, leading to a successful application of consistency regularization in the semi-supervised semantic segmentation setting and reaching state-of-the-art results in several standard datasets.
I NTRODUCTION
Semi-supervised learning offers the tantalizing promise of training a machine learning model with limited amounts of labelled training data and large quantities of unlabelled data.
These situations often arise in practical computer vision problems where large quantities of images are readily available and generating ground truth labels acts as a bottleneck due to the cost and labour required.
Consistency regularization (Sajjadi et al., 2016b; Laine & Aila, 2017; Miyato et al., 2017; Oliver et al., 2018) describes a class of semi-supervised learning algorithms that have yielded state-ofthe-art results in semi-supervised classiﬁcation, while being conceptually simple and often easy to implement.
The key idea is to encourage the network to give consistent predictions for unlabeled inputs that are perturbed in various ways.
The effectiveness of consistency regularization is often attributed to the smoothness assumption (Miyato et al., 2017; Luo et al., 2018) or cluster assumption (Chapelle & Zien, 2005; Sajjadi et al., 2016a; Shu et al., 2018; Verma et al., 2019).
The smoothness assumption states that samples close to each other are likely to have the same label.
The cluster assumption — a special case of the smoothness assumption — states that decision surfaces should lie in low density regions, not crossing high density regions.
This typically holds in classiﬁcation tasks, where most successes of consistency regularization have been reported so far.
On a high level, semantic segmentation is classiﬁcation, where each pixel is classiﬁed based on its neighborhood.
It is therefore intriguing that consistency regularization has not been clearly beneﬁcial in this context.
We make the observation that the distance between patches centered on neighboring pixels varies smoothly even when the class of the center pixel changes, and thus there are no low-density regions on class boundaries.
This alarming observation leads us to investigate the conditions that can allow consistency regularization to operate even in these conditions.
Our key insight is that this is indeed possible, and that previous attempts have yielded little success primarily because the perturbations they used were not strong enough, and especially not high-dimensional enough.
We show that a ﬂexibly masked variant of CutMix (Yun et al., 2019), which we call CowMix based on the mask appearance, does realize signiﬁcant gains in semi-supervised semantic segmentation.
This result clearly signposts a direction where further improvements are likely to be available.
B ACKGROUND
Our work relates to prior art in three areas: recent regularization techniques for classiﬁcation, semisupervised classiﬁcation with a focus on consistency regularization, and semantic segmentation.
2. M IX U P , C UTOUT , AND C UT M IX
The MixUp regularizer of Zhang et al.
(2018) improves the performance of supervised image, speech and tabular data classiﬁers by using interpolated samples during training.
The inputs and target labels of two randomly chosen examples are blended using the same randomly chosen factor.
The Cutout regularizer of DeVries & Taylor (2017) augments an image by masking a rectangular region to zero.
The recently proposed CutMix regularizer of Yun et al.
(2019) combines aspects of
MixUp and Cutout, cutting a rectangular region from B and pasting it over A .
MixUp, Cutout, and
CutMix improve supervised classiﬁcation performance, with CutMix outperforming the other two
. 2. S EMI - SUPERVISED CLASSIFICATION
The Π -model of Laine & Aila (2017) passes each unlabeled sample through a classiﬁer twice, applying two realizations of a stochastic perturbation process, and minimizes the difference between the resulting class probability predictions.
Their temporal model maintains a per-sample moving average of historical predictions and encourages subsequent predictions to be consistent with the average.
Sajjadi et al.
(2016b) similarly encourage consistency between the current and historical predictions.
Miyato et al.
(2017) improve the results by replacing the stochastic perturbations with adversarial directions, thus focusing on perturbations that are closer to the decision boundaries.
The mean teacher model of Tarvainen & Valpola (2017) encourages consistency between predictions of a student network and a teacher network.
The teacher’s weights are an exponential moving average (Polyak & Juditsky, 1992) of those of the student, leading to a faster convergence and improved results.
French et al.
(2018) adapt the mean teacher approach for domain adaptation.
Interpolation consistency training (ICT) (Verma et al., 2019) and MixMatch (Berthelot et al., 2019)
both combine MixUp (Zhang et al., 2018) with consistency regularization.
ICT uses the mean teacher model and applies MixUp to unsupervised samples, blending input images along with teacher class predictions to produce a blended input and target to train the student.
MixMatch stochastically perturbs each sample multiple times and averages the predictions to produce unsupervised targets; MixUp is applied to labeled as well as unlabeled samples.
2. S EMANTIC SEGMENTATION
Long et al.
(2015) ﬁnetune a pre-trained VGG-16 (Simonyan & Zisserman, 2014) image classiﬁer to produce a dense set of predictions for overlapping input windows.
They effectively transform the image classiﬁer into a fully convolutional network that can be used to segment input images of arbitrary size.
Various methods were proposed for increasing the localization accuracy of the results (Long et al., 2015;
Chen et al., 2014; Mostajabi et al., 2014) until the introduction of encoderdecoder networks (Badrinarayanan et al., 2015; Ronneberger et al., 2015) led to a solution where the output resolution natively matches the input.
In these recent methods the encoder downsamples the input progressively, similarly to image classiﬁers, the decoder performs progressive upsampling, and skip connections route data between the matching resolutions of the two networks, improving the ability to accurately segment ﬁne details.
A number of approaches for semi-supervised semantic segmentation use additional data.
Kalluri et al.
(2018) use data from two datasets from different domains, maximizing the similarity between per-class embeddings from each dataset.
Stekovic et al.
(2018) use depth images and enforced geometric constraints between multiple views of a 3D scene.
Relatively few approaches operate in a strictly semi-supervised setting.
Hung et al.
(2018) employ adversarial learning, using a discriminator network that distinguishes real from predicted segmentation maps to guide learning.
Perone &
Cohen-Adad (2018) apply consistency regularization to a MRI volume dataset and their method is the only successful application of consistency regularization to segmentation that we are aware of.
[Figure]
C ONSISTENCY REGULARIZATION FOR SEMANTIC SEGMENTATION
Consistency regularization adds a consistency loss term L cons to the loss that is minimized during training (Oliver et al., 2018).
In a classiﬁcation task, L cons measures a distance d ( · , · ) between the predictions resulting from applying a neural network f θ to an unsupervised sample x and a perturbed version ˆ x of the same sample, i.e., L cons =
d ( f θ ( x ) , f θ (ˆ x )) .
The perturbation used to generate ˆ x depends on the variant of consistency regularization used.
A variety of distance measures d ( · , · ) have been used, e.g., squared distance (Laine & Aila, 2017) or cross-entropy (Miyato et al., 2017).
Athiwaratkun et al.
(2019) analyze a simpliﬁed version of the Π -model (Laine & Aila, 2017) in which perturbation consists of additive Gaussian noise so that ˆ x = x + (cid:15)h , where h ∼ N (0 , I ) and d ( · , · ) is a squared Euclidean distance.
For small constant (cid:15) , the expected value of the consistency loss term L cons is approximately proportional to the square of the Frobenius norm of the Jacobian
J f θ ( x ) of the networks outputs with respect to its inputs:
[Equation]
Thus, minimizing L cons directly ﬂattens the decision function in the vicinity of unsupervised samples.
This illustrates clearly the mechanism by which consistency regularization encourages the network to move the decision boundary — and its surrounding region of high gradient — into regions of low sample density.
3. W HY SEMI - SUPERVISED SEMANTIC SEGMENTATION IS CHALLENGING
We attribute the infrequent success of consistency regularization in semantic segmentation problems to the observation that low density regions in input data do not align well with class boundaries.
As illustrated in Figure 1, the cluster assumption is clearly violated: how much the raw pixel content of the receptive ﬁeld of one pixel differ from the contents of the receptive ﬁeld of a neighboring pixel has effectively no correlation with whether the patches’ center pixels belong to the same class or not.
For the cluster assumption to hold, we would require the distances between the contents of receptive ﬁelds to be large between classes and small within classes, which is not the case here.
The lack of variation in the patchwise distances is easy to explain from a signal processing perspective.
With patch shape B , the L 2 distance between the pixel content of two overlapping patches centered at, say, horizontally neighboring pixels can be written as (cid:112) B ∗ (∆ x x ) 2 , where ∗ denotes convolution and ∆ x x is the horizontal gradient of the input image x .
The squared gradient image is thus low-pass ﬁltered by a B -shaped box ﬁlter, which suppresses any ﬁne details and leads to smoothly varying sample density across the image.
We test this experimentally in Appendix B.
Our decision to compare the raw pixel content of image patches is motivated by the fact that the perturbations used to generate ˆ x from x in prior work operate in raw pixel (input) space.
The success of virtual adversarial training (Miyato et al., 2017) in semi-supervised classiﬁcation – in which ˆ x is
[Figure]
an adversarial example chosen to maximise the difference in prediction from that of x – illustrates the signiﬁcance of the input space distribution in semi-supervised learning.
3. C ONSISTENCY REGULARIZATION WITHOUT CLUSTER ASSUMPTION
Let us analyze a simple toy example of learning to classify 2D points in a semi-supervised fashion.
Figure 2a illustrates the setup where cluster assumption holds and there is a gap between the unsupervised samples belonging to the two different classes.
The perturbation used for the consistency loss is a simple Gaussian nudge to both coordinates, and as expected, the learned decision boundary settles neatly between the two clusters.
In Figure 2b, the cluster assumption is violated and there are no density differences in the set of unsupervised samples.
In this case, the consistency loss does more harm than good — even though it successfully ﬂattens the neighborhood of the decision function, it does so also across the true class boundary.
In order for the consistency regularization to be a net win, it would have to perturb the samples as much as possible, but at the same time avoid crossing the true class boundary.
In Figure 2c, we plot the contours of the distance to the true class boundary, suggesting a potentially better mechanism for perturbation.
Indeed, when perturbations are done only along these contours, the probability of crossing the true class boundary is negligible compared to the regularization potential in the remaining dimension.
Figure 2d shows that the resulting learned decision boundary aligns well with the true class boundary.
Low-density regions provide an effective signal that guides consistency regularization by providing areas into which a decision boundary can settle.
This illustrative toy example demonstrates an alternative mechanism; the orientation of the decision boundary can be constrained to lie parallel to the directions of perturbation.
We therefore argue that consistency regularization can be successful even when the cluster assumption is violated, if the following guidelines are observed: 1) the perturbations must be varied in order to cover as much of the input space in the same class as possible, 2) they must be high-dimensional in order to sufﬁciently constrain the orientation of the decision boundary in the high-dimensional space of natural imagery, 3) the probability of a perturbation crossing the true class boundary must be very small compared to the amount of exploration in other dimensions, and 4) the perturbed inputs should be plausible, i.e., they should not be grossly outside the manifold of real inputs. [Figure]
3. C UT O UT AND C UT M IX FOR SEMANTIC SEGMENTATION
If we consider the classical augmentation-like perturbations such as translation, rotation, scaling, and brightness/contrast changes, it is evident that these have a low chance of confusing the output class (Athiwaratkun et al., 2019) but they also provide very little variation.
Note that in context of semantic segmentation, all geometric transformations need to be applied in reverse for the result image before computing the loss (Ji et al., 2018).
As such, translation turns into a no-op, unlike in classiﬁcation tasks where it remains a useful perturbation (see Appendix D.1.1 for more details).
Adding noise is another questionable perturbation strategy — although high-dimensional, such perturbations are very unlikely to lie on the manifold of natural images.
Out of previously proposed perturbation methods for consistency regularization, we identify CutOut and CutMix as promising candidates for semantic segmentation as they provide a large variety of possible outputs and are class preserving.
Both approaches use a mask M with a randomly chosen rectangular region.
Our masks have M = 1 inside the rectangle, and M = 0 otherwise.
CutOut.
To apply CutOut in a semantic segmentation task, we mask the input pixels with M and disregard the consistency loss for pixels masked to 0 by M .
Using square distance as the metric, we have L cons =
|| M (cid:12) ( f θ ( M (cid:12) x ) − f θ ( x ))
|| 2 , where (cid:12) denotes an elementwise product.
CutMix.
CutMix requires two input images that we shall denote x a and x b that we mix with the mask M .
Following ICT (Verma et al.
we run both input images as well as the mixed variant through the network.
The consistency loss is then taken between the segmentation of the mixed image and the mix between the segmented input images.
To simplify the notation, let us deﬁne function mix ( a, b, M )
=
(1 − M ) (cid:12) a + M (cid:12) b that selects the output pixel based on mask
M .
We can now write the consistency loss for segmentation CutMix as
[Equation]
So far we have assumed that both original and perturbed images are segmented using the same network f θ .
In this sense, our approach is similar to the Π -network Laine & Aila (2017), although mean teacher (Tarvainen & Valpola, 2017) is known to outperform it in image classiﬁcation tasks.
Our preliminary tests indicated that the same is true for semantic segmentation, and therefore all the experiments in this paper use the mean teacher framework.
Speciﬁcally, in CutOut, we segment the unperturbed image x using the teacher network and the perturbed image M (cid:12) x using the student network.
In CutMix — following Verma et al.
(2019) — we similarly segment the original images x a and x b using the teacher network, and the mixed image using the student network.
The computation is illustrated in Figure 3.
To analyze how much variation these perturbations provide, we note that the original CutOut always masks out a ﬁxed-size square at a random position, so the resulting mask has only two degrees [Figure]
of freedom The original CutMix has three degrees of freedom in choosing the position and size of the rectangle (the aspect ratio is ﬁxed), and in addition it replaces the contents of the rectangle from another training image, providing one further degree of freedom.
Probably more importantly, the CutOut-perturbed images are rather implausible because real data rarely contains axis-aligned constant-color rectangles, whereas the content-ﬁlled rectangles of CutMix are not nearly as conspicuous.
In semantic segmentation, we deviate from the original methods in that we choose all four parameters of the rectangle at random in order to obtain as much variation as possible.
3. C OW O UT AND C OW M IX
The use of a rectangular mask restricts the dimensionality of the perturbations that CutOut and
CutMix can produce.
Intuitively, a more complex mask that has more degrees of freedom should provide better exploration of the plausible input space.
We propose combining the semantic CutOut and CutMix regularizers introduced above with a novel mask generation method, giving rise to two regularization methods that we dub CowOut and CowMix due to the Friesian cow -like texture of the masks.
To generate a mask with a given proportion p of pixels having M = 0 , we start by sampling a
Gaussian noise image N (0 , I ) , convolve it with a Gaussian smoothing kernel, and threshold the result at τ = erf − 1 (2 p − 1) √ 2 S σ + S µ , where S µ and S σ are the mean and standard deviation of the smoothed noise, respectively.
The standard deviation σ of the Gaussian smoothing kernel determines the average size of features in the mask and is drawn from a log-uniform distribution: σ ∼ log U ( σ min , σ max ) .
Figure 4 shows example masks with varying values of σ .
To estimate how much variation these masks contain, we ﬁrst observe that if no smoothing were performed before thresholding, we would obtain a binary per-pixel Bernoulli mask.
This has an exponential amount of variation with respect to the area of the receptive ﬁeld: a receptive ﬁeld with area A has 2 A possible masks.
Applying the smoothing ﬁlter has a similar effect as zooming the mask — if the smoothing constant σ is doubled, we can expect to obtain roughly 4 × fewer “bits” of variation (see, e.g., Figure 4, σ = 8 vs. σ = 16 ).
The suitable value for σ is thus an empirical tradeoff between the amount of variation, i.e., the strength of the resulting perturbation, and plausibility of the masked/mixed image.
If too little ﬁltering is performed, we obtain a huge amount of variation but the perturbed images are unrealistic, and with too much ﬁltering the images are very realistic but the amount of variation is small.
As a geometric interpretation, we hypothesize that the more varied masks help the network to cope with occlusions better than the rectangular masks that can mimic occlusions only along horizontal and vertical edges.
Considering potential future work, we note that it should be possible to improve the results further with a more principled method for choosing σ so that its distribution is automatically adapted to the training data.
E XPERIMENTS
We will now describe describe our experiments and main results.
We will start by describing the training setup, followed by an investigation of various perturbation methods in the context of semisupervised semantic segmentation, and conclude with a comparison against the state-of-the-art. [Table]
T RAINING SETUP
We use two segmentation networks in our experiments: 1) U-Net (Ronneberger et al., 2015) with a
ResNet-50 (He et al., 2016) based encoder that was pre-trained using ImageNet and a randomly initialized decoder (Appendix D.2), and 2) DeepLab v2 network (Chen et al., 2017) based on ResNet101 and pre-trained for semantic segmentation using the COCO (Lin et al., 2014) dataset, as used by Hung et al.
We minimise the sum of the supervised cross-entropy loss and consistency loss L cons with weights of 1 and 10 respectively.
We use the ean teacher algorithm (Tarvainen & Valpola, 2017), as detailed in Appendix D.3.
We replace the sigmoidal ramp-up of the consistency regularization weight (Laine & Aila, 2017; Tarvainen & Valpola, 2017) with the average thresholded conﬁdence of the teacher network (see Appendix D.3.4), which increases as the training progresses (French et al., 2018).
Our implementation uses the PyTorch framework and will be made available.
4. C OMPARISON OF PERTURBATION METHODS
The C AM V ID (Brostow et al., 2008) training set consists of 367 images.
We chose 10 subsets of 30 labeled images that were used for the supervised loss, while all training images were used to compute the consistency loss.
The U-net setup is used in this test.
The results for different perturbation methods are given in Table 1.
Perturbations based on standard data augmentation (ﬂips, rotations, brightness, etc., see Appendix D.1) and Interpolation Consistency Training (ICT) resulted in no measurable improvement in the mean IoU score compared to the baseline of supervised training using only the labeled samples.
The Cutout and CutMix experiments used masks with a single random rectangle, and led to clear improvements over the baseline.
For CowOut and CowMix, we generated the masks using σ ∈ [4 , 16] .
For CowOut, we found that choosing the masked pixel proportion randomly with p ∼ U (0 , 1) produced the best results, whereas for CowMix it was optimal to always use p = 0 . 5 .
CowMix led to the best result in this experiment, bridging approximately half of the gap between the supervised baseline and the fully supervised reference result.
We tuned all hyper-parameters using the C AM V ID dataset due to its small size and fast run-time.
We found that these hyper-parameters also worked well for other, larger datasets.
4. R ESULTS ON C ITYSCAPES AND P ASCAL VOC
We will now compare our results against the state-of-the-art in semi-supervised semantic segmentation, which is currently the adversarial training approach of Hung et al.
We use two datasets in our experiments.
C ITYSCAPES consists of urban scenery and has 2975 images in its training set.
P ASCAL VOC
(Everingham et al., 2012) is more varied, but includes only 1464 training images, and thus we follow the lead of Hung et al.
and augment it using S EMANTIC B OUNDARIES (Hariharan et al., 2011), resulting in 10582 training images.
The practical beneﬁts of semisupervised learning are realised by reducing the required quantity of supervised training data as much as possible, hence we tested our approch using less supervised data than in Hung et al.
We note that we generated CowMix masks using σ ∈ [8 , 32] due to the larger visual features present in C ITYSCAPES and P ASCAL VOC.
Our results are given in Tables 2 and 3 as mean intersection-over-union (mIoU) percentages, where higher is better.
The supervised baseline results between Hung et al.
and our DeepLab implementa-
[Table]
[Table] [Figure] [Table] tion are based on the same setup, provided by the authors, but differ slightly in practice due to the different choice of optimizer, etc.
We can see that using the same DeepLab setup, CowMix outperforms the adversarial training approach in both datasets, with the exception of P ASCAL when using a large number of labeled samples.
The difference is particularly signiﬁcant when only a small number of labeled samples is available, e.g., 52% vs 38% mIoU with 100 labeled samples in P ASCAL .
When using our U-Net architecture — in which the decoder has not been pre-trained — we were unable to successfully apply the adversarial approach.
U-Net improves the performance of CowMix on C ITYSCAPES and continues to yield signiﬁcant gains relative to the supervised baseline in the case of P ASCAL .
While the performance on the more varied P ASCAL dataset signiﬁcantly beneﬁts the pre-training of DeepLab v2, it also provides a good test environment for semi-supervised learning algorithms when the decoder is randomly initialized.
4. D ISCUSSION
We attribute the fact that the adversarial approach (Hung et al.
(2018)) is ineffective with small numbers of labeled samples to the requirements imposed by its discriminator network.
A small set of ground truth labels lacks the variation necessary to effectively train the discriminator to distinguish ground truth from predicted segmentation maps, preventing it from effectively guiding the segmentation network.
In contrast, consistency regularization minimizes variation in prediction over class preserving perturbation, effectively propagating labels between unlabelled samples.
It therefore does not impose similar requirements on the size of the labeled data set.
C ONCLUSIONS
We have shown that consistency regularization is a viable solution for semi-supervised semantic segmentation, despite the lack of low-density regions between classes.
The proposed CowMix regularization leads to very high-dimensional perturbations that enable state-of-the-art results, while being considerably easier to implement and use than the previous methods based on adversarial training.
Even better and more varied perturbation strategies are an obvious avenue for future work.
Additionally, it could be fruitful to investigate when the CowMix approach could also be beneﬁcial in the context of classiﬁcation.
A 2D TOY EXPERIMENTS
The neural networks used in our 2D toy experiments are simple classiﬁers in which samples are 2D x, y points ranging from -1 to 1.
Our networks are multi-layer perceptrons consisting of 3 hidden layers of 512 units, each followed by a ReLU non-linearity.
The ﬁnal layer is a 2-unit classiﬁcation layer.
We use the mean teacher (Tarvainen & Valpola, 2017) semi-supervised learning algorithm with binary cross-entropy as the consistency loss function, a consistency loss weight of 10 and conﬁdence thresholding (French et al., 2018) with a threshold of 0.97.
The ground truth decision boundary was derived from a hand-drawn 512 ×pixel image.
The distance map shown in
Figure 2(c) was computed using scipy.
ndimage.morphology.distance transform edt
, with distances negated for regions assigned to class 0.
Each pixel in the distance map therefore has a signed distance to the ground truth class boundary.
This distance map was used to generate the countours seen as lines in
Figure 2(c) and used to support the constrained consistency regularization experiment illustrated in
Figure 2(d).
The constrained consistency regularization experiment described in Section 3.2 required that a sample x should be perturbed to ˆ x such that they are at the same — or similar — distance to the ground truth decision boundary.
This was achieved by drawing isotropic perturbations from a normal distrubtion ˆ x = x + h where h ∼ N (0 , 0 . 117) ( 0 .
117 ≈ 30 pixels in the source image), determining the distances m ( x ) and m (ˆ x ) from x and ˆ x to the ground truth boundary (using a precomputed distance map) and discarding the perturbation – by masking consistency loss for x to 0 – if | m (ˆ x ) − m ( x ) | > 0 .
≈ 4 pixels in the source image).
B D ISTRIBUTION OF PATCHES IN THE C ITYSCAPES DATASET
In Section 3.1 we observed that if we extract overlapping neighboring patches from an image, where the patches are centered on neighboring pixels and compute the L 2 distance between the raw pixel content of neighboring patches, we see that the distances vary smoothly across the image.
This is due to the fact that the distance between these patches can be efﬁciently computed by applying a box/uniform ﬁlter to the squared gradient image.
This equivalence between the patch-to-patch distance and a uniform ﬁlter (that suppresses high frequencies and therefore the ﬁne details necessary for low-density regions / large inter-patch distances to lie along class/object/texture boundaries in an image) suggests that these distance maps will be smooth for natural images in general.
[Figure]
We will now use the C ITYSCAPES dataset to experimentally test our observations concerning the distribution of image patches.
Figure 5 illustrates the distributions of distances between pairs of patches centred on neighbouring pixels that have the same class in comparison to pairs of patches centred on pixels that lie either side of a class boundary.
The higher proportion of lower pixel content distances between neighboring patches that have the same class can be explained by the presence of large lightly textured regions such as sky that have few class boundaries and patches with similar pixel content.
Further analysis of the C ITYSCAPES dataset revealed that in terms of pixel content distance, the nearest neighbours to a patch are other patches centred on neighboring and nearby pixels.
We chose 1000 pairs of patches Q i and P i where i ≤ 1000 from C ITYSCAPES whose central pixels lie either side of a class boundary.
For each query patch Q i we computed the distance to all other patches in the entire dataset.
Figure 6 shows that the closest neighbour extracted from another image is typically around 3 × the distance from Q i as the immediate neighbor P i .
B. E FFICIENT COMPUTATION OF SLIDING WINDOW PATCH DISTANCES
Computing the distance from patch Q i to every patch centred on each pixel in the complete
C ITYSCAPES dataset would be prohibitively expensive if performed in a brute-force fashion.
There are 2975 training images in C ITYSCAPES with each image having a resolution of 1024 × 512 pixels (downsampled by a factor of 2 from the original resolution, as used in our segmentation experiments).
Centering a patch on each pixel results in 1,559,756,800 patches.
Given a query patch Q and a patch A where A is extracted from an image I , the squared distance (cid:107) A − Q (cid:107) 2 can be expanded to Q · Q + A · A − 2( Q · A ) .
Q · Q is a constant, A
· A can be computed for all patches in I in sliding window fashion using the box ﬁlter trick from Section 3.1; A · A = B ∗ Iwhere B is a box-ﬁlter kernel, I 2 is the element-wide squared image and ∗ represents convolution.
Q · A can be computed using correlation; given that the correlation kernel Q is typically large, we use FFT-based convolution.
C P SEUDOCODE
The mask generation function is given in the form of Python/PyTorch code in Listing 1, and the training functions for semi-supervised CutOut and CutMix are given in Listings 2 and 3, respectively.
The latter incorporate the mean teacher framework (Tarvainen & Valpola, 2017) and conﬁdence thresholding (French et al., 2018) as used in our experiments.
In keeping with Yun et al.
DeVries & Taylor (2017); Verma et al.
, our mask generator allows the proportion of pixels that come from each source image to vary.
Although we found via experimentation that for CowMix the best proportion was always p = 0 . 5 , varying the proportion is beneﬁcial for CowOut as detailed in Section 4.2.
1 import numpy as np 2 from scipy.ndimage.filter import gaussian_filter 3 from scipy.special import erfinv 4 5 def generate_mixing_mask(img_size, sigma_min, sigma_max, p_min, p_max): # Randomly draw sigma from log-uniform distribution sigma = np.exp(np.random.uniform(np.log(sigma_min), np.log(sigma_max)))
p = np.random.uniform(p_min, p_max)
# Randomly draw proportion p
N = np.random.normal(size=img_size)
# Generate noise image
Ns = gaussian_filter(N, sigma) # Smooth with a Gaussian # Compute threshold t = erfinv(p*2 - 1) * (2**
5) * Ns.
+ Ns mean() return
# Apply threshold and return
Listing 1: Python/NumPy code for cow mask generation 1 def cutout_loss
, teacher_
, student_
): """x is input of
), mask is mask of shape
)""" # Apply teacher model and softmax to get per-pixel class probability
y_t
= softmax(teacher_model(x), dim=1)
# Apply student model to masked image
y_s = softmax(student_model(x * mask), dim=1)
# Confidence thresholding factor prob.
mean factor probability predictions mask
ym s xm factor confidence ym t prob.
error confidence factor return
Listing
D S EMANTIC SEGMENTATION EXPERIMENTS
D.
D
ATA AUGMENTATION
Our data augmentation scheme that we used for standard augmentation based perturbation consists of an afﬁne transformation composed of horizontal ﬂips, translation in the range [ − 4 , 4] pixels,
[Figure]
uniform scaling in the range [0 . 8 , 1 . 25] and rotation in the range [ − 15 ◦ , 15 ◦ ] .
We also modify the brightness and contrast by adding a value b ∼ N (0 , 0 . 1) and scaling by a factor c ∼ e N (0 ,ln (1 . 1)) .
D.1. A FFINE TRANSFORMATION BASED DATA AUGMENTATION FOR SEMANTIC SEGMENTATION
In Section 3.3 we stated that for the purpose of semantic segmentation, all geometric transformations need to be applied in reverse for the result image before computing the loss and that translation effectively becomes a no-op.
This occurs in a similar manner in both the supervised and unsupervised pathways.
Let us assume that the image x is augmented by applyng the afﬁne transformation matrix M ; ˆ x = transform ( x, A ) .
We apply our segmentation network f θ to get our predictions ˆ y pred =
f θ (ˆ x ) .
We note that the pixel-wise predictions ˆ y pred were generated from the transformed image ˆ x and are therefore are not aligned with our ground truth annotations y .
There are two ways in which we can compute the supervised loss L ce ; we can apply the transformation A to the ground truths ˆ y = transform ( y, A ) to align them with ˆ y pred and compute L ce = cross entropy (ˆ y pred , y ) or we can apply the inverse transformation
A − 1 to our predictions to bring them into alignment with the original image y pred = transform (ˆ y pred , A − 1 ) .
Should the transformation A include a translation, it will be countered by applying the equivalent translation to the pixel-wise ground truth in the ﬁrst case or will be undone by applying A − 1 to the predictions in the second case.
We must adopt a similar strategy when computing the unsupervised loss term L cons when using afﬁne augmentation as perturbation.
If the image x is transformed using two different afﬁne transformations A and B producing ˆ x and ˜ x .
Our network f θ will generate predictions ˆ y pred and ˜ y
pred that are not aligned with one another.
In order to compute the consistency loss L cons we must align the pixel-wise predictions with one another.
This can be done by transforming ˜
y pred into the same frame of reference as ˆ y pred by applying the transformation AB − 1 ; ¯ y pred = transform (˜ y pred , AB − 1 ) .
We would like to note that these constraints limit the utility of translations used as an augmentation for semantic segmentation.
Classiﬁcation problems are typically translation invariant; applying a translation will not change the target class.
This is not true for semantic segmentation which is translation variant.
For example, consider applying a translation to a patch that is centered on a pixel that is very close to a class boundary that separates a car from a road in an image.
The translated patch may now have a different class.
Computing either supervised or unsupervised loss without countering this translation will either cause the network to learn from erroneous training data.
D. U-N ET NETWORK ARCHITECTURE
Our U-Net network is shown in Figure 7 and Table 4.
[Table]
D. T RAINING DETAILS D.3. E XPERIMENTS USING U- NET ARCHITECTURE
In keeping with Long et al.
we use a batch size of 1.
We freeze the batch normalization layers within the ResNet encoder
[Table]
using the pre-trained running mean and variance rather than computing per-batch mean and variance during training.
We use the Adam Kingma & Ba (2015) optimization algorithm with a learning rate of 1 × 10 − 4 .
As per the mean teacher algorithm Tarvainen & Valpola (2017), after each iteration the weights w t of the teacher network are updated to be the exponential moving average of the weights w s of the student: w t = α t w t + (1 − α t ) w s , where α t = 0 . 99 .
The Cityscapes images were downsampled to half resolution ( 1024 × 512 ) prior to use, as in Hung et al.
When using Cityscapes we trained for 100,000 iterations using a batch size of 1.
We tuned our approach and selected hyper-parameters using the C AM V ID dataset due to its small size and fast run-time, after which we applied the same hyper-parameters to Cityscapes.
We did not run the augmentation based perturbation experiments on the Cityscapes dataset due to the long run-time involved.
D.3. E XPERIMENTS USING D EEP L AB V 2 ARCHITECTURE
We found that we had to decrease the learning rate to 1 × 10 − 5 to get good performance with
DeepLab v2.
We note that the PyTorch implementation of DeepLab v3 worked best with 1 × 10 − 4 .
We believe that this is because the U-Net and DeepLab v3 networks accept input images with zeromean unit-variance, where DeepLab v2 accepts images whose values are in the range 0 to 255 with only mean subtraction.
For the P ASCAL VOC experiments, we extracted 321 × 321 random crops and used a batch size of 10, in keeping with Hung et al.
For the C ITYSCAPES experiments we used full image crops and a batch size of 2.
D.3. A BLATION
The augmentation based perturbation experiments performed on the C AM V ID dataset were trained for 50 epochs.
The CutMix and CowOut experiments were trained for 100.
D.3. C ONFIDENCE THRESHOLDING
French et al.
(2018) apply conﬁdence thresholding, in which they mask consistency loss to 0 for samples whose conﬁdence as predicted by the teacher network is below a threshold of 0.968.
In the
[Figure]
D. D ETAILED PERFORMANCE TABLES
The detailed per-class performance on the C ITYSCAPES dataset are presented in Table 5 and visualized in Figure 8, ad on the augmented P ASCAL VOC dataset are presented in Table 6 and visualized in Figure 9.
D. H YPER - PARAMETER CHOICE
We explored the hyper-parameters for CowMix using the C AM V ID dataset.
We present our results in the form of bar plots.
Black and green bars at left and right end are the performance of the baseline and fully supervised setups respectively while blue bars show response to hyper-parameter values.
The performance is strong when consistency weight has a value in the range of 3 to 30, withbeing the optimum choice, as seen in Figure 10.
The EMA α value used to update the teacher network had little effect, as seen in Figure 11.
The σ used for Gaussian smoothing yielded good results when between 4 and 8, but was best when drawn from σ ∼ log U (4 , 16) , as seen in Figure 12.
[Figure] [Figure]
[Figure] [Figure]
[Table] [Table]