U NIVERSAL W EAKLY S UPERVISED S EGMENTATION BY P IXEL - TO -S EGMENT C ONTRASTIVE L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Weakly supervised segmentation is challenging as sparsely labeled pixels do not provide sufﬁcient supervision: A semantic segment may contain multiple distinctive regions whereas adjacent segments may appear similar.
Common approaches use the few labeled pixels in all training images to train a segmentation model, and then propagate labels within each image based on visual or feature similarity.
Instead, we treat segmentation as a semi-supervised pixel-wise metric learning problem, where pixels in different segments are mapped to distinctive features.
Naturally, our unlabeled pixels participate not only in data-driven grouping within each image, but also in discriminative feature learning within and across images.
Our results on Pascal VOC and DensePose datasets demonstrate our substantial accuracy gain on various forms of weak supervision including image-level tags, bounding boxes, labeled points, and scribbles.
I NTRODUCTION
Consider semantic segmentation given sparsely labeled pixels in Fig.
1: Each body part is labeled with a single seed pixel and the task is to segment out the entire person by individual body parts.
This task is challenging, as not only a single body part could contain several visually distinctive areas, e.g., head consists of eyes , nose , mouth , beard ), but two adjacent body parts could also have the same visual appearance, e.g., upper arm , lower arm , and hand have the same skin appearance.
Such a seeded segmentation problem belongs to a family of weakly supervised segmentation problems, where the task is to assign a semantic label to each pixel despite that only partial supervision is available during training.
It addresses an extremely practical issue of acquiring as little human annotations as possible, instead of dense pixel-wise labeling for the sake of learning segmentation.
Such weak supervision takes many forms, from image-level tags (Kolesnikov & Lampert, 2016;
Ahn & Kwak, 2018; Huang et al., 2018; Lee et al., 2019), bounding boxes (Dai et al., 2015; Khoreva et al., 2017; Song et al., 2019), keypoints (Bearman et al., 2016), to scribbles (Lin et al., 2016; Tang et al., 2018a;b), with an increasing level of weak supervision.
Image tags only provide semantic information without any localization information, bounding boxes further outline the object sizes and locations, whereas keypoints and scribbles indicate the object foregrounds and rough outreach.
They vary in the amount of and the spatial precision of partial annotations.
Weakly supervised semantic segmentation can be regarded as a
-supervised problem: Some pixels have labels, most don’t, and the key is how to propagate annotations from coarsely and sparsely labeled pixels to unlabeled pixels.
Earlier works tackle this problem by generating pseudo dense supervision (Weston et al., 2012), and using image segmentation techniques such as graph cuts (Boykov et al., 2001), normalized cuts (Shi & Malik, 2000), and Conditional Random Fields (CRF)
(Kr ¨ ahenb ¨ uhl & Koltun, 2011).
While such pseudo-supervision is imperfect, it effectively propagates labels from labeled pixels to unlabeled pixels within individual images, and a consensus from the entire image helps mitigate errors.
Recent works incorporate these ideas as an additional unsupervised loss on the feature learned for segmentation (Tang et al., 2018b): While seeded pixels receive labeling supervision, pixels in different segments shall have distinctive feature representations.
Unlike traditional segmentation based on a ﬁxed hand-crafted feature, with deep learning, seeded segmentation is no longer a seeded diffusion process in the pixel space, but a self-organizing discriminative learning in the feature space.
We propose a Semi-supervised Pixel-wise Metric Learning (SPML) model that can handle all these weak supervision varieties with a single pixel-to-segment contrastive learning formulation (Fig. 2).
Instead of classifying pixels, our metric learning model learns a pixel-wise feature embedding based on common grouping relationships derived from any form of weak supervision.
Our key insight is to integrate unlabeled pixels into both supervised labeling and discriminative feature learning.
They shall participate not only in data-driven grouping within each image, but also in discriminative feature learning within and more importantly across images.
Intuitively, seed pixels are labeled not meant for themselves alone, but also for their surround pixels that share visual similarity.
On the other hand, unlabeled pixels are not just passively brought into discriminative learning induced by sparsely labeled pixels, they themselves are organized based on bottom-up grouping cues (such as grouping by color similarity and separation by strong contours), and when they are examined across images, repeated patterns of frequent occurrences would also form a cluster that demand active discrimination from other patterns.
We capture the above insight in a single learning objective in the pixel-wise metric learning framework for segmentation, the goal of which is to map each pixel into a point in the feature space so that pixels in the same (different) semantic groups are close (far) in the feature space.
Our model extends
SegSort (Hwang et al., 2019) from its fully supervised and unsupervised segmentation settings to a universal weakly-supervised segmentation setting.
With a single consistent feature learning criterion, such a model sorts pixels discriminatively within individual images and sorts segment clusters discriminatively across images, both steps minimizing the same feature discrimination loss.
Our experiments on Pascal VOC (Everingham et al., 2010) and DensePose (Alp G¨uler et al., 2018) demonstrate consistent gains over the state-of-the-art (SOTA), and the gain is substantial especially for the sparsest keypoint supervision.
R ELATED W ORK
Semi-supervised learning.
(Weston et al., 2012) treats it as joint learning with labeled and unlabeled data.
The goal is to learn better feature representations by observing the common structures between labeled and unlabeled sets.
Two lines of research have emerged.
One is to leverage generative models (Kingma et al., 2014; Rasmus et al., 2015) for learning the underlying structure of unlabeled data.
Such methods require good generative models and domain transfer methods to generate reliable training data.
The other is to add regularization through consistency loss, e.g., adversarial ensembling (Miyato et al., 2018), imitation learning and distillation (Tarvainen & Valpola, 2017), cross-view ensembling (Clark et al., 2018).
Regularization approaches for weakly-supervised segmentation are most related to transductive learning (Joachims, 2003;
Zhou et al., 2004;
Fergus et al., 2009; Liu et al., 2019), where labels are propagated to unlabeled data via clustering in the pre-trained feature space.
Our work does transductive learning in an adaptively learned feature space.
Weakly-supervised semantic segmentation.
Partial annotations include scribbles (Lin et al., 2016;
Tang et al., 2018a;b; Wang et al., 2019), bounding boxes (Dai et al., 2015; Khoreva et al., 2017;
Song et al., 2019), points (Bearman et al., 2016), or image tags (Papandreou et al., 2015; Kolesnikov & Lampert, 2016; Ahn & Kwak, 2018; Huang et al., 2018; Li et al., 2018; Lee et al., 2019; Shimoda & Yanai, 2019; Zhang et al., 2019; Yao & Gong, 2020; Chang et al., 2020; Araslanov & Roth, 2020; Wang et al., 2020; Fan et al., 2020; Sun et al., 2020).
Xu et al. (2015) formulates all types of weak supervision as linear constraints on a SVM.
Recent works (Lin et al., 2016; Kolesnikov & Lampert, 2016; Pathak et al., 2015) typically use Class Activation Map (CAM)
(Zhou et al., 2016) to obtain an initial dense mask and then train a model iteratively. GAIN (Li et al., 2018) utilizes image tag or bounding box annotations to reﬁne these class-speciﬁc activation maps. Sun et al. (2020) considers within-image relationships and explores the idea of co-segmentation.
Fan et al.
(2020) estimates the foreground and background for each category, with which the network learns to generate more precise CAMs.
Regularization is enforced at either the image level (Lin et al., 2016; Kolesnikov &
Lampert, 2016; Pathak et al., 2015) or the feature level (Tang et al., 2018a;b) to produce better dense masks.
We incorporate this concept into adaptive feature learning and train the model only once.
It is worth noting that each of the annotations carries different assumptions, we propose to unify all these types of weak annotations in a single contrastive learning framework.
Non-parametric segmentation.
Prior to deep learning, non-parametric models (Russell et al., 2009;
Tighe & Lazebnik, 2010; Liu et al., 2011) usually use designed features with statistical or graphical models to segment images.
Recently, inspired by non-parametric models for recognition (Wu et al., 2018b;a), SegSort (Hwang et al., 2019) captures pixel-to-segment relationships via a pixelwise embedding and develops the ﬁrst deep non-parametric semantic segmentation for supervised and unsupervised settings.
Our work builds upon SegSort and thus have the ﬂexibility of a nonparametric model at capturing data relationships and modeling subclusters within a category.
S EMI -S UPERVISED P IXEL - WISE M ETRIC L EARNING M ETHOD
Metric learning develops a feature representation based on data grouping and separation cues.
Our method (Fig. 3) segments an image by learning a pixel-wise embedding with a contrastive loss between pixels and segments.
We start from deﬁning two disjoint sets – positive and negative segments (exemplars) with respect to pixel i .
Our goal is to group i with positive segments while separating it from negative ones.
Given latent feature φ ( i ) at pixel i and a speciﬁc distance metric, the objective is to decrease (increase) the distance between φ ( i ) and its positive (negative) segments.
Under the supervised setting, we can deﬁne the positive and negative sets as the same- and differentcategory pixels with respect to pixel i , which are denoted as C + and C − .
However, this idea is not applicable to weakly- or un-supervised setting, when the label at x i is unknown.
Particularly for weakly-supervised segmentation, we only have sparsely labeled pixels in the image, resulting in much smaller sets of C + and C − and degraded learning efﬁciency.
Our basic idea is to enlarge the sets of C + and C − to improve the learning efﬁcacy.
By exploring different relationships and assumptions in the image data, we are able to generate abundant positive and negative segments for any pixel at the same time, providing more supervision in the latent feature space.
We use four types of pixel-to-segment semantic relationships (Fig. 4):
Low-level image similarity : Positive and negative segments can be generated by selecting pixel i ’s own segment or all the other segments, respectively.
Semantic annotation : Pseudo-labeled segments can be generated by their majority pixels’ annotations.
Here, the annotations can be dense masks, scribbles, points, or the localization cues generated by Class Activation Maps (Zhou et al., 2016).
Semantic co-occurrence : Positive (negative) segments can be generated by using other images that include (exclude) the same category as in pixel i ’s image.
Feature afﬁnity : Pseudo-labeled segments can be generated by propagating labels from known pixels to their nearest neighbor segments in the feature space, assuming that semantic clusters emerge during training and their territories are thus expanded.
3. P IXEL - TO -S EGMENT S EMANTIC R ELATIONSHIPS
We consider propagating semantic labels from labeled data C to unlabeled data U by exploiting the aforementioned relationships.
We denote C and U as the sets of segment indices.
Our label propagation is driven by grouping and separating data in a learned feature space.
We now describe each pixel-to-segment semantic relationship for augmenting the sets of positive/negative segments using both labeled and unlabeled pixels.
Low-level image similarity.
We propagate the labels within visually coherent regions, as visual similarity often goes with semantic similarity.
To generate an over-segmentation, we follow
Hwang et al.
by using the HED contour detector (Xie & Tu, 2015) (pre-trained on BSDS dataset (Arbelaez et al., 2010)) and the procedure in gPb-owt-ucm (Arbelaez et al., 2010).
Such bottom-up segmentation techniques consider both local and global appearance afﬁnity without semantic information.
Some of the segments could contain pixels from different categories.
For learning pixel embedding at pixel i , we deﬁne positive and negative segments w.r.t.
i as i ’s own segment and all the other segments, denoted as V + and V − , respectively.
Such a formulation helps learn embeddings that respect low-level visual cues.
In our implementation, we only consider segments in the same image as i ’s image as positive and negative samples.
We follow SegSort Hwang et al.
(2019) to align the contour-based over-segmentations with segmentations generated by K-Means clustering in SegSort.
Semantic annotation.
We contrast the pixel embedding based on semantic annotations.
For box and image-level tag annotations, which do not provide precise localization, we instead produce semantic cues with CAM (Zhou et al., 2016).
We align these semantic annotations with oversegmentations induced by the SegSort feature.
We then deﬁne segments carrying same-category and differentcategory labels as positive and negative samples.
To reduce noise, we ignore all the unlabeled segments.
We denote C + and C − as all the same-category and different-category regions.
Semantic co-occurrence.
Semantic context characterizes the co-occurrences of different objects and can be used to group and separate pixels.
We deﬁne semantic context as the union of object classes in each image.
Even without the location of labels, we can leverage semantic context to impose global regularization in the latent feature space, i.e., the pixel-wise feature embedding should be separated from all the other pixels from images without overlapping object categories.
We denote O + as the set of segments among images with overlapping object categories.
For example, if pixel i is labeled as sofa and another image also contains sofa, all the segments from that image are included in O +
; otherwise they are considered negative segments O − .
This semantic context relationship does not require localization annotations or cues yet enhances the global and higher-level regularization on the pixel embedding.
Feature afﬁnity.
We consider propagating the labels from an annotated set to an unlabeled set by nearest neighbor search in the feature space.
We assume that semantic clusters emerge during training with sparse supervision, reinforced by aforementioned pixel-to-segment relationships.
By propagating labels in the feature space, we reinforce the learning of semantic clusters.
To be speciﬁc, we assign a semantic label for each unlabeled segment by ﬁnding its nearest labeled segment in the feature space.
We denote this new labeled set (together with the original labeled set) as ˆ C , where ˆ C + denotes all the same-category segments other than segment s in ˆ C and pixel i belongs to the segment s .
Such a formulation is based on three assumptions: 1) The size of the original labeled set is large enough to cover the feature space, 2) the labeled segments are distributed uniformly in the feature space, and 3) the embedding already encodes certain semantic information.
Therefore, we only apply this relationship to propagate the keypoint annotations in the DensePose dataset, where each body part is annotated by a point.
3. P IXEL - WISE M ETRIC L EARNING L OSS
SegSort (Hwang et al., 2019) is an end-to-end image segmentation model that generates a pixel-wise embedding and a consequent over-segmentation.
It assumes that each segment has an independent normal distribution on a hypersphere.
Spherical K-Means clustering (Banerjee et al., 2005) is used for segmenting an image and learning the discriminative feature clustering jointly.
It is worth noting that, such an assumption indicates homogeneous representations within each segment.
To learn the embedding, SegSort formulates a maximum likelihood loss that maximizes the discrimination between segments.
In addition, soft neighborhood assignment (Goldberger et al., 2005) is incorporated to enforce grouping of semantically similar segments.
During inference, the segment labels are predicted by K-Nearest Neighbor retrievals.
Spherical K-Means clustering (Banerjee et al., 2005) is an EM framework, which generates segmentations from a set of pixel embeddings, or E = { eee i } =
{ φ ( i ) } in an image.
During M-step, the mean embedding, or the prototype , ppp s is calculated for segment s , where ppp s
[Equation]
[Figure]
[Equation]
and
RRR s denotes the region of segment s .
E-step assigns pixel embedding eee i to the segment of nearest prototype ppp s , where z i = arg max s ppp (cid:62) s eee i is the segment index of pixel i .
Therefore, the posterior probability of pixel i belonging to segment s can be derived as p ( z i = s | eee i , Θ
[Equation]
=
[Equation]
where κ denotes the concentration hyper-parameter and Ω represents the set of all segments.
The SegSort objective is to minimize the negative log-likelihood loss:
[Equation]
To strengthen the grouping of same-category segments, Hwang et al.
(2019) further adopt the soft neighborhood assignments (Goldberger et al., 2005):
[Equation]
where c + is the segment index of all the other same-category segments as s .
We also denote c − as the index of all the different-category segments from s .
By letting C + = { c + } and C − = { c − } w.r.t pixel i , the corresponding loss is formulated:
[Equation]
For our weakly supervised segmentation, the total pixel-to-segment contrastive loss for pixel i consists of 4 terms, one for each of the 4 pixel-to-segment attraction and repulsion relationships:
[Equation]
Fig.
5 shows how our metric learning differs from previous classiﬁcation methods at utilizing labeled and unlabeled pixels and segments.
By easily integrating a full range of pixel-to-segment attraction and repulsion relationships from low-level image similarity to mid-level feature afﬁnity, and to highlevel semantic co-occurrence, we go far beyond the direct supervision from semantic annotations.
E XPERIMENTS
Dataset: Pascal VOC 2012.
Pascal VOC 2012 dataset (Everingham et al., 2010) includes 20 object categories and one background class.
Following Chen et al.
(2017), we use the augmented train-
[Table]
Dataset: DensePose.
DensePose (Alp G¨uler et al., 2018) is a human pose parsing dataset based on MSCOCO (Lin et al., 2014).
The dataset is annotated with 14 body part classes.
We extract the keypoints from the center of each part segmentation.
The training set includes 26,437 images and we use minival2014 set for testing, which includes 1,508 images.
See Appendix for more details.
Architecture, training and testing.
For all the experiments on VOC, we base our architecture as DeepLab (Chen et al., 2017) with ResNet101 (He et al., 2016) as backbone network.
For the experiments on DensePose dataset, we adopt PSPNet (Zhao et al., 2017) as backbone network.
We only use models pre-trained on ImageNet (Deng et al., 2009) dataset.
For training our models, we set λ
I , λ C , λ O and λ A according to different types of annotations and datasets, which is shown in table 1.
For inference, we follow SegSort (Hwang et al., 2019) to perform k-nearest neighbor retrievals.
See Appendix for more detail of hyper-parameters and setting for training and testing.
Table 1 summarizes the four relationships we experiment with for different annotation types and datasets.
They can be formulated jointly in a single pixel-wise metric learning framework (Fig. 3).
Pascal: Image tag annotations.
On Pascal VOC dataset, our method outperforms others by a large margin.
Table 2 shows that, without additional saliency labels, our method still achieves SOTA.
Compared to (Chang et al., 2020), we improves mIoU by a sizable 4 . 5% .
Pascal: Box annotations.
Table 2 shows that, with the same DeepLab/ResNet101 backbone network, our method outperforms others: On the validation set, our performance is 2 .
9% higher.
[Table]
Our SPML -
[Table]
Pascal: Scribble annotations.
Table 3 shows that, without CRF post-processing, we get 74 .
1% mIoU, achieving 97 .
6% of full supervision performance; with CRF post-processing, we reach new
SOTA: We get 75 .
9% mIoU, achieving 98 .
6% of full supervision performance.
Pascal: Varying sparsity of scribble and point annotations.
Exploiting metric learning with different relationships in the data frees us from the classiﬁcation framework, which requires a huge number of annotations.
We investigate whether we can retain the segmentation performance with even sparser annotations.
Table 3 shows that, as we shorten the length of scribbles from 100% , 80% , 50% , 30% to 0% (points), we reach 98 .
1% and 92 .
5% of full supervision performance.
Compared to the scribble annotations, our performance only drops by 2 .
6% with using point labels, which is signiﬁcantly better than the baseline.
DensePose
[Table]
WvF
Tang et al
[Table]
(2018b
[Table]
Our
[Table]
Pascal: Learning from different weak annotations.
DensePose: Point annotations.
We train our baseline using the code released by Tang et al.
(2018b).
Tableshows that, without CRF post-processing, we are able to [Table] reach 81 .
8% of full supervision performance.
Note that our method does not require a two-stage training procedure as in the baseline.
Visual quality comparison and Ablation study.
Fig.shows that our results are well aligned with image boundary and visually close to fully-supervised counterpart.
Fig.
7 shows signiﬁcant improvement of our results by additively introducing different relationships for more regularization.
Please refer to the Appendix for more details and ablation studies.
S UMMARY
We propose a novel universal weakly-supervised semantic segmentation method via Semisupervised Pixel-wise Metric Learning.
Four common types of pixel-to-segment attraction and repulsion relationships can be derived, whether the partial annotation is coarse image tags and bounding boxes, or sparse keypoints and scribbles.
Our results on PASCAL VOC and DensePose show consistent and substantial gains over SOTA, especially for the sparsest keypoint supervision.
A A PPENDIX
We propose a single pixel-to-segment contrastive learning loss formulation for weakly supervised semantic segmentation.
We explore different types of visual relationships to group and separate pixels within and across images.
We demonstrate state-of-the-art performance using our proposed method with different types of annotations.
Here, we include more details on the following aspects:
• We present the visual results of our method in A.1.
• We showcase the semantic cues generated by CAM for image tag and bounding box annotations in A.2.
• We illustrate the data pre-processing used for DensePose dataset in A.3.
• We describe the details of our experimental settings and hyper-parameters in A.4.
• We present the ablation study regarding hyper-parameters in A.5.
• We present mIoU performance with varying sparsity of scribble annotations on Pascal dataset in A.6.
• We present per-category results with scribble and point annotations on Pascal and DensePose dataset in A.7.
• We provide detailed experimental results using image tag, bounding box and scribble annotations on Pascal dataset in table 13, 12 and 11, respectively.
A. V ISUALIZATION
We present the visual results on VOC (with image tags, bounding boxes and scribbles) and DensePose (with keypoints) dataset in ﬁgure 8.
We observe that our segmentation results are especially well aligned with image boundary.
When visual evidence is prominent, our weakly-supervised results are even better than the fully-supervised counterpart.
We then demonstrate the efﬁcacy of each visual relationship in ﬁgure 9.
By adding semantic annotation , low-level image similarity and feature afﬁnity progressively, we observe consistent improvement of our results.
The predicted segmentation becomes more coherent and better aligned with image boundary.
We lastly showcase that our method implicitly encodes semantic contexts.
In ﬁgure 10, We observe that retrieved segments appear in the similar semantic context as the query segments.
For examples, given a bottle next to a desktop, our model retrieves bottles also next to a desktop; a set of sofas in a living room can be retrieved using one sofa query example; screens of a desktop can also be retrieved likewise.
A. S EMANTIC A NNOTATIONS
Since image tag and bounding box annotation do not provide any of precisely localized semantic information, we adopt CAM (Zhou et al., 2016) to produce localized semantic cues.
Without using additional saliency labels, we use the classiﬁer trained by Ahn & Kwak (2018) to generate
CAM.
Let M c ( I ) be the activation map of class c in image I .
Proposed in Ahn & Kwak (2018),
M c ( I ) is normalized to the range between 0 and 1, where M c ( I ) =
M c ( I ) max M c ( I ) .
The background conﬁdence M bg ( I ) can then be estimated by M bg ( I ) = { 1 − max c ∈ C M c ( I ) } α , where α is the hyper-parameter adjusting background conﬁdence and C is the set of all object categories.
In our experiments, we set α to 32 and conﬁdence threshold to 0 .
3 and 0 .
2 for image tag and bounding box annotations.
For bounding box annotations, we further restrict all the regions outside bounding boxes as “background”.
See ﬁgure 11 for more visual examples.
A. D ATA PRE - PROCESSING FOR D ENSE P OSE DATASET
We next illustrate our pre-processing to generate training labels given keypoint annotations in DensePose dataset.
As shown in ﬁgure 12, we ﬁrst assume a Gaussian heat map from every keypoint.
By thresholding, we derive 3 regions from every Gaussian blob: labelled, unknown and background region.
In labelled region, pixels are annotated as each body part.
We then propagate labels, including background class, to pixels in the unknown region.
The std of Gaussian heat map is estimated from instance size, and we use ground-truth information in our paper.
A. H YPER - PARAMETERS AND E XPERIMENTAL S ETUP Architecture and training.
For all the experiments on VOC, we base our architecture as
DeepLab (Chen et al., 2017) with ResNet101 (He et al., 2016) as backbone network.
For the experiments on DensePose dataset, we adopt PSPNet (Zhao et al., 2017) as backbone network.
We only use models pre-trained on ImageNet (Deng et al., 2009) dataset.
We next describe the hyper-parameters used for each experiment.
For all the experiments, we set “batchsize” to 8 , “cropsize” to 336 × 336 and 480 × 480 for VOC and DensePose dataset.
Following (Chen et al., 2017), we adopt poly learning rate policy by multiplying base learning rate by 1 − ( iter max iter ) 0 .
We set initial learning rate to 0 .
003 , momentum to 0 .
For the hyper-parameters in SegSort framework, we use unit-length normalized embeddings of dimension 64 .
We iterate KMeans clustering for 10 iterations and generate 25 and 144 clusters on VOC and DensePose dataset.
The concentration parameter κ is set to 6 , 16 , 12 , 12 for semantic annotation , low-level image similarity , semantic co-occurrence and feature afﬁnity , respectively.
Shown in table 1, we also set λ
I , λ C , λ O and λ A according to different types of annotations and datasets.
We train for 30 k and 45 k iterations on VOC and DensePose dataset for all the experiments.
We use additional memory banks to cache up previous 2 batches.
Inference and testing.
Following Hwang et al.
(2019), we conduct k-nearest neighbor retrievals for inference.
We produce oversegmentations and calculate the prototypes in the training set.
For
Pascal VOC and DensePose dataset, We partition each image into 144 and 576 clusters, respectively.
We set k = 20 for nearest neighbor retrievals, and then, we infer the semantic label of each queried segment with majority label of retrievals.
Notably, we do not use left-right ﬂipped and multi-scale inputs as data augmentation for inference.
For fair comparison, we also use CRF post-processing (if speciﬁed) to improve our performance on Pascal VOC dataset under different settings of supervision.
[Table]
A. A BLATION STUDY OF HYPER - PARAMETERS
We conduct ablation study over different regularizations on Pascal VOC and DensePose dataset.
As shown in table 5, we achieve the most optimal performance on Pascal VOC dataset with λ I = 0 .and
λ O = 0 .
We also observe performance drops 0 .
3 of mIoU by adding feature afﬁnity regularization.
We argue that scribble/box/point annotations are not uniformly distributed across object instance and background, and results in noisy label propagation.
In table 6, we also demonstration
[Table]
the efﬁcacy of each loss.
We show that regularization on semantic annotation, low-level image similarity and feature afﬁnity improve the performance signiﬁcantly, whereas, our model doesn’t beneﬁt from semantic co-occurrence regularization
[Table]
[Table] [Table]
A. MEAN I O U PERFORMANCE WITH VARYING SPARSITY OF SCRIBBLES .
We also report absolute mIoU performance by varying sparsity of scribbles on Pascal VOCvalidation set.
The results are summarized in table 7.
Our results are higher than baseline by larger margin when the annotation is sparser.
A. P ER - CATEGORY M I O U ON P ASCAL VOC AND D ENSE P OSE DATASET .
We next present per-category results on Pascal VOC and Denspose dataset.
In table
[Table]
we compare with Tang et al.
(2018b) on VOC validation set.
Without- and with CRF post-processing, our method outperform the baseline method among most categories by large margin.
We further conduct
[Table] [Table]
Our SPML
DeepLab/ResNet
[Table]
Our SPML
DeepLab
[Table]
Our SPML
PSPNet/ResNet
[Table]
Our SPML
PSPNet
[Table]
Our SPML
[Table]
Our SPML
[Table] [Table]
Khoreva
[Table]
DeepLab/ResNet (cid:88)
[Table]
Song
[Table]
al.
DeepLab/ResNet (cid:88) 70.
Our SPML
DeepLab/ResNet71.
Our SPML
DeepLab/ResNet (cid:88) 73.
Our SPML
PSPNet/ResNet73.
Our SPML
PSPNet/ResNet (cid:88) 74.
experiments on VOC
, using we can retrieve most w supervision
We also compare
DensePose in table 10.
We train our baseline method using the code released by Tang et al
2018b).
We outperform the baseline method large margin every category
[Table]
DeepLab/ResNet - - 64.
Chang et al.
DeepLab/ResNet - (cid:88) 66.
Our SPML
DeepLab/ResNet - - 68.
Our SPML
DeepLab/ResNet - (cid:88) 70.
Our SPML
PSPNet/ResNet - - 70.
Our SPML
PSPNet/ResNet - (cid:88) 72.