L EARNING TO A NNOTATE P ART S EGMENTATION WITH G RADIENT M ATCHING
Anonymous authors
Paper under double-blind review
A BSTRACT
The success of state-of-the-art deep neural networks heavily relies on the presence of large-scale labeled datasets, which are extremely expensive and time-consuming to annotate. This paper focuses on reducing the annotation cost of part segmentation by generating high-quality images with a pre-trained GAN and labeling the generated images with an automatic annotator.
In particular we formulate the annotator learning as the following learning-to-learn problem. Given a pre-trained GAN the annotator learns to label object parts in a set of randomly generated images such that a part segmentation model trained on these synthetic images with automatic labels obtains superior performance evaluated on a small set of manually labeled images.
We show that this nested-loop optimization problem can be reduced to a simple gradient matching problem, which is further efﬁciently solved with an iterative algorithm.
As our method suits the semi-supervised learning setting, we evaluate our method on semi-supervised part segmentation tasks.
Experiments demonstrate that our method signiﬁcantly outperforms other semi-supervised competitors, especially when the amount of labeled examples is limited.
I NTRODUCTION
In recent years, deep neural networks have shown a remarkable ability to learn complex visual concepts from large quantities of labeled data.
However, collecting manual annotations for a growing body of visual concepts remains an extremely expensive practice, especially for pixelwise prediction tasks such as semantic segmentation. A common and effective way of reducing the dependence on manual labels is ﬁrst learning representations from unlabeled data (Pathak et al., 2016; Noroozi & Favaro, 2016; Gidaris et al., 2018; He et al., 2020) and then deploying learned representations to a supervised learning task (Chen et al., 2020; Zhai et al., 2019).
The insight of this paradigm is that some of the transferred representations are informative, easily identiﬁed, and disentangled, such that small amount of labeled data sufﬁces to train the target task. A promising direction for learning such representations is using generative models (Kingma &
Welling, 2014; Goodfellow et al., 2014), since they attain the capability of synthesizing photorealistic images and disentangling factors of variation in dataset.
Recent methods (Goetschalckx et al., 2019; Shen et al., 2020; Karras et al., 2019; Plumerault et al., 2020; Jahanian et al., 2020; Voynov & Babenko, 2020; Spingarn-Eliezer et al., 2021; H¨ark¨onen et al., 2020) show that certain factors such as object shape and position in the images synthesized by generative adversarial networks (GANs) (Goodfellow et al., 2014) can be individually controlled by manipulating latent features. Building on the success of powerful GAN models, Zhang et al.
(2021) and Tritrong et al.
(2021) demonstrate that feature maps in StyleGAN (Karras et al., 2019; 2020b;a) can be mapped to semantic segmentation masks or keypoint heatmaps through a shallow decoder, denoted as annotator in this paper. Remarkably the annotator can be trained after few synthesized images are labeled, and the joint generator and trained annotator can be used as an inﬁnite labeled data generator. Nonetheless, labeling synthesized images with “human in the loop” has two shortcomings: (i) it can be difﬁcult to label the generated images of low quality; (ii) it requires continual human effort whenever the generator is changed.
Motivated by these limitations, we remove the necessity of noisy and challenging annotation of the generated images by replacing them with the same number of annotated realistic images.
However, it is non-trivial to learn the annotator from labeled images, because images are not always possible to be inverted into generator features while the annotator only receives generator features as input. Therefore, we formulate the annotator learning as a learning-to-learn problem – the annotator learns to label a set of randomly generated images such that a model, denoted as student, trained on these automatically labeled images obtains superior performance evaluated on a small set of manually labeled images.
This problem involves solving a nested-loop optimization, where the inner loop optimizes the student and the outer loop optimizes the annotator based on the solution of optimization of student.
In contrast to existing learning-to-learn algorithms (Li et al., 2019; Pham et al., 2021) that employs end-to-end gradient-based meta learning techniques, we reduce this complex and expensive optimization problem into a simple gradient matching problem and efﬁciently solve it with an iterative algorithm.
We show that our method obtains performance comparable to the supervised learning counterparts (Zhang et al., 2021; Tritrong et al., 2021) yet overcomes their shortcomings such that labeled data within a broader range can be utilized.
Our method requires GANs pre-trained on large quantity of unlabeled and learns annotator using a small amount of labeled data, which suits a semi-supervised learning setting.
Furthermore, GANs mostly show their promise to synthesize images of particular object classes.
Therefore, we benchmark our method with respect to semi-supervised object part segmentation task on a variety of part segmentation datasets.
Experimental results show that our method outperforms other semi-supervised segmentation methods with a large margin, especially under the circumstances of extremely limited labeled data.
R ELATED W ORK
Semi-supervised learning Semi-supervised learning (SSL) augments the training of neural networks from a small amount of labeled data with a large-scale unlabeled dataset.
A rich body of work regularize networks on unlabeled data with consistency regularization.
In particular, networks are required to make consistent prediction over previous training iterations (Laine & Aila, 2017; Tarvainen & Valpola, 2017; Izmailov et al., 2018), noisy inputs (Miyato et al., 2018), or augmented inputs (Berthelot et al., 2019b;a; Sohn et al., 2020).
This paradigm is also interpreted as a teacherand-student framework where teacher produces pseudo labels for unlabelled data to supervise the training of students.
Pseudo labels can be enhanced with heuristics (Lee et al., 2013; Berthelot et al., 2019a) or optimized towards better generalization (Pham et al., 2021;
Li et al., 2019).
Apart from image classiﬁcation tasks, the above ideas are also successfully applied to semantic segmentation tasks (Hung et al., 2018; Ouali et al., 2020; Ke et al., 2020; French et al., 2020).
Our work can also be interpreted from teacher-and-student perspective.
In contrast to the above methods that adopt homogeneous network structures for teacher and student, our work employ generative models as teacher and discriminative models as student.
Other SSL approaches explore the utility of unlabelled data from representation learning perspective.
In particular, deep neural networks are ﬁrst optimized on the unlabeled data with self-supervised learning tasks (Pathak et al., 2016; Noroozi & Favaro, 2016;
Gidaris et al., 2018;
He et al., 2020) to gain powerful and versatile representations, and then ﬁnetuned on the labeled data to perform target tasks (Chen et al., 2020; Zhai et al., 2019).
In addition, such efﬁcient and transferable representations can also be learned with generative models that are typically trained without labels.
Therefore, other works (Li et al., 2021; Zhang et al., 2021; Tritrong et al., 2021)
decode generative features into dense annotations such as segmentation masks and keypoint heatmap to facilitate SSL.
Our work is most related to DatasetGAN (Zhang et al., 2021) and RepurposeGAN (Tritrong et al., 2021) while addresses their limitations that labeling generated images is required.
Semantic Part Segmentation Semantic part segmentation provides dense form of representations for both rigid and non-rigid objects, which is of great signiﬁcance in tremendous computer-vision tasks such as human parsing (Dong et al., 2014; Nie et al., 2018; Fang et al., 2018;
Gong et al., 2017), pose estimation (Yang & Ramanan, 2011; Shotton et al., 2011; Xia et al., 2017) and 3D object understanding (Yi et al., 2016; Song et al., 2017).
The challenge of part segmentation arises from viewpoint changes, occlusion, and lack of clear boundaries for certain parts.
Existing datasets manage to alleviate the problem by enrich the data and annotations.
CelebAMask-HQ (Lee et al., 2020a) is a large-scale face image dataset annotated with masks of face components.
Chen et al.
(2014) provides additional part segmentation upon PASCAL VOC dataset (Everingham et al., 2010).
Those datasets meet the demands of latest data-driven and deep neural network based method (Lee et al., 2020b; Wang et al., 2015; Wang & Yuille, 2015).
However, a well-curated part segmentation dataset containing ﬁne-grained pixel-wise annotation usually requires time-consuming manual labeling work, which holds back the development of this ﬁeld.
This problem also motivates the latest work (Zhang et al., 2021;
Li et al., 2021; Tritrong et al., 2021) to leverage a generative model as an inﬁnite data generator in a semi-supervised learning setting, which gain promising results.
Gradient matching Modern deep learning optimization methods are based on the gradients of loss with respect to network parameters.
Gradients contain rich information such that training data might be recovered from gradients (Zhao et al., 2020; Yin et al., 2021). Moreover, gradient matching has been widely discussed in meta learning (Li et al., 2018), dataset condensation (Zhao et al., 2021; Zhao & Bilen, 2021) etc .
Our work is interestingly connected to the recent work on dataset condensation (DC)
(Zhao et al., 2021; Zhao & Bilen, 2021) that condenses a large-scale dataset into a small synthetic one with gradient matching.
While DC shows that gradient matching problem formulates the synthesis of a small dataset to represent a large dataset, our work shows that gradient matching problem can also be employed to learn annotations for GAN generated images from a few labeled examples.
M ETHOD 3. P RELIMINARY
Let G denote a generative model trained from a dataset D tot without labels.
For example, G can be trained via adversarial learning (Goodfellow et al., 2014) such that it can generate photo-realistic images.
The generation process is formulated as
[Equation]
where z denotes a random variable drawn from a prior distribution P z which can be a uniform or Gaussian distribution, x denotes the generated image, and h denotes the hidden features in G .
An annotator
A ω is constructed to decode h into labels
[Equation]
where ω parameterizes the annotator.
Let D l denote a manually labeled dataset.
Given a ﬁxed G , our problem is to learn annotator A ω from D l such that A ω can annotate images generated from G .
Revisiting supervised learning manner DatasetGAN (Zhang et al., 2021) and RepurposeGAN (Tritrong et al., 2021) train annotators in a supervised learning manner.
First, a set of synthetic images { x i } Ni =1 generated from G are selected and annotated by humans and their generator features { h i } N i =1 are reserved.
These features and manual labels { ˆ y i } N i =1 constitute the labeled dataset D l = { ( x i , h i , ˆ y i ) } Ni =1 .
Second, an annotator is trained by minimizing the loss computed between annotator prediction and ground truth labels,
[Equation]
where f denotes the loss function for speciﬁc tasks, e.g . per-pixel cross-entropy function for segmentation tasks.
This method, however, requires D l to be a synthetic and generator-speciﬁc set, which has the following two defects.
(i) Details might be illegible in synthetic images, which increases the difﬁculty of annotation.
(ii) Data needs to be re-labeled whenever the generator is changed.
Although projecting labeled images into GAN latent space can alleviate the above drawbacks, it leads to degenerate performance due to inexact inversion (see Section C.1 in the appendix).
Learning to learn We formulate the problem of learning A ω as a learning-to-learn problem like Li et al.
Pham et al.
In particular, we introduce a student model S θ which is a deep neural network parameterized with θ . It takes as input an image x and outputs a segmentation mask [Equation] [Figure]
𝜽 𝓛 𝒈 (𝜽,𝝎) 𝜵 𝜽 𝓛 𝒍 (𝜽) [Figure]
𝜃 [Figure] 𝜔) [Figure]
𝜃)
[Figure] y .
This student model learns the target task only from the automatically labeled generated images, which is formulated as
[Equation]
where f denotes the loss function for the target task. The optimal solution of Equation 4 depends on the ω , which is therefore denoted as θ ∗ ( ω ) .
The performance of learned student can be evaluated with loss on the labeled dataset D l as [Equation] The aim of learning the annotator is to produce high-quality labels such that the learned student network achieve minimal loss on D l , which is formulated as
[Equation]
where θ [Equation] [Equation] This is a nested-loop optimization problem, where the inner loop optimizes student (Equation 4) and the outer loop optimizes annotator based on the solution of the inner loop.
It is noteworthy that this optimization problem does not require manual labeling of generated images. To this end, any labeled images, including real labeled images , can serve as D l .
While this problem can be solved with end-to-end gradient-based meta learning approaches (Li et al., 2019; Pham et al., 2021), its solution involves expensive unrolling of the computational graph over multiple θ states, which is memory and computation intensive.
Hence, we reduce it into a simple gradient matching problem as follows.
Gradient matching Inspired by Li et al.
(2018), we reduce the nested-loop optimization problem (Equation 6) into a gradient matching problem.
First, as commonly done in gradient-based meta learning approaches, the optimization of student (Equation 4) is approximated with one-step gradient descent,
[Equation]
where η denotes the learning rate and θ 0 denotes the initialized parameter.
By plugging this equation into L l ( θ ∗ ( ω )) , unfolding it as Taylor series and omitting the higher order term, we have
[Equation]
[Equation]
The last approximation is conditioned on (cid:107) η ∇ θ L g ( θ t , ω ) (cid:107) ≤ (cid:15) , where (cid:15) is a very small amount.
By further omitting the constant term L l ( θ 0 ) in Equation 8, we obtain the approximation of Equationas
[Equation]
Inputs :
G trained generator
D l set of labeled examples ω , A ω , η ω initial annotator parameters, annotator, and learning rate for annotator θ , S θ , η θ initial student parameters, student, and learning rate for student
T , K , and B total number of optimization steps, interval of updating student, and batch size 1 for t ← 1 to T do // update annotator A ω
B l ← { ( x i , ˆ y i ) ∼ D l } Bi = // sample a batch of labeled examples ∇ θ L l ← 1 B (cid:80) Bi =1 ∇ θ f ( S θ ( x i ) , ˆ y i ) //
compute gradients on B l
B g ← { ( x i , A ω ( h i )) } Bi = // sample a batch of synthetic data ∇ θ L g ← 1 B (cid:80) Bi
=1 ∇ θ f
( S θ ( x i ) , A ω ( h i )) // compute gradients on B g
L gm ← D ( ∇ θ L g , ∇ θ L l ) // compute gradient matching loss ω ← ω − η ω ∇ ω
L gm // update annotator parameters // update student {+S θ if t mod K = 
 then+}
B g ← { ( x i , A ω ( h i )) } Bi = // sample a batch of synthetic data L g ← 1 B (cid:80) Bi
=1 f ( S θ ( x i ) , A ω ( h i )) // compute loss on synthetic data θ ← θ − η θ ∇ θ L g // update segmentor parameters end 13 endOutput : annotator A ω and segmentor S θ This constrained optimization problem is further relaxed into the following unconstrained optimization problem
[Equation] where to minimize this objective, one needs to simultaneously maximize the dot product between the gradients and minimize the norm of ∇ θ L g ( θ 0 , ω ) . Note that − ∇ (cid:62) θ L l ( θ 0 ) ∇ θ L g
[Equation]
θ 0 ,ω ) [Equation] η ∇ θ L g [Equation]
θ 0 ,
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] , and the constant term and constant coefﬁcient can be omitted without changing the optimal solution. We re-write the learning objective of Equation 10 as
[Equation]
which computes the cosine distance between the gradients of student network on automatically labeled generated data and manually labeled data (see Section B in appendix for dealing with gradient matching in multi-layer neural networks) To this end, the nested-loop optimization (Equation 6) is reduced into a simple one that minimizes the gradient matching loss (Equation 11). Illustration of this procedure is presented in Figure 1. Algorithm Note that the gradient matching problem learns an annotator for a particular network parameterized with θ 0 .
However, a desirable annotator should produce universal automatic labels for training any student with previously unseen parameters. Hence, we present an alternate training algorithm of annotator and student in Algorithm 1, where the student is iteratively updated using the automatically labeled synthetic data and the annotator is iteratively updated with gradient matching. This algorithm allows the annotator to be optimized over various student states, and therefore leads to a student-agnostic annotator. The gradient matching problem for each student parameter is solved with K -step gradient descent and typically K = 1 works well in practice.
Implementation details are available in the appendix (see Section B). [Figure]
Our compared to mean Guided Collaborative Training -supervised Learning SemanticGAN al varied of labelled examples
E XPERIMENTS 4. S ETUP
Datasets We benchmark our methods on several part segmentation datasets, CelebA , PascalHorse , and Pascal-Aeroplane .
CelebA
(Liu et al., 2015) is a large scale human face dataset from which 30,000 images are annotated with up to 19 part classes, denoted as CelebAMask-HQ (Lee et al., 2020a).
Following Li et al.
(2021), we only consider a subset of 8 face classes and use the ﬁrst 28,000 images in CelebAMask-HQ as unlabeled set, the last 500 images as test set, and the restimages as labeled set.
We construct Pascal-Horse and Pascal-Aeroplane by taking horse images and aeroplane images from Pascal Part dataset Chen et al.
(2014) which provides images of 20 object categories annotated with parts.
The selected images are further cropped according to bounding box annotations.
Full process details are available in the appendix.
We ﬁnally have 180 images as labeled set, 34 images as validation set, and 225 images as test set in Pascal-Horse, and 180 images as labeled set, 78 images as validation set and 269 images as test set in Pascal-Aeroplane.
In Pascal-Horse and
Pascal-Aeroplane, additional images of horse and airplane from LSUN (Yu et al., 2015) are taken as unlabeled set.
Furthermore, we conduct comprehensive experiments and make thorough comparison to DatasetGAN (Zhang et al., 2021) on Car-20 , Cat-16 , and Face-34 , which are originally released by Zhang et al.
and used to evaluate DatasetGAN.
Car-20, Cat-16, and Face-34 are manually annotated 20, 16, and 34 part classes for car, cat, and human face, respectively.
Car-20 containsimages as training set, 10 images as test set; Cat-16 contains 30 images as training set, 20 images as test set; Face-34 contains 16 images as training set, 20 images as test set. Pre-trained generators We use models of StyleGAN family (Karras et al., 2019; 2020b;a) that are either trained by ourselves or publicly available. Details are provided in the appendix (see Section B).
Evaluation We use mean intersection over union (mIoU) to evaluate the performance of our segmentation network.
In CelebA, Pascal-Horse, and Pascal-Aeroplane, we use validation set to select checkpoints and report mIoU across all foreground classes, denoted as “FG-mIoU”, on test set.
In Car-20, Cat-16 and Face-34, we follow the setting of DatasetGAN (Zhang et al., 2021) and report the cross validation mIoU across all classes including background on test set.
4. S EMI -S UPERVISED P ART S EGMENTATION
Benchmark results As our methods have ﬂexible restrictions on the source of labeled examples, we benchmark our methods on datasets consisting of realistic images with different number of labeled [Table] plus a background class. examples.
Following SemanticGAN
(Li et al., 2021), our methods are compared against SSL methods that have codes available 2 , including Mean Teacher (MT) (Tarvainen & Valpola, 2017; Li et al., 2020), Guided Collaborative Training (GCT) (Ke et al., 2020), an adversarial-learning-based semi-supervised segmentation method (AdvSSL) (Hung et al., 2018), as well as SemanticGAN (Li et al., 2021).
We benchmark under different number of labeled examples and present the results in Figure 2.
Qualitative results are available in the appendix (see Figure D.7, D.8, D.9).
Our methods are shown to signiﬁcantly outperform other methods.
Notably, when the number of labeled examples reduces, the performance of our methods drop slightly while performance of other methods decreases drastically.
We attribute the reason to our utilization of generator features which are highly disentangled.
SemanticGAN (Li et al., 2021) also exploits the generative features to produce segmentation masks but its performance still degrades faster than our methods as the number of labeled examples decreases.
We believe it is due to the adversarial learning employed by SemanticGAN which requires more labels than our methods to achieve fair performance.
Comparison to DatasetGAN We comprehensively compare our methods to a supervised learning counterpart, DatasetGAN (Zhang et al., 2021).
In particular, DatasetGAN and our methods are evaluated on Car-20, Cat-16, and Face-34 with different network structures and quantitative results are presented in Table 1.
Qualitative comparison is available in the appendix (see Figure D.12, D.11,
D.10).
First, our method achieve same performance level compared to DatasetGAN on Face-34 and
Cat-16.
Second, since our method remove the necessity of labeling generated images, we upgrade the generator to the up-to-date StyleGAN2 version without requiring extra human effort.
The results show such update of generator only brings modest improvement on Face-34 and Cat-16 and even a bit decrease on Car-20.
This suggests that synthesis quality measured by metrics like FID can not precisely depict the disentanglement of generator features.
Finally, our methods could adapt to different segmentation architectures, and generally, DeepLab would gives slightly higher performance than U-Net.
We further use our trained annotator to generate a synthetic segmentation dataset from which downstream segmentation networks could be trained.
Table 1 shows the results when different source networks, which are used for training annotator with gradient matching, and different downstream networks are used.
First, our learned synthetic datasets can be used to train different structure of downstream segmentation networks: DeepLab and U-Net.
Results show that downstream segmentation networks can achieve good performance, which indicates that our learned synthetic dataset is not architecture-speciﬁc.
Second, DeepLab as source network generally produce higher performance [Figure] [Figure]
than U-Net as source network.
This suggests that in our method the student network inﬂuences the annotator learning.
Cross-domain annotator learning We further consider a challenge where labeled examples are out-of-domain images. In particular, we render a bunch of car images from annotated 3D CAD models provided by CGPart 3 (Liu et al., 2021) using graphic tools ( e.g . blender). Since 3D models are annotated with parts, the ground truth part segmentation of these rendered images are available via 3D projection. These rendered images vary in viewpoints and textures and have signiﬁcant domain gap compared to realistic car images due to inexact 3D models, unrealistic textures, unnatural lighting etc ., raising challenges for learning annotators.
Despite so, as presented in Figure 3, our method still learns reasonable annotators that produce quality segmentation labels for GAN generated images. It is noteworthy that it is nearly impossible for DatasetGAN (Zhang et al., 2021) and RepurposeGAN (Tritrong et al., 2021) to utilize such out-of-domain labeled data.
The frequency of updating student We conduct an ablation study w.r.t. the interval of updating student network, K , and present the results in Figure 4. We run experiments on Face-34 and Catwith DeepLab as student network and StyleGAN as generator such that pairs of generator features and ground truth segmentation are available to quantitatively measure the performance of annotator.
The results show that annotator signiﬁcantly learns faster when student network is updated more frequently.
From the visualization, it can be seen that very detailed labels, e.g . pupils in Face-34 and [Table] eyes in Cat-16, emerge earlier if student network is updated more frequently.
It can be concluded that it is more effective to match gradients for various student states than optimize gradient matching for single student state. Gradient matching on partial student parameters As the implementation of gradient matching requires maintenance of computation graph of backpropagation through student networks, one may concern the training efﬁciency for very deep neural networks.
Here we present a simple strategy to trade off the training efﬁciency and overall performance: gradient matching can be done only on part of student parameters. An ablation study on Cat-16 using StyleGAN2 as generator and
DeepLab as student network is done and the results are shown in Table 2.
We consider network blocks in DeepLab (DeepLabv3 with ResNet-101 as backbone) from bottom to top: Res1 , Res2 ,
Res3 , Res4 and Res5 that are residual blocks in the backbone, and Head that is the segmentation head.
Results show that by skipping the bottom blocks for gradient matching, the training speed is improved while the segmentation performance is affected little. We conjecture the reasons can be two folds: (i) the backbone parameters are pre-trained such that little needs to be changed during training; (ii) the gradients accumulate more randomness as they backpropagate to the bottom blocks of deep neural networks.
Therefore, in practice, we only match the gradients of Head in DeepLab by default.
Additional results are available in the appendix (see Section C.3).
C ONCLUSION
We propose a gradient-matching-based method to learn annotator which is able to label generated images with part segmentation by decoding the generator features into segmentation masks.
Unlike existing methods that require labeling generated images for training an annotator, our method allows usage of a broader range of labeled data including realistic images, generated images, and even rendered images from 3D CAD models.
On the benchmark of semi-supervised part segmentation, our method signiﬁcantly outperform other semi-supervised segmentation methods, especially under the circumstances of extremely limited labeled data.
With the rapid progress of generative modeling, in the future, we would like to employ more powerful generative models with our method to tackle more varieties of computer vision problems under the low-supervision settings.
R EPRODUCIBILITY S TATEMENT
We provide all the details about the datasets, network structure, and training curriculum in the appendix.
Code reproduce our main results will be made public.
A D ATASET D ETAILS
CelebA Following Li et al.
(2021), we only consider 8 part classes: background, ear, eye, eyebrow, skin, hair, mouth, and nose.
The protocol for merging the 19 part classes into 8 classes is presented in
Table A.1(Left).
CelebAMask-HQ contains 30,000 annotated images in total.
We split this dataset into unlabeled set, training set, validation set and test set as in Table A.1(Right). [Table] Pascal-Horse & Pascal-Aeroplane Pascal Part (Chen et al., 2014) provides part segmentation annotations of 20 object classes for images in Pascal VOC 2010. Following other work on part segmentation (Tsogkas et al., 2015; Zhao et al., 2019; Tritrong et al., 2021), we merge the ﬁne-grained part classes into 6 classes for horse and aeroplane.
The merging protocol is presented in Table A.2.
We crop the image patches that contain the object of interest according to bounding box annotations.
Concretely, bounding boxes that have IoU greater than 0.05 with other boxes or empty annotations are discarded to ensure single object appears in single image patch.
The patches that have any side less a certain number of pixels (32 for horse; 50 for aeroplane) are also abandoned.
The crop regions are extended from bounding boxes to square boxes and the regions outside images are padded with zero.
These processed patches are ﬁnally resized to 256 × 256 resolution to serve the training and evaluation process.
For horse and aeroplane, We further split the patches in ofﬁcial VOCtrain split in training and validation set, and take the patches in ofﬁcial VOC 2010 val split as test set.
This procedure ﬁnally provides 180 images as training set ( i.e . labeled set), 33 images as validation set, and 223 images as test set in Pascal-Horse, and 180 images as labeled set, 78 images as validation set and 266 images as test set in Pascal-Aeroplane.
As unlabeled images, we center-crop the 200,000 images in the large-scale LSUN (Yu et al., 2015) archive and resize them to 256 ×resolution for both horse and aeroplane.
[Table]
CGPart As a part segmentation dataset composed of 3D CAD models from 5 vehicle categories with 3D part manual annotations, CGPart (Liu et al., 2021) manage to render a large amount of image dataset with part mask labels by adopting blender, a well-known graphical software.
In this paper, we select category car as training and evaluation dataset and make moderate modiﬁcations upon the rendering pipeline.
Parts are merged into 15 classes, where excessively ﬁne-grained parts that seldom emerge are combined together to avoid sample imbalance.
Part merging protocol is presented in Table
A.3.
Viewpoints are sampled around the object with azimuth ranged in full 360 ◦ .
Elevation angle is varied from 0 ◦ to 80 ◦ , yet higher sampling probability is assigned to interval [0 ◦ , 30 ◦ ] .
The distance from object to viewpoint is carefully set so that the object is located at center and full in frame.
Random color is painted upon the object and random texture sampled from COCO dataset (Lin et al.) and Pixar-One-Twenty-Eight (Sisson, 2018) is rendered on the ground and the surroundings.
4 ,images of 512 × 384 are rendered as full train dataset.
When training, only 1 , 000 of them in total are randomly sampled as ofﬁcial trainset.
The original val split of CGPart is adopted as test dataset, consisting of 40 images.
Test masks are merged under the same merging protocol in inference and evaluation phases.
[Table]
B I MPLEMENTATION D ETAILS
Gradient matching loss Following DC
(Zhao et al., 2021), the distance between gradients is measured with cosine similarity.
The computation of gradient matching is quite similar as in DC.
Here, we re-state this procedure for clarity.
In particular, considering a multi-layer neural network S θ that is parameterized with θ , the gradient matching loss is computed as an average over layerwise losses as D ( ∇ θ L l , ∇ θ L g )
=
1 L (cid:80) Li =
1 d ( ∇ θ ( i ) L l , ∇ θ
( i ) L g ) , where i denotes the layer index, L denotes the number of layers for gradient matching and
[Equation]
where A j and B j are ﬂattened gradient vectors for each neural node j .
The minor difference of our practice compared DC is that we average the layerwise gradient distances whereas DC sum them.
It is easier to handle numerical issues by using our practice in our problem.
In terms of gradient matching for normalization layers such as Batch Normalization (BN), we follow the practice of DC to ignore the gradient matching of learnable parameters in BN.
The BN layer is set as train mode during gradient matching.
We ﬁnd it works well.
Pre-trained GAN We use StyleGAN family (Karras et al., 2019; 2020b;a) as our pre-trained
GANs.
For each dataset, We either train GANs by ourselves or use publicly available models.
The training conﬁgurations or links of the pre-trained GANs are listed in Table B.4.
For GANs trained by ourselves, we use the checkpoints with lowest FID in the historical ones.
Annotator structure Annotator is a neural network that takes as input the generator features and output segmentation masks.
DatasetGAN (Zhang et al., 2021) and RepurposeGAN (Tritrong et al., 2021)
ﬁrst upsample feature maps to full resolution (resolution of output image) and then concatenate
[Table]
these feature maps along channel.
This procedure results in extremely high-dimensional feature vectors for every pixel.
These features are fed into either a pixelwise multiple layer perceptrons (MLPs) as in DatasetGAN (Zhang et al., 2021) or a convolutional neural networks (CNNs) as in (Tritrong et al., 2021).
Nonetheless, this practice consumes a lot of GPU memory and run painfully slow even on high-end modern GPUs.
That is also why DatasetGAN cannot consume all image feature vectors in a batch during training.
Therefore, to make training annotator more efﬁcient, as commonly done in segmentation or detection network architectures, we fuse the multi-scale feature maps with a feature pyramid network (FPN) (Lin et al., 2017) structure and decode the fused features into segmentation masks with consecutive convolutional layers.
This FPN structure saves a lot of GPU memories and computation, which allows us to consume multiple full images in one mini-batch during training.
An illustration of our annotator structure is presented in Figure B.1.
For generator features that are fed into annotator, we use outputs of all convolutional layers in StyleGAN as in DatasetGAN (Zhang et al., 2021) and use outputs of all synthesis blocks, each of which typically contains two consecutive convolutional layers, in StyleGAN2.
[Figure]
Segmentation network structures In our experiments, we employ two prevalent segmentation network structures: U-Net Ronneberger et al.
and DeepLabv3 (Chen et al., 2017) with ResNet101 pretrained on ImageNet as backbone.
For U-Net, we reference the implementation from and train it from scratch on target datasets.
For DeepLabv3, we use the the built-in implementation in
PyTorch library 4 .
Training details of our method We use pixelwise cross entropy loss as f ( · , · ) for training student network and computing gradients.
Both the annotator and the student are optimized with SGD optimizer with learning rate 0 .
001 and momentum 0 .
By default, we jointly train an annotator and a student with K = 1 and batch size 2 for 150 , 000 steps.
Training details of competing semi-supervised methods For MT, GCT and AdvSSL, we train the networks for 10 , 000 iterations with batch size 16 for labeled data and 8 for unlabeled data.
Learning rate is decayed with power 0 .
9 every iteration.
For the rest of the hyper-parameters, we use the default settings as in PixelSSL.
C F URTHER A NALYSIS C. C OMPARISON TO I NVERSION B ASELINE
An alternative way to make the supervised learning of annotator be able to utilize existing labeled images is to (i) project images into GAN latent space to acquire generator features, (ii) use the joint generator features and ground truth labels to train an annotator.
We call this method “inversion baseline”.
We evaluate the performance of this method on Cat-16, Face-34, and Car-20 using
StyleGAN2 as generator and make a comparison to our method in Table C.5.
Concretely, we ﬁrst use the projection method provided by Karras et al.
(2020b) to compute the latent style codes that result in a reconstruction of labeled images.
Second, the latent style codes are forwarded to the generator to acquire generator features.
We present examples of such reconstruction in Figure C.2 which shows that the reconstruction is quite accurate.
However, the performance of inversion baseline fails to combat our method and the automatic labels for a randomly generated image are of low quality (see
Figure C.3).
We argue that it is non-trivial to inverse the GAN generation process to acquire features for speciﬁc images.
Even though the reconstruction of images is satisfying, one is still unable to exactly recover the generator features, which leads to an degraded learning of annotator.
[Table]
[Figure]
C. T HE I MPACT OF S TUDENT S TATES
As we present an alternate learning algorithm between annotator and student, we study the impact of updating student states.
In particular, we investigate the training convergence and overall performance of annotator when different series of student states are employed for gradient matching.
We present in Figure C.4 & Figure C.5 the evolution of annotator performance evaluated on training set during training process, where “baseline” denotes the default setting and the following settings are further compared.
(i)
The student parameters are randomly re-initialized rather than learned from generated images, denoted as “random re-init”.
(ii)
The student is pretrained to acquire a fair performance ( ∼ 40 .
7% mIoU evaluated on test set), denoted as “pre-trained”.
(iii)
The student is a DeepLabv network that has backbone randomly initialized rather than pretrained on ImageNet, denoted as “scratch”.
(iv) The student parameters, either randomly initialized or pre-trained, are ﬁxed, denoted as “random init & ﬁxed” and “pre-trained & ﬁxed”, respectively.
First, “Fixed” students result in a gradient matching problem on single student state, which learns annotator with very poor performance compared to the default setting.
This result suggests that matching gradients for a series of student states is vital to learning annotator.
Second, even though it is crucial to require annotator to produce student-agnostic labels, a randomly re-initialization strategy does not work well (see results of “random re-init” v.s. “baseline”).
Moreover, the results of “pre-trained & ﬁxed” is better than “random init & ﬁxed”, which suggest that a well-performed student provides more informative gradients such that by gradient matching the annotator can be more effectively learned.
Third, the conclusion that a well-performed student provides more informative gradients can be further veriﬁed by the fact that matching gradients of pre-trained students accelerates the learning of annotator (see results of “baseline” v.s. “pre-trained” and “baseline” v.s. “scratch”).
C. T RADE OFF T RAINING E FFICIENCY WITH P ERFORMANCE
Gradient matching on partial network parameters We present additional results on in Table C.6.
U-Net takes as input an image and ﬁrst produce feature map at multiple downsampled scales with consecutive convolutional and pooling layers.
These feature maps are then processed by consecutive convolutional and upsampling layers to be mapped back to original input scale.
Notably, skip connection is employed to connect the multi-scale feature maps along the downsampling path and upsampling path.
We consider the following particular group of layers in U-Net: (i) Up v.s.
Down , (ii) Input v.s. Output , and (iii) U1 v.s. U2 .
Up and Down denote the convolutional layers along the upsampling and downsampling path, respectively.
Input and Output denote the input convolutional layer and the output convolutional layer, respectively.
U1 and U2 denote the convolutional layers that process feature maps at 1 / 1 and 1 / 2 scales, respectively.
From the results
[Figure]
[Figure]
we can see that matching gradients of layers near the output end is more effective than doing so at layers near the input end.
[Figure]
[Table]
Resolution of input to student network Another way to improve the training efﬁciency is to reduce the resolution of input to student network.
Note that the resize operation can be implemented with interpolation which is differentiable.
We show in Figure C.6 and Table C.7 that by reducing the resolution, the training speed improves signiﬁcantly, but the annotator performance and downstream segmentation networks compromises.
Therefore, in practice, the student resolution can be tuned by trading off the training efﬁciency and effectiveness.
[Table]
Training speed is measured on a single RTX 3090.
D Q UALITATIVE R ESULTS
[Figure]
I npu t
G T
[Figure]
f
[Figure]
a b e l e d exa
[Figure]
p l es
[Figure]
[Figure]
I npu t
G T
N u m b e r o f l a b e l e d exa m p l es
I npu t
G T
{+N u m b e r o f l a b e l e d exa m p l es    
+}
[Figure]
I npu t
G T
D a t ase t
G AN
O u r s
D a t ase t G
AN
O u r s
[Figure]
I npu t
G T
D a t ase t
G AN
O u r s
D a t ase t G
AN
O u r s
[Figure]