F INDING AN U NSUPERVISED I MAGE S EGMENTER IN E ACH OF Y OUR D EEP G ENERATIVE M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent research has shown that numerous human-interpretable directions exist in the latent space of GANs.
In this paper, we develop an automatic procedure for ﬁnding directions that lead to foreground-background image separation, and we use these directions to train an image segmentation model without human supervision.
Our method is generator-agnostic, producing strong segmentation results with a wide range of different GAN architectures.
Furthermore, by leveraging
GANs pretrained on large datasets such as ImageNet, we are able to segment images from a range of domains without further training or ﬁnetuning.
Evaluating our method on image segmentation benchmarks, we compare favorably to prior work while using neither human supervision nor access to the training data.
Broadly, our results demonstrate that automatically extracting foregroundbackground structure from pretrained deep generative models can serve as a remarkably effective substitute for human supervision
I NTRODUCTION
Recent years have seen rapid progress in the ﬁeld of deep generative modeling of images, driven by a proliferation of research into Generative Adversaial Networks (GANs) (Goodfellow et al., 2014).
Nowadays, it is possible to generate high-resolution images of realistic objects and scenes (Karras et al., 2019; Brock et al., 2019).
However, with the exception of generation for computer graphics, there has been limited research into how we might be able to leverage the representations learned by these powerful generative models to enhance other tasks, particularly those involving semantic understanding (e.g. classiﬁcation or segmentation).
Generally, deep generative models learn to map latent codes to images, imposing simple statistical structures on the distribution of the latent codes, such as assuming an i.i.d.
Gaussian distribution.
Due to this structure, in some cases code dimensions acquire speciﬁc meanings which can be related to human-interpretable concepts (e.g., the rotation or size of an object); however, the code space in high-quality generators (e.g., BigGAN (Brock et al., 2019), BigBiGANs (Donahue & Simonyan, 2019), StyleGAN (Karras et al., 2019)) is usually not easily interpretable.
Nonetheless, it is intuitive that an efﬁcient generative process should account for the structure of natural images, including for example the fact that images often comprise distinct foreground and background regions.
In this paper, we validate this hypothesis by learning to separate foreground and background image regions from generator networks.
Our approach starts from an arbitrary, off-the-shelf high-quality generator network trained on a large corpus of (unlabeled) images.
While these generator networks are not explicitly trained for foreground/background segmentation, we show that such a separation emerges implicitly as a step to efﬁciently encode realistically-looking images.
Speciﬁcally, we design a probing scheme that can extract such foreground/background information automatically , i.e. without manual supervision, from the generated images.
This scheme works as follows (cf.
Fig. 1).
We start from a random code in latent space and learn a ﬁxed, global offset that results in a change in the generated images.
The offset is learned to alter the appearance of foreground and background such that a mask can be extracted from the changes in image space.
The resulting masks provide segmentation maps for the generated images, but they cannot yet be used to segment images from the real world.
Given a natural image, the obvious approach would
G e n e r a t e d i m
a g e
s a n d e x t r a c t e d m a s k s
R e a l - w o r l d
i m a g e s a n d p r e d
i c t e d m a s k s
G e n e r a t e d i m
a g e
s a n d e x t r a c t e d m a s k s
R e a l - w o r l d
i m a g e s a n d p r e d
i c t e d m a s k s be to ﬁnd the corresponding code in the latent space of the generator, and then obtain a mask with our method.
Unfortunately, this inversion process is less than trivial.
In fact, recent work provides strong evidence that the expressiveness of GANs is insufﬁcient to encode arbitrary images (Bau et al., 2019), meaning that the inversion problem has no solution in general.
As we aim to build a general-purpose segmentation method, we take a different approach: we generate a labelled image dataset with foreground/background segmentations and use the generated dataset to train a standard segmentation network.
With this, we show that our method can successfully learn accurate foreground-background segmentation networks with no manually provided labels at all.
Differently from prior work in GAN-based image segmentation, we neither design a new GAN architecture speciﬁcally for the task of segmentation nor use manual supervision to extract segmentation information from an existing GAN.
Thus, we can discover meaningful latent directions for any GAN with no need for model-speciﬁc manual intervention.
Extensive experiments on ﬁve segmentation datasets across twelve different GANs demonstrate the effectiveness and generalizability of our approach.
Moreover, by constructing our image segmenters from generator networks trained on a generic large-scale datasets such as ImageNet, our method can learn to generically segment objects from a wide range of visual domains.
Speciﬁcally, when we apply our generic segmenters to the CUB200 (Welinder et al., 2010) and Oxford Flowers (Nilsback, 2009) datasets, we attain very strong foreground-background results despite not training on this data.
Similarly, when we apply our generic segmenters to three saliency detection benchmarks, our method approaches and sometimes even exceeds the performance of supervised and handcrafted saliency detection methods.
An analysis of our results also shows that segmentation performance directly correlates with the quality of the underlying GAN, suggesting that foreground/background separation is an important concept in learning generative models.
Finally, we demonstrate that our method may be used as a drop-in replacement for saliency networks for the purpose of learning pixel-wise semantic image representations.
These pixel-wise representations can then be clustered to obtain fully unsupervised semantic segmentations, extending our method beyond foreground-background segmentation.
Thus, we not only demonstrate for the ﬁrst time that it is possible to obtain semantic segmentations using GANs, but also that this information may be extracted from a wide range of generic GANs trained on general-purpose image datasets.
R ELATED WORK
Below, we describe how our method relates to recent work in interpreting generative models and object segmentation.
Due to space constraints, we include additional related work in the Appendix.
Interpreting Deep Generative Models.
Several works have proposed methods for decomposing the latent space of a generative model into interpretable or disentangled directions.
Early work included Beta-VAE (Higgins et al., 2017), which modiﬁed the variational ELBO in the original VAE formulation, and InfoGAN (Chen et al., 2016), which maximized the mutual information between a subset of the latent code and the generated data.
Later work has sought to disentangle factors of variation by mixing latent codes (Hu et al., 2018), adding additional adversarial losses (Mathieu et al., 2016), and using contrastive learning (Ren et al., 2021).
Our work follows a recent line of research that looks for structure in large, pretrained generative models.
Shen & Zhou (2020) perform a direct decomposition of model weights to ﬁnd disentangled directions, while Peebles et al.
(2020) penalize nonzero second-order interactions between different latent dimensions, and Voynov & Babenko (2020) ﬁnd interpretable directions by introducing an additional reconstruction network.
Differently from the works above, we conduct a deep study of one speciﬁc type of structure (foreground/background separation) encoded in the latent space.
Other works have taken this approach in the context of extracting 3D structure from 2D images; for example, IG-GAN (Lunz et al., 2020) uses a neural renderer to recover 3D (voxel-based) representations of scenes, and GAN2Shape (Pan et al., 2021) exploits viewpoint and lighting variations in generated images to recover 3D shapes.
Unsupervised Object Segmentation.
Prior work on unsupervised object segmentation can be divided into two categories: those that employ generative models to obtain segmentation masks and those that employ purely discriminative methods such as contrastive learning (Ji et al., 2019a; Ouali et al., 2020).
Here, we focus on generative approaches.
Nearly all generative approaches are based on the idea of decomposing the generative process in a layer-wise fashion; in general, the foreground and background of an image are generated separately and then combined to obtain a ﬁnal image.
Speciﬁcally, ReDo (Chen et al., 2019) trains a generator to re-draw new objects on top of old objects, and enforces realism through adversarial training.
(Bielski & Favaro, 2019) generates a background, a foreground, and a foreground mask separately and composite them together; they prevent degenerate outputs (i.e. the foreground and background being the same) by randomly shifting the foreground relative to the background.
CopyPaste GAN (Arandjelovi´c & Zisserman, 2019) receives two images as input and copies parts of one image onto the other.
OneGAN (Benny & Wolf, 2020) learns to simultaneously generate, cluster, and segment images with a combination of GANs, VAEs, and additional encoders.
Equivariant Layered GAN (Yang et al., 2021)
ﬁrst trains a new layerwise GAN and then trains a segmentation network on synthetic data.
Labels4Free (Abdal et al., 2021) proposes a new layerwise network inspired by StyleGAN for foreground-background segmenation.
Common difﬁculties with these layer-wise approaches above include the challenges of training new GANs and scaling beyond simple datasets (e.g. CUB, Flowers).
One work along different lines is by Voynov et al.
(2021), which uses a pretrained BigBiGAN (Donahue & Simonyan, 2019) generator rather than proposing a new layer-wise GAN.
Voynov et al.
(2021) use the method from Voynov & Babenko (2020) to decompose the latent space into interpretable directions. A recent version of their method also picks out this direction without supervision using the idea that a foreground-background separating direction should be decomposable into two afﬁne operators acting on different sets of pixels.
Our approach is based on generative modeling, but it differs from other approaches in that we seek a general-purpose method to ﬁnd foreground/background structure implicitly encoded in standard, non-layer-wise GANs rather than encoding it explicitly or searching for it manually.
This enables us to leverage any of the numerous existing generators that have already been pretrained on millions of high-resolution images, rather than developing a new GAN architecture for this speciﬁc task.
Differently from layer-wise approaches, our approach does not require training new GANs and it does not rely on the assumption that the foreground and background of an image are independent.
Differently from Voynov et al.
(2021), we show through experiments that our method can be applied to a wide range of different GANs and ﬁnd that it delivers superior performance on four out of ﬁve object segmentation and saliency detection benchmarks.
Finally, unlike any of these previous works, we demonstrate that our foreground-background segmentation network can be used as a drop-in replacement for saliency networks for the task of learning dense semantic image representations.
By clustering these representations, we are able to extend our method from foregound-background segmentation to semantic segmentation.
M ETHOD
Let x ∈ R 3 × H × W be a (color) image.
A generator (network) is a function G : R D
→ R 3 × H × W that maps code variables z to images x = G ( z ) .
Optionally, some generative models come with an encoder function E : R 3 × H × W
→ R D which computes an approximate inverse of the generator (i.e. G ( E ( x ))
≈ x ).
A challenge in generating images is that individual pixels exhibit complex correlations, caused by the fact that the images are obtained as the composition of a number of different objects.
For example,
[Figure]
Sobel [Figure] G Step 1: Finding informative code variations z
G z + v l ℒ consistency ℒ segmentation
Perturbation in direction v
G z
G z + v l G z + v d
S ℒ cross-entropy
Prediction
Synthetic Ground Truth
Brightnessmask
Step 2: Learning a segmentation model
Perturbation in direction v d
Perturbation in direction v l all pixels that belong to a dog have a similar color, characteristic of dog’s instance.
However, the correlation is much less strong between pixels that belong to different objects.
This is because, while object in a scene are not entirely independent, their correlation is much weaker than within the structure of objects.
Intuitively, an image generator must learn to account for such correlations in order to generate realistically-looking images.
In particular, we expect the generator to capture the idea that pixels that belong to the same object have a related appearance, whereas the appearance of pixels that belong to different objects or, as it may be, to a foreground object and its background, should be much more statistically independent.
Given a generator function G , it is then natural to ask whether such correlations can be extracted and used not just for the purpose of generating images, but also for analyzing them.
In order to explore this idea, we consider perturbing the code z via a small increment (cid:15)v ∈ R D , where (cid:15) ∈ R and v ∈ S D − 1 is a unit vector.
Because the dimension D of the embedding space is typically much smaller than the dimension 3 HW of the generated images, codes provide highly-compressed views of the data (for example, D = 120 for BigBiGAN (Donahue & Simonyan, 2019) and the self-conditioned GAN).
As such, most changes in the code are likely to affect most if not all pixels in the image.
However, if the generator did in fact learn to compose objects, then one could hope too ﬁnd speciﬁc variations v that only affect only portions of the image, corresponding to individual objects, and use the latter to highlight and segment them.
Empirically, we ﬁnd that the situation is not as simple.
Speciﬁcally, it is not easy to ﬁnd changes in the code that leave part of the pixels exactly constant while changing other pixels.
However, we ﬁnd that there are directions that affect foreground and background regions in a systematic and characteristic manner.
Furthermore, we show that these directions are ‘universal’, in the sense that the same v works for all codes z , and are thus characteristic of a given generator network G .
3. F INDING INFORMATIVE CODE VARIATIONS
Next, we introduce an automated criterion to select informative changes v in code space.
To this end, we consider an image x = G ( z ) generated from a random code z ∼ Z , where Z is the code distribution characteristic of the generator (e.g. an i.i.d.
Gaussian).
We then consider a modiﬁed image x (cid:48)
=
G ( z + (cid:15)v ) and observe the change x
→ x (cid:48) .
We compare the two images using two criteria.
The ﬁrst one preserves the structure of the image x .
We capture the latter by imposing that x and x (cid:48) generate approximately the same edges when fed to a simple edge detector.
The intuition is that we wish v to affect the appearance of objects without changing their shape.
By preventing objects from ‘moving around’ the image or deforming, we make it signiﬁcantly easier to extract an image segmentation from the change x → x (cid:48) .
This loss takes the form:
[Equation] [Table]
[Table]
where z i ∼ Z and S is the Sobel-Feldman operator:
[Equation]
and g = [1 2 1] · [1 0 − 1] (cid:62) .
This loss encourages x and x (cid:48) to be similar.
We thus also need a loss that encourages the direction v to explore a non-zero change of the image.
We consider an image contrast variation and additionally exploit the photographer bias, that objects are often placed in the middle of the image.
This is captured by the loss:
[Equation]
where r ∈ R H
× W is a ‘radial’ prior:
[Equation]
with normalization factor α = 14 (cid:112) ( H − 1)
+ ( W − 1) 2 that linearly interpolates from 1 in the center of the image to − 1 at the boundary.
This encourages ﬁnding a direction v that changes the brightness in the center of the image opposite to the border.
In order to learn v , the two losses are combined with a weighting factor λ .
[Equation]
Given this fully-automatic procedure, the latent code direction v may be thought of as a function of the generator G and the weighting factor λ .
3.1. C OMBINING INFORMATIVE CODES
Optimizing Eq.
(1) with λ > 0 encourages the network to produce a shift v that brightens the foreground and darkens the background of an image.
However, there is no constraint that λ need be positive; by negating λ and optimizing a second time, we obtain another direction v that shifts the foreground dark and the background light.
Although using only one direction sufﬁces for our method, we ﬁnd that we can improve performance by using both.
As a result, for the remainder of the paper, let v l represent the direction that shifts the foreground lighter, and v d represent the direction that shifts the foreground darker.
3. L EARNING A SEGMENTATION MODEL
Once the latent directions v d and v l have been found, the process of extracting a segmentation mask is straightforward: we label as foreground regions the pixels in which the image generated with the foreground-lighter shifted latent code is lighter than the image generated with the foreground-darker shifted latent code.
That is, for a generated image x = G ( z ) , we have:
[Equation]
Alternatively, if we use only a single direction v l or v d , M ( z ) is set to either:
[Equation]
Given the learned direction v , we use it to generate a training set as follows:
[Equation]
This dataset may then be used to train any dense segmentation network Ψ (i.e., a UNet (Ronneberger et al., 2015)) in the standard fashion.
That is, we minimize the pixel-wise binary cross-entropy loss between the segmentation output Ψ( G ( z ))
∈
R H × W and the (synthesized) mask M ( z ) :
[Equation]
where p ( m | s )
=
mσ ( s )+(1 − m ) σ ( − s )
, u is a pixel index and sign is the sign function.
Unlike previous object segmentation methods, our method requires no additional losses or constraints to ensure the stability of training.
By sampling z , we can generate an ‘inﬁnite’ dataset for learning the network Ψ .
Although we described the procedure above for unconditional GANs, our method applies just as well for weakly-supervised conditional GANs, where the generator G ( z, y ) also depends on a class label; we simply sample a label y uniformly at random for each generated image.
3.2. R EFINING THE GENERATED DATASET
An advantage of training with GAN-generated data is that the dataset size is inﬁnite, which means that one is free to curate one’s dataset and discard uninformative training examples.
In our case, we found that it was helpful to reﬁne the dataset by (1) discarding images with masks that were too large, (2) discarding images for which the latent code shift did not produce a signiﬁcant change in brightness, and (3) removing small connected components from the mask.
The exact details are given in the Supplementary Material.
E XPERIMENTS
In this section, we present an extensive set of experiments demonstrating the method’s performance, its wide applicability across image datasets, and its generalizability across GAN architectures.
4. E XPERIMENTAL S ETUP
As our method is generator-agnostic, we apply our method to twelve generators, including three unconditional and nine conditional GANs.
For the three unconditional GANs (BigBiGAN (Donahue & Simonyan, 2019), SelfCondGAN (Liu et al., 2020), and UncondGAN (Liu et al., 2020)), our procedure is completely unsupervised.
For conditional GANs, our method is still unsupervised but the GAN naturally relies on class supervision for training.
To demonstrate the efﬁcacy of our method across resolutions and datasets, we implement both,
GANs trained on ImageNet (Deng et al., 2009) at a resolution of 128px, and GANs trained on the smaller TinyImageNet dataset (100,000 images split into 200 classes) at a resolution of 64px.
All experiments performed across all GANs utilize the same set of hyperparameters for both optimization and segmentation.
This is a key advantage of our method relative to other unsupervised/weaklysupervised image segmentation methods (Chen et al., 2019; Bielski & Favaro, 2019; Benny & Wolf, 2020; Arandjelovi´c & Zisserman, 2019), which are sensitive to dataset-speciﬁc hyperparameters. 4. E VALUATION DATA
We evaluate the performance of our model on three standard saliency detection benchmarks (DUTOmrom (Yang et al., 2013), DUTS (Wang et al., 2017), ECSSD (Shi et al., 2016)) and two standard
[Table] [Table]
refers to TinyImageNet at resolution 64px
[Table]
object segmentation benchmarks (CUB (Welinder et al., 2010), and Flowers-102 (Nilsback & Zisserman, 2009)).
For the saliency datasets, we evaluate using (pixel-wise) accuracy, mean intersectionover-union (IoU), and F β -score with β 2 = 0 .
For the object segmentation datasets, we evaluate using accuracy, IoU, and max F β (the maximum F β score over a range of 255 uniformly distributed binarization thresholds between 0 and 1 ).
4. R ESULTS
Performance on Benchmarks.
In Table 1, we compare our method to other recent work.
We emphasize that our method uses the same model for all datasets and has not seen any of the (training or evaluation) data for these datasets before.
Our method delivers strong performance across datasets, approaching and even outperforming some supervised/handcrafted saliency detection methods.
In comparison to Voynov et al. (2021), we perform better on four out of ﬁve benchmarks (all except
CUB), even though we do not rely on humans to hand-pick latent directions.
In comparison to layerwise GANs (Chen et al., 2019; Bielski & Favaro, 2019; Benny & Wolf, 2020; Arandjelovi´c & Zisserman, 2019), we perform similarly on CUB and Flowers-102, but we cannot compare our method with layerwise GANs on complex datasets (e.g. DUTS) because they do not produce any meaningful results.
Due to the difﬁculty of training GANs, they are only ever trained on datasets consisting of images from a single domain with a single main subject, such as birds or ﬂowers.
By contrast, our ability to leverage pre-trained generators means that our method scales to complex and diverse datasets, such as those used for saliency detection.
Performance across Generators.
We investigate the generality of our method by performing the same optimization and training pipeline with twelve different GANs.
For each generator, we optimize to obtain a latent direction v l , train a segmentation model using this direction, and evaluate its performance across the ﬁve datasets above.
The same hyperparameters are kept constant for all GANs, including λ = 5 .
during the optimization phase.
Results are shown in Table 2; BigBiGAN performs best, but all networks, even those using relatively weak TinyImageNet-trained GANs (e.g., GGAN (Lim & Ye, 2017)), deliver reasonable segmentation performance.
This highlights the beneﬁts of our fully-automatic segmentation pipeline; our method performs well across a wide range of generators trained on different datasets at different resolutions.
Naturally, the quality of a ﬁnal segmentation network produced by our method is related to the quality of the underlying generator.
Figure 4 plots the Frechet Inception Distance (FID) score of nine conditional GANs versus the average accuracy of the corresponding segmentation networks produced by our method.
Lower FID scores, which correspond to better GANs, correlate with improved accuracy.
This correlation suggests that as better GANs architectures are developed, our method will continue to produce better unsupervised segmentation networks.
Ablation: Comparing latent directions.
We compare the performance of a segmentation masks using the two latent directions v d and v l together (Eq.
(2)), or each of them individually (Eq.
(3)) visually in Fig. 3a.
In Table 6, we quantitatively compare the results of these three methods along with a fourth method in which we ensemble the ﬁnal segmentation networks produced by v d and v l individually.
The foreground-lighter ( v l ) and foreground-darker ( v d ) directions yield similar results when used individually.
The combination ( v l and v d ) provides superior results, on par with the ensemble.
Unlike the ensemble, which requires training two networks, the combination of v d and v l adds minimal overhead compared to training with one direction.
Ablation: Varying λ , (cid:15) , the Central Prior, Random Initializations, Amount of Generated Data.
The two hyperparameters in the optimization stage of our method are λ , which controls the trade-off between brightness and consistency, and (cid:15) , which controls the magnitude of the perturbation.
We ﬁnd that the process is only modestly sensitive to changes in these hyperparameters.
We also ﬁnd that using the central prior compared to a spatially-agnostic loss term is better, but only moderately.
Detailed numerical results, including the ablation on the central prior, are given in Section A.3.
In Appendix B.3, we also investigate how segmentation performance varies with the number of generated images, ﬁnding a clear log-linear relationship.
Qualitative Results.
By inspection, we ﬁnd that our optimization procedure is able to edit images in such a manner that the foreground becomes lighter and the background becomes darker.
In numerous cases, the network appears to convert the scene from daytime to nighttime.
Furthermore, better GANs generally produce qualitativley better segmentations.
Please refer to Appendix A and
Section A.1.3 for illustrations.
4. E XTENSION T O S EMANTIC S EGMENTATION
Finally, we demonstrate that our network may be extended from binary foreground-background segmentation to semantic segmentation, the task of assigning each pixel in an image into one of K
[Table]
[Table] semantic categories.
Due to the challenging nature of this task, it has not been attempted by any previous works in the GAN-based segmentation space
[Table]
We extend our method by following the dense contrastive learning approach proposed by Van Gansbeke et al.
In this approach, binary masks are extracted from a set of images using a foreground-background segmentation model, sometimes called a “mid-level visual prior.”
Then, a network is trained to generate pixel-wise features using a mask-based contrastive loss: features corresponding to pixels in the foreground of the image are pulled toward the features of other pixels in the mask and pushed away from the features of background pixels.
After training, semantic segmentations may be extracted by clustering these pixel-wise features across an entire dataset.
Importantly, Van Gansbeke et al. (2021) uses saliency detection networks to generate their object masks.
Although these saliency detectors are sometimes called “unsupervised,” they are actually initialized using pretrained semantic segmentation networks 1 .
We propose to use our object segmentation network as a drop-in replacement for these saliency networks, making the entire process entirely unsupervised.
We leave the rest of their method (i.e. the dense contrastive learning and the evaluation procedure) unchanged.
We perform experiments on the PASCAL VOC dataset, which contains 20 semantic classes.
Experimental details are included in Section A.1.3 and K-Means clustering results are shown in Table 3.
We compare to a range of baselines, along with two models from Van Gansbeke et al.
(2021) using different levels of supervised pretraining.
Our network is competitive with Van Gansbeke et al.
(2021) despite being entirely unsupervised.
Furthermore, in Table 4 in Section A.1.3 we show that this procedure works for a wide range of GANs, demonstrating the generalizability of our approach.
C ONCLUSIONS
We ﬁnd that extracting a salient object segmentation from the latent space of a GAN is not only possible without supervision but also leads to state-of-the-art unsupervised segmentation performance on several benchmark datasets.
In contrast to existing methods that have been engineered speciﬁcally for this task, we extract segmentations from a network trained for a very different purpose — generating images.
Surprisingly, we are able to generalize to a wide range of segmentation benchmarks without directly training on any real images, and even extend our results to semantic segmentation.
Our ﬁndings directly prompt future research questions about what other concepts of the physical world can be automatically extracted from generative models, and to what extent we can use such extracted concepts to replace human supervision in other computer vision tasks.
R EPRODUCIBILITY S TATEMENT
We aim to ensure that our experiments are entirely and easily reproducible.
We upload code to the
Supplementary Material to fully reproduce all experiments.
This code contains a README ﬁle with a detailed description of the code structure, which should help enable others to reproduce and later extend upon our work.
We also take care to describe all hyperparameters and implementation details in the Appendix.
Our results do not require extremely large amounts of compute; they can be reproduced with a single GPU by researchers with computational constraints.
E THICS S TATEMENT
It is important to discuss the potential ethical issues involved with training large-scale generative models and segmentation networks along the lines proposed by our paper.
First of all, the task of segmentation is predicated upon classifying objects into predetermined (either binary or semantic) categories; the deﬁnition of these categories, especially in the case of semantic segmentation, may introduce biases into the task itself.
Second, when training models on largescale datasets, it is essential to consider the biases and data privacy issues introduced in the data collection process.
For example, the PASCAL-VOC dataset, which we use for the task of semantic segmentation, is composed of images scraped from Flickr.
As a result, it is composed primarily of photographs from the United States and Europe, and the “person” class contains primarily images of white individuals.
Additionally, it is not clear whether the individuals in these photographs consent to being used to train image segmentation models, bringing up the issue of data privacy.
From an ethical perspective, our method is slightly different from standard segmentation models because it is trained solely on GAN-generated images; this is not to say that it is ethically better or worse, but that it involves different ethical considerations.
On the one hand, this might alleviate some data privacy concerns, as the segmentation training data is synthetic.
However, since this training data is generated by a GAN, one has to examine the data and methodology originally used to pretrain the GAN; any biases present in this data will likely be reproduced or ampliﬁed by the
GAN .
For example, if one uses a GAN trained on ImageNet to perform object segmentation, it may perform better on white individuals than individuals of other races due to the disproportionate percentage of white individuals in the training data (Steed & Caliskan, 2021).
Investigating biases introduced by GANs remains an active area of research in the machine learning community (Jain et al., 2020; Tan et al., 2020), and these ethical discussions extend to our GAN-based segmentation method.
A A PPENDIX A. I MPLEMENTATION D ETAILS A.1. O PTIMIZATION
First, we optimize for the latent code vectors v d and v l .
We generate latent codes z ∼ N (0 , 1) and optimize the vector v l (or v d ) by gradient descent with the Adam (Kingma & Ba, 2014) optimizer and learning rate 0 .
We use λ = 5 for the light direction v l and λ = − 5 for the dark direction v b .
We perform 1000 optimization steps, by which point v l (or v d ) has converged.
A.1. O BJECT S EGMENTATION
To generate data, we sample z ∼ N (0 , 1) , produce the mask m =
M ( z ) , and reﬁne the mask as described in the main paper.
We use a simple UNet (Ronneberger et al., 2015) with bilinear down/upsampling as our segmentation network.
Following (Voynov et al., 2021), we train for 12000 steps using Adam with learning rate 10 − 3 and batch size 95 , decaying the learning rate by a factor of 0 .at iteration 8000 .
A.1. S EMANTIC S EGMENTATION
We follow the procedure from (Van Gansbeke et al., 2021), but we use binary masks produced by our object segmentation model as a drop-in replacement for their saliency detection masks.
Under this procedure, we begin by applying our segmentation network to extract masks from the trainaug split of PASCAL-VOC (10582 images).
We then train a dilated ResNet-50 model with DeepLabv3 head, initialized with a pretrained MoCo ResNet-50.
As in (Van Gansbeke et al., 2021), this network has a main head and an auxiliary head: the main head predicts dense pixel-wise features of dimension 32 , and the auxiliary head predicts the object segmentation mask for the purpose of regularization.
We train the output of the main head using a pixel-wise version of the InfoNCE loss.
Speciﬁcally, for a feature z i corresponding to a pixel i in the foreground mask M i , we have:
[Equation]
where z + is the average embedding of other pixels in the mask M i , and z − is the average embeding of pixels in the background:
[Equation]
where N is the number of pixels in the image.
We use the same hyperparameters as (Van Gansbeke et al., 2021) for the contrastive learning procedure.
We train for 40 epochs using random crops of size 224, learning rate 0.01, and output dimension 32.
After training, we evaluate with K-Means clustering with 21 clusters (20 semantic categories and 1 background category).
These clusters are computed globally over the entire validation set and matched to the ground truth labels using the Hungarian algorithm.
We repeat this same procedure identically for a large range of GANs.
Figure 9 compares the result of ﬁnal segmentation networks for different GANs: all images produce sensible segmentations, with better GANs generally producing better segmentations.
A. D ATASETS
Below, we give statistics of our evaluation datasets.
Note that these are only used for evaluation, as only GAN-generated images are seen during training:
[Table] [Table]
[Table]
A. A DDITIONAL A BLATIONS
Ablation: Comparing latent directions.
In addition to comparing the networks resulting from v l and v b , we also compare the actual latent directions v l and v b .
Due to the nonlinearity of the generator function, the optimal unit directions v l and v d are not necessarily negations of one another; indeed, we found in practice that they were close to but not exactly antiparallel.
Table 9 in
Section A.3 gives exact numbers for a variety of different generator architectures.
Ablation over λ .
The most notable hyperparameter in the optimization stage of our method is λ , which controls the trade-off between brightness and consistency.
Table 7 compares segmentation results for BigBiGAN with λ = 10 , 5 , 2 . 5 , and 1 . 25 , with 5 performing best.
Ablation over (cid:15) .
In Table 8, we show ablation results for changing (cid:15) during the optimization process.
Note that since the GAN used in this set of experiments (BigBiGAN) has a 120-dimensional latent space, the distribution of the norm of the N (0 , 1) latent vectors used to train the GAN is concentrated around (approximately) 11.
That is to say, a shift of magnitude (cid:15)
= 6 in the latent space is very large.
Ablation: Random initialization.
Given the importance of the latent direction optimization procedure in our segmentation pipeline, it is natural to ask whether different initializations yield materially different directions in the latent space.
To answer this question, we optimized ten vectors v on
[Table] [Table]
BigBiGAN from different initializations and measured their pairwise cosine similarities.
We found that the cosine similarities had mean 0 .
9866 with standard deviation 0 .
0044 , which is extremely small relative to the values in Table 9.
Thus, the random initialization of the optimization procedure does not have a large impact on the ﬁnal direction.
Ablation:
On the necessity of the center prior.
Here, we conduct a small experiment to investigate the necessity of the central prior in the optimization step of our segmentation pipeline.
We replace it with a spatially-agnostic loss term with no central prior: it simply encourages the shifted image to have high variance in brightness.
We ﬁnd that this spatially-agnostic loss still yields good results, although not quite as good as the spatial prior loss.
Numerical results are shown in Table 10.
A. A DDITIONAL R ELATED W ORK
Saliency Detection.
Object segmentation is closely related to saliency detection, the problem of ﬁnding signiﬁcant (salient) objects in an image.
Although most saliency detection models are trained with pixel-level supervision (or human eye movement data), the past few years have seen some research into unsupervised/weakly-supervised saliency detection (Zhang et al., 2018; Nguyen et al., 2019; Zeng et al., 2019).
These methods work by ensembling strong hand-crafted priors and distilling them into a deep network.
In practice, they also initialize their networks with pretrained (supervised) image classiﬁers or semantic segmentation networks.
Learning from Synthetic Data.
Finally, our method can be viewed from the perspective of learning from synthetic data.
Motivated by the costly and time-consuming nature of data labeling, there has been a plethora of recent work on synthesizing, training with, and adapting from synthetic datasets.
For example, one widely-studied line of research (Hoffman et al., 2018;
Tsai et al., 2018;
Zou et al., 2018; Vu et al., 2019; Toldo et al., 2020) tackles the task of semantic segmentation by training on data generated from video games (e.g. GTA5).
With regard to adversarially-generated training data speciﬁcally (Shrivastava et al., 2017) uses a GAN-like network to enhance the realism of synthetic images while preserving label information.
Although we train our segmentation network using generated images only, we show in the experiments below that it generalizes to real-world images without the need for additional adaptation.
Such additional adaptation could be an interesting avenue for future research.
[Table]
[Table]
A. A DDITIONAL E XAMPLES A.5. E XAMPLES A CROSS D ATASETS
In Figure
Figure 7,aand Figure we show the results of applying our ﬁnal segmentation network to random images from each of the ﬁve datasets on which we evaluated
. A.5. E XAMPLES A CROSS G ENERATORS
In Figure we show examples of pairs of generated images and the corresponding extracted samples for a range of different GANs.
A.5. S EMANTIC S EGMENTATION E XAMPLES
In Figure we provide randomly selected examples of our semantic segmentations after K -Means clustering on the PASCAL-VOC dataset.
We see that our masks are of similar visual quality to those from MaskContrast (Van Gansbeke et al., 2021), despite the fact that our method is entirely unsupervised.
Finding informative code variations
[Figure]
ContraGAN
SAGAN
[Figure]
Perturbation Amount ( 𝜺 ) 0.
[Figure] [Figure]
CUB Flowers DUT
[Figure]
OMRON
[Figure] [Figure] [Figure]
B F URTHER E XPERIMENTS
Based on the helpful comments of our reviewers, we have conducted new experiments and created more visualizations to better understand how our method functions and performs.
B. A DDITIONAL E XPERIMENTS ON F LOWERS
In this section, we provide a comprehensive explanation and analysis of our performance on the Flowers dataset. The Flowers dataset is different from the other datasets investigated in our paper (CUB, DUTS, DUT-OMRON, ECSSD, PASCAL VOC) in that the “ground-truth” masks for Flowers were obtained using an automated segmentation method rather than human annotations.
As a result, the ground truth are very unreliable.
In fact, there are many instances in the Flowers dataset in which the ground truth mask is completely or nearly completely empty despite there clearly being a ﬂower in the image.
This issue is also mentioned by Chen et al.
(2019), who claim that their method “provide[s] better masks” than the ground-truth in cases of disagreement.
Below, we present an additional experiment to improve our performance on Flowers.
We switch the underlying GAN in our method from BigBiGAN (trained on ImageNet) to an unconditional
StyleGAN2 trained on Flowers.
This experiment aims to show that our method works well on in-domain data (i.e. using a GAN trained on Flowers to segment Flowers) in addition to out-of-domain data (i.e. using a GAN trained on ImageNet to segment Flowers).
Speciﬁcally, we found publicly available weights on GitHub for an (unconditional) StyleGAN model Karras et al.
(2019) trained on the Flowers dataset at a resolution of 256 px, and we applied our method directly to this model.
We use perturbation radius r = 0 .
{+2 rather than r = 2 . 
+}
as in BigBiGAN because the latent space of StyleGAN2 has a different scale.
Apart from this singular parameter, however we apply our method without any hyperparameter tuning or additional modiﬁcations for the new StyleGAN2.
Results are shown in Table 11.
We ﬁnd that using StyleGAN2 dramatically improves the visual quality of our segmentations both quantitatively and qualitatively.
Quantitatively, our performance under this setup is similar to other unsupervised methods which have been extensively ﬁnetuned for this task, such as Chen et al.
We show examples of failure cases in Fig.
13 and Fig.
The most severe failure cases involve empty ground-truth masks.
Similarly to ReDo, we ﬁnd that when our method and the ground truth disagree, our segmentations are often visually superior.
At this level of performance, the high level of label noise means that exact mIoU/accuracy numbers are not meaningful.
However, there is no doubt that switching generators to StyleGAN tremendously improves our ﬂower segmentation performance.
Additionally, we sought to identify the extent to which this improvement was due purely to the increased resolution ( 256 px) as opposed to the underlying GAN.
We train a segmentation model 128 px-downsampled versions of the StyleGAN2 generated images and show the results in Table 15.
Higher resolution training improves performance modestly on the Flowers dataset, while the bulk of the improvement is due to the use of StyleGAN.
This result supports our hypothesis that better
GANs correspond to better unsupervised object segmentation performance under our model.
B. A DDITIONAL E XPERIMENTS ON F ACES
In this section, we provide additional experiments on the domain of face images.
This section is designed with comparison to Abdal et al.
(2021) in mind.
However, it is difﬁcult to compare with Abdal et al.
(2021) because they evaluate on non-standard datasets, they do not release any information about their data splits, and they do not release code or pretrained models.
In particular, they conduct evaluations on CelebA-HQ-Mask using 1000 randomly selected images from the dataset, but they do not specify which images these are.
Nevertheless, we tried to recreate their setup as best as possible to establish a comparison.
We applied our method to a StyleGAN2 trained on FFHQ.
We use resolution 256 px due to time constraints (and 1024 px would likely improve results).
We use a perturbation radius r = 0 . 2 , which is copied directly from the experiment in the section above (because both experiments use StyleGAN2).
Again, with the exception of this single parameter, we did not change a single other hyperparameter in our entire learning setup to demonstrate the generalizability of our approach.
Results are shown in Table 12.
Qualitatively, we ﬁnd that our model tends to segment the hair and face but not the clothes, which is an equally valid segmentation but does not align with the ground truth where the person’s body is included in the foreground mask.
This result could mean that
StyleGAN2 internally models the clothing in a different manner from the face which would lead to this behavior in latent space.
B. D ATA S IZE
In this section, we investigate the relationship between the amount of synthetic training data and segmentation performance.
All experiments in the main paper use 1,000,000 generated examples.
We train models for the same number of total iterations using a varying number of synthetic training images from 100 to 1,000,000.
Results are shown in Table 14 and Fig.
We ﬁnd consistently across all datasets that - similar to supervised learning - performance scales logarithmically with the dataset size.
Differently from supervised learning, our labels are entirely free to generate, so we can scale our synthetic dataset as large as desired.
B. S EMI - SUPERVISED L EARNING
In this section, we investigate whether our synthetic data can be used to augment existing supervised datasets in a semi-supervised learning setup.
We consider a subset of the CUB dataset with 1000 labeled examples along with synthetic data from our BigBiGAN (ImageNet-pretrained) model.
Results are shown in Table 13.
First, we train a supervised model on the 1000 images with ground truth segmentations as a baseline.
Similarly, we train a fully unsupervised model on our generated data alone.
Now, in the semisupervised setting, we have access to 1000 labeled images.
When training on the combination of both datasets we can already see an improvement over both baselines in IoU (which is the harder and less saturated metric).
With access to some in-domain data, we can also take another step to reduce the domain gap between the generated and real images: we can ﬁlter our generated images by their distance to the supervised samples.
Speciﬁcally, we select the 50000 nearest neighbors in our generated set (according to cosine similarity using a self-supervised ResNet-50 He et al.
, and we refer to these as the “kNN generated images” in Table 13.
We see that a model trained on the combined ground truth and kNN samples surpasses both baselines and the supervised model.
This result demonstrates that our pipeline can play a valuable role in semi-supervised object segmentation in addition to unsupervised object segmentation.
B.4. F AILURE C ASES
In this section, we describe common failure cases of our models.
Fig.
12, Fig. 13, Fig.
14, and Fig.
15 show examples of failure cases of our model.
One of the main failure cases of our method is to segment additional foreground objects beyond the “main object” in the image.
For example, on the CUB dataset (Fig. 12), we often segment objects such as branches and bird feeders alongside the bird in the image.
Across all datasets, one of the other main sources of errors is the object boundary that is not as precise as the object.
This could potentially be improved in the future through the use of higher-resolution GANs or by postprocessing (e.g. with a CRF).
The Flowers dataset is a special case in which the largest source of error is label noise in the ground truth annotations.
We discuss this issue extensively in the section on Flowers above.
Fig.
13 shows our top 16 predictions by mIoU error in the Flowers dataset: the ground truth segmentations are empty.
Fig.
13 shows top failure cases on Flowers where the ground truth mask is not empty, which still mainly consists of low quality ground truth annotations.
Occasionally, our method misses to segment a ﬂower in the background or segments too much of the stem.
With regard to failure cases of mask generation, since there is no ground-truth associated with these generated images, these failure cases were found by manually ﬁltering approximately 500 generated images (Fig. 15).
Common failure cases include segmenting too much or too little of the image and are often attributable to low-quality images generated by the GAN.
Thus, improvements in the generation quality of GANs and the performance of our method are closely linked.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
the best results.
[Table]
[Table]
different input image resolutions.
Using a higher resolution improves performance slightly.
[Figure] [Figure] [Figure] [Figure]