ComGAN: Unsupervised Disentanglement and Segmentation via Image Composition
Abstract
We propose ComGAN, a simple unsupervised generative model, which simultaneously generates realistic images and high semantic masks under an adversarial loss and a binary regularization.
In this paper, we first investigate two kinds of trivial solutions in the compositional generation process, and demonstrate their source is vanishing gradients on the mask.
Then, we solve trivial solutions from the perspective of architecture.
Furthermore, we redesign two fully unsupervised modules based on ComGAN (DS-ComGAN), where the disentanglement module associates the foreground, background and mask with three independent variables, and the segmentation module learns object segmentation.
Experimental results show that (i) ComGAN’s network architecture effectively avoids trivial solutions without any supervised information and regularization; (ii) DS-ComGAN achieves remarkable results and outperforms existing semi-supervised and weakly supervised methods by a large margin in both the image disentanglement and unsupervised segmentation tasks.
It implies that the redesign of ComGAN is a possible direction for future unsupervised work.
Introduction
Generative adversarial networks (GANs) [1] have been successful in realistic image generation [2, 3], and recent work on GAN-based image composition [4, 5, 6, 7, 8, 9] has shown that GANs identify and disentangle a class of objects from the background.
To direct that GANs learn the distinction between foreground and background, this kind of work commonly requires the following assumption:
Assumption 1 (Image composition [10] )
An image x taken from the world is typically composed of foreground x f and background x b , which can be decomposed by the following equation:
[Equation]
where x m is the mask, and the ⊙ denotes element wise multiplication operator.
It is notable that the Assumption 1 is mild and achieves significant outcomes in many areas.
For example, the models [11, 12] capture meaningful foreground segmentation masks via exactly coupling different foregrounds and backgrounds.
C3-GAN [13] proposes a scene decomposition-based method to enforce the model to learn the features of foreground objects.
Yang et al.
[14, 15] detect moving objects in video by minimizing the mutual information of foreground-background and training in an adversarial manner.
A class of methods [16, 17] explores and perturbs the latent space of pre-trained
GANs to find foreground masks.
However, it is observed that under the Assumption 1, the generation process is always accompanied by trivial solutions [10, 18, 11, 19, 20, 21, 22, 12].
Trivial solutions [11] can be considered as meaningless masks generated by models.
To our knowledge, existing works do not indicate that the source of the trivial solutions and they alleviate this issue in two ways.
One way is to add supervised information [23, 21, 19, 18], such as CGN [19] avoids trivial solutions by adding pre-trained U2-Net [24].
Another is to design clever regularization and fine-tune the parameters [10, 11, 20, 22, 12].
For example, PerturbGAN [11] proposes a regularization based on image recombination with arbitrarily small relative shifts.
Yang et al.
[20] propose a regularization based on mutual information maximization.
These methods are sensitive to object scale, object category and datasets, and therefore consume computational resources and time costs to perform hyperparameters fine-tuning.
For our solution, we propose an unsupervised generative model called
ComGAN, which simultaneously generates realistic images and high semantic masks.
In addition, our model effectively avoids trivial solutions from the perspective of architecture, which implies that the model does not require any supervised information and explicit regularization.
More specifically,
ComGAN is a generic way to generalize two typical image compositional generation methods and alleviate the shortcomings coming from the above two methods.
Furthermore, we place unique restrictions on each module, which help the model avoid trivial solutions.
Trivial solutions also have a negative role in related tasks.
For example, in the image disentanglement, low semantic masks cause foreground and background conflicts [25, 26], which reduce the synthesis quality.
In unsupervised segmentation, the regularization to alleviate trivial solutions may induce overfitting of the segmentation masks [14, 15, 12].
To highlight the flexibility and robustness of ComGAN, we extend it to the above two tasks.
The ComGAN-based variant is called DS-ComGAN, which contains both disentanglement and segmentation modules.
For multi-factor disentanglement, existing methods [26, 21, 27] rely on additional supervised information to learn the distinction of image regions.
Notice that ComGAN achieves foreground-background disentanglement in an unsupervised way.
As a result, the disentanglement module is designed as follows: we add more global information to the shared features in ComGAN and maximize the mutual information between variables and images.
This module simplifies the previous hierarchical generative network and outperforms the state-of-the-art (SOTA) semi-supervised and weakly supervised image disentanglement methods.
For unsupervised segmentation, existing methods [14, 15, 11, 12] rely on strong assumptions, such as that the foreground and background are largely independent, which limits their applicability.
Different from these methods, we train a segmentation network using the images and semantic masks synthesized by the disentanglement module.
By adversarial training strategy, the image distribution and the mask distribution are aligned.
Furthermore, a consistency regularization is introduced to ensure that the predicted masks are consistent with the inputs.
This segmentation module relies only on mild Assumption 1 and outperforms the SOTA unsupervised segmentation methods.
The main contributions are highlighted as follows:
To the best of our knowledge, we are the first to find that the source of trivial solutions is vanishing gradients on the mask.
Furthermore, we propose ComGAN, a simple unsupervised generative model, which is the first to solve trivial solutions from the perspective of architecture and achieve foreground-background disentanglement with only an adversarial loss and a binary regularization.
We propose DS-ComGAN, a variant network based on ComGAN, which achieves image disentanglement and object segmentation in a fully unsupervised manner.
Experiments show that DS-ComGAN is robust to various datasets and outperforms SOTAs in both tasks.
Related Work
GAN-based Image Composition.
Image composition can be regarded as combining multiple visual areas to construct a realistic image [28].
A series of typical methods built on GANs [1] perform image composition by utilizing Assumption 1 to achieve various functions.
LR-GAN [10] composites realistic images by learning to generate the backgrounds and foregrounds separately and recursively.
FineGAN [25] hierarchically composites images, which disentangles the background, object shape, and object appearance.
PerturbGAN [11] learns segmentation masks by training a generative model of a layered scene and composing foregrounds and backgrounds.
Similarly, SEIGAN [18] performs the cut, paste and inpaint operations consistent with semantic information to obtain segmentation masks.
Although the compositional image generation performs excellently in several tasks under
Assumption 1, these methods require additional regularization and hyperparameters tuning to avoid trivial solutions.
Controllable Image Disentanglement.
The goal of controllable image disentanglement is to disentangle factors of variation (e.g., object shape and object appearance) towards controllable generation.
InfoGAN [29] proposes semantic image generation control by imposing regularization of mutual information.
A series of works [25, 30, 31, 26] hierarchically generates images and disentangles the background, the object’s shape and its appearance by bounding box annotation.
MixNMatch [31] extends the FineGAN [25], which disentangles and encodes four variable factors (background, object pose, shape, and texture) from real images.
Along this direction, PartGAN [30] achieves part-level decomposition by learning a part generator.
Benny and Wolf [21] further propose a complex
GAN-based generative model, which requires a set of clean background images to simultaneously solve several tasks including image disentanglement and unsupervised segmentation.
These methods [32, 33] learn the independent latent characteristics of an object, especially its appearance and pose.
CGN [19] decomposes the image generation process into independent causal mechanisms, and allows for generating counterfactual images.
SSC-GAN [27] is a semi-supervised single-stage generative model considers three factors of variation via class labels, namely independent variables, cross-class variables, and class variables.
Although these methods exhibit strong performance, they all rely on additional supervised information to achieve multi-factor disentanglement.
[Figure] [Figure]
Unsupervised Object Segmentation.
Unsupervised object segmentation aims to extract a useful interpretation from an image in an unsupervised way.
W-net [34] extracts foreground objects through an encoder-decoder framework and minimizes reconstruction errors.
ReDO [35] captures the mask of the object, relying on the assumption that changing the foreground objects does not affect the overall data distribution.
Assuming that the foreground and background are largely independent, these methods [14, 15, 11, 12] obtain segmentation masks by reconstructing the realistic images, that is, precisely combining the different foregrounds and backgrounds.
Labels4Free [12] performs unsupervised object segmentation based on the idea that exact coupling between foreground and background is highly nontrivial.
A similar assumption is implemented In method [36], where IEM learns image segmentation from an information information-theoretic by maximizing inpainting error.
Similarly, Yang et al.
[20] synthesize paired realistic images and segmentation masks by maximizing mutual information between generated images and latent variables.
These methods frequently require stringent assumptions and are not necessarily applicable to complex realworld data.
Proposed Approach 3.1 Trivial Solutions and ComGAN
Source of trivial solutions.
Two kinds of trivial solutions were introduced in [11].
The first trivial solution degrades the whole scene into foreground or background, and the mask is always full or empty.
The second trivial solution makes the foreground and background identical, and the mask is meaningless for the composite scene.
Then, we state the source of two kinds of trivial solutions.
Lemma 1 ( Vanishing gradients on the mask ).
Let L all be the overall loss and ¯ x m be the synthesized mask.
Consider a model that composes images utilizing Assumption 1.
There exist vanishing gradients on the mask, i.e. ∂L all /∂ ¯ x m =
if and only if the model converges to two kinds of trivial solutions.
Proof: See Appendix B.1.
□
Network architecture of ComGAN.
An overview of our proposed ComGAN is illustrated in Fig.
The generator of ComGAN consists of two parts.
The feature generator denoted as Φ synthesizes features conditioned on the variable z , i.e. ϕ =
Φ ( z ) where z is sampled from a prior distribution p 0 .
The three subnetworks are defined as F , B and M .
They synthesize the foreground, background and mask, that is, ¯ x f = F ( ϕ ) , ¯ x b = B ( ϕ ) and ¯ x m =
M ( ϕ ) .
Driven by Assumption 1, the composite image ¯ x is defined as:
[Equation]
Remark
This form generalizes two typical image compositional generation methods:
• If only Φ( · ) in B is an identity map, i.e. B (Φ( z ))
=
B ( z ) , then this form is equivalent to the model Π 1 , that is, two independent generators synthesize a composite image where shared features exist in the foreground and mask generation.
For example, FineGAN [25], C3-GAN [13] and Labels4Free [12], etc. can be formulated as the model Π 1 .
• If Φ( · ) is an identity map, i
e. Φ( z =
z , then this form is equivalent to the model Π 2 , that is, three independent generators synthesize a composite image where foreground, background and mask are generated by three generators respectively.
For example, PerturbGAN [11] and CGN [19], etc. can be formulated as the model Π 2 .
From the above observations, the two typical methods have a common shortcoming, that is, models Π 1 and Π 2 both contain an independent background generation process.
The term ignores the association of foreground, background and mask on the features.
Moreover, if the model converges to trivial solutions, then the model degrades to an original GAN that suffers from known shortcomings, such as mode collapse.
As a result, our proposed method is a generic way and mitigates the above shortcoming.
Then, the adversarial training loss L advD is defined as follows:
[Equation]
Obviously, Lemma 1 points us to the limitation of the compositional generation under Assumption 1.
Then, we illustrate how to address the issue from the perspective of architecture.
Theorem 1 Given a generation G θ composed of a decoder Φ θ ϕ : Z → ϕ and three subnets F θ f , B θ b and M θ m : ϕ → X .
Let D be a discriminator and D ∗ ( G ∗ ( · )) be Nash equilibrium.
If the composite images satisfy ¯ x = F (Φ( z ))
⊙M (Φ( z ))
+ B (Φ( z ))
⊙ (1 −M (Φ( z ))) , ∥ D ( G ( · ))
− D ∗ ( G ∗ ( · ))
∥ < ϵ , max { E ϕ ∼ p (Φ( z ))
(cid:104)(cid:13)(cid:13) J θ f F ( ϕ ) (cid:13)(cid:13) 2 2 (cid:105) , E ϕ ∼ p (Φ( z )) (cid:104) ∥
J θ b B ( ϕ ) ∥ 22 (cid:105) } ≤ δ 2 and ∥∇ θ L advD ∥ 2 ≥ σ , then
[Equation]
Proof: See Appendix B.1.
□
We denote that ρ = σ 2 − δ 2 ϵ 2 (1 / 2 − ϵ ) 2 and trivial masks as ¯ x ∗ m .
If ρ > 0 , then the updated masks are ¯ x + m =
M θ + m (Φ θ + ϕ ( z )) , which means the model escape from the first trivial solution, i.e. ¯ x + m ̸
=
¯ x ∗ m .
Corollary
The following modules restrictions help the model avoid trivial solutions: the F and B are lightweight and differential, the M is a shallow network and the capacity of Φ is enough.
Proof: See Appendix B.1.
□
Upsample
Shared Feature
Compose f b f b
Back-Subnet
Mask-Subnet
Fore-Subnet z~p
Generator G
Discsriminator
Syn.mask
Real/Fake
Discsriminator
SegmentationNetworks S
D f
D b
D m
D z x m x
ResBlock 3  3 Conv 3  3 Conv + Sigmoid 3  3 Conv + Tanh
L D adv
L info
No gradient flow
Syn.image
Real image
Real/Fake
Syn.image
Real image x
_ z x _ m x
^ m x x _ x _
3.2 The Disentanglement Module of DS-ComGAN
Image generation is controlled by the mask variable z , foreground variable f and background variable b , where f and b are both N dimensional random one-hot vectors.
The generator consists of two parts, namely a feature generator denoted as G and three subnetworks defined as F , B and M .
Global information for shared features.
The features should contain global information so that the model captures a wide range of distinctions between foreground and background, rather than focusing on abrupt regions.
To add global information to the shared features of G , we synthesize realistic images for adversarial training, that is, ¯ x z is synthesized by G ( z ) and fed to the discriminator D z .
Then, we extract the activation map from the last three layers inside G , denoted as a 1 , a 2 and a 3 .
The shared features with global information can be written as follows:
[Equation]
where U is a spatially upsample to maintain the same size for all activation maps and ⊕ is a concatenation along the channel dimension.
In order to control image disentanglement, F , B and
M synthesize foreground, background, and mask conditioned on f , b and z , that is, ¯ x f = F ( ϕ, f ) , ¯ x b = B ( ϕ, b ) and ¯ x m = M ( ϕ ) .
Driven by Assumption 1, the composite image ¯ x are defined as:
[Equation]
Then, the discriminator D z is trained to classify images as real or fake and the adversarial training loss L advD z is defined as follows:
[Equation]
Controllable regularization for foreground-background subnetworks.
A mutual information regularization is incorporated into the F and B to associate the variables f and b with the synthesized ¯ x f and ¯ x b .
Similar to InfoGAN [29], we obtain similar results by maximizing the lower bound on the variance of the mutual information.
[Equation]
where I ( · ) is the mutual information, two discriminators D f and D
b approximate posterior distribution of p ( f | ¯ x f ) and p ( b | ¯ x b ) .
As implemented in [37, 20, 25], the I ( · ) is regarded as a regularization of a class of unsupervised clusters.
In other words, D f guides F to focus on generating foreground regions and learning the features of objects.
The similar analysis applies to D b .
According to [10, 38], it is difficult for GANs to learn a set of disconnected manifolds.
Intuitively, F and B learn simpler and continuous distributions, and thereby L info improves the quality of image generation.
3.3 The Segmentation Module of DS-ComGAN
The following two networks are defined: a segmentation network S predicts segmentation masks conditioned on the input images; a discriminator D m classifies masks as real or fake.
Our key idea is to exploit the high semantic masks synthesized by disentanglement module to align the image distribution and the mask distribution via adversarial training.
Image distribution alignment.
In the disentanglement module, D z is trained to identify x sampled from p data and ¯ x composited by our model.
When the model achieve Nash equilibrium, ¯ x has the identical distribution as x , i.e. p data ( x ) = p (¯ x ) .
Hence, image distribution alignment is accomplished.
Mask distribution alignment.
In the first step, we input the real image x and the synthetic image ¯ x into the segmentation network to obtain the corresponding prediction masks, that is, x m = S ( x ) and ˆ x m = S (¯ x ) .
Then, a mask triplet ( ¯ x m , ˆ x m , x m ) is fed into the discriminator D m where the synthesized mask x m is regarded as real and both the predicted masks ˆ x and x m are regarded as fake.
Mask distributions of x m , ˆ x m and ¯ x m are aligned via adversarial training,
[Equation]
The mask distribution alignment makes the predicted masks clearer, but this adversarial training strategy causes the issue that predicted masks are inconsistent with the input images.
Hence, the corresponding consistency loss L cons is introduced to ensure that the predicted masks are consistent with the inputs, rather than merely learning the mask distribution.
[Equation]
We transform the unsupervised segmentation task into two adversarial tasks and acquire segmentation masks by enhancing image and mask distribution consistency.
3.4 Model Training
The whole training process of ComGAN is simple, with only an adversarial loss and a binary regularization.
The learning objective is as follows:
[Equation]
where L binary is the pixel-wise binary entropy of mask from [19].
An explicit formulation of L binary is shown in Appendix B.2.
The sharpness of the semantic mask is modified by fine-tuning β .
The results of the visualization on fine-tuning β are available in Appendix C.1.
All the constituent networks of DS-ComGAN perform two kinds of unsupervised tasks: image disentanglement and object segmentation.
Overall, we optimize the following loss:
[Equation]
For the image disentanglement task, the model is performed in one stage.
Then, the synthesized images and semantic masks are fed into the segmentation network.
Hence, the objective segmentation process is composed of two stages.
Experiments 4.1 Experimental Setup
Datasets and implementation details.
The experiments are conducted on five fine-grained image datasets and a multi-object dataset: CUB [39], FS-100 [40], Stanford-Cars [41].
Stanford-Dogs [41], Flowers [42], CLEVR6 [43].
The ground truth segmentation masks of Stanford-Cars and
Stanford-Dogs are approximated by following the practice in [21, 44, 20].
More details about datasets and the implementation are available in the supplementary material.
[Figure]
ℱ(cid:4666)Φ(cid:4666)𝑧(cid:4667)(cid:4667)ℬ(cid:4666)Φ(cid:4666)𝑧(cid:4667)(cid:4667)ℳ(cid:4666)Φ(cid:4666)𝑧(cid:4667)(cid:4667)
[Figure]
[Figure]
Baseline and evaluation protocol.
In order to analyze the effectiveness of the adopted components, we established a baseline model,
i
[Figure]
e. SimpleGAN [25].
SimpleGAN aims to generate a realistic image
[Figure]
and its network contains a generator and a discriminator.
For a fair comparison, we use
SimpleGAN as the feature generator of ComGAN and DS-ComGAN.
We assess synthesis quality in terms of Inception Score ( IS ) [45] and Fréchet Inception Distance ( FID ) [46], which are computed on 20K randomly synthesized images.
To quantitatively evaluate the quality of the predicted masks, both the Intersecion of Union ( IoU ) and Dice score ( Dice ) [47] are used as the evaluation metrics.
4.2 Trivial Solutions Avoidance based on ComGAN
Avoidance of trivial solutions.
To show that ComGAN solves trivial solutions, we select two typical networks represent model Π 1 and Π 2 for comparison.
All models only include identical adversarial loss and binary regularization.
As a typical network for model Π 1 , FineGAN has two generator where a generates foreground and mask and another generates background.
PerturbGAN is selected as the typical network for model Π 2 , which consists of three generators that generates foreground, background and mask, respectively.
For a fair comparison, we make the following two things: (i) we use the code provided by the authors in FineGAN and retrain it with the identical training parameters; (ii) we simplify StyleGAN [48] with SimpleGAN in PerturbGAN and retrain it.
As shown in Fig.
2, the mask synthesized by ComGAN avoids convergence to trivial solutions and has non-trivial semantic information.
In addition, we track the gradient 2-norm of the three models.
It is clear from
Fig.
4 that the gradient norms of mask networks in the model Π 1 and Π 2 converge to zero and both models degrade to an original GAN, while our method effectively avoids vanishing gradients.
4.3 Image Disentanglement based on DS-ComGAN
Quantitative evaluation of image generation.
We compare DS-ComGAN with the competing
GAN-based generative models without any advanced GAN training strategy.
As illustrated in Table 1, the results show that DS-ComGAN significantly outperforms all the SOTA weakly supervised and semi-supervised methods.
FineGAN and MixNMatch may be limited by the foreground-background conflict, such as trivial masks.
SSC-GAN might have trouble in learning a set of disconnected manifolds.
To emphasize that DS-ComGAN is fully unsupervised and adapts to complex multi-object scenarios, we further perform our model on diverse datasets.
Stanford-Dogs and Flowers typically include a single foreground object, whereas CLEVR6 contains a variety of occluded and truncated objects.
Table 3 shows that our model is robust on various datasets and outperforms the SOTA image
V a r y i n g va r i a b
[Figure]
CLEVR
CUB
[Table]
Methods
[Table] [Table]
EnhancedTGAN
[Table] [Table]
disentanglement methods.
Fig 5 shows the qualitative generation results for each dataset, from which it is evident that our model generates not only realistic images but also clear semantic masks.
[Figure]
[Table]
Controllable image disentanglement.
Fig
6 illustrates that by varying three independent variables, DS-ComGAN synthesizes realistic and diverse images.
This indicates that the model achieves controlled image disentanglement.
We see that variable z is associated with the masks.
The pose of the object is altered with the varying variable z .
The colors and textures of the foreground and background are associated with variables f and b .
To clarify the role of f and b in image disentanglement, Fig 7 further analyzes the disentanglement of foreground and background.
In summary, DS-ComGAN associates three independent variables ( f , b and z ) with the foreground, background and mask, respectively.
Single Object
Multi-Object
Stanfor
[Table]
Dogs
Flowers
CLEVR
Methods
FID ↓
IS
[Table]
IS ↑
[Table]
IS
[Table]
SSC-GAN § [27]
[Table] [Table]
FineGAN § [25]
[Table]
MixNMatch § [31]
[Table]
CLEVR6 lacks bounding box annotation and labels.
Consequently, only our model is suitable for CLEVR6.
[Figure]
Effectiveness of model components To evaluate the contribution of each proposed component, we analyze quantitatively the performance margin between the baseline model and ComGAN.
As demonstrated in Table 2, the incorporation of F , B and M significantly improves the performance of the baseline model.
It illustrates that ComGAN has an excellent image generation performance.
As analyzed in section 3.2, the L info causes a decrease in FID by about 4 points.
4.4 Unsupervised Segmentation based on DS-ComGAN
To demonstrate that our method applies to complex real-world domains and multi-object scenarios with challenging spatial configurations, along the lines of [44], DS-ComGAN performs the unsupervised segmentation task on the various datasets.
Then, we perform a comparison between
DS-ComGAN and several state-of-the-art un(weakly-)supervised segmentation methods.
Qualitative analysis.
In Fig. 8, qualitative segmentation results of DS-ComGAN are presented.
We observe that the predicted masks segment visual details more precisely than the ground-truth masks, such as the legs of birds and the rearview mirrors of cars.
Although the predicted masks properly detect and segment foreground objects on the Flowers and Stanford-Cars datasets, the masks are inconsistent with the foreground images, which is mitigated by increasing λ .
Quantitative comparisons to prior work.
Table 4 demonstrates that DS-ComGAN exhibit robustness to diverse datasets and achieve superior performance across all metrics and datasets.
This implies that the model pays more attention to the discrepancy between the objects and background rather than the number of objects.
In addition, the result benefits from the images and high semantic masks synthesized by DS-ComGAN, which ensure two high-quality distribution alignments.
[Table]
Conclusion
As the limitation of method, DS-ComGAN has struggled to achieve the desired performance when the foreground object features are highly diverse (e.g., HKU-IS [56]).
We conjecture that the limitation might relate to the capacity of model.
DS-ComGAN performs excellently in controlled image synthesis tasks, which may cause the incidence of image falsification.
We first analyze that the source of trivial solutions is vanishing gradients on the mask.
Subsequently, we propose ComGAN, a simple unsupervised generative model, which simultaneously generates realistic images and high semantic masks, and effectively avoids trivial solutions.
Then, we design DSComGAN based on ComGAN, which exhibits excellent performance in both image disentanglement and unsupervised segmentation tasks.
Notably, DS-ComGAN obtains these excellent performances by relying on its architecture rather than complex and clever regularization.
It implies that ComGAN has the potential to shine in many tasks.
Acknowledgments
This work was supported by the Natural Science Foundation of China under Grant 62076255 and 62177047; Open Research Projects of Zhejiang Lab (NO. 2022RC0AB07); Hunan Provincial Science and Technology Plan Project 2020SK2059; Key projects of Hunan Education Department 20A88;
National Science Foundation of Hunan Province 2021JJ30082, 2019JJ20025 and 2019JJ40406;
National Social Science Fund of China (No. 20&ZD120).
A Details on Datasets and Hyperparameters A.1 Dataset Details
Caltech-UCSD Birds-200-2011.
This dataset contains about 12K bird images of 200 classes annotated with bounding boxes.
The ground truth segmentation masks of the foreground object are annotated by humans and provided in the official release [39].
As in the previous work [25, 31, 44],
We use the provided bounding box to extract a center square from the image, and scale it to 128 ×pixels.
Stanford Dogs.
This dataset consists of about 20K dog images of 120 classes annotated with bounding boxes [41].
Similar to [25, 31, 44], We use the provided bounding box to extract a center square from the image, and scale it to 128 × 128 pixels.
Since the ground-truth segmentation masks are not provided, we adopt the approximate ground-truth masks, generated by Yu et al.
The specific generation process is as follows.
Yu et al.
follow the practice in Benny and Wolf[21] to approximate ground-truth masks with Mask R-CNN[57], pre-trained on the COCO[58] dataset with a
ResNet-101[59] backend.
The pre-trained model is acquired from the detectron2 [60] toolkit.
Along the filtering strategy of [44], there exist 5,024 images with a clear foreground-background setup and high-quality mask.
Stanford Cars.
This dataset consists of about 16K car images of 196 classes annotated with bounding boxes [41].
We use the provided bounding box to extract a center square from the image, and scale it to 128 × 128 pixels.
To approximate the ground-truth masks, we adopt a similar process and filtering strategy to the Stanford Dogs dataset.
Finally, there exist 12,322 images for experimentation.
FS-100.
This dataset consists of about 15K human face images from the 100 largest classes [40, 27] of FaceScrub[45].
FaceScrub comprises a total of 107,818 face images of 530 celebrities.
For a fair comparison, we follow the [27] by scaling all the images to 64 × 64 pixels.
Flowers.
This dataset consists of about 8K flower images of 102 classes [42].
The ground truth segmentation masks are created by an automated approach designed exclusively for segmenting flowers in color photographs [61].
We scale all the images to 128 × 128 pixels.
CLEVR6 This dataset is a subset of the CLEVR dataset [43] with masks generated by Greff et al.
We take the first 8K samples from CLEVR and scale these samples to 128 × 128 pixels.
A.2 Hyperparameters and Training Details.
We adopt the ADAM optimizer[62] with a learning rate of ς = 2 × 10 − 4 and momentum parameters of β 1 = 0 .
5 and β 2 = 0 .
Following FineGAN[25] and DRC[44], we augment the real image with a horizontal flip and a random resizing crop.
The variable dimensions are set as follows: (1)
CUB: N = 200 ; (2) FS-100: N = 100 ; (3) Stanford-Dogs: N = 150 ; (4) Stanford-Cars: N = 200 ; (5) Flowers: N = 150 ; (6) CLEVR6: N = 20 .
We empirically set λ = 5 defined in equation (12) for all datasets.
The maximum training epochs are set to 500, where the disentanglement module is first trained for 450 epochs with a batch size of 32, and the segmentation module is second trained forepochs with a batch size of 64.
All experiments are performed under the environments of NVIDIA
GeForce RTX 3090 GPUs and an Intel Xeon Gold 6130 CPU.
B Details on Proofs and Regularization B.1 Proofs of things
Proof of Lemma 1: (Sufficiency) Under Assumption 1, the composes image ¯ x at each pixel i, j ∈ R can be written as follows:
[Equation]
where ¯ x f is the foreground and ¯ x b is the background.
As the model converges to the first trivial solution, the composite image degenerates to the foreground or background, i.e. ¯ x [ i, j ] =
¯ x f [ i, j ] or ¯ x
[ i, j ] =
¯ x b [ i, j ] .
It implies that the compositional generation process degenerates into a single foreground or background generation, which causes vanishing gradients on the mask.
Let ¯ G θ m be masks generator and θ m be the network parameters of ¯ G θ m .
Then, by using the chain rule, it can be obtained that
[Equation]
As the model converges to the second trivial solution, the foreground and background are identical, i.e. ¯ x f [ i, j ] =
¯ x b [ i, j ] , which causes vanishing gradients on the mask.
(Necessity)
Vanishing gradients on the mask means that no gradient is passed to the mask generator.
The synthesized mask is meaningless and fails to achieve foreground-background disentanglement.
Therefore, the model converges to the first trivial solution when the synthesized mask is full or empty.
In other cases, the model converges to the second trivial solution.
□
Proof of Theorem 1: ∥∇ θ L advD ∥ 2 can be written as
[Equation]
For simplicity of the following analysis, we assume the model converges to the first trivial solution and ¯ x m = 1 .
Under Assumption 1, it can be obtained that ¯ x = ¯ x f and ∂L all /∂ ¯ x m = ∂L all /∂ ¯ x b = 0 .
Hence, ∇ θ L all can be decomposed as ∇ ( θ ϕ ,θ f ) L all , and the gadient norms (15) can be written as:
[Equation]
[Equation]
Along the lines from [63, 64], we use Jensen’s inequality and the chain rule:
[Equation]
Then
[Equation]
it follows that
[Equation]
Notice that the mask can be written as M (Φ( z )) , which means
[Equation]
A similar discussion can be applied to the case of ¯ x m =□
Proof of Corollary 1: By Theorem 1, the larger the value of ρ = σ 2 − δ 2 ϵ 2 (1 / 2 − ϵ ) 2 , the easier it is for the model to escape from trivial solutions.
We observe that the δ 2 is decreased by reducing the parameter of θ f and θ b from the inequality max { E ϕ ∼ p (Φ( z ))
∥ J θ f F ( ϕ ) ∥ 22 , E ϕ ∼ p (Φ( z ))
∥ J θ b B ( ϕ ) ∥ 22 } ≤ δ 2 .
It is notable that if the θ f
and θ b have few parameters or even no parameters, our model degrades into an original GAN.
Therefore, the F and B should be designed as lightweight modules (e.g. adding residual connections), so as to enhance the capacity of the module with fewer parameters.
Furthermore, the ρ is the gradient norm from the decoder Φ .
It is a natural idea to increase ρ by raising the capacity of Φ .
In addition, the update of Φ does not necessarily mean the update of M .
If M is a deep neural network, it may not be able to map the fluctuations of features to the mask space, that is, M ( ϕ )
≈ M ( ϕ + △ ϕ ) .
Therefore, M should be designed as a shallow neural network.
For example, when M is a sigmoid layer, it is easy to get that M ( ϕ )
̸
= M ( ϕ + △ ϕ ) by the monotonicity of the sigmoid function.
As for the second trivial solution, ¯ x f = ¯ x b is a fragile equilibrium.
We can break the identical mapping F θ f ( ϕ ) =
B θ b ( ϕ ) , by changing parameters of ( θ f , θ b ) or modifying the structure of the F and B .
□
B.2 Regularization
The binary regularization L binary enforces the output to be close to either 0 or 1, which is as follows:
[Equation]
C Additional results about ComGAN
[Figure] [Figure]
We compare the visual effects of masks synthesized by ComGAN under various β .
The comparison results are illustrated in Fig.
Without any supervised information, it is difficult for the model to distinguish which part of the image belongs to the foreground or the background.
Therefore, we introduce the binary regularization βL binary to separate the foreground-background regions.
[Figure]
C.2 Imbalance of foreground and background subnetworks
In contrast to the background, the foreground is not occluded and contains complete information.
Therefore, we modify the foreground and background subnetworks with identical structures to make the two subnetworks have unbalanced network capacity.
For simplicity, we define a ComGAN variant called ComGAN ( N F − N B , N M ), where N F , N B and N M denote the number of ResBlocks in the foreground, background and mask subnetworks, respectively.
As shown in Fig.
10, a sizeable visual gap exists between the masks produced by ComGAN(3-1,0) and ComGAN (3-3,0).
The masks in
Fig.
10(a) contain more foreground information, such as branches and rocks, even if the information is insignificant.
In contrast, the masks Fig.
10(b)
lack partial details of objects, especially the bird’s feet.
It seems that we obtain higher-quality segmentation masks by trade-offs between the capacity of the foreground and background subnetworks.
C.3 Relaxing constraints on mask subnetwork
Corollary 1 requires that the M is a shallow network.
The property is a strong constraint on the network design.
Therefore, we show that a ComGAN variant avoids trivial solutions even when
[Figure]
Methods
[Table]
Triple-GAN
[Table] [Table]
EnhancedTGAN
[Table] [Table]
Triangle-GAN [50]
[Table] [Table]
SSC
[Table]
FineGAN
[Table] [Table]
MixNMatch
[Table] [Table]
SN
[Table] [Table]
[Table]
ComGAN
[Table] [Table]
[Table]
IODINE
[Table]
Slot
[Table]
Attn.
[Table] [Table]
IEM+SegNet [36]
[Table]
DRC [44] 56.
[Table]
ComGAN (3-3,
[Table]
the mask subnetwork is not a sigmoid function.
ComGAN variants are designed by increasing the number of ResBlocks in the mask subnetwork.
It can be seen from Fig.
10(b)
and Fig 11 that the extension of the mask subnetwork does not result in trivial solutions and provides masks that retain more details.
C.4 Quantitative evaluation of image generation.
As summarized in Table 5, ComGAN variants can achieve comparable FID scores.
We conjecture that improved image synthesis quality is attributable to ComGAN‘s network architecture.
The mask subnetwork and binary regularization force foreground and background subnetworks to focus on learning simpler and continuous foreground and background regions.
Furthermore, the experimental results on GAN-based image composition models such as FineGAN and MixNMatch indicate that trivial solutions degrade the quality of image generation.
Specifically, the first trivial solution degrades the generation process of composing the foreground and background to a single generation.
The second trivial solution induces interference between the foreground and background, decreasing the composite image quality.
C.5 Quantitative evaluation of segmentation masks.
To highlight that ComGAN variants generate high semantic masks, we obtain the predicted segmentation masks by the method in Section 3.3.
As summarized in Table 6, ComGAN variants are simple and exhibit comparable performance with only two regularizations.
Possibly limited by the known drawbacks of GANs such as unstable training, ComGAN variants have differences in segmentation results.
Moreover, we observe that the quality of the predicted segmentation masks is improved when the image distribution is closer (i.e. the lower FID scores).
N =100 N =150 N = 200 N =250 N =
FID ↓ 16.
IS ↑ 4.
D Additional discussion about DS-ComGAN D.1 Influence of dimension of variables
Table 7 shows the FID and IS of DS-ComGAN trained on the CUD by varying the dimension of variables N .
We observe that our model achieves comparable performance with various N .
As discussed in section 3.2, the N can be considered as the number of categories in clusters.
Therefore, when N equals 200, model performance is marginally improved.
D.2 Ablation study on global feature extraction
[Figure]
As shown in Fig.
12, the model performs global feature extraction on the CUB dataset, which leads to a rapid convergence and decrease of 1 to 2 points in the FID scores.
Of note is that the extraction of global features consumes a large amount of GPU video memory and slightly improves the quality of image generation.
However, the operation is necessary for specific datasets such as Stanford Cars, because it permits the mask to capture the entire distinction between the foreground and background, as illustrated in Fig.
[Figure]
D.3 Additional unsupervised segmentation results
See Fig.
14, Fig. 15, Fig.
16, Fig. 17, and Fig.
18 for more unsupervised segmentation results by our model on each dataset.
From top to bottom, we display: (i) observed images, (ii) ground-truth masks, and (iii) predicted masks in each figure.