ComGAN: Unsupervised Disentanglement and Segmentation via Image Composition
Anonymous Author(s)
AffiliationAddress email
Abstract
We propose ComGAN, a simple unsupervised generative model, which simultaneously generates realistic images and high semantic masks under an adversarial lossand a binary regularization.
In this paper, we first investigate two kinds of trivialsolutions in the compositional generation process, and demonstrate their sourceis vanishing gradients on the mask.
Then, we solve trivial solutions from the perspective of architecture. Furthermore, we redesign two fully unsupervised modulesbased on ComGAN (DS-ComGAN), where the disentanglement module associatesthe foreground, background and mask with three independent variables, and thesegmentation module learns object segmentation.
Experimental results show that(i) ComGAN’s network architecture effectively avoids trivial solutions without anysupervised information and regularization; (ii) DS-ComGAN achieves remarkableresults and outperforms existing semi-supervised and weakly supervised methodsby a large margin in both the image disentanglement and unsupervised segmentation tasks.
It implies that the redesign of ComGAN is a possible direction forfuture unsupervised work.
Introduction
Generative adversarial networks (GANs) [1] have been successful in realistic image generation [2, 3],and recent work on GAN-based image composition [4, 5, 6, 7, 8, 9] has shown that GANs identifyand disentangle a class of objects from the background.
To direct that GANs learn the distinctionbetween foreground and background, this kind of work commonly requires the following assumption:
Assumption 1 (Image composition [10] )
An image x taken from the world is typically composedof foreground x f and background x b , which can be decomposed by the following equation:
[Equation]
where x m is the mask, and the ⊙ denotes element wise multiplication operator.
It is notable that the assumption 1 is mild and achieves significant outcomes in many areas.
Forexample, the models [11, 12] capture meaningful foreground segmentation masks via exactly couplingdifferent foregrounds and backgrounds.
C3-GAN [13] proposes a scene decomposition-based methodto enforce the model to learn the features of foreground objects.
Yang et al.
[14, 15] detect movingobjects in video by minimizing the mutual information of foreground-background and training in anadversarial manner.
A class of methods [16, 17] explores and perturbs the latent space of pre-trained
GANs to find foreground masks.
However, it is observed that under the assumption 1, the generationprocess is always accompanied by trivial solutions [10, 18, 11, 19, 20, 21, 22, 12].
Trivial solutions[11] can be considered as meaningless masks generated by models.
To our knowledge, existing works do not indicate that the source of the trivial solutions and they alleviate this issue in two ways.
One way is to add supervised information [23, 21, 19, 18], such as CGN [19] avoids trivial solutionsby adding pre-trained U2-Net [24].
Another is to design clever regularization and fine-tune theparameters [10, 11, 20, 22, 12].
For example, PerturbGAN [11] proposes a regularization based onimage recombination with arbitrarily small relative shifts.
Yang et al.
[20] propose a regularizationbased on mutual information maximization.
These methods are sensitive to object scale, objectcategory and datasets, and therefore consume computational resources and time costs to performhyperparameters fine-tuning.
For our solution, we propose an unsupervised generative model called ComGAN, which simultaneously generates realistic images and high semantic masks.
In addition,our model effectively avoids trivial solutions from the perspective of architecture, which implies thatthe model does not require any supervised information and explicit regularization.
More specifically, ComGAN is a generic way to generalize two typical image compositional generation methods andalleviate the shortcomings coming from the above two methods.
Furthermore, we place uniquerestrictions on each module, which help the model avoid trivial solutions. Trivial solutions also have a negative role in related tasks. For example, in the image disentanglement,low semantic masks cause foreground and background conflicts [25, 26], which reduce the synthesisquality.
In unsupervised segmentation, the regularization to alleviate trivial solutions may induce overfitting of the segmentation masks [14, 15, 12]. To highlight the flexibility and robustness of ComGAN,we extend it to the above two tasks. The ComGAN-based variant is called DS-ComGAN, whichcontains both disentanglement and segmentation modules.
For multi-factor disentanglement, existingmethods [26, 21, 27] rely on additional supervised information to learn the distinction of imageregions.
Notice that ComGAN achieves foreground-background disentanglement in an unsupervisedway. As a result, the disentanglement module is designed as follows: we add more global informationto the shared features in ComGAN and maximize the mutual information between variables andimages.
This module simplifies the previous hierarchical generative network and outperforms thestate-of-the-art (SOTA) semi-supervised and weakly supervised image disentanglement methods.
For unsupervised segmentation, existing methods [14, 15, 11, 12] rely on strong assumptions, suchas that the foreground and background are largely independent, which limits their applicability.
Different from these methods, we train a segmentation network using the images and semantic maskssynthesized by the disentanglement module. By adversarial training strategy, the image distributionand the mask distribution are aligned.
Furthermore, a consistency regularization is introduced toensure that the predicted masks are consistent with the inputs.
This segmentation module relies onlyon mild assumption 1 and outperforms the SOTA unsupervised segmentation methods.
The main contributions are highlighted as follows:
To the best of our knowledge, we are the first to find that the source of trivial solutions is vanishinggradients on the mask.
Furthermore, we propose ComGAN, a simple unsupervised generativemodel, which is the first to solve trivial solutions from the perspective of architecture and achieveforeground-background disentanglement with only an adversarial loss and a binary regularization
We propose DS- ComGAN a variant network based on ComGAN, which achieves image disentanglement and object segmentation in a fully unsupervised manner.
Experiments show that DS ComGAN is robust to various datasets and outperforms SOTAs in both tasks.
Related Work
GAN-based image composition.
Image composition can be regarded as combining multiple visualareas to construct a realistic image [28].
A series of typical methods built on GANs [1] performimage composition by utilizing assumption 1 to achieve various functions.
LR-GAN [10] compositesrealistic images by learning to generate the backgrounds and foregrounds separately and recursively.
FineGAN [25] hierarchically composites images, which disentangles the background, object shape,and object appearance.
PerturbGAN [11] learns segmentation masks by training a generative modelof a layered scene and composing foregrounds and backgrounds.
Similarly, SEIGAN [18] performsthe cut, paste and inpaint operations consistent with semantic information to obtain segmentationmasks.
Although the compositional image generation performs excellently in several tasks underassumption 1, these methods require additional regularization and hyperparameters tuning to avoidtrivial solutions.
Controllable Image Disentanglement.
The goal of controllable image disentanglement is to disentangle factors of variation (e.g., object shape and object appearance) towards controllable generation.
InfoGAN [29] proposes semantic image generation control by imposing regularization of mutualinformation.
A series of works [25, 30, 31, 26] hierarchically generates images and disentanglesthe background, the object’s shape and its appearance by bounding box annotation.
MixNMatch[31] extends the FineGAN [25], which disentangles and encodes four variable factors (background,object pose, shape, and texture) from real images.
Along this direction, PartGAN [30] achievespart-level decomposition by learning a part generator. Benny and Wolf [21] further propose a complex GAN-based generative model, which requires a set of clean background images to simultaneouslysolve several tasks including image disentanglement and unsupervised segmentation.
These methods[32, 33] learn the independent latent characteristics of an object, especially its appearance and pose.
CGN [19] decomposes the image generation process into independent causal mechanisms, and allowsfor generating counterfactual images.
SSC-GAN [27] is a semi-supervised single-stage generativemodel considers three factors of variation via class labels, namely independent variables, cross-classvariables, and class variables.
Although these methods exhibit strong performance, they all rely onadditional supervised information to achieve multi-factor disentanglement.
[Figure] [Figure]
Unsupervised Object Segmentation.
Unsupervised object segmentation aims toextract a useful interpretation from animage in an unsupervised way.
W-net[34] extracts foreground objects throughan encoder-decoder framework and minimizes reconstruction errors.
ReDO [35]captures the mask of the object, relyingon the assumption that changing the foreground objects does not affect the overalldata distribution.
Assuming that the foreground and background are largely independent, these methods [14, 15, 11, 12]obtain segmentation masks by reconstructing the realistic images, i.e., preciselycombining the different foregrounds andbackgrounds. Labels4Free [12] performsunsupervised object segmentation basedon the idea that exact coupling betweenforeground and background is highly nontrivial.
A similar assumption is implemented In method [36], where IEM learnsimage segmentation from an informationinformation-theoretic by maximizing inpainting error.
Similarly, Yang et al.
[20]synthesize paired realistic images and segmentation masks by maximizing mutualinformation between generated images andlatent variables.
These methods frequentlyrequire stringent assumptions and are notnecessarily applicable to complex realworld data.
Proposed Approach
3.1 Trivial solutions and ComGAN
Source of trivial solutions.
Two kinds of trivial solutions were introduced in [11].
The first trivialsolution degrades the whole scene into foreground or background, and the mask is always full orempty.
The second trivial solution makes the foreground and background identical, and the mask ismeaningless for the composite scene.
Then, we state the source of two kinds of trivial solutions.
Lemma 1 ( Vanishing gradients on the mask ).
Let L all be the overall loss function and ¯ x m be thesynthesized mask.
Consider a model that composes an image ¯ x utilizing assumption 1.
If there existvanishing gradients on the mask, i.e., ∂L all /∂ ¯ x m = 0 , then the model learns the trivial solutions.
Proof: See Appendix B.
□
Network architecture of ComGAN.
An overview of our proposed ComGAN is illustrated in Fig.
The generator of ComGAN consists of two parts.
The feature generator denoted as Φ synthesizesfeatures conditioned on the variable z , i.e., ϕ =
Φ ( z ) where z is sampled from a prior distributionp 0 .
The three independent subnetworks are defined as F , B and M .
They synthesize the foreground,background and mask respectively, i.e., ¯ x f = F ( ϕ ) , ¯ x b = B ( ϕ ) and ¯ x m = M ( ϕ ) .
Driven byassumption 1, the composite image ¯ x is defined as:
[Equation]
{+Remark 
+}
This form generalizes two typical image compositional generation methods:
• If only Φ( · ) in B is an identity map,
i
e., B (Φ( z ))
=
B ( z ) , then this form is equivalent to model Π 1 ,
that is, two independent generators synthesize a composite image where shared features exist in the
foreground and mask generation.
• If Φ( · ) is an identity map,
i
e., Φ( z ) =
z , then this form is equivalent to model Π 2 , that is, threeindependent generators synthesize a composite image where foreground, background and mask are
generated by three generators respectively.
From the above observations, we notice that the two typical methods have a common shortcoming,that is, models Π 1 and Π 2 both contain an independent background generation process.
This termignores the association of foreground, background and mask on the features.
Moreover, if the modelconverges to trivial solutions, it implies that the model degrades to an original GAN that suffers fromknown shortcomings, such as mode collapse.
As a result, our proposed method is a generic way andmitigates the above shortcoming.
Then, the adversarial training loss L advD is defined as follows:
[Equation]
Obviously, Lemma 1 points us to the limitation of the compositional generation under assumption 1.
Then, we illustrate how and why to address the issue from the perspective of architecture.
Theorem 1 Given a generation G θ composed of a decoder Φ θ ϕ : Z → ϕ and three subnets F θ f , B θ band M θ m : ϕ → X . Let D be a discriminator and D ∗ ( G ∗ ( · )) be Nash equilibrium. If the compositeimages satisfy ¯ x = F (Φ( z )) ⊙M (Φ( z ))
+ B (Φ( z )) ⊙ (1 −M (Φ( z ))) , ∥ D ( G ( · )) − D ∗ ( G ∗ ( · )) ∥ < ϵ ,max { E ϕ ∼ p (Φ( z ))
(cid:104)(cid:13)(cid:13) J θ f F ( ϕ ) (cid:13)(cid:13) 2 2 (cid:105) , E ϕ ∼ p (Φ( z )) (cid:104) ∥
J θ b B ( ϕ ) ∥ 22 (cid:105) } ≤ δ 2 and ∥∇ θ L advD ∥ 2 ≥ σ , then(cid:13)
[Equation]
∇ ( θ ϕ ,θ m [Equation] E z ∼ p
[Equation] z ) [log (1 − D ( F (Φ( z [Equation]
))] (cid:13) [Equation] > σ
[Equation]
− [Equation] [Equation] Proof: See Appendix B.
□
We denote that ρ = σ 2 − δϵ 2 (1 / 2 − ϵ ) 2 and trivial masks as ¯ x ∗ m = M θ ∗ m (Φ θ ∗ ϕ ( z )) .
If ρ > 0 , whichimplies that the Φ and M are updated.
The updated masks are ¯ x + m = M θ + m (Φ θ + ϕ ( z )) , which meansthat the model can escape from the first trivial solutions, i.e. ¯ x + m ̸[-= 
-]
= ¯ x ∗ m and ∂L advD /∂ ¯ x + m ̸ = 0 .
{+Corollary 
+}
The following modules restrictions help the model avoid trivial solutions: the F and Bare lightweight and differential, the M is a shallow network and the capacity of Φ is enough.
Proof: See Appendix B.
□
3.2 The Disentanglement Module of DS-ComGAN
Image generation is controlled by a mask variable z , a foreground variable f and a backgroundvariable b , where f and b are both N dimensional random one-hot vectors.
The generator of This
Upsample
Shared Feature
Compose f b f b
Back-Subnet
Mask-Subnet
Fore-Subnet z~p
Generator G
Discsriminator
Syn.mask
Real/Fake
Discsriminator
SegmentationNetworks S
D f
D b
D m
D z x m x
ResBlock 3  3 Conv 3  3 Conv + Sigmoid 3  3 Conv + Tanh
L D adv
L info
No gradient flow
Syn.image
Real image
Real/Fake
Syn.image
Real image x
_ z x _ m x
^ m x x _ x _ module consists of two parts, namely a feature generator denoted as G and three independentsubnetworks defined as F , B and M .
Global features for shared network.
The features should contain global information so that themodel captures a wide range of distinctions between foreground and background, rather than focusingon abrupt regions.
To add global information to the features of G , we synthesize realistic imagesfor adversarial training, i.e., ¯ x z is synthesized by G ( z ) and fed to the discriminator D z . Then, weextract the activation map from the last three layers inside G , denoted as a 1 , a 2 and a 3 .
The sharedfeatures with global information can be written as follows:
[Equation] [Equation]
where U is a spatially upsample to maintain the same size for all activation maps and ⊕ is aconcatenation along the channel dimension.
In order to control image disentanglement, F , B and
M synthesize foreground, background, and mask conditioned on f , b and z , i.e., ¯ x f = F ( ϕ, f ) ,¯ x b = B ( ϕ, b ) and ¯ x m = M ( ϕ ) .
Driven by assumption 1, the composite image ¯ x are defined as:
[Equation]
Then, the discriminator D z is trained to classify images as real or fake and the adversarial trainingloss L advD z is defined as follows:
[Equation]
Controllable regularization for foreground-background subnetworks.
A mutual informationregularization is incorporated into the F and B to associate the variables f and b with the synthesized¯ x f and ¯ x b .
Similar to InfoGAN [29], we obtain similar results by maximizing the lower bound on thevariance of the mutual information.
[Equation]
where I ( · ) is the mutual information, two discriminators D f and D
b approximate posterior distribution of p ( f | ¯ x f ) and p ( b | ¯ x b ) .
As implemented in [37, 20, 25], the I ( · ) is regarded as a regularizationof a class of unsupervised clusters.
In other words, D f guides F to focus on generating foregroundregions and learning the features of objects.
The similar analysis applies to D b .
According to [10, 38],it is difficult for GANs to learn a set of disconnected manifolds.
Intuitively, F and B learn simplerand continuous distributions, and thereby L info improves the quality of image generation.
3.3 The Segmentation Module of DS-ComGAN
The following two networks are defined: a segmentation network S predicts segmentation masksconditioned on the input images; a discriminator D m classifies masks as real or fake.
Our key idea is
[Figure]
to exploit the high semantic masks synthesized by our model to align the image distribution and themask distribution via adversarial training.
Image distribution alignment.
In the disentanglement module, D z is trained to identify x sampledfrom p data and ¯ x composited by our model.
When the model achieve Nash equilibrium, ¯ x has theidentical distribution as x , i.e., p data ( x ) = p (¯ x ) .
Hence, image distribution alignment is accomplished.
Mask distribution alignment.
In the first step, we input the real image x and the synthetic image ¯ xinto the segmentation network to obtain the corresponding prediction masks, i.e., x m = S ( x ) andˆ x m = S (¯ x ) .
Then, a mask triplet ( ¯ x , ˆ x , x m ) is fed into the discriminator D m where the synthesizedmask x m is regarded as real and both the predicted masks ˆ x and x m are regarded as fake.
Maskdistributions of x m , ˆ x m and ¯ x m are aligned via adversarial training,
[Equation]
The mask distribution alignment makes the predicted masks clearer, but this adversarial trainingstrategy causes the issue that predicted masks are inconsistent with the input images.
Hence, thecorresponding consistency loss L cons is introduced to ensure that the predicted masks are consistentwith the inputs, rather than merely learning the mask distribution. The L cons is as follows:
[Equation]
where λ is the weight of L cons .
We transform the unsupervised segmentation task into two adversarialtasks and acquire segmentation masks by enhancing image and mask distribution consistency.
Experiments
4.1 Experimental Setup
Datasets and implementation details.
The experiments are conducted on five fine-grained imagedatasets and a multi-object dataset: CUB [39], FS-100 [40], Stanford-Cars [41].
Stanford-Dogs[41], Flowers [42], CLEVR6 [43].
The ground truth segmentation masks of Stanford-Cars and
Stanford-Dogs are approximated by following the practice in [21, 44, 20].
More details about datasetsand the implementation are available in the supplementary material.
Baseline and evaluation protocol.
In order to analyze the effectiveness of the adopted components,we established a baseline model, i.e., SimpleGAN [25].
SimpleGAN aims to generate a realisticimage, and its network contains a generator and a discriminator.
For a fair comparison, we use
SimpleGAN as the feature generator of ComGAN and DS-ComGAN.
We assess synthesis quality interms of Inception Score ( IS ) [45] and Fréchet Inception Distance ( FID ) [46], which are computedon 20K randomly synthesized images.
To quantitatively evaluate the quality of the predicted masks,both the Intersecion of Union ( IoU ) and Dice score ( Dice ) [47] are used as the evaluation metrics.
4.2 Trivial solution avoidance based on ComGAN
Avoidance of trivial solutions.
To show that ComGAN solves trivial solutions, we select two typicalnetworks represent model Π 1 and Π 2 for comparison.
All models only include identical adversarialloss and binary regularization.
As a typical network for model Π 1 , FineGAN has two generatorwhere a generates foreground and mask and another generates background. PerturbGAN is selected [Figure]
ℱ(cid:4666)Φ(cid:4666)𝑧(cid:4667)(cid:4667)ℬ(cid:4666)Φ(cid:4666)𝑧(cid:4667)(cid:4667)ℳ(cid:4666)Φ(cid:4666)𝑧(cid:4667)(cid:4667)
[Figure] [Figure]
as the typical network for model Π 2 , which consists of three generators that generates foreground,background and mask, respectively.
For a fair comparison, we make the following two things: (i) weuse the code provided by the authors in FineGAN and retrain it with the identical training parameters;(ii)
we simplify StyleGAN [48] with SimpleGAN in PerturbGAN and retrain it.
As shown in Fig.2, the mask synthesized by ComGAN avoids convergence to trivial solutions and has non-trivialsemantic information.
In addition, we track the gradient 2-norm of the three models.
It is clear from
Figure 5 that the gradients norm of mask networks in model Π 1 and model Π 2 converge to zero andthe model degrades to a raw GAN, while our method effectively avoids vanishing gradients.
4.3 Image Disentanglement based on DS-ComGAN
Controllable image disentanglement of independent variables.
Fig.
6 illustrates that by varyingthree independent variables, DS-ComGAN synthesizes realistic and diverse images such that themodel achieves controlled image disentanglement.
We see that variable z is associated with themasks.
The colors and textures of the foreground and background are associated with variables f andb .
To clarify the role of f and b in image disentanglement, Fig 7 further analyzes the disentanglementof foreground and background.
In summary, DS-ComGAN associates three independent variables ( f ,b and z ) with the foreground, background and mask, respectively.
Quantitative evaluation of image generation.
We compare DS-ComGAN with the competing
GAN-based generative models without any advanced GAN training strategy.
As illustrated in Table1, the results show that DS-ComGAN significantly outperforms all the SOTA weakly supervised andsemi-supervised methods.
FineGAN and MixNMatch may be limited by the foreground-backgroundconflict, such as trivial masks.
SSC-GAN might have trouble in learning a set of disconnectedmanifolds.
To emphasize that DS-ComGAN is fully unsupervised and adapts to complex multi-objectscenarios, we further perform our model on diverse datasets.
Stanford-Dogs and Flowers typicallyinclude a single foreground object, whereas CLEVR6 contains a variety of occluded and truncated [Table] [Table] objects.
Table 3 shows that our model is robust on various datasets and outperforms the SOTA imagedisentanglement methods. Fig 4 shows the qualitative generation results for each dataset, from whichit is evident that our model generates not only realistic images but also clear semantic masks. Single Object [Table]
Object
Stanfor-Dogs
Flowers
CLEVR
Methods
FID ↓
IS ↑
FID ↓
IS ↑
FID ↓
IS ↑
SSC-GAN § [27] 64.
8.97 ± 0.
3.41 ± 0.
FineGAN § [25] 69.
[Table]
.27 ±
[Table]
MixNMatch § [31]
[Table]
[Table]
DS-ComGAN §
[Table]
CLEVR6 lacks bounding box annotation and labels.
Consequently, only our model is suitable for CLEVR6.
[Figure]
Method
FID
[Table]
IS
[Table]
Baseline
[Table]
+ Independent
[Table]
+ L info -based Reg.
[Table]
+ Extract global features.
[Table]
[Table]
Improvement [Table] [Table] Effectiveness of model components To eval- uate the contribution of each proposed component, we analyze quantitatively the performance margin between the baseline modeland ComGAN.
As demonstrated in Table2, the incorporation of F , B and M significantly improves the performance of the baseline model.
It illustrates that ComGAN hasan excellent image generation performance.
As analyzed in section 3.2, the L info causesa decrease in FID by about 3 points.
4.4 UnsupervisedSegmentation based on DS-ComGAN
To demonstrate that our method appliesto complex real-world domains and multiobject scenarios with challenging spatial configurations, along the lines of [44], DS-ComGANperforms the unsupervised segmentation task on the various datasets.
Then, we perform a comparisonbetween DS-ComGAN and several state-of-the-art un(weakly-)supervised segmentation methods.
Qualitative analysis.
In Fig. 8, qualitative segmentation results of DS-ComGAN are presented.
We observe that the predicted masks segment visual details more precisely than the ground-truthmasks, such as the legs of birds and the rearview mirrors of cars.
Although the predicted masksproperly detect and segment foreground objects on the Flowers and Stanford-Cars datasets, the masksare inconsistent with the foreground images, which is mitigated by increasing λ .
[Figure]
Single Object
Multi-Object
CUB
Stanford [Table] Dogs Stanford-Cars
Flowers
CLEVR
Methods
[Table]
W-Net [34] 24.
GrabCut [53] 30.
ReDO † ⋇ [35] 46.
OneGAN ⋄ ⋇ [21] 55.
IODINE † [54] 30.
PerturbGAN [11] 38.
Slot-Attn.
IEM+SegNet [36] 55.
DRC [44] 56.
DS-ComGAN [Table]
Quantitative comparisons to prior work.
Table 4 demonstrates that DS-ComGAN outperformsall unsupervised and weakly unsupervised baselines significantly.
The result benefits from the highsemantic masks synthesized by DS-ComGAN.
In other words, ComGAN-based network structurereduces the noise of regularization and trivial solutions.
Conclusion
As the limitation of method, DS-ComGAN has struggled to achieve the desired performance when theforeground object features are highly diverse (e.g., HKU-IS [56]).
We conjecture that the limitationmight relate to the capacity of model.
DS-ComGAN performs excellently in controlled imagesynthesis tasks, which may cause the incidence of image falsification.
We first analyze that the source of trivial solutions is vanishing gradients on the mask.
Subsequently,we propose ComGAN, a simple unsupervised generative model, which simultaneously generatesrealistic images and high semantic masks, and effectively avoids trivial solutions.
Then, we design DSComGAN based on ComGAN, which exhibits excellent performance in both image disentanglementand unsupervised segmentation tasks.
Notably, DS-ComGAN obtains these excellent performancesby relying on its architecture rather than complex and clever regularization.
It implies that ComGANhas the potential to shine in many tasks.
Checklist
[Table]
For all authors...
[Table] (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See Section 3.1, Section 4.2, Section 4.3, Section 4.4.
(b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
[Table] (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 1.(b)
Did you include complete proofs of all theoretical results?
[Yes] See the Lemma 1,Proposition ?? .
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- [Table]
imental results (either in the supplemental material or as a URL)?
[Yes] Code and [Table]
data are available at https://anonymous.4open.science/r/ComGAN-D174 .
We
provide detailed instructions of how to reproduce the main results in the supplemental [Table]
material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were
chosen)?
[Yes] See Section 4.1, Section 4.2, Section 4.3, Section 4.4 and supplementalmaterial.(c)
Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] See supplemental material. [Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... [Table]
(a) If your work uses existing assets , did you cite the
creators
[Yes
](b) Did you mention the license of the assets?
[N/ A](c)
Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A](e)
Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[N/A]5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A](b)
Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
Caltech-UCSD Birds-200-2011. This dataset contains about 12K bird images of 200 classes annotated with bounding boxes.
The ground truth segmentation masks of the foreground object areannotated by humans and provided in the official release [39].
As in the previous work [25, 31, 44], We use the provided bounding box to extract a center square from the image, and scale it to 128 × 128pixels. Stanford Dogs. This dataset consists about 20K dog images of 120 classes annotated with boundingboxes.
Similar to [25, 31, 44], We use the provided bounding box to extract a center square from theimage, and scale it to 128 × 128 pixels.
Since the ground-truth segmentation masks are not provided,we adopt the approximate ground-truth masks, generated by Yu et al.
The specific generationprocess is as follows.
Yu et al.
follow the practice in Benny and Wolf[21] to approximate ground-truthmasks with Mask R-CNN[57], pre-trained on the COCO[58] dataset with a ResNet-101[59] backend.
The pre-trained model is acquired from the detectron2 [60] toolkit.
Along the filtering strategy of[44], there exist 5,024 images with a clear foreground-background setup and high quality mask.
Stanford Cars.
This dataset consists about 16K car images of 196 classes annotated with boundingboxes.
We use the provided bounding box to extract a center square from the image, and scale it to128 × 128 pixels.
To approximate the ground-truth masks, we adopt the similar process and filteringstrategy as for the Stanford Dogs dataset.
Finally, there exist 12,322 images for experimentation.
FS-100.
This dataset consists about 15K human face images from the 100 largest classes of
FaceScrub[45].
FaceScrub comprises a total of 107,818 face images of 530 celebrities.
For afair comparison, we follow the [27] by scaling all the images to 64 × 64 pixels.
Flowers.
This dataset consists about 8K flower images of 102 classes.
The ground truth segmentationmasks created by an automated approach designed exclusively for segmenting flowers in colorphotographs [61].
We scale all the images to 128 × 128 pixels.
CLEVR6 This dataset is a subset of the CLEVR dataset [43] with masks, generated by Greff et al.[54].
We taks the first 8K samples from CLEVR and scale these samples to 128 × 128 pixels.
A.2 Hyperparameters and Training Details.
We adopt the ADAM optimizer[62] with a learning rate of ς = 2 × 10 − 4 and momentum parametersof β 1 = 0 .
5 and β 2 = 0 .
Following FineGAN[25] and DRC[44], we augment the real imagewith a horizontal flip and a random resizing crop.
The variable dimensions are set as follows: (1)
CUB: N = 200 ; (2) FS-100: N = 100 ; (3) Stanford-Dogs: N = 150 ; (4) Stanford-Cars: N = 200 ;(5) Flowers: N = 150 ; (6) CLEVR6: N = 20 .
We empirically set λ = 5 defined in equation (13) forall datasets.
The maximum training epochs is set to 500, where the disentanglement module is firsttrained for 450 epochs with a batch size of 32 and the segmentation module is second trained for 50epochs with a batch size of 64.
All experiments are performed under the environments of NVIDIA
GeForce RTX 3090 GPUs and an Intel Xeon Gold 6130 CPU.
B Additional results about ComGAN
some words to speak
B.1 Imbalance of foreground and background subnetworks
In contrast to the background, the foreground is not occluded and contains complete information.
Therefore, we modify the foreground and background subnetworks with identical structures to makethe two subnetworks have unbalanced network capacity.
For simplicity, we define a ComGAN variantcalled ComGAN ( N F − N B , N M ), where N F , N B and N M denote the number of ResBlocks in theforeground, background and mask subnetworks, respectively.
As shown in Fig.
9, a sizeable visualgap exists between the masks produced by ComGAN(3-1,0) and ComGAN (3-3,0).
The masks in
Fig.
9(a) contain more foreground information, such as branches and rocks, even if the information isinsignificant.
In contrast, the masks Fig. 9(b) lack partial details of objects, especially the bird’s feet.
[Figure]
[Figure]
It seems that we obtain higher-quality segmentation masks by trade-offs between the capacity of theforeground and background subnetworks.
B.2 Relaxing constraints on mask subnetwork
In Proposition ??
, we utilize the property that the mask subnetwork is an invertible function, but thisproperty is a strong constraint on the network design.
Therefore, we show that the ComGAN variantsavoid trivial solutions when the mask subnetwork is not just a sigmoid function.
ComGAN variantsare designed by increasing the number of ResBlocks in the mask subnetwork.
It can be seen from
Fig. 9(b) and Fig 10 that the extension of the mask subnetwork does not result in trivial solutions andprovides masks that retain more details.
Methods
Sup.
FID
[Table]
Triple-GAN[49]
Semi.
[Table]
94 ±
[Table]
EnhancedTGAN[40] Semi.
[Table]
[Table]
Triangle-GAN[50]
Semi. 96.42 4.36 ± [Table] R 3 -CGAN[51]
Semi.
[Table]
SSC-GAN[27]
Semi.
[Table]
[Table]
68 ±
[Table]
FineGAN[25]
Weak.
[Table]
[Table]
[Table]
MixNMatch[31]
Weak.
[Table] SN-GAN[52] Unsup. 160.09 4.21 ± 0.
ComGAN(3-3,0) Unsup [Table] [Table]
Quantitative evaluation of image generation As summarized in Table 5, we compare ComGAN variants with a series of competing GAN-basedgenerative models on the CUB dataset.
ComGAN variants can achieve comparable FID scores. Weconjecture that improved image synthesis quality is attributable to ComGAN‘s network architecture. The mask subnetwork and binary regularization force foreground and background subnetworks tofocus on learning simpler and continuous foreground and background regions. Furthermore, theexperimental results on GAN-based image composition models such as FineGAN and MixNMatchindicate that trivial solutions degrade the quality of image generation.
Specifically, the first trivialsolution degrades the generation process of composing the foreground and background to a singlegeneration.
The second trivial solution induces interference between the foreground and background,decreasing the composite image quality. B.4 Quantitative evaluation of segmentation masks. To highlight that ComGAN variants generate high semantic masks, we obtain the predicted segmentation masks by the method in Section 3.3. As summarized in Table 6, ComGAN variants aresimple and exhibit comparable performance with only two regularizations.
However, it is notablethat, limited by the known drawbacks of GAN, our model suffers from unstable training.
C Additional discussion about DS-ComGAN
[Figure]
C.1 Ablation study on global feature extraction
[Figure]
[Figure]
C.2 Additional unsupervised segmentation results
[Figure]