Z ERO - SHOT S YNTHESIS WITH G ROUP -S UPERVISED L EARNING
A BSTRACT
Visual cognition of primates is superior to that of artiﬁcial neural networks in its ability to “envision” a visual object, even a newly-introduced one, in different attributes including pose, position, color, texture, etc.
To aid neural networks to envision objects with different attributes, we propose a family of objective functions, expressed on groups of examples , as a novel learning framework that we term Group-Supervised Learning (GSL).
GSL allows us to decompose inputs into a disentangled representation with swappable components, that can be recombined to synthesize new samples.
For instance, images of red boats & blue cars can be decomposed and recombined to synthesize novel images of red cars .
We propose an implementation based on auto-encoder, termed group-supervised zeroshot synthesis network (GZS-Net) trained with our learning framework, that can produce a high-quality red car even if no such example is witnessed during training.
We test our model and learning framework on existing benchmarks, in addition to a new dataset that we open-source.
We qualitatively and quantitatively demonstrate that GZS-Net trained with GSL outperforms state-of-the-art methods.
I NTRODUCTION
Primates perform well at generalization tasks.
If presented with a single visual instance of an object, they often immediately can generalize and envision the object in different attributes, e.g., in different 3D pose (Logothetis et al., 1995).
Primates can readily do so, as their previous knowledge allows them to be cognizant of attributes.
Machines, by contrast, are most-commonly trained on sample features (e.g., pixels), not taking into consideration attributes that gave rise to those features.
To aid machine cognition of visual object attributes, a class of algorithms focuses on learning disentangled representations (Kingma & Welling, 2014;
Higgins et al., 2017; Burgess et al., 2018;
Kim & Mnih, 2018; Chen et al., 2018), which map visual samples onto a latent space that separates the information belonging to different attributes.
These methods show disentanglement by interpolating between attribute values (e.g., interpolate pose, etc).
However, these methods usually process one sample at a time, rather than contrasting or reasoning about a group of samples.
We posit that semantic links across samples could lead to better learning.
We are motivated by the visual generalization of primates.
We seek a method that can synthesize realistic images for arbitrary queries (e.g., a particular car, in a given pose, on a given background), which we refer to as controlled synthesis .
We design a method that enforces semantic consistency of attributes, facilitating controlled synthesis by leveraging semantic links between samples.
Our method maps samples onto a disentangled latent representation space that (i) consists of subspaces, each encoding one attribute (e.g., identity, pose, ...), and, (ii) is such that two visual samples that share an attribute value (e.g., both have identity “car”) have identical latent values in the shared attribute subspace (identity), even if other attribute values (e.g., pose) differ.
To achieve this, we propose a general learning framework: Group Supervised Learning (GSL, Sec. 3), which provides a learner (e.g., neural network) with groups of semantically-related training examples, represented as multigraph .
Given a query of attributes, GSL proposes groups of training examples with attribute combinations that are useful for synthesizing a test example satisfying the query (Fig. 1).
This endows the network with an envisioning capability.
In addition to applications in graphics, controlled synthesis can also augment training sets for better generalization on machine learning tasks (Sec. 6.3).
ImageImage- ImageImage-
ImageImage- ImageImage-
ImageImage-
Imageidentity pose background identity pose expression content font color backcolor size font (
[Figure]
) (b) (c)
Encoder Decoder
Disentangled Latent Features
As an instantiation of GSL, we propose an encoder-decoder network for zero-shot synthesis: GroupSupervised Zero-Shot Synthesis Network (GZS-Net, Sec. 4).
While learning (Sec. 4.2 & 4.3), we repeatedly draw a group of semantically-related examples, as informed by a multigraph created by GSL.
GZS-Net encodes group examples, to obtain latent vectors, then swaps entries for one or more attributes in the latent space across examples , through multigraph edges, then decodes into an example within the group (Sec. 4.2).
Our contributions are: (i)
We propose Group-Supervised Learning (GSL), explain how it casts its admissible datasets into a multigraph, and show how it can be used to express learning from semantically-related groups and to synthesize samples with controllable attributes; (ii) We show one instantiation of GSL: Group-supervised Zero-shot Synthesis Network (GZS-Net), trained on groups of examples and reconstruction objectives; (iii)
We demonstrate that GZS-Net trained with GSL outperforms state-of-the-art alternatives for controllable image synthesis on existing datasets; (iv)
We provide a new dataset, Fonts 1 , with its generating code.
It contains 1.56 million images and their attributes.
Its simplicity allows rapid idea prototyping for learning disentangled representations.
R ELATED W ORK
We review research areas, that share similarities with our work, to position our contribution.
Self-Supervised Learning (e.g., Gidaris et al.
(2018)) admits a dataset containing features of training samples (e.g., upright images) and maps it onto an auxiliary task (e.g., rotated images): dataset examples are drawn and a random transformation (e.g., rotate 90 ◦ ) is applied to each.
The task could be to predict the transformation (e.g., =90 ◦ ) from the transformed features (e.g., rotated image).
Our approach is similar, in that it also creates auxiliary tasks, however, the tasks we create involve semantically-related group of examples , rather than from one example at a time.
Disentangled Representation Learning are methods that infer latent factors given example visible features, under a generative assumption that each latent factor is responsible for generating one semantic attribute (e.g. color).
Following Variational Autoencoders (VAEs, Kingma & Welling, 2014), a class of models (including, Higgins et al., 2017; Chen et al., 2018) achieve disentanglement implicitly , by incorporating into the objective, a distance measure e.g. KL-divergence, encouraging the latent factors to be statistically-independent.
While these methods can disentangle the factors without knowing them beforehand, unfortunately, they are unable to generate novel combinations not witnessed during training (e.g., generating images of red car , without any in training).
On the other hand, our method requires knowing the semantic relationships between samples (e.g., which objects are of same identity and/or color), but can then synthesize novel combinations (e.g., by stitching latent features of “any car” plus “any red object”).
Conditional synthesis methods can synthesize a sample (e.g., image) and some use information external to the synthesized modalities, e.g.,
(2018) or class label Mirza & Osindero (2014); Tran et al.
Ours differ, in that our “external information” takes the form of semantic relationships between samples.
There are methods based on
GAN Goodfellow et al.
(2014) that also utilize semantic relationships including Motion Re-targeting (Yang et al., 2020), which unfortunately requires domain-speciﬁc hand-engineering (detect and track human body parts).
On the other hand, we design and apply our method on different tasks (including people faces, vehicles, fonts; see Fig. 1).
Further, we compare against two recent GAN methods starGAN (Choi et al., 2018) and ELEGANT (Xiao et al., 2018), as they are state-of-the-art GAN methods for amending visual attributes onto images.
While they are powerful in carrying local image transformations (within a small patch, e.g., changing skin tone or hair texture).
However, our method better maintains global information: when rotating the main object, the scene also rotates with it, in a semantically coherent manner.
Importantly, our learning framework allows expressing simpler model network architectures, such as feed-forward auto-encoders, trained with only reconstruction objectives, as opposed to GANs, with potential difﬁculties such as lack of convergence guarantees.
Zero-shot learning also consumes side-information.
For instance, models of Lampert (2009);
Atzmon & Chechik (2018) learn from object attributes, like our method.
However, (i) these models are supervised to accurately predict attributes, (ii) they train and infer one example at a time, and (iii) they are concerned with classifying unseen objects.
We differ in that (i) no learning gradients (supervision signal) are derived from the attributes, as (ii) these attributes are used to group the examples (based on shared attribute values), and (iii) we are concerned with generation rather than classiﬁcation: we want to synthesize an object in previously-unseen attribute combinations.
Graph Neural Networks (GNNs) (Scarselli et al., 2009) are a class of models described on graph structured data.
This is applicable to our method, as we propose to create a multigraph connecting training samples.
In fact, our method can be described as a GNN, with message passing functions (Gilmer et al., 2017) that are aware of the latent space partitioning per attribute (explained in Sec. 4).
Nonetheless, for self-containment, we introduce our method in the absence of the GNN framework.
G ROUP -S UPERVISED L EARNING 3. D ATASETS ADMISSIBLE BY GSL
Formally, a dataset admissible by GSL containing n samples D
= { x ( i ) } ni =1 where each example is accompanied with m attributes D
a
=
( a ( i ) 1 , a ( i ) 2 , . . .
a ( i ) m ) } ni =1 .
Each attribute value is a member of a countable set: a j ∈ A j .
For instance, pertaining to visual scenes, A 1 can denote foreground-colors
A 1 = { red , yellow , . . .
} , A 2 could denote background colors, A 3 could correspond to foreground identity, A 4 to (quantized) orientation.
Such datasets have appeared in literature, e.g. in Borji et al.
; Matthey et al.
; Langner et al.
Lai et al.
3. A UXILIARY TASKS VIA M ULTIGRAPHS
Given a dataset of n samples and their attributes, we deﬁne a multigraph M with node set [1 ..n ] .
Two nodes, i, k ∈ [1 ..n ] with i (cid:54) =
k are connected with edge labels M ( i, k ) ⊆
[1 ..m ] as:
[Equation]
In particular, M deﬁnes a multigraph, with | M ( i, k ) | denoting the number of edges connecting nodes i and k , which is equals the number of their shared attributes.
Fig.
2 depicts a (sub-)multigraph for the Fonts dataset (Sec. 5.1).
Deﬁnition 1 C OVER ( S, i ) :
Given node set S ⊆
[1 .. |D g | ] and node i ∈ [1 .. |D g | ] we say set S covers node i if every attribute value of i is in at least one member of S .
Formally:
[Equation]
When C OVER ( S, i ) holds, there are two mutually-exclusive cases: either i ∈ S , or i / ∈ S , respectively shaded as green and blue in Fig.
2 (b).
The ﬁrst case trivially holds even for small S , e.g.
C OVER ( { i } , i ) holds for all i .
However, we are interested in non-trivial sets where | S | > 1 , as sets with | S | = 1 would cast our proposed network (Sec. 4) to a standard Auto-Encoder.
The second case is crucial for zero-shot synthesis.
Suppose the (image) features of node i (in Fig.
2 (b)) are not given, we can search for S 1 , under the assumption that if C OVER ( S 1 , i ) holds, then S 1 contains sufﬁcient information to synthesize i ’s features as they are not given ( i / ∈ S 1 ).
Until this point, we made no assumptions how the pairs ( S, i ) are extracted (mined) from the multigraph s.t.
C OVER ( S, i ) holds.
In the sequel, we train with | S | = 2 and i ∈ S .
We ﬁnd that this particular specialization of GSL is easy to program, and we leave-out analyzing the impact of mining different kinds of cover sets for future work.
G ROUP -S UPERVISED Z ERO -S HOT S YNTHESIS N ETWORK (GZS-N ET )
We now describe our ingredients towards our goal: synthesize holistically-semantic novel images.
4. A UTO -E NCODING ALONG RELATIONS IN M
Auto-encoders ( D ◦ E ) :
X → X are composed of an encoder network E : X → R d and a decoder network D : R d → X .
Our networks further utilize M emitted by GSL.
GZS-Net consists of an encoder E : X
× M → R d × M ; and a decoder D : R d × M → X .
M denotes the space of sample pairwise-relationships.
GSL realizes such ( X, M ) ⊂
X × M , where
X contains (a batch of) training samples and M the (sub)graph of their pairwise relations.
Rather than passing as-is the output of E into D , one can modify it using algorithm A by chaining: D ◦ A ◦ E .
For notation brevity, we fold A into the encoder E , by designing a swap operation, next.
4. D ISENTANGLEMENT BY SWAP O PERATION
While training our auto-encoder D ( E ( X, M )) , we wish to disentangle the latents output by E , to provide use for using D to decode samples not given to E .
D (/ E ) outputs (/ inputs) one or more images, onto (/ from) the image space.
Both networks can access feature and relationship information.
At a high level, GZS-Net aims to swap attributes across images by swapping corresponding entries across their latent representations.
Before any training, we ﬁx partitioning of the the latent space
Z = E
( X, M ) .
Let row-vector z (1) =
[ g (1)1 , g (1)2 , . . .
, g (1) m ] be the concatenation of m row vectors { g (1) j ∈ R d j } mj =1 where d
= (cid:80) mj
=1 d j and the values of { d j } mj =1 are hyperparameters.
To simplify the notation to follow, we deﬁne an operation swap : R d × R d × [1 ..m ]
→ R d × R d , which accepts two latent vectors (e.g., z (1) and z (2) ) and an attribute (e.g., 2) and returns the input vectors except that the latent features corresponding to the attribute are swapped.
E.g., swap ( z (1) , z (2) , 2) = swap ([ g (1)1 , g (1)2 , g (1)3 , . .
, g (1) m ] , [ g (2)1 , g (2)2 , g (2)3 , . .
, g (2) m ] , 2) =
[ g (1)1 , g
(2)2 , g (1)3 , . . .
, g (1) m ] , [ g (2)1 , g (1)2 , g (2)3 , . .
, g (2) m ]
One-Overlap Attribute Swap .
To encourage disentanglement in the latent representation of attributes, we consider group S and example x s.t.
C OVER ( S, x ) holds, and for all x o ∈ S, x (cid:54) =
x o , the
𝐿 !
Group Images: (a) (b)
E
D 𝒙
Self Reconstruction:
OneOverlapAttributeSwap
E
D
E
Same id 𝒙
D ?
E
D
E
D
CycleAttributeSwap 𝒙 𝒙#
E
D
E
D 𝒙# "#$ 𝒙 "#$ 𝒙# %&’ 𝒙 %&’
Same id 𝒙 𝒙# Same pose
Same back No overlap (c)
Random CorrespondingSwap
Encoder
Decoder
DisentangledLatent Features
FeatureSwap ?
Reconstruction Loss
SwapReconstruction Loss
Cycle Swap Reconstruction Loss
E
D
Framed Images : Real images; No framed Images: Synthesis images
Image copy
Same pose 𝒙
E
D
E
D
E
D
E
D
Same back 𝒙 𝐿 #!
𝐿 #! 𝐿 #!
𝐿 #! 𝐿 #!
𝐿 #!
𝐿 (#! 𝐿 (#! 𝐿 !
𝐿 #!
𝐿 (#! cycle enforces that double-swap on j does not destroy information for other attributes.
pair ( x o , x ) share exactly one attribute value ( | M ( x o , x ) | = 1 ).
Encoding those pairs, swapping the latent representation of the attribute, and decoding should then be a no-op if the swap did not affect other attributes (Fig. 3b).
Speciﬁcally, we would like for a pair of examples, x (red border in Fig.
3b) and x o (blue border) sharing only attribute j (e.g., identity) 2 , with z = E ( x ) and z o = E ( x o ) , be s.t.
D ( z s )
≈ x and D (cid:16) z
( o ) s (cid:17)
≈ x ( o ) ; with z s , z ( o ) s = swap
[Equation]
z o
[Equation]
j ) .
If, for each attribute, sufﬁcient sample pairs share only that attribute, and Eq.
3 holds for all with zero residual loss, then disentanglement is achieved for that attribute (on the training set).
Cycle Attribute Swap .
This operates on all example pairs, regardless of whether they share an attribute or not.
Given two examples and their corresponding latent vectors, if we swap latent information corresponding to any attribute , we should end up with a sensible decoding.
However, we may not have ground-truth supervision samples for swapping all attributes of all pairs.
For instance, when swapping the color attribute between pair orange truck and white airplane , we would like to learn from this pair, even without any orange airplanes in the dataset.
To train from any pair, we are motivated to follow a recipe similar to CycleGAN (Zhu et al., 2017).
As shown in Fig.
3c, given two examples x and ¯ x : (i) sample an attribute j ∼ U [1 ..m ] ; (ii) encode both examples, z = E ( x ) and ¯
z = E (¯ x ) ; (iii) swap features corresponding to attribute j with z s ,
¯ z s = swap ( z, ¯ z, j ) ; (iv) decode, (cid:98) x
= D ( z s ) and (cid:98) ¯ x = D (¯ z s ) ; (v) on a second round (hence, cycle), encode again as (cid:98)
z
= E ( (cid:98) x ) and (cid:98) ¯
z = E ( (cid:98) ¯ x ) ; (vi) another swap, which should reverse the ﬁrst swap, (cid:98)
z s , (cid:98) ¯
z s = swap ( (cid:98) z, (cid:98) ¯ z, j ) ; (vii) ﬁnally, one last decoding which should approximately recover the original input pair, such that:
[Equation]
If, after the two encode-swap-decode, we are able to recover the input images, regardless of which attribute we sample, this implies that swapping one attribute does not destroy latent information for other attributes.
As shown in Sec. 5, this can be seen as a data augmentation, growing the effective training set size by adding all possible new attribute combinations not already in the training set.
[Table] [Table]
[Table] [Table]
[Table] [Table]
Cycle reconstruction loss 9 L r ← || D ( E ( x ))
− x || l 1 # Standard reconstruction loss
4. T RAINING AND O PTIMIZATION
Algorithm 1 lists our sampling strategy and calculates loss terms, which we combine into a total loss
[Equation]
where L r , L sr and L csr , respectively are the reconstruction, swap-reconstruction, and cycle construction losses.
Scalar coefﬁcients λ sr , λ csr > 0 control the relative importance of the loss terms.
The total loss L can be minimized w.r.t.
parameters of encoder ( E ) and decoder ( D ) via gradient descent.
Q UALITATIVE E XPERIMENTS
We qualitatively evaluate our method on zero-shot synthesis tasks, and on its ability to learn disentangled representations, on existing datasets (Sec. 5.2), and on a dataset we contribute (Sec. 5.1).
GZS-Net architecture.
For all experiments, the encoder E is composed of two convolutional layers with stride 2, followed by 3 residual blocks, followed by a convolutional layer with stride 2, followed by reshaping the response map to a vector, and ﬁnally two fully-connected layers to output 100-dim vector as latent feature.
The decoder D mirrors the encoder, and is composed of two fully-connected layers, followed by reshape into cuboid, followed by de-conv layer with stride 2, followed byresidual blocks, then ﬁnally two de-conv layers with stride 2, to output a synthesized image.
5. F ONTS D ATASET & Z ERO - SHOT SYNTHESIS P ERFORMANCE
Design Choices.
Fonts is a computer-generated image datasets.
Each image is of an alphabet letter and is accompanied with its generating attributes: Letters (52 choices, of lower- and upper-case
English alphabet); size (3 choices); font colors (10); background colors (10); fonts (100); giving a total of 1.56 million images, each with size (128 × 128) pixels.
We propose this dataset to allow fast testing and idea iteration on zero-shot synthesis and disentangled representation learning.
Samples from the dataset are shown in Fig.
Details and source code are in the Appendix.
We partition the 100-d latents equally among the 5 attributes.
We use a train:test split of 75:25.
Baselines.
We train four baselines:
• The ﬁrst three are a standard Autoencoder, a β -VAE (Higgins et al., 2017), and β -TCVAE (Chen et al., 2018).
β -VAE and β -TCVAE show reasonable disentanglement on the dSprites dataset (Matthey et al., 2017).
Yet, they do not make explicit the assignment between latent variables and attributes, which would have been useful for precisely controlling the attributes (e.g. color, orientation) of synthesized images.
Therefore, for these methods, we designed a best-effort approach by exhaustively searching for the assignments.
Once assignments are known, swapping attributes between images might become possible with these VAEs and hopefully enabling for controllable-synthesis.
We denote these three baselines with this E xhaustive S earch, using sufﬁx + ES .
Details on Exhaustive Search are in the Appendix.
shown in supervision
5. Z ERO - SHOT SYNTHESIS ON ILAB -20M AND R A FD
iLab-20M (Borji et al., 2016): is an attributed dataset containing images of toy vehicles placed on a turntable using 11 cameras at different viewing points.
There are 3 attribute classes: vehicle identity: 15 categories, each having 25-160 instances; pose; and backgrounds: over 14 for each identity: projecting vehicles in relevant contexts.
Further details are in the Appendix.
We partition the 100-d latent space among attributes as: 60 for identity, 20 for pose and 20 for background.
iLab-20M has limited attribute combinations (identity shows only in relevant background; e g., cars on roads but not in deserts), GZS-Net can disentangle these three attributes and reconstruct novel combinations (e.g., cars on desert backgrounds) Fig.
5 shows qualitative generation results.
We compare against (AE+DS), conﬁrming that maintains discriminative information, and against two state-of-the-art GAN baselines:
starGAN (Choi et al., 2018) and ELEGANT (Xiao et al., 2018).
GAN baselines are strong in knowing what to change but not necessarily how to change it : Where change is required, pixels are locally perturbed (within a patch) but the perturbations often lack global correctness (on the image).
See Appendix for further details and experiments on these GAN methods.
RaFD (Radboud Faces Database, Langner et al., 2010): contains pictures of 67 models displaying 8 emotional expressions taken by 5 different camera angles simultaneously.
There are 3 attributes: identity, camera position (pose), and expression.
We partition the 100-d latent space among the attributes as 60 for identity, 20 for pose, and 20 for expression.
We use a 80:20 split for train:test, and use GZS-Net to synthesize images with novel combination of attributes (Fig. 6).
The synthesized images can capture the corresponding attributes well, especially for pose and identity.
Q UANTITATIVE E XPERIMENTS 6. Q UANTIFYING D ISENTANGLEMENT THROUGH ATTRIBUTE CO - PREDICTION
Can latent features of one attribute predict the attribute value?
Can it also predict values for other attributes?
Under perfect disentanglement, we should answer always for the ﬁrst and never for the second.
Our network did not receive attribute information through supervision, but rather, through swapping.
We quantitatively assess disentanglement by calculating a model-based confusion matrix between attributes: We analyze models trained on the Fonts dataset.
We take the Test examples from
Font, and split them 80:20 for train DR :test DR .
For each attribute pair j, r ∈ [1 ..m ] × [1 ..m ] , we train a classiﬁer (3 layer MLP) from g j of train DR to the attribute values of r , then obtain the accuracy of each attribute by testing with g j of test DR .
Table 1 compares how well features of each attribute (row) can predict an attribute value (column): perfect should be as close as possible to Identity matrix, with off-diagonal entries close to random (i.e., 1 /
|A
r | ).
GZS-Net outperforms other methods, except for (AE + DS) as its latent space was Directly Supervised for this particular task, though it shows inferior synthesis performance.
6. D ISTANCE OF SYNTHESIZED IMAGE TO GROUND TRUTH
The construction of the Fonts dataset allows programmatic calculating ground-truth images corresponding to synthesized images (recall, Fig. 4).
We measure how well do our generated images compare to the ground-truth test images.
Table 2 shows image similarity metrics, averaged over the test set, comparing our method against baselines.
Our method signiﬁcantly outperforms baselines
6. GZS-N ET B OOST O BJECT R ECOGNITION
We show-case that our zero-shot synthesised images by GZS-Net can augment and boost training of a visual object recognition classiﬁer (Ge et al., 2020).
Two different training datasets (Fig. 7a) are tailored from iLab-20M, pose and background unbalanced datasets ( D UB ) (half classes withposes per object instance, other half with only 2 poses; as we cut poses, some backgrounds are also
[Table]
[Table]
eliminated), as well as pose and background balanced dataset ( D B ) (all classes with all 6 poses per object instance).
We use GZS-Net to synthesize the missing images of D UB and synthesize a new (augmented) balanced dataset D B-s .
We alternatively use common data augmentation methods (random crop, horizontal ﬂip, scale resize, etc) to augment the D UB dataset to the same number of images as D B-s , called D UB-a .
We show object recognition performance on the test set using these four datasets respectively.
Comparing
D B-s with D UB shows ≈ 7% points improvements on classiﬁcation performance, due to augmentation with synthesized images for missing poses in the training set, reaching the level of when all real poses are available ( D B ).
Our synthesized poses outperform traditional data augmentation ( D UB-a )
C ONCLUSION
We propose a new learning framework, Group Supervised Learning (GSL), which admits datasets of examples and their semantic relationships.
It provides a learner groups of semantically-related samples, which we show is powerful for zero-shot synthesis.
In particular, our Group-supervised Zero-Shot synthesis network (GZS-Net) is capable of training on groups of examples, and can learn disentangled representations by explicitly swapping latent features across training examples, along edges suggested by GSL.
We show that, to synthesize samples given a query with custom attributes, it is sufﬁcient to ﬁnd one example per requested attribute and to combine them in the latent space.
We hope that researchers ﬁnd our learning framework useful and extend it for their applications.
A CKNOWLEDGMENTS
This work was supported by C-BRIC (one of six centers in JUMP, a Semiconductor Research
Corporation (SRC) program sponsored by DARPA), the Army Research Ofﬁce (W911NF2020053), and the Intel and CISCO Corporations.
The authors afﬁrm that the views expressed herein are solely their own, and do not represent the views of the United States government or any agency thereof.
B B ASELINES B. E XHAUSTIVE S EARCH (ES) AFTER TRAINING A UTO -E NCODER BASED METHODS
After training the baselines: standard Autoencoder, a β -VAE (Higgins et al., 2017), and TC-VAE (Chen et al., 2018).
We want to search for the assignment between latent variables and attributes, as these VAEs do not make explicit the assignment.
This knowing the assignment should hypothetically allow us to trade attributes between two images by swapping feature values belonging to the attribute we desire to swap.
To discover the assignment from latent dimension to attribute, we map all n training images through the encoder, giving a 100D vector per training sample ∈ R n × 100 .
We make an 80: split on the vectors, obtaining X train ES ∈ R 0 .
8 n × 100 and X test ES ∈ R 0 .
2 n × 100 .
Then, we randomly sample K different partitionings P of the 100D space evenly among the 5 attributes.
For each partitioning p ∈ P , we create 5 classiﬁcation tasks, one task per attribute, according to p : (cid:8)(cid:0)
X train ES [: , p j ] ∈ R 0 .
8 n × 20 , X test ES [: , p j ] ∈ R 0 .
2 n × 20 (cid:1)(cid:9) 5 j =1 .
For each task j , we train a 3-layer MLP to map X train ES [: , p j ] to their known attribute values and measure its performance on X test ES [: , p j ] .
Finally, we commit to the partitioning p ∈ P with highest average performance on the 5 attribute tasks.
This p represents our best effort to determine which latent feature dimensions correspond to which attributes.
For zero-shot synthesis with baselines, we swap latent dimensions indicated by partitioning p .
We denote three baselines with this E xhaustive S earch, using sufﬁx + ES (Fig. 4).
B. D IRECT S UPERVISION (DS) ON A UTO - ENCODER LATENT SPACE
The last baseline (AE+DS) directly uses attribute labels to supervise the latent disentangled representation of the auto-encoder by adding auxiliary classiﬁcation modules.
Speciﬁcally, the encoder maps an image sample x ( i ) to a 100-d latent vector z ( i ) =
E ( x ( i ) ) , equally divided into 5 partitions corresponding to 5 attributes: z ( i ) =
[ g ( i ) 1 , g ( i ) 2 , . .
, g ( i ) 5 ] .
Each attribute partition has a attribute label, [ y ( i ) 1 , y
( i ) 2 , . . .
, y ( i ) 5 ] , which represent the attribute value (e.g. for font color attribute, the label represent different colors: red, green, blue,.etc).
We use 5 auxiliary classiﬁcation modules to predict the corresponding class label given each latent attribute partitions as input.
We use Cross Entropy loss as the classiﬁcation loss and the training goal is to minimize both the reconstruction loss and classiﬁcation loss.
After training, we have assignment between latent variables and attributes, so we can achieve attribute swapping and controlled synthesis (Fig.
4 (AE+DS)).
The inferior synthesis performance demonstrates that: The supervision (classiﬁcation task) preserves discriminative information that is insufﬁcient for photo-realistic generation.
While our GZS-Net uses one attribute swap and cross swap which enforce disentangled information to be sufﬁcient for photo-realistic synthesis.
B. ELEGANT (X IAO ET AL ., 2018)
We utilize the author’s open-sourced code: https://github.com/Prinsphield/ELEGANT .
For ELEGANT and starGAN (Section B.4), we want to synthesis a target image has same identity as id provider image , same background as background provider image , and same pose as pose provider image .
To achieve this, we want to change the background and pose attribute of id image.
Although ELEGANT is strong in making image transformations that are local to relatively-small neighborhoods, however, it does not work well for our datasets, where image-wide transformations are required for meaningful synthesis.
This can be conﬁrmed by their model design: their ﬁnal output is a pixel-wise addition of a residual map, plus the input image.
Further, ELEGANT treats all attribute values as binary: they represent each attribute value in a different part of the latent space, whereas our method devotes part of the latent space to represents all values for an attribute .
For investigation, we train dozens of ELEGANT models with different hyperparameters, detailed as:
• For iLab-20M, the pose and background contain a total of 117 attribute values (6 for pose,for background).
As such, we tried training it on all attribute values (dividing their latent space among 117 attribute values).
We note that this training regime was too slow and the loss values do not seem to change much during training, even with various learning rate choices (listed below)
• To reduce the difﬁculty of the task for ELEGANT, we ran further experiments restricting attribute variation to only 17 attribute values (6 for pose, 11 for background) and this shows more qualitative promise than 117 attributes.
This is what we report.
• Fig 10 shows that ELEGANT ﬁnds more challenge in changing the pose than in changing the background.
We now explain how we generated Columns 3 and 4 of Fig 10 for modifying the background.
We modify the latent features for the identity image before decoding.
Since the Identity input image and the Background input image have known but different background values, their background latent features are represented in two different latent spaces.
One can swap on one or on both of these latent spaces.
Column 3 and 4 of Fig.10 swap only on one latent space.
However, in Fig.
5 of the main paper, we swap on both positions.
We also show swapping only the pose attribute (across 2 latent spaces) in Column 1 of Fig.10 and swapping both pose and background in Column 2.
• To investigate if the model’s performance is due to poor convergence of the generator, we qualitatively assess its performance on the training set .
Fig.
11 shows output of ELEGANT on training samples.
We see that the reconstruction (right) of input images (left) shows decent quality, suggesting that the generator network has converged to decently good parameters.
Nonetheless, we see artefacts in its outputs when amending attributes, particularly located in pixel locations where a change is required.
This shows that the model setup of ELEGANT is aware that these pixel values need to be updated, but the actual change is not coherent across the image.
• For the above, we applied a generous sweep of training hyperparameters, including: – Learning rate : author’s original is 2e-4, we tried several values between 1e-5 and 1e-3, including different rates for generator and discriminator.
– Objective term coefﬁcients : There are multiple loss terms for the generator, adversarial loss and reconstruction loss.
We used a grid search method by multiplying the original parameters by a number from [0.2, 0.5, 2, 5] for each of the loss terms and tried several combinations.
– The update frequency of weights on generator (G) and discriminator (D).
Since D is easier to learn, we performing k update steps on G for every update step on D. We tried k = 5 , 10 , 15 , 20 , 30 , 40 , 50 .
We report ELEGANT results showing best qualitative performance.
Overall, ELEGANT does not work well for holistic image manipulation (though works well for local image edits , per experiments by authors (Xiao et al., 2018)).
B. STAR GAN (C HOI ET AL ., 2018)
We utilize the author’s open-sourced code: https://github.com/yunjey/stargan .
Unlike
ELEGANT (Xiao et al., 2018) and our method, starGAN only accepts one input image and an edit information: the edit information, is not extracted from another image – this is following their method and published code.
C Z ERO - SHOT SYNTHESIS P ERFORMANCE ON D S PRITES DATASET
We qualitatively evaluate our method, Group-Supervised Zero-Shot Synthesis Network (GZS-Net), against three baseline methods, on zero-shot synthesis tasks on the dSprites dataset.
C. D S PRITES
dSprites (Matthey et al., 2017) is a dataset of 2D shapes procedurally generated from 6 ground truth independent latent factors.
These factors are color, shape, scale, rotation, x- and y-positions of a sprite.
All possible combinations of these latents are present exactly once, generating 737280 total images.
Latent factor values (Color: white; Shape: square, ellipse, heart; Scale: 6 values linearly spaced in [0.5, 1];
Orientation: 40 values in [0, 2 pi]; Position X: 32 values in [0, 1];
Y:values in [0, 1])
C. E XPERIMENTS OF B ASELINES AND GZS-N ET
We train a 10-dimensional latent space and partition the it equally among the 5 attributes: 2 for shape, 2 for scale, 2 for orientation, 2 for position X , and 2 for position Y .
We use a train:test split of 75:25.
We train 3 baselines: a standard Autoencoder, a β -VAE (Higgins et al., 2017), and TC-VAE (Chen et al., 2018).
To recover the latent-to-attribute assignment for these baselines, we utilize the Exhaustive
Search best-effort strategy, described in the main paper: the only difference is that we change the dimension of Z space from 100 to 10.
Once assignments are known, we utilize these baseline VAEs by attribute swapping to do controlled synthesis.
We denote these baselines using sufﬁx + ES .
As is shown in Figure 2, GZS-Net can precisely synthesize zero-shot images with new combinations of attributes, producing images similar to the groud truth.
The baselines β -VAE and TC-VAE produce realistic images of good visual quality, however, not satisfying the requested query: therefore, they cannot do controllable synthesis even when equipped with our best-effort Exhaustive Search to discover the disentanglement.
Standard auto-encoders can not synthesis meaningful images when combining latents from different examples, giving images outside the distribution of training samples (e.g. showing multiple sprites per image).