A DVERSARIAL M IXUP R ESYNTHESIZERS
A BSTRACT
In this paper, we explore new approaches to combining information encoded within the learned representations of autoencoders.
We explore models that are capable of combining the attributes of multiple inputs such that a resynthesised output is trained to fool an adversarial discriminator for real versus synthesised data.
Furthermore, we explore the use of such an architecture in the context of semi-supervised learning, where we learn a mixing function whose objective is to produce interpolations of hidden states, or masked combinations of latent representations that are consistent with a conditioned class label.
We show quantitative and qualitative evidence that such a formulation is an interesting avenue of research.
I NTRODUCTION
The autoencoder is a fundamental building block in unsupervised learning.
Autoencoders are trained to reconstruct their inputs after being processed by two neural networks: an encoder which encodes the input to a high-level representation or bottleneck , and a decoder which performs the reconstruction using the representation as input.
One primary goal of the autoencoder is to learn representations of the input data which are useful (Bengio, 2012), which may help in downstream tasks such as classiﬁcation (Zhang et al., 2017;
Hsu et al., 2019) or reinforcement learning (van den Oord et al., 2017;
Ha & Schmidhuber, 2018).
The representations of autoencoders can be encouraged to contain more ‘useful’ information by restricting the size of the bottleneck, through the use of input noise (e.g., in denoising autoencoders, Vincent et al., 2008), through regularisation of the encoder function (Rifai et al., 2011), or by introducing a prior (Kingma & Welling, 2013).
Another goal is in learning interpretable representations (Chen et al., 2016; Jang et al., 2016).
In unsupervised learning, learning often involves qualitative objectives on the representation itself, such as disentanglement of latent variables (Liu et al., 2017; Thomas et al., 2017) or maximisation of mutual information (Chen et al., 2016; Belghazi et al., 2018; Hjelm et al., 2019).
Mixup (Zhang et al., 2018) and manifold mixup (Verma et al., 2018) are regularisation techniques that encourage deep neural networks to behave linearly between two data samples.
These methods artiﬁcially augment the training set by producing random convex combinations between pairs of examples and their corresponding labels and training the network on these combinations.
This has the effect of creating smoother decision boundaries, which can have a positive effect on generalisation performance.
In Verma et al.
(2018), the random convex combinations are computed in the hidden space of the network.
This procedure can be viewed as using the high-level representation of the network to produce novel training examples and provides improvements over strong baselines in the supervised learning.
Furthermore, Verma et al.
(2019) propose a simple and efﬁcient method for semi-supervised classiﬁcation based on random convex combinations between unlabeled samples and their predicted labels.
In this paper we explore the use of a wider class of mixing functions for unsupervised learning, mixing in the bottleneck layer of an autoencoder.
These mixing functions could consist of continuous interpolations between latent vectors such as in Verma et al.
(2018), to binary masking operations to even a deep neural network which learns the mixing operation.
In order to ensure that the output of the decoder given the mixed representation resembles the data distribution at the pixel level, we leverage adversarial learning (Goodfellow et al., 2014), where here we train a discriminator to distinguish between decoded mixed and unmixed representations.
This technique affords a model the ability to simulate novel data points (such as those corresponding to combinations of annotations not present in the training set).
Furthermore, we explore our approach in the context of semi-supervised learning, where we learn a mixing function whose objective is to produce interpolations of hidden states consistent with a conditioned class label.
1. R ELATED WORK
Our method can be thought of as an extension of autoencoders that allows for sampling through mixing operations, such as continuous interpolations and masking operations.
Variational autoencoders (VAEs, Kingma & Welling, 2013) can also be thought of as a similar extension of autoencoders, using the outputs of the encoder as parameters for an approximate posterior q ( z | x ) which is matched to a prior distribution p ( z ) through the evidence lower bound objective (ELBO).
At test time, new data points are sampled by passing samples from the prior, z ∼ p ( z ) , through the decoder.
In contrast, our we sample a random mixup operation between the representations of two inputs from the encoder.
The Adversarially Constrained Autoencoder Interpolation (ACAI) method is another approach which involves sampling interpolations as part of an unsupervised objective (Berthelot* et al., 2019).
ACAI uses a discriminator network to predict the mixing coefﬁcient from the decoder output of the mixed representation, and the autoencoder tries to ‘fool’ the discriminator, making interpolated points indistinguishable from real ones.
The GAIA algorithm (Sainburg et al., 2018) uses a BEGAN framework with an additional interpolation-based adversarial objective.
What primarily differentiates our work from theirs is that we perform an exploration into different kinds of mixing functions, including a semi-supervised variant which uses an MLP to produce mixes consistent with a class label.
F ORMULATION
Let us consider an autoencoder model F ( · ) , with the encoder part denoted as f ( · ) and the decoder g ( · ) .
In an autoencoder we wish to minimise the reconstruction, which is simply:
[Equation]
Because autoencoders trained by input-reconstruction loss tend to produce images which are slightly blurry, one can train an adversarial autoencoder (Makhzani et al., 2016), but instead of putting the adversary on the bottleneck, we put it on the reconstruction, and the discriminator (denoted D ) tries to distinguish between real and reconstructed x , and the autoencoder (which is analogous to the generator) tries to construct ‘realistic’ reconstructions so as to fool the discriminator.
Because of this, we coin the term ‘ARAE’ (adversarial reconstruction autoencoder).
This can be written as:
[Equation]
where (cid:96) GAN is a GAN-speciﬁc loss function.
In our case, (cid:96) GAN is the binary cross-entropy loss, which corresponds to the Jenson-Shannon GAN (Goodfellow et al., 2014).
One way to use the autoencoder to generate novel samples would be to encode two inputs h 1 = f ( x 1 ) and h 2 = f ( x 2 ) into their latent representation, perform some combination between them, and then run the result through the decoder g ( · ) .
There are many ways one could combine the two latent representations, and we denote this function Mix ( h 1 , h 2 ) .
Manifold mixup (Verma et al., 2018) implements mixing in the hidden space through convex combinations:
[Equation]
where λ ∈ [0 , 1] ( bs, ) is sampled from a Uniform (0 , 1) distribution and bs denotes the minibatch size.
In contrast, here we explore a strategy in which we randomly retain some components of the hidden representation from h 1 and use the rest from h 2 , and in this case we would randomly sample a binary mask m ∈ { 0 , 1 } ( bs × f ) (where f denotes the number of feature maps) and perform the following operation:
[Equation]
where m is sampled from a Bernoulli ( p ) distribution ( p can simply be sampled uniformly).
With this in mind, we propose the adversarial mixup resynthesiser (AMR), where part of the autoencoder’s objective is to produce mixes which, when decoded, are indistinguishable from real images.
The generator and the discriminator of AMR are trained by the following mixture of loss components:
[Equation]
Note that the mixing consistency loss is simply the reconstruction between the mix ˜ h mix =
Mix ( f ( x ) , f ( x (cid:48) )) and the re-encoding of it f
( g (˜ h mix )) , where x and x (cid:48) are two randomly sampled images from the training set.
This may be necessary as without it the decoder may simply output an image which is not semantically consistent with the two images which were mixed (refer to Section 5.2 for an in-depth explanation and analysis of this loss).
Both the generator and discriminator are trained by the decoded image of the mix g ( Mix ( f ( x ) , f
( x (cid:48) ))) .
The discriminator D is trained to label it as a fake image by minimising its probability and the generator F is trained to fool the discriminator by maximising its probability.
Note that the coefﬁcient λ controls the reconstruction and the coefﬁcient β controls the mixing consistency in the generator.
See Figure 1 for a visualisation of the AMR model.
2. U SING LABELS
While it is interesting to generate new examples via random mixing strategies in the hidden states, we also explore a supervised mixing formulation in which we learn a mixing function that can produce mixes between two examples such that they are consistent with a particular class label .
We make this possible by backpropagating through a classiﬁer network p ( y | x ) which branches off the end of the discriminator, i.e., an auxiliary classiﬁer GAN (Odena et al., 2017).
Let us assume that for some image x , we have a set of binary attributes y associated with it, where y ∈ { 0 , 1 } k (and k ≥ 1 ).
We introduce a mixing function Mix sup ( h 1 , h 2 , y ) , which is an MLP that maps y to Bernoulli parameters p ∈ [0 , 1] bs × f .
These parameters are used to sample a Bernoulli mask m ∼ Bernoulli ( p ) to produce a new combination ˜ h mix =
mh 1 + (1 − m ) h 2 , which is consistent with the class label y .
Note that the conditioning class label should be semantically meaningful with respect to both of the conditioned hidden states.
For example, if we’re producing mixes based on the gender attribute and both h 1 and h 2 are male, it would not make sense to condition on the ‘female’ label.
To enforce this constraint, we simply make the conditioning label a convex combination ˜ y mix =
α y 1 + (1 − α )
y 2 as well, using α ∼ Uniform (0 , 1) .
To make this more concrete, the autoencoder and discriminator, in addition to their losses described in Equation 5, try to minimise the following losses:
[Equation]
where ˜ y mix =
α y 1 + (1 − α )
y 2 and ˜ h mix =
Mix sup ( f ( x 1 ) , f ( x 2 ) , ˜ y mix ) (6)
Note that for the consistency loss the same coefﬁcient β is used.
See Figure 2 for a visualisation of the supervised AMR model.
R ESULTS
We use ResNets (He et al., 2016) for both the generator and discriminator.
The precise architectures for generator and discriminator can be found here.
The datasets evaluated on are:
• UT Zappos50K (Yu & Grauman, 2014; 2017): a large dataset comprising 50k images of shoes, sandals, slippers, and boots.
Each shoe is centered on a white background and in the same orientation, which makes it convenient for generative modelling purposes.
• CelebA
(Liu et al., 2015): a large-scale and highly diverse face dataset consisting of 200K images.
We use the aligned and cropped version of the dataset downscaled to 64px, and only consider (via the use of a keypoint-based heuristic) frontal faces.
It is worth noting that despite this, there is still quite a bit of variation in terms of the size and position of the faces, which can make mixing between faces a more difﬁcult task since the faces are not completely aligned.
[Figure]
[Figure]
As seen in Figures 3 and all of the mixup variants produce more realistic-looking interpolations than in pixel space.
Due to background details in CelebA however, it is slightly harder to distinguish the quality between the different methods.
Though this may not be the most ideal metric to use in our case (see discussion at end of this section) we use the Frechet Inception Distance (FID) by Heusel et al.
(2017), which is based on features extracted from a pre-trained CelebA classiﬁer, to compute the distance between samples from the dataset and ones from our autoencoders 2 .
Concretely, we compute (on the validation set) two scores: the FID between validation samples and their [Figure] [Figure] reconstructions (denoted in the table as the FID between validation samples and randomly sampled interpolations (denoted in the table as FID(data, mix) ).
In the latter case, we repeat this ﬁve times (over ﬁve different sets of randomly sampled interpolations) for three different random seeds, resulting in 5 × 3 = 15 FID scores from which we compute the mean and standard deviation.
These results are shown in Tables 1 and 2 for the mixup and Bernoulli mixup formulations, respectively. [Table] [Table]
Lower FID is usually considered to be better.
However, FID may not be the most appropriate metric to use in our case.
Because the FID is a measure of distance between two distributions, one can simply obtain a very low FID by simplying autoencoding the data, as shown in Tables 1 and 2.
In the case of mixing, one situation which may favour a lower FID is if g ( αf ( x 1 ) + (1 − α ) f
( x 2 ))
≈ g
( f ( x 1 )) or g ( f ( x 2 )) ; in other words, the supposed mix simply decodes into one of the original examples x 1 or x 2 , which clearly lie on the data manifold.
To avoid having the mixed features α x 1 + (1 − α )
x 2 being decoded back into samples which lie on the data manifold, we leverage the consistency loss, which is tuned by coefﬁcient β .
The lower the coefﬁcient, the more likely that decoded mixes are projected back onto the manifold, but if this constraint is too weak then it may not necessarily be desirable if one wants to create novel data points.
(For more details, see Section 5.2 in the appendix.)
Despite potential shortcomings of using FID, it seems reasonable to use such a metric to compare against baselines without any mixing losses, such as the adversarial reconstruction autoencoder (ARAE), which we indeed outperform for both mixup and Bernoulli mixup.
For Bernoulli mixup, the FID scores appear to be higher than those in the mixup case (Table 1) because the sampled
Bernoulli mask m is also across the channel axis, i.e., it is of the shape ( bs, f ) , whereas in mixup α has the shape ( bs, ) .
Because the mixing is performed on an extra axis, this produces a greater degree of variability in the mixes, and we have observed similar FID scores to the Bernoulli mixup case by evaluating on a variant of mixup where the α has the shape ( bs, f ) instead of ( bs, ) .
3. S UPERVISED C ELEB A
We present some qualitative results with the supervised formulation.
We train our supervised AMR variant using a subset of the attributes in CelebA (‘is male’, ‘is wearing heavy makeup’, and ‘is wearing lipstick’).
We consider pairs of examples { ( x 1 , y 1 ) , ( x 2 , y 2 ) }
(where one example is male and the other female) and produce random convex combinations of the attributes ˜ y mix =
α y 1 +(1 − α )
y 2 and decode their resulting mixes
Mix sup ( f ( x 1 ) , f ( x 2 ) , ˜ y mix ) .
This can be seen in Figure 6.
[Figure]
We can see that for the most part, the class mixer function has been able to produce decent mixes between the two faces consistent with the desired attributes.
There are some issues – namely, the model does not seem to disentangle the lipstick and makeup attributes well – but this may be due to the strong correlation between lipstick and makeup (lipstick is makeup!), or be in part due to the classiﬁcation performance of the auxiliary classiﬁer part of the discriminator (while its accuracy on both training and validation was as high as 95%, there may still be room for improvement).
We also achieved better results by simply having the embedding function produce a mask m ∈ [0 , 1] rather than { 0 , 1 } , most likely because such a formulation allows a greater degree of ﬂexibility when it comes to mixing.
Indeed, one priority is to conduct further hyperparameter tuning in order to improve these results.
For a visualisation of the Bernoulli parameters output by the embedding function, see Section 5.3 in the appendix.
C ONCLUSION
In this paper, we proposed the adversarial mixup resynthesiser and showed that it can be used to produce realistic-looking combinations of examples by performing mixing in the bottleneck of an autoencoder.
We proposed several mixing functions, including one based on sampling from a uniform distribution and the other a Bernoulli distribution.
Furthermore, we presented a semisupervised version of the Bernoulli variant in which one can leverage class labels to learn a mixing function which can determine what parts of the latent code should be mixed to produce an image consistent with a desired class label.
While our technique can be used to leverage an autoencoder as a generative model, we conjecture that our technique may have positive effects on the latent representation and therefore downstream tasks, though this is yet to be substantiated.
Future work will involve more comparisons to existing literature and experiments to determine the effects of mixing on the latent space itself and downstream tasks.
A PPENDIX 5. E XPERIMENTAL CONFIGURATION
We will provide a summary of our experimental setup here, though we also provide links to (and encourage viewers to look at) various parts of the code such as the networks used for the generator and discriminator and the optimiser hyperparameters.
We use a residual network for both the generator and discriminator.
The discriminator uses spectral normalisation (Miyato et al., 2018), with ﬁve discriminator updates being performed for each generator update.
We use ADAM for our optimiser with α = 2 e − 4 , β 1 = 0 .
and β 2 = 0 .
5. C ONSISTENCY LOSS
In order to examine the effect of the consistency loss, we explore a simple two-dimensional spiral dataset, where points along the spiral are deemed to be part of the data distribution and points outside it are not.
With the mixup loss enabled and λ = 10 , we try values of β ∈ { 0 , 0 . 1 , 10 , 100 } .
After 100 epochs of training, we produce decoded random mixes and plot them over the data distribution, which are shown as orange points (overlaid on top of real samples, shown in blue).
This is shown in
Figure 7.
As we can see, the lower β is, the more likely interpolated points will lie within the data manifold (i.e. the spiral).
This is because the consistency loss competes with the discriminator loss – as β is decreased, there is a relatively greater incentive for the autoencoder to try and fool the discriminator with interpolations, forcing it to decode interpolated points such that they lie in the spiral.
Ideally however we would want a bit of both: we want high consistency so that interpolations in hidden states are semantically meaningful (and do not decode into some other random data point), while also having those decoded interpolations look realistic.
We also compare our formulation to ACAI (Berthelot* et al., 2019), which does not explicitly have a consistency loss term.
Instead, the discriminator tries to predict what the mixing coefﬁcient α is, and the autoencoder tries to fool it into thinking interpolations have a coefﬁcient of 0.
In Figure 8 we compare this to our formulation in which β
= 0 .
This is shown in Figure 8a
(right ﬁgure).
It appears that ACAI also prefers to place points in the spiral, although not as strongly as AMR with β =
(though this may be because ACAI needs to trained for longer – ACAI and AMR were trained for the same number of epochs).
In Figure 8b we can see that over the course of training the consistency losses for both ACAI and AMR gradually rise, indicating both models’ preference for moving interpolated points closer to the data manifold.
Note that here we are only observing the consistency loss during training, and it is not used in the generator’s loss.
Lastly, in Figure 9 we show some side-by-side comparisons of our model interpolating between faces when β = 50 and β = 0 .
We can see that when β = 0 interpolations between faces are not as smooth in terms of colour and lighting.
This somewhat slight discontinuity in the interpolation may be explained by the decoder pushing these interpolated points closer to the data manifold, since there is no consistency loss enforced.
[Figure]
[Figure]
5. B ERNOULLI PARAMETERS FROM SUPERVISED AMR
To recap, the class mixer in the supervised formulation internally maps from a label ˜
y mix to Bernoulli parameters p ∈
[0 , 1] K , from which a Bernoulli mask m ∼ Bernoulli ( p ) is sampled.
The resulting Bernoulli parameters p are shown in Figure 10, where each row denotes some combination of attributes y ∈ { 000 , 001 , 010 , . . . } and the columns denote the index of p (spread out across four images, such that the ﬁrst image denotes p 1:128 , second image p 128:256 , etc.).
We can see that each attribute combination spells out a binary combination of feature maps, which allows one to easily glean which feature maps contribute to which attributes.
5. A DDITIONAL SAMPLES
In this section we show additional samples of the AMR model (using mixup and Bernoulli mixup variants) on Zappos and CelebA datasets.
We compare AMR against linear interpolation in pixel space (pixel), adversarial reconstruction autoencoder (ARAE), and adversarialy contrained autoencoder interpolation (ACAI).
As can be observed in the following images, the interpolations of pixel and ARAE are less realistic and suffer from more artifacts.
AMR and ACAI produce more realisticlooking results, while AMR generates a smoother transition between the two samples.
• Figure 11: AMR on Zappos (mixup) • Figure 12: AMR on Zappos (Bernoulli mixup) • Figure 13: AMR on CelebA (mixup) • Figure 14: AMR on CelebA (Bernoulli mixup) • Figure 15: AMR on CelebA-HQ (Bernoulli mixup) • Figure 16: AMR on Zappos-HQ (Bernoulli mixup)
[Figure]
[Figure] [Figure]
[Figure] [Figure]