G ROUP E QUIVARIANT G ENERATIVE A DVERSARIAL N ETWORKS
A BSTRACT
Recent improvements in generative adversarial visual synthesis incorporate real and fake image transformation in a self-supervised setting, leading to increased stability and perceptual ﬁdelity.
However, these approaches typically involve image augmentations via additional regularizers in the GAN objective and thus spend valuable network capacity towards approximating transformation equivariance instead of their desired task.
In this work, we explicitly incorporate inductive sym - metry priors into the network architectures via group-equivariant convolutional networks
Group-convolutions have higher expressive power with fewer sam - ples and lead to better gradient feedback between generator and discriminator.
We show that group-equivariance integrates seamlessly with recent techniques for
GAN training across regularizers, architectures and loss functions.
We demon- strate the utility of our methods for conditional synthesis by improving generation in the limited data regime across symmetric imaging datasets and even ﬁnd bene ﬁts for natural images with preferred orientation.
I NTRODUCTION
Generative visual modeling is an area of active research, time and again ﬁnding diverse and creative applications.
A prevailing approach is the generative adversarial network (GAN), wherein density estimation is implicitly approximated by a min-max game between two neural networks (Goodfellow et al., 2014).
Recent GANs are capable of high-quality natural image synthesis and scale dramatically with increases in data and compute (Brock et al., 2018).
However, GANs are prone to instability due to the difﬁculty of achieving a local equilibrium between the two networks.
Frequent failures include one or both networks diverging or the generator only capturing a few modes of the empirical distribution.
Several proposed remedies include modifying training objectives (Arjovsky et al., 2017; Jolicoeur-Martineau, 2018), hierarchical methods (Karras et al., 2017), instance selection (Sinha et al., 2019; 2020), latent optimization (Wu et al., 2019), and strongly regularizing one or both networks (Gulrajani et al., 2017; Miyato et al., 2018; Dieng et al., 2019), among others.
In practice, one or all of the above techniques are ultimately adapted to speciﬁc use cases.
Further, limits on data quantity empirically exacerbate training stability issues more often due to discriminator overﬁtting.
Recent work on GANs for small sample sizes can be roughly divided into transfer learning approaches (Wang et al., 2018; Noguchi & Harada, 2019; Mo et al., 2020;
Zhao et al., 2020a) or methods which transform/augment the available training data and provide the discriminator with auxiliary tasks.
For example, Chen et al.
(2019) propose a multi-task discriminator which additionally predicts the degree by which an input image has been rotated, whereas Zhang et al.
Zhao et al.
(2020c) incorporate consistency regularization where the discriminator is penalized towards similar activations for transformed/augmented real and fake images.
However, with consistency regularization and augmentation, network capacity is spent learning equivariance to transformation as opposed to the desired task and equivariance is not guaranteed.
In this work, we consider the problem of training tabula rasa on limited data which possess global and even local symmetries.
We begin by noting that GANs ubiquitously use convolutional layers which exploit the approximate translation invariance and equivariance of image labels and distributions, respectively.
Equivariance to geometric transformations is key to understanding image representations (Bietti & Mairal, 2019).
Unfortunately, other symmetries (e.g., rotations and reﬂections) inherent to modalities such as astronomy and medical imaging where galaxies and cells can be in arbitrary orientations are not accounted for by standard convolutional layers.
To this end,
Cohen & Welling (2016) proposed a group-theoretic generalization of convolutional layers (groupconvolutions) which in addition to translation, exploit other inherent symmetries and increase the expressive capacity of a network thereby increasing its sample efﬁciency signiﬁcantly in detection (Winkels & Cohen, 2019), classiﬁcation (Veeling et al., 2018), and segmentation (Chidester et al., 2019).
Importantly, equivariant networks outperform standard CNNs trained with augmentations from the corresponding group (Veeling et al., 2018, Table 1), (Lafarge et al., 2020a, Fig. 7).
See
Cohen et al.
; Esteves (2020) for a formal treatment of equivariant CNNs.
Equivariant features may also be constructed via scattering networks consisting of non-trainable
Wavelet ﬁlters, enabling equivariance to diverse symmetries (Mallat, 2012; Bruna & Mallat, 2013;
Sifre & Mallat, 2013).
Generative scattering networks include Angles & Mallat (2018) where a standard convolutional decoder is optimized to reconstruct images from an embedding generated by a ﬁxed scattering network and Oyallon et al.
(2019) who show preliminary results using a standard convolutional GAN to generate scattering coefﬁcients.
We note that while both approaches are promising, they currently yield suboptimal synthesis results not comparable to modern GANs.
Capsule networks (Hinton et al., 2011; Sabour et al., 2017) are also equivariant and emerging work has shown that using a capsule network for the GAN discriminator (Jaiswal et al., 2019; Upadhyay &
Schrater, 2018) improves synthesis on toy datasets.
However, capsule GANs and generative scattering approaches require complex training strategies, restrictive architectural choices not compatible with recent insights in GAN training, and have not yet been shown to scale to real-world datasets.
In this work, we improve the generative modeling of images with transformation invariant labels by using an inductive bias of symmetry.
We replace all convolutions with group-convolutions thereby admitting a higher degree of weight sharing which enables increased visual ﬁdelity, especially with limited-sample datasets.
To our knowledge, we are the ﬁrst to use group-equivariant layers in the
GAN context and to use symmetry-driven considerations in both generator and discriminator architectures.
Our contributions are as follows,
We introduce symmetry priors via group-equivariance to generative adversarial networks.
We show that recent insights in improving GAN training are fully compatible with groupequivariance with careful reformulations.
We improve class-conditional image synthesis across a diversity of datasets, architectures, loss functions, and regularizations.
These improvements are consistent for both symmetric images and even natural images with preferred orientation.
M ETHODS 2. P RELIMINARIES
Groups and group-convolutions.
A group is a set with an endowed binary function satisfying the properties of closure, associativity, identity, and invertibility.
A two-dimensional symmetry group is
[Figure]
the set of all transformations under which a geometric object is invariant with an endowed operation of composition.
Given a group G and a map Φ : X → Y between two G -sets X and Y , Φ is said to be equivariant i.f.f.
Φ( g · x ) = g · Φ( x ) , ∀ x ∈ X, ∀ g ∈ G .
Colloquially, an equivariant map implies that transforming an input and applying the map yields the same result as applying the map and then transforming the output.
Analogously, invariance requires that Φ( g · x ) =
Φ( x ) , ∀ x ∈ X, ∀ g ∈ G .
In deep networks, equivariance to a planar symmetry group can be achieved by either transforming ﬁlters (Cohen & Welling, 2016) or feature maps (Dieleman et al., 2016).
Our work utilizes the plane symmetry groups p 4 (all compositions of 90-degree rotations and translations) and p 4 m (all compositions of 90-degree rotations, reﬂections, and translations) (Schattschneider, 1978).
These groups can be parameterized neatly following Cohen & Welling (2016), g
[Equation] [Equation]
[Equation]
where g ( r, u, v ) parameterizes p 4 , g (cid:48)
( m, r, u, v ) parameterizes p 4 m , 0 ≤ r < 4 (the number of 90degree rotations),
m ∈ { 0 , 1 } (the number of reﬂections), and ( u, v ) ∈
Z 2 (integer translations).
The group operation is matrix multiplication for both groups.
The matrix g ( r, u, v ) rotates and translates a point (expressed as homogeneous coordinate vector) in pixel space via left-multiplication.
Analogous intuition follows for g (cid:48) ( m, r, u, v ) .
We now brieﬂy deﬁne G -equivariant convolutions.
We note that formally these are correlations and not convolutions and that the literature uses the terms interchangeably.
A G -convolution between a vector-valued K -channel image f : Z 2 → R K and ﬁlter ψ : Z 2 → R K with f = ( f 1 , f 2 , . . .
, f k ) and ψ = ( ψ 1 , ψ 2 , . . .
, ψ k ) can be expressed as [ f ∗ ψ ]( g ) =
(cid:80)
y ∈
Z 2 (cid:80)
Kk =
1 f k ( y ) ψ k ( g − 1 y ) .
For standard reference, if one considers G to be the translation group on Z 2 , we have g − 1 y
=
y − g and recover the standard convolution.
After the ﬁrst layer of a G -CNN, we see that ( f ∗ ψ ) is a function on G , necessitating that ﬁlter banks also be functions on G .
Subsequent G -convolutional layers are therefore deﬁned as [ f ∗ ψ ]( g ) =
(cid:80) h ∈ G (cid:80)
Kk =
1 f k ( h ) ψ k ( g − 1 h ) .
Finally, for tasks where the output is an image, it is necessary to bring the domain of feature maps from G back to Z 2 .
We can pool the feature map for each ﬁlter over the set of transformations, corresponding to average or max pooling over the group of rotations (or roto-reﬂections as appropriate).
GAN optimization and stability.
As we focus on the limited data setting where training instability is exacerbated, we brieﬂy describe the two major stabilizing methods used in all experiments here.
We regularize the discriminator by using a zero-centered gradient penalty (GP) on the real data as proposed by Mescheder et al.
(2018) of the form, R 1 :=
γ 2 E x ∼ P real [ (cid:107)∇ D ( x ) (cid:107) 22 ] , where γ is the regularization weight, x is sampled from the real distribution P real , and D is the discriminator.
This
GP has been shown to cause convergence (in toy cases), alleviate catastrophic forgetting (ThanhTung & Tran, 2018), and strongly stabilize GAN training.
However, empirical work has found that this GP achieves stability at the cost of worsening GAN evaluation scores (Brock et al., 2018).
A widely used technique for GAN stabilization is spectral normalization (Miyato et al., 2018), which constrains the discriminator to be 1-Lipschitz, thereby improving gradient feedback to the generator (Zhou et al., 2019; Chu et al., 2020).
With spectral normalization, each layer is rescaled as, W SN =
W/σ ( W ) , where W is the weight matrix for a given layer and σ ( W ) is its spectral norm.
In practice, σ ( W ) is estimated via a power iteration method as opposed to computing the full singular value decomposition during each training iteration.
Finally, applying spectral normalization to both generator and discriminator empirically improves training signiﬁcantly (Zhang et al., 2018).
2. G ROUP E QUIVARIANT G ENERATIVE A DVERSARIAL N ETWORKS
Here, we outline how to induce a symmetry prior into the GAN framework.
Implementations are available at https://github.com/neel-dey/equivariant-gans .
The literature has developed several techniques for normalization and conditioning of the individual networks, along with unique architectural choices - we extend these developments to the equivariant setting.
We start by replacing all convolutional layers with group-convolutional layers where ﬁlters and feature maps are functions on a symmetry group G .
Batch normalization moments (Ioffe & Szegedy, 2015) are calculated per group-feature map as opposed to spatial feature maps.
Pointwise nonlinearities preserve equivariance for the groups considered here.
Pre-activation residual blocks common to modern GANs are used freely as the sum of equivariant feature maps on G is also equivariant.
Generator.
The generator is illustrated at a high-level in Figure 2.
We use a fully connected layer to linearly project and reshape the concatenated noise vector z ∼ N (0 , I ) and class embedding c into spatial feature maps on Z 2 .
We then use spectrally-normalized group-convolutions, interspersed with pointwise-nonlinearities, and nearest-neighbours upsampling to increase spatial extent.
We use upsampling followed by group-convolutions instead of transposed group-convolutions to reduce checkerboard artefacts (Odena et al., 2016).
We further use a novel group-equivariant classconditional batch normalization layer (described below) to normalize and class-condition image generation while also projecting the latent vector z to each level of the group-convolutional hierarchy.
We ﬁnally max-pool over the set of transformations to obtain the generated image x .
Discriminator.
The group-equivariant discriminator receives an input x , which it maps to a scalar indicating whether it is real or fake.
We do this via spectrally normalized group-convolutions, pointwise-nonlinearities, and spatial-pooling layers to decrease spatial extent.
After the ﬁnal groupconvolutional layer, we pool over the group and use global average pooling to obtain an invariant representation at the output.
Finally, we condition the discriminator output via the projection method proposed by Miyato & Koyama (2018).
Importantly, the equivariance of group-convolutions depends on the convolutional stride.
Strided convolutions were commonly used for downsampling in early GANs (Radford et al., 2015).
However, stride values must be adjusted to the dataset to preserve equivariance, which makes comparisons to equivalent non-equivariant GAN architectures difﬁcult.
We therefore use pooling layers over the plane (commonly used in recent GANs) to downsample in all settings to preserve equivariance and enable a fair comparison.
Spectral Normalization.
As the singular values of a matrix are invariant under compositions of 90degree rotations, transpositions, and reﬂections - spectral normalization on a group-weight matrix preserves equivariance and we use it freely.
Class-conditional Batch Normalization.
Conditional batch normalization (Perez et al., 2018) replaces the scale and shift of features with an afﬁne transformation learned from the class label (and optionally from the latent vector as well (Brock et al., 2018)) via linear dense layers, and is widely used in generative networks.
We propose a group-equivariance preserving conditional normalization by learning the afﬁne transformation parameters per group-feature map, rather than each spatial feature.
As we use fewer group-ﬁlters than equivalent non-equivariant GANs, we use fewer dense parameters to learn conditional scales and shifts.
E XPERIMENTS
Common setups.
In each subsection, we list speciﬁc experimental design choices with full details available in App. C. For each comparison, the number of group-ﬁlters in each layer is divided by the square root of the cardinality of the symmetry set to ensure a similar number of parameters to the standard CNNs to enable fair comparison.
We skew towards stabilizing training over absolute
[Table] [Table]
performance to compare models under the same settings to obviate extensive checkpointing typically required for BigGAN-like models.
Optimization is performed via Adam (Kingma & Ba, 2014) with β 1 = 0 . 0 and β 2 = 0 .
9 , as in Zhang et al.
; Brock et al.
Unless otherwise noted, all discriminators are updated twice per generator update and employ unequal learning rates for the generator and discriminator following Heusel et al.
We use an exponential moving average ( α = 0 . 9999 ) of generator weights across iterations when sampling images as in Brock et al.
All initializations use the same random seed, except for RotMNIST where we average over 3 random seeds.
An overview of the small datasets considered here is presented in Table 1.
Evaluation methodologies.
GANs are commonly evaluated by embedding the real and generated images into the feature space of an ImageNet pre-trained network where similarity scores are computed.
The Fr´echet Inception Distance (FID) (Heusel et al., 2017) jointly captures sample ﬁdelity and diversity and is presented for all experiments.
To further evaluate both aspects explicitly, we present the improved precision and recall scores (Kynk¨a¨anniemi et al., 2019) for ablations on real-world datasets.
As the medical imaging datasets (ANHIR and LYSTO) are not represented in
ImageNet, we ﬁnetune Inception-v3 (Szegedy et al., 2016) prior to feature extraction for FID calculation as in Huang et al.
For RotMNIST, we use features derived from the ﬁnal pooling layer of the p 4 -CNN deﬁned in Cohen & Welling (2016) to replace Inception-featurization.
An analogous approach was taken in Binkowski et al.
(2018) in their experiments on the canonical
MNIST dataset.
Natural image datasets (Food-101 and CIFAR-10) are evaluated with the ofﬁcial
Tensorflow Inception-v3 weights.
Importantly, we perform ablation studies on all datasets to evaluate group-equivariance in either or both networks.
We note that the FID estimator is strongly biased (Binkowski et al., 2018) and work around this limitation by always generating the same number of samples as the validation set as recommended in Binkowski et al.
An alternative Kernel Inception Distance (KID) with negligible bias has
[Figure]
P r o p o s e d
B a s e li n e
P r o p o s e d
B a s e li n e
P r o p o s e d
B a s e li n e
Equivariant GANs interpolate intuitively between samples, whereas standard GANs do not
Selected inter-label linear interpolations between two staining dyes in synthesized ANHIR images.
The standard model ( top ) changes both structure and dye between the generated samples, whereas the equivariant model ( bottom ) better preserves structure while translating between dyes.
been proposed (Binkowski et al., 2018), yet large-scale evaluation (Kurach et al., 2019) ﬁnds that
KID correlates strongly with FID.
We thus focus on FID in our experiments in the main text.
3. S YNTHETIC E XPERIMENTS : R OTATED MNIST
Rotated MNIST (Larochelle et al., 2007) provides random rotations of the MNIST dataset and is a common benchmark for equivariant CNNs which we use to measure sensitivity to dataset size, loss function, and equivariance in either network to motivate choices for real-world experiments.
We experiment with four different proportions of training data: 10%, 33%, 66%, and 100%.
Additionally, the non-saturating loss (Goodfellow et al., 2014) (NSGAN), the Wasserstein loss (Arjovsky et al., 2017) (WGAN), and the relativistic average loss (Jolicoeur-Martineau, 2018) (RaGAN) are tested.
For the equivariant setting, all convolutions are replaced with p 4 -convolutions.
p 4 m is precluded as some digits do not possess mirror symmetry.
All settings were trained for 20,000 generator iterations with a batch size of 64.
Implementation details are available in Appendix C.2.1.
Results.
Fr ´ echet distance of synthesized samples to the validation set is calculated at every thousand generator iterations.
As shown in Table 2, we ﬁnd that under nearly every conﬁguration of loss and data availability considered, using p 4 -convolutions in either network improves both the mean and minimum Fr ´ echet distance.
As data availability increases, the best-case minimum and mean
FID scores improve.
With { 33% , 66% , 100% } of the data, most improvements come from using a p 4 -discriminator, with the further usage of a p 4 -generator only helping in a few cases.
At 10% data, having an equivariant generator is more impactful than an equivariant discriminator.
These trends are further evident from App.
A Fig.
6, where we see that GANs with p 4 -discriminators converge faster than non-equivariant counterparts.
The NSGAN-GP and RAGAN-GP losses perform similarly, with WGAN-GP underperforming initially and ultimately achieving comparable results.
Qualitatively, the equivariant model learns better representations as shown in Figure 3(a).
Holding the class-label constant and interpolating between samples, we ﬁnd that the standard GAN changes the shape of the digit in order to rotate it, whereas the equivariant model learns rotation in the latent space.
Holding the latent constant and interpolating between classes shows that our model learns an intuitive interpolation between digits, whereas the standard GAN transforms the image immediately.
3. R EAL - WORLD E XPERIMENTS
Datasets.
p 4 and p 4 m -equivariant networks are most useful when datasets possess global roto(- reﬂective) symmetry, yet have also been shown to beneﬁt generic image representation due to local symmetries (Cohen & Welling, 2016; Romero et al., 2020).
To this end, we experiment with two
[Table]
[Figure]
[Table]
types of real-world datasets as detailed in Table 1: (1) sets with roto(-reﬂective) symmetry, such that the image label is invariant under transformation; (2) natural images with preferred orientation (e.g., the boat class of images in CIFAR-10 cannot be upside-down).
Brieﬂy, they are:
ANHIR provides high-resolution pathology slides stained with 5 different dyes to highlight different cellular properties (Borovec et al., 2020; 2018).
We extract 128 × 128 foreground patches from images of different scales, as described in App.
C.1.2.
We use the staining dye as conditioning.
LYSTO is a multi-organ pathology benchmark for the counting of immunohistochemistry stained lymphocytes (Ciompi et al., 2019).
We re-purpose it here for conditional synthesis at a higher resolution of 256 × 256 .
As classiﬁcation labels are not provided, we use the organ source as class labels.
The use of organ sources as classes is validated in App.
C.1.1.
The high image resolution in addition to the limited sample size of 20,000 make LYSTO a challenging dataset for GANs.
CIFAR-10 is a natural image vision benchmark of both small resolution and sample size (Krizhevsky et al., 2009).
Previous work (Weiler & Cesa, 2019; Romero et al., 2020) ﬁnds that equivariantnetworks improve classiﬁcation accuracy on CIFAR-10 and we include here it as a GAN benchmark.
Food-101 is a small natural image dataset of a 101 categories of food taken in various challenging settings of over/under exposure, label noise, etc.
(Bossard et al., 2014).
Further, datasets with a high number of classes are known to be challenging for GANs (Odena, 2019).
Importantly, even though the objects in this dataset have a preferred pose due to common camera orientations, we speculate that roto-equivariance may be beneﬁcial here as food photography commonly takes an en face or mildly oblique view.
We resize the training set to 64 × 64 resolution for our experiments.
Fig.
encouraged
Baseline architecture.
To produce a strong non-equivariant baseline, we face several design choices.
State-of-the-art GANs follow either BigGAN (Brock et al., 2018) or StyleGAN2 (Karras et al., 2020b) in design.
As StyleGAN2 has not yet been demonstrated to scale to conditional generation with a large number of classes (to our knowledge), we follow a BigGAN-like construction despite the stability of StyleGAN2.
For our small datasets, we make the following modiﬁcations: (1) we use fewer channels; (2) we do not use orthogonal regularization; (3) we do not use hierarchical latent projection as we ﬁnd in early testing that projecting the entire latent to each normalization layer achieves similar results; (4) we do not use attention as equivariant attention is an area of active research (Romero & Hoogendoorn, 2019; Romero et al., 2020) but currently has prohibitively high memory requirements and may not yet scale to GANs.
Further details are available in App. C.2.
We then modify either generator ( G ) and/or discriminator ( D ) as in Section 2.2 to obtain the corresponding equivariant settings.
We note that a discriminator invariant to roto-reﬂections would assign the same amount of realism to an upright natural image versus a rotated/reﬂected copy of the same image, allowing the generator to synthesize images at arbitrary orientations.
Therefore, for CIFAR10 and Food-101 we pool over rotations before the last residual block to enable the discriminator to detect when generated images are not in their canonical pose while maintaining most of the beneﬁts of equivariance as studied in Weiler & Cesa (2019).
We use p 4 m -equivariance for ANHIR and
LYSTO and p 4 -equivariance for CIFAR-10 and Food-101 to reduce training time.
Comparisons.
A natural comparison would be against standard GANs using augmentations drawn from the same group our model is equivariant to.
However, augmentation on the real images alone would lead to the augmentations “leaking” into the generated images, e.g., vertical ﬂip augmentation may lead to generated images being upside-down.
Zhao et al.
(2020c) propose balanced consistency regularization (bCR) for augmentations of both real and generated samples to alleviate this issue, and we thus use it as a comparison.
We restrict the augmentations used in bCR to 90-degree rotations or 90-degree rotations and reﬂections as appropriate to enable a fair comparison against equivariant GANs.
Using additional augmentations would help all methods across the board.
We further compare against auxiliary rotations (AR) GAN (Chen et al., 2019) where real and fake images are augmented with 90-degree rotations and the discriminator is tasked with predicting their orientation.
We do not use AR for ANHIR and LYSTO as they have no canonical orientation.
For completeness, we also evaluate standard augmentation (reals only) for all datasets.
Results.
Quantitative FID results of ablations and comparisons against baselines are presented in Table 3.
Equivariant networks (G-CNNs) outperform methods which use standard CNNs with or without augmentation across all datasets.
For ANHIR and LYSTO, we ﬁnd that p 4 m -equivariance in either network improves FID evaluation, with the best results coming from modifying both networks.
However, for the upright datasets CIFAR-10 and Food-101, we ﬁnd that having a p 4 -equivariant discriminator alone helps more than having both networks be p 4 -equivariant.
We speculate that this effect is in part attributable to their orientation bias.
With bCR and AR GANs, we ﬁnd that standard CNNs improve signiﬁcantly, yet are still outperformed by equivariant nets using no augmentation.
We include a mixture of equivariant GANs and bCR for completeness and ﬁnd that for ANHIR and Food-101, they have an additive effect, whereas they do not for LYSTO and CIFAR-10, indicating a dataset-sensitivity.
Of note, we found that bCR with its suggested hyperparameters lead to immediate training collapse on ANHIR, LYSTO, and CIFAR-10, which was ﬁxed by decreasing the strength of the regularization substantially.
This may be due to the original work using several different types of augmentation and not just roto-reﬂections.
Standard augmentation (i.e., augmenting training images alone) lead to augmentation leakage for CIFAR-10 and Food-101.
Qualitatively, as class differences in ANHIR should be stain (color) based, we visualize inter-class interpolations between synthesized samples in Figure 3(b).
We ﬁnd that our model better preserves structure while translating between stains, whereas the non-equivariant GAN struggles to do so.
In our ablation study in terms of precision and recall in Figure 5, using p 4 m -equivariance in G and D achieves consistently higher recall for ANHIR and LYSTO.
For Food-101, we ﬁnd that G-CNN in G and D achieves higher precision, whereas CNN in G and G-CNN in D achieves higher recall.
For CIFAR-10 precision and recall, we ﬁnd no discernable differences between the two settings with lowest FID.
Interestingly, for CIFAR-10 adding p 4 -equivariance to G but not D worsens FID but noticeably improves precision.
These observations are consistent with our FID ﬁndings as FID tends to correlate better with recall (Karras et al., 2020a).
Finally, we plot
FID vs. generator updates in Figure 5, ﬁnding that the proposed framework converges faster than the baseline as a function of training iterations (for all datasets except ANHIR).
Convergence plots for all datasets and all methods compared can be found in App.
A Figure 7, showing similar trends.
D ISCUSSION
Future work.
We present improved conditional image synthesis using equivariant networks, opening several potential future research directions: (1) As efﬁcient implementations of equivariant attention develop, we will incorporate them to model long-range dependency; (2) Equivariance to continuous groups may yield further increased data efﬁciency and more powerful representations.
However, doing so may require non-trivial modiﬁcations to current GAN architectures as memory limitations could bottleneck continuous group-equivariant GANs at relevant image sizes.
Further, adding more discretizations beyond 4 rotations on a continuous group such has SE (2) may show diminishing returns (Lafarge et al., 2020a, Fig.7); (3) In parallel to our work, Karras et al.
(2020a) propose a differentiable augmentation scheme for limited data GANs pertaining to which transformations to apply and learning the frequency of augmentation for generic images, with similar work presented in Zhao et al.
(2020b).
Our approach is fully complementary to these methods when employing transformations outside the considered group and will be integrated into future work; (4)
Contemporaneously, Lafarge et al.
(2020b) propose equivariant variational autoencoders allowing for control over generated orientations via structured latent spaces which may be used for equivariant GANs as well; (5) The groups considered here do not capture all variabilities present in natural images such as small diffeomorphic warps.
Scattering networks may provide an elegant framework to construct GANs equivariant to a wider range of symmetries and enable higher data-efﬁciency.
Conclusion.
We present a ﬂexible framework for incorporating symmetry priors within GANs.
In doing so, we improve the visual ﬁdelity of GANs in the limited-data regime when trained on symmetric images and even extending to natural images.
Our experiments conﬁrm this by improving on conventional GANs across a variety of datasets, ranging from medical imaging modalities to real-world images of food.
Modifying either generator or discriminator generally leads to improvements in synthesis, with the latter typically having more impact.
To our knowledge, our work is the ﬁrst to show clear beneﬁts of equivariant learning over standard GAN training on high-resolution conditional image generation beyond toy datasets.
While this work is empirical, we believe that it strongly motivates future theoretical analysis of the interplay between GANs and equivariance.
Finally, improved results over augmentation-based strategies are presented, demonstrating the beneﬁts of explicit transformation equivariance over equivariance-approximating regularizations. Acknowledgements.
Neel Dey thanks Mengwei Ren, Axel Elaldi, Jorge Ono, and Guido Gerig.
A S UPPLEMENTARY R ESULTS
[Figure] [Figure]
P l a n e
C a r
B i r d
C a t
D ee r
D og
F r og
H
o r s e
B oa t
T r u c k
[Table]
B I MAGE - TO -I MAGE T RANSLATION
To show the generic utility of equivariance in generative adversarial network tasks, we present a pilot study employing p 4 -equivariance in supervised image-to-image translation to learn mappings between visual domains.
Using the popular Pix2Pix model of Isola et al.
(2017) as a baseline, we replace both networks with p 4 -equivariant models.
For completeness, we also evaluate whether employing p 4 -equivariance in just the discriminator achieves comparable results to modifying both networks, as in the natural image datasets in the main text.
We use the 256 × 256 Maps dataset ﬁrst introduced in (Isola et al., 2017), consisting of 1096 training and 1098 validation images of pairs of Google maps images and their corresponding satellite/aerial view images.
As FID has a highly biased estimator, its use for evaluating generation with onlyvalidation samples is contraindicated (Binkowski et al., 2018).
We instead use the Kernel Inception
Distance (KID) proposed by Binkowski et al.
(2018) which exhibits low bias for small sample sizes and is adopted in recent image translation studies (Kim et al., 2020).
Brieﬂy, as in FID, KID embeds real and fake images into the feature-space of an appropriately chosen network and computes the squared maximum-mean discrepancy (with a polynomial kernel) between their embeddings.
Lower values of KID are better.
We use the ofﬁcial Tensorflow implementation and weights 1 .
For baseline Pix2Pix , we use pre-trained weights provided by the authors 2 .
Interestingly, we ﬁnd that their architectures can be optimized for improved performance by replacing transposed convolutions with resize-convolutions, reducing the number of parameters by swapping 4 × 4 convolutional kernels for 3 × 3 kernels, and removing dropout.
For equivariant models, we replace convolutions with p 4 -convolutions in this optimized architecture and halve the number of ﬁlters to keep the number of parameters similar across settings.
Architectures are given in Tables 15 and 16.
We leave all other experimental details identical to Isola et al.
(2017) for all models, such as training forepochs with random crops under a cross-entropy GAN loss.
Quantitative results are presented in Table 4 which shows that p 4 -equivariance in either setting improves over both original baseline and optimized baseline by a wide margin, with the best results coming from p 4 -equivariance in the discriminator alone.
Qualitative results are presented in Figure 12 showing improved translation ﬁdelity, further supporting our hypothesis that equivariant networks beneﬁt GAN tasks generically.
C E XPERIMENTAL DETAILS C. D ATA PREPARATION C.1. LYSTO CLASS CONDITIONING
To validate the assumption of the organ source being a discriminative feature, a suitable test would be to train a classiﬁer to distinguish between sources.
We partition the original training set with a 60/40 train/test split.
The original testing set is not used as it has no publicly available organ source information.
The dataset has 3 classes - colon, breast, and prostate.
Holding out 20% of the new constructed training set for validation, we ﬁne-tune ImageNet-pretrained VGG16 (Simonyan &
Zisserman, 2014) and achieve 98% organ classiﬁcation test accuracy, thus validating our assumption.
C.1. ANHIR PATCH EXTRACTION
To extract patches for image synthesis, we choose the lung-lesion images from the larger ANHIR dataset, as these images are provided at different scales and possess diverse staining.
The images were cropped to the nearest multiples of 128, and 128 × 128 patches were then extracted.
Foreground/background masking was performed via K-means clustering, followed by morphological dilation.
The images were then gridded into 128 × 128 patches, i.e., there was no overlap between patches.
If a patch contained less than 10% foreground pixels, it was excluded from consideration.
C. A DDITIONAL I MPLEMENTATION D ETAILS
The following subsections list dataset-speciﬁc training strategies.
Unless noted, all layers use orthogonal initializations.
Batch normalization momentum is set to 0.1, and LeakyReLU slopes are set to 0.2 (if used).
Spectral normalization is used everywhere except for the dense layer which learns the class embedding as speciﬁed in the BigGAN PyTorch GitHub repository 3 .
For ablation studies, as GANs consist of two networks (the generator and discriminator), we replace group-equivariant layers (convolutional, normalization, and pooling) with the corresponding standard layers in either generator or discriminator to evaluate which network beneﬁts the most from equivariant learning.
When we remove equivariant layers from both networks, we recover our baseline comparison.
All settings use roughly the same number of parameters, with a very small difference in parameter count arising from the p 4 (or p 4 m ) class-conditional batch normalization layers requiring fewer afﬁne scale and shift parameters than their corresponding standard normalization layers.
Tangentially, we note that the equivariant networks require higher amounts of computation time.
For example, for a ﬁxed number of training iterations on ANHIR, p 4 m -equivariant GANs currently require approximately four times the amount of computation time.
To identify a common shared stable hyperparameter conﬁguration for all ablations of our method on real datasets, a grid search was performed for the ANHIR dataset over learning rates for generator and discriminator ( η g , η d ) : ( { 10 − 4 , 4 × 10 − 4 } , { 5 × 10 − 5 , 2 × 10 − 4 } ) , gradient penalty strengths ( γ = { 0 . 01 , 0 . 1 , 1 . 0 , 10 . 0 } ), and binary choices as to whether to use batch normalization in the discriminator or not, whether to use average-pooling or max-pooling to reduce spatial extent in the discriminator, and whether to use a Gaussian latent space or a Bernoulli latent space.
We use the identiﬁed hyperparameter conﬁguration as an initial starting point for all datasets, modifying them as appropriate as described below.
For ANHIR, LYSTO, and Food-101 we use the relativistic average adversarial loss (JolicoeurMartineau, 2018) for its stability and for CIFAR-10 we use the Hinge loss (Lim & Ye, 2017; Tran et al., 2017) to remain consistent with the literature for that dataset.
For our implementation of auxiliary rotations GAN (Chen et al., 2019), we use the suggested regularization weights.
For balanced consistency regularization (bCR) (Zhao et al., 2020c), we ﬁnd that dataset-speciﬁc tuning of the regularization strength was required
C.2. R OT MNIST
Given the low resolution of Rotated MNIST, we take a straightforward approach to synthesis without residual connections.
In the generator, we sample from a 64 D Gaussian latent space, concatenate class embeddings, and linearly project as described in Section 2.2.
Four spectrally-normalized convolutional layers are then used with class-conditional batch normalization employed after every convolution except for the ﬁrst and last layer.
The discriminator uses three spectrally normalized convolutional layers, with leaky ReLU non-linearities.
Average pooling is used to reduce the spatial extent of the feature maps, with global average pooling and conditional projection used at the end of the sequence.
For NSGAN and RaGAN, we use the R 1 GP, conservatively setting γ = 0 . 1 .
For WGAN, we use the GP deﬁned in Gulrajani et al.
(2017) to ensure the 1-Lipschitz constraint with the recommended weight of 10 . 0 .
Learning rates were set to η G = 0 . 0001 and η D = 0 . 0004 , respectively.
For the p 4 -equivariant models, max-pooling over rotations is used after the last groupconvolutional layer in both generator and discriminator to get planar feature maps.
Architectures are presented in Tables 5 and 6.
C.2. ANHIR
We sample from a 128 D Gaussian latent space with a batch size of 32.
The generator consists of 6 pre-activation residual blocks followed by a ﬁnal convolutional layer to obtain a 3-channel output.
We use class-conditional batch normalization after every convolution, except at the ﬁnal layer.
The discriminator uses 5 pre-activation residual blocks, followed by global average pooling and conditional projection.
In the equivariant settings, we use residual blocks with p 4 m -convolutions for roto-reﬂective symmetries.
We train with the relativisitic average loss and use the R 1 GP with γ = 0 . 1 .
Learning rates are set to η G = 0 . 0001 and η D = 0 . 0004 .
All models were trained for approximately 60,000 generator iterations.
bCR weights for comparison were set to λ real = 0 .and
λ fake = 0 .
for roto-reﬂective augmentations, with higher values collapsing training.
Architectures are presented in Tables 7 and 8.
C.2. LYSTO
Implementation for LYSTO is similar to that of App.
C.2.2, with some key differences due to the greater difﬁculty of training.
Due to memory constraints, we use a batch size of 16.
We increase the number of residual blocks to 6 in both generator and discriminator and halve the number of ﬁlters.
The equivariant settings used the p 4 m roto-reﬂective symmetries.
We initially experienced low sample diversity across a variety of hyperparameter settings.
Contrary to recent literature, we ﬁnd that using batch normalization in the discriminator in addition to spectral normalization greatly improves training for this dataset.
Further, halving the learning rates for both networks to η G = 0 .
00005 and η D = 0 .
and increasing the strength of the gradient penalty to 1.0 were necessary for ensuring training stability.
As in App.
C.2.2, all models were trained for approximately 60, generator iterations and bCR weights were set to λ real = 0 .
1 and λ fake = 0 .
for roto-reﬂective augmentations.
As test set labels are not publicly available for LYSTO, we evaluate FID, Precision, and Recall to the training set itself as done in a subset of experiments within Jolicoeur-Martineau (2018) and Zhao et al.
(2020b).
Architectures are presented in Tables 9 and 10.
C.2. CIFAR-
For CIFAR-10, we make the following changes to our training parameters to be in accordance with prior art for BigGAN-like designs for this dataset: (1) layer weights are now initialized from
N (0 , 0 . 02) ; (2) average pooling is used in the discriminator instead of max pooling; (3) learning rates η G and η D are now equal and set to 0.0002; (4) the discriminator is updated four times per generator update; (5) architectures are modiﬁed as in Tables 11 and 12; (6) we use the Hinge loss instead of the relativistic average loss.
We use a batch size of 64.
Karras et al.
(2020a) suggest an R
GP weight of γ = 0 .
01 for CIFAR-10 which we use here.
We train all CIFAR-10 GANs for 100K generator iterations.
bCR weights were set to λ real = 0 .
1 and λ fake = 0 .
1 for 90-degree rotation augmentations.
For the p 4 -equivariant discriminators, we move the pooling over the group to before the last residual block as stated in the main text.
Alternatively, we experimented with using a single additional standard convolutional layer with 32 ﬁlters after the p 4 -residual blocks as a lightweight alternative
Upsample 2x 1x1 p4m-p4m
GConvSN p4m GBatchNorm p4m GBatchNorm +
Upsample 2x
ResBlockG +
ReLU
ReLU 3x3 p4m-p4m
GConvSN 3x3 p4m-p4m
GConvSN
ReLU p4m GBatchNorm p4m GBatchNorm + 1x1 p4m-p4m
GConvSN 3x3 p4m-p4m GConvSN 3x3 p4m-p4m
GConvSN +
ResBlockD 2x2 plane-groupMaxPool
ReLU 2x2 plane-groupMaxPool
o p t i o n a l o p t i o n a l (latent, class) embedding
Linearprojection γ , β γ , β Linearprojection to making an entire residual block non-equivariant but this worsened FID evaluation.
Interestingly, we ﬁnd that substituting Global Average Pooling for Global Sum Pooling in the CIFAR-10 discriminators lead to an improvement of ∼ 5 - 8 in terms of FID across the board.
This architectural change to the ResNet-based GANs from Gulrajani et al.
(2017) was originally made in Miyato et al.
(2018), but to our knowledge has not been noted in the literature previously.
C.2. F OOD -
Compared to the residual synthesis models in App.
C.2.2 and C.2.3, we make several changes.
We sample from a 64 D latent Gaussian to lower the number of dense parameters and substantially increase the width of the residual blocks to account for the high number of image classes.
We ﬁnd that an 8 × increase in the number of channels for the initial projection from the latent vector and class embedding improves training signiﬁcantly.
We use 4 residual blocks each in both generator and discriminator.
For the equivariant setting, we use only p 4 rotational symmetries to reduce training time.
Importantly, we increase the batch size to 64 and the R 1 GP to γ = 1 . 0 , both of which improve the evaluation of all experimental settings.
We train all GANs for ∼ 45K generator iterations.
The suggested bCR weights of λ real = 10 . 0 and λ fake = 10 . 0 from Zhao et al.
(2020c) were used here for 90-degree rotation augmentations.
However, when bCR with default parameters was combined with p 4 -equivariance in G and D , augmentations start to ‘leak’ into the generated images (e.g., G generating upside-down plates), necessitating lower weights of λ real = 0 .
5 and λ fake = 0 .
C. A RCHITECTURES
Architectures for the Rotated MNIST experiments are given in Tables 5 and 6, ANHIR in Tablesand 8, and LYSTO in Tables 9 and 10.
The residual blocks used in the ANHIR, LYSTO, CIFAR10, and Food-101 experiments are given in Figure 13.
SN refers to spectral normalization and ( z 2 − p 4) , ( p 4 − p 4) , ( z 2 − p 4 m ) , ( p 4 m − p 4 m ) refer to the type of convolution used.
[Table]
LeakyReLU
[Table]
Global Average Pool into f
Embed y ∈ { 0 , ..., 9 } into
[Table]
Projection step
[Table] [Table]
LeakyReLU,
[Table] [Table]
[Table] [Table]
[Table]
tanh
[Table]
[Table] [Table]
[Table] [Table]
y
[Table]
Concatenate z and ˆ y into h ∈ R
Project and reshape h to 4 × 4 ×z 2 − p 4 m ResBlockG, 128 →p 4 m − p 4 m ResBlockG, 181 →p 4 m − p 4 m ResBlockG, 90 →p 4 m − p 4 m ResBlockG, 45 →p 4 m − p 4 m ResBlockG, 22 →p 4 m − p 4 m ResBlockG, 11 →p 4 m -BN; ReLU 3 × 3 p 4 m − p 4 m GConvSN
[Table]
5 →
[Table]
p
[Table]
m -Max Pool
[Table]
[Table] [Table]
[Table] [Table]
Concatenate
[Table]
and ˆ
[Table]
Project and reshape h to 4 × 4 ×
[Table]
ResBlockG
[Table] [Table]
[Table]
ResBlockG
[Table] [Table]
ResBlockG
[Table] [Table]
[Table] [Table]
Embed y ∈ { 0 , ..., 100 } into ˆ y ∈ R
Concatenate z and ˆ y into h ∈ R
Project and reshape h to 4 × 4 ×
[Table]
Projection step
[Table]
[Table]
→ 512; Concatenate hz 2 − z 2 UpBlock, 512 → 256; Concatenate hz 2 − z 2 UpBlock, 256 → 128;
Concatenate hz 2 − z 2 UpBlock, 128 → 64; Concatenate h
Upsample 2 × , 3 × 3 Conv, 64 →tanh()
Discriminator
Input RGB image x ∈ R 256 × 256 ×
Input RGB image y ∈ R 256 × 256 ×
Concatenate x and y feature-wise z 2 − z 2 DownBlock, 3 →z 2 − z 2 DownBlock, 64 →z 2 − z 2 DownBlock, 128 →z 2 − z 2 3 × 3 Conv 256 → 512,
BatchNorm, Leaky ReLU z 2 − z 2 3 × 3 Conv 512 → 1, tanh()
[Table] [Table]
[Table]
Upsample 2 × , 3 × 3 GConv, 64 →p 4 -average pool, tanh()
[Table]
p 4 − p 4 DownBlock,
[Table]
p 4 − p 4 DownBlock, 128 →p 4 − p
[Table]
3 × 3 Conv 256 → 512,
BatchNorm, Leaky ReLU p 4 − p 4 3 × 3 Conv 512 → 1,
[Table]
pool tanh
[Table]
3 × 3 p 4 -Convolution, p 4 -Batch Normalization
and Leaky ReLU activation