Original: W HITENING AND C OLORING TRANSFORM FOR GAN S
Revised: W HITENING AND C OLORING TRANSFORM FOR GAN S

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Batch Normalization (BN) is a common technique used to speed-up and stabilize training.
Revised: Batch Normalization (BN) is a common technique used to speed-up and stabilize training.

Original: On the other hand, the learnable parameters of BN are commonly used in conditional Generative Adversarial Networks (cGANs) for representing classspeciﬁc information using conditional Batch Normalization (cBN).
Revised: On the other hand, the learnable parameters of BN are commonly used in conditional Generative Adversarial Networks (cGANs) for representing classspeciﬁc information using conditional Batch Normalization (cBN).

Original: In this paper we propose to generalize both BN and cBN using a Whitening and Coloring based batch normalization.
Revised: In this paper we propose to generalize both BN and cBN using a Whitening and Coloring based batch normalization.

Original: We show that our conditional Coloring can represent categorical conditioning information which largely helps the cGAN qualitative results.
Revised: We show that our conditional Coloring can represent categorical conditioning information which largely helps the cGAN qualitative results.

Original: Moreover, we show that full-feature whitening is important in a general GAN scenario in which the training process is known to be highly unstable.
Revised: Moreover, we show that full-feature whitening is important in a general GAN scenario in which the training process is known to be highly unstable.

Original: We test our approach on different datasets and using different GAN networks and training protocols, showing a consistent improvement in all the tested frameworks.
Revised: We test our approach on different datasets and using different GAN networks and training protocols, showing a consistent improvement in all the tested frameworks.

Original: Our
Revised: Our

Original: CIFAR-10 supervised results are higher than all previous works on this dataset.
Revised: CIFAR-10 supervised results are higher than all previous works on this dataset.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Generative Adversarial Networks (GANs) (Goodfellow et al.
Revised: Generative Adversarial Networks (GANs) (Goodfellow et al.

Original: (2014)) have drawn much attention in the last years due to their proven ability to generate sufﬁciently realistic short videos (Vondrick et al.
Revised: (2014)) have drawn much attention in the last years due to their proven ability to generate sufﬁciently realistic short videos (Vondrick et al.

Original: (2016)) or still images (Gulrajani et al.
Revised: (2016)) or still images (Gulrajani et al.

Original: (2017)), possibly conditioned on some input information such as: a class label (Odena et al.
Revised: (2017)), possibly conditioned on some input information such as: a class label (Odena et al.

Original: (2017)), an image (Isola et al.
Revised: (2017)), an image (Isola et al.

Original: (2017)), a textual description (Reed et al.
Revised: (2017)), a textual description (Reed et al.

Original: or structured data (Ma et al.
Revised: or structured data (Ma et al.

Original: ; Siarohin et al.
Revised: ; Siarohin et al.

Original: When the generation process depends on some input data, the framework is commonly called conditional GAN (cGAN).
Revised: When the generation process depends on some input data, the framework is commonly called conditional GAN (cGAN).

Original: In this paper we deal with both conditional and unconditional GANs and we propose to replaceBatch
Revised: In this paper we deal with both conditional and unconditional GANs and we propose to replace

Original:  Normalization (Ioffe & Szegedy (2015))
Revised: Batch Normalization (Ioffe & Szegedy (2015))

Original: (BN) with a Whitening and Coloring (WC) transform (Hossain (2016))
Revised: (BN) with a Whitening and Coloring (WC) transform (Hossain (2016))

Original: in order to jointly speed-up training and increase the network representation capacity.
Revised: in order to jointly speed-up training and increase the network representation capacity.

Original: BN, proposed in (Ioffe & Szegedy (2015)) for discriminative tasks and then adopted in many discriminative and generative networks, is based on feature standardization of all the network layers.
Revised: BN, proposed in (Ioffe & Szegedy (2015)) for discriminative tasks and then adopted in many discriminative and generative networks, is based on feature standardization of all the network layers.

Original: The original motivation behind the introduction of BN is to alleviate the internal Covariate Shift problem caused by the continuous modiﬁcation of each layer’s input representation during training (Ioffe & Szegedy (2015)).
Revised: The original motivation behind the introduction of BN is to alleviate the internal Covariate Shift problem caused by the continuous modiﬁcation of each layer’s input representation during training (Ioffe & Szegedy (2015)).

Original: However, recent works (Santurkar et al.
Revised: However, recent works (Santurkar et al.

Original:  Kohler et al.
Revised: ; Kohler et al.

Original: (2018)) showed that a major motivation of the empirical success of BN relies also on a higher training stability, the latter being due to a better-conditioning of the Jacobian of the corresponding loss when BN is used compared to training without BN.
Revised: (2018)) showed that a major motivation of the empirical success of BN relies also on a higher training stability, the latter being due to a better-conditioning of the Jacobian of the corresponding loss when BN is used compared to training without BN.

Original: BN has very recenlty been extended in (Huang et al.
Revised: BN has very recently been extended in (Huang et al.

Original: (2018)) to full-feature decorrelation using ZCA whitening (Kessy et al.
Revised: to full-feature decorrelation using ZCA whitening (Kessy et al.

Original: in a discriminative scenario, showing promising results.
Revised: in a discriminative scenario.

Original: On the other hand, GAN training is known to be particularly unstable, and this instability problem is more severe than in discriminative networks.
Revised: On the other hand, GAN training is known to be particularly unstable, and this instability problem is more severe than in discriminative networks.

Original: Odena et al.
Revised: Odena et al.

Original: (2018) show the relation between stability of GAN training and conditioning of the Jacobian.
Revised: (2018) show the relation between stability of GAN training and conditioning of the Jacobian.

Original: This motivates our proposal to extend BN to full-feature whitening in a GAN scenario.
Revised: This motivates our proposal to extend BN to full-feature whitening in a GAN scenario (more details on this in Sec. 6).

Original: Differently from Huang et al.
Revised: Differently from Huang et al.

Original: (2018), our feature whitening is based on the Cholesky decomposition (Dereniowski & Marek (2004)), and we empirically show that our method is much faster and achieves better results in GAN training with respect to ZCA whitening.
Revised: (2018), our feature whitening is based on the Cholesky decomposition (Dereniowski & Marek (2004)), and we empirically show that our method is much faster and achieves better results in GAN training with respect to ZCA whitening.

Original: Importantly, while in (Huang et al.
Revised: Importantly, while in (Huang et al.

Original: feature whitening is followed by a per-dimension scaling and shifting of each feature, as in the original BN (Ioffe & Szegedy (2015)), we propose to use a coloring transform (Hossain (2016)) to keep the network representation capacity unchanged with respect to a non-whitened layer.
Revised: feature whitening is followed by a per-dimension scaling and shifting of each feature, as in the original BN (Ioffe & Szegedy (2015)), we propose to use a coloring transform (Hossain (2016)) to keep the network representation capacity unchanged with respect to a non-whitened layer.

Original: Our coloring projects each whitened feature vector in a new distribution using learnable multivariate ﬁlters and we show that this transformation is critical when using whitening for GANs.
Revised: Our coloring projects each whitened feature vector in a new distribution using learnable multivariate ﬁlters and we show that this transformation is critical when using whitening for GANs.

Original: The second contribution of this paper is based on the aforementioned coloring transform to represent class-speciﬁc information in a cGAN scenario.
Revised: The second contribution of this paper is based on the aforementioned coloring transform to represent class-speciﬁc information in a cGAN scenario.

Original: Formally, given a set of class labels Y =
Revised: Formally, given a set of class labels Y =

Original: { y 1 , ..., y n } , we want to generate an image
Revised: { y 1 , ..., y n } , we want to generate an image

Original: I = G
Revised: I = G

Original: ( z , y ) , where the generator G is input with a noise vector z and a class label y .
Revised: ( z , y ) , where the generator G is input with a noise vector z and a class label y .

Original: For instance, if y = cat , we want a foreground object in I depicting a cat, if y = dog , then I should represent a dog, etc.
Revised: For instance, if y = cat , we want a foreground object in I depicting a cat, if y = dog , then I should represent a dog, etc.

Original: The categorical conditional information y is commonly represented in G using conditional batch normalization (cBN) (Dumoulin et al. 
Revised: The categorical conditional information y is commonly represented in G using conditional batch normalization (cBN) (Dumoulin et al. (2016b); Gulrajani et al.

Original: (2016b)), where a class-speciﬁc pair of scaling-shifting parameters is learned for each class y .
Revised: (2017)), where a class-speciﬁc pair of scaling-shifting parameters is learned for each class y .

Original: In our proposed conditional Whitening and Coloring (cWC) the class-speciﬁc scaling-shifting parameters are replaced with a combination of class-agnostic and class-speciﬁc coloring ﬁlters.
Revised: In our proposed conditional Whitening and Coloring (cWC) the class-speciﬁc scalingshifting parameters are replaced with a combination of class-agnostic and class-speciﬁc coloring ﬁlters.

Original: The intuitive idea behind the proposed cWC is that multivariate transformation ﬁlters are more informative than the scalar parameters proposed in (Dumoulin et al.
Revised: The intuitive idea behind the proposed cWC is that multivariate transformation ﬁlters are more informative than the scalar parameters proposed in (Dumoulin et al.

Original: (2016b))
Revised: (2016b))

Original: and, thus, they can more accurately represent class-speciﬁc information.
Revised: and, thus, they can more accurately represent class-speciﬁc information.

Original: Finally, in order to have a set of class-speciﬁc coloring ﬁlters ( { Γ y } ) which grows sub-linearly with respect to the number of classes n , we represent each Γ y using a common, restricted dictionary of class-independent ﬁlters together with class-ﬁlter soft-assignment weights.
Revised: Finally, in order to have a set of class-speciﬁc coloring ﬁlters ( { Γ y } ) which grows sub-linearly with respect to the number of classes n , we represent each Γ y using a common, restricted dictionary of class-independent ﬁlters together with class-ﬁlter soft-assignment weights.

Original: The use of a classindependent dictionary of ﬁlters, together with a class-agnostic branch in the architecture of our cGAN generators (see Sec. 4) makes it possible to share parameters over all the classes and to reduce the ﬁnal number of weights which need to be learned.
Revised: The use of a classindependent dictionary of ﬁlters, together with a class-agnostic branch in the architecture of our cGAN generators (see Sec. 4) makes it possible to share parameters over all the classes and to reduce the ﬁnal number of weights which need to be learned.

Original: In summary, our contributions are the following:
Revised: In summary, our contributions are the following:

Original: • We propose a whitening-coloring transform for batch normalization to improve stability of GAN training which is based on the Cholesky decomposition and a re-projection of the whitened features in a learned distribution.
Revised: • We propose a whitening-coloring transform for batch normalization to improve stability of GAN training which is based on the Cholesky decomposition and a re-projection of the whitened features in a learned distribution.

Original: • In a cGAN framework we propose: (1) A class-speciﬁc coloring transform which generalizes cBN (Dumoulin et al.
Revised: • In a cGAN framework we propose: (1) A class-speciﬁc coloring transform which generalizes cBN (Dumoulin et al.

Original: (2016b))
Revised: (2016b))

Original: and (2) A soft-assignment of classes to ﬁlters to reduce the network complexity.
Revised: and (2) A soft-assignment of classes to ﬁlters to reduce the network complexity.

Original: • Using different datasets and basic GAN-cGAN frameworks, we show that our WC and cWC consistently improve the original frameworks’ results and training speed.
Revised: • Using different datasets and basic GAN-cGAN frameworks, we show that our WC and cWC consistently improve the original frameworks’ results and training speed.

Original: A link to our code will be made publicly available after the acceptance of this article.
Revised: A link to our code will be made publicly available after the acceptance of this article.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Batch Normalization.
Revised: Batch Normalization.

Original: BN (Ioffe & Szegedy (2015)) is based on a batch-based (per-dimension) standardization of all the network’s layers.
Revised: BN (Ioffe & Szegedy (2015)) is based on a batch-based (per-dimension) standardization of all the network’s layers.

Original: Formally, given a batch of d -dimensional samples B = { x 1 , ..., x m } , input to a given network layer, BN transforms each sample x i ∈ R d in B using:
Revised: Formally, given a batch of d -dimensional samples B = { x 1 , ..., x m } , input to a given network layer, BN transforms each sample x i ∈ R d in B using:

Original: [Equation]
Revised: [Equation]

Original: where: k ( 1 ≤ k ≤ d ) indicates the k -th dimension of the data, µ B,k and σ B,k are, respectively, the mean and the standard deviation computed with respect to the k -th dimension of the samples in B and (cid:15) is a constant used to prevent numerical instability.
Revised: where: k ( 1 ≤ k ≤ d ) indicates the k -th dimension of the data, µ B,k and σ B,k are, respectively, the mean and the standard deviation computed with respect to the k -th dimension of the samples in B and (cid:15) is a constant used to prevent numerical instability.

Original: Finally, γ k and β k are learnable scaling and shifting parameters, which are introduced in (Ioffe & Szegedy (2015))
Revised: Finally, γ k and β k are learnable scaling and shifting parameters, which are introduced in (Ioffe & Szegedy (2015))

Original: in order to prevent losing the original representation capacity of the network.
Revised: in order to prevent losing the original representation capacity of the network.

Original: More in detail, γ k and β k are used to guarantee that the BN transform can be inverted, if necessary, in order to represent the identity transform (Ioffe & Szegedy (2015)).
Revised: More in detail, γ k and β k are used to guarantee that the BN transform can be inverted, if necessary, in order to represent the identity transform (Ioffe & Szegedy (2015)).

Original: BN is commonly used in both discriminative and generative networks and has been extended in many directions.
Revised: BN is commonly used in both discriminative and generative networks and has been extended in many directions.

Original: For instance, Ba et al.
Revised: For instance, Ba et al.

Original: propose Layer Normalization, where normalization is performed using all the layer’s activation values of a single sample.
Revised: propose Layer Normalization, where normalization is performed using all the layer’s activation values of a single sample.

Original: Luo (2017) proposes to whiten the activation values using a learnable whitening matrix.
Revised: Luo (2017) proposes to whiten the activation values using a learnable whitening matrix.

Original: Differently from (Luo (2017)), our whitening matrix only depends on the speciﬁc batch of data.
Revised: Differently from (Luo (2017)), our whitening matrix only depends on the speciﬁc batch of data.

Original: Similarly to our work, batch-dependent full-feature decorrelation is performed also by Huang et al.
Revised: Similarly to our work, batch-dependent full-feature decorrelation is performed also by Huang et al.

Original: (2018), who use a ZCA-whitening and show promising results in a discriminative scenario.
Revised: (2018), who use a ZCA-whitening and show promising results in a discriminative scenario.

Original: However, the whitening part of our method is based on the Cholesky decomposition and we empirically show that, in a GAN scenario where training is known to be highly unstable, our method produces better results and it is much faster to compute.
Revised: However, the whitening part of our method is based on the Cholesky decomposition and we empirically show that, in a GAN scenario where training is known to be highly unstable, our method produces better results and it is much faster to compute.

Original: Moreover, while in (Huang et al.
Revised: Moreover, while in (Huang et al.

Original: only scaling and shifting parameters are used after feature decorrelation, we propose to use a coloring transform.
Revised: only scaling and shifting parameters are used after feature decorrelation, we propose to use a coloring transform.

Original: Analogously to the role played by γ k and β k in Eq. 1, coloring can potentially invert the whitening transform if that were the optimal thing to do, in this way preserving the original representation capacity of the network .
Revised: Analogously to the role played by γ k and β k in Eq. 1, coloring can potentially invert the whitening transform if that were the optimal thing to do, in this way preserving the original representation capacity of the network .

Original: We empirically show (see Sec. 5.1.1) that whitening without coloring performs poorly.
Revised: We empirically show (see Sec. 5.1.1) that whitening without coloring performs poorly.

Original: Last but not least, our conditional coloring can be used to represent rich class-speciﬁc information in a cGAN framework (see below).
Revised: Last but not least, our conditional coloring can be used to represent rich class-speciﬁc information in a cGAN framework (see below).

Original: Representing conditional information in cGANs.
Revised: Representing conditional information in cGANs.

Original: cGANs have been widely used to produce images conditioned on some input.
Revised: cGANs have been widely used to produce images conditioned on some input.

Original: For instance, Isola et al.
Revised: For instance, Isola et al.

Original: (2017) transform a given image in a second image represented in another “channel”.
Revised: (2017) transform a given image in a second image represented in another “channel”.

Original: Siarohin et al.
Revised: Siarohin et al.

Original: (2018) extend this framework to deal with structured conditional information (the human body pose).
Revised: (2018) extend this framework to deal with structured conditional information (the human body pose).

Original: In (Reed et al.
Revised: In (Reed et al.

Original: a textual description of an image is used to condition the image generation.
Revised: a textual description of an image is used to condition the image generation.

Original: When the conditioning data y is a categorical, unstructured variable, as in the case investigated in this paper, information about y can be represented in different ways.
Revised: When the conditioning data y is a categorical, unstructured variable, as in the case investigated in this paper, information about y can be represented in different ways.

Original: cBN (Dumoulin et al.
Revised: cBN (Dumoulin et al.

Original: (2016b)) uses y to select class-speciﬁc scaling and shifting parameters in the generator (more details in Sec. 4).
Revised: (2016b)) uses y to select class-speciﬁc scaling and shifting parameters in the generator (more details in Sec. 4).

Original: Another common approach is to represent y using a one-hot vector, which is concatenated with the input layer of both the discriminator and the generator (Mirza & Osindero (2014); Perarnau et al.
Revised: Another common approach is to represent y using a one-hot vector, which is concatenated with the input layer of both the discriminator and the generator (Mirza & Osindero (2014); Perarnau et al.

Original: or with the ﬁrst convolutional layer of the discriminator (Perarnau et al. (2016)).
Revised: or with the ﬁrst convolutional layer of the discriminator (Perarnau et al. (2016)).

Original: Miyato & Koyama (2018) split the last discriminator layer in two branches.
Revised: Miyato & Koyama (2018) split the last discriminator layer in two branches.

Original: The ﬁrst branch estimates the class-agnostic probability that the input image is real, while the second branch represents class-conditional information.
Revised: The ﬁrst branch estimates the class-agnostic probability that the input image is real, while the second branch represents class-conditional information.

Original: This solution is orthogonal to our proposed cWC and we show in Sec.
Revised: This solution is orthogonal to our proposed cWC and we show in Sec.

Original: 5.2 that the two approaches can be combined for boosting the results obtained in (Miyato & Koyama (2018)).
Revised: 5.2 that the two approaches can be combined for boosting the results obtained in (Miyato & Koyama (2018)).

Original:  T HE W HITENING AND C OLORING TRANSFORM FOR B ATCH N ORMALIZATION
Revised:  T HE W HITENING AND C OLORING TRANSFORM FOR B ATCH N ORMALIZATION

Original: We start describing the details of our Whitening and Coloring transform in the unconditional case, which we refer to as WC in the rest of the paper.
Revised: We start describing the details of our Whitening and Coloring transform in the unconditional case, which we refer to as WC in the rest of the paper.

Original: We always assume an image domain and convolutional generator/discriminator networks.
Revised: We always assume an image domain and convolutional generator/discriminator networks.

Original: However, most of the solutions we propose can potentially be applied to non-visual domains and non-convolutional networks.
Revised: However, most of the solutions we propose can potentially be applied to non-visual domains and non-convolutional networks.

Original: Let F ∈ R h × w ×
Revised: Let F ∈ R h × w ×

Original: d be the tensor representing the activation values of the convolutional feature maps for a given image and layer, with d channels and h × w spatial locations.
Revised: d be the tensor representing the activation values of the convolutional feature maps for a given image and layer, with d channels and h × w spatial locations.

Original: Similarly to BN, we consider the d -dimensional activations of each location in the h × w convolutional grid as a separate instance x i ∈ R d in B .
Revised: Similarly to BN, we consider the d -dimensional activations of each location in the h × w convolutional grid as a separate instance x i ∈ R d in B .

Original: This is done also to have a larger cardinality batch which alleviates instability issues when computing the batch-related parameters.
Revised: This is done also to have a larger cardinality batch which alleviates instability issues when computing the batch-related parameters.

Original: In fact, a mini-batch of m (cid:48) images corresponds to m =
Revised: In fact, a mini-batch of m (cid:48) images corresponds to m =

Original: | B | = m (cid:48) × h × w (Ioffe & Szegedy (2015)).
Revised: | B | = m (cid:48) × h × w (Ioffe & Szegedy (2015)).

Original: Using a vectorial notation, our proposed WC-based batch normalization is given by WC ( x i ) =
Revised: Using a vectorial notation, our proposed WC-based batch normalization is given by WC ( x i ) =

Original: Coloring (ˆ x i ) , where ˆ x i =
Revised: Coloring (ˆ x i ) , where ˆ x i =

Original: Whitening ( x i ) :
Revised: Whitening ( x i ) :

Original: [Equation]
Revised: [Equation]

Original: In Eq.
Revised: In Eq.

Original: 3, the vector µ B is the mean of the elements in B (being µ B,k in Eq.
Revised: 3, the vector µ B is the mean of the elements in B (being µ B,k in Eq.

Original: 1 its k -th component), while the matrix W B is such that: W (cid:62) B W B = Σ − 1 B , where Σ B is the covariance matrix computed using B .
Revised: 1 its k -th component), while the matrix W B is such that: W (cid:62) B W B = Σ − 1 B , where Σ B is the covariance matrix computed using B .

Original: This transformation performs the full whitening of x i and the resulting set of vectors ˆ
Revised: This transformation performs the full whitening of x i and the resulting set of vectors ˆ

Original: B = { ˆ x 1 , ..., ˆ x m } lies in a spherical distribution (i.e., with a covariance matrix equal to the identity matrix).
Revised: B = { ˆ x 1 , ..., ˆ x m } lies in a spherical distribution (i.e., with a covariance matrix equal to the identity matrix).

Original: Note that, similarly to µ B,k and σ B,k in Eq. 1, µ B and W B are completely data-dependent, being computed using only B and without any learnable parameter involved.
Revised: Note that, similarly to µ B,k and σ B,k in Eq. 1, µ B and W B are completely data-dependent, being computed using only B and without any learnable parameter involved.

Original: On the other hand,
Revised: On the other hand,

Original: Eq.
Revised: Eq.

Original: 2 performs the coloring transform (Hossain (2016)), which projects the elements in ˆ B onto a multivariate Gaussian distribution with an arbitrary covariance matrix (see Sec. 2 for the motivation behind this projection).
Revised: 2 performs the coloring transform (Hossain (2016)), which projects the elements in ˆ B onto a multivariate Gaussian distribution with an arbitrary covariance matrix (see Sec. 2 for the motivation behind this projection).

Original: Eq.
Revised: Eq.

Original: 2 is based on the learnable parameters β
Revised: 2 is based on the learnable parameters β

Original: = ( β 1 , ..., β k , ..., β d ) (cid:62) and Γ = (cid:16) ...
Revised: = ( β 1 , ..., β k , ..., β d ) (cid:62) and Γ = (cid:16) ...

Original: c k ... (cid:17) , where c k is a vector of parameters (a coloring ﬁlter ) which generalizes the scalar parameter γ k in Eq. 1.
Revised: c k ... (cid:17) , where c k is a vector of parameters (a coloring ﬁlter ) which generalizes the scalar parameter γ k in Eq. 1.

Original: Coloring is a linear operation and can be simply implemented using 1 × 1 × d convolutional ﬁlters c k and the corresponding biases β k .
Revised: Coloring is a linear operation and can be simply implemented using 1 × 1 × d convolutional ﬁlters c k and the corresponding biases β k .

Original: In Sec.
Revised: In Sec.

Original: 3.1 we show how to compute W B .
Revised: 3.1 we show how to compute W B .

Original: WC is plugged before each convolutional layer of our generators ( G ).
Revised: WC is plugged before each convolutional layer of our generators ( G ).

Original: In most of our experiments, we use a ResNet-like architecture (He et al.
Revised: In most of our experiments, we use a ResNet-like architecture (He et al.

Original: for G with 3 blocks (Gulrajani et al.
Revised: for G with 3 blocks (Gulrajani et al.

Original: and a ﬁnal convolutional layer (i.e. 3 × 2 + 1 = 7 convolutional layers, corresponding to 7 WC layers).
Revised: and a ﬁnal convolutional layer (i.e. 3 × 2 + 1 = 7 convolutional layers, corresponding to 7 WC layers).

Original: Fig.
Revised: Fig.

Original: 1(a) shows a simpliﬁed block scheme.
Revised: 1(a) shows a simpliﬁed block scheme.

Original: We do not use WC nor we introduce any novelty in the discriminators (more details in Sec. 5).
Revised: We do not use WC nor we introduce any novelty in the discriminators (more details in Sec. 5).

Original: Finally, at inference time, once the network has been trained, W B and µ B in Eq.
Revised: Finally, at inference time, once the network has been trained, W B and µ B in Eq.

Original: 3 are replaced with the expected W E and µ E , computed over the whole training set and approximated using the running average as in (Ioffe & Szegedy (2015)).
Revised: 3 are replaced with the expected W E and µ E , computed over the whole training set and approximated using the running average as in (Ioffe & Szegedy (2015)).

Original: In more detail, at each mini-batch training iteration t , we compute an updated version of Σ tE and µ tE by means of:
Revised: In more detail, at each mini-batch training iteration t , we compute an updated version of Σ tE and µ tE by means of:

Original: [Equation]
Revised: [Equation]

Original: When training is over, we use Σ E and the steps described in Sec.
Revised: When training is over, we use Σ E and the steps described in Sec.

Original: 3.1 to compute W E (speciﬁcally, we replace ˆΣ B with Σ E in Eq.
Revised: 3.1 to compute W E (speciﬁcally, we replace ˆΣ B with Σ E in Eq.

Original: 5, see below).
Revised: 5, see below).

Original: Note that this operation needs to be done only once, after that W E and µ E are ﬁxed and can be used for inference.
Revised: Note that this operation needs to be done only once, after that W E and µ E are ﬁxed and can be used for inference.

Original: 3. C OMPUTING THE WHITENING MATRIX USING THE C HOLESKY DECOMPOSITION
Revised: 3. C OMPUTING THE WHITENING MATRIX USING THE C HOLESKY DECOMPOSITION

Original: In order to compute the whitening matrix W B in Eq.
Revised: In order to compute the whitening matrix W B in Eq.

Original: 3, we ﬁrst need to compute the batch-dependent covariance matrix Σ B .
Revised: 3, we ﬁrst need to compute the batch-dependent covariance matrix Σ B .

Original: Since this computation may be unstable, we use the Shrinkage estimator (Schfer & Strimmer (2005)) which is based on blending the empirical covariance matrix ˆΣ B with a regularizing matrix (in our case we use I , the identity matrix):
Revised: Since this computation may be unstable, we use the Shrinkage estimator (Schfer & Strimmer (2005)) which is based on blending the empirical covariance matrix ˆΣ B with a regularizing matrix (in our case we use I , the identity matrix):

Original: [Equation]
Revised: [Equation]

Original: , where:
Revised: , where:

Original: [Equation]
Revised: [Equation]

Original: Once Σ B is computed, we use the Cholesky decomposition (Dereniowski & Marek (2004)) in order to compute W B in Eq.
Revised: Once Σ B is computed, we use the Cholesky decomposition (Dereniowski & Marek (2004)) in order to compute W B in Eq.

Original: 3 such that W (cid:62) B W B = Σ − 1 B .
Revised: 3 such that W (cid:62) B W B = Σ − 1 B .

Original: Note that the Cholesky decomposition is unique given Σ B and that it is differentiable, thus we can back-propagate the gradients through our whitening transform (more details in Appendix B).
Revised: Note that the Cholesky decomposition is unique given Σ B and that it is differentiable, thus we can back-propagate the gradients through our whitening transform (more details in Appendix B).

Original: Many modern platforms for deep-network developing such as TensorFlow TM include tools for computing the Cholesky decomposition.
Revised: Many modern platforms for deep-network developing such as TensorFlow TM include tools for computing the Cholesky decomposition.

Original: Moreover, its computational cost is smaller than other alternatives such as the ZCA whitening (Kessy et al.
Revised: Moreover, its computational cost is smaller than other alternatives such as the ZCA whitening (Kessy et al.

Original: (2017)) used, for instance, in (Huang et al.
Revised: (2017)) used, for instance, in (Huang et al.

Original: We empirically show in Sec.
Revised: We empirically show in Sec.

Original: 5.1.1 that, when used to train a GAN generator, our whitening transform is much faster and achieves better results than the ZCA whitening.
Revised: 5.1.1 that, when used to train a GAN generator, our whitening transform is much faster and achieves better results than the ZCA whitening.

Original: In more details, W B is computed using a lower triangular matrix L and the following steps:
Revised: In more details, W B is computed using a lower triangular matrix L and the following steps:

Original: We start with LL (cid:62) =
Revised: We start with LL (cid:62) =

Original: Σ B , which is equivalent to L −(cid:62)
Revised: Σ B , which is equivalent to L −(cid:62)

Original: L − 1 = Σ − 1 B . 2.
Revised: L − 1 = Σ − 1 B . 2.

Original: We use the Cholesky decomposition in order to compute L and L (cid:62) from Σ B . 3.
Revised: We use the Cholesky decomposition in order to compute L and L (cid:62) from Σ B . 3.

Original: We invert
Revised: We invert

Original: L and we eventually get W B =
Revised: L and we eventually get W B =

Original: L − 1 which is used in Eq. 3.
Revised: L − 1 which is used in Eq. 3.

Original: 3. C OMPUTATIONAL OVERHEAD
Revised: 3. C OMPUTATIONAL OVERHEAD

Original: The computational cost of Whitening () is given by the sum of computing ˆΣ B ( O ( md 2 ) ) plus the
Revised: The computational cost of Whitening () is given by the sum of computing ˆΣ B ( O ( md 2 ) ) plus the

Original: Cholesky decomposition ( O (2 d 3 ) ), the inversion of L ( O ( d 3 ) ) and the application of Eq.
Revised: Cholesky decomposition ( O (2 d 3 ) ), the inversion of L ( O ( d 3 ) ) and the application of Eq.

Original: 3 to the m samples in B ( O ( md 2 ) ).
Revised: 3 to the m samples in B ( O ( md 2 ) ).

Original: Note that optimized matrix multiplication can largely speed-up most of the involved operations.
Revised: Note that optimized matrix multiplication can largely speed-up most of the involved operations.

Original: On the other hand, Coloring () is implemented as a convolutional operation which can also be largely sped-up using common GPUs.
Revised: On the other hand, Coloring () is implemented as a convolutional operation which can also be largely sped-up using common GPUs.

Original: Overall, WC is O
Revised: Overall, WC is O

Original: ( d 3 + md 2 ) .
Revised: ( d 3 + md 2 ) .

Original: Using the aforementioned ResNet architecture with 7 WC layers, in our CIFAR-10 experiments ( 32 ×images), we need 0.82 seconds per iteration on average on a single NVIDIA Titan X. This includes both the forward and the backward pass for 1 generator and 5 discriminator updates (Gulrajani et al.
Revised: Using the aforementioned ResNet architecture with 7 WC layers, in our CIFAR-10 experiments ( 32 ×images), we need 0.82 seconds per iteration on average on a single NVIDIA Titan X. This includes both the forward and the backward pass for 1 generator and 5 discriminator updates (Gulrajani et al.

Original: In comparison, the standard BN, with the same architecture, takes on average 0.62 seconds.
Revised: In comparison, the standard BN, with the same architecture, takes on average 0.62 seconds.

Original: The ratio WC/BN is 1.32, which is a minor relative overhead.
Revised: The ratio WC/BN is 1.32, which is a minor relative overhead.

Original: At inference time, since W E and µ E are pre-computed, WC is reduced to the application of Eq.
Revised: At inference time, since W E and µ E are pre-computed, WC is reduced to the application of Eq.

Original: 3 and the convolutional operations implementing coloring.
Revised: 3 and the convolutional operations implementing coloring.

Original: [Figure]
Revised: [Figure]

Original:  C ONDITIONAL C OLORING TRANSFORM
Revised:  C ONDITIONAL C OLORING TRANSFORM

Original: In a cGAN framework, the conditional information y , input to G , is commonly represented using cBN (Dumoulin et al.
Revised: In a cGAN framework, the conditional information y , input to G , is commonly represented using cBN (Dumoulin et al.

Original: (2016b)), where Eq. 1 is replaced with:
Revised: (2016b)), where Eq. 1 is replaced with:

Original: [Equation]
Revised: [Equation]

Original: Comparing Eq.
Revised: Comparing Eq.

Original: 1 with Eq.
Revised: 1 with Eq.

Original: 6, the only difference lies in the class-speciﬁc γ and β parameters, which are used to project each standardized dimension into a class-speciﬁc univariate Gaussian distribution.
Revised: 6, the only difference lies in the class-speciﬁc γ and β parameters, which are used to project each standardized dimension into a class-speciﬁc univariate Gaussian distribution.

Original: We propose to replace Eq.
Revised: We propose to replace Eq.

Original: 6 with our conditional Whitening and Coloring transform (cWC):
Revised: 6 with our conditional Whitening and Coloring transform (cWC):

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation] (ˆ x i ) and ˆ x i =
Revised: CondColoring (ˆ x i ) and ˆ x i =

Original: Whitening ( x i ) , where:
Revised: Whitening ( x i ) , where:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term in Eq.
Revised: The ﬁrst term in Eq.

Original: 7 is based on class-speciﬁc learnable parameters Γ y and β y .
Revised: 7 is based on class-speciﬁc learnable parameters Γ y and β y .

Original: Analogously to (Dumoulin et al.
Revised: Analogously to (Dumoulin et al.

Original: (2016b)), in the forward-pass (both at training and at inference time), conditional information y , input to G , is used to select the correct (Γ y , β y ) pair to apply.
Revised: (2016b)), in the forward-pass (both at training and at inference time), conditional information y , input to G , is used to select the correct (Γ y , β y ) pair to apply.

Original: Similarly, during training, gradient information is backpropagated through only the corresponding pair used in the forward pass.
Revised: Similarly, during training, gradient information is backpropagated through only the corresponding pair used in the forward pass.

Original: The second term in Eq.
Revised: The second term in Eq.

Original: 7 is based on class-agnostic learnable parameters Γ and β , similarly to the unconditional coloring transform in Eq.2.
Revised: 7 is based on class-agnostic learnable parameters Γ and β , similarly to the unconditional coloring transform in Eq.

Original: The motivation behind the use of this second term is the higher instability of the class-speciﬁc (Γ y , β y ) parameter computation with respect to the corresponding class-agnostic weights.
Revised: This second term is used because the class-speciﬁc (Γ y , β y ) parameters are trained with less data then the corresponding class-agnostic weights.

Original: In fact, in a generative task with n classes and a training protocol with batches of m elements, on average only mn instances in B are associated with label y .
Revised: In fact, in a generative task with n classes and a training protocol with batches of m elements, on average only m n instances in B are associated with label y .

Original: Thus, on average, at each training iteration, the parameters (Γ y , β y ) are updated using an effective batch of mn elements versus a batch of m elements used for all the other class-independent parameters.
Revised: Thus, on average, at each training iteration, the parameters (Γ y , β y ) are updated using an effective batch of mn elements versus a batch of m elements used for all the other class-independent parameters.

Revised: The class-agnostic term is then used to enforce sharing of the parameters of the overall coloring transform.

Original: In Sec.
Revised: In Sec.

Original: 5.2.1 we empirically show the importance of this second term.
Revised: 5.2.1 we empirically show the importance of this second term.

Original: Fig. 1(b) shows the two separate network branches corresponding to the two terms in Eq.
Revised: Fig. 1(b) shows the two separate network branches corresponding to the two terms in Eq.

Original: 4. C LASS - FILTER SOFT ASSIGNMENT
Revised: 4. C LASS - FILTER SOFT ASSIGNMENT

Original: When the number of classes n is small and the number of real training images is relatively large, learning a separate Γ y for each y is not a problem.
Revised: When the number of classes n is small and the number of real training images is relatively large, learning a separate Γ y for each y is not a problem.

Original: However, with a large n and/or few training data, this may lead to an over-complex generator network which can be hard to train.
Revised: However, with a large n and/or few training data, this may lead to an over-complex generator network which can be hard to train.

Original: Indeed, the dimensionality of each Γ y is d × d and learning nd 2 parameters may be difﬁcult with a large n .
Revised: Indeed, the dimensionality of each Γ y is d × d and learning nd 2 parameters may be difﬁcult with a large n .

Original: In order to solve this issue, when n is large, we compute Γ y using a weighted sum over the elements of a class-independent dictionary D .
Revised: In order to solve this issue, when n is large, we compute Γ y using a weighted sum over the elements of a class-independent dictionary D .

Original: Each row of D contains a ﬂattened version of a Γ matrix.
Revised: Each row of D contains a ﬂattened version of a Γ matrix.

Original: More formally, D is an s × d 2 matrix, with s << n .
Revised: More formally, D is an s × d 2 matrix, with s << n .

Original: The j -th row in D is given by the concatenation of d d -dimensional ﬁlters:
Revised: The j -th row in D is given by the concatenation of d d -dimensional ﬁlters:

Original: D j =
Revised: D j =

Original: [ c j 1 , ..., c jk , ...
Revised: [ c j 1 , ..., c jk , ...

Original: c jd ] , being c jk ∈
Revised: c jd ] , being c jk ∈

Original: R
Revised: R

Original: d a coloring ﬁlter for the k -th output dimension.
Revised: d a coloring ﬁlter for the k -th output dimension.

Original: D is shared over all the classes.
Revised: D is shared over all the classes.

Original: On the other hand, given y as input, Γ y is computed using:
Revised: On the other hand, given y as input, Γ y is computed using:

Original: [Equation]
Revised: [Equation]

Original: where y is a one-hot representation of class y (i.e., a vector of zero elements except one in the y -th component) and it is used to select the y -th row ( A y ) of the n × s association matrix A .
Revised: where y is a one-hot representation of class y (i.e., a vector of zero elements except one in the y -th component) and it is used to select the y -th row ( A y ) of the n × s association matrix A .

Original: During training, given y , only the weights in A y are updated (i.e., all the other rows A y (cid:48) , with y (cid:48) (cid:54) =
Revised: During training, given y , only the weights in A y are updated (i.e., all the other rows A y (cid:48) , with y (cid:48) (cid:54) =

Original: y are not used and do not change).
Revised: y are not used and do not change).

Original: However, all the elements in D are updated independently of y .
Revised: However, all the elements in D are updated independently of y .

Original: In the rest of the paper, we call (plain) cWC the version in which n different class-speciﬁc matrices Γ y are learned, one per class, and cWC sa the version in which Γ y is computed using Eq. 8.
Revised: In the rest of the paper, we call (plain) cWC the version in which n different class-speciﬁc matrices Γ y are learned, one per class, and cWC sa the version in which Γ y is computed using Eq. 8.

Original: In all our experiments (Sec. 5.2), we ﬁx s = (cid:100)√ n (cid:101) .
Revised: In all our experiments (Sec. 5.2), we ﬁx s = (cid:100)√ n (cid:101) .

Original: The computational cost of both versions ( cWC and cWC sa ) is basically the same as WC (Sec. 3.2).
Revised: The computational cost of both versions ( cWC and cWC sa ) is basically the same as WC (Sec. 3.2).

Original: In cWC sa , the only additional overhead of the class-speciﬁc branch is given by the computation of A (cid:62) y D , which is O ( sd 2 ) .
Revised: In cWC sa , the only additional overhead of the class-speciﬁc branch is given by the computation of A (cid:62) y D , which is O ( sd 2 ) .

Original: We again point out that optimized matrix multiplication can largely speed-up most of the involved operations.
Revised: We again point out that optimized matrix multiplication can largely speed-up most of the involved operations.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We split our experiments in two main scenarios: unconditional and conditional image generation.
Revised: We split our experiments in two main scenarios: unconditional and conditional image generation.

Original: In each scenario we ﬁrst compare our approach with the state of the art and then we analyze the impact of each element of our proposal.
Revised: In each scenario we ﬁrst compare our approach with the state of the art and then we analyze the impact of each element of our proposal.

Original: We use 4 datasets: CIFAR-10, CIFAR-100 (Torralba et al.
Revised: We use 4 datasets: CIFAR-10, CIFAR-100 (Torralba et al.

Original: , STL-10 (Coates et al.
Revised: , STL-10 (Coates et al.

Original: and Tiny ImageNet 1 .
Revised: and Tiny ImageNet 1 .

Original: The latter is composed of 64 ×images taken from 200 ImageNet categories (500 images per class).
Revised: The latter is composed of 64 ×images taken from 200 ImageNet categories (500 images per class).

Original: In our evaluation we use the two most common metrics for image generation tasks: Inception Score (IS) (Salimans et al.
Revised: In our evaluation we use the two most common metrics for image generation tasks: Inception Score (IS) (Salimans et al.

Original: (the higher the better) and the Fr ´ echet Inception Distance (FID) (Heusel et al.
Revised: (the higher the better) and the Fr ´ echet Inception Distance (FID) (Heusel et al.

Original: (the lower the better).
Revised: (the lower the better).

Original: In the Appendix we also show: (1) a comparison based on human judgments as an additional evaluation criterion, (2) qualitative results for all the datasets used in this section, (3) additional experiments on other smaller datasets.
Revised: In the Appendix we also show: (1) a comparison based on human judgments as an additional evaluation criterion, (2) qualitative results for all the datasets used in this section, (3) additional experiments on other smaller datasets.

Original: In our experiments we use different generator and discriminator architectures in combination with different discriminator training protocols and we plug our WC/cWC in the generator of each adopted framework.
Revised: In our experiments we use different generator and discriminator architectures in combination with different discriminator training protocols and we plug our WC/cWC in the generator of each adopted framework.

Original: The goal is to show that our whitening-coloring transforms can consistently improve the quality of the results with respect to different basic GAN networks, training protocols and and image generation tasks.
Revised: The goal is to show that our whitening-coloring transforms can consistently improve the quality of the results with respect to different basic GAN networks, training protocols and and image generation tasks.

Original: Note that, if not otherwise explicitly mentioned, we usually adopt the same hyperparameter setting of the framewoks we compare with.
Revised: Note that, if not otherwise explicitly mentioned, we usually adopt the same hyperparameter setting of the framewoks we compare with.

Original: Altough better results can be achieved using, for instance, an ad-hoc learning rate policy (see Sec.
Revised: Altough better results can be achieved using, for instance, an ad-hoc learning rate policy (see Sec.

Original: 5.2 and Tab. 2 (right))
Revised: 5.2 and Tab. 2 (right))

Original: , we want to emphasize that our whitening-coloring transforms can be easily plugged into existing systems.
Revised: , we want to emphasize that our whitening-coloring transforms can be easily plugged into existing systems.

Original: In all our experiments, WC , cWC and cWC sa , as well as the simpliﬁed baselines presented in the ablation studies, are applied before each convolutional layer of our generators.
Revised: In all our experiments, WC , cWC and cWC sa , as well as the simpliﬁed baselines presented in the ablation studies, are applied before each convolutional layer of our generators.

Original: 5. U NCONDITIONAL IMAGE GENERATION
Revised: 5. U NCONDITIONAL IMAGE GENERATION

Original: In the unconditional scenario we use two different training strategies for the discriminator : (1)
Revised: In the unconditional scenario we use two different training strategies for the discriminator : (1)

Original: WGAN with Gradient Penalty (Gulrajani et al.
Revised: WGAN with Gradient Penalty (Gulrajani et al.

Original: ( GP ) and (2) GAN with Spectral Normalization (Miyato et al.
Revised: ( GP ) and (2) GAN with Spectral Normalization (Miyato et al.

Original: ( SN ).
Revised: ( SN ).

Original: Remind that our WC is used only in the generator (Sec. 3).
Revised: Remind that our WC is used only in the generator (Sec. 3).

Original: When
Revised: When

Original: WC is used in combination with GP , we call it WC GP , while, in combination with SN , we refer to it as WC SN .
Revised: WC is used in combination with GP , we call it WC GP , while, in combination with SN , we refer to it as WC SN .

Original: Moreover, we use either ResNet (He et al.
Revised: Moreover, we use either ResNet (He et al.

Original: ; Gulrajani et al.
Revised: ; Gulrajani et al.

Original: or
Revised: or

Original: DCGAN (Radford et al.
Revised: DCGAN (Radford et al.

Original: as the basic generator architecture.
Revised: as the basic generator architecture.

Original: In order to make the comparison with the corresponding GP and SN frameworks as fair as possible and to show that the quantitative result improvements are due only to our batch normalization approach, we strictly follow the architectural and training protocol details reported in (Gulrajani et al.
Revised: In order to make the comparison with the corresponding GP and SN frameworks as fair as possible and to show that the quantitative result improvements are due only to our batch normalization approach, we strictly follow the architectural and training protocol details reported in (Gulrajani et al.

Original: ; Miyato et al.
Revised: ; Miyato et al.

Original: For instance, in WC SN + DCGAN we use the same DCGAN-based generator and discriminator used in SN +
Revised: For instance, in WC SN + DCGAN we use the same DCGAN-based generator and discriminator used in SN +

Original: DCGAN (Miyato et al.
Revised: DCGAN (Miyato et al.

Original: (2018)), with the same number of feature-map dimension ( d ) in each layer in both the generator and the discriminator, the same learning rate policy (which was optimized for SN + DCGAN but not for our framework), etc.
Revised: (2018)), with the same number of feature-map dimension ( d ) in each layer in both the generator and the discriminator, the same learning rate policy (which was optimized for SN + DCGAN but not for our framework), etc.

Original: We provide implementation details in Appendix A.1.
Revised: We provide implementation details in Appendix A.1.

Original: [Figure]
Revised: [Figure]

Original: In Tab. 1 we show that WC improves both the IS and the FID values of all the tested frameworks in both the CIFAR-10 and the STL-10 dataset.
Revised: In Tab. 1 we show that WC improves both the IS and the FID values of all the tested frameworks in both the CIFAR-10 and the STL-10 dataset.

Original: Speciﬁcally, independently of the basic architecture (either ResNet or DCGAN), our WC transform improves the corresponding SN results.
Revised: Speciﬁcally, independently of the basic architecture (either ResNet or DCGAN), our WC transform improves the corresponding SN results.

Original: Similarly, our WC GP results are better than the corresponding GP values.
Revised: Similarly, our WC GP results are better than the corresponding GP values.

Original: [Table]
Revised: [Table]

Original: In Tab. 2 (left) we compare WC SN + ResNet with other methods on CIFAR-10 and we show that we reach results almost on par with the state of the art on this dataset (Karras et al.
Revised: In Tab. 2 (left) we compare WC SN + ResNet with other methods on CIFAR-10 and we show that we reach results almost on par with the state of the art on this dataset (Karras et al.

Original: Note that the average IS values over 10 runs reported in Karras et al.
Revised: Note that the average IS values over 10 runs reported in Karras et al.

Original: (2018) is exactly the same as our average
Revised: (2018) is exactly the same as our average

Original: IS value: 8.56 (the results reported in the table refer to the best values for each framework).
Revised: IS value: 8.56 (the results reported in the table refer to the best values for each framework).

Original: Moreover, the method proposed in Karras et al.
Revised: Moreover, the method proposed in Karras et al.

Original: (2018) is based on a much more sophisticated generator architecture and training protocol.
Revised: (2018) is based on a much more sophisticated generator architecture and training protocol.

Original: Finally
Revised: Finally

Original: [Table]
Revised: [Table]

Original: in Fig.
Revised: in Fig.

Original: 2 we plot the IS and the FID values computed at different mini-batch training iterations.
Revised: 2 we plot the IS and the FID values computed at different mini-batch training iterations.

Original: These plots empirically show that the proposed batch normalization approach signiﬁcantly speeds-up the training process.
Revised: These plots empirically show that the proposed batch normalization approach signiﬁcantly speeds-up the training process.

Original: For instance, the IS value of WC SN + ResNet after 20k iterations is already higher than the IS value of SN + ResNet after 50k iterations.
Revised: For instance, the IS value of WC SN + ResNet after 20k iterations is already higher than the IS value of SN + ResNet after 50k iterations.

Original: [Table]
Revised: [Table]

Original: 5.1. A BLATION STUDY
Revised: 5.1. A BLATION STUDY

Original: We use CIFAR-10 to analyze the inﬂuence of each WC element and an SN + ResNet framework (Miyato et al.
Revised: We use CIFAR-10 to analyze the inﬂuence of each WC element and an SN + ResNet framework (Miyato et al.

Original: , omitting ’SN’ and ’ResNet’ from the names for simplicity.
Revised: , omitting ’SN’ and ’ResNet’ from the names for simplicity.

Original: Our initial baseline is W-only , in which we use Eq.
Revised: Our initial baseline is W-only , in which we use Eq.

Original: 3 to project the features in a spherical distribution without coloring .
Revised: 3 to project the features in a spherical distribution without coloring .

Original: In WC-diag , after whitening (Eq. 3), we use Eq. 2 with diagonal Γ matrices.
Revised: In WC-diag , after whitening (Eq. 3), we use Eq. 2 with diagonal Γ matrices.

Original: This corresponds to learning scalar γ - β parameters for a dimension-wise scaling-shifting transform as in
Revised: This corresponds to learning scalar γ - β parameters for a dimension-wise scaling-shifting transform as in

Original: BN (Eq. 1).
Revised: BN (Eq. 1).

Original: On the other hand, in std-C we standardize the features as in BN (Eq. 1) but replacing the γ - β parameters with our coloring transform (Eq. 2) and in C-only , Eq. 2 is applied directly to the original, non-normalized batch instances.
Revised: On the other hand, in std-C we standardize the features as in BN (Eq. 1) but replacing the γ - β parameters with our coloring transform (Eq. 2) and in C-only , Eq. 2 is applied directly to the original, non-normalized batch instances.

Original: Finally, WC is our full-pipeline.
Revised: Finally, WC is our full-pipeline.

Original: The results reported in Tab.
Revised: The results reported in Tab.

Original: 3 show that feature whitening without coloring performs poorly.
Revised: 3 show that feature whitening without coloring performs poorly.

Original: This is probably due to the fact that the activation values, once normalized with W-only , have zero mean and thus, on average, half of them are “cut” by the subsequent ReLU nonlinearity.
Revised: This is probably due to the fact that the activation values, once normalized with W-only , have zero mean and thus, on average, half of them are “cut” by the subsequent ReLU nonlinearity.

Original: After the feature normalization step, it is crucial for the network to re-project the features in a new (learned) distribution with a sufﬁcient representation capacity.
Revised: After the feature normalization step, it is crucial for the network to re-project the features in a new (learned) distribution with a sufﬁcient representation capacity.

Original: This is done in standard BN using the γ - β parameters.
Revised: This is done in standard BN using the γ - β parameters.

Original: Analogously, in our WC, this role is played by the coloring step.
Revised: Analogously, in our WC, this role is played by the coloring step.

Original: In fact, with a diagonal-matrix based coloring, WC-diag improves the results with respect to W-only .
Revised: In fact, with a diagonal-matrix based coloring, WC-diag improves the results with respect to W-only .

Original: Moreover, a drastic improvement is obtained by the full-pipeline ( WC ) with respect to WC-diag , showing the importance of learning a full covariance matrix in the coloring step.
Revised: Moreover, a drastic improvement is obtained by the full-pipeline ( WC ) with respect to WC-diag , showing the importance of learning a full covariance matrix in the coloring step.

Original: On the other hand, the comparison between
Revised: On the other hand, the comparison between

Original: WC with both C-only and std-C shows that feature normalization before coloring is also important, and that the full advantage of coloring is obtained when used in combination with whitening.
Revised: WC with both C-only and std-C shows that feature normalization before coloring is also important, and that the full advantage of coloring is obtained when used in combination with whitening.

Original: In the same table we report the results obtained by replacing our Cholesky decomposition (Sec. 3.1) with the ZCA-based whitening ( W zca C ), which is used in (Huang et al.
Revised: In the same table we report the results obtained by replacing our Cholesky decomposition (Sec. 3.1) with the ZCA-based whitening ( W zca C ), which is used in (Huang et al.

Original: (2018)), keeping all the rest of our method unchanged (coloring included).
Revised: (2018)), keeping all the rest of our method unchanged (coloring included).

Original: The comparison with WC shows that the procedure proposed in Sec.
Revised: The comparison with WC shows that the procedure proposed in Sec.

Original: 3.1 achieves better results than W zca C .
Revised: 3.1 achieves better results than W zca C .

Revised: The results reported in Tab.

Revised: 3 correspond to the best values achieved by W zca C during training and in Appendix D we show that W zca C can be highly unstable.

Original: Moreover, W zca C is, on average, much slower than WC.
Revised: Moreover, W zca C is, on average, much slower than WC.

Original: Speciﬁcally, the whitening only part, when computed using the procedure proposed in Sec. 3.1, is more than 11 times faster than the ZCA-based whitening, while, considering the whole pipeline (most of which is shared between WC and W zca C ), WC is more than 3.5 times faster than W zca C .
Revised: Speciﬁcally, the whitening only part, when computed using the procedure proposed in Sec. 3.1, is more thantimes faster than the ZCA-based whitening, while, considering the whole pipeline (most of which is shared between WC and W zca C ), WC is more than 3.5 times faster than W zca C .

Original: However, note that in the Decorrelated Batch Normalization (DBN) (Huang et al.
Revised: However, note that in the Decorrelated Batch Normalization (DBN) (Huang et al.

Original: (2018)), no coloring is used and the ZCA-based whitening is applied to only the ﬁrst layer of a classiﬁcation network, trained in a standard supervised fashion.
Revised: (2018)), no coloring is used and the ZCA-based whitening is applied to only the ﬁrst layer of a classiﬁcation network, trained in a standard supervised fashion.

Original: Hence, W zca C has to be considered as our proposed WC in which whitening is performed using the procedure proposed in (Huang et al.
Revised: Hence, W zca C has to be considered as our proposed WC in which whitening is performed using the procedure proposed in (Huang et al.

Original: Although investigating discriminative scenarios is out of the scope of this paper, in

Original: Appendix D we adapted our unconditional WC to work in classiﬁcation networks and we show that, in a direct comparison with DBN, we obtain comparable results

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: 5. C ONDITIONAL IMAGE GENERATION
Revised: 5. C ONDITIONAL IMAGE GENERATION

Original: In the cGAN scenario, conditional information is represented in the generator as explained in Sec.and using either cWC or cWC sa (Sec. 4.1).
Revised: In the cGAN scenario, conditional information is represented in the generator as explained in Sec.and using either cWC or cWC sa (Sec. 4.1).

Original: The purpose of this section is to show the advantage of using our conditional coloring with respect to other cGAN methods for representing categorical information input to a generator.
Revised: The purpose of this section is to show the advantage of using our conditional coloring with respect to other cGAN methods for representing categorical information input to a generator.

Original: In Tab. 5 we use different basic discriminator and training-protocol frameworks and we report results with and without our conditional batch normalization approach in the corresponding generator.
Revised: In Tab. 5 we use different basic discriminator and training-protocol frameworks and we report results with and without our conditional batch normalization approach in the corresponding generator.

Original: Similarly to the experiments in Sec. 5.1, in our cWC and cWC sa based GANs we follow the implementation details of the reference frameworks and use their hyper-parameters (e.g., the learning rate policy, etc.) which are not tuned for our method.
Revised: Similarly to the experiments in Sec. 5.1, in our cWC and cWC sa based GANs we follow the implementation details of the reference frameworks and use their hyper-parameters (e.g., the learning rate policy, etc.) which are not tuned for our method.

Original: The only difference with respect to the approaches we compare with (apart from our conditional batch normalization) is a different ratio between the number of ﬁlters in the discriminator and those used in the generator in the SN + Proj.
Revised: The only difference with respect to the approaches we compare with (apart from our conditional batch normalization) is a different ratio between the number of ﬁlters in the discriminator and those used in the generator in the SN + Proj.

Original: Discr.
Revised: Discr.

Original: experiments (but keeping constant the total number of ﬁlters, see Appendix A.2 for more details).
Revised: experiments (but keeping constant the total number of ﬁlters, see Appendix A.2 for more details).

Original: Tab.shows that both cWC and cWC sa improve the IS and the FID values with respect to the basic frameworks in both the CIFAR-10 and the CIFAR-100 dataset.
Revised: Tab.shows that both cWC and cWC sa improve the IS and the FID values with respect to the basic frameworks in both the CIFAR-10 and the CIFAR-100 dataset.

Original: On CIFAR-10, cWC outperforms cWC sa , probably because, with a small number of classes ( n ), learning class-speciﬁc coloring ﬁlters ( Γ y ) is relatively easy and more informative than using a class-independent dictionary.
Revised: On CIFAR-10, cWC outperforms cWC sa , probably because, with a small number of classes ( n ), learning class-speciﬁc coloring ﬁlters ( Γ y ) is relatively easy and more informative than using a class-independent dictionary.

Original: However, with n = 200 , as in the case of the Tiny ImageNet dataset, cWC sa obtains signiﬁcantly better results than cWC (see Tab. 4).
Revised: However, with n = 200 , as in the case of the Tiny ImageNet dataset, cWC sa obtains signiﬁcantly better results than cWC (see Tab. 4).

Original: In Tab. 4, cBN corresponds to our re-implementation of (Miyato & Koyama (2018)), with a ResNet generator + cBN (Dumoulin et al.
Revised: In Tab. 4, cBN corresponds to our re-implementation of (Miyato & Koyama (2018)), with a ResNet generator + cBN (Dumoulin et al.

Original: (2016b))
Revised: (2016b))

Original: and an SN + ResNet + Projection
Revised: and an SN + ResNet + Projection

Original: Discriminator (Miyato & Koyama (2018)).
Revised: Discriminator (Miyato & Koyama (2018)).

Original: Note that both cWC and cWC sa signiﬁcantly improve both the IS and the FID values with respect to cBN on this large and challenging dataset.
Revised: Note that both cWC and cWC sa signiﬁcantly improve both the IS and the FID values with respect to cBN on this large and challenging dataset.

Original: Finally, in Tab. 2 (right) we compare cWC and cWC sa with other methods on CIFAR-10, where cWC achieves the best so far published supervised results on this dataset.
Revised: Finally, in Tab. 2 (right) we compare cWC and cWC sa with other methods on CIFAR-10, where cWC achieves the best so far published supervised results on this dataset.

Original: Note that cWC SN +
Revised: Note that cWC SN +

Original: Proj.
Revised: Proj.

Original: Discr.
Revised: Discr.

Original: in Tab. 2 (right) and in Tab. 5 indicate exactly the same framework, the only difference being the learning rate policy (see Appendix A.2).
Revised: in Tab. 2 (right) and in Tab. 5 indicate exactly the same framework, the only difference being the learning rate policy (see Appendix A.2).

Original: In fact, to emphasize the comparison, in Tab.
Revised: In fact, to emphasize the comparison, in Tab.

Original: we adopted the same learning rate policy used by Miyato & Koyama (2018), which has been customized to our approach only in Tab.
Revised: we adopted the same learning rate policy used by Miyato & Koyama (2018), which has been customized to our approach only in Tab.

Original: 2 (right).
Revised: 2 (right).

Original: [Table]
Revised: [Table]

Original: 5.2. A BLATION STUDY
Revised: 5.2. A BLATION STUDY

Original: We analyze all the main aspects of the conditional coloring introduced in Sec.
Revised: We analyze all the main aspects of the conditional coloring introduced in Sec.

Original: 4 using an SN +
Revised: 4 using an SN +

Original: ResNet + Proj.
Revised: ResNet + Proj.

Original: Discr.
Revised: Discr.

Original: framework for all the baselines presented here.
Revised: framework for all the baselines presented here.

Original: The whitening part (when used) is obtained using Eq. 3.
Revised: The whitening part (when used) is obtained using Eq. 3.

Original: We call cWC-cls-only a version of CondColoring () with only the class-speciﬁc term in Eq.
Revised: We call cWC-cls-only a version of CondColoring () with only the class-speciﬁc term in Eq.

Original: This is implemented simply removing the class-agnostic branch in
Revised: This is implemented simply removing the class-agnostic branch in

Original: Fig. 1(b) and learning a speciﬁc ( Γ y , β y ) pair per class, without class-ﬁlter soft assignment.
Revised: Fig. 1(b) and learning a speciﬁc ( Γ y , β y ) pair per class, without class-ﬁlter soft assignment.

Original: cWC sacls-only is the same as cWC-cls-only but using the soft assignment in Eq. 8.
Revised: cWC sacls-only is the same as cWC-cls-only but using the soft assignment in Eq. 8.

Original: Similarly [Table] we call cWCdiag the baseline in which each Γ y in Eq.
Revised: Similarly, we call cWCdiag the baseline in which each Γ y in Eq.

Original: 7 is replaced with a diagonal Γ y matrix.
Revised: 7 is replaced with a diagonal Γ y matrix.

Original: This corresponds to learning a set of d independent scalar parameters for each class y
Revised: This corresponds to learning a set of d independent scalar parameters for each class y

Original: ( { ( γ y,k , β y,k ) } dk =1 ) to represent class-speciﬁc information similarly to cBN (see Eq. 6).
Revised: ( { ( γ y,k , β y,k ) } dk =1 ) to represent class-speciﬁc information similarly to cBN (see Eq. 6).

Original: In cWC-diag we also use the class-agnostic branch.
Revised: In cWC-diag we also use the class-agnostic branch.

Original: Note that a class-agnostic branch without any kind of class-speciﬁc parameters cannot represent at all class information in the generator.
Revised: Note that a class-agnostic branch without any kind of class-speciﬁc parameters cannot represent at all class information in the generator.

Original: As a reference, in the ﬁrst row of Tab. 6 we also report the results of exactly the same framework but with our cWC replaced with cBN (Dumoulin et al.
Revised: As a reference, in the ﬁrst row of Tab. 6 we also report the results of exactly the same framework but with our cWC replaced with cBN (Dumoulin et al.

Original: (2016b)).
Revised: (2016b)).

Original: Moreover, we call c-std-C a version of our method in which feature normalization is performed using standardization instead of whitening and then the two coloring branches described in Sec.
Revised: Moreover, we call c-std-C a version of our method in which feature normalization is performed using standardization instead of whitening and then the two coloring branches described in Sec.

Original: (Eq. 7) are applied to the standardized features.
Revised: (Eq. 7) are applied to the standardized features.

Original: Finally, c-std-C sa corresponds to the softassignment version of our conditional coloring (Sec. 4.1) used together with feature standardization.
Revised: Finally, c-std-C sa corresponds to the softassignment version of our conditional coloring (Sec. 4.1) used together with feature standardization.

Original: We ﬁrst analyze the whitening-based baselines (second part of Tab. 6).
Revised: We ﬁrst analyze the whitening-based baselines (second part of Tab. 6).

Original: The results reported in
Revised: The results reported in

Original: Tab. 6 show that both versions based on a single coloring branch ( cls-only ) are signiﬁcantly worse than cWC-diag which includes the class-agnostic branch.
Revised: Tab. 6 show that both versions based on a single coloring branch ( cls-only ) are signiﬁcantly worse than cWC-diag which includes the class-agnostic branch.

Original: On the other hand, while on CIFAR10 cWC-diag , cWC and cWC sa reach comparable results, on CIFAR-100 cWC-diag drastically underperforms the other two versions which are based on full-coloring matrices.
Revised: On the other hand, while on CIFAR10 cWC-diag , cWC and cWC sa reach comparable results, on CIFAR-100 cWC-diag drastically underperforms the other two versions which are based on full-coloring matrices.

Original: This shows the representation capacity of our conditional coloring, which is emphasized in CIFAR-100 where the number of classes ( n ) is one order of magnitude larger than in CIFAR-10:
Revised: This shows the representation capacity of our conditional coloring, which is emphasized in CIFAR-100 where the number of classes ( n ) is one order of magnitude larger than in CIFAR-10:

Original: With a large n , the scalar “ γ ” parameters are not sufﬁcient to represent class-speciﬁc information.
Revised: With a large n , the scalar “ γ ” parameters are not sufﬁcient to represent class-speciﬁc information.

Original: Also the comparison between cWC and cWC sa depends on n , and the advantage of using cWC sa instead of cWC is more evident on the Tiny ImageNet experiment (see Tab.4).
Revised: Also the comparison between cWC and cWC sa depends on n , and the advantage of using cWC sa instead of cWC is more evident on the Tiny ImageNet experiment (see Tab.4).

Original: Finally, comparing the standardization-based versions c-std-C and c-std-C sa with the corresponding whitening-based ones ( cWC and cWC sa ), the importance of full-feature decorrelation before coloring becomes clear.
Revised: Finally, comparing the standardization-based versions c-std-C and c-std-C sa with the corresponding whitening-based ones ( cWC and cWC sa ), the importance of full-feature decorrelation before coloring becomes clear.

Revised: Note that c-std-C and c-std-C sa have exactly the same number of parameters than cWC and cWC sa , respectively, which shows that the advantage of our method does not depend only on the increase of the ﬁlter number.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C OMPARISON WITH A DISCRIMINATIVE SCENARIO

Revised: In this section we plug our WC into the classiﬁcation networks used in (Huang et al. and we compare with their DBN (Sec. 2) in a discriminative scenario. DBN basically consists of a ZCAbased feature whitening followed by a scaling-and-shifting transform (Sec. 2).

Revised: Huang et al.

Revised: (2018) plug their DBN in the ﬁrst layer of different ResNet architectures (He et al.

Revised: with varying depth.

Revised: DBN is based also on grouping , which consists in computing smaller (and, hence, more stable) covariance matrices by grouping together different feature dimensions.

Revised: For a fair comparison, we also plug our WC in only the ﬁrst layer of each network but we do not use grouping.

Revised: Tab. 7 shows the results averaged over 5 runs.

Revised: Both WC and DBN achieve lower errors than BN in all the conﬁgurations.

Revised: DBN is slightly better than WC, however the absolute error difference is quite marginal.

Revised: Moreover, note that the maximum error difference over all the tested conﬁgurations is lower than 1% .

Revised: This empirical analysis, compared with the signiﬁcant boost we obtain using WC and cWC in Sec. 5, shows that full-feature whitening and coloring is more helpful in a GAN scenario than in a discriminative setting.

Revised: As mentioned in Sec.

Original: In this paper we proposed a whitening and coloring transform which extends both BN (Ioffe &
Revised: 1-2, we believe that this is due to the higher instability of GAN training. Feature normalization makes the loss landscape smoother (Santurkar et al. Kohler et al. (2018)), making the gradient-descent weight updates closer to Newton updates (Huang et al.

Original: Szegedy (2015)) and cBN (Dumoulin et al. 
Revised: ; LeCun et al. (2012); S. Wiesler (2011)).

Revised: Since full-feature whitening performs a more principled normalization, this helps adversarial training in which instability issues are more critical than in a discriminative scenario (Odena et al.

Original: (2016b))
Revised: [Table]

Original:  for GAN/cGAN training, respectively.
Revised: In this paper we proposed a whitening and coloring transform which extends both BN and cBN for GAN/cGAN training, respectively.

Original: In the ﬁrst case, we generalize BN introducing full-feature whitening and a multivariate feature reprojection.
Revised: In the ﬁrst case, we generalize BN introducing full-feature whitening and a multivariate feature re-projection.

Original: In the second case, we exploit our learnable feature re-projection after normalization to represent strong categorical conditional information in the generator.
Revised: In the second case, we exploit our learnable feature re-projection after normalization to represent signiﬁcant categorical conditional information in the generator.

Original: Our empirical results show that the proposed approach can speed-up the training process and improve the image generation quality of different basic frameworks on different datasets in both the conditional and the unconditional scenario.
Revised: Our empirical results show that the proposed approach can speed-up the training process and improve the image generation quality of different basic frameworks on different datasets in both the conditional and the unconditional scenario.

Original: A I MPLEMENTATION DETAILS
Revised: A I MPLEMENTATION DETAILS

Original: In our experiments we use 2 basic network architectures: ResNet (Gulrajani et al.
Revised: In our experiments we use 2 basic network architectures: ResNet (Gulrajani et al.

Original: and DCGAN (Miyato et al.
Revised: and DCGAN (Miyato et al.

Original: We strictly follow the training protocol details reported in (Gulrajani et al.
Revised: We strictly follow the training protocol details reported in (Gulrajani et al.

Original: ; Miyato et al.
Revised: ; Miyato et al.

Original: and provided in the two subsections below.
Revised: and provided in the two subsections below.

Original: Moreover, we adopt the same architectural details suggested in (Gulrajani et al.
Revised: Moreover, we adopt the same architectural details suggested in (Gulrajani et al.

Original: ; Miyato et al.
Revised: ; Miyato et al.

Original: (e.g., the number of blocks, the number of 3 × 3 convolutional ﬁlters, etc.), except the use of our WC/cWC layers in our generators.
Revised: (e.g., the number of blocks, the number of 3 × 3 convolutional ﬁlters, etc.), except the use of our WC/cWC layers in our generators.

Original: We use the same hyper-parameter setting of the works we compare with, a choice which likely favours the baselines used for comparison.
Revised: We use the same hyper-parameter setting of the works we compare with, a choice which likely favours the baselines used for comparison.

Original: The only difference with respect to the architectural setup in (Gulrajani et al.
Revised: The only difference with respect to the architectural setup in (Gulrajani et al.

Original: ; Miyato et al.
Revised: ; Miyato et al.

Original: (2018)) is emphasized in Appendix A.2 and concerns a different ratio between the number of 3 × 3 convolutional ﬁlters in the generator versus the number of discriminator’s convolutional ﬁlters in the SN + Proj.
Revised: (2018)) is emphasized in Appendix A.2 and concerns a different ratio between the number of 3 × 3 convolutional ﬁlters in the generator versus the number of discriminator’s convolutional ﬁlters in the SN + Proj.

Original: Discr experiments.
Revised: Discr experiments.

Original: A. U NCONDITIONAL IMAGE GENERATION EXPERIMENTS
Revised: A. U NCONDITIONAL IMAGE GENERATION EXPERIMENTS

Original: In all our GP -based experiments we use Adam with learning rate α = 2 e - 4 , ﬁrst momentum β 1 =and second momentum β 2 = 0 .
Revised: In all our GP -based experiments we use Adam with learning rate α = 2 e - 4 , ﬁrst momentum β 1 =and second momentum β 2 = 0 .

Original: The learning rate is linearly decreased till it reaches 0.
Revised: The learning rate is linearly decreased till it reaches 0.

Original: In all the GP -based experiments we use the WGAN loss + Gradient Penalty (Gulrajani et al.
Revised: In all the GP -based experiments we use the WGAN loss + Gradient Penalty (Gulrajani et al.

Original: (2017)), while in the SN -based experiment we use the hinge loss (Miyato et al.
Revised: (2017)), while in the SN -based experiment we use the hinge loss (Miyato et al.

Original: In all the GP experiments we train the networks for 100k iterations following (Gulrajani et al.
Revised: In all the GP experiments we train the networks for 100k iterations following (Gulrajani et al.

Original: In the
Revised: In the

Original: CIFAR-10 experiments, SN + ResNet was trained for 50k iterations (Miyato et al.
Revised: CIFAR-10 experiments, SN + ResNet was trained for 50k iterations (Miyato et al.

Original: In the
Revised: In the

Original: STL-10 experiments, we train the networks for 100k iterations following (Miyato et al.
Revised: STL-10 experiments, we train the networks for 100k iterations following (Miyato et al.

Original: Each iteration is composed of 1 generator update with batch size 128 and 5 discriminator updates with batch size 64+64 (i.e., 64 generated and 64 real images), same numbers are used in (Gulrajani et al.
Revised: Each iteration is composed of 1 generator update with batch size 128 and 5 discriminator updates with batch size 64+64 (i.e., 64 generated and 64 real images), same numbers are used in (Gulrajani et al.

Original: ; Miyato et al.
Revised: ; Miyato et al.

Original: When training the DCGAN architectures we use two times more iterations.
Revised: When training the DCGAN architectures we use two times more iterations.

Original: However, in DCGAN each iteration is composed of 1 generator update with batch sizeand 1 discriminator update with batch size 64 + 

Revised: However, in DCGAN each iteration is composed of 1 generator update with batch sizeand 1 discriminator update with batch size 64 +

Original: (Miyato et al.
Revised: (Miyato et al.

Original: In all the experiments we use the original image resolution, except for STL-10 where we downsample all images to 48 ×following (Miyato & Koyama (2018)).
Revised: In all the experiments we use the original image resolution, except for STL-10 where we downsample all images to 48 ×following (Miyato & Koyama (2018)).

Original: We provide below the architectural details of our networks.
Revised: We provide below the architectural details of our networks.

Original: We denote with C Sd a block consisting of the following layer sequence: (1) either a WC or a cWC layer (depending on the unconditional/conditional scenario, respectively), (2) a ReLU and (3) a convolutional layer with d output ﬁlters.
Revised: We denote with C Sd a block consisting of the following layer sequence: (1) either a WC or a cWC layer (depending on the unconditional/conditional scenario, respectively), (2) a ReLU and (3) a convolutional layer with d output ﬁlters.

Original: The output of this block has the same ( S ) spatial resolution of the input.
Revised: The output of this block has the same ( S ) spatial resolution of the input.

Original: C Dd is similar to C Sd , but the output feature map is downsampled ( D ): the convolutional layer uses stride = 2 .
Revised: C Dd is similar to C Sd , but the output feature map is downsampled ( D ): the convolutional layer uses stride = 2 .

Original: C Ud is an upconvolution ( U ) block with fractional stride = 0 .
Revised: C Ud is an upconvolution ( U ) block with fractional stride = 0 .

Original: We use a similar notation for the
Revised: We use a similar notation for the

Original: ResNet blocks: R Sd is a ResNet block (see Fig.1(a)) which preserves spatial resolution.
Revised: ResNet blocks: R Sd is a ResNet block (see Fig.1(a)) which preserves spatial resolution.

Original: R Dd downsamples the feature maps and R Ud upsamples the feature maps.
Revised: R Dd downsamples the feature maps and R Ud upsamples the feature maps.

Original: Also we denote with: (1) D k a dense layer with k outputs, (2) G a global-sum-pooling layer, (3) F a ﬂattening layer, (4) P h a layer that permutes the dimensions of the input feature map to the target spatial resolution h × h .
Revised: Also we denote with: (1) D k a dense layer with k outputs, (2) G a global-sum-pooling layer, (3) F a ﬂattening layer, (4) P h a layer that permutes the dimensions of the input feature map to the target spatial resolution h × h .

Original: Given the above deﬁnitions, we show in Tab. {+
+}
Revised: Given the above deﬁnitions, we show in Tab. {+
+}

Original: the architectures used for the unconditional image generation experiments.
Revised: the architectures used for the unconditional image generation experiments.

Original: We again emphasize that the methods we compare with in Tab. 1 have the same network structure except the use of WC layers in the generator.
Revised: We again emphasize that the methods we compare with in Tab. 1 have the same network structure except the use of WC layers in the generator.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: soft assignment.
Revised: soft assignment.

Original: A. C ONDITIONAL IMAGE GENERATION EXPERIMENTS
Revised: A. C ONDITIONAL IMAGE GENERATION EXPERIMENTS

Original: In the conditional image generation experiments, most of the hyper-parameter values are the same as in Appendix A.1.
Revised: In the conditional image generation experiments, most of the hyper-parameter values are the same as in Appendix A.1.

Original: In the CIFAR-10 experiments with GP + ACGAN , we train our networks for 100k iterations following (Gulrajani et al.
Revised: In the CIFAR-10 experiments with GP + ACGAN , we train our networks for 100k iterations following (Gulrajani et al.

Original: In the CIFAR-10 and CIFAR-100 experiments with SN and Projection Discriminator we train our networks for 50k iterations following (Miyato & Koyama (2018)).
Revised: In the CIFAR-10 and CIFAR-100 experiments with SN and Projection Discriminator we train our networks for 50k iterations following (Miyato & Koyama (2018)).

Original: In the Tiny ImageNet experiments, we train our networks for 100k iterations.
Revised: In the Tiny ImageNet experiments, we train our networks for 100k iterations.

Original: All the other hyper-parameters are the same as in Appendix A.1.
Revised: All the other hyper-parameters are the same as in Appendix A.1.

Original: Tab. 8 shows the architectures used in the conditional image generation experiments.
Revised: Tab. 9 shows the architectures used in the conditional image generation experiments.

Original: Both cWC SN + Proj.
Revised: Both cWC SN + Proj.

Original: Discr.
Revised: Discr.

Original: and cWC sa SN + Proj.
Revised: and cWC sa SN + Proj.

Original: Discr.
Revised: Discr.

Original: used in CIFAR-10 and CIFAR-100 (see
Revised: used in CIFAR-10 and CIFAR-100 (see

Original: Tab. 5) have an opposite generator/discriminator ﬁlter-number ratio with respect to the ratio used in the original SN + Proj.
Revised: Tab. 5) have an opposite generator/discriminator ﬁlter-number ratio with respect to the ratio used in the original SN + Proj.

Original: Discr.
Revised: Discr.

Original: by Miyato & Koyama (2018) ( 128 / 256 vs. 256 / 128 ).
Revised: by Miyato & Koyama (2018) ( 128 / 256 vs. 256 / 128 ).

Original: In fact, we decrease the number of 3 × 3 convolutional ﬁlters in each generator block from 256, used in (Miyato & Koyama (2018)), to 128.
Revised: In fact, we decrease the number of 3 × 3 convolutional ﬁlters in each generator block from 256, used in (Miyato & Koyama (2018)), to 128.

Original: Simultaneously, we increase the number of 3 × 3 convolutional ﬁlters in the discriminator from 128 (Miyato & Koyama (2018)) to 256.
Revised: Simultaneously, we increase the number of 3 × 3 convolutional ﬁlters in the discriminator from 128 (Miyato & Koyama (2018)) to 256.

Original: This is the only difference between our cWC-based frameworks and (Miyato & Koyama (2018)) in the experiments of Tab. 5.
Revised: This is the only difference between our cWC-based frameworks and (Miyato & Koyama (2018)) in the experiments of Tab. 5.

Original: The reason behind this architectural choice is that we hypothesize that our cWC generators are more powerful than the corresponding discriminators.
Revised: The reason behind this architectural choice is that we hypothesize that our cWC generators are more powerful than the corresponding discriminators.

Original: Using the same generator/discriminator ﬁlter ratio as in (Miyato & Koyama (2018)), we perform on par with SN + Proj.
Revised: Using the same generator/discriminator ﬁlter ratio as in (Miyato & Koyama (2018)), we perform on par with SN + Proj.

Original: Discr.
Revised: Discr.

Original: (Miyato & Koyama (2018)).
Revised: (Miyato & Koyama (2018)).

Original: In Tab. 6, cBN refers to SN + Proj.
Revised: In Tab. 6, cBN refers to SN + Proj.

Original: Discr.
Revised: Discr.

Original: with the same aforementioned 128 / 256 ﬁlter ratio and used in the cWC baselines.
Revised: with the same aforementioned 128 / 256 ﬁlter ratio and used in the cWC baselines.

Original: In Fig.
Revised: In Fig.

Original: 3 we show a schematic representation of the main baselines reported in Tab. 6.
Revised: 3 we show a schematic representation of the main baselines reported in Tab. 6.

Original: Finally, in Tab. 2 (right) and in Tab. 5, cWC SN + Proj.
Revised: Finally, in Tab. 2 (right) and in Tab. 5, cWC SN + Proj.

Original: Discr. indicates exactly the same framework, the only difference being a different learning rate policy.
Revised: Discr. indicates exactly the same framework, the only difference being a different learning rate policy.

Original: The results of cWC SN + Proj.
Revised: The results of cWC SN + Proj.

Original: Discr.
Revised: Discr.

Original: reported in Tab. 5 have been obtained using the same learning rate policy used by Miyato & Koyama (2018).
Revised: reported in Tab. 5 have been obtained using the same learning rate policy used by Miyato & Koyama (2018).

Original: However, our model already achieved its peak performance after 30k iterations.
Revised: However, our model already achieved its peak performance after 30k iterations.

Original: Hence, we decreased the learning rate by a factor of 10 at iteration 30k
Revised: Hence, we decreased the learning rate by a factor of 10 at iteration 30k

Original: and then we trained cWC SN + Proj.
Revised: and then we trained cWC SN + Proj.

Original: Discr.
Revised: Discr.

Original: for additional 1k iterations.
Revised: for additional 1k iterations.

Original: This brought to a performance boost, as reported in Tab.
Revised: This brought to a performance boost, as reported in Tab.

Original: (right).
Revised: (right).

Original: A. E VALUATION METRICS
Revised: A. E VALUATION METRICS

Original: In our experiments we use 2 evaluation metrics: Inception Score (IS) (Salimans et al.
Revised: In our experiments we use 2 evaluation metrics: Inception Score (IS) (Salimans et al.

Original: and the Fr´echet Inception Distance (FID) (Heusel et al.
Revised: and the Fr´echet Inception Distance (FID) (Heusel et al.

Original: These metrics are the two most widely adopted evaluation criteria for generative methods.
Revised: These metrics are the two most widely adopted evaluation criteria for generative methods.

Original: While most of generative works report quantitative results using only one of these two criteria, we chose to report them both to show the agreement of these two metrics: in most of the presented comparisons and ablation studies, when IS is larger,
Revised: While most of generative works report quantitative results using only one of these two criteria, we chose to report them both to show the agreement of these two metrics: in most of the presented comparisons and ablation studies, when IS is larger,

Original: FID is lower.
Revised: FID is lower.

Original: IS was originally introduced by Salimans et al.
Revised: IS was originally introduced by Salimans et al.

Original: (2016), who suggest to measure IS using 5k generated images, repeat this generation 10 times and then average the results.
Revised: (2016), who suggest to measure IS using 5k generated images, repeat this generation 10 times and then average the results.

Original: We follow exactly the same protocol.
Revised: We follow exactly the same protocol.

Original: FID is a more recent metric and the evaluation protocol is less consolidated.
Revised: FID is a more recent metric and the evaluation protocol is less consolidated.

Original: Miyato & Koyama (2018) suggest to compute FID using 10k images from the test set together with 5k generated images.
Revised: Miyato & Koyama (2018) suggest to compute FID using 10k images from the test set together with 5k generated images.

Original: Heusel et al.
Revised: Heusel et al.

Original: (2017) suggest to use a sample sizes of at least 10k real and 10k generated images.
Revised: (2017) suggest to use a sample sizes of at least 10k real and 10k generated images.

Original: In all our tables, FID 5k refers to the former protocol (Miyato & Koyama (2018)) and FID 10k refers to the second protocol (Heusel et al.
Revised: In all our tables, FID 5k refers to the former protocol (Miyato & Koyama (2018)) and FID 10k refers to the second protocol (Heusel et al.

Original: B B ACK - PROPAGATION
Revised: B B ACK - PROPAGATION

Original: In our implementation, the gradient back-propagation through our WC/cWC layers is obtained by relying on the TensorFlow TM automatic differentiation.
Revised: In our implementation, the gradient back-propagation through our WC/cWC layers is obtained by relying on the TensorFlow TM automatic differentiation.

Original: However, for completeness and to make our method fully-reproducible, in this section we provide a reverse mode automatic differentiation (AD) (Giles (2008); Ioffe & Szegedy (2015);
Revised: However, for completeness and to make our method fully-reproducible, in this section we provide a reverse mode automatic differentiation (AD) (Giles (2008); Ioffe & Szegedy (2015);

Original: Walter (2012)) of the Whitening back-propagation steps.
Revised: Walter (2012)) of the Whitening back-propagation steps.

Original: Note that, since (conditional and unconditional) Coloring is implemented using convolution operations, back-propagation for Coloring is straightforward.
Revised: Note that, since (conditional and unconditional) Coloring is implemented using convolution operations, back-propagation for Coloring is straightforward.

Original: In the following we adopt the notation used by Giles (2008) that we shortly present below.
Revised: In the following we adopt the notation used by Giles (2008) that we shortly present below.

Original: Given a scalar function of a matrix f
Revised: Given a scalar function of a matrix f

Original: ( A ) , ¯ A indicates the reverse mode AD sensitivities with respect to A ,
Revised: ( A ) , ¯ A indicates the reverse mode AD sensitivities with respect to A ,

Original: i.e.: ¯ A =
Revised: i.e.: ¯ A =

Original: ∂f ( A ) ∂A .
Revised: ∂f ( A ) ∂A .

Original: Given a batch B , we subtract the mean µ B from each sample x i ∈ B .
Revised: Given a batch B , we subtract the mean µ B from each sample x i ∈ B .

Original: Reverse mode AD meansubtraction is trivial and described in (Ioffe & Szegedy (2015)).
Revised: Reverse mode AD meansubtraction is trivial and described in (Ioffe & Szegedy (2015)).

Original: We then stack all the meansubtracted samples in a matrix X ∈ R
Revised: We then stack all the meansubtracted samples in a matrix X ∈ R

Original: d
Revised: d

Original: × m (where we drop the subscript B for simplicity).
Revised: × m (where we drop the subscript B for simplicity).

Original: Using this notation, the empirical covariance matrix in Eq.
Revised: Using this notation, the empirical covariance matrix in Eq.

Original: 5 is given by: Σ
Revised: 5 is given by: Σ

Original: [Equation]
Revised: [Equation]

Original: In the Cholesky decomposition we have: Σ =
Revised: In the Cholesky decomposition we have: Σ =

Original: L T L , W =
Revised: L T L , W =

Original: L − 1 .
Revised: L − 1 .

Original: We can rewrite Eq.
Revised: We can rewrite Eq.

Original: 3 as a matrix multiplication and obtain Y
Revised: 3 as a matrix multiplication and obtain Y

Original: =
Revised: =

Original: WX , where i -th column of Y is ˆ x i .
Revised: WX , where i -th column of Y is ˆ x i .

Original: In reverse mode AD, ¯ Y is given and we need to compute ¯ X .
Revised: In reverse mode AD, ¯ Y is given and we need to compute ¯ X .

Original: Using the formulas in (Giles (2008))
Revised: Using the formulas in (Giles (2008))

Original: we obtain:
Revised: we obtain:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Now, using the formulas in (Walter (2012)):
Revised: Now, using the formulas in (Walter (2012)):

Original: [Equation]
Revised: [Equation]

Original: where:
Revised: where:

Original: [Equation]

Original:  =    · · ·   ...
Revised: P =    · · ·   ...

Original: · · ·    and ◦ is Hadamard product.
Revised: · · ·    and ◦ is Hadamard product.

Original: Referring again to (Giles (2008)), and combining the reverse sensitives with respect to X from the 2 streams we get:[Equation]
Revised: Referring again to (Giles (2008)), and combining the reverse sensitives with respect to X from the 2 streams we get:

Original: [Equation]
Revised: [Equation]

Original: C U SER S TUDY
Revised: C U SER S TUDY

Original: In order to further emphasize the improvement we have when using cWC sa instead of cBN, we show in Tab.
Revised: In order to further emphasize the improvement we have when using cWC sa instead of cBN, we show in Tab.

Original: 9 a user study performed with 21 persons.
Revised: 10 a user study performed with 21 persons.

Original: Speciﬁcally, the dataset used is Tiny ImageNet and the cBN baseline is the same used for the experiments reported in Tab. 4.
Revised: Speciﬁcally, the dataset used is Tiny ImageNet and the cBN baseline is the same used for the experiments reported in Tab. 4.

Original: We show to each user 50 paired image-sets (using 50 randomly selected classes), generated using either cWC sa or cBN.
Revised: We show to each user 50 paired image-sets (using 50 randomly selected classes), generated using either cWC sa or cBN.

Original: Each image-set consists of 16 stacked images of the same class, used to increase the robustness of the user’s choice.
Revised: Each image-set consists of 16 stacked images of the same class, used to increase the robustness of the user’s choice.

Original: The user selects the image-set which he/she believes is more realistic.
Revised: The user selects the image-set which he/she believes is more realistic.

Original: These results clearly show the superiority of cWC sa against cBN.
Revised: These results clearly show the superiority of cWC sa against cBN.

Original: [Table]
Revised: [Table]

Original: D D ISCRIMINATIVE EXPERIMENTS
Revised: D C OMPARISON BETWEEN THE C HOLESKY AND THE ZCA- BASED WHITENING PROCEDURES

Original: Although the purpose of this paper is to show that feature whitening and coloring improves training in an adversarial framework (GAN and cGAN), characterized by a high instability (see Sec.
Revised: In this section we compare the stability properties of our Cholesky decomposition with the ZCAbased whitening proposed in (Huang et al.

Original: 1-2), we also apply our WC to the classiﬁcation networks used in (Huang et al.
Revised: Speciﬁcally, we use W zca C which is the variant of our method introduced in Sec.

Original: and we compare with their DBN in a discriminative scenario.
Revised: 5.1.1, where we replace our Cholesky decomposition with the

Original: DBN basically consists of a ZCA-based feature whitening followed by a scaling-and-shifting transform (Sec. 2).
Revised: ZCA-based whitening.

Original: Huang et al.
Revised: In Fig.

Original: (2018) plug their DBN in the ﬁrst layer of different ResNet architectures (He et al.
Revised: 4 we plot the IS/training-iteration curves corresponding to both WC and W zca C .

Original: with varying depth.
Revised: This graph shows that W zca C is highly unstable, since training can frequently degenerate.

Original: DBN is based also on grouping , which consists in computing smaller (and, hence, more stable) covariance matrices by grouping together different feature dimensions. For a fair comparison, we also plug our WC in only the ﬁrst layer of each network but we do not use grouping.
Revised: When this happens, W zca C collapses to a model that always produces a constant, uniform grey image. The reason for this behaviour is likely related to the way in which the gradients are backpropagated through the ZCA-based whitening step.

Original: Tab. 10 shows the results averaged over 5 runs.
Revised: Speciﬁcally, following (Huang et al.

Original: Both WC and DBN achieve lower errors than BN in all the conﬁgurations.
Revised: (2018)), the gradient of the loss with respect to the covariance matrix depends from the inverse of the difference of the covariance-matrix singular values (Huang et al. (2018), Appendix A.2).

Original: DBN is slightly better than WC, however the absolute error difference is quite marginal.
Revised: As a consequence, if some of the singular values are identical or very close to each other, this computation is ill-conditioned.

Original: We refer to Sec. 1-2 for the methodological differences between DBN and WC and cWC.
Revised: Conversely, our whitening procedure based on the Cholesky decomposition (Sec. 3.1) has never showed these drastic instability phenomena (Fig. 4).

Original: E E XPERIMENTS USING SMALL - SIZE DATASETS
Revised: E E XPERIMENTS USING SMALL - SIZE DATASETS

Original: In this section we show additional experiments using two small-size datasets: the MNIST (LeCun et al.
Revised: In this section we show additional experiments using two small-size datasets: the MNIST (LeCun et al.

Original: and the Fashion-MNIST dataset (Xiao et al.
Revised: and the Fashion-MNIST dataset (Xiao et al.

Original: (see Fig.
Revised: (see Fig.

Original: 7 for some generated image examples).
Revised: 9 for some generated image examples).

Original: In both cases we use the SN + ResNet architecture described in Tab. 11.
Revised: In both cases we use the SN + ResNet architecture described in Tab. 11.

Original: On
Revised: On

Original:  MNIST we train for 5k iterations with a constant learning rate, while on Fashion-MNIST we train[Table] for 20k iterations with a linearly decaying learning rate (see Appendix A.1).
Revised: [Figure] MNIST we train for 5k iterations with a constant learning rate, while on Fashion-MNIST we train for 20k iterations with a linearly decaying learning rate (see Appendix A.1).

Original: All the other hyperparameters are the same as in Appendix A.1.
Revised: All the other hyperparameters are the same as in Appendix A.1.

Original: The results are reported in Tab. 12.
Revised: The results are reported in Tab. 12.

Original: Unconditional image generation on MNIST is the only setup in which we have not observed an improvement using our WC.
Revised: Unconditional image generation on MNIST is the only setup in which we have not observed an improvement using our WC.

Original: However, on the more challenging
Revised: However, on the more challenging

Original: Fashion-MNIST dataset, we did observe an improvement, especially in the conditional scenario.
Revised: Fashion-MNIST dataset, we did observe an improvement, especially in the conditional scenario.

Original: [Table]
Revised: [Table]

Revised: and the Fashion-MNIST datasets

Original: [Table]
Revised: [Table]

Original: F Q UALITATIVE RESULTS
Revised: F Q UALITATIVE RESULTS

Original: In this section we show some qualitative results obtained using our WC/cWC methods.
Revised: In this section we show some qualitative results obtained using our WC/cWC methods.

Original: Speciﬁcally, examples of images generated using the CIFAR-10 dataset are shown in Fig.
Revised: Speciﬁcally, examples of images generated using the CIFAR-10 dataset are shown in Fig.

Original: Fig.
Revised: Fig.

Original: 5 (a)-Fig.(b) show examples generated on the STL-10 and the CIFAR-100 dataset, respectively.
Revised: 6 (a)-Fig.(b) show examples generated on the STL-10 and the CIFAR-100 dataset, respectively.

Original: In Fig.
Revised: In Fig.

Original: 6 we show Tiny ImageNet samples.
Revised: 8 we show Tiny ImageNet samples, while Fig.

Revised: 9 shows MNIST and Fashion-MNIST generated images.

Original: Finally,  Fig.
Revised: Finally, in Fig.

Original: 7 shows MNIST and Fashion-MNIST generated images.
Revised: 7 we show some images generated using CIFAR-10 and CIFAR-100 in which the value of the noise vector z is kept ﬁxed while varying the conditioning class y .

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]