Original: G ENERATIVE NETWORKS AS INVERSE PROBLEMS WITH SCATTERING TRANSFORMS
Revised: G ENERATIVE NETWORKS AS INVERSE PROBLEMS WITH SCATTERING TRANSFORMS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Generative Adversarial Nets (GANs) and Variational Auto-Encoders (VAEs) provide impressive image generations from Gaussian white noise, but the underlying mathematics are not well understood.
Revised: Generative Adversarial Nets (GANs) and Variational Auto-Encoders (VAEs) provide impressive image generations from Gaussian white noise, but the underlying mathematics are not well understood.

Original: We compute deep convolutional network generators by inverting a ﬁxed embedding operator.
Revised: We compute deep convolutional network generators by inverting a ﬁxed embedding operator.

Original: Therefore, they do not require to be optimized with a discriminator or an encoder.
Revised: Therefore, they do not require to be optimized with a discriminator or an encoder.

Original: The embedding is Lipschitz continuous to deformations so that generators transform linear interpolations between input white noise vectors into deformations between output images.
Revised: The embedding is Lipschitz continuous to deformations so that generators transform linear interpolations between input white noise vectors into deformations between output images.

Original: This embedding is computed with a wavelet scattering transform.
Revised: This embedding is computed with a wavelet scattering transform.

Original: Numerical experiments demonstrate that the resulting scattering generators have similar properties as GANs or VAEs, without learning a discriminative network or an encoder.
Revised: Numerical experiments demonstrate that the resulting scattering generators have similar properties as GANs or VAEs, without learning a discriminative network or an encoder.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Generative Adversarial Networks (GANs) and Variational Auto-Encoders (VAEs) allow training generative networks to synthesize images of remarkable quality and complexity from Gaussian white noise.
Revised: Generative Adversarial Networks (GANs) and Variational Auto-Encoders (VAEs) allow training generative networks to synthesize images of remarkable quality and complexity from Gaussian white noise.

Original: This work shows that one can train generative networks having similar properties to those obtained with GANs or VAEs without learning a discriminator or an encoder.
Revised: This work shows that one can train generative networks having similar properties to those obtained with GANs or VAEs without learning a discriminator or an encoder.

Original: The generator is a deep convolutional network that inverts a predeﬁned embedding operator which is Lipschitz continuous to deformations in order to reproduce relevant properties of GAN image synthesis.
Revised: The generator is a deep convolutional network that inverts a predeﬁned embedding operator which is Lipschitz continuous to deformations in order to reproduce relevant properties of GAN image synthesis.

Original: This embedding operator is implemented with a wavelet scattering transform.
Revised: This embedding operator is implemented with a wavelet scattering transform.

Original: Deﬁning image generators as the solution of an inverse problem provides a mathematical framework, which is closer to standard probabilistic models such as Gaussian autoregressive models.
Revised: Deﬁning image generators as the solution of an inverse problem provides a mathematical framework, which is closer to standard probabilistic models such as Gaussian autoregressive models.

Original: GANs were introduced by Goodfellow et al.
Revised: GANs were introduced by Goodfellow et al.

Original: (2014) as an unsupervised learning framework to estimate implicit generative models of complex data (such as natural images) by training a generative model (the generator) and a discriminative model (the discriminator) simultaneously.
Revised: (2014) as an unsupervised learning framework to estimate implicit generative models of complex data (such as natural images) by training a generative model (the generator) and a discriminative model (the discriminator) simultaneously.

Original: A generative model of the random vector X is an operator (cid:98) G which transforms a Gaussian white noise random vector Z into a model (cid:98) X of X ,
Revised: A generative model of the random vector X is an operator (cid:98) G which transforms a Gaussian white noise random vector Z into a model (cid:98) X of X ,

Original: i.e., (cid:98) X = (cid:98)
Revised: i.e., (cid:98) X = (cid:98)

Original: G ( Z ) .
Revised: G ( Z ) .

Original: When (cid:98) G is a deep convolutional network it is called a generative network.
Revised: When (cid:98) G is a deep convolutional network it is called a generative network.

Original: Radford et al.
Revised: Radford et al.

Original: (2016) have introduced deep convolutional architectures for the generator and the discriminator, which result in high-quality image synthesis.
Revised: (2016) have introduced deep convolutional architectures for the generator and the discriminator, which result in high-quality image synthesis.

Original: When the latent variable z is linearly modiﬁed the resulting image (cid:98) x = (cid:98) G ( z ) is progressively deformed.
Revised: When the latent variable z is linearly modiﬁed the resulting image (cid:98) x = (cid:98) G ( z ) is progressively deformed.

Original: Mathematical models as in Goodfellow et al.
Revised: Mathematical models as in Goodfellow et al.

Original: (2014) and Arjovsky et al.
Revised: (2014) and Arjovsky et al.

Original: (2016) argue that GANs select the generator (cid:98) G by minimizing estimations of the Jensen-Shannon divergence or the Wasserstein distance calculated from empirical estimations of these distances with generated and training images.
Revised: (2016) argue that GANs select the generator (cid:98) G by minimizing estimations of the Jensen-Shannon divergence or the Wasserstein distance calculated from empirical estimations of these distances with generated and training images.

Original: However, Arora et al.
Revised: However, Arora et al.

Original: (2017) prove that this explanation fails to pass the curse of dimensionality since estimates of Jensen-Shannon or Wasserstein distances do not generalize with a number of training examples which is a polynomial of the dimension.
Revised: (2017) prove that this explanation fails to pass the curse of dimensionality since estimates of Jensen-Shannon or Wasserstein distances do not generalize with a number of training examples which is a polynomial of the dimension.

Original: Therefore, the reason behind the generalization capacities of generative networks remains an open problem.
Revised: Therefore, the reason behind the generalization capacities of generative networks remains an open problem.

Original: VAEs, introduced by Kingma & Welling (2014), provide an alternative approach to GANs, by optimizing (cid:98) G together with its inverse on the training samples, instead of using a discriminator.
Revised: VAEs, introduced by Kingma & Welling (2014), provide an alternative approach to GANs, by optimizing (cid:98) G together with its inverse on the training samples, instead of using a discriminator.

Original: The inverse Φ is an embedding operator (the encoder) that is trained to transform X into a Gaussian white noise Z .
Revised: The inverse Φ is an embedding operator (the encoder) that is trained to transform X into a Gaussian white noise Z .

Original: A signiﬁcant disadvantage of VAEs is that the resulting generative models produce blurred images compared with GANs.
Revised: A signiﬁcant disadvantage of VAEs is that the resulting generative models produce blurred images compared with GANs.

Original: The loss function to train a VAE is also based on probabilistic distances which suffer from the same dimensionality curse shown in Arora et al.
Revised: The loss function to train a VAE is also based on probabilistic distances which suffer from the same dimensionality curse shown in Arora et al.

Original: It has been shown in Creswell & Bharath (2016) that GAN generators can be inverted, even if this is not imposed in their optimization.
Revised: It has been shown in Creswell & Bharath (2016) that GAN generators can be inverted, even if this is not imposed in their optimization.

Original: Generative Latent Optimization (GLO) was introduced in Bojanowski et al.
Revised: Generative Latent Optimization (GLO) was introduced in Bojanowski et al.

Original: (2017) to eliminate the need for a GAN discriminator while restoring sharper images than VAEs.
Revised: (2017) to eliminate the need for a GAN discriminator while restoring sharper images than VAEs.

Original: GLO still uses an autoencoder computational structure, where the latent space variables z are optimized together with the generator G .
Revised: GLO still uses an autoencoder computational structure, where the latent space variables z are optimized together with the generator G .

Original: Despite good results, linear variations of the embedding space variables are not mapped as clearly into image deformations as in GANs, which reduces the quality of generated images.
Revised: Despite good results, linear variations of the embedding space variables are not mapped as clearly into image deformations as in GANs, which reduces the quality of generated images.

Original: GANs and VAEs raise many questions.
Revised: GANs and VAEs raise many questions.

Original: What are the properties of the embedding operator Φ ?
Revised: What are the properties of the embedding operator Φ ?

Original: Where are the deformation properties coming from?
Revised: Where are the deformation properties coming from?

Original: Why do these algorithms seem to generalize despite the curse of dimensionality?
Revised: Why do these algorithms seem to generalize despite the curse of dimensionality?

Original: This paper shows that the embedding Φ does not need to be learned and can be speciﬁed by a priori properties.
Revised: This paper shows that the embedding Φ does not need to be learned and can be speciﬁed by a priori properties.

Original: Indeed, the embedding should Gaussianize X and be Lipschitz continuous to translations and deformations so that modiﬁcations of the input noise result in deformations of X . Section 2.2 reviews these properties.
Revised: Indeed, the embedding should Gaussianize X and be Lipschitz continuous to translations and deformations so that modiﬁcations of the input noise result in deformations of X . Section 2.2 reviews these properties.

Original: We then deﬁne the generative model as an inversion of the embedding on training data, regularized by the deep convolutional network architecture.
Revised: We then deﬁne the generative model as an inversion of the embedding on training data, regularized by the deep convolutional network architecture.

Original: Lipschitz continuity to translations and deformations together with Gaussianization are strong constraints that allow to specify suitable embedding operators.
Revised: Lipschitz continuity to translations and deformations together with Gaussianization are strong constraints that allow to specify suitable embedding operators.

Original: Lipschitz continuity to deformations requires separating the signal variations at different scales, which leads to the use of wavelet transforms.
Revised: Lipschitz continuity to deformations requires separating the signal variations at different scales, which leads to the use of wavelet transforms.

Original: We concentrate on wavelet scattering transforms (Mallat, 2012), which linearize translations and provide appropriate Gaussianization, but other multiscale embedding representations may also be used.
Revised: We concentrate on wavelet scattering transforms (Mallat, 2012), which linearize translations and provide appropriate Gaussianization, but other multiscale embedding representations may also be used.

Original: The generator (cid:98) G is thus estimated by inverting the scattering operator on training data.
Revised: The generator (cid:98) G is thus estimated by inverting the scattering operator on training data.

Original: We implement this inversion with a deep convolutional network having the same architecture as the generator of a DCGAN (Radford et al., 2016).
Revised: We implement this inversion with a deep convolutional network having the same architecture as the generator of a DCGAN (Radford et al., 2016).

Original: Experiments in Section 4 show that these Generative
Revised: Experiments in Section 4 show that these Generative

Original: Scattering Networks have similar properties as GAN generators.
Revised: Scattering Networks have similar properties as GAN generators.

Original:  C OMPUTING A G ENERATOR FROM AN E MBEDDING 2. G ENERATOR C ALCULATION AS AN I NVERSE P ROBLEM
Revised:  C OMPUTING A G ENERATOR FROM AN E MBEDDING 2. G ENERATOR C ALCULATION AS AN I NVERSE P ROBLEM

Original: Unsupervised learning consists on estimating a random model (cid:98) X of a random vector X from n realizations { x i } i ≤ n of X .
Revised: Unsupervised learning consists on estimating a random model (cid:98) X of a random vector X from n realizations { x i } i ≤ n of X .

Original: Autoregressive Gaussian processes are simple models (cid:98)
Revised: Autoregressive Gaussian processes are simple models (cid:98)

Original: X = (cid:98)
Revised: X = (cid:98)

Original: G ( Z ) , computed from an input Gaussian white noise Z by estimating a parametrized linear operator (cid:98) G .
Revised: G ( Z ) , computed from an input Gaussian white noise Z by estimating a parametrized linear operator (cid:98) G .

Original: This operator is obtained by inverting a linear operator, whose coefﬁcients are calculated from the realizations { x i } i ≤ n of X .
Revised: This operator is obtained by inverting a linear operator, whose coefﬁcients are calculated from the realizations { x i } i ≤ n of X .

Original: We shall similarly compute image models (cid:98)
Revised: We shall similarly compute image models (cid:98)

Original: X = (cid:98)
Revised: X = (cid:98)

Original: G ( Z ) from a Gaussian white noise Z , by estimating a parametrized operator (cid:98) G , but which is a deep convolutional network instead of a linear operator.
Revised: G ( Z ) from a Gaussian white noise Z , by estimating a parametrized operator (cid:98) G , but which is a deep convolutional network instead of a linear operator.

Original: It is calculated by inverting an embedding { Φ( x i ) } i ≤ n of the realizations { x i } i ≤ n of X , with a predeﬁned operator Φ( x ) .
Revised: It is calculated by inverting an embedding { Φ( x i ) } i ≤ n of the realizations { x i } i ≤ n of X , with a predeﬁned operator Φ( x ) .

Original: Let us denote by G the set of all parametrized convolutional network generators deﬁned by a particular architecture.
Revised: Let us denote by G the set of all parametrized convolutional network generators deﬁned by a particular architecture.

Original: We impose that (cid:98) G (Φ( x i ))
Revised: We impose that (cid:98) G (Φ( x i ))

Original: ≈ x
Revised: ≈ x

Original: i by minimizing a mean-square error in the convolutional network class G :
Revised: i by minimizing a mean-square error in the convolutional network class G :

Original: [Equation]
Revised: [Equation]

Original: The resulting generator (cid:98) G depends upon the training examples { x i } i ≤ n and on the regularization imposed by the network class G .
Revised: The resulting generator (cid:98) G depends upon the training examples { x i } i ≤ n and on the regularization imposed by the network class G .

Original: We shall say that the network generalizes over Φ( X ) if E ( (cid:107) X − (cid:98) G (Φ( X ))
Revised: We shall say that the network generalizes over Φ( X ) if E ( (cid:107) X − (cid:98) G (Φ( X ))

Original: (cid:107) 2 ) is small and comparable to the empirical error n − 1 (cid:80) ni
Revised: (cid:107) 2 ) is small and comparable to the empirical error n − 1 (cid:80) ni

Original: =1 (cid:107) x
Revised: =1 (cid:107) x

Original: i − G (Φ( x i ))
Revised: i − G (Φ( x i ))

Original: (cid:107) 2 .
Revised: (cid:107) 2 .

Original: We say that the network generalizes over a Gaussian white noise Z if realizations of (cid:98) X
Revised: We say that the network generalizes over a Gaussian white noise Z if realizations of (cid:98) X

Original: = (cid:98) G ( Z ) are realizations of X .
Revised: = (cid:98) G ( Z ) are realizations of X .

Original: If (cid:98) G generalizes over Φ( X ) then a sufﬁcient condition to generalize over Z is that Φ transforms X into a Gaussian white noise.
Revised: If (cid:98) G generalizes over Φ( X ) then a sufﬁcient condition to generalize over Z is that Φ transforms X into a Gaussian white noise.

Original: Besides this condition, the role of the embedding Φ is to specify the properties of x = (cid:98) G ( z ) that should result from modiﬁcations of z .
Revised: Besides this condition, the role of the embedding Φ is to specify the properties of x = (cid:98) G ( z ) that should result from modiﬁcations of z .

Original: We shall deﬁne Φ =
Revised: We shall deﬁne Φ =

Original: A Φ , where Φ is a ﬁxed normalized operator and A is an afﬁne operator which performs a whitening and a projection to a lower dimensional space.
Revised: A Φ , where Φ is a ﬁxed normalized operator and A is an afﬁne operator which performs a whitening and a projection to a lower dimensional space.

Original: We impose that { Φ( x i ) } i ≤ n are realizations of a general Gaussian process and that A transforms this process into a lower dimensional Gaussian white noise.
Revised: We impose that { Φ( x i ) } i ≤ n are realizations of a general Gaussian process and that A transforms this process into a lower dimensional Gaussian white noise.

Original: We normalize Φ by imposing that Φ(0) = 0 and that it is contractive, this is:
Revised: We normalize Φ by imposing that Φ(0) = 0 and that it is contractive, this is:

Original: [Equation]
Revised: [Equation]

Original: where D is the dimension of the space where X takes its values.
Revised: where D is the dimension of the space where X takes its values.

Original: To guarantee that the embedding of the training samples { Φ( x i ) } i ≤ n has a stable inverse, the normalized operator Φ should not contract too much the distances between the { x i } i ≤ n .
Revised: To guarantee that the embedding of the training samples { Φ( x i ) } i ≤ n has a stable inverse, the normalized operator Φ should not contract too much the distances between the { x i } i ≤ n .

Original: This means that Φ is a bi-Lipschitz embedding of these samples, and hence that there exists a constant α > 

Revised: This means that Φ is a bi-Lipschitz embedding of these samples, and hence that there exists a constant α >

Original: so that for all samples x i and x i (cid:48) in { x i } i ≤ n we have:
Revised: so that for all samples x i and x i (cid:48) in { x i } i ≤ n we have:

Original: [Equation]
Revised: [Equation]

Original: The afﬁne operator A performs a whitening of the distribution of the { Φ( x i ) } i ≤ n by subtracting the empirical mean µ and normalizing the largest eigenvalues of the empirical covariance matrix Σ .
Revised: The afﬁne operator A performs a whitening of the distribution of the { Φ( x i ) } i ≤ n by subtracting the empirical mean µ and normalizing the largest eigenvalues of the empirical covariance matrix Σ .

Original: For this purpose, we calculate the eigendecomposition of the covariance Σ =
Revised: For this purpose, we calculate the eigendecomposition of the covariance Σ =

Original: QSQ T .
Revised: QSQ T .

Original: Let P V d = Q d Q Td be the orthogonal projection in the space generated by the d eigenvectors with eigenvalues above a small threshold (cid:15) > 0 .
Revised: Let P V d = Q d Q Td be the orthogonal projection in the space generated by the d eigenvectors with eigenvalues above a small threshold (cid:15) > 0 .

Original: We choose (cid:15) and hence d so that Φ d ( x ) =
Revised: We choose (cid:15) and hence d so that Φ d ( x ) =

Original: P V d Φ( x ) still satisﬁes the bi-Lipschitz stability bound (3) up to a minor modiﬁcation of α .
Revised: P V d Φ( x ) still satisﬁes the bi-Lipschitz stability bound (3) up to a minor modiﬁcation of α .

Original: We choose A = Σ − 1 / 2 d ( Id − µ ) with Σ 1 / 2 d = Q d S 1 / 2 d hence Σ − 1 / 2 d = S − 1 / 2 d
Revised: We choose A = Σ − 1 / 2 d ( Id − µ ) with Σ 1 / 2 d = Q d S 1 / 2 d hence Σ − 1 / 2 d = S − 1 / 2 d

Original: Q Td , and Id denoting the identity.
Revised: Q Td , and Id denoting the identity.

Original: We obtain:
Revised: We obtain:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: which computes a d -dimensional whitening of the { Φ( x i ) } i ≤ n .
Revised: which computes a d -dimensional whitening of the { Φ( x i ) } i ≤ n .

Original: The generator (cid:98) G which inverts Φ can be factorized by inverting the eq.
Revised: The generator (cid:98) G which inverts Φ can be factorized by inverting the eq.

Original: [Equation]
Revised: [Equation]

Original: where µ d = P V d µ and G is a deep convolutional network which inverts Φ d over the training samples:
Revised: where µ d = P V d µ and G is a deep convolutional network which inverts Φ d over the training samples:

Original: [Equation]
Revised: [Equation]

Original: The deep convolutional networks in G (cid:48) have one layer less than in G .
Revised: The deep convolutional networks in G (cid:48) have one layer less than in G .

Original: Indeed, the ﬁrst layer is computed in (5) with Σ 1 / 2 d ( Id + µ d ) , it reshapes and adjusts the mean and covariance of the input white noise Z of dimension d .
Revised: Indeed, the ﬁrst layer is computed in (5) with Σ 1 / 2 d ( Id + µ d ) , it reshapes and adjusts the mean and covariance of the input white noise Z of dimension d .

Original: If d is sufﬁciently large, we can compute G independently from the dimensionality reduction by replacing Φ d by Φ in (6).
Revised: If d is sufﬁciently large, we can compute G independently from the dimensionality reduction by replacing Φ d by Φ in (6).

Original: If Φ is invertible, the operator G should not be confused with the inverse Φ − 1 .
Revised: If Φ is invertible, the operator G should not be confused with the inverse Φ − 1 .

Original: The model (cid:98) X
Revised: The model (cid:98) X

Original: = (cid:98) G
Revised: = (cid:98) G

Original: ( Z ) computed from G has typically a much smaller entropy than (cid:101)
Revised: ( Z ) computed from G has typically a much smaller entropy than (cid:101)

Original: X
Revised: X

Original: = Φ − 1 (Σ 1 / 2 d Z + µ d ) because
Revised: = Φ − 1 (Σ 1 / 2 d Z + µ d ) because

Original: G is regularized by the convolutional network architecture.
Revised: G is regularized by the convolutional network architecture.

Original: Since G is convolutional, it transforms a stationary process into a stationary process.
Revised: Since G is convolutional, it transforms a stationary process into a stationary process.

Original: Therefore, the convolutional non-linear operator G is capturing stationary components, factorized at multiple scales by the deep convolutional network.
Revised: Therefore, the convolutional non-linear operator G is capturing stationary components, factorized at multiple scales by the deep convolutional network.

Original: The non-stationarity part of the model (cid:98) X
Revised: The non-stationarity part of the model (cid:98) X

Original: = (cid:98) G ( Z ) is captured by the afﬁne transformation Σ 1 / 2 d Z + µ d .
Revised: = (cid:98) G ( Z ) is captured by the afﬁne transformation Σ 1 / 2 d Z + µ d .

Original: We shall see that this is sufﬁcient to model highly non-stationary processes such as centered faces.
Revised: We shall see that this is sufﬁcient to model highly non-stationary processes such as centered faces.

Original: Associative Memory:
Revised: Associative Memory:

Original: The existence of a known embedding operator Φ allows one to use these generative networks as associative or content addressable memories.
Revised: The existence of a known embedding operator Φ allows one to use these generative networks as associative or content addressable memories.

Original: The input Z can be interpreted as an address, of lower dimension d than the generated image (cid:98) X .
Revised: The input Z can be interpreted as an address, of lower dimension d than the generated image (cid:98) X .

Original: Any training image x
Revised: Any training image x

Original: i is associated to the address
Revised: i is associated to the address

Original: z i = Σ − 1 / 2 d (Φ x i − µ ) .
Revised: z i = Σ − 1 / 2 d (Φ x i − µ ) .

Original: The network is trained to generate the training images { x i } i ≤ n from these lower dimensional addresses.
Revised: The network is trained to generate the training images { x i } i ≤ n from these lower dimensional addresses.

Original: The inner network coefﬁcients thus include a form of distributed memory of these training images.
Revised: The inner network coefﬁcients thus include a form of distributed memory of these training images.

Original: If the network generalizes then one can approximately reconstruct a realization x of the random process X from its embedding address z = Σ − 1 / 2 d (Φ x − µ ) .
Revised: If the network generalizes then one can approximately reconstruct a realization x of the random process X from its embedding address z = Σ − 1 / 2 d (Φ x − µ ) .

Original: In that sense, the memory is content addressable.
Revised: In that sense, the memory is content addressable.

Original: 2. G AUSSIANIZATION AND C ONTINUITY TO D EFORMATIONS
Revised: 2. G AUSSIANIZATION AND C ONTINUITY TO D EFORMATIONS

Original: We now describe the properties of the normalized embedding operator Φ to build a generator having similar properties as GANs or VAEs.
Revised: We now describe the properties of the normalized embedding operator Φ to build a generator having similar properties as GANs or VAEs.

Original: We mentioned that we would like Φ( X ) to be nearly a
Revised: We mentioned that we would like Φ( X ) to be nearly a

Original: Gaussian white noise and since Φ =
Revised: Gaussian white noise and since Φ =

Original: A Φ where A is afﬁne then Φ( X ) should be Gaussian.
Revised: A Φ where A is afﬁne then Φ( X ) should be Gaussian.

Original: Therefore, { Φ( x i ) } i ≤ n should be realizations of a Gaussian process and hence be concentrated over an ellipsoid.
Revised: Therefore, { Φ( x i ) } i ≤ n should be realizations of a Gaussian process and hence be concentrated over an ellipsoid.

Original: However, this is not the only constraint.
Revised: However, this is not the only constraint.

Original: The generator G is a deep convolutional network which is covariant to translations, and we compute
Revised: The generator G is a deep convolutional network which is covariant to translations, and we compute

Original: G such that G (Φ( x i ))
Revised: G such that G (Φ( x i ))

Original: ≈ x i .
Revised: ≈ x i .

Original: As previously explained, G will not be an exact inverse of Φ because the generator is regularized by its architecture.
Revised: As previously explained, G will not be an exact inverse of Φ because the generator is regularized by its architecture.

Original: However, we want to ensure that G is as close as possible to a left inverse of Φ .
Revised: However, we want to ensure that G is as close as possible to a left inverse of Φ .

Original: A convolutional network G maps a multichannel image z (cid:48) deﬁned over a subsampled spatial grid into a full resolution image x .
Revised: A convolutional network G maps a multichannel image z (cid:48) deﬁned over a subsampled spatial grid into a full resolution image x .

Original: It is covariant to translations in the sense that if z (cid:48) is translated then x is also translated.
Revised: It is covariant to translations in the sense that if z (cid:48) is translated then x is also translated.

Original: The inverse of such a network should also be covariant to translations.
Revised: The inverse of such a network should also be covariant to translations.

Original: Consequently, Φ is required to be covariant to translations.
Revised: Consequently, Φ is required to be covariant to translations.

Original: Another important property is that images x = G ( z ) appear to be translated and deformed when z is modiﬁed.
Revised: Another important property is that images x = G ( z ) appear to be translated and deformed when z is modiﬁed.

Original: To enforce this property, we require Φ to be Lipschitz continuous to translations and deformations.
Revised: To enforce this property, we require Φ to be Lipschitz continuous to translations and deformations.

Original: A translation and deformation of an image x ( u ) can be written as x τ ( u ) =
Revised: A translation and deformation of an image x ( u ) can be written as x τ ( u ) =

Original: x ( u − τ ( u )) , where u denotes the spatial coordinates.
Revised: x ( u − τ ( u )) , where u denotes the spatial coordinates.

Original: Let | τ | ∞ =
Revised: Let | τ | ∞ =

Original: max u | τ ( u ) | be the maximum translation amplitude.
Revised: max u | τ ( u ) | be the maximum translation amplitude.

Original: Let ∇ τ ( u ) be the Jacobian of τ at u and |∇ τ ( u ) | be the norm of this Jacobian matrix.
Revised: Let ∇ τ ( u ) be the Jacobian of τ at u and |∇ τ ( u ) | be the norm of this Jacobian matrix.

Original: The deformation induced by τ is measured by |∇ τ | ∞ =
Revised: The deformation induced by τ is measured by |∇ τ | ∞ =

Original: max u |∇ τ ( u ) | , which speciﬁes the maximum scaling factor induced by the deformation.
Revised: max u |∇ τ ( u ) | , which speciﬁes the maximum scaling factor induced by the deformation.

Original: It deﬁnes a metric on diffeomorphisms (Mallat, 2012) and thus speciﬁes the deformation “size”.
Revised: It deﬁnes a metric on diffeomorphisms (Mallat, 2012) and thus speciﬁes the deformation “size”.

Original: The operator Φ is said to be Lipschitz continuous to translations and deformations over domains of scale 2 J if there exists a constant C such that for all images x we have:
Revised: The operator Φ is said to be Lipschitz continuous to translations and deformations over domains of scale 2 J if there exists a constant C such that for all images x we have:

Original: [Equation]
Revised: [Equation]

Original: This implies that translations of x that are small relative to 2 J and small deformations are mapped by Φ into small quasi-linear variations of z = Φ( x ) .
Revised: This implies that translations of x that are small relative to 2 J and small deformations are mapped by Φ into small quasi-linear variations of z = Φ( x ) .

Original: We also mentioned that { Φ( x i ) } i ≤ n should be concentrated on an ellipsoid.
Revised: We also mentioned that { Φ( x i ) } i ≤ n should be concentrated on an ellipsoid.

Original: This Gaussianization property is always true if n ≤ d , but can be difﬁcult to achieve if n (cid:29) d .
Revised: This Gaussianization property is always true if n ≤ d , but can be difﬁcult to achieve if n (cid:29) d .

Original: In one dimension x ∈ R , an invertible differentiable operator Φ which Gaussianizes a random variable X can be computed as the solution of a differential equation that transports the histogram into a Gaussian (Friedman, 1987).
Revised: In one dimension x ∈ R , an invertible differentiable operator Φ which Gaussianizes a random variable X can be computed as the solution of a differential equation that transports the histogram into a Gaussian (Friedman, 1987).

Original: In higher dimension, this strategy has been extended by iteratively performing a Gaussianization of one-dimensional variables, through independent component analysis (Chen & Gopinath, 2000) or with random rotations (Laparra et al., 2011).
Revised: In higher dimension, this strategy has been extended by iteratively performing a Gaussianization of one-dimensional variables, through independent component analysis (Chen & Gopinath, 2000) or with random rotations (Laparra et al., 2011).

Original: However, such strategies might not apply here directly because they do not necessarily lead to operators that are translation covariant and Lipschitz continuous to translations and deformations.
Revised: However, such strategies might not apply here directly because they do not necessarily lead to operators that are translation covariant and Lipschitz continuous to translations and deformations.

Original: Another Gaussianization strategy comes from the central limit theorem by averaging enough nearly independent random variables having variances of the same order of magnitude.
Revised: Another Gaussianization strategy comes from the central limit theorem by averaging enough nearly independent random variables having variances of the same order of magnitude.

Original: The advantage of this approach is that an averaging can be covariant to translations if implemented with convolutions with a low-pass ﬁlter.
Revised: The advantage of this approach is that an averaging can be covariant to translations if implemented with convolutions with a low-pass ﬁlter.

Original: Moreover, this averaging leads to Lipschitz continuity to translations and deformations.
Revised: Moreover, this averaging leads to Lipschitz continuity to translations and deformations.

Original: An averaging operator is contractive; therefore, it loses all high-frequency information, and hence confuses images.
Revised: An averaging operator is contractive; therefore, it loses all high-frequency information, and hence confuses images.

Original: To deﬁne an operator Φ that discriminates large collections of images and is bi-Lipschitz (3), we must preserve high-frequency information despite the averaging.
Revised: To deﬁne an operator Φ that discriminates large collections of images and is bi-Lipschitz (3), we must preserve high-frequency information despite the averaging.

Original: The next section explains how to do so with a Scattering transform.
Revised: The next section explains how to do so with a Scattering transform.

Original:  G ENERATIVE S CATTERING N ETWORKS 3. G AUSSIANIZATION BY M ULTISCALE S CATTERING
Revised:  G ENERATIVE S CATTERING N ETWORKS 3. G AUSSIANIZATION BY M ULTISCALE S CATTERING

Original: We will show that scattering transforms (Mallat, 2012; Bruna & Mallat, 2013) are good candidates to construct an image embedding for image generation.
Revised: We will show that scattering transforms (Mallat, 2012; Bruna & Mallat, 2013) are good candidates to construct an image embedding for image generation.

Original: It is explained in Mallat (2012) that to deﬁne an operator that is Lipschitz continuous to deformations and hence satisfy (7), one must separate variations of x at different scales.
Revised: It is explained in Mallat (2012) that to deﬁne an operator that is Lipschitz continuous to deformations and hence satisfy (7), one must separate variations of x at different scales.

Original: Indeed, the ﬁrst order term of the non-rigid part of a deformation is a scaling; controlling the impact of this scaling term over signal frequencies requires decomposing the signal at different scales, which is done by a wavelet transform.
Revised: Indeed, the ﬁrst order term of the non-rigid part of a deformation is a scaling; controlling the impact of this scaling term over signal frequencies requires decomposing the signal at different scales, which is done by a wavelet transform.

Original: The linearization of translations at a scale 2 J is obtained by an averaging at this scale.
Revised: The linearization of translations at a scale 2 J is obtained by an averaging at this scale.

Original: Non-linearities appear as a necessity to create interactions across multiple scale coefﬁcients; these interactions reduce the information loss due to averaging, and thus help to avoid modifying too much the Euclidean distances (3).
Revised: Non-linearities appear as a necessity to create interactions across multiple scale coefﬁcients; these interactions reduce the information loss due to averaging, and thus help to avoid modifying too much the Euclidean distances (3).

Original: The averaging at the scale 2 J also Gaussianizes the random vector X .
Revised: The averaging at the scale 2 J also Gaussianizes the random vector X .

Original: Therefore, the scale 2 J adjusts the trade-off between Gaussianization and contraction of distances.
Revised: Therefore, the scale 2 J adjusts the trade-off between Gaussianization and contraction of distances.

Original: Let us emphasize that the scattering transform is not a unique solution to this problem.
Revised: Let us emphasize that the scattering transform is not a unique solution to this problem.

Original: Multiscale scattering operators can be implemented with a deep convolutional network architecture, using predeﬁned wavelet ﬁlters (Mallat, 2016); however, other deep convolutional networks might have the same properties.
Revised: Multiscale scattering operators can be implemented with a deep convolutional network architecture, using predeﬁned wavelet ﬁlters (Mallat, 2016); however, other deep convolutional networks might have the same properties.

Original: Also, other multiscale representations such as the ones in Portilla & Simoncelli (2000); Lyu & Simoncelli (2009); Malo & Laparra (2010) may also be used.
Revised: Also, other multiscale representations such as the ones in Portilla & Simoncelli (2000); Lyu & Simoncelli (2009); Malo & Laparra (2010) may also be used.

Original: We concentrate here on scattering operators because their mathematical properties are relatively well understood.
Revised: We concentrate here on scattering operators because their mathematical properties are relatively well understood.

Original: A scattering operator S j transforms x ( u ) into a tensor x j ( u, k ) where the spatial parameter u is sampled at intervals 2 j and the channels are indexed by k .
Revised: A scattering operator S j transforms x ( u ) into a tensor x j ( u, k ) where the spatial parameter u is sampled at intervals 2 j and the channels are indexed by k .

Original: The number K j of channels increases with j to partially compensate for the loss of spatial resolution.
Revised: The number K j of channels increases with j to partially compensate for the loss of spatial resolution.

Original: These K j channels are computed by a non-linear translation covariant transformation Ψ j .
Revised: These K j channels are computed by a non-linear translation covariant transformation Ψ j .

Original: In this paper, Ψ j is computed as successive convolutions with complex two-dimensional wavelets and modulus, with no channel interactions.
Revised: In this paper, Ψ j is computed as successive convolutions with complex two-dimensional wavelets and modulus, with no channel interactions.

Original: Following Bruna & Mallat (2013), we choose a Morlet wavelet ψ , scaled by 2 (cid:96) and rotated along Q angles θ = qπ/Q :
Revised: Following Bruna & Mallat (2013), we choose a Morlet wavelet ψ , scaled by 2 (cid:96) and rotated along Q angles θ = qπ/Q :

Original: [Equation]
Revised: [Equation]

Original: q < Q.
Revised: q < Q.

Original: To obtain an order two scattering operator S j , the operator Ψ j computes sequences of up to two wavelet convolutions and complex modulus:
Revised: To obtain an order two scattering operator S j , the operator Ψ j computes sequences of up to two wavelet convolutions and complex modulus:

Original: [Equation]
Revised: [Equation]

Original: Therefore, there are K j = 1+ Qj + Q 2 j ( j − 1) / 2 channels.
Revised: Therefore, there are K j = 1+ Qj + Q 2 j ( j − 1) / 2 channels.

Original: A scattering transform is then obtained by averaging each channel with a Gaussian low-pass ﬁlter φ j ( u ) = 2 − 2 j φ (2 − j u ) , whose spatial width is proportional to 2 j :
Revised: A scattering transform is then obtained by averaging each channel with a Gaussian low-pass ﬁlter φ j ( u ) = 2 − 2 j φ (2 − j u ) , whose spatial width is proportional to 2 j :

Original: [Equation]
Revised: [Equation]

Original: Convolutions by φ j are followed by a subsampling by 2 j ; as a result, if x has p pixels then S j is of dimension p
Revised: Convolutions by φ j are followed by a subsampling by 2 j ; as a result, if x has p pixels then S j is of dimension p

Original: α j where:
Revised: α j where:

Original: [Equation]
Revised: [Equation]

Original: The maximum scale 2 j is limited by the image width 2 j ≤ p 1 / 2 .
Revised: The maximum scale 2 j is limited by the image width 2 j ≤ p 1 / 2 .

Original: For our experiments we used
Revised: For our experiments we used

Original: Q = 8 and images x of size p = 64 2 .
Revised: Q = 8 and images x of size p = 64 2 .

Original: In this case we have α 4 = 1 .
Revised: In this case we have α 4 = 1 .

Original: 63 and α 5 = 0 . 66 .
Revised: 63 and α 5 = 0 . 66 .

Original: Moreover, α j > 1 for j ≤ 4 and so S j ( x ) has more coefﬁcients than x .
Revised: Moreover, α j > 1 for j ≤ 4 and so S j ( x ) has more coefﬁcients than x .

Original: Based on this coefﬁcient counting, we expect S j to be invertible for j ≤ 4 , but not for j ≥ 5 .
Revised: Based on this coefﬁcient counting, we expect S j to be invertible for j ≤ 4 , but not for j ≥ 5 .

Original: The wavelets separate the variations of x at different scales 2 (cid:96) along different directions qπ/Q , and second order coefﬁcients compute interactions across scales.
Revised: The wavelets separate the variations of x at different scales 2 (cid:96) along different directions qπ/Q , and second order coefﬁcients compute interactions across scales.

Original: Because of this scale separation, one can prove that S j is Lipschitz continuous to translations and deformations (Mallat, 2012) in the sense of eq.
Revised: Because of this scale separation, one can prove that S j is Lipschitz continuous to translations and deformations (Mallat, 2012) in the sense of eq.

Original: Wavelets are also deﬁned with a Littlewood-Paley condition which guarantees that the wavelet transforms and also S j are contractive operators (Mallat, 2012).
Revised: Wavelets are also deﬁned with a Littlewood-Paley condition which guarantees that the wavelet transforms and also S j are contractive operators (Mallat, 2012).

Original: The wavelets ψ (cid:96),q and the low-pass ﬁlter φ j can be seen as a cascade of convolutions with small size ﬁlters and subsamplings followed by a modulus non-linearity.
Revised: The wavelets ψ (cid:96),q and the low-pass ﬁlter φ j can be seen as a cascade of convolutions with small size ﬁlters and subsamplings followed by a modulus non-linearity.

Original: As a result, a scattering transform is obtained by applying convolution matrices V j and then a modulus non-linearity (Mallat, 2016):
Revised: As a result, a scattering transform is obtained by applying convolution matrices V j and then a modulus non-linearity (Mallat, 2016):

Original: [Equation]
Revised: [Equation]

Original: it is thus an instance of a deep convolutional network whose ﬁlters are speciﬁed by wavelets and where the non-linearity is chosen to be a modulus.
Revised: it is thus an instance of a deep convolutional network whose ﬁlters are speciﬁed by wavelets and where the non-linearity is chosen to be a modulus.

Original: If wavelet coefﬁcients become nearly independent when they are sufﬁciently far away then S j ( X ) becomes progressively more Gaussian as the scale 2 j increases, because of the spatial averaging by φ j .
Revised: If wavelet coefﬁcients become nearly independent when they are sufﬁciently far away then S j ( X ) becomes progressively more Gaussian as the scale 2 j increases, because of the spatial averaging by φ j .

Original: So, if X ( u ) is independent from X ( v ) for | x − v | ≥ ∆ then the Central Limit Theorem proves that S j ( X ) converges to a Gaussian distribution when 2 j / ∆ increases.
Revised: So, if X ( u ) is independent from X ( v ) for | x − v | ≥ ∆ then the Central Limit Theorem proves that S j ( X ) converges to a Gaussian distribution when 2 j / ∆ increases.

Original: However, as the scale increases there are instabilities created by the contractions (as in eq.
Revised: However, as the scale increases there are instabilities created by the contractions (as in eq.

Original: (3)) induced by S j .
Revised: (3)) induced by S j .

Original: This trade-off between Gaussianization and stability deﬁnes the optimal choice of the scale 2 j.
Revised: This trade-off between Gaussianization and stability deﬁnes the optimal choice of the scale 2 j

Original: 3. I NVERSE S CATTERING G ENERATOR WITH D EEP C ONVOLUTIONAL N ETWORKS
Revised: 3. I NVERSE S CATTERING G ENERATOR WITH D EEP C ONVOLUTIONAL N ETWORKS

Original: A scattering generator network is computed by choosing the normalized embedding operator Φ to be a scattering operator S J at some predeﬁned scale 2 J .
Revised: A scattering generator network is computed by choosing the normalized embedding operator Φ to be a scattering operator S J at some predeﬁned scale 2 J .

Original: As explained in Section 2.1, we compute in (6) a scattering generator by ﬁnding a deep convolutional network G which minimizes the MSE over all training samples.
Revised: As explained in Section 2.1, we compute in (6) a scattering generator by ﬁnding a deep convolutional network G which minimizes the MSE over all training samples.

Original: The resulting network G is a regularized approximate inversion of S J .
Revised: The resulting network G is a regularized approximate inversion of S J .

Original: The deep convolutional network architecture that deﬁnes the family G (cid:48) has depth J and follows the shapes of only the convolutional layers of a DCGAN (Radford et al., 2016) generator:
Revised: The deep convolutional network architecture that deﬁnes the family G (cid:48) has depth J and follows the shapes of only the convolutional layers of a DCGAN (Radford et al., 2016) generator:

Original: [Equation]
Revised: [Equation]

Original: The non-linearity ρ is a ReLU, and each (cid:102) W j consists of multiplying by W j , which is an upsampling followed by a multichannel convolutional operator along the spatial variables, and then adding perchannel biases.
Revised: The non-linearity ρ is a ReLU, and each (cid:102) W j consists of multiplying by W j , which is an upsampling followed by a multichannel convolutional operator along the spatial variables, and then adding perchannel biases.

Original: The weights of this network are optimized by minimizing a mean-square error loss:
Revised: The weights of this network are optimized by minimizing a mean-square error loss:

Original: [Equation]
Revised: [Equation]

Original: with stochastic gradient descent.
Revised: with stochastic gradient descent.

Original: We explained that S J ( x ) is computed from x = S 0 ( x ) by iteratively calculating S j
Revised: We explained that S J ( x ) is computed from x = S 0 ( x ) by iteratively calculating S j

Original: ( x ) from S j − 1 ( x ) with (10), which reduces the spatial resolution while increasing the number of channels.
Revised: ( x ) from S j − 1 ( x ) with (10), which reduces the spatial resolution while increasing the number of channels.

Original: We can thus interpret (11) as a progressive inversion of this decomposition.
Revised: We can thus interpret (11) as a progressive inversion of this decomposition.

Original:  N UMERICAL EXPERIMENTS
Revised:  N UMERICAL EXPERIMENTS

Original: In this section, we test the ability to reconstruct training images from their scattering coefﬁcients and the generalization properties of the obtained generative networks.
Revised: In this section, we test the ability to reconstruct training images from their scattering coefﬁcients and the generalization properties of the obtained generative networks.

Original: We shall also verify that modiﬁcations of the input noise produce a morphing of the generated images as in GANs.
Revised: We shall also verify that modiﬁcations of the input noise produce a morphing of the generated images as in GANs.

Original: We consider three datasets that have different levels of variabilities: CelebA (Liu et al., 2015), a 64 2 downsampled version of ImageNet (Russakovsky et al., 2015) and Polygon5.
Revised: We consider three datasets that have different levels of variabilities: CelebA (Liu et al., 2015), a 64 2 downsampled version of ImageNet (Russakovsky et al., 2015) and Polygon5.

Original: The last dataset consists of images of random convex polygons of at most ﬁve vertices, with random colors.
Revised: The last dataset consists of images of random convex polygons of at most ﬁve vertices, with random colors.

Original: All datasets consist of RGB color images with shape 64 2 × 3 .
Revised: All datasets consist of RGB color images with shape 64 2 × 3 .

Original: For each dataset, we consider onlytraining images and 16384 test images.
Revised: For each dataset, we consider onlytraining images and 16384 test images.

Original: In all experiments, we chose the scattering averaging scale to be 2 J = 2 

Revised: In all experiments, we chose the scattering averaging scale to be 2 J = 2

Original: = 16 , which is 4 times smaller than the image width 64 .
Revised: = 16 , which is 4 times smaller than the image width 64 .

Original: This operator is stable to deformations over big domains.
Revised: This operator is stable to deformations over big domains.

Original: S 4 ( x ) is computed over the 3 color channels of each image x .
Revised: S 4 ( x ) is computed over the 3 color channels of each image x .

Original: Because of the subsampling by 2 − 4 , S 4 ( x ) has a spatial resolution of 4 by 4 , with 417 × 3 = 1251 channels, and is thus of dimension
Revised: Because of the subsampling by 2 − 4 , S 4 ( x ) has a spatial resolution of 4 by 4 , with 417 × 3 = 1251 channels, and is thus of dimension

Original: ≈ 2 × 10 4 .
Revised: ≈ 2 × 10 4 .

Original: It has more coefﬁcients than the input image x and numerical experiments indicate that S 4 is invertible.
Revised: It has more coefﬁcients than the input image x and numerical experiments indicate that S 4 is invertible.

Original: This dimension is reduced to d = 100 by the whitening operator.
Revised: This dimension is reduced to d = 100 by the whitening operator.

Original: All scattering transforms are computed with PyScatWave (Oyallon et al., 2017).
Revised: All scattering transforms are computed with PyScatWave (Oyallon et al., 2017).

Original: The operator S 4 is a bi-Lipschitz embedding of the training images of the three datasets in the sense of (3), with a Lipschitz constant α = 5 and over 99 .
Revised: The operator S 4 is a bi-Lipschitz embedding of the training images of the three datasets in the sense of (3), with a Lipschitz constant α = 5 and over 99 .

Original: 5% of the distances between image pairs ( x i , x i (cid:48) ) are preserved with a Lipschitz constant smaller than 3 .
Revised: 5% of the distances between image pairs ( x i , x i (cid:48) ) are preserved with a Lipschitz constant smaller than 3 .

Original: The architecture of the generator is shown in Fig.
Revised: The architecture of the generator is shown in Fig.

Original: The input Gaussian white noise Z is of dimension d = 100 .
Revised: The input Gaussian white noise Z is of dimension d = 100 .

Original: All the convolutional layers have ﬁlters of size 5 , use symmetric padding at the boundaries and are preceded by a bilinear upsampling of their inputs.
Revised: All the convolutional layers have ﬁlters of size 5 , use symmetric padding at the boundaries and are preceded by a bilinear upsampling of their inputs.

Original: The number of channels of each layer is 512 , 256 , 128 and 3 .
Revised: The number of channels of each layer is 512 , 256 , 128 and 3 .

Original: As in Radford et al.
Revised: As in Radford et al.

Original: (2016), we use a hyperbolic tangent instead of a ReLU for the ﬁnal non-linearity.
Revised: (2016), we use a hyperbolic tangent instead of a ReLU for the ﬁnal non-linearity.

Original: We minimize the loss in eq.
Revised: We minimize the loss in eq.

Original: (12) using the Adam optimizer (Kingma & Ba, 2014) with default hyperparameters.
Revised: (12) using the Adam optimizer (Kingma & Ba, 2014) with default hyperparameters.

Original: Figure 2 shows that we can indeed recover a precise approximation of an image x from S 4 ( x ) .
Revised: Figure 2 shows that we can indeed recover a precise approximation of an image x from S 4 ( x ) .

Original: This reconstruction was also computed with PyScatWave (Oyallon et al., 2017), which recovers ˆ x by iteratively minimizing (cid:107) S 4 (ˆ x ) − S 4 ( x ) (cid:107) with a gradient descent, initialized with white noise.
Revised: This reconstruction was also computed with PyScatWave (Oyallon et al., 2017), which recovers ˆ x by iteratively minimizing (cid:107) S 4 (ˆ x ) − S 4 ( x ) (cid:107) with a gradient descent, initialized with white noise.

Original: The convolutional network operator G is very different from the inverse scattering S − 1despite
Revised: The convolutional network operator G is very different from the inverse scattering S − 1despite

Original: the fact that they both recover the same training images.
Revised: the fact that they both recover the same training images.

Original: Figures 2(c) and 2(d) are respectively realizations of (cid:98) X = (cid:98) G ( Z )
Revised: Figures 2(c) and 2(d) are respectively realizations of (cid:98) X = (cid:98) G ( Z )

Original: =
Revised: =

Original: G (Σ 1 / 2 d Z + µ d ) and of (cid:101)
Revised: G (Σ 1 / 2 d Z + µ d ) and of (cid:101)

Original: X
Revised: X

Original: = S − 1 4 (Σ 1 / 2 d Z + µ d ) , for the same realization of the Gaussian white noise Z .
Revised: = S − 1 4 (Σ 1 / 2 d Z + µ d ) , for the same realization of the Gaussian white noise Z .

Original: The convolutional network generator G restores a face image, which is barely the case for S − 1 4 .
Revised: The convolutional network generator G restores a face image, which is barely the case for S − 1 4 .

Original: However, the image in 2(d) shows that the non-stationary component provided by Σ d and µ d does provide some global shape information on faces.
Revised: However, the image in 2(d) shows that the non-stationary component provided by Σ d and µ d does provide some global shape information on faces.

Original: [Figure]
Revised: [Figure]

Original: We now evaluate the training and testing errors of the generator (cid:98) G to evaluate its generalization properties on Φ( X ) =
Revised: We now evaluate the training and testing errors of the generator (cid:98) G to evaluate its generalization properties on Φ( X ) =

Original: Σ − 1 / 2 d
Revised: Σ − 1 / 2 d

Original: ( S 4 ( X ) − µ ) .
Revised: ( S 4 ( X ) − µ ) .

Original: This is done by comparing x with the image (cid:98) G ( z ) generated from z =
Revised: This is done by comparing x with the image (cid:98) G ( z ) generated from z =

Original: Φ( x ) , where x may be a training or a test image.
Revised: Φ( x ) , where x may be a training or a test image.

Original: Figures 3 and 4 show such reconstructions on training and test images.
Revised: Figures 3 and 4 show such reconstructions on training and test images.

Original: Table 1 gives the average training and test errors for each dataset.
Revised: Table 1 gives the average training and test errors for each dataset.

Original: The training error is between 1.5db and 3db above the test error, which is a sign of overﬁtting.
Revised: The training error is between 1.5db and 3db above the test error, which is a sign of overﬁtting.

Original: However, this overﬁtting is not large compared to the variability of errors from one dataset to the next.
Revised: However, this overﬁtting is not large compared to the variability of errors from one dataset to the next.

Original: Overﬁtting is not good for unsupervised learning where the intent is to model a probability density, but if we consider this network as an associative memory, it is not a bad property.
Revised: Overﬁtting is not good for unsupervised learning where the intent is to model a probability density, but if we consider this network as an associative memory, it is not a bad property.

Original: Indeed, it means that the network performs a better recovery of known images used in training than unknown images in the test, which is needed for high precision storage of particular images.
Revised: Indeed, it means that the network performs a better recovery of known images used in training than unknown images in the test, which is needed for high precision storage of particular images.

Original: Polygons are simple images that are much better recovered than faces in CelebA, which are much simpler images than the ones in ImageNet.
Revised: Polygons are simple images that are much better recovered than faces in CelebA, which are much simpler images than the ones in ImageNet.

Original: This simplicity is related to the sparsity of their wavelet coefﬁcients, which is higher for polygons than for faces or ImageNet images.
Revised: This simplicity is related to the sparsity of their wavelet coefﬁcients, which is higher for polygons than for faces or ImageNet images.

Original: Wavelet sparsity drives the properties of scattering coefﬁcients, which are localized l 1 norms of wavelet coefﬁcients.
Revised: Wavelet sparsity drives the properties of scattering coefﬁcients, which are localized l 1 norms of wavelet coefﬁcients.

Original: The network regularizes the inversion by storing information on training images, which is a form of memorization.
Revised: The network regularizes the inversion by storing information on training images, which is a form of memorization.

Original: ImageNet images require more memorization because their wavelet coefﬁcients are less sparse than polygons or faces; this might explain the difference of accuracies over datasets, but the link between sparsity and the network memory capacity is not fully understood.
Revised: ImageNet images require more memorization because their wavelet coefﬁcients are less sparse than polygons or faces; this might explain the difference of accuracies over datasets, but the link between sparsity and the network memory capacity is not fully understood.

Original: The generative network has itself sparse activations with about 70% of them being equal to zero on average over all images of the three datasets.
Revised: The generative network has itself sparse activations with about 70% of them being equal to zero on average over all images of the three datasets.

Original: Sparsity thus seems to be an important element of the regularization given by the network.
Revised: Sparsity thus seems to be an important element of the regularization given by the network.

Original: [Table]
Revised: [Table]

Original: Similarly to VAEs, scattering image generations can be slightly blurred, even on training samples.
Revised: Similarly to VAEs, scattering image generations can be slightly blurred, even on training samples.

Original: This blurriness is probably due to a memorization issue because we observe that the blur is smaller on images of polygons which are simpler.
Revised: This blurriness is probably due to a memorization issue because we observe that the blur is smaller on images of polygons which are simpler.

Original: Also, if the dataset has only 16 images, the generator can fully recover all high frequencies, but learning is very slow and requires many iterations.
Revised: Also, if the dataset has only 16 images, the generator can fully recover all high frequencies, but learning is very slow and requires many iterations.

Original: Forimages, ﬁtting the high frequencies does not seem to be possible with Adam even after 2000 epochs.
Revised: Forimages, ﬁtting the high frequencies does not seem to be possible with Adam even after 2000 epochs.

Original: Therefore, this difﬁculty can be due to the memory capacity of the network which is not sufﬁcient to capture all the training images, but it may also be partly due to the optimizer.
Revised: Therefore, this difﬁculty can be due to the memory capacity of the network which is not sufﬁcient to capture all the training images, but it may also be partly due to the optimizer.

Original: To evaluate the network generalization on Gaussian white noise Z , Figure 5 shows images (cid:98)
Revised: To evaluate the network generalization on Gaussian white noise Z , Figure 5 shows images (cid:98)

Original: X = (cid:98)
Revised: X = (cid:98)

Original: G ( Z ) generated from a random sampling of the input Gaussian white noise Z .
Revised: G ( Z ) generated from a random sampling of the input Gaussian white noise Z .

Original: Generated images have strong similarities with the ones in the training set for polygons and faces.
Revised: Generated images have strong similarities with the ones in the training set for polygons and faces.

Original: The network recovers colored geometric shapes for the polygon images even tough they are not exactly polygons
Revised: The network recovers colored geometric shapes for the polygon images even tough they are not exactly polygons

Original: and it recovers faces for CelebA with a blurred background.
Revised: and it recovers faces for CelebA with a blurred background.

Original: For ImageNet, it recovers blurred images, showing that in this case the model is not good.
Revised: For ImageNet, it recovers blurred images, showing that in this case the model is not good.

Original: Figure 6 shows images reconstructed from the generator G by interpolating the latent variables z i
Revised: Figure 6 shows images reconstructed from the generator G by interpolating the latent variables z i

Original: = S 4 ( x i ) and z i (cid:48)
Revised: = S 4 ( x i ) and z i (cid:48)

Original: =
Revised: =

Original: S 4 ( x i (cid:48) ) of two images x i and x i (cid:48) :
Revised: S 4 ( x i (cid:48) ) of two images x i and x i (cid:48) :

Original: z = αz i + (1 − α )
Revised: z = αz i + (1 − α )

Original: z i (cid:48) with α ∈ [0 , 1] .
Revised: z i (cid:48) with α ∈ [0 , 1] .

Original: This linear interpolation results in a continuous deformation from one image to the other while colors and image intensities are also adjusted.
Revised: This linear interpolation results in a continuous deformation from one image to the other while colors and image intensities are also adjusted.

Original: This behavior is similar to the one obtained by GANs, in our case it is a consequence of the Lipschitz continuity to deformations of the scattering transform.
Revised: This behavior is similar to the one obtained by GANs, in our case it is a consequence of the Lipschitz continuity to deformations of the scattering transform.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper shows that most properties of GANs and VAEs can be reproduced with an embedding computed with a scattering transform, which avoids using a discriminator as in GANs or learning the embedding as in VAEs.
Revised: This paper shows that most properties of GANs and VAEs can be reproduced with an embedding computed with a scattering transform, which avoids using a discriminator as in GANs or learning the embedding as in VAEs.

Original: It also provides a mathematical framework to analyze the statistical properties of these generators through the resolution of an inverse problem, regularized by the convolutional network architecture and the sparsity of the obtained activations.
Revised: It also provides a mathematical framework to analyze the statistical properties of these generators through the resolution of an inverse problem, regularized by the convolutional network architecture and the sparsity of the obtained activations.

Original: Because the embedding function is known, numerical results can be evaluated on training as well as testing samples.
Revised: Because the embedding function is known, numerical results can be evaluated on training as well as testing samples.

Original: We report preliminary numerical results with no hyperparameter optimization.
Revised: We report preliminary numerical results with no hyperparameter optimization.

Original: Test on noise vector arithmetic and conditional generations have not yet been carried.
Revised: Test on noise vector arithmetic and conditional generations have not yet been carried.

Original: Potential numerical improvements have not been explored either.
Revised: Potential numerical improvements have not been explored either.

Original: In particular, the architecture of the convolutional generator can be adapted to the properties of the forward scattering operator S j as j increases.
Revised: In particular, the architecture of the convolutional generator can be adapted to the properties of the forward scattering operator S j as j increases.

Original: Also, the paper uses a “plain” scattering transform which does not take into account interactions between angle and scale variables, which may also improve the representation as explained in Oyallon & Mallat (2015).
Revised: Also, the paper uses a “plain” scattering transform which does not take into account interactions between angle and scale variables, which may also improve the representation as explained in Oyallon & Mallat (2015).