Original: A MBIENT GAN: G ENERATIVE MODELS FROM LOSSY MEASUREMENTS
Revised: A MBIENT GAN: G ENERATIVE MODELS FROM LOSSY MEASUREMENTS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Generative models provide a way to model structure in complex distributions and have been shown to be useful for many tasks of practical interest.
Revised: Generative models provide a way to model structure in complex distributions and have been shown to be useful for many tasks of practical interest.

Original: However, current techniques for training generative models require access to fully-observed samples.
Revised: However, current techniques for training generative models require access to fully-observed samples.

Original: In many settings, it is expensive or even impossible to obtain fullyobserved samples, but economical to obtain partial, noisy observations.
Revised: In many settings, it is expensive or even impossible to obtain fullyobserved samples, but economical to obtain partial, noisy observations.

Original: We consider the task of learning an implicit generative model given only lossy measurements of samples from the distribution of interest.
Revised: We consider the task of learning an implicit generative model given only lossy measurements of samples from the distribution of interest.

Original: We show that the true underlying distribution can be provably recovered even in the presence of per-sample information loss for a class of measurement models.
Revised: We show that the true underlying distribution can be provably recovered even in the presence of per-sample information loss for a class of measurement models.

Original: Based on this, we propose a new method of training Generative Adversarial Networks (GANs) which we call
Revised: Based on this, we propose a new method of training Generative Adversarial Networks (GANs) which we call

Original: AmbientGAN.
Revised: AmbientGAN.

Original: On three benchmark datasets, and for various measurement models, we demonstrate substantial qualitative and quantitative improvements.
Revised: On three benchmark datasets, and for various measurement models, we demonstrate substantial qualitative and quantitative improvements.

Original: Generative models trained with our method can obtain 2 - 4 x higher inception scores than the baselines.
Revised: Generative models trained with our method can obtain 2 - 4 x higher inception scores than the baselines.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Generative models are powerful tools to concisely represent the structure in large datasets.
Revised: Generative models are powerful tools to concisely represent the structure in large datasets.

Original: An implicit generative model is a mechanism that only speciﬁes a stochastic procedure to produce samples from a probability distribution.
Revised: An implicit generative model is a mechanism that only speciﬁes a stochastic procedure to produce samples from a probability distribution.

Original: These models are attractive since they do not require an explicit parametrization of the probability distribution they are trying to model.
Revised: These models are attractive since they do not require an explicit parametrization of the probability distribution they are trying to model.

Original: Recently, there has been substantial progress in neural-network based implicit generative models within the autoregressive and the adversarial framework.
Revised: Recently, there has been substantial progress in neural-network based implicit generative models within the autoregressive and the adversarial framework.

Original: The adversarial framework was pioneered by Generative Adversarial Networks (GANs)
Revised: The adversarial framework was pioneered by Generative Adversarial Networks (GANs)

Original: [Goodfellow et al.
Revised: [Goodfellow et al.

Original: In these models, a generator network attempts to map samples from a simple low-dimensional distribution (such as standard
Revised: In these models, a generator network attempts to map samples from a simple low-dimensional distribution (such as standard

Original: Gaussian) to points in a high-dimensional space that resemble the learned data distribution.
Revised: Gaussian) to points in a high-dimensional space that resemble the learned data distribution.

Original: At the same time, a discriminator network attempts to distinguish between real and generated samples.
Revised: At the same time, a discriminator network attempts to distinguish between real and generated samples.

Original: By setting up a min-max game between them, the two networks are jointly trained.
Revised: By setting up a min-max game between them, the two networks are jointly trained.

Original: The latent probability distribution along with the learned generator network deﬁne a stochastic procedure that can produce new samples.
Revised: The latent probability distribution along with the learned generator network deﬁne a stochastic procedure that can produce new samples.

Original: The adversarial framework has been shown to be extremely successful in modeling complex distributions [Berthelot et al.
Revised: The adversarial framework has been shown to be extremely successful in modeling complex distributions [Berthelot et al.

Original: ; Vondrick et al.
Revised: ; Vondrick et al.

Original: ; Pascual et al.
Revised: ; Pascual et al.

Original: ; Wu et al.
Revised: ; Wu et al.

Original: (2016)], and the priors induced by these models are useful for various applications [Shrivastava et al.
Revised: (2016)], and the priors induced by these models are useful for various applications [Shrivastava et al.

Original: ; Ho & Ermon (2016)].
Revised: ; Ho & Ermon (2016)].

Original: This procedure for training generative models requires access to a large number of fully-observed samples from the desired distribution.
Revised: This procedure for training generative models requires access to a large number of fully-observed samples from the desired distribution.

Original: Unfortunately, obtaining multiple high-resolution samples can be expensive or impractical for some applications.
Revised: Unfortunately, obtaining multiple high-resolution samples can be expensive or impractical for some applications.

Original: For example, many sensing and tomography problems ( e.g. MRI, CT Scan) require a large number of projections for good reconstruction.
Revised: For example, many sensing and tomography problems ( e.g. MRI, CT Scan) require a large number of projections for good reconstruction.

Original: Compressed sensing [Donoho (2006); Candes et al.
Revised: Compressed sensing [Donoho (2006); Candes et al.

Original: (2006)] attempts to ameliorate this problem using  models of the data structure.
Revised: (2006)] attempts to ameliorate this problem using [Figure] models of the data structure.

Original: Recent work has shown that generative models can be particularly effective for easier sensing [Bora et al.
Revised: Recent work has shown that generative models can be particularly effective for easier sensing [Bora et al.

Original: (2017); Mardani et al.
Revised: (2017); Mardani et al.

Original: (2017)]—but if sensing is expensive in the ﬁrst place, how can we collect enough data to train a generative model to start with?
Revised: (2017)]—but if sensing is expensive in the ﬁrst place, how can we collect enough data to train a generative model to start with?

Original: This work solves this chicken-and-egg problem by training a generative model directly from noisy or incomplete samples.
Revised: This work solves this chicken-and-egg problem by training a generative model directly from noisy or incomplete samples.

Original: We show that our observations can be even projections or more general measurements of different types and the unknown distribution is still provably recoverable.
Revised: We show that our observations can be even projections or more general measurements of different types and the unknown distribution is still provably recoverable.

Original: A critical[Figure] assumption for our framework and theory to work is that the measurement process is known and satisﬁes certain technical conditions.
Revised: A critical assumption for our framework and theory to work is that the measurement process is known and satisﬁes certain technical conditions.

Original: We present several measurement processes for which it is possible to learn a generative model from a dataset of measured samples, both in theory and in practice.
Revised: We present several measurement processes for which it is possible to learn a generative model from a dataset of measured samples, both in theory and in practice.

Original: Our approach uses a new way of training GANs, which we call AmbientGAN.
Revised: Our approach uses a new way of training GANs, which we call AmbientGAN.

Original: The idea is simple: rather than distinguish a real image from a generated image as in a traditional GAN, our discriminator must distinguish a real measurement from a simulated measurement of a generated image; see Figure 1.
Revised: The idea is simple: rather than distinguish a real image from a generated image as in a traditional GAN, our discriminator must distinguish a real measurement from a simulated measurement of a generated image; see Figure 1.

Original: We empirically demonstrate the effectiveness of our approach on three datasets and a variety of measurement models.
Revised: We empirically demonstrate the effectiveness of our approach on three datasets and a variety of measurement models.

Original: Our method is able to construct good generative models from extremely noisy observations and even from low dimensional projections with drastic per-sample information loss.
Revised: Our method is able to construct good generative models from extremely noisy observations and even from low dimensional projections with drastic per-sample information loss.

Original: We show this qualitatively by exhibiting samples with good visual quality, and quantitatively by comparing inception scores [Salimans et al.
Revised: We show this qualitatively by exhibiting samples with good visual quality, and quantitatively by comparing inception scores [Salimans et al.

Original: (2016)] to baseline methods.
Revised: (2016)] to baseline methods.

Original: Theoretical results.
Revised: Theoretical results.

Original: We ﬁrst consider measurements that are noisy, blurred versions of the desired images.
Revised: We ﬁrst consider measurements that are noisy, blurred versions of the desired images.

Original: That is, we consider convolving the original image with a Gaussian kernel and adding independent Gaussian noise to each pixel (our actual theorem applies to more general kernels and noise distributions).
Revised: That is, we consider convolving the original image with a Gaussian kernel and adding independent Gaussian noise to each pixel (our actual theorem applies to more general kernels and noise distributions).

Original: Because of the noise, this process is not invertible for a single image.
Revised: Because of the noise, this process is not invertible for a single image.

Original: However, we show that the distribution of measured images uniquely determines the distribution of original images.
Revised: However, we show that the distribution of measured images uniquely determines the distribution of original images.

Original: This implies that a pure Nash equilibrium for the GAN game must ﬁnd a generative model that matches the true distribution.
Revised: This implies that a pure Nash equilibrium for the GAN game must ﬁnd a generative model that matches the true distribution.

Original: We show similar results for a dropout measurement model, where each pixel is set to zero with some probability p , and a random projection measurement model, where we observe the inner product of the image with a random Gaussian vector.
Revised: We show similar results for a dropout measurement model, where each pixel is set to zero with some probability p , and a random projection measurement model, where we observe the inner product of the image with a random Gaussian vector.

Original: Empirical results.
Revised: Empirical results.

Original: Our empirical work also considers measurement models for which we do not have provable guarantees.
Revised: Our empirical work also considers measurement models for which we do not have provable guarantees.

Original: We present results on some of our models now and defer the full exploration to Section 8.
Revised: We present results on some of our models now and defer the full exploration to Section 8.

Original: In Fig.
Revised: In Fig.

Original: 2, we consider the celebA dataset of celebrity faces [Liu et al.
Revised: 2, we consider the celebA dataset of celebrity faces [Liu et al.

Original: (2015)] under randomly placed occlusions, where a randomly placed square containing 1 / 4 of the pixels is set to zero.
Revised: (2015)] under randomly placed occlusions, where a randomly placed square containing 1 / 4 of the pixels is set to zero.

Original: It is hard to inpaint individual images, so cleaning up the data by inpainting and then learning a GAN on the result yields signiﬁcant artifacts.
Revised: It is hard to inpaint individual images, so cleaning up the data by inpainting and then learning a GAN on the result yields signiﬁcant artifacts.

Original: By incorporating the measurement process into the GAN training, we can produce much better samples.
Revised: By incorporating the measurement process into the GAN training, we can produce much better samples.

Original: In Fig.
Revised: In Fig.

Original: 3a we consider learning from noisy, blurred version of images from the celebA dataset.
Revised: 3a we consider learning from noisy, blurred version of images from the celebA dataset.

Original: Each image is convolved with a Gaussian kernel and then IID Gaussian noise is added to each pixel.
Revised: Each image is convolved with a Gaussian kernel and then IID Gaussian noise is added to each pixel.

Original: Learning a GAN on images denoised by Wiener deconvolution leads to poor sample quality while our models are able to produce cleaner samples.
Revised: Learning a GAN on images denoised by Wiener deconvolution leads to poor sample quality while our models are able to produce cleaner samples.

Original: In Fig.
Revised: In Fig.

Original: 3b, we consider learning a generative model on the 2 D images in the MNIST handwritten digit dataset [LeCun et al.
Revised: 3b, we consider learning a generative model on the 2 D images in the MNIST handwritten digit dataset [LeCun et al.

Original: from pairs of 1 D projections.
Revised: from pairs of 1 D projections.

Original: That is, measurements consist of picking two random lines and projecting the image onto each line, so the observed value along the line is the sum of all pixels that project to that point.
Revised: That is, measurements consist of picking two random lines and projecting the image onto each line, so the observed value along the line is the sum of all pixels that project to that point.

Original: We consider two variants: in the ﬁrst, the choice of line is forgotten, while in the second the measurement includes the choice of line.
Revised: We consider two variants: in the ﬁrst, the choice of line is forgotten, while in the second the measurement includes the choice of line.

Original: We ﬁnd for both variants that AmbientGAN recovers a lot of the underlying structure, although the ﬁrst variant cannot identify the distribution up to rotation or reﬂection.
Revised: We ﬁnd for both variants that AmbientGAN recovers a lot of the underlying structure, although the ﬁrst variant cannot identify the distribution up to rotation or reﬂection.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: There are two distinct approaches to constructing neural network based implicit generative models; autoregressive [Kingma & Welling (2013);
Revised: There are two distinct approaches to constructing neural network based implicit generative models; autoregressive [Kingma & Welling (2013);

Original: Oord et al.
Revised: Oord et al.

Original: (2016b;a)], and adversarial [Goodfellow et al.
Revised: (2016b;a)], and adversarial [Goodfellow et al.

Original: Some combination approaches have also been successful [Mescheder et al.
Revised: Some combination approaches have also been successful [Mescheder et al.

Original: The adversarial framework has been shown to be extremely powerful in modeling complex data distributions such as images [Radford et al.
Revised: The adversarial framework has been shown to be extremely powerful in modeling complex data distributions such as images [Radford et al.

Original: Arjovsky et al.
Revised: Arjovsky et al.

Original: ; Berthelot et al.
Revised: ; Berthelot et al.

Original: (2017)], video
Revised: (2017)], video

Original: [Liang et al.
Revised: [Liang et al.

Original: ; Vondrick et al.
Revised: ; Vondrick et al.

Original: (2016)], and 3D models [Achlioptas et al.
Revised: (2016)], and 3D models [Achlioptas et al.

Original: ; Wu et al.
Revised: ; Wu et al.

Original: A learned generative model can be useful for many applications.
Revised: A learned generative model can be useful for many applications.

Original: A string of papers [Bora et al.
Revised: A string of papers [Bora et al.

Original: ; Zhu et al.
Revised: ; Zhu et al.

Original: ; Yeh et al.
Revised: ; Yeh et al.

Original: (2016)] explore the utility of generative priors to solve ill-posed inverse problems.
Revised: (2016)] explore the utility of generative priors to solve ill-posed inverse problems.

Original: [Shrivastava et al.
Revised: [Shrivastava et al.

Original: (2016)] demonstrate that synthetic data can be made more realistic using GANs.
Revised: (2016)] demonstrate that synthetic data can be made more realistic using GANs.

Original: [Isola et al.
Revised: [Isola et al.

Original: and [Zhu et al.
Revised: and [Zhu et al.

Original: (2017)] show how to translate images from one domain to another using GANs.
Revised: (2017)] show how to translate images from one domain to another using GANs.

Original: The idea of operating generators and discriminators on different spaces has been proposed before.
Revised: The idea of operating generators and discriminators on different spaces has been proposed before.

Original: [Neyshabur et al.
Revised: [Neyshabur et al.

Original: (2017)] explores an interesting connection of training stability with low dimensional projections of samples.
Revised: (2017)] explores an interesting connection of training stability with low dimensional projections of samples.

Original: They show that training a generator against an array of discriminators, each operating on a different low-dimensional projection of the data can improve stability.
Revised: They show that training a generator against an array of discriminators, each operating on a different low-dimensional projection of the data can improve stability.

Original: Our work is also closely related to [Gadelha et al.
Revised: Our work is also closely related to [Gadelha et al.

Original: (2016)] where the authors create 3D object shapes from a dataset of 2D projections.
Revised: (2016)] where the authors create 3D object shapes from a dataset of 2D projections.

Original: We note that their setup is a special case of the AmbientGAN framework where the measurement process creates 2D projections using weighted sums of voxel occupancies.
Revised: We note that their setup is a special case of the AmbientGAN framework where the measurement process creates 2D projections using weighted sums of voxel occupancies.

Original:  N OTATION AND OUR APPROACH
Revised:  N OTATION AND OUR APPROACH

Original: Throughout, we use superscript ‘ r ’ to denote real or true distribution, superscript ‘ g ’ for the generated distributions, ‘ x ’ for the underlying space and ‘ y ’ for measurements.
Revised: Throughout, we use superscript ‘ r ’ to denote real or true distribution, superscript ‘ g ’ for the generated distributions, ‘ x ’ for the underlying space and ‘ y ’ for measurements.

Original: Let p rx be a real underlying distribution over R n .
Revised: Let p rx be a real underlying distribution over R n .

Original: We observe lossy measurements performed on samples from p rx .
Revised: We observe lossy measurements performed on samples from p rx .

Original: If we let m be the size of each observed measurement, then, each measurement is an output of some measurement function f θ : R n → R m , parameterized by θ .
Revised: If we let m be the size of each observed measurement, then, each measurement is an output of some measurement function f θ : R n → R m , parameterized by θ .

Original: We allow the measurement function to be stochastic by letting the parameters of the measurement functions have a distribution p θ .
Revised: We allow the measurement function to be stochastic by letting the parameters of the measurement functions have a distribution p θ .

Original: With this notation, for a given x and θ , the measurements are given by y =
Revised: With this notation, for a given x and θ , the measurements are given by y =

Original: f θ ( x ) .
Revised: f θ ( x ) .

Original: We assume that it is easy to sample Θ ∼ p θ and to compute f θ ( x ) for any x and θ .
Revised: We assume that it is easy to sample Θ ∼ p θ and to compute f θ ( x ) for any x and θ .

Original: The distributions p rx and p θ naturally induce a distribution over the measurements y which we shall denote by p ry .
Revised: The distributions p r x and p θ naturally induce a distribution over the measurements y which we shall denote by p ry .

Original: In other words, if X ∼ p rx and Θ ∼ p θ , then Y =
Revised: In other words, if X ∼ p rx and Θ ∼ p θ , then Y =

Original: f Θ ( X ) ∼ p ry .
Revised: f Θ ( X ) ∼ p ry .

Original: Our task is the following: there is some unknown distribution p rx and a known distribution p θ .
Revised: Our task is the following: there is some unknown distribution p rx and a known distribution p θ .

Original: We are given a set of IID realizations { y 1 , y 2 , . . .
Revised: We are given a set of IID realizations { y 1 , y 2 , . . .

Original: , y s } from the distribution p ry .
Revised: , y s } from the distribution p ry .

Original: Using these, our goal is to create an implicit generative model of p rx , i.e. , a stochastic procedure that can sample from p rx .
Revised: Using these, our goal is to create an implicit generative model of p rx , i.e. , a stochastic procedure that can sample from p rx .

Original: Our main idea is to combine the measurement process with the adversarial training framework, as shown in Fig.
Revised: Our main idea is to combine the measurement process with the adversarial training framework, as shown in Fig.

Original: Just like in the standard GAN setting, let Z ∈ R k , Z ∼ p
Revised: Just like in the standard GAN setting, let Z ∈ R k , Z ∼ p

Original: z be a random latent vector for a distribution p z
Revised: z be a random latent vector for a distribution p z

Original: that is easy to sample from, such as IID Gaussian or IID uniform.
Revised: that is easy to sample from, such as IID Gaussian or IID uniform.

Original: Let
Revised: Let

Original: G : R k
Revised: G : R k

Original: → R n be a generator.
Revised: → R n be a generator.

Original: Let X g =
Revised: Let X g =

Original: G ( Z ) , and let p gx be the distribution of X g .
Revised: G ( Z ) , and let p gx be the distribution of X g .

Original: Thus, our goal is to learn a generator G such that p gx is close to p rx .
Revised: Thus, our goal is to learn a generator G such that p gx is close to p rx .

Original: However, unlike the standard GAN setting, we do not have access to the desired objects ( X ∼ p rx ).
Revised: However, unlike the standard GAN setting, we do not have access to the desired objects ( X ∼ p rx ).

Original: Instead, we only have a dataset of measurements (samples from Y ∼ p ry ).
Revised: Instead, we only have a dataset of measurements (samples from Y ∼ p ry ).

Original: Our main idea is to simulate random measurements on the generated objects X g , and use the discriminator to distinguish real measurements from fake measurements.
Revised: Our main idea is to simulate random measurements on the generated objects X g , and use the discriminator to distinguish real measurements from fake measurements.

Original: Thus, we sample a random measurement function f Θ by sampling Θ ∼ p θ and apply it on X g to obtain Y g = f Θ
Revised: Thus, we sample a random measurement function f Θ by sampling Θ ∼ p θ and apply it on X g to obtain Y g = f Θ

Original: ( X g ) =
Revised: ( X g ) =

Original: f Θ ( G ( Z )) .
Revised: f Θ ( G ( Z )) .

Original: Let p gy be the distribution of Y g .
Revised: Let p gy be the distribution of Y g .

Original: We set up the discriminator to predict if a given y is a sample from the real measurement distribution p ry as opposed to the generated measurement distribution p gy .
Revised: We set up the discriminator to predict if a given y is a sample from the real measurement distribution p ry as opposed to the generated measurement distribution p gy .

Original: Thus, the discriminator is a function D : R m → R .
Revised: Thus, the discriminator is a function D : R m → R .

Original: We let q ( · ) be the quality function that is used to deﬁne the objective, based on the discriminator output.
Revised: We let q ( · ) be the quality function that is used to deﬁne the objective, based on the discriminator output.

Original: For vanilla GAN, q ( x ) = log( x ) and for Wasserstein GAN [Arjovsky et al.
Revised: For vanilla GAN, q ( x ) = log( x ) and for Wasserstein GAN [Arjovsky et al.

Original: (2017)], q ( x ) = x .
Revised: (2017)], q ( x ) = x .

Original: Accordingly, the AmbientGAN objective is the following:
Revised: Accordingly, the AmbientGAN objective is the following:

Original: [Equation]
Revised: [Equation]

Original: We additionally require f θ to be differentiable with respect to its inputs for all θ .
Revised: We additionally require f θ to be differentiable with respect to its inputs for all θ .

Original: We implement
Revised: We implement

Original: G and D as feedforward neural networks.
Revised: G and D as feedforward neural networks.

Original: With these assumptions, our model is end-to-end differentiable and can be trained using an approach similar to the standard gradient-based GAN training procedure.
Revised: With these assumptions, our model is end-to-end differentiable and can be trained using an approach similar to the standard gradient-based GAN training procedure.

Original: In each iteration, we sample Z ∼ p
Revised: In each iteration, we sample Z ∼ p

Original: z , Θ ∼ p θ , and Y r ∼ U NIF { y 1 , y 2 , . . .
Revised: z , Θ ∼ p θ , and Y r ∼ U NIF { y 1 , y 2 , . . .

Original: , y s } to use them to compute stochastic gradients of the objective with respect to parameters in G and D by backpropagation.
Revised: , y s } to use them to compute stochastic gradients of the objective with respect to parameters in G and D by backpropagation.

Original: We alternate between updates to parameters of D and updates to parameters of G .
Revised: We alternate between updates to parameters of D and updates to parameters of G .

Original: We note that our approach is compatible with and complementary to the various improvements proposed to the GAN objective, network architectures, and the training procedures.
Revised: We note that our approach is compatible with and complementary to the various improvements proposed to the GAN objective, network architectures, and the training procedures.

Original: Additionally, we can easily incorporate additional information, such as per sample labels, in our framework through conditional versions of the generator and discriminator.
Revised: Additionally, we can easily incorporate additional information, such as per sample labels, in our framework through conditional versions of the generator and discriminator.

Original: This is exempliﬁed in our experiments, where we use unconditional and conditional versions of DCGAN
Revised: This is exempliﬁed in our experiments, where we use unconditional and conditional versions of DCGAN

Original: [Radford et al.
Revised: [Radford et al.

Original: unconditional Wasserstein GAN with gradient penalty [Gulrajani et al.
Revised: unconditional Wasserstein GAN with gradient penalty [Gulrajani et al.

Original: (2017)], and an Auxiliary Classiﬁer
Revised: (2017)], and an Auxiliary Classiﬁer

Original: Wasserstein GAN [Odena et al.
Revised: Wasserstein GAN [Odena et al.

Original: (2016)] with gradient penalty.
Revised: (2016)] with gradient penalty.

Original:  M EASUREMENT M ODELS
Revised:  M EASUREMENT M ODELS

Original: Now, we describe the measurement models that we use for our theoretical and empirical results.
Revised: Now, we describe the measurement models that we use for our theoretical and empirical results.

Original: We primarily focus on 2 D images and thus our measurement models are tailored to this setting.
Revised: We primarily focus on 2 D images and thus our measurement models are tailored to this setting.

Original: The
Revised: The

Original: AmbientGAN learning framework, however, is more general and can be used for other data formats and other measurement models as well.
Revised: AmbientGAN learning framework, however, is more general and can be used for other data formats and other measurement models as well.

Original: For the rest of this section, we assume that input to the measurement function ( x ) is a 2 D image.
Revised: For the rest of this section, we assume that input to the measurement function ( x ) is a 2 D image.

Original: We consider the following measurement models:
Revised: We consider the following measurement models:

Original: Block-Pixels : Each pixel is independently set to zero with probability p .
Revised: Block-Pixels : Each pixel is independently set to zero with probability p .

Original: Convolve+Noise :
Revised: Convolve+Noise :

Original: Let k be a convolution kernel and let Θ ∼ p θ be the distribution of noise.
Revised: Let k be a convolution kernel and let Θ ∼ p θ be the distribution of noise.

Original: Then the measurements are given by f Θ ( x ) =
Revised: Then the measurements are given by f Θ ( x ) =

Original: k ∗ x +Θ , where ∗ is the convolution operator.
Revised: k ∗ x +Θ , where ∗ is the convolution operator.

Original: Block-Patch : A randomly chosen k × k patch is set to zero.
Revised: Block-Patch : A randomly chosen k × k patch is set to zero.

Original: Keep-Patch : All pixels outside a randomly chosen k × k patch are set to zero.
Revised: Keep-Patch : All pixels outside a randomly chosen k × k patch are set to zero.

Original: Extract-Patch : A random k × k patch is extracted.
Revised: Extract-Patch : A random k × k patch is extracted.

Original: Note that unlike the previous measurement function, the information about the location of the patch is lost.
Revised: Note that unlike the previous measurement function, the information about the location of the patch is lost.

Original: Pad-Rotate-Project : We pad the image on all four sides by zeros.
Revised: Pad-Rotate-Project : We pad the image on all four sides by zeros.

Original: Then we rotate the image by a random angle ( θ ) about its center.
Revised: Then we rotate the image by a random angle ( θ ) about its center.

Original: The padding is done to make sure that the original pixels stay within the boundary.
Revised: The padding is done to make sure that the original pixels stay within the boundary.

Original: Finally, for each channel in the image, we sum the pixels along the vertical axis to get one measurement vector.
Revised: Finally, for each channel in the image, we sum the pixels along the vertical axis to get one measurement vector.

Original: PadRotate-Project- θ : This is the same as the previous measurement function, except that along with the projection values, the chosen angle is also included in the measurements.
Revised: PadRotate-Project- θ : This is the same as the previous measurement function, except that along with the projection values, the chosen angle is also included in the measurements.

Original: Gaussian-Projection : We project onto a random Gaussian vector which is included in the measurements.
Revised: Gaussian-Projection : We project onto a random Gaussian vector which is included in the measurements.

Original: So, Θ ∼ N (0 , I n ) , and f Θ ( x ) =
Revised: So, Θ ∼ N (0 , I n ) , and f Θ ( x ) =

Original: (Θ , (cid:104) Θ , x (cid:105) ) .
Revised: (Θ , (cid:104) Θ , x (cid:105) ) .

Original:  T HEORETICAL R ESULTS
Revised:  T HEORETICAL R ESULTS

Original: We show that we can provably recover the true underlying distribution p rx for certain measurement models.
Revised: We show that we can provably recover the true underlying distribution p rx for certain measurement models.

Original: Our broad approach is to show that there is a unique distribution p rx consistent with the observed measurement distribution p ry , i.e. , the mapping of distributions of samples p rx to distribution of measurements p ry is invertible even though the map from an individual image x to its measurements f θ ( x ) is not.
Revised: Our broad approach is to show that there is a unique distribution p rx consistent with the observed measurement distribution p ry , i.e. , the mapping of distributions of samples p rx to distribution of measurements p ry is invertible even though the map from an individual image x to its measurements f θ ( x ) is not.

Original: If this holds, then the following lemma immediately gives a consistency guarantee with the AmbientGAN training procedure.
Revised: If this holds, then the following lemma immediately gives a consistency guarantee with the AmbientGAN training procedure.

Original: Lemma 5.1.
Revised: Lemma 5.1.

Original: As in Section 3, let p rx be the data distribution, p θ be the distribution over parameters of the measurement functions and p ry be the induced measurement distribution.
Revised: As in Section 3, let p rx be the data distribution, p θ be the distribution over parameters of the measurement functions and p ry be the induced measurement distribution.

Original: Further, assume that for the given p θ , there is a unique probability distribution p rx that induces the given measurement distribution p ry .
Revised: Further, assume that for the given p θ , there is a unique probability distribution p rx that induces the given measurement distribution p ry .

Original: Then, for the vanilla GAN model [Goodfellow et al.
Revised: Then, for the vanilla GAN model [Goodfellow et al.

Original: (2014)], if the Discriminator D is optimal, so that D ( · ) = 
Revised: (2014)], if the Discriminator D is optimal, so that D ( · ) = [Equation]

Original: p ry ( · ) p ry ( · )+ p gy ( · ) , then a generator G is optimal iff p gx = p rx .
Revised: a generator [Equation] optimal [Equation]

Original: All proofs including this one are deferred to Appendix A. Note that the previous lemma makes a non-trivial assumption of uniqueness of the true underlying distribution given the measurement distribution.
Revised: All proofs including this one are deferred to Appendix A. Note that the previous lemma makes a non-trivial assumption of uniqueness of the true underlying distribution given the measurement distribution.

Original: The next few theorems show that this assumption is satisﬁed under Gaussian-Projection,
Revised: The next few theorems show that this assumption is satisﬁed under Gaussian-Projection,

Original: Convolve+Noise and Block-Pixels measurement models, thus showing that that we can recover the true underlying distribution with the AmbientGAN framework.
Revised: Convolve+Noise and Block-Pixels measurement models, thus showing that that we can recover the true underlying distribution with the AmbientGAN framework.

Original: Theorem 5.2.
Revised: Theorem 5.2.

Original: For the Gaussian-Projection measurement model (Section 4), there is a unique underlying distribution p r x that can induce the observed measurement distribution p r y .
Revised: For the Gaussian-Projection measurement model (Section 4), there is a unique underlying distribution p rx that can induce the observed measurement distribution p ry .

Original: Theorem 5.3.
Revised: Theorem 5.3.

Original: Let F ( · ) denote the Fourier transform and let supp( · ) be the support of a function.
Revised: Let F ( · ) denote the Fourier transform and let supp( · ) be the support of a function.

Original: Consider the Convolve+Noise measurement model (Section 4) with the convolution kernel k and additive noise distribution p θ .
Revised: Consider the Convolve+Noise measurement model (Section 4) with the convolution kernel k and additive noise distribution p θ .

Original: If supp( F ( k )) c = φ and supp( F ( p θ ))
Revised: If supp( F ( k )) c = φ and supp( F ( p θ ))

Original: c
Revised: c

Original: = φ , then there is a unique distribution p rx that can induce the measurement distribution p ry .
Revised: = φ , then there is a unique distribution p rx that can induce the measurement distribution p ry .

Original: We remark that the required conditions in the preceding theorem are easily satisﬁed for the common setting of Gaussian blurring kernel with additive Gaussian noise.
Revised: We remark that the required conditions in the preceding theorem are easily satisﬁed for the common setting of Gaussian blurring kernel with additive Gaussian noise.

Original: The same guarantee can be generalized for any continuous and invertible function instead of a convolution.
Revised: The same guarantee can be generalized for any continuous and invertible function instead of a convolution.

Original: We omit the details.
Revised: We omit the details.

Original: Our next theorem makes an assumption of a ﬁnite discrete set of pixel values.
Revised: Our next theorem makes an assumption of a ﬁnite discrete set of pixel values.

Original: This assumption holds in most practical scenarios since images are represented with a ﬁnite number of discrete values per channel.
Revised: This assumption holds in most practical scenarios since images are represented with a ﬁnite number of discrete values per channel.

Original: In this setting, in addition to a consistency guarantee, we also give a sample complexity result for approximately learning the distributions in the AmbientGAN framework.
Revised: In this setting, in addition to a consistency guarantee, we also give a sample complexity result for approximately learning the distributions in the AmbientGAN framework.

Original: Theorem 5.4.
Revised: Theorem 5.4.

Original: Assume that each image pixel takes values in a ﬁnite set P. Thus x ∈ P n ⊂ R n .
Revised: Assume that each image pixel takes values in a ﬁnite set P. Thus x ∈ P n ⊂ R n .

Original: Assume 0 ∈ P , and consider the Block-Pixels measurement model (Section 4) with p being the probability of blocking a pixel.
Revised: Assume 0 ∈ P , and consider the Block-Pixels measurement model (Section 4) with p being the probability of blocking a pixel.

Original: If p < 1 , then there is a unique distribution p rx that can induce the measurement distribution p ry .
Revised: If p < 1 , then there is a unique distribution p rx that can induce the measurement distribution p ry .

Original: Further, for any (cid:15) > 0 , δ ∈ (0 , 1] , given a dataset ofs = Ω (cid:32) | P | 2 n (1 − p ) 2 n (cid:15) 2 log (cid:18)
Revised: Further, for any (cid:15) > 0 , δ ∈ (0 , 1] , given a dataset of

Original: | P | n δ (cid:19)(cid:33)
Revised: [Equation]

Original: IID measurement samples from p ry , if the discriminator D is optimal, then with probability ≥ 1 − δ over the dataset, any optimal generator G must satisfy d TV ( p gx , p rx ) ≤ (cid:15) , where d TV ( · , · ) is the total variation distance.
Revised: IID measurement samples from p ry , if the discriminator D is optimal, then with probability ≥ 1 − δ over the dataset, any optimal generator G must satisfy d TV ( p gx , p rx ) ≤ (cid:15) , where d TV ( · , · ) is the total variation distance.

Original:  D ATASETS AND M ODEL A RCHITECTURES
Revised:  D ATASETS AND M ODEL A RCHITECTURES

Original: We used three datasets for our experiments.
Revised: We used three datasets for our experiments.

Original: MNIST is a dataset of 28 × 28 images of handwritten digits [LeCun et al.
Revised: MNIST is a dataset of 28 × 28 images of handwritten digits [LeCun et al.

Original: CelebA is a dataset of face images of celebrities [Liu et al.
Revised: CelebA is a dataset of face images of celebrities [Liu et al.

Original: We use an aligned and cropped version where each image is 64 × 64 RGB.
Revised: We use an aligned and cropped version where each image is 64 × 64 RGB.

Original: The CIFAR-10 dataset consists of 32 × 32 RGB images from 10 different classes [Krizhevsky & Hinton (2009)].
Revised: The CIFAR-10 dataset consists of 32 × 32 RGB images from 10 different classes [Krizhevsky & Hinton (2009)].

Original: We brieﬂy describe the generative models we used for our experiments.
Revised: We brieﬂy describe the generative models we used for our experiments.

Original: More details on architectures and hyperparameters can be found in the appendix.
Revised: More details on architectures and hyperparameters can be found in the appendix.

Original: For the MNIST dataset, we use two
Revised: For the MNIST dataset, we use two

Original: GAN models.
Revised: GAN models.

Original: The ﬁrst model is a conditional DCGAN which follows the architecture in [Radford et al.
Revised: The ﬁrst model is a conditional DCGAN which follows the architecture in [Radford et al.

Original: (2015)] 1 , while the second model is an unconditional Wasserstein GAN with gradient penalty (WGANGP) which follows the architecture in [Gulrajani et al.
Revised: (2015)] 1 , while the second model is an unconditional Wasserstein GAN with gradient penalty (WGANGP) which follows the architecture in [Gulrajani et al.

Original: For the celebA dataset, we use an unconditional DCGAN and follow the architecture in [Radford et al.
Revised: For the celebA dataset, we use an unconditional DCGAN and follow the architecture in [Radford et al.

Original: For the CIFAR-10 dataset, we use an Auxiliary Classiﬁer Wasserstein GAN with gradient penalty (ACWGANGP) which follows the residual architecture in [Gulrajani et al.
Revised: For the CIFAR-10 dataset, we use an Auxiliary Classiﬁer Wasserstein GAN with gradient penalty (ACWGANGP) which follows the residual architecture in [Gulrajani et al.

Original: For measurements with 2 D outputs, i.e. Block-Pixels, Block-Patch, Keep-Patch, Extract-Patch, and
Revised: For measurements with 2 D outputs, i.e. Block-Pixels, Block-Patch, Keep-Patch, Extract-Patch, and

Original: Convolve+Noise (see Section 4), we use the same discriminator architectures as in the original work.
Revised: Convolve+Noise (see Section 4), we use the same discriminator architectures as in the original work.

Original: For 1 D projections, i.e. Pad-Rotate-Project, Pad-Rotate-Project- θ , we use fully connected discriminators.
Revised: For 1 D projections, i.e. Pad-Rotate-Project, Pad-Rotate-Project- θ , we use fully connected discriminators.

Original: The architecture of the fully connected discriminator used for the MNIST dataset was 25 - 25 - 1 and for the celebA dataset was 100 - 100 - 1 .
Revised: The architecture of the fully connected discriminator used for the MNIST dataset was 25 - 25 - 1 and for the celebA dataset was 100 - 100 - 1 .

Original:  B ASELINES
Revised:  B ASELINES

Original: Now, we describe some baseline approaches that we implemented to evaluate the relative performance of the AmbientGAN framework.
Revised: Now, we describe some baseline approaches that we implemented to evaluate the relative performance of the AmbientGAN framework.

Original: Recall that we have a dataset of IID samples { y 1 , y 2 , . . .
Revised: Recall that we have a dataset of IID samples { y 1 , y 2 , . . .

Original: y s } from the measurement distribution p ry and our goal is to create an implicit generative model for p rx .
Revised: y s } from the measurement distribution p ry and our goal is to create an implicit generative model for p rx .

Original: A crude baseline is to ignore that any measurement happened at all.
Revised: A crude baseline is to ignore that any measurement happened at all.

Original: In other words, for cases where the measurements lie in the same space as the full-samples (for example Convolve+Noise)
Revised: In other words, for cases where the measurements lie in the same space as the full-samples (for example Convolve+Noise)

Original: we can learn a generative model directly on the measurements and test how well it approximates the true distribution p rx .
Revised: we can learn a generative model directly on the measurements and test how well it approximates the true distribution p rx .

Original: We call this the “ignore” baseline.
Revised: We call this the “ignore” baseline.

Original: A stronger baseline is based on the following observation: If the measurement functions f θ were invertible, and we observed θ i for each measurement
Revised: A stronger baseline is based on the following observation: If the measurement functions f θ were invertible, and we observed θ i for each measurement

Original: y i in our dataset, we could just invert the functions to obtain full-samples x i = f
Revised: y i in our dataset, we could just invert the functions to obtain full-samples x i = f

Original: − 1 θ i
Revised: − 1 θ i

Original: ( y i ) .
Revised: ( y i ) .

Original: Then we could directly learn a generative model using these full-samples.
Revised: Then we could directly learn a generative model using these full-samples.

Original: Notice that both assumptions are violated in the AmbientGAN setting.
Revised: Notice that both assumptions are violated in the AmbientGAN setting.

Original: First, we may not observe θ i and second, the functions may not be invertible.
Revised: First, we may not observe θ i and second, the functions may not be invertible.

Original: Indeed all the measurement models in Section 4 violate one of the assumptions.
Revised: Indeed all the measurement models in Section 4 violate one of the assumptions.

Original: However, we can try to approximate an inverse function and use the inverted samples to train a generative model.
Revised: However, we can try to approximate an inverse function and use the inverted samples to train a generative model.

Original: Thus, given a measurement y i =
Revised: Thus, given a measurement y i =

Original: f θ i ( x i ) , we try to “unmeasure” it and obtain (cid:98) x
Revised: f θ i ( x i ) , we try to “unmeasure” it and obtain (cid:98) x

Original: i , an estimate of x i .
Revised: i , an estimate of x i .

Original: We then learn a generative model with the estimated inverse samples and test how well it approximates p rx .
Revised: We then learn a generative model with the estimated inverse samples and test how well it approximates p rx .

Original:  For the measurement models described in Section 4, we now describe the methods we used to obtain approximate inverse functions: (a) For the Block-Pixels measurements, a simple approximate inverse function is to just blur the image so that zero pixels are ﬁlled in from the surrounding.
Revised: [Figure] For the measurement models described in Section 4, we now describe the methods we used to obtain approximate inverse functions: (a) For the Block-Pixels measurements, a simple approximate inverse function is to just blur the image so that zero pixels are ﬁlled in from the surrounding.

Original: We also implemented a more sophisticated approach to ﬁll in the pixels by using total variation inpainting.
Revised: We also implemented a more sophisticated approach to ﬁll in the pixels by using total variation inpainting.

Original: (b) For Convolve+Noise measurements with a Gaussian kernel and additive Gaussian
Revised: (b) For Convolve+Noise measurements with a Gaussian kernel and additive Gaussian

Original: [Figure]

Original: Noise, we approximate the inverse by a Wiener deconvolution.
Revised: Noise, we approximate the inverse by a Wiener deconvolution.

Original: (c) For Block-Patch measurements, we use the Navier Stokes based inpainting method [Bertalmio et al.
Revised: (c) For Block-Patch measurements, we use the Navier Stokes based inpainting method [Bertalmio et al.

Original: (2001)] to ﬁll in the zero pixels.
Revised: (2001)] to ﬁll in the zero pixels.

Original: For other measurement models, it is unclear how to obtain an approximate inverse function.
Revised: For other measurement models, it is unclear how to obtain an approximate inverse function.

Original: For the
Revised: For the

Original: Keep-Patch measurement model, no pixels outside a box are known and thus inpainting methods are not suitable.
Revised: Keep-Patch measurement model, no pixels outside a box are known and thus inpainting methods are not suitable.

Original: Inverting Extract-Patch measurements is even harder since the information about the position of the patch is also lost.
Revised: Inverting Extract-Patch measurements is even harder since the information about the position of the patch is also lost.

Original: For the Pad-Rotate-Project- θ measurements, a conventional technique is to sample many angles, and use techniques for inverting the Radon transform [Deans (2007)].
Revised: For the Pad-Rotate-Project- θ measurements, a conventional technique is to sample many angles, and use techniques for inverting the Radon transform [Deans (2007)].

Original: However, since we observe only a few projections at a time, these methods aren’t readily applicable.
Revised: However, since we observe only a few projections at a time, these methods aren’t readily applicable.

Original: Inverting Pad-Rotate-Project measurements is even harder since it lacks information about θ .
Revised: Inverting Pad-Rotate-Project measurements is even harder since it lacks information about θ .

Original: So, on this subset of experiments, we report only the results with the AmbientGAN models.
Revised: So, on this subset of experiments, we report only the results with the AmbientGAN models.

Original:  Q UALITATIVE R ESULTS
Revised:  Q UALITATIVE R ESULTS

Original: We present some samples generated by the baselines and our models.
Revised: We present some samples generated by the baselines and our models.

Original: For each experiment, we show the samples from the dataset of measurements ( Y r ) available for training, samples generated by the baselines (when applicable) and the samples generated by our models ( X g ).
Revised: For each experiment, we show the samples from the dataset of measurements ( Y r ) available for training, samples generated by the baselines (when applicable) and the samples generated by our models ( X g ).

Original: We show samples only for a selected value of parameter settings.
Revised: We show samples only for a selected value of parameter settings.

Original: More results are provided in the appendix.
Revised: More results are provided in the appendix.

Original: All results on
Revised: All results on

Original: MNIST are deferred to the appendix.
Revised: MNIST are deferred to the appendix.

Original: Block-Pixels : Fig.
Revised: Block-Pixels : Fig.

Original: 4 shows results on celebA with DCGAN and Fig.
Revised: 4 shows results on celebA with DCGAN and Fig.

Original: 6 on CIFAR-10 with ACWGANGP.
Revised: 6 on CIFAR-10 with ACWGANGP.

Original: We see that the samples are heavily degraded in our measurement process (left image).
Revised: We see that the samples are heavily degraded in our measurement process (left image).

Original: Thus, it is challenging for baselines to invert the measurements process, and correspondingly,the they do not produce good samples (middle image).
Revised: Thus, it is challenging for baselines to invert the measurements process, and correspondingly, they do not produce good samples (middle image).

Original: Our models are able to produce images with good visual quality (right image).
Revised: Our models are able to produce images with good visual quality (right image).

Original: Convolve+Noise :
Revised: Convolve+Noise :

Original: We use a Gaussian kernel and IID Gaussian noise.
Revised: We use a Gaussian kernel and IID Gaussian noise.

Original: Fig.
Revised: Fig.

Original: 3a  on celebA with DCGAN.
Revised: 3a shows results on celebA with DCGAN.

Original: We see that the measurements are drowned in noise (left image) and the baselines struggle to extract the original image, giving samples of low quality (middle image).
Revised: We see that the measurements are drowned in noise (left image) and the baselines struggle to extract the original image, giving samples of low quality (middle image).

Original: Our models are able to produce samples that resemble digits on MNIST, and clear faceson celebA (right image).
Revised: Our models are able to produce samples with clear faces (right image).

Original: Block-Patch, Keep-Patch : Fig.
Revised: Block-Patch, Keep-Patch : Fig.

Original: 2 shows the results for Block-Patch and Fig.
Revised: 2 shows the results for Block-Patch and Fig.

Original: 5a for Keep-Patch measurements on celebA with DCGAN.
Revised: 5a for Keep-Patch measurements on celebA with DCGAN.

Original: On both measurement distributions, our models are able to create coherent images by observing only parts of one image at a time (right image).
Revised: On both measurement distributions, our models are able to create coherent faces (right image) by observing only parts of one image at a time.

Original: 1D projections : Pad-Rotate-Project and Pad-Rotate-Project- θ measurement models exhibit drastic signal degradation, where most of the information in a sample is lost during the measurements process.
Revised: 1D projections : Pad-Rotate-Project and Pad-Rotate-Project- θ measurement models exhibit drastic signal degradation; most of the information in a sample is lost during the measurements process.

Original: For our experiments, we use two measurements at a time.
Revised: For our experiments, we use two measurements at a time.

Original: Fig.
Revised: Fig.

Original: 3b shows the results on MNIST with DCGAN.
Revised: 3b shows the results on MNIST with DCGAN.

Original: While the ﬁrst model is able to learn only up to rotation and reﬂection (left image), we note that generated digits have similar orientations and chirality within each class without any explicit incentive.
Revised: While the ﬁrst model is able to learn only up to rotation and reﬂection (left image), we note that generated digits have similar orientations and chirality within each class without any explicit incentive.

Original: We hypothesize that the model prefers this mode because it is easier to learn with consistent orientation per class.
Revised: We hypothesize that the model prefers this mode because it is easier to learn with consistent orientation per class.

Original: The second measurement model contains the rotation angle and thus produces upright digits (right image).
Revised: The second measurement model contains the rotation angle and thus produces upright digits (right image).

Original: While in both cases, the generated images are of lesser visual quality, our method demonstrates that we can produce images of digits given only 1 D projections.
Revised: While in both cases, the generated images are of lesser visual quality, our method demonstrates that we can produce images of digits given only 1 D projections.

Original: Failure case : In Fig.
Revised: Failure case : In Fig.

Original: 5b, we show the samples obtained from our model trained on celebA dataset with Pad-Rotate-Project- θ measurements with a DCGAN.
Revised: 5b, we show the samples obtained from our model trained on celebA dataset with Pad-Rotate-Project- θ measurements with a DCGAN.

Original: We see that the model has learned a very crude outline of a face, but lacks details.
Revised: We see that the model has learned a very crude outline of a face, but lacks details.

Original: This highlights the difﬁculty in learning complex distributions with just 1 D projections and a need for better understanding of distribution recovery under projection measurement model as well as better methods for training GANs.
Revised: This highlights the difﬁculty in learning complex distributions with just 1 D projections and a need for better understanding of distribution recovery under projection measurement model as well as better methods for training GANs.

Original:  Q UANTITATIVE R ESULTS
Revised:  Q UANTITATIVE R ESULTS

Original: We report inception scores [Salimans et al.
Revised: We report inception scores [Salimans et al.

Original: to quantify the quality of the generative models learned in the AmbientGAN framework.
Revised: to quantify the quality of the generative models learned in the AmbientGAN framework.

Original: For the CIFAR-10 dataset, we use the Inception model[Szegedy et al.
Revised: For the CIFAR-10 dataset, we use the Inception model[Szegedy et al.

Original: (2016)] trained on the ImageNet dataset [Deng et al.
Revised: (2016)] trained on the ImageNet dataset [Deng et al.

Original: For computing a similar score on MNIST, we trained a classiﬁcation model with two conv+pool layers followed by two fully connected layers 2 .
Revised: For computing a similar score on MNIST, we trained a classiﬁcation model with two conv+pool layers followed by two fully connected layers 2 .

Original: The ﬁnal test set accuracy of this model was 99 . 2% .
Revised: The ﬁnal test set accuracy of this model was 99 . 2% .

Original: 9. MNIST
Revised: 9. MNIST

Original: For Block-Pixels measurements on MNIST, we trained several models with our approach and the baselines, each with a different probability p of blocking pixels.
Revised: For Block-Pixels measurements on MNIST, we trained several models with our approach and the baselines, each with a different probability p of blocking pixels.

Original: For each model, after convergence, we computed the inception score using the network described above.
Revised: For each model, after convergence, we computed the inception score using the network described above.

Original: A plot of the inception scores as a function of p is shown in Fig.
Revised: A plot of the inception scores as a function of p is shown in Fig.

Original: 7 (left).
Revised: 7 (left).

Original: We note that at p = 0 , i.e. if no pixels are blocked, our model is equivalent to a conventional GAN.
Revised: We note that at p = 0 , i.e. if no pixels are blocked, our model is equivalent to a conventional GAN.

Original: As we increase p , the baseline models quickly start to perform poorly, while the AmbientGAN models continue to perform relatively well. 
Revised: As we increase p , the baseline models quickly start to perform poorly, while the AmbientGAN models continue to perform relatively well. [Figure] [Figure]

Original: For the Convolve+Noise measurements with a Gaussian kernel of radius 1 pixel, and additive Gaussian noise with zero mean and standard deviation σ , we trained several models on MNIST by varying the value of σ .
Revised: For the Convolve+Noise measurements with a Gaussian kernel of radius 1 pixel, and additive Gaussian noise with zero mean and standard deviation σ , we trained several models on MNIST by varying the value of σ .

Original: A plot of the inception score as a function of σ is shown in Fig.
Revised: A plot of the inception score as a function of σ is shown in Fig.

Original: 7 (right).
Revised: 7 (right).

Original: We see that for small variance of additive noise, Wiener deconvolution and the “ignore” baseline perform quite well.
Revised: We see that for small variance of additive noise, Wiener deconvolution and the “ignore” baseline perform quite well.

Original: However, as we start to increase the noise levels, these baselines quickly deteriorate in performance, while the AmbientGAN models maintain a high inception score.[Figure] [Figure]
Revised: However, as we start to increase the noise levels, these baselines quickly deteriorate in performance, while the AmbientGAN models maintain a high inception score.

Original: For 1 D projection measurements, we report the inception scores for the samples produced by the
Revised: For 1 D projection measurements, we report the inception scores for the samples produced by the

Original: AmbientGAN models trained with two projection measurements at a time.
Revised: AmbientGAN models trained with two projection measurements at a time.

Original: The Pad-Rotate-Project model produces digits at various orientations and thus does quite poorly, achieving an inception score of just 4 .
Revised: The Pad-Rotate-Project model produces digits at various orientations and thus does quite poorly, achieving an inception score of just 4 .

Original: The model with Pad-Rotate-Project- θ measurements produces well-aligned digits and achieves an inception score of 8 . 12 .
Revised: The model with Pad-Rotate-Project- θ measurements produces well-aligned digits and achieves an inception score of 8 . 12 .

Original: For comparison, the vanilla GAN model trained with fullyobserved samples achieves an inception score of 8 . 99 .
Revised: For comparison, the vanilla GAN model trained with fullyobserved samples achieves an inception score of 8 . 99 .

Original: Thus, the second model comes quite close to the performance of the fully-observed case while being trained only on 1 D projections.
Revised: Thus, the second model comes quite close to the performance of the fully-observed case while being trained only on 1 D projections.

Original: 9. CIFAR- 
Revised: 9. CIFAR-

Original: In Fig. 8 (left), we show a plot of inception score vs the probability of blocking pixels p in the BlockPixels measurement model on CIFAR-10.
Revised: In Fig. 8 (left), we show a plot of inception score vs the probability of blocking pixels p in the BlockPixels measurement model on CIFAR-10.

Original: We note that the total variation inpainting method is quite slow and the performance on MNIST was about the same as unmeasure-blur baseline.
Revised: We note that the total variation inpainting method is quite slow and the performance on MNIST was about the same as unmeasure-blur baseline.

Original: So, we do not run inpainting baselines on the CIFAR-10 dataset.
Revised: So, we do not run inpainting baselines on the CIFAR-10 dataset.

Original: From the plots, we see a trend similar to the plot obtained with MNIST (Fig. 7, left), showing the superiority of our approach over baselines.
Revised: From the plots, we see a trend similar to the plot obtained with MNIST (Fig. 7, left), showing the superiority of our approach over baselines.

Original: We show the inception score as a function of training iteration in Fig.
Revised: We show the inception score as a function of training iteration in Fig.

Original: 8 (right).
Revised: 8 (right).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Generative models are powerful tools, but constructing a generative model requires a large, highquality dataset of the distribution of interest.
Revised: Generative models are powerful tools, but constructing a generative model requires a large, highquality dataset of the distribution of interest.

Original: We show how to relax this requirement, by learning a distribution from a dataset that only contains incomplete, noisy measurements of the distribution.
Revised: We show how to relax this requirement, by learning a distribution from a dataset that only contains incomplete, noisy measurements of the distribution.

Original: We hope that this will allow for the construction of new generative models of distributions for which no high-quality dataset exists.
Revised: We hope that this will allow for the construction of new generative models of distributions for which no high-quality dataset exists.

Revised: A CKNOWLEDGEMENTS

Revised: We would like to thank Philipp Kr¨ahenb¨uhl, Ajil Jalal, Surbhi Goel and Jessica Hoffmann for helpful discussions.

Revised: This research has been supported by NSF Grants CCF 1344364, 1407278, 1422549, 1618689, and ARO YIP W911NF-14-1-0258.

Original: A PPENDIX A 10. P ROOF OF L EMMA 5. 
Revised: A PPENDIX A 10. P ROOF OF L EMMA 5.

Original: Lemma.
Revised: Lemma.

Original: As in Section 3, let p rx be the data distribution, p θ be the distribution over parameters of the measurement functions and p ry be the induced measurement distribution.
Revised: As in Section 3, let p rx be the data distribution, p θ be the distribution over parameters of the measurement functions and p ry be the induced measurement distribution.

Original: Further, assume that for the given p θ , there is a unique probability distribution p rx that induces the given measurement distribution p ry .
Revised: Further, assume that for the given p θ , there is a unique probability distribution p rx that induces the given measurement distribution p ry .

Original: Then, for the vanilla GAN model [Goodfellow et al.
Revised: Then, for the vanilla GAN model [Goodfellow et al.

Original: (2014)], if the Discriminator D is optimal,
Revised: (2014)], if the Discriminator D is optimal,

Original: so that
Revised: so that

Original: [Equation]
Revised: [Equation]

Original: then a generator G is optimal iff p gx = p rx .
Revised: then a generator G is optimal iff p gx = p rx .

Original: Proof.
Revised: Proof.

Original: From the same argument as in Theorem 1 in [Goodfellow et al.
Revised: From the same argument as in Theorem 1 in [Goodfellow et al.

Original: (2014)], it follows that p gy = p ry .
Revised: (2014)], it follows that p gy = p ry .

Original: Then, since there is a unique probability distribution p rx that can induce the given measurement distribution, it follows that p gx = p rx .
Revised: Then, since there is a unique probability distribution p rx that can induce the given measurement distribution, it follows that p gx = p rx .

Original: The converse is trivially true.
Revised: The converse is trivially true.

Original: 10. P ROOF OF T HEOREM 5. 
Revised: 10. P ROOF OF T HEOREM 5.

Original: Theorem.
Revised: Theorem.

Original: For the Gaussian-Projection measurement model (Section 4), there is a unique underlying distribution p rx that can induce the observed measurement distribution p ry .
Revised: For the Gaussian-Projection measurement model (Section 4), there is a unique underlying distribution p rx that can induce the observed measurement distribution p ry .

Original: Proof.
Revised: Proof.

Original: We note that Since Θ ∼ N (0 , I n ) , all possible directions for projections are covered.
Revised: We note that Since Θ ∼ N (0 , I n ) , all possible directions for projections are covered.

Original: Further, since the measurement model includes the projection vector Θ as a part of the measurements, in order to match the measurement distribution, the underlying distribution p rx must be such that all 1 D marginals are matched.
Revised: Further, since the measurement model includes the projection vector Θ as a part of the measurements, in order to match the measurement distribution, the underlying distribution p rx must be such that all 1 D marginals are matched.

Original: Thus, by Cramer-Wold theorem [Cram ´ er & Wold (1936)], any sequence of random vectors that match the 1D marginals must converge in distribution to the true underlying distribution.
Revised: Thus, by Cramer-Wold theorem [Cram ´ er & Wold (1936)], any sequence of random vectors that match the 1D marginals must converge in distribution to the true underlying distribution.

Original: Thus, in particular, there is a unique probability distribution p rx that can match all 1 D marginals obtained with the Gaussian projection measurements.
Revised: Thus, in particular, there is a unique probability distribution p rx that can match all 1 D marginals obtained with the Gaussian projection measurements.

Original: 10. P ROOF OF T HEOREM 5. 
Revised: 10. P ROOF OF T HEOREM 5.

Original: Theorem.
Revised: Theorem.

Original: Let F ( · ) denote the Fourier transform and let supp( · ) be the support of a function.
Revised: Let F ( · ) denote the Fourier transform and let supp( · ) be the support of a function.

Original: Consider the Convolve+Noise measurement model (Section 4) with the convolution kernel k and additive noise distribution p θ .
Revised: Consider the Convolve+Noise measurement model (Section 4) with the convolution kernel k and additive noise distribution p θ .

Original: If supp( F ( k )) c = φ and supp( F ( p η ))
Revised: If supp( F ( k )) c = φ and supp( F ( p θ ))

Original: c
Revised: c

Original: = φ , then there is a unique distribution p rx that can induce the measurement distribution p ry .
Revised: = φ , then there is a unique distribution p rx that can induce the measurement distribution p ry .

Original: Proof.
Revised: Proof.

Original: Let X ∼ p x .
Revised: Let X ∼ p x .

Original: Let Θ ∼ p θ .
Revised: Let Θ ∼ p θ .

Original: Let Z = k ∗ X so that Z ∼ p z , and Y =
Revised: Let Z = k ∗ X so that Z ∼ p z , and Y =

Original: Z +Θ , so Y ∼ p y .
Revised: Z +Θ , so Y ∼ p y .

Original: With a slight abuse of notation, we will denote the probability density functions (pdf) also by p subscripted with the variable name.
Revised: With a slight abuse of notation, we will denote the probability density functions (pdf) also by p subscripted with the variable name.

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: where the penultimate step follows since by assumption, F ( k ) is nowhere 0 .
Revised: where the penultimate step follows since by assumption, F ( k ) is nowhere 0 .

Original: In the last step, F −is the inverse Fourier transform.
Revised: In the last step, F −is the inverse Fourier transform.

Original: Thus, there is a bijective map between X and Z .
Revised: Thus, there is a bijective map between X and Z .

Original: Since the Fourier and the inverse Fourier are continuous transformations, this map is also continuous.
Revised: Since the Fourier and the inverse Fourier are continuous transformations, this map is also continuous.

Original: So, we can write Z = h ( X ) , where h is a bijective, differentiable function.
Revised: So, we can write Z = h ( X ) , where h is a bijective, differentiable function.

Original: So, the pdfs of X and Z are related as
Revised: So, the pdfs of X and Z are related as

Original: [Equation]
Revised: [Equation]

Original: where J h (˜ x ) is the Jacobian of h evaluated at ˜ x .
Revised: where J h (˜ x ) is the Jacobian of h evaluated at ˜ x .

Original: Now, note that since Y is a sum of two random variables, its pdf is a convolution of the individual probability density functions.
Revised: Now, note that since Y is a sum of two random variables, its pdf is a convolution of the individual probability density functions.

Original: So we have:
Revised: So we have:

Original: [Equation]
Revised: [Equation]

Original: Taking the Fourier transform on both sides, we have
Revised: Taking the Fourier transform on both sides, we have

Original: [Equation]
Revised: [Equation]

Original: where the penultimate step follows since by assumption, F ( p θ ) is nowhere 0 .
Revised: where the penultimate step follows since by assumption, F ( p θ ) is nowhere 0 .

Original: Combining the two results, we have a reverse map from the measurement distribution p y to a sample distribution p x .
Revised: Combining the two results, we have a reverse map from the measurement distribution p y to a sample distribution p x .

Original: Thus, the reverse map uniquely determines the true underlying distribution p x , concluding the proof.
Revised: Thus, the reverse map uniquely determines the true underlying distribution p x , concluding the proof.

Original: 10. P ROOF OF T HEOREM 5. 
Revised: 10. P ROOF OF T HEOREM 5.

Original: We ﬁrst state a slightly different version of Theorem 1 from [Goodfellow et al.
Revised: We ﬁrst state a slightly different version of Theorem 1 from [Goodfellow et al.

Original: for the discrete setting.
Revised: for the discrete setting.

Original: We shall use [ n ] to denote the set { 1 , 2 , . . . n } , and use I ( · ) to denote the indicator function.
Revised: We shall use [ n ] to denote the set { 1 , 2 , . . . n } , and use I ( · ) to denote the indicator function.

Original: Lemma 10.1.
Revised: Lemma 10.1.

Original: Consider a dataset of measurement samples { y 1 , y 2 , . . .
Revised: Consider a dataset of measurement samples { y 1 , y 2 , . . .

Original: y s } , where each y i ∈
Revised: y s } , where each y i ∈

Original: [ t ] .
Revised: [ t ] .

Original: We deﬁne the empirical version of the vanilla GAN objective as
Revised: We deﬁne the empirical version of the vanilla GAN objective as

Original: [Equation]
Revised: [Equation]

Original: For j ∈ [ t ] , let ˆ p ry ( j )
Revised: For j ∈ [ t ] , let ˆ p ry ( j )

Original: =
Revised: =

Original: (cid:80) I ( y i =
Revised: (cid:80) I ( y i =

Original: j ) /s be the empirical distribution of samples.
Revised: j ) /s be the empirical distribution of samples.

Original: Then the optimal discriminator for the empirical objective is such that
Revised: Then the optimal discriminator for the empirical objective is such that

Original: [Equation]
Revised: [Equation]

Original: Additionally, if we ﬁx the discriminator to be optimal, then any optimal generator must satisfy p gy
Revised: Additionally, if we ﬁx the discriminator to be optimal, then any optimal generator must satisfy p gy

Original: = ˆ p ry .
Revised: = ˆ p ry .

Original: Proof.
Revised: Proof.

Original: The Empirical Risk Minimization (ERM) version of the loss is equivalent to the taking expectation of the data dependent term with respect to the empirical distribution.
Revised: The Empirical Risk Minimization (ERM) version of the loss is equivalent to the taking expectation of the data dependent term with respect to the empirical distribution.

Original: Replacing the real data distribution with the empirical version in the proof of Theorem 1 from [Goodfellow et al.
Revised: Replacing the real data distribution with the empirical version in the proof of Theorem 1 from [Goodfellow et al.

Original: (2014)], we obtain the result.
Revised: (2014)], we obtain the result.

Original: Now we give a proof of Theorem 5.4.
Revised: Now we give a proof of Theorem 5.4.

Original: Theorem.
Revised: Theorem.

Original: Assume that each image pixel takes values in a ﬁnite set P. Thus x ∈ P n ⊂ R n .
Revised: Assume that each image pixel takes values in a ﬁnite set P. Thus x ∈ P n ⊂ R n .

Original: Assume 0 ∈ P , and consider the Block-Pixels measurement model (Section 4) with p being the probability of blocking a pixel.
Revised: Assume 0 ∈ P , and consider the Block-Pixels measurement model (Section 4) with p being the probability of blocking a pixel.

Original: If p < 1 , then there is a unique distribution p rx that can induce the measurement distribution p ry .
Revised: If p < 1 , then there is a unique distribution p rx that can induce the measurement distribution p ry .

Original: Further, for any (cid:15) > 0 , δ ∈ (0 , 1] , given a dataset of
Revised: Further, for any (cid:15) > 0 , δ ∈ (0 , 1] , given a dataset of

Original: [Equation]
Revised: [Equation]

Original: IID measurement samples from p ry , if the discriminator D is optimal, then with probability ≥ 1 − δ over the dataset, any optimal generator G must satisfy d TV ( p gx , p rx ) ≤ (cid:15) , where d TV ( · , · ) is the total variation distance.
Revised: IID measurement samples from p ry , if the discriminator D is optimal, then with probability ≥ 1 − δ over the dataset, any optimal generator G must satisfy d TV ( p gx , p rx ) ≤ (cid:15) , where d TV ( · , · ) is the total variation distance.

Original: Proof.
Revised: Proof.

Original: We ﬁrst consider a more general case and apply that to the Block-Pixels model.
Revised: We ﬁrst consider a more general case and apply that to the Block-Pixels model.

Original: Consider a discrete distribution p x over [ t ] .
Revised: Consider a discrete distribution p x over [ t ] .

Original: We apply random measurement functions to samples from p x to obtain measurements.
Revised: We apply random measurement functions to samples from p x to obtain measurements.

Original: Assume that each measurement also belongs to the same set, i.e. [ t ] .
Revised: Assume that each measurement also belongs to the same set, i.e. [ t ] .

Original: Let A ∈ R t
Revised: Let A ∈ R t

Original: ×
Revised: ×

Original: t be the transition matrix so that A ij is the probability (under the randomness in measurement functions) that measurement i was produced by sample j .
Revised: t be the transition matrix so that A ij is the probability (under the randomness in measurement functions) that measurement i was produced by sample j .

Original: Then the distribution over measurements p y can be written in terms of p x and A as:
Revised: Then the distribution over measurements p y can be written in terms of p x and A as:

Original: [Equation]
Revised: [Equation]

Original: Thus, if the matrix A is invertible, we can guarantee that the distribution p x is recoverable from p y .
Revised: Thus, if the matrix A is invertible, we can guarantee that the distribution p x is recoverable from p y .

Original: Assuming A is invertible, we now turn to the sample complexity.
Revised: Assuming A is invertible, we now turn to the sample complexity.

Original: Let λ be the minimum of magnitude of eigenvalues of A .
Revised: Let λ be the minimum of magnitude of eigenvalues of A .

Original: Since A is invertible, λ > 0 .
Revised: Since A is invertible, λ > 0 .

Original: Let the dataset of measurements be { y 1 , y 2 , . . .
Revised: Let the dataset of measurements be { y 1 , y 2 , . . .

Original: y s } .
Revised: y s } .

Original: For j ∈ [ t ] and for k ∈ [ s ] , Let Y jk
Revised: For j ∈ [ t ] and for k ∈ [ s ] , Let Y jk

Original: =
Revised: =

Original: I ( y k = j ) .
Revised: I ( y k = j ) .

Original: Then for any (cid:15) > 0 , we have
Revised: Then for any (cid:15) > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: where we used union bound and Chernoff inequalities.
Revised: where we used union bound and Chernoff inequalities.

Original: Setting this to δ , we get
Revised: Setting this to δ , we get

Revised: [Equation]

Revised: =

Original: [Equation]
Revised: [Equation]

Original: From Lemma 10.1, we know that the optimal generator must satisfy p gy
Revised: From Lemma 10.1, we know that the optimal generator must satisfy p gy

Original: = ˆ p ry .
Revised: = ˆ p ry .

Original: By invertibility of A , we know that p gx =
Revised: By invertibility of A , we know that p gx =

Original: A − 1 p gy , and that p rx =
Revised: A − 1 p gy , and that p rx =

Original: A − 1 p ry .
Revised: A − 1 p ry .

Original: Thus, we obtain that with probability ≥ 1 − δ ,
Revised: Thus, we obtain that with probability ≥ 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: Now we turn to the speciﬁc case of Block-Pixels measurement.
Revised: Now we turn to the speciﬁc case of Block-Pixels measurement.

Original: We proceed by dividing the set of all possible | P | n images into n +1 classes.
Revised: We proceed by dividing the set of all possible | P | n images into n +1 classes.

Original: The i -th class has those images that have exactly i pixels with zero value.
Revised: The i -th class has those images that have exactly i pixels with zero value.

Original: We sort the images according to their class number (arbitrary ordering within the class) and consider the transition matrix A .
Revised: We sort the images according to their class number (arbitrary ordering within the class) and consider the transition matrix A .

Original: Note that given an image from class i it must have j ≥ i zero pixels after the measurement.
Revised: Note that given an image from class i it must have j ≥ i zero pixels after the measurement.

Original: Also, no image in class i can produce another image in the same class after measurements.
Revised: Also, no image in class i can produce another image in the same class after measurements.

Original: Thus, the transition matrix is lower triangular.
Revised: Thus, the transition matrix is lower triangular.

Original: Since each pixel is blocked independently with probability p and since there are n pixels, the event that no pixels are blocked occurs with probability (1 − p ) n .
Revised: Since each pixel is blocked independently with probability p and since there are n pixels, the event that no pixels are blocked occurs with probability (1 − p ) n .

Original: Thus, every image has at least (1 − p ) n chance of being unaffected by the measurements.
Revised: Thus, every image has at least (1 − p ) n chance of being unaffected by the measurements.

Original: Any unaffected image maps to itself and thus forms diagonal entries in the transition matrix.
Revised: Any unaffected image maps to itself and thus forms diagonal entries in the transition matrix.

Original: So, we observe that the diagonal entries of the transition matrix are strictly positive and their minimum value is (1 − p ) n .
Revised: So, we observe that the diagonal entries of the transition matrix are strictly positive and their minimum value is (1 − p ) n .

Original: For a triangular matrix, the diagonal entries are precisely the eigenvalues and hence we have proved that A is invertible and the smallest eigenvalue is (1 − p ) n .
Revised: For a triangular matrix, the diagonal entries are precisely the eigenvalues and hence we have proved that A is invertible and the smallest eigenvalue is (1 − p ) n .

Original: Combined with the result above, by setting λ
Revised: Combined with the result above, by setting λ

Original: = (1 − p ) n , and t = | P | n , we conclude the proof.
Revised: = (1 − p ) n , and t = | P | n , we conclude the proof.

Original: [Equation] 10. M ODEL A RCHITECTURES DETAILS
Revised: A PPENDIX B 10. M ODEL A RCHITECTURE DETAILS

Original: The DCGAN model on MNIST follows the architecture in [Radford et al.
Revised: The DCGAN model on MNIST follows the architecture in [Radford et al.

Original: The noise input to the generator ( Z ) has 100 dimensions where each coordinate is sampled IID Uniform on [ − 1 , 1] .
Revised: The noise input to the generator ( Z ) has 100 dimensions where each coordinate is sampled IID Uniform on [ − 1 , 1] .

Original: The generator uses two linear layers followed by two deconvolutional layers.
Revised: The generator uses two linear layers followed by two deconvolutional layers.

Original: The labels are concatenated with the inputs of each layer.
Revised: The labels are concatenated with the inputs of each layer.

Original: The discriminator uses two convolutional layers followed by two linear layers.
Revised: The discriminator uses two convolutional layers followed by two linear layers.

Original: As with the generator, the labels are concatenated with the inputs of each layer.
Revised: As with the generator, the labels are concatenated with the inputs of each layer.

Original: Batch-norm is used in both generator and the discriminator.
Revised: Batch-norm is used in both generator and the discriminator.

Original: The WGANGP model on MNIST follows the architecture in [Gulrajani et al.
Revised: The WGANGP model on MNIST follows the architecture in [Gulrajani et al.

Original: The generator takes in a latent vector of 128 dimensions where each coordinate is sampled IID Uniform on [ − 1 , 1] .
Revised: The generator takes in a latent vector of 128 dimensions where each coordinate is sampled IID Uniform on [ − 1 , 1] .

Original: The generator then applies one linear and three deconvolutional layers.
Revised: The generator then applies one linear and three deconvolutional layers.

Original: The discriminator uses three convolutional layers followed by one linear layer.
Revised: The discriminator uses three convolutional layers followed by one linear layer.

Original: Batch-norm is not used.
Revised: Batch-norm is not used.

Original: The unconditional DCGAN model on celebA follows the architecture in [Radford et al.
Revised: The unconditional DCGAN model on celebA follows the architecture in [Radford et al.

Original: The latent vector has 100 dimensions where each coordinate is Uniform on [ − 1 , 1] .
Revised: The latent vector has 100 dimensions where each coordinate is Uniform on [ − 1 , 1] .

Original: The generator applies one linear layer followed by four deconvolutional layers.
Revised: The generator applies one linear layer followed by four deconvolutional layers.

Original: The discriminator uses four convolutional layers followed by a linear layer.
Revised: The discriminator uses four convolutional layers followed by a linear layer.

Original: Batch-norm is used in both generator and the discriminator.
Revised: Batch-norm is used in both generator and the discriminator.

Original: The ACWGANGP model on CIFAR-10 follows the residual architecture in [Gulrajani et al.
Revised: The ACWGANGP model on CIFAR-10 follows the residual architecture in [Gulrajani et al.

Original: The latent vector has 128 dimensions where each coordinate is sampled from IID standard Gaussian distribution.
Revised: The latent vector has 128 dimensions where each coordinate is sampled from IID standard Gaussian distribution.

Original: The generator has a linear layer followed by three residual blocks.
Revised: The generator has a linear layer followed by three residual blocks.

Original: Each residual block consists of two repetitions of the following three operations: conditional batch normalization followed by a nonlinearity followed by an upconvolution layer.
Revised: Each residual block consists of two repetitions of the following three operations: conditional batch normalization followed by a nonlinearity followed by an upconvolution layer.

Original: The residual blocks are followed by another conditional batch normalization, a ﬁnal convolution, and a ﬁnal tanh non-linearity.
Revised: The residual blocks are followed by another conditional batch normalization, a ﬁnal convolution, and a ﬁnal tanh non-linearity.

Original: The discriminator consists of one residual block with two convolutional layers followed by three residual blocks, and a ﬁnal linear layer.
Revised: The discriminator consists of one residual block with two convolutional layers followed by three residual blocks, and a ﬁnal linear layer.

Original: A PPENDIX C
Revised: A PPENDIX C

Original: Here, we present some more results for various measurement models.
Revised: Here, we present some more results for various measurement models.

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original: I
Revised: I

Original: [Figure]
Revised: [Figure]

Original: p t i o n s c o r
Revised: p t i o n s c o r

Original: [Figure]
Revised: [Figure]

Original: 10. R OBUSTNESS TO MEASUREMENT  MISMATCH
Revised: 10. R OBUSTNESS TO MEASUREMENT MODEL MISMATCH

Original: So far, in our analysis and experiments, we assumed that the parametric form of the measurement function and the distribution of those parameters is exactly known.
Revised: So far, in our analysis and experiments, we assumed that the parametric form of the measurement function and the distribution of those parameters is exactly known.

Original: This was then used for simulating the stochastic measurement process in our setup.
Revised: This was then used for simulating the stochastic measurement process.

Original: Here, we consider the case where the parameter distribution is only approximately known.
Revised: Here, we consider the case where the parameter distribution is only approximately known.

Original: In such cases, one would like the  quality of the learned generator to be close to the case where the parameter distribution is exactly known.
Revised: In this case, one would like the training process to be robust, i.e. the quality of the learned generator to be close to the case where the parameter distribution is exactly known.

Original: We empirically demonstrate the robustness of the AmbientGAN approach  to systematic mismatches in the parameter distribution through the following experiment.
Revised: Through the following experiment, we empirically demonstrate that the AmbientGAN approach is robust to systematic mismatches in the parameter distribution of the measurement function.

Original: Consider the Block-Pixels measurement model (Section 4).
Revised: Consider the Block-Pixels measurement model (Section 4).

Original: We use the MNIST dataset.
Revised: We use the MNIST dataset.

Original: Pixels are blocked with probability p ∗ = 0 .
Revised: Pixels are blocked with probability p ∗ = 0 .

Original: 5 to obtain a dataset of measurements.
Revised: 5 to obtain a dataset of measurements.

Original: For several values of blocking probability p for the measurement function applied to the output of the generator, we train AmbientGAN models with this dataset.
Revised: For several values of blocking probability p for the measurement function applied to the output of the generator, we train AmbientGAN models with this dataset.

Original: After training, we compute the inception score of the learned generators and plot it as a function of p in Fig.
Revised: After training, we compute the inception score of the learned generators and plot it as a function of p in Fig.

Original: 18 (left).
Revised: 16 (left).

Original: We note that the plot peaks at p = p ∗ = 0 .
Revised: We note that the plot peaks at p = p ∗ = 0 .

Original: 5 and gradually drops on both sides.
Revised: 5 and gradually drops on both sides.

Original: This suggests that our method is somewhat robust to parameter distribution mismatch.
Revised: This suggests that our method is somewhat robust to parameter distribution mismatch.

Original: 10. C OMPRESSED SENSING USING A MBIENT GAN
Revised: 10. C OMPRESSED SENSING USING A MBIENT GAN

Original: We provide further evidence that the generator learned through AmbientGAN approach captures the data distribution well.
Revised: We provide further evidence that the generator learned through AmbientGAN approach captures the data distribution well.

Original: Generative models have been shown to improve sensing over sparsitybased approaches [Bora et al.
Revised: Generative models have been shown to improve sensing over sparsitybased approaches [Bora et al.

Original: We attempt to use the GAN learned using our procedure for compressed sensing.
Revised: We attempt to use the GAN learned using our procedure for compressed sensing.

Original: We trained an AmbientGAN with Block-Pixels measurement model (Section 4) on MNIST with p = 0 . 5 .
Revised: We trained an AmbientGAN with Block-Pixels measurement model (Section 4) on MNIST with p = 0 . 5 .

Original: Using the learned generator, we followed the rest of the procedure in [Bora et al.
Revised: Using the learned generator, we followed the rest of the procedure in [Bora et al.

Original: using their code 3 .
Revised: using their code 3 .

Original: Fig.
Revised: Fig.

Original: 18 (right) shows a plot of reconstruction error vs the number of measurements, comparing Lasso with AmbienGAN.
Revised: 16 (right) shows a plot of reconstruction error vs the number of measurements, comparing Lasso with AmbienGAN.

Original: Thus, we observe a similar reduction in the number of measurements while using AmbientGAN  instead of  regular GAN.
Revised: Thus, we observe a similar reduction in the number of measurements while using AmbientGAN trained with corrupted samples instead of a regular GAN trained with fully observed samples.