Original: Generator Surgery for Compressed Sensing
Revised: Generator Surgery for Compressed Sensing

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Recent work has explored the use of generator networks with low latent dimensionas signal priors for image recovery in compressed sensing.
Revised: Recent work has explored the use of generator networks with low latent dimension as signal priors for image recovery in compressed sensing.

Original: However, the recoveryperformance of such models is limited by high representation error.
Revised: However, the recovery performance of such models is limited by high representation error.

Original: We introduce amethod to reduce the representation error of such generator signal priors by cuttingone or more initial blocks at test time and optimizing over the resulting higherdimensional latent space.
Revised: We introduce a method to reduce the representation error of such generator signal priors by cutting one or more initial blocks at test time and optimizing over the resulting higherdimensional latent space.

Original: Experiments demonstrate signiﬁcantly improved recoveryfor a variety of architectures.
Revised: Experiments demonstrate signiﬁcantly improved recovery for a variety of architectures.

Original: This approach also works well for out-of-trainingdistribution images and is competitive with other state-of-the-art methods.
Revised: This approach also works well for out-of-trainingdistribution images and is competitive with other state-of-the-art methods.

Original: Ourexperiments show that test-time architectural modiﬁcations can greatly improvethe recovery quality of generator signal priors for compressed sensing.
Revised: Our experiments show that test-time architectural modiﬁcations can greatly improve the recovery quality of generator signal priors for compressed sensing.

Original:  Introduction
Revised:  Introduction

Original: In inverse imaging problems, we recover an image signal from undersampled measurements.
Revised: In inverse imaging problems, we recover an image signal from undersampled measurements.

Original: Asthe problem is underdetermined, additional structural assumptions are required.
Revised: As the problem is underdetermined, additional structural assumptions are required.

Original: Classical methodsleverage sparsity or compressibilty in a known basis [1, 2].
Revised: Classical methods leverage sparsity or compressibilty in a known basis [1, 2].

Original: Recent developments in deep generativemodeling, such as generative adversarial networks (GANs [3]) and variational autoencoders (VAEs,[4]) have led to their use as a signal prior for these problems.
Revised: Recent developments in deep generative modeling, such as generative adversarial networks (GANs [3]) and variational autoencoders (VAEs, [4]) have led to their use as a signal prior for these problems.

Original: While Bora et al.
Revised: While Bora et al.

Original: [5] showed thatgenerator signal priors outperform sparsity priors at low undersampling ratios, recovery quality islimited by low latent dimension and high representation error (formal deﬁnitions below).
Revised: [5] showed that generator signal priors outperform sparsity priors at low undersampling ratios, recovery quality is limited by low latent dimension and high representation error (formal deﬁnitions below).

Original: We focuson improving these generator-based signal priors.
Revised: We focus on improving these generator-based signal priors.

Original: To reduce representation error, it is natural to seek models with high latent dimension, but existingoptimizers cannot reliably train such models.
Revised: To reduce representation error, it is natural to seek models with high latent dimension, but existing optimizers cannot reliably train such models.

Original: This has motivated a variety of approaches for obtaininggenerator signal priors with more degrees of freedom [6–11], though these methods are oftencomputationally expensive or complicated.
Revised: This has motivated a variety of approaches for obtaining generator signal priors with more degrees of freedom [6–11], though these methods are often computationally expensive or complicated.

Original: For example, Gu et al.
Revised: For example, Gu et al.

Original: [6] optimize multiple latent codes,composing them at an intermediate layer with optimizable weights.
Revised: [6] optimize multiple latent codes, composing them at an intermediate layer with optimizable weights.

Original: Abu Hussein et al.
Revised: Abu Hussein et al.

Original: [7] optimizeboth the latent code and the generator’s weights.
Revised: [7] optimize both the latent code and the generator’s weights.

Original: Untrained methods such as Deep Image Prior[11] and Deep Decoder [10] recover an image by optimizing the weights of a randomly-initializednetwork.
Revised: Untrained methods such as Deep Image Prior [11] and Deep Decoder [10] recover an image by optimizing the weights of a randomly-initialized network.

Original: We address the problem of high representation error in generator signal priors and substantiallyimprove their performance in compressed sensing with a simple and inexpensive technique we call
Revised: We address the problem of high representation error in generator signal priors and substantially improve their performance in compressed sensing with a simple and inexpensive technique we call

Original: Generator Surgery . Using a pre-trained generator with low latent dimension, we ‘cut’ one or moreearly blocks from the network at test time, and treat the intermediate activations as a new latent imagerepresentation. By cutting early blocks at test time, we trade the generative sampling capability of themodel for increased recovery quality. Our method is applicable for a wide variety of network architectures, and is competitive with otherrecent baselines.
Revised: Optimization parameters:

Original: We also show that our method can even recover high-quality images that are outof-training-distribution; this is important as real-world applications like MRI imaging often includetarget images with novel features. Our results suggest more broadly that test-time architectural [Figure]
Revised: Cut Index:

Original: Input : Pre-trained G 0 , c , A , y , T , α
Revised: Input : Pre-trained G 0 , c , A , y , T , α

Original: G c ← cut ( G 0 , c ) ; // cut c blocks
Revised: G c ← cut ( G 0 , c ) ; // cut c blocks

Original: Initialize z (0) ; for t = 0 to T − 

Revised: Initialize z (0) ; for t = 0 to T −

Original: do x ( t ) ← G c ( z ( t ) ) ;
Revised: do x ( t ) ← G c ( z ( t ) ) ;

Original: L ← (cid:107) y − Ax ( t ) (cid:107) ; z ( t +1) ← z ( t ) − α ∇ z ( t ) L ; end
Revised: L ← (cid:107) y − Ax ( t ) (cid:107) ; z ( t +1) ← z ( t ) − α ∇ z ( t ) L ; end

Original: Output: ˆ x = x ( T ) modiﬁcations can allow a model trained on one task (image generation) to perform well on a differenttask (image recovery).
Revised: Output: ˆ x = x ( T ) Generator Surgery . Using a pre-trained generator with low latent dimension, we ‘cut’ one or more early blocks from the network at test time, and treat the intermediate activations as a new latent image representation.

Original: Method
Revised: By cutting early blocks at test time, we trade the generative sampling capability of the model for increased recovery quality. Our method is applicable for a wide variety of network architectures, and is competitive with other recent baselines. We also show that our method can even recover high-quality images that are outof-training-distribution; this is important as real-world applications like MRI imaging often include target images with novel features.

Original: [Figure]
Revised: Our results suggest more broadly that test-time architectural modiﬁcations can allow a model trained on one task (image generation) to perform well on a different task (image recovery).

Original:   2.1 Generator Surgery (GS)
Revised:  Method 2.1 Generator Surgery (GS)

Original: We deﬁne a generator network G 0 : R k 0 (cid:55)→ R n that maps a low-dimensional latent code z 0 ∈ R k 0 toan image x = G 0 ( z 0 ) .
Revised: We deﬁne a generator network G 0 : R k 0 (cid:55)→ R n that maps a low-dimensional latent code z 0 ∈ R k 0 to an image x = G 0 ( z 0 ) .

Original: The generator is composed of d blocks:
Revised: The generator is composed of d blocks:

Original: [Equation]
Revised: [Equation]

Original:  where we deﬁne z i ∈ R k
Revised: (1) where we deﬁne z i ∈ R k

Original: i as the input to block B i .
Revised: i as the input to block B i .

Original: Each block B i can consist of arbitrary neuralnetwork operations (such as convolution, upsampling, and activation functions).
Revised: Each block B i can consist of arbitrary neural network operations (such as convolution, upsampling, and activation functions).

Original: We assume k i ≥k 0 , ∀ i = 1 , . . .
Revised: We assume k i ≥ k 0 , ∀ i = 1 , . . .

Original: d − 1 , which holds for all architectures considered in this paper.
Revised: d − 1 , which holds for all architectures considered in this paper.

Original: We refer to Eqn.
Revised: We refer to Eqn.

Original: (1)as an ‘uncut’ generator or ‘No Generator Surgery (no GS)’.
Revised: (1) as an ‘uncut’ generator or ‘No Generator Surgery (no GS)’.

Original: After training the model in Eqn. 1 or using pre-trained weights, we cut away the ﬁrst c ≥ 1 blocks,resulting in a ‘cut generator’ G c : R k c (cid:55)→ R n = B
Revised: After training the model in Eqn. 1 or using pre-trained weights, we cut away the ﬁrst c ≥ 1 blocks, resulting in a ‘cut generator’ G c : R k c (cid:55)→ R n = B

Original: d − 1 ◦ . . .
Revised: d − 1 ◦ . . .

Original: ◦ B c which accepts a latent code z c ∈ R k cas input.
Revised: ◦ B c which accepts a latent code z c ∈ R k c as input.

Original: We refer to this model as ‘Generator Surgery (GS)’, and we apply it on inverse problems.
Revised: We refer to this model as ‘Generator Surgery (GS)’, and we apply it on inverse problems.

Original: It is important to note that the cut generator is not a generative model.
Revised: It is important to note that the cut generator is not a generative model.

Original: In effect, we are intentionallytrading our ability to approximately sample from the learned distribution over images for increasedexpressivity of the latent representation in inversion tasks.
Revised: In effect, we are intentionally trading our ability to approximately sample from the learned distribution over images for increased expressivity of the latent representation in inversion tasks.

Original: Furthermore, since we remove blocksrelative to the original generator, the cut generator requires fewer backpropagation steps and thus lesscomputation at inversion time.
Revised: Furthermore, since we remove blocks relative to the original generator, the cut generator requires fewer backpropagation steps and thus less computation at inversion time.

Original: Additional inputs Our method is applicable to many different architectures, including models withskip connections or class-conditional models.
Revised: Additional inputs Our method is applicable to many different architectures, including models with skip connections or class-conditional models.

Original: Speciﬁcally, if we cut the ﬁrst c blocks of a modeland the new initial layer B c requires multiple inputs, all of these inputs are treated as optimizableparameters at inversion time.
Revised: Speciﬁcally, if we cut the ﬁrst c blocks of a model and the new initial layer B c requires multiple inputs, all of these inputs are treated as optimizable parameters at inversion time.

Original: We use several architectures in experiments, one of which includes skipconnections (BEGAN [12]).
Revised: We use several architectures in experiments, one of which includes skip connections (BEGAN [12]).

Original: 2.2 Compressed Sensing
Revised: 2.2 Compressed Sensing

Original: We consider the problem of noisy compressed sensing (CS).
Revised: We consider the problem of noisy compressed sensing (CS).

Original: All norms (cid:107) · (cid:107) are L 2 unless otherwisestated.
Revised: All norms (cid:107) · (cid:107) are L 2 unless otherwise stated.

Original: For an unknown target image x ∈ R n , we are given a measurement matrix A ∈ R m × n , m <<n and measurements y =
Revised: For an unknown target image x ∈ R n , we are given a measurement matrix A ∈ R m × n , m (cid:28) n and measurements y =

Original: Ax + η , where η is noise.
Revised: Ax + η , where η is noise.

Original: Our goal is to ﬁnd ˆ x ∈ R n that minimizes(cid:107) x − ˆ x (cid:107) .
Revised: Our goal is to ﬁnd ˆ x ∈ R n that minimizes (cid:107) x − ˆ x (cid:107) .

Original: Note that the problem is underdetermined, since m < n .
Revised: Note that the problem is underdetermined, since m < n .

Original: Using a generator G : R k (cid:55)→ R n ,we can estimate ˆ x by ﬁnding a latent code z that optimizes the loss function:
Revised: Using a generator G : R k (cid:55)→ R n , we can estimate ˆ x by ﬁnding a latent code z that optimizes the loss function:

Original: [Equation]
Revised: [Equation]

Original: We study the case of a Gaussian measurement matrix A ij ∼ N (0 , 1 /m ) and i.i.d.
Revised: We study the case of a Gaussian measurement matrix A ij ∼ N (0 , 1 /m ) and i.i.d.

Original: Gaussian noiseη ∼ N (0 , σ 2 ) .
Revised: Gaussian noise η ∼ N (0 , σ 2 ) .

Original: We follow Bora et al.
Revised: We follow Bora et al.

Original: [5] and set σ = 0 .
Revised: [5] and set σ = 0 .

Original: 1 for 64 px images, and scale the noiselevel for larger images such that E (cid:2) (cid:107) η (cid:107) 2 / (cid:107) Ax (cid:107) 2 (cid:3) is kept constant.
Revised: 1 for 64 px images, and scale the noise level for larger images such that E (cid:2) (cid:107) η (cid:107) 2 / (cid:107) Ax (cid:107) 2 (cid:3) is kept constant.

Original: Applying ‘Generator Surgery’ tocompressed sensing and using gradient descent to optimize 
Revised: Applying ‘Generator Surgery’ to compressed sensing and using gradient descent to optimize Eqn.

Original: [Figure]
Revised: (2) results in Alg.

Original: 2.3 Theoretical Motivation
Revised: 2.3 Theoretical Motivation

Original: Deﬁne the representation error of G 0 w.r.t.
Revised: Deﬁne the representation error of G 0 w.r.t.

Original: an image x as Err rep ( G 0 , x ) =
Revised: an image x as Err rep ( G 0 , x ) =

Original: min z ∈ R k 0 (cid:107) x − G 0 ( z ) (cid:107) ,and analogously for G c .
Revised: min z ∈ R k 0 (cid:107) x − G 0 ( z ) (cid:107) , and analogously for G c .

Original: Bora et al.
Revised: Bora et al.

Original: [5] show in their Theorem 1.2 that for an L -Lipschitz network,overall error in image recovery using a neural network signal prior is bounded w.h.p.
Revised: [5] show in their Theorem 1.2 that for an L -Lipschitz network, overall error in image recovery using a neural network signal prior is bounded w.h.p.

Original: by four terms:representation error, the norm of the measurement noise η , optimization error (cid:15) , and an additive errorδ based on the sensing matrix.
Revised: by four terms: representation error, the norm of the measurement noise η , optimization error (cid:15) , and an additive error δ based on the sensing matrix.

Original: Consider the performance of  uncut generator G 0 and cut generator G c applied to the same setof measurements y .
Revised: Consider the performance of the uncut generator G 0 and cut generator G c applied to the same set of measurements y .

Original: Since we use the same measurements y , both the sensing error δ and themeasurement noise η are equal for the two generators.
Revised: Since we use the same measurements y , both the sensing error δ and the measurement noise η are equal for the two generators.

Original: We can assume that both models are optimizedto below a ﬁxed error level (cid:15) (we empirically ﬁnd that optimization error is indeed low).
Revised: We can assume that both models are optimized to below a ﬁxed error level (cid:15) (we empirically ﬁnd that optimization error is indeed low).

Original: Thus, inorder to compare overall recovery quality between the two models, we only need to compare theirrepresentation error:
Revised: Thus, in order to compare overall recovery quality between the two models, we only need to compare their representation error:

Original: [Equation]
Revised: [Equation]

Original: The inequality in Eqn.
Revised: The inequality in Eqn.

Original: (3) arises by comparing the minimization of z ∈ R k c to the constrainedminimization of ˜
Revised: (3) arises by comparing the minimization of z ∈ R k c to the constrained minimization of ˜

Original: z ∈ R k 0 for k c ≥ k 0 .
Revised: z ∈ R k 0 for k c ≥ k 0 .

Original: Intuitively, we can see that cutting blocks increases the rangeof the model and thereby reduces representation error.
Revised: Intuitively, we can see that cutting blocks increases the range of the model and thereby reduces representation error.

Original:  Experiments
Revised:  Experiments

Original: Experimental Design.
Revised: Experimental Design.

Original: We apply Generator Surgery (GS) to DCGAN [13], BEGAN [12], and
Revised: We apply Generator Surgery (GS) to DCGAN [13], BEGAN [12], and

Original: VAE [4] architectures.
Revised: VAE [4] architectures.

Original: We measure recovery performance using average peak signal-to-noise ratio(PSNR).
Revised: We measure recovery performance using average peak signal-to-noise ratio (PSNR).

Original: All models are trained on CelebA [14].
Revised: All models are trained on CelebA [14].

Original: We measure average peak-signal-to-noise ratio(PSNR) on recovery of test images from CelebA and COCO 2017 [15].
Revised: We measure average PSNR on recovery of test images from CelebA and COCO 2017 [15].

Original: For each architecture, wechoose the cut index c that maximizes average PSNR over 100 validation images on CelebA. For allexperiments, we use 100 images per dataset, and run Alg. 1 with 3 random restarts.
Revised: For each architecture, we choose the cut index c that maximizes average PSNR over 100 validation images on CelebA. For all experiments, we useimages per dataset, and run Alg. 1 with 3 random restarts.

Original: We ﬁrst compare against the standard sparsity-based Lasso-DCT to demonstrate utility [16].
Revised: We ﬁrst compare against the standard sparsity-based Lasso-DCT to demonstrate utility [16].

Original: LassoDCT solves the Lasso optimization ˆ z =
Revised: LassoDCT solves the Lasso optimization ˆ z =

Original: min z (cid:107)
Revised: min z (cid:107)

Original: y − A Φ z (cid:107) 22 + 0 .
Revised: y − A Φ z (cid:107) 22 + 0 .

Original: 01 (cid:107) z (cid:107) 1 .
Revised: 01 (cid:107) z (cid:107) 1 .

Original: Next, we compare againstseveral recent learned and unlearned methods: IAGAN [7], Deep Decoder (DD) [10], and mGANprior[6].
Revised: Next, we compare against several recent learned and unlearned methods: IAGAN [7], Deep Decoder (DD) [10], and mGANprior [6].

Original: To compare GS against baselines, we use the same generator architecture for Generator Surgery ,
Revised: To compare GS against baselines, we use the same generator architecture for Generator Surgery ,

Original: IAGAN, and mGANprior.
Revised: IAGAN, and mGANprior.

Original: For Deep Decoder (DD), we use a similar number of parameters as
Revised: For Deep Decoder (DD), we use a similar number of parameters as

Original: Generator Surgery for a fair comparison.
Revised: Generator Surgery for a fair comparison.

Original: We follow hyperparameters given in the literature, exceptfor mGANprior as the given settings were not applicable.
Revised: We follow hyperparameters given in the literature, except for mGANprior as the given settings were not applicable.

Original: Generator Surgery for CS.
Revised: Generator Surgery for CS.

Original: Fig.
Revised: Fig.

Original: 2 shows recovery on images from CelebA test set as well as
Revised: 2 shows recovery on images from CelebA test set as well as

Original: COCO.
Revised: COCO.

Original: For CelebA, the “No GS" model matches the results of Bora et al.
Revised: For CelebA, the “No GS" model matches the results of Bora et al.

Original: [5], outperforming
Revised: [5], outperforming

Original: Lasso-DCT at low measurement regimes for all models.
Revised: Lasso-DCT at low measurement regimes for all models.

Original: The “GS" model improves recovery performance substantially for all architectures, and outperforms the Lasso-DCT baseline over almostall undersampling ratios.
Revised: The “GS" model improves recovery performance substantially for all architectures, and outperforms the Lasso-DCT baseline over almost all undersampling ratios.

Original: For COCO images, cutting blocks still gives a large jump in recovery quality.
Revised: For COCO images, cutting blocks still gives a large jump in recovery quality.

Original: The performanceincrease over “No GS” is similar to CelebA images; this may indicate that much of the bias towardsthe training domain is contained in the early blocks of the generator.
Revised: The performance increase over “No GS” is similar to CelebA images; this may indicate that much of the bias towards the training domain is contained in the early blocks of the generator.

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Baseline Comparisons.
Revised: Baseline Comparisons.

Original: Fig.
Revised: Fig.

Original: 3 shows thecomparison to baselines for BEGAN.
Revised: 3 shows the comparison to baselines for BEGAN.

Original: We emphasize that our focus is not to produce thehighest performance but to be competitive withother methods while being simpler.
Revised: We emphasize that our focus is not to produce the highest performance but to be competitive with other methods while being simpler.

Original: GS generally beats mGANprior and performs similarly to
Revised: GS generally beats mGANprior and performs similarly to

Original: DD (GS is slightly worse at lower measurementsand slightly better at higher measurements).
Revised: DD (GS is slightly worse at lower measurements and slightly better at higher measurements).

Original: IAGAN outperforms all methods by a fair margin; this is expected as IAGAN is highly overparametrized.
Revised: IAGAN outperforms all methods by a fair margin; this is expected as IAGAN is highly overparametrized.

Original: In general, performance increaseswith increasing overparametrization ratio (see
Revised: In general, performance increases with increasing overparametrization ratio (see

Original: Table 1).
Revised: Table 1).

Original: Qualitative comparisons are providedin the Appendix.
Revised: Qualitative comparisons are provided in the Appendix.

Original: 3.1 Understanding GS Effect
Revised: 3.1 Understanding GS Effect

Original: Optimization and Representation Error.
Revised: Optimization and Representation Error.

Original: We explore the relative magnitude of sources of errorin our models.
Revised: We explore the relative magnitude of sources of error in our models.

Original: We set A
Revised: We set A

Original: = I and η = 0 to eliminate measurement error and sensing error, leavingonly representation error and optimization error.
Revised: = I and η = 0 to eliminate measurement error and sensing error, leaving only representation error and optimization error.

Original: We then evaluate recovery performance for cutand uncut models on images generated from the model (for which representation error is zero bydeﬁnition), as well as CelebA train, test, and COCO images (Fig. 4, left).
Revised: We then evaluate recovery performance for cut and uncut models on images generated from the model (for which representation error is zero by deﬁnition), as well as CelebA train, test, and COCO images (Fig. 4, left).

Original: The uncut generator G 0achieves a very high PSNR on generated images, conﬁrming that it has low optimization error, butperforms poorly on all real images, indicating it suffers from high representation error.
Revised: The uncut generator Gachieves a very high PSNR on generated images, conﬁrming that it has low optimization error, but performs poorly on all real images, indicating it suffers from high representation error.

Original: By contrast,the cut generator G c achieves high quality recovery on generated images, indicating low optimizationerror, and also performs well on real images, indicating reduced representation error.
Revised: By contrast, the cut generator G c achieves high quality recovery on generated images, indicating low optimization error, and also performs well on real images, indicating reduced representation error.

Original: [Figure]
Revised: [Figure]

Original: Do Learned Weights Matter?
Revised: Do Learned Weights Matter?

Original: We examine whether the performance increase from GS is solelydue to increased degrees of freedom, or whether this performance beneﬁts from the learned weights.
Revised: We examine whether the performance increase from GS is solely due to increased degrees of freedom, or whether this performance beneﬁts from the learned weights.

Original: Fig.
Revised: Fig.

Original: 4 (right) shows recovery performance using a trained and untrained BEGAN generator in Alg.1.
Revised: 4 (right) shows recovery performance using a trained and untrained BEGAN generator in Alg.1.

Original: The pre-trained model weights produce signiﬁcantly higher recovery quality in both image domains.
Revised: The pre-trained model weights produce signiﬁcantly higher recovery quality in both image domains.

Original: Other considerations All results for this subsection are provided in the Appendix.
Revised: Other considerations All results for this subsection are provided in the Appendix.

Original: We conﬁrmthat directly training a cut generator fails for a variety of hyperparameter settings, indicating thatcutting blocks must be done at test time.
Revised: We conﬁrm that directly training a cut generator fails for a variety of hyperparameter settings, indicating that cutting blocks must be done at test time.

Original: We also show that there is a tradeoff in choosing how manyblocks to cut; while cutting initial blocks increases the generator’s expressivity and reduces biastowards the training domain, cutting too many blocks results in poor image quality.
Revised: We also show that there is a tradeoff in choosing how many blocks to cut; while cutting initial blocks increases the generator’s expressivity and reduces bias towards the training domain, cutting too many blocks results in poor image quality.

Original: We also considerdifferent initialization strategies and how they impact performance.
Revised: We also consider different initialization strategies and how they impact performance.

Original: Finally, we show samples fromthe cut generator, demonstrating it no longer behaves 
Revised: Finally, we show samples from the cut generator, demonstrating it no longer behaves as a generative model.

Original:  Appendix 
Revised: A Appendix A.1 Baseline Comparisons

Original: A.1 Baseline Comparisons

Original: O r i g i n a l m G A N p r
Revised: O r i g i n a l m G A N p r

Original: i o r
Revised: i o r

Original: DD
Revised: DD

Original: N o G S
Revised: N o G S

Original: G S
Revised: G S

Original: O r i g i n a l m G A N p r
Revised: O r i g i n a l m G A N p r

Original: i o r
Revised: i o r

Original: DD
Revised: DD

Original: N o G S
Revised: N o G S

Original: G S
Revised: G S

Original: Fig.
Revised: Fig.

Original: 5 show recovered images from CelebA and COCO for compressed sensing at m/n ≈ 0 .
Revised: 5 show recovered images from CelebA and COCO for compressed sensing at m/n ≈ 0 .

Original: Weuse BEGAN for GS.
Revised: We use BEGAN for GS.

Original: First, we can clearly see that “no GS" recovers a clearly different face than theoriginal for CelebA and hallucinates a non-existent face for COCO (this is expected as the modelsare trained on CelebA).
Revised: First, we can clearly see that “no GS" recovers a clearly different face than the original for CelebA and hallucinates a non-existent face for COCO (this is expected as the models are trained on CelebA).

Original: This qualitatively demonstrates the large representation error of the uncutgenerator.
Revised: This qualitatively demonstrates the large representation error of the uncut generator.

Original: mGANprior produces slightly lower quality results than GS; it is grainier and sometimescreates different colors than the original.
Revised: mGANprior produces slightly lower quality results than GS; it is grainier and sometimes creates different colors than the original.

Original: DD and GS show subtle qualitative differences.
Revised: DD and GS show subtle qualitative differences.

Original: DD oftenproduces smoother images, sometimes losing ﬁne textural details such as eyes, hair strands, and clocknumerals; GS produces grainier images but recovers such ﬁne details.
Revised: DD often produces smoother images, sometimes losing ﬁne textural details such as eyes, hair strands, and clock numerals; GS produces grainier images but recovers such ﬁne details.

Original: A.2 Choice of Initialization
Revised: A.2 Choice of Initialization

Original: [Figure]
Revised: [Figure]

Original: We consider different initialization strategies in Fig.
Revised: We consider different initialization strategies in Fig.

Original: “Zero” is the all-zeros vector and “ CN (0 , 1) ”denotes a standard normal distribution censored to the range [ − 1 , 1] .
Revised: “Zero” is the all-zeros vector and “ CN (0 , 1) ” denotes a standard normal distribution censored to the range [ − 1 , 1] .

Original: “LassoInv” denotes using thesolution to the Lasso-DCT optimization (the latent vector z ) as initialization.
Revised: “LassoInv” denotes using the solution to the Lasso-DCT optimization (the latent vector z ) as initialization.

Original: N (0 , 5) , N (0 , 10) are normal distributions with increasing variance.
Revised: N ( 5) , N ( 10) are normal distributions with increasing variance.

Original: All initializations were run with the best of 3 randomrestarts.
Revised: All initializations were run with the best of 3 random restarts.

Original: We ﬁnd that performance varies widely for different initialization strategies, supporting ourclaim of the source of regularization.
Revised: We ﬁnd that performance varies widely for different initialization strategies, supporting our claim of the source of regularization.

Original: We ﬁnd that CN (0 , 1) and “Zero” perform best, though “Zero”has slightly higher variance.
Revised: We ﬁnd that CN (0 , 1) and “Zero” perform best, though “Zero” has slightly higher variance.

Original: We use CN (0 , 1) for all experiments.
Revised: We use CN (0 , 1) for all experiments.

Original: A.3 Choosing Number of Blocks to Cut
Revised: A.3 Choosing Number of Blocks to Cut

Original: [Figure]
Revised: [Figure]

Original: We ﬁnd the best number of cuts c for each model by evaluating compressed sensing performance onvalidation images from CelebA and COCO.
Revised: We ﬁnd the best number of cuts c for each model by evaluating compressed sensing performance on validation images from CelebA and COCO.

Original: We use 100 images and compute the average PSNR foreach dataset.
Revised: We use 100 images and compute the average PSNR for each dataset.

Original: Fig.
Revised: Fig.

Original: 7 shows the average PSNR for these datasets when applying Generator Surgery to
Revised: 7 shows the average PSNR for these datasets when applying Generator Surgery to

Original: DCGAN, BEGAN, and VAE models.
Revised: DCGAN, BEGAN, and VAE models.

Original: For all other experiments, we select the c which achieves thehighest PSNR on CelebA here, which is c = 1 for DCGAN, c = 2 for BEGAN, and c = 1 for VAE.
Revised: For all other experiments, we select the c which achieves the highest PSNR on CelebA here, which is c = 1 for DCGAN, c = 2 for BEGAN, and c = 1 for VAE.

Original: A.4 Generator samples with and without Generator Surgery
Revised: A.4 Generator samples with and without Generator Surgery

Original: [Figure]
Revised: [Figure]

Original: , BEGAN, and VAE with varying cut index.

Original: We give samples from our pre-trained generators, before Generator Surgery (“cuts = 0 "), and aftercutting 1 , 2 , or 3 blocks from each model.
Revised: We give samples from our pre-trained generators, before Generator Surgery (“cuts = 0 "), and after cutting 1 , 2 , or 3 blocks from each model.

Original: The high quality samples in the ﬁrst rows show that our original generators have been successfully trained, while the low quality samples from the cutgenerators show that these no longer behave as a generative model after Generator Surgery .
Revised: The high quality samples in the ﬁrst rows show that our original generators have been successfully trained, while the low quality samples from the cut generators show that these no longer behave as a generative model after Generator Surgery .

Original: We produce samples from cut generators by sampling from a censored normal p ( z c ) =
Revised: We produce samples from cut generators by sampling from a censored normal p ( z c ) =

Original: CN (0 , 1) k cin the latent space of a cut generator.
Revised: CN (0 , 1) k c in the latent space of a cut generator.

Original: We note that sampling from the prior p ( z 0 ) =
Revised: We note that sampling from the prior p ( z 0 ) =

Original: CN (0 , 1) k 0 in theoriginal latent space, and pushing that vector through the removed blocks would recover the originalgenerative sampling procedure.
Revised: CN (0 , 1) k 0 in the original latent space, and pushing that vector through the removed blocks would recover the original generative sampling procedure.

Original: There may exist a method for sampling from this "pushforward prior"without saving all parameters from the removed blocks, and such a method could provide a cheapand effective initialization for image recovery, but we defer this exploration to future work.
Revised: There may exist a method for sampling from this "pushforward prior" without saving all parameters from the removed blocks, and such a method could provide a cheap and effective initialization for image recovery, but we defer this exploration to future work.

Original: A.5 Surgery before Training
Revised: A.5 Surgery before Training

Original: [Figure]
Revised: [Figure]

Original: To conﬁrm that layers cannot be cut at training time, we try training BEGAN and DCGAN with avariety of hyperparameter settings for at least 20 epochs after cutting c = 1 blocks.
Revised: To conﬁrm that layers cannot be cut at training time, we try training BEGAN and DCGAN with a variety of hyperparameter settings for at least 20 epochs after cutting c = 1 blocks.

Original: In Fig.
Revised: In Fig.

Original: 9, eachrow represents a single hyperparameter setting and each column is an independent random samplefrom after just one epoch.
Revised: 9, each row represents a single hyperparameter setting and each column is an independent random sample from after just one epoch.

Original: All attempts appear to result in mode collapse with poor quality outputimages.
Revised: All attempts appear to result in mode collapse with poor quality images.

Original: Note the stark difference when compared to the samples from successful training in the ﬁrstrows of Fig. 8.
Revised: Note the stark difference when compared to the samples from successful training in the ﬁrst rows of Fig. 8.

Original: A.6 Experiment Details
Revised: A.6 Experiment Details

Original: We use L-BFGS for all problems and use the following hyperparamters:
Revised: We use L-BFGS for all problems and use the following hyperparamters:

Original: Model
Revised: Model

Original: Task
Revised: Task

Original: Learning Rate
Revised: Learning Rate

Original: Optimizer Steps
Revised: Optimizer Steps

Original: DCGAN
Revised: DCGAN

Original: CS 0.
Revised: CS 0.

Original: Raw Reconstruction
Revised: Raw Reconstruction

Original: BEGAN
Revised: BEGAN

Original: CS
Revised: CS

Original: Raw Reconstruction
Revised: Raw Reconstruction

Original: VAE
Revised: VAE

Original: CS
Revised: CS

Original: Raw Reconstruction
Revised: Raw Reconstruction