From: NeurIPS.cc/2022/Conference/Program_Chairs
title: Paper Decision
decision: Accept

From: NeurIPS.cc/2022/Conference/Paper12375/Area_Chair_MMs9
metareview: **Summary**: This paper proposes an approach for using existing generator networks in an energy based model. The authors introduce a "hat" network that accepts output of the generator and a residual image, and returns a scalar representing the energy value. The parameters of the hat network (and potentially of the generator) can be trained using persistent contrastive divergence, where negative samples are generated by using a Gibbs sampler with Langevin updates to generate both the latent variable for the generator and the residual. The authors demonstrate how this approach can be used to refine samples from an existing generator, generate images by jointly training the generator and the energy function, allow sampling over deterministic generators, and use the EBM for anomaly detection. 

**Strengths**: Reviewers [9epW] and [45fe] appreciate the interesting and novel formulation of an energy-based model, which can inspire further work on EBMs for generative learning. Moreover, reviewers agree [9epW,45fe,oJLY] that the proposes formulation is very general and is technically sound. [N2ii] and [45fe] note the fairly clear presentation, with good background on topics such as cooperative learning. All reviewers [9epW, N2ii, 45fe, oJLY] agree that experiments on refinement, generation and OOD detection show improvements relative to baselines. 

**Weaknesses:** While reviewers were overall appreciative of this submission, they also note weaknesses with respect to clarity, discussion, and experimental results. One issue that reviewers feel is overlooked is the additional computation cost associated with the approach [9epW,N2ii]. Reviewer [9epW] states that while main idea is presented clearly, parts of the paper are difficult to parse (multiple examples given). Reviewer [45fe] similarly finds discussion lacking and that experimental details are missing (4 examples given).

Reviewer [9epW] notes that experimental results are also not strong across the board. Sample refine does not appear to a strong suit of hat EBM. FID scores are barely an improvement. This is particularly relevant becase refinement baselines do not require training, whereas paper does not compare to refinement baselines like DOT and DGflow. Moreover, there are some problems with baselines. Discussion and comparison against GEBM is missing, which is a model that also combines a base generator and an energy function to draw/refine samples. EBM baselines in Table 2 are not properly discussed in the paper. While synthesis results are impressive, discussion of why the proposed approach works well are missing. 

**Author Reviewer Discussion**: Authors provided clarifications to all reviewers, and amended the paper to reference related work on GEBM and DGflow. The authors indicate that they will also attempt to compare against these baselines. 

**Reviewer AC Discussion**: The AC indicated that this appears to be highly borderline paper with a significant variance in reviewer scores. Unfortunately, only reviewer [9epW] engaged with the AC during the discussion phase. This reviewer indicated that while the proposed is interesting, they remain of the opinion that the related work section and baselines need work, and that the paper would benefit from a clearer motivation for the proposed approach. On balance they remain of the opinion that this paper is on the borderline. 

**Overall Recommendation:** Based on the numerical average of the scores, this submission would be just about above the threshold for acceptance. At the same time, the AC is concerned with the comments about clarity and in particular with the comments about the lack of discussion of related work. In this context, the sparse discussion and the fact that no reviewers have championed the paper is somewhat worrying. On balance, the AC recommendation is to consider this paper as borderline. It is narrowly above the bar for acceptance, but may have to be cut in favor of other papers.
recommendation: Accept
confidence: Less certain
award: No

From: NeurIPS.cc/2022/Conference/Paper12375/Authors
title: Thanks for your review
comment: We are glad to hear that you maintain your positive evaluation of our work. Contextual information about MCMC implementation will certainly improve the presentation of our method and make our work accessible to a wider audience. We will include a section that addresses your questions in future revisions of the appendix, along with more information about computational costs and scalability.

From: NeurIPS.cc/2022/Conference/Paper12375/Authors
title: FID Score Comparison
comment: We were careful to ensure that all FID scores reported in Table 2 are consistent with the FID scores presented in the Mimicry repo for GAN replication [1], which we believe represents correct FID implementation under the standard FID framework. In particular, the Inception Network is from the original TensorFlow 1 model release, TF1 resizing is used to upsample images to the Inception input size of 224x224, resized images from data sources are anti-aliased correctly when downsampling before being loaded in the TF1 code, and images are saved in a lossless format before processing, as recommended by [2]. The official GEBM code repo [3] does not contain pretrained models but does contain reference samples. We retrained the GEBM model using official code and evaluated scores with our validated FID pipeline for consistent FID reports. We further observe that the samples from our GEBM reimplementation and samples from the GEBM paper/git repo are visually similar and that samples from our work (see Figure 4 in Appendix) have a noticeably sharper visual appearance, although this is a subjective evaluation.

The differences in GEBM scores are likely due to differences in FID implementations, plus potential differences from retraining. The networks used for our Hat EBM, the original GEBM report, and our GEBM reimplementation are nearly identical (ResNet SNGAN architecture). We will release pretrained models for the Hat EBM results reported in the paper to ensure that future researchers can directly validate Hat EBM FID scores.

[1] https://github.com/kwotsin/mimicry  
[2] On Aliased Resizing and Surprising Subtleties in GAN Evaluation. https://arxiv.org/pdf/2104.11222.pdf  
[3] https://github.com/MichaelArbel/GeneralizedEBM

From: NeurIPS.cc/2022/Conference/Paper12375/Reviewer_45fe
title: Confirm accept score
comment: Thanks for your clarifications, I will keep my accept score.
I believe it would be useful to address some of these points (as well as other relevant comments from other reviewers on scalablilty) in the revised version of the paper.

From: NeurIPS.cc/2022/Conference/Paper12375/Reviewer_9epW
title: Post Rebuttal Comment
comment: Thank you for the response.

Follow-up regarding GEBM results: The results reported in GEBM paper vs. what the authors have reported in the revision are considerably different. Are the ResNet architectures significantly different?

From: NeurIPS.cc/2022/Conference/Paper12375/Authors
title: Response to Reviewer oJLY
comment: Thank you for your insightful review and for raising important questions about the relative performance of EBM and GAN models and our contribution toward high-quality unconditional image synthesis.

* Training EBMs can be more compute-intensive than other generative models. However, this cost is ubiquitous across most models that sample during learning [1, 2, 3]. When compared directly to GAN training, our CIFAR-10 EBM is about 4 to 6 times slower. Despite this, we scale even better than some GAN-based methods for more complex datasets like ImageNet. At ImageNet 128x128 we can train our model 25-50% faster than SS-GAN. A reason for this is that EBMs can be learned with relatively small batch sizes (128 for Hat EBM vs 2048 for SS-GAN). The bottleneck for EBM learning is the sampler, which can be run in parallel on each XLA-enabled device (TPU, GPU, etc). For more detailed information please see Appendix D: Computational Requirements. 


* Performance comparison with GANs and relevance of results: We acknowledge that GANs generally produce better images for datasets with lower resolution and fewer images, such as CIFAR-10. A surprising and compelling result of our work is showing that EBMs can, in some circumstances, outperform GANs on large-scale and high-diversity datasets like unconditional ImageNet. In Table 2, we look at two different ImageNet 128x128 experiments. In the first experiment, we compare the EBM with SNGAN, SSGAN, and InfoMax GAN, all using networks that are very similar to the original 128x128 SNGAN. Our EBM achieves an FID of 40.4, outperforming the other methods with the same architecture (SNGAN FID is 65.7, SSGAN FID is 43.9, InfoMax GAN is 58.9). Further, we refer to the GAN replication repo [4] (we took great care to ensure our EBM FID results are directly comparable to [4]) and find even more benefits of using the EBM compared to SSGAN, since SSGAN in the replication repo has a much higher score than the original report (FID 62.4 in replication repo vs. FID 43.9 in original report). This might indicate that a significant portion of strong GAN scores could come from engineering and intensive computation rather than the GAN model.  Although our largest EBM (SNGAN architecture with double the channels for each convolution layer) does not quite match the best SSGAN score (EBM FID 29 vs. SSGAN FID 23), the large SSGAN uses a BigGAN architecture that was aggressively optimized for GAN learning, and a much larger batch size of 2,048 which we were unable to use due to resource constraints. Our paper provides compelling evidence that EBMs can have very strong performance on large unconditional datasets, with significant room for improvement with engineering/architecture/compute.

—

[1] Yilun Du and Igor Mordatch. Implicit Generation and Generalization in Energy-Based Models. NeurIPS 2019.  
[2] Erik Nijkamp, Mitch Hill, Tian Han, Ying Nian Wu, and Song-Chun Zhu. On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Models. AAAI 2020.  
[3] Will Grathwohl, Kuan-Chieh Wang, Joern-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly and energy-based model and you should treat it like one. ICLR 2020.  
[4] https://github.com/kwotsin/mimicry

From: NeurIPS.cc/2022/Conference/Paper12375/Authors
title: Response to Reviewer 45fe
comment: Thanks for your supportive evaluation of our work. We address your questions about the technical aspects of EBM training below.

* Number of MCMC steps: Previous EBM works such as [1, 2, 3, 4] (see below) have empirically established that the range of 50 to 100 steps provides an optimal balance between stability and computation for MCMC-based EBM learning. In particular, training with fewer than 40 to 50 steps can lead to model instability, while training with more than 100 steps is more expensive and typically does not improve results beyond the 50 to 100 step range. This range appears to be quite consistent across many EBM works.
* MALA and HMC: Prior EBM work has typically found that a Metropolis-Hastings adjustment for the Langevin Chains (i.e. MALA instead of unadjusted Langevin dynamics) or full HMC does not improve results [2]. One explanation for why MALA and HMC are unnecessary is that our Hat EBMs are learning in the “non-convergent” regime described in [3], meaning that the model can produce realistic samples with a small number of MCMC steps even when steady-state samples have very poor quality. We note that nearly all EBMs are learned in a non-convergent regime, and that MALA and HMC have not been observed to overcome the poor convergence of MCMC samples during EBM learning.
* Size of Persistent Bank: Standard practice from prior works suggests using a persistent bank with size 10,000 [2, 3]. We did not observe any major changes once our bank was sufficiently large. A bank of size 1,000 led to instability, while a bank of 50,000 gave about the same results.

—

[1] Jianwen Xie, Yang Lu, Song-Chun Zhu, and Ying Nian Wu. A theory of generative ConvNet. ICML 2016.  
[2] Yilun Du and Igor Mordatch. Implicit Generation and Generalization in Energy-Based Models. NeurIPS 2019.  
[3] Erik Nijkamp, Mitch Hill, Tian Han, Ying Nian Wu, and Song-Chun Zhu. On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Models. AAAI 2020.  
[4] Will Grathwohl, Kuan-Chieh Wang, Joern-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly and energy-based model and you should treat it like one. ICLR 2020.

From: NeurIPS.cc/2022/Conference/Paper12375/Authors
title: Response to Reviewer 9epW (Part 2 of 2)
comment: * Original Motivation of Hat EBM: The original motivation of the Hat EBM was the retrofit experiment, where we aimed to build a probabilistic model and sample from the latent space of an arbitrary generator model. This is the most unique application of the Hat EBM that does not appear in prior work. Refinement can be considered a special case of retrofit where the pretrained generator can provide initial realistic image samples. Later experiments with the conditional Hat EBM show that the method is also capable of high-quality synthesis.


* Line 232 and importance of residual $Y$: This statement is more fully explored in Appendix J (called Appendix H in the original appendix version). One could imagine a Hat EBM refinement experiment where $Y$ is not present, the negative samples are $X_i^- = G(Z_{i,K})$ where $Z_{i,K}$ comes from Langevin dynamics, and we posit that $X_i^+ = G(Z_i^+)$ for some unknown $Z_i^+$. While the $Z_i^+$ does not exist when the generator cannot perfectly reconstruct the data, we can run the Hat EBM training using this set-up. Interestingly, this leads to unstable training, as shown in Figure 2 of the appendix. This shows that $Y$ is important for our method for practical reasons as well as the theoretical formulation, even when $Y$ is barely visible. A possible explanation is that the energy network can distinguish between $X_i^+$ and $X_i^-$, which means that it can tell apart data and samples directly on the generator image manifold. This destabilizes training as the energy network learns to assign very low values to $X_i^+$ and very high values to $X_i^-$ until training collapses.

From: NeurIPS.cc/2022/Conference/Paper12375/Authors
title: Response to Reviewer 9epW (Part 1 of 2)
comment: Thank you for your careful review of our work and for your useful comments. Your feedback will lead to positive changes in our work. We address the related work that you have brought to our attention and other weaknesses and questions.

* Historical Generator Bank Update and Pseudocode: We agree that the exposition of the conditional Hat EBM for synthesis, especially the generator update, would be improved with pseudocode. Appendix G contains further details about the historical generator update mentioned at line 180 of the original main paper version. We included a new section of the supplementary material (Appendix H in revised appendix) to provide pseudocode for training the Conditional Hat EBM for image synthesis.

* Other text suggestions: Thank you for bringing up these points, we edited the text to express them more clearly. Typos in lines 112 and 167 were fixed. Other changes:  
    * What is $\sigma$?: This is the standard deviation of a hypothetical normal prior that we could put on the residual of the Hat EBM. Since we do not use this prior in our experiments, we chose to remove that text to avoid confusion.
    * $\phi$ not yet defined: Thanks for pointing this out. We intended for Section 3.3 to discuss the conditional hat EBM for a fixed generator with no weight variable notation, and for Section 3.4 to introduce $\phi$. We removed \phi from Section 3.3.
    * Line 158, $X_t$ is a typo, we changed it to say $X_K$.

* Related refinement methods: The DOT and DGflow methods are very relevant and we have included references in the current revision. We are working on a direct comparison with DOT and DGflow using our SNGAN baseline and will include this in future revisions. We refer to comparisons in the most recent work, DGflow, to get a rough idea of the relative performance of DOT, DDLS, and DGflow for context. On unconditional CIFAR-10 with a shared baseline net with FID 20.9, DOT achieves refinement FID of 15.78, and DGflow achieves and FID of 15.30 (Table 2 of DGflow paper). On unconditional CIFAR-10 with a shared baseline net with Inception Score 8.38, DDLS achieves an Inception Score of 9.09 and DGflow achieves an Inception Score of 9.35 (Table 3 of DGflow paper). The evaluations in the DGflow paper generally show that DOT, DDLS, and DGflow have similar refinement abilities, with DGflow having an edge. Since Hat EBM gives an edge of improvement over DDLS, this suggests that Hat EBM performance is comparable to other methods for generator refinement.  
We acknowledge that Hat EBM requires additional computation for training the network. While the cost can be viewed as a weakness of our method, the flexibility of refining a generator without its partner discriminator, which may not be available, can also be seen as a strength. We also note that our Hat EBM can be used to sample from both probabilistic generators (refinement) and non-probabilistic generators (retrofit). DGflow can refine probabilistic generators with an initial latent sampling method, but does not convert non-probabilistic generators into probabilistic models.

* GEBM and Table 2 Methods: Thank you for pointing out the GEBM work. We have included this as a reference in the revision. We implemented GEBM training using the official code from [a] with the same ResNet SNGAN architecture as our CIFAR-10 Hat EBM for synthesis, evaluated the model with the official Tensorflow FID code, and obtained an FID score of 30.1. We included this score in Table 2 of the main paper. We updated the supplementary material to include a thorough comparison of the EBM methods from Table 2, including GEBM, in Appendix I.

* Performance Improvement vs. Baseline EBMs: Our network architectures are fair comparisons to other methods. We use a ResNet SNGAN with spectral normalization removed from the EBM and with batch normalization removed from the generator across all experiments (except for retrofit which uses a modified generator). This network has 4M parameters for CIFAR-10. This is the EBM network as the GEBM, and smaller than the EBM in other works such as IGEBM (18M parameters in network from official CIFAR-10 code) and JEM (37M parameter WideResNet). The Hat EBM formulation treats the observed images as the sum of (1) unobserved latent variables passed through a generator network, and, (2) the residual random variable which may span the gap between the ancestral output of the generator model, and, the underlying true image manifold. We believe this is the root cause of the empirical improvement in performance, but we will conduct further studies in the future.

—

[a] https://github.com/MichaelArbel/GeneralizedEBM

From: NeurIPS.cc/2022/Conference/Paper12375/Authors
title: Response to Reviewer N2ii
comment: Thank you for your thoughtful assessment of our work. We are glad that you found the contextual information in the appendix helpful. We address your comments about compute requirements and the appearance of $Y$ below.

* Compute Requirements: It is true that EBM methods can be computationally expensive relative to other methods. The cost of training a hat EBM for refinement or synthesis is on par with the cost of other MCMC-based methods for EBMs, and our network architectures are on par with or smaller than those used by other works (for CIFAR-10, our hat network has 4M params, IGEBM has 18M params, JEM has 37M). Training a Hat EBM for retrofit is about 2.5x slower than training for refinement or synthesis because one latent Langevin step is used for each residual image Langevin step, and the latent step must backpropagate through both the EBM and generator.  On simpler datasets like CIFAR-10, EBM training is about 4x to 6x slower than GAN training. For complex datasets like ImageNet, however, our large-scale method only requires about 25% to 50% of the resources used for the state-of-the-art SSGAN, making the method computationally efficient relative to GANs when scaling up to harder datasets with larger model sizes. This desirable scaling behavior can be attributed to 1) efficient scaling of EBMs across parallel devices, since the bottleneck comes from Langevin compute rather than data processing, and 2) because our EBM achieves strong synthesis results with small batches while the large-scale SSGAN uses a batch size of 2,048.
* Appearance of $G(Z)$ and $Y$: Your understanding is correct. When we examine the images $G(Z)$ and $Y$ separately, we observe that $G(Z)$ looks very similar to the final image and that $Y$ is usually close to the constant gray image where all pixels value 0 (note we use a pixel range of [-1, 1]). This is illustrated in Figure 3, where the pixel intensities of $Y$ have been multiplied by 5 for improved visualization. Refinement experiments had the least visible residuals, synthesis experiments had the most visible residuals, and retrofit residual magnitude was in the middle.

From: NeurIPS.cc/2022/Conference/Paper12375/Reviewer_N2ii
rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary: This paper proposes the Hat EBM, which is a method to incorporate a sampler (such as a pre-trained generator) into an energy-based model (EBM). The basic idea is to define a joint probability distribution over both the latent space of the pre-trained generator and a “residual space” – the Hat EBM takes as input a synthesized image from the generator combined with a “residual image,” and uses the output of the Hat EBM in an MCMC sampler. The distribution over these residual images is also learned jointly via MCMC. This approach allows for unconditional image generation, image refinement from a pretrained generator such as a GAN, defining implicit samplers for deterministic decoders, and out-of-distribution (OOD) detection.  Hat EBM achieves strong empirical results on complex datasets such as ImageNet 128x128.

strengths_and_weaknesses: Strengths:
- The paper’s empirical results are strong across a wide range of tasks.
- The presentation is fairly clear and easy to follow. They also provided good background material on concepts that were important for understanding the paper, such as cooperative learning.

Weaknesses:
- The proposed approach hinges on some theoretical assumptions and appears to be quite compute-intensive. However, the authors included a thorough description of the computational limitations of their method in Appendix B and D. Appendix E also included a description of the theoretical assumptions required for their method (e.g. correct model specification and convergence of MCMC), as well as precautions necessary for incorporating the residual vector Y during training in Appendix H. I appreciated the care they took here.


questions: Questions:
- Clarification: what does it mean on l.107 when the text says that the “majority of the appearance of G(Z) + Y” comes from the generator network? Is that just saying that the generator is responsible for most of the content/variation in the image?

Very minor miscellaneous things:
- “We propose [to] initialize Langevin sampling in l. 167

limitations: The authors included a description of the potential negative societal impacts in Appendix B, though it could have been a bit more thorough.
ethics_flag: No
soundness: 3 good
presentation: 3 good
contribution: 3 good

From: NeurIPS.cc/2022/Conference/Paper12375/Reviewer_9epW
rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary: The paper proposes “Hat EBM”, an energy-based generative model that incorporates a generator network that maps points from a latent space to the data space into the EBM. The resulting EBM energy is a function of the image generated by the generator and an image residual. The authors propose methods to: train the EBM on top of a fixed generator and train both the generator and the EBM jointly. The authors show the applicability of the proposed method to sample refinement, synthesis and out-of-distribution detection tasks.
strengths_and_weaknesses: [Strengths]

- The paper presents an interesting and novel formulation of an energy-based model.
- The proposed method is technically sound.
- Experiments have been presented on three different tasks of refinement, generation, and OOD detection. The results show a significant improvement over the baseline EBM models for the image synthesis task.

[Weaknesses]

- While the the key idea of the paper is presented clearly, parts of the paper are difficult to parse. For example, the paragraph beginning at line 180 lacks clarity. Following are some other comments and suggestions:
    - It appears from the text that training algorithm is not straightforward. Thus, the main text would benefit significantly from a separate algorithm/pseudocode of the proposed method.
    - Section 2: Instead of saying “This work [D]…” consider using “XYZ et al. [D]”.
    - Line 110: What is \sigma?
    - Line 112: log determinant of **the Jacobian** of G.
    - Line 139: \phi has not been defined yet.
    - Line 158: What does the subscript t indicate in X_t? Is this a typo?
    - Line 167: propose **to** initialize…
- From the results in Table 1, it appears that sample refinement is not a strong suit of hat EBM. The FID scores are barely an improvement. This is particularly important because the baseline refinement methods (e.g., DDLS) do not require training of another network unlike hat EBM. Furthermore, the current manuscript doesn’t discuss/compare against other sample refinement baselines, e.g., DOT [1] and DGflow [2]. In particular, DGflow also studies a method to refine samples from all kinds of probabilistic generators.
- A discussion of and comparison against the very relevant GEBM [3] is missing. GEBM also uses a base generator and an energy function on top of the generator to draw/refine samples. Furthermore, EBM baselines considered in Table 2 have not been properly discussed in the paper. This discussion is important to correctly contextualize this work.
- While the synthesis results are impressive, the overall significance is marred by a lack of discussion. Why does hat EBM perform so well against these baseline EBMs? Is the comparison fair (e.g., in terms of networks)?

[1] Tanaka, Akinori. "Discriminator optimal transport." *Advances in Neural Information Processing Systems*
 32 (2019).

[2] Ansari, Abdul Fatir, Ming Liang Ang, and Harold Soh. "Refining deep generative models via discriminator gradient flow." arXiv preprint arXiv:2012.00780 (2020).

[3] Arbel, Michael, Liang Zhou, and Arthur Gretton. "Generalized energy based models." arXiv preprint arXiv:2003.05033 (2020).
questions: - What was the original motivation for the image residual formulation of the EBM? It appears that the original motivation was the problem of refinement; however, the proposed method doesn’t refine samples better than the baseline (which does not require additional training).
- Line 232: How does the incorporation of the residual Y help prevent the assignment of “increasingly high energy to generator samples”?
- Do the authors have any comments on why hat EBM performs so well against baseline EBMs on image synthesis?
limitations: The authors have not discussed limitations of their work. I suggest that they add a few lines to discuss the shortcomings of the proposed work and potential future directions to fix these limitations.
ethics_flag: No
soundness: 3 good
presentation: 2 fair
contribution: 2 fair

From: NeurIPS.cc/2022/Conference/Paper12375/Reviewer_45fe
rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary: This paper shows how one can use existing generator networks within an energy based model. The authors introduce a *hat network* that takes as input the output of the generator plus a residual image, and returns a scalar representing the energy value. The parameters of the hat network (and potentially of the generator) can be trained with Maximum Likelihood learning where negative samples can be obtained with MCMC steps (Gibbs sampler with Langevin updates) 

 

The authors explore different learning scenarios and show how this framework can be used to 1) Refine samples of an existing pretrained generator; 2) Generate images by learning the whole EBM model (generator and hat network) from scratch; 3) Allowing sampling over deterministic generators; 4) Use the EBM for anomaly detection. 
strengths_and_weaknesses: STRENGTHS 

1. There has been an increased focus lately on using EBM for generative learning. This work presents a novel idea that I believe can inspire further research in the neurips community. 

2. This is a quite general formulation that can be used for any type of deep generative model 

3. Learning does not require posterior inference over the latent states, making the model simpler and more scalable.  

4. The authors show empirically that the hat EMB outperforms competing methods in a number of different tasks. It also generates compelling unconditional Imagenet samples.  

5. The paper is well written and clear 

 

WEAKNESSES 

The authors should provide more discussions/experimental details, particularly to make this framework more accessible to people not familiar with MCMC (and its pitfalls): 

1. In your experiment you chose 50/100 MCMC steps. What is a good strategy to choose this hyperparameter and ensure that the chains have converged well enough for learning to be efficient? 

2. In the paper you do not mention the accept-reject step needed when doing the Langevin updates to define a valid Metropolis-Hastings sampler. Do you monitor the acceptance rate during training? 

3. What would it take to use Hamiltonian Monte Carlo instead of the Langevin updates to improve the convergence of the chains? 

4. How robust is the algorithm to the choice of the size of bank memory? 

 

 
questions: See questions in the weaknesses section above.
limitations: Yes
ethics_flag: No
soundness: 4 excellent
presentation: 3 good
contribution: 4 excellent

From: NeurIPS.cc/2022/Conference/Paper12375/Reviewer_oJLY
rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary: This work proposes a method for using a generator network as the foundation for an Energy-Based Model (EBM). In essence, the paper considers that the output of the generator network is superimposed with a latent variable defined in the image space; this produces the final image. The generator network is presented with a low-dimensional latent variable input. The low dimensionality of this input space renders the modeling problem less multimodal, thus more tractable. 
The described formulation of the model renders it amenable to MCMC-type sampling (Langevin gradients) for performing training, in a contrastive divergence type of maximum likelihood. 
strengths_and_weaknesses: Strengths: The model is solid and the training algorithm is straightforward and correct. The authors provide a variety of experiments that deal with different application types. In all cases, they use standard benchmark data to experiment with. They compare to related EBM-type approaches, as well as some state of the art alternatives based on the GAN principle.

Weaknesses: The authors do not explain how the method compares to the alternatives in terms of computational costs, both for training and at test time. In addition, the results are not fully convincing of the usefulness of the method. Specifically, while the method improves over previous EBMs, it seems of lackluster performance compared to GAN-type methods. How do the authors explain this shortcoming? Why would someone prefer their method over the GAN-type alternatives?
questions: How the method compares to the alternatives in terms of computational costs, both for training and at test time?
How do the authors explain the better performance of GAN-type alternatives?
What is the main merit of the proposed approach that adds important value to the state of the art?
limitations: n/a
ethics_flag: No
soundness: 3 good
presentation: 2 fair
contribution: 3 good