Paper:
Energy-Based Models (EBMs) assign unnormalized log-probability to data samples. This functionality has a variety of applications, such as sample synthesis, data denoising, sample restoration, outlier detection, Bayesian reasoning, and many more. But training of EBMs using standard maximum likelihood is extremely slow because it requires sampling from the model distribution. Score matching potentially alleviates this problem. In particular, denoising score matching (Vincent, 2011) has been successfully used to train EBMs. Using noisy data samples with one fixed noise level, these models learn fast and yield good results in data denoising (Saremi and Hyvarinen, 2019). However, demonstrations of such models in high quality sample synthesis of high dimensional data were lacking. Recently, Song and Ermon (2019) have shown that a generative model trained by denoising score matching accomplishes excellent sample synthesis, when trained with data samples corrupted with multiple levels of noise. Here we provide analysis and empirical evidence showing that training with multiple noise levels is necessary when the data dimension is high. Leveraging this insight, we propose a novel EBM trained with multi-scale denoising score matching. Our model exhibits data generation performance comparable to state-of-the-art techniques such as (Song and Ermon, 2019) and GANs, and sets a new baseline for EBMs. The proposed model also provides density information and performs well in an image inpainting task.

Review:
This paper presents a method of learning of energy based models using denoising score matching. This technique has been used before but only with limited success. The authors hypothesize that this is due to the fact that the matching was only performed over a single noise scale. The main idea of this work is to employ a range of scales to learn a single energy function. This trick helps to alleviate the problem of noisy samples concentrating in a low-volume region of the ambient space.
It seems that the paper draws significant inspiration from the work by Song & Ermon, 19. The difference between the two appears to be minor:
1) The density is represented as a Boltzman distribution and therefore the score function is reduced to the gradient of the energy function (this has been done before)
2) Instead of conditioning the energy on the noise level the authors propose to use explicit scaling by the inverse temperature
 
Pros:
+ The paper is mostly well-written.
+ I think Section 2 does a good job at illustrating challenges in training energy-based models using denoising score matching with a single noise scale. 
+ As compared to (Song & Ermon, 19) using the Boltzman distribution ensures that the learned score is an actual conservative vector field. Arguably, learning an image to scalar network is easier than learning an image to image one.
+ Samples from the model are of competitive visual quality.
Cons:
- Scaling energy by the inverse temperature seems to be one of the most important aspects of the paper but is only justified by “intuition from physics”. I’m not entirely sure that this is a valid assumption. In contrast, (Song & Ermon, 19) don’t put any hard constraints on the values of the score for different noise levels besides that they are produced by a single conditional network. I would appreciate if the authors discussed that difference in more detail.
- The authors don’t provide any analysis as to whether the annealed Langevin MC procedure leads to the samples from the right distribution.
- The quantitative results don’t seem to be better (actually, they are worse) than those from (Song & Ermon, 19).
Notes/questions:
* Abstract: “unmormalized” -> “unnormalized”
* Section 2.1, (1): \tilde{x} -> \tilde{\mathbf{x}}
* Section 2.2, paragraph 2: What does superscript C mean in the noisy manifold? Never defined.
* Section 2.2, paragraph 4: “some example” -> “some examples” (?)
* Section 3, paragraph 1: “CIFAT-10” -> “CIFAR-10”
* Section 4, paragraph 2: “for each T as a separate model”. I don’t think this is a correct statement. (Song & Ermon, 19) use a single conditional model for all the noise levels.
* Section 4, paragraph 2: “does not rely on explicit receive noise magnitude” -> “does not rely on receiving noise magnitude explicitly” (?) I also don’t quite understand this entire sentence. Does the model really infer the noise magnitude from a given image? It seems like in Equation (7) there is an assumption that the temperature T is equal to 1. I don’t feel like there is a lot of difference between the proposed model and (Song & Ermon, 19) when it comes to supplying noise information. I’d appreciate if the authors could clarify that bit for me.
My main concern about this paper is that it doesn’t seem like a big step from its starting point (Song & Ermon, 19). The modifications are shown to work empirically but don’t result in a significantly better model. Moreover, I feel like the paper could do a better job at justifying those changes. I’m giving a borderline score but willing to increase it if the authors address my questions.