Paper: Diffusion Random Feature Model (ICLR-2024)

Reviewer comment (Novelty):
The proposed framework is inspired by both the denoising diffusion probabilistic model introduced by Ho et al. (2020), and the semi-random features introduced by Kawaguchi et al. (2018). It seems the proposed Diffusion Random Feature Model (DRFM) is incremental in integrating the above two works. Specifically, in Kawaguchi et al. (2018), a parameterization of one single-hidden layer is defined in a similar form to Eq (15). The authors employ this parameterization for the noise-prediction network in training DDPM.

Author response:
-  We would like to highlight the fact that our model architecture is capable of incorporating time as an input with a separate set of weights associated with it. This is much different from the architecture proposed in Kawaguchi et al, (2018) where the authors only consider the input $\mathbf{x}$.

- We would like to point out that although our final result is based on results from Chen et. al. (2022), we believe that the quantification of $\varepsilon_{\text{score}}$ is a significant contribution to understand the number of parameters needed (with respect to the input dimension and number of timesteps) to obtain a small error. This quantification helps us to choose the appropriate number of model parameters before training. In particular, note that for appropriate choices of $T$ and $h$, the bounds in Chen et. al. (2022), can be atmost reduced to $TV(p_{\theta}(0),q)\leq\mathcal{O}(\varepsilon+\varepsilon_{\text{score}})$. Our results show that we can also choose the parameters of the model so that $TV(p_{\theta}(0),q)\leq\mathcal{O}(\varepsilon)$.

Reviewer comment (Theory):
Throughout the paper, it is not straightforward to understand the advantage of the parameterization in Eq (15) in terms of interpretability and computation complexity. Any analysis or experimental results should be provided to address these challenges stated in the introduction.

Author response:
- The advantage of the proposed architecture lies in being able to write it as a random feature model which helps us to obtain the results given in the paper and quantifying the the error bounds. The bounds show that by choosing $N\gg Kd$ for our model, we can reduce the bound $TV(p_{\theta}(0),q)\leq\mathcal{O}(\varepsilon+\varepsilon_{\text{score}})$ to $TV(p_{\theta}(0),q)\leq\mathcal{O}(\varepsilon)$. We included a brief description of the signicance of our work in the revised version of the paper.

- Our choice of the model in (15) is based on Fourier-type feature functions as they represent the Gaussian kernel when the components of matrix $\mathbf{W}$ are sampled from Gaussian distribution and bias vector $\mathbf{b}$ is sampled from the uniform distribution. Since the aim of diffusion models is to learn the mean (and covariance) of the reverse process whose transition kernels are Gaussian distributions, we choose this particular form. For more details on the relation between Fourier-type features and Gaussian kernel, we refer to Rahimi and Recht (2007).

Reviewer comment (Evaluation):
The experiments validate the efficacy of the proposed DRFM. However, when considering the challenges diffusion models face in terms of interpretability and computation, the experiments seem to omit direct comparisons. Additionally, many of the presented results are not quantified using specific metrics, as observed in Fig 2. This makes it challenging to identify the model's superior performance.

Author response:
- Table 1 (see revised version of the paper and also given below) gives the Fr\'echet Inception Distance (FID) calculated using a batch of 50 images from the training dataset and 15 of the generated images by each of the models. Note that the scores values given are for the sake of comparing the four methods and more samples may improve the FID score. We see that for the generative task, the proposed DRFM architecture gives the lowest scores. The more commonly used U-Net model is at the third position after the NN model. For the denoising task, we see that the NN gives the best results, (although some of the images are inconsistently noisy), followed by U-Net and then DRFM. Note that for U-Net, while most images are denoised perfectly, some images are incompletely formed which leads to a lower FID score. Moreover, we see that all the FID scores are also consistent with the qualitative assessment made in Figures 2 and 3 (updated version of the paper).

| Model | Timesteps | FID Score |
|---------|-------------|-------------|
| DRFM | 100 | **453.87** |
| U-Net | 100 | 463.28 |
| NN | 100 | 457.21 |
| RF | 100 | 470.12 |

Table: FID scores for generative task

| Model | Timesteps | FID Score |
|---------|-------------|-------------|
| DRFM | 100 | 394.99 |
| U-Net | 100 | 388.38 |
| NN | 100 | **378.18** |
| RF | 100 | 450.23 |

Table: FID scores for denoising task

Reviewer comment (Experiments):
The paper currently compares the DRFM solely with a fully-connected layer. Given the prominence of both the Unet and transformer architectures in noise prediction networks, it would be beneficial for the authors to discuss the relationship of DRFM to these models. In particular, considering that transformer layers utilize a combination of the attention mechanism and fully-connected MLPs, they align more closely with the scope of the fully-connected layer baseline. It would be valuable to explore this connection further.

Author response:
- As per the suggestion, we have now added a few more results using a simple U-Net architecture with the number of trainable parameters similar to the DRFM model. As we see in Figure 2 in the revised version of the paper, for generative tasks, the U-Net performs very similar to that of DRFM.

Reviewer comment (Experiments):
Since the authors have generalized the random features with the properties of score-matching, it remains unknown whether DRFM can improve the performance in the tasks evaluated in Kawaguchi et al. (2018).

Author response:
- The tasks in Kawaguchi et al. (2018) are different from the goal of this paper. Our goal is to particularly develop an interpretable model that can be potentially used for understanding diffusion models. Moreover, the model architecture in Kawaguchi et al. (2018) does not involve a trainable time parameter which makes our proposed model much different.

Reviewer comment (Theory):
In the abstract, the authors state that diffusion models suffer from a lack of theoretical justification. This assertion is somewhat unclear and could benefit from further specificity. It would be helpful if the authors could elucidate whether this deficiency refers to aspects within statistical, optimization, or other theoretical domains.

Author response:
We mention that diffusion models suffer from a lack of theoretical justification to indicate that two aspects:
- Parameterized models are a black box: The models used to train diffusion models are too complex w.r.t. the number of layers, architecture, etc and can not be analyzed for generalization bounds.
- Optimization process: The number of results analyzing the training/sampling process is limited with some significant results based on SDE formulation of DDPM given in Chen et. al. (2022) , Lee et. al. (2022).
For our proposed work, we focus on the first aspect as there are already existing results based on the optimization aspect.

Reviewer comment (Writing):
Mathematical notations are redundant and inconsistent. For example, $\tilde \epsilon_i$ does not seem to be used anywhere in section 2.1 and the following sections.

Author response:
When we use the reparameterization trick, we use the notation $\tilde{\epsilon}_i$ to differentiate it from $\epsilon_i$. Note that although $\tilde{\epsilon}_i$ is also a Gaussian distribution, it is obtained by taking a sum of Gaussian distributions of the previous and current timestep. Although in practice both $\epsilon$ and $\tilde{\epsilon}_i$ are sampled from $\mathcal{N}(0,I)$, they are derived differently and hence the separate notation.

Reviewer comment (Experiments):
On the numerical side, there are no quantitative comparisons between the different approaches, which makes it unconvincing that the proposed approach really outperforms the one-hidden-layer neural network.

Author response:
As per the suggestion, we have now added a few more results using a simple U-Net architecture with the number of trainable parameters similar to the DRFM model. As we see in Figure 2, for generative tasks, the U-Net performs very similar to that of DRFM. We have also added Table 3 that gives the Fr\'echet Inception Distance (FID) calculated using a batch of 50 images from the training dataset and 15 of the generated images by each of the models. Note that the scores values given are for the sake of comparing the four methods and more samples may improve the FID score. We see that for the generative task, the the proposed DRFM architecture gives the lowest scores. The more commonly used U-Net model is at the third position after the NN model. For the denoising task, we see that the NN gives the best results, (although some of the images are inconsistently noisy), followed by U-Net and then DRFM. Note that for U-Net, while most images are denoised perfectly, some images are incompletly formed which leads to a lower FID score. Moreover, we see that all the FID scores are also consistent with the qualitative assessment made in Figures 2 and 3.

- Table with FID scores for generative task
| Model | Timesteps | FID Score|
| ----------- | ----------- |---------|
| DRFM | 100 | **453.87**|
|U-Net | 100 | 463.28|
| NN | 100 | 457.21|
| RF | 100 | 470.12|

- Table with FID scores for denoising task
| Model | Timesteps | FID Score|
| ----------- | ----------- |---------|
| DRFM | 100 | 394.99|
|U-Net | 100 | 388.38|
| NN | 100 | **378.18**|
| RF | 100 |450.23|

Reviewer comment (Presentation):
The presentation mixes discrete and continuous frameworks for diffusion models, and it is suggested to choose one.

Author response:
Our initial purpose for including the continous SDE is to recall the use the contious framework to show the connection between the loss function used to predict noise (Eq.8 in our paper) and the score matching formula (Eq. 13 in our paper). Apparently, there is a direct connection without using the continuos framework. Our training model is based on the discrete framework.

Reviewer comment (Writing):
There is a typo on page 4: "features [if -> of] the input data."

Author response:
Fixed.

Reviewer comment (Writing):
On page 5, the phrasing "random and trainable" is confusing.

Author response:
We remove "random" in the phrase for clarity.

Reviewer comment (Presentation):
The definition of \( \mathcal{F}_\omega \) should be placed in the main text if it is used in a lemma.

Author response:
Done.

Reviewer comment (Experiments):
The authors only experiment with very simple and small datasets such as fashion MNIST. It is not that clear what happens for more complex datasets. Here, in the diffusion RF model, feature learning is absent because W is fixed at initialization. It is hard to believe that the model can be successful for more complex tasks.

Author response:
- The fixing of weights and biases in first layer is a common approach in random feature models. The idea is based on using large enough weights and biases (fixed) such that they can approximate shift invariant kernels, see Rahimi and Recht (2007) for more details. For example, Fourier-type feature functions represent the Gaussian kernel when the components of matrix $\mathbf{W}$ are sampled from Gaussian distribution and bias vector $\mathbf{b}$ is sampled from the uniform distribution. Since the aim of diffusion models is to learn the mean (and covariance) of the reverse process whose transition kernels are Gaussian distributions, we choose this particular form. For more details on the relation between Fourier-type features and Gaussian kernel, we refer to Rahimi and Recht (2007).
 - In addition to the fashion MNIST dataset, we also provide some experiments based on music samples to test our method for different types of datasets. Additionally, we have added experiments using U-Net, a more commonly used architecture for diffusion models (Figure 2 and 3 in revised version) along with a table of FID scores (Table 1 and also given below). 
- Table with FID scores for generative task
| Model | Timesteps | FID Score|
| ----------- | ----------- |---------|
| DRFM | 100 | **453.87**|
|U-Net | 100 | 463.28|
| NN | 100 | 457.21|
| RF | 100 | 470.12|

- Table with FID scores for denoising task
| Model | Timesteps | FID Score|
| ----------- | ----------- |---------|
| DRFM | 100 | 394.99|
|U-Net | 100 | 388.38|
| NN | 100 | **378.18**|
| RF | 100 |450.23|

- We see that for the generative task, the the proposed DRFM architecture gives the lowest scores. The more commonly used U-Net model is at the third position after the NN model. For the denoising task, we see that the NN gives the best results, (although some of the images are inconsistently noisy), followed by U-Net and then DRFM. Note that for U-Net, while most images are denoised perfectly, some images are incompletely formed which leads to a lower FID score. To summarize, we see that all the three methods i.e., DRFM, NN and U-Net are comparable although the last two do not enjoy the theoretical interpretability that DRFM does.

Reviewer comment (Novelty):
The main goal of the paper is not clear. It is unclear whether the authors are proposing the RF model as a tool to decrease computational complexity (like Rahimi and Recht) or as some theoretical tool to analyze a more sophisticated phenomenon (like Mei and Montanari, 2019). It is also unclear whether the authors expect such models to achieve similar accuracies to state of the art methods.

Author response:
- **Goal:** The goal of this paper is to propose a model that can be used as a theoretical tool to understand the training process of diffusion models and have a quantified method to bound the sampling error based on the properties of the model. For example, reformulation of DRFM into random features shows that there is one random feature model corresponding to each timestep. Furthermore, by choosing $N\geq Kd$ (as we chose in the experiments as well), we can bound the sampling error $TV(p_{\theta}(0),q)\leq\mathcal{O}(\varepsilon)$ which would otherwise be $TV(p_{\theta}(0),q)\leq\mathcal{O}(\varepsilon+\varepsilon_{\text{score}})$.

 - For this current work, we have compared to some similar methods and also added some additional experiments with more commonly used architectures such as the U-Net in the revised version of the paper. In the current setting, we do not see a significant difference between the performance of DRFM and the other models, although none of the other models such as NN and U-Net enjoy the interpretability as DRFM. We leave further exploration using more complex models and tasks for future work.

Reviewer comment (Novelty):
The purpose of the model in describing a particular phenomenon is not clear. While Theorem 3.3 is interesting, it is unclear what phenomenon this model is trying to describe. For example, Mei and Montanari (2019) and others use RF models to analyze double descent and to show that this phenomenon exists even in very simple models.

Author response:
In Theorem 3.3 we quantify the upper bounds for the total variation between the true distribution and learned distribution. In particular, we show that when DRFM is used for learning the distribution, we can choose the number of features $N$ in such a way that the error due to the parameterized model becomes small.

Reviewer comment (Theory):
The motivation for the particular choice of random features is unclear. Specifically, why was W chosen to be the random weights while the other parameters are trained?

Author response:
- We focus mainly on Fourier-type feature functions as they represent the Gaussian kernel when the components of matrix $\mathbf{W}$ are sampled from Gaussian distribution and bias vector $\mathbf{b}$ is sampled from the uniform distribution. Since the aim of diffusion models is to learn the mean (and covariance) of the reverse process whose transition kernels are Gaussian distributions, we choose this particular form. For more details on the relation between Fourier-type features and Gaussian kernel, we refer to Rahimi and Recht (2007).
 - The idea of fixing first layer weights and training the others is based on the training of random feature models. The feature function with fixed weights is a way to approximate a kernel depending on the distribution from which the weights are sampled. Detailed explanation and derivations can be found in Rahimi and Recht (2007, 2008) and Rudi and Rosasco (2017).

Reviewer comment (Theory):
In the model analyzed, it may be more interesting to study the case where the dimension d is proportional to N. The authors should explain what will happen in that regime to the bound in (19). In general, a more thorough discussion after Theorem 3.3 would benefit the paper. It is unclear what we understand from the generalization bound.

Author response:
- Note that the results of random features in general hold in the overparameterized setting. Unless some additional information is known about the target function or the underlying distribution, the number of features $N$ can not be reduced arbitrarily. For example, for a simple function approximation problem, if the target function is a sparse additive function, then the number of features $N$ can be reduced as explained in Hashemi et. al. (2023), Rudi and Rosasco (2017).

- The quantification in Theorem 3.3 helps us to choose the appropriate number of model parameters before training. In particular, note that for appropriate choices of $T$ and $h$, the bounds in Chen et. al. (2022) can be atmost reduced to $TV(p_{\theta}(0),q)\leq\mathcal{O}(\varepsilon+\varepsilon_{\text{score}})$. Our results show that we can also choose the parameters $N\gg Kd$ in the model so that $TV(p_{\theta}(0),q)\leq\mathcal{O}(\varepsilon)$. We have briefly pointed this out in Section 1 of our revised version.

Reviewer comment (Experiments):
How do other choices of random feature maps behave compared to the sin/cos map chosen in the current paper? A sensitivity analysis would improve the validity of the paper.

Author response:
- Theoretically, any activation function (instead of sin/cos) would suffice as long as the activation function is bounded as indicated in Rahimi and Recht (2007). 
- While numerical results might differe slightly based on the choice of feature functions, we focus mainly on Fourier-type feature ($\sin(.)$ for the feature type) functions as they represent the Gaussian kernel when the components of matrix $\mathbf{W}$ are sampled from Gaussian distribution and bias vector $\mathbf{b}$ is sampled from the uniform distribution. Since the aim of diffusion models is to learn the mean (and covariance) of the reverse process whose transition kernels are Gaussian distributions, we choose this particular form. For more details on the relation between Fourier-type features and Gaussian kernel, we refer to Rahimi and Recht (2007). 
- Our choice of cosine function for representing the time parameter reflects the idea of positional encoding as commonly used in classical diffusion models. However, theoretically as mentioned above, any bounded function would suffice.

Reviewer comment (Novelty):
The random features model was first introduced as an approximation scheme for the kernel method and later used as a toy model for studying neural networks. I personally understand it as a theoretical tool rather than a practical algorithm. Therefore, I think that the idea of using the random features model in generative modeling is not particularly interesting unless it provides a fundamentally new understanding of diffusion-based generative modeling. While the paper provides some theoretical results, they appear to be simple extensions of previously known results.

Author response:
- Using random features for diffusion models could be one of the ways in which diffusion models can be understood. Note that random features are an approximation of (shift invariant) kernels depending on the choice of distribution of the fixed weights. Since transition kernels are essentially Gaussian distributions in diffusion models, our proposed method is the first step to exploring diffusion models with interpretable architectures.

- The performance of random feature models can be considered superior to other similar surrogate models such as neural networks. They have a function space similar to that of shallow neural networks (the function class of random feature models is dense in the space of two-layer neural networks) and are computationally cheaper and easier to implement. In comparison to deeper neural networks which are a black-box, random feature models can quantify the generalization bound based on the number of features and measurements required.   

- Although the results may seem extensions of existing works, we would like to point out that the proposed architecture is not explicitly a random feature model. Firstly, it involves two layers of training. Secondly, it involves a certain involvement of time parameter which is uncommon to the classical random feature or the semi-random feature approach. Our major contribution lies in reducing the proposed architecture to a random feature-based model and applying the extended approximation bound (Lemma 3.2) to conclude that the bounds in Chen et. al. (2022) can be reduced from $TV(p_{\theta}(0),q)\leq\mathcal{O}(\varepsilon+\varepsilon_{\text{score}})$ to $TV(p_{\theta}(0),q)\leq\mathcal{O}(\varepsilon)$ with an appropriate choice of the number of parameters.

Reviewer comment (Experiments):
To my understanding, the finding of the experiment section can be summarized as "training only $\theta^1, \theta^2$ showed better performance than training all parameters or training only $\theta^2$". Can the authors provide results for different $N$'s? The current experiment is using huge $N$, and it is not surprising that restricting the model class can lead to a better generalization.

Author response:
- From the theoretical results presented in the paper, we need to choose $N\gg Kd$ to achieve a small error for our proposed method. Thus for our experiments we choose $N=80000$ ($N\gg Kd$, where $Kd = 100\times 28\times 28$). From previous works on random features in Rahimi and Recht (2007, 2008), Rudi and Rosasco (2017), the results for random feature models hold in an overparameterized setting.

Reviewer comment (Evaluation):
Can the authors provide other popular evaluation metrics for generative models such as inception score, Frechet inception distance, etc.?

Author response:
Thank you for the suggestion. We have now included the FID scores comparing different methods in the revised version of the paper.

- Table 1 in the updated version of the paper (also given below) gives the Fr\'echet Inception Distance (FID) calculated using a batch of 50 images from the training dataset and 15 of the generated images by each of the models. Note that the scores values given are for the sake of comparing the four methods and more samples may improve the FID score. We see that for the generative task, the the proposed DRFM architecture gives the lowest scores. The more commonly used U-Net model is at the third position after the NN model. For the denoising task, we see that the NN gives the best results, (although some of the images are inconsistently noisy), followed by U-Net and then DRFM. Note that for U-Net, while most images are denoised perfectly, some images are incompletely formed which leads to a lower FID score. Moreover, we see that all the FID scores are also consistent with the qualitative assessment made in Figures 2 and 3 (see updated version of the paper).

- Table with FID scores for generative task
| Model | Timesteps | FID Score|
| ----------- | ----------- |---------|
| DRFM | 100 | 453.87|
|U-Net | 100 | 463.28|
| NN | 100 | 457.21|
| RF | 100 | 470.12|

- Table with FID scores for denoising task
| Model | Timesteps | FID Score|
| ----------- | ----------- |---------|
| DRFM | 100 | 394.99|
|U-Net | 100 | 388.38|
| NN | 100 | 378.181|
| RF | 100 |450.23|

Reviewer comment (Theory):
I did not get the point of focusing on random Fourier-type feature models in Eq. (15). Following, the proof of the paper, it is possible to include more general activation functions. I did not find the motivation why we only consider this specific random feature model defined in Eq. (15) and Eq. (16). The authors may need to provide additional explanation of the choice of the random feature or empirical evidence of the advantage of this specific random feature model.

Author response:
1. In general, any activation function would suffice as long as the activation function is bounded as indicated in Rahimi and Recht (2007). 

2. We focus mainly on Fourier-type feature functions as they represent the Gaussian kernel when the components of matrix $\mathbf{W}$ are sampled from Gaussian distribution and bias vector $\mathbf{b}$ is sampled from the uniform distribution. Since the aim of diffusion models is to learn the mean (and covariance) of the reverse process whose transition kernels are Gaussian distributions, we choose this particular form. For more details on the relation between Fourier-type features and Gaussian kernel, we refer to Rahimi and Recht (2007).

Reviewer comment (Theory):
The theory of this paper seems to directly come from Chen et al. (2022) and Ranhimi & Recht (2008b; a). The authors may need to emphasize the difference between the current paper and the references, and the difficulty in the analysis in the current paper. In particular, the authors derived the generalization bounds via the approximation error of the random feature model, which, I think, lacks explanations in the proof in the Appendix.

Author response:
Thank you for the suggestions. We have added the following discussion to the references in the Introduction section. 

- Firstly, we show that our proposed architecture is a random feature model which helps us to obtain the results given in the paper. Also, we would like to highlight the fact that our model architecture is capable of incorporating time as an input with a separate set of weights associated with it. This is much different from the architecture proposed in Kawaguchi et al. (2018) where the authors only consider the input $\mathbf{x}$.

- Secondly, we would like to point out that although our final result is based on results from Chen et. al. (2022), we believe that the quantification of $\varepsilon_{\text{score}}$ is a significant contribution to understanding the number of parameters needed (with respect to the input dimension and number of timesteps) to obtain a small error. This quantification helps us to choose the appropriate number of model parameters before training. In particular, note that for appropriate choices of $T$ and $h$, the bounds in Chen et al. (2022) can be atmost reduced to $TV(p_{\theta}(0),q)\leq\mathcal{O}(\varepsilon+\varepsilon_{\text{score}})$. Our results show that we can also choose the parameters of the model so that $TV(p_{\theta}(0),q)\leq\mathcal{O}(\varepsilon)$.

- Also, we explicitly extend existing function approximation bounds to hold for $d-$ dimensional function. Although trivial, this result forms the basis for the main result as well which has not been used in any previous works.

- Lastly, as for the proof in the appendix, we try to be as brief as possible to avoid repetition of steps which can be found in the references. For example, the main steps for proving Theorem 3.3 can be found in Chen et al. (2022) and thus we omit the details. However, we provide statements and proofs of all the results corresponding to our proposed method (Equation (16) in Section 3.1, Lemma 3.1 and Lemma 3.2) which are key for the result in Theorem to hold.

Reviewer comment (Experiments):
Although there may not be a theory, for completeness, the authors may need to provide additional simulations for deeper neural networks with random features, showing the advantage of this architecture, and comparing it with more commonly used U-Net or other neural network models. The random feature models will reduce the computation complexity but I am not sure the performance is still comparable with conventional diffusion models.

Author response:
We have now added a few more results using a simple U-Net architecture with the number of trainable parameters similar to the DRFM model. For the generative task, the performance of DRFM and U-Net are comparable (see Figure 2 in the updated file). For the denoising task, both U-Net and NN perform marginally better, but neither of them has the interpretable properties of DRFM.

Reviewer comment (Writing):
It is a standard practice to introduce an abbreviation by providing its full form for the first time. For instance, DDPM appears in the paper first time without explaining the full name.

Author response:
Thank you for pointing out this typo. It is fixed.

Reviewer comment (Writing):
Between (5) and (6), there are typos in $q(x_{k-1}|x_k,x_0)=\mathcal{N}(\tilde{\mu}_t,\tilde{\beta}_t)$: you should specify $t$ and insert identity matrix for covariance. And the same issue for (6).

Author response:
Thank you for pointing out this typo. It is fixed.

Reviewer comment (Writing):
Please explain ''U-Nets not only preserve the dimension of input data, they also apply techniques such as downsampling using convolution which helps to learn the features if the input data.'' on page 4.

Author response:
The architecture of U-Net is designed in a way that it preserves the input dimension of the data i.e., the output and input dimension are the same. Additionally, U-Nets generally consist of a contracting (downsampling) and expanding (upsampling) path. The contracting path consists of sequences of convolutional and max-pooling layers that downsample the input images and extracts the layers.

Reviewer comment (Reproducibility):
In Algorithm 1, line 4, did you use uniformly sampling from $K$ time points to update the training parameters? For line 7, how do you minimize the loss $L$? Did you assume it will attain some global minimizer after training? Further explanations may be needed.

Author response:
- We used uniform sampling from 100 timesteps to update our parameters. This is consistent with the algorithm outlined in DDPM.
-We minimize the mean-squared error loss to update the weights using the ADAM optimizer.
-Note that training using random features is a convex method and thus we expect our proposed model to reach a global minimizer post-training.

Reviewer comment (Writing):
In Lemma 3.1, please explain the notion $\mathcal{F}_{\omega}$ and $\mathbf{\theta}_j^{(2)}$.

Author response:
We have updated the paper by rearranging some of the definitions which was initially given later. We denote $\boldsymbol{\theta}_j^{(2)}$ the $j^{th}$ row of the matrix $\boldsymbol{\theta}^{(2)}\in\mathbb{R}^{N\times d},\rho(\boldsymbol{\omega})$ the probability density function from which $\boldsymbol{\omega}$ is sampled, and 

$\mathcal{F}_{\boldsymbol{\omega}}$ be the set of all functions $\boldsymbol{f}(\mathbf{x})$ of the form:

$\sum\limits_{j=1}^N \boldsymbol{\alpha_j} \,\phi(\mathbf{x}^T\boldsymbol{\omega}_j)$

such that $||\boldsymbol{\alpha_j}||_{\infty}\leq \frac{C}{N}$ for all $j=1,\ldots N$.

Reviewer comment (Writing):
In Lemma 3.2, is $\rho(\omega)$ the density function of $\rho$?

Author response:
We have updated the paper by rearranging some of the definitions which was initially given later. We denote $\boldsymbol{\theta}_j^{(2)}$ the $j^{th}$ row of the matrix $\boldsymbol{\theta}^{(2)}\in\mathbb{R}^{N\times d},\rho(\boldsymbol{\omega})$ the probability density function from which $\boldsymbol{\omega}$ is sampled, and 

$\mathcal{F}_{\boldsymbol{\omega}}$ be the set of all functions $\boldsymbol{f}(\mathbf{x})$ of the form:

$\sum\limits_{j=1}^N \boldsymbol{\alpha_j} \,\phi(\mathbf{x}^T\boldsymbol{\omega}_j)$

such that $||\boldsymbol{\alpha_j}||_{\infty}\leq \frac{C}{N}$ for all $j=1,\ldots N$.

Reviewer comment (Theory):
In Theorem 3.3, you assume a bounded second moment for $q(x_0)$ but Chen et al. (2022) require $(2+\eta)$-th moment bound for some $\eta>0$. Is there any extra work to relax this assumption in your proof? Right now, there is a lack of explanation of the proof and how you applied Lemmas 3.1, 3.2, and A.2 to conclude the main theorem. Lemma 3.2 shows the approximation error of the random feature model but in the DRFM, how do you use (18) to finish the proof of Theorem 3.3 and ensure $|\theta^{(2)}_{ij}|$ are always uniformly bounded by some constant?

Author response:
-In the paper the authors specify, "For technical reasons, we need to assume that $q$ has a finite moment of order slightly but strictly bigger than 2, but our quantitative bounds will only depend on the second moment." Also, we used the proof from a different version of the paper which uses the second moment for the proof (as also mentioned by the authors). However, for consistent results, we have updated the paper to reflect the changes.

-Also, in Lemma 3.1 we show that the class of functions approximated by our proposed method is the same as the space of functions approximated by a random feature model. Further Lemma 3.2 gives the approximation bound for a $d-$dimensional function approximated by random features which will be the same for DRFM. The upper bound leads to the satisfaction of the third assumption of Lemma A.2 which leads to the stated conclusion in Theorem 3.3.

Reviewer comment (Evaluation):
Any explanation for why neural networks may fail in denoising images in Figure 3 sometimes but DRFM works better and is stable?

Author response:
Training neural networks is a non-convex problem while training DRFM is a convex problem so DRFM is more stable.

Reviewer comment (Reproducibility):
There should be more details of how you trained the models in the captions of Figures 2-7, e.g. which optimizers are used.

Author response:
Thank you for the suggestion. We have added more details on the training process both in the main text of the paper and captions of figures.