Paper: Manifold Diffusion Fields (ICLR-2024)

Reviewer comment (Novelty):
The main concern, as stated in the Weakness section, is the comparison of this work with [1]. To me, the difference of this work with [1] is only the different ways of representing points on the Riemannian manifold, where [1] directly use some coordinate since the manifolds we consider are always embedded, and this work uses the leading eigen-function of the LBO to approximate the points. It remains not very clear why and how LBO sign-functions outperforms a plain coordinate approach, or a local chart approach (since for every point on the manifold we have a local coordinate chart). Calculating eigen-functions of LBO potentially also consumes time. A more elaborative discussion of different approaches to do the embedding seems necessary.

Author response:
* We thank the reviewer for raising this point, which deserves clarification. In practical terms, Tab. 1-2-3-4 in the paper show a comparison between MDF (which uses the eigen-functions of the LBO) and DPF (which uses a plain coordinate approach). Our results, show that MDF consistently outperforms DPF (ref. Tab. 1-2-3-4). We attribute this to the fact that representing points through the eigen-functions of the LBO explicitly provides the model with information about the intrinsic geometry of the manifold. Furthermore, in Sect. A.6.2  in the appendix we show that DPF completely breaks under rigid (Fig. 12) or isometric (Fig. 13) transformations of the manifold, while MDF is invariant to these transformations since it uses an intrinsic coordinate system. 
    
   * In addition, note that although one needs to compute the eigen-functions of the LBO, this computation: (i) Is very efficient, since Laplacians are sparse matrices and efficient eigen-system solvers can be used. (ii) Only needs to be performed once before training, caching eigen-functions for usage during training. In practice, the cost of computing the LBO eigen-functions is negligible compared to training the diffusion model.


    * We have included a new section in the Appendix (Sect. A.3) discussing at length the different ways to compute embeddings.

Reviewer comment (Experiments):
The authors choose the top k LBO eigen-functions to do the embedding. Is there a guidance on how to determine k?

Author response:
* Conceptually speaking, $k$ is usually dependent the specific manifold under consideration and its intrinsic geometry. Manifolds with high frequency geometry typically require larger $k$. In Figure 7, we illustrate the impact of varying $k$ on performance metrics (COV and MMD). Notably, we observe satisfactory performance even for small values of $k$. In MDF, we choose $k$ experimentally but one could also choose $k$ based on the overall variance explained of the manifold (similar to PCA).

Reviewer comment (Writing):
Regarding Algorithm 2: First, what’s $M_q$, which seems to exist in [1] but not here? Second, why do we need a random subset $C_{t}$? I thought we just need the same context and query sets as in Algorithm 1.

Author response:
* In [1], $M_q$,    denotes the stacked coordinates of query pairs. In our context, we use $\varphi(\mathbf{X}_{q})$ to denote the stacked eigen-function corresponding to query pairs. Note that during inference, one can select a random subset of query pairs as context. In particular, it can be equivalent to the one used during training but it does not need to be.

Reviewer comment (Experiments):
Regarding the numerical experiments, I’m particularly interested in the ERA5 dataset, which seems to be the only natural instance of manifold-supported distributions. It would be interesting to see how the proposed algorithm performs in terms of COV and MMD for this dataset, comparing to other algorithms as DPF and GASP.

Author response:
* Tab 5 provides further insights on the ERA5 dataset, demonstrating that MDF outperforms GASP in terms of both COV and MMD.

Reviewer comment (Theory):
While the authors theoretically discuss the possibility to use Laplace-Beltrami Operators as the source for a functional basis on the manifolds, this is only explored as graph Laplacians, which are a discrete approximation of manifolds. Understandably, in many scenarios it is not possible to analytically solve this problem, but it is unclear how challenging it is to adapt the ideas presented here to scenarios where it is possible (or how beneficial that is).

Author response:
* We thank the reviewer for raising this point which we completely agree deserves clarification. We have clarified in the contributions that the bulk of our experimental evaluation covers graphs, meshes and pointclouds (ref. Tab 6 and Fig. 8) as discrete approximations of manifolds, which is important in most practical settings.

   * We would also like to point the reviewer to Tab 5. where we show results of MDF on the ERA5 dataset, which consists of temperatures on on a parametric manifold (eg. earth represented as a sphere). In this case, the analytical solution for the eigenfunctions of the LBO are given by spherical harmonics (second paragraph of page 7). Our results show that using spherical harmonics MDF outperforms previous approaches to learn functions on the sphere, like GASP [2]. We have clarified this in the paper.

   * A final clarification point is that this approach has a theoretical connection to the analytical LBO, as the eigenvectors and eigenvalues of the graph Laplacian converge to the eigenfunctions and eigenvalues of the analytical LBO as the number of vertices/nodes tends to infinity [1].

    [1] Convergence of Laplacian eigenmaps. in Adv. Neur. In.: Proceedings of the 2006 Conference, vol. 19, p. 129. The MIT Press.

    [2] Dupont, Emilien, Yee Whye Teh, and Arnaud Doucet. "Generative models as distributions of functions." arXiv preprint arXiv:2102.04776 (2021).

Reviewer comment (Evaluation):
A simple fix to the above issue, in my opinion, is to claim what is being extensively evaluated experimentally: graphs/meshes as an approximation of a manifold.

Author response:
* We thank the reviewer for raising this point which we completely agree deserves clarification. We have clarified in the contributions that the bulk of our experimental evaluation covers graphs, meshes and pointclouds (ref. Tab 6 and Fig. 8) as discrete approximations of manifolds, which is important in most practical settings.

   * We would also like to point the reviewer to Tab 5. where we show results of MDF on the ERA5 dataset, which consists of temperatures on on a parametric manifold (eg. earth represented as a sphere). In this case, the analytical solution for the eigenfunctions of the LBO are given by spherical harmonics (second paragraph of page 7). Our results show that using spherical harmonics MDF outperforms previous approaches to learn functions on the sphere, like GASP [2]. We have clarified this in the paper.

   * A final clarification point is that this approach has a theoretical connection to the analytical LBO, as the eigenvectors and eigenvalues of the graph Laplacian converge to the eigenfunctions and eigenvalues of the analytical LBO as the number of vertices/nodes tends to infinity [1].

    [1] Convergence of Laplacian eigenmaps. in Adv. Neur. In.: Proceedings of the 2006 Conference, vol. 19, p. 129. The MIT Press.

    [2] Dupont, Emilien, Yee Whye Teh, and Arnaud Doucet. "Generative models as distributions of functions." arXiv preprint arXiv:2102.04776 (2021).

Reviewer comment (Evaluation):
Is the model's robustness to rigid transformations on the manifold a sufficient explanation for its performance for molecule generation? Is it possible that it is also robust to other types of transformations which are useful in solving this task?

Author response:
* We thank the reviewer for raising this very interesting point. It is true that the robustness to rigid transformations can be helpful for conformer generation, we do believe that this is not the only explanation for the performance of MDF. Our current hypothesis is that the eigen-functions of the LBO provide a function basis that helps the model find representations that can be aligned across molecules and in turn this boosts performance. Note that generating conformers can be thought of in a way as a non-rigid transformation of a base molecular structure and MDF appears to be quite good at discovering this set of non-linear symmetries. We will fully uncover and understanding of this behaviour in future work.

Reviewer comment (Writing):
Sec. 3.3: were $<$ and $>$ the intended symbols when writing $f$ as a linear combination of the basis?

Author response:
* Thanks for pointing out this typo, we have fixed it in the new version of the paper.

Reviewer comment (Writing):
Sec. 3.3: "making MDF strictly operate the manifold" => "on the"?

Author response:
* We fixed the typo in the new version of the paper.

Reviewer comment (Presentation):
p. 7: "available at [URL]" I think a footnote here is less disruptive for reading.

Author response:
* We have moved this into the appendix to keep the paper at 9 pages.