Paper: Scalable Diffusion for Materials Generation (ICLR-2024)

Reviewer comment (Experiments):
The authors claim that the proposed representation is capable of capturing any crystal structure, as well as being "scalable and flexible." While the completeness of the representation is clear and well-articulated, there appears to be a lack of detailed discussion regarding its redundancy or compactness. It would be beneficial if the authors could provide further explanations or experimental insights that illuminate how compact or redundant this representation might be in practice.

Author response:
> How compact or redundant is the UniMat representation.

The answer to this question depends on the types of materials we are modeling. For small crystals with only a few atoms in a unit cell, the UniMat representation is not compact and in fact quite redundant (e.g., 99% atoms are null atoms). However, this representation allows handling of complex structures with up to thousands of atoms. We have included this discussion in the conclusion section.

Reviewer comment (Theory):
In the section discussing conditional generation, the use of conditioning variables seems somewhat unclear. Specifically, on Page 4 under "Conditioned Diffusion with UniMat," the conditioning variables are directly concatenated with the noisy material along the last dimension. However, this approach raises questions as there appears to be a disparity in the feature spaces of the conditioning variables and the noisy material. For reference, in image conditional generation, cross-attention modules are commonly used to align the input condition (like text) with the image space effectively. Without the incorporation of a similar module in this work, the mechanism by which the conditioning variables guide the generation process remains unclear.

Author response:
> Align input conditions and generations effectively.

UniMat in fact uses self-attention layers to attend over the concatenated conditioning variable and noising materials. The skip connections in a UNet can been seen as a form of cross-attention (conditioning variables are passed to attention layer through the previous skip connection). This approach to conditioning is common in spatial and temporal super-resolution diffusion models [1] [2].

[1] Video Diffusion Models. Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, David J. Fleet.
[2] Imagen Video: High Definition Video Generation with Diffusion Models. Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, Tim Salimans.

Reviewer comment (Evaluation):
Regarding the method's performance, it is noted that the proposed method does not attain 100% validity on larger datasets like MP20, in contrast to simpler approaches such as CDVAE. It would enhance the paper if the authors could delve deeper into this issue, offering more insights or explanations. Including potential solutions or future directions in addressing this limitation in the limitation section would also be quite valuable.

Author response:
> Low validity on MP20.

We have found MP20 to be a more difficult dataset where scaling up the model helps with validity (Figure 4). The validity metric from CDVAE is only a proxy metric that does not fully capture the generation quality. For instance, 100% validity could be due to overfitting to the training set and mostly generating training compositions. Hence, converge metrics and property statistics should be take into account, for which UniMat has better performance. We have included this limitation in the conclusion and limitation section.

Reviewer comment (Presentation):
In Figure 1, it might be beneficial to improve clarity by adding more descriptions or labels to the element table. It actually takes me some time to decipher that it represents an element table. Providing a more explicit explanation regarding the motivation and benefits of utilizing the element table for representation would also be advantageous. It seems that one evident benefit is the shared similarities in properties among neighboring elements in the table, potentially providing a useful prior for generation. Incorporating such observations and expanding on the motivations in the Introduction section would also be helpful.

Author response:
> Clarity of Figure 1 and motivation of periodic table representation.

We have updated Figure 1 to indicate each cells are entries in the periodic table. We have added additional motivations for the periodic table representation in the Introduction (second paragraph on Page 2).

Reviewer comment (Writing):
In Section 2.2, it could be helpful to include references to DDPM and extend the discussion slightly to incorporate considerations of other diffusion models, elaborating on why DDPM was the chosen approach. The discussion doesn’t need to be overly complicated: a straightforward explanation, such as the effective performance of DDPM in the authors’ use case, accompanied by some contextual background, would enhance the readers' understanding.

Author response:
> The choice of DDPM.

Since the denoising process of a DDPM nicely corresponds to the process of gradually moving atoms in space until they reach their target location, we choose DDPM over other diffusion models (e.g., denoising score matching). We included this intuition on Page 4.

Reviewer comment (Theory):
In the section "Conditioned Diffusion with UniMat" on Page 4, the statement "While the unconditional ... training distribution" could be refined for precision and accuracy. It might be more accurate to state that DDPM primarily learns the score function rather than directly learning the training distribution, making it challenging to quantify the extent of overlap between the learned and training distributions.

Author response:
> In the section "Conditioned Diffusion with UniMat" on Page 4, the statement "While the unconditional ... training distribution" could be refined for precision and accuracy.

Thank you for catching this detail. We have updated the text on Page 4.

Reviewer comment (Presentation):
The section "Drawbacks of Learning Based Evaluations" in Section 2.3 is quite motivating. However, it might be more seamlessly integrated by briefly mentioning its main points in the Introduction. This could help prepare the reader for the detailed discussion that follows in Section 2.3.

Author response:
> Mentioning drawbacks in learning based evaluations in the Introduction.

Thank you for the suggestion. We have updated the Intro section (second to last paragraph on Page 2).

Reviewer comment (Novelty):
There is no innovation in the diffusion model and the AI part. This paper just uses the standard diffusion model, and the conditional diffusion model to generate crystal structures.

Author response:
Thank you for the feedback. Please let us know if our response below addresses your concerns.

> Lack of innovation.

We disagree that this work has “no innovation in the diffusion model and the AI part”. Despite the advances in generative models, their adoption to material science surfaces many unanswered questions. For instance, whether crystals should be represented using voxel images, pointclouds, phase fields, or graphs is unclear. How to jointly model discrete atoms and continuous locations has not been previously addressed. How to properly evaluate generated materials has been largely unattended. Whether generative models can scale to large material datasets has not been considered. Our work provides the first set of answers to many of these questions.

Reviewer comment (Evaluation):
In Sec. 2.3, evaluating the generated materials using energy, is purely material science and has nothing related to AI. AI Researchers probably cannot evaluate the correctness and novelty of Sec. 2.3.

Author response:
We disagree that “evaluating generated materials using energy is purely material science and has nothing related to AI.”  How to properly evaluate generations lies at the core of generative models, where many metrics such as the Fréchet inception distance has been studied by the core AI community. Instead of leaving evaluation to material scientists, we believe it is more constructive for AI researchers to also understand and engage in application-specific evaluations of generative models.

Reviewer comment (Novelty):
For the AI community, we do not learn any novel AI knowledge from this paper.

Author response:
We also disagree that “we do not learn any novel AI knowledge from this paper”. In addition to the answers this work provides to crystal representations, joint modeling of discrete atoms and continuous locations, scaling up, and proper evaluations, this work illustrates diffusion models’ capability in modeling relationships other than spatial and temporal in images and videos (for which diffusion models were invented), opening up wider applications of diffusion models.

Reviewer comment (Evaluation):
The utilized benchmarks seem to be saturated with values close to 100% performance. The performance gain is marginal and therefore could be a random improvement. Also, in some of the cases, the previous work has already achieved 100%, so there is no room for improvement.

Author response:
> Existing benchmark saturated.

This is indeed the case. We have updated the text to reflect this (the last sentence of the first paragraph on Page 7).  Saturation of existing benchmarks and unreliability of learning based evaluations (elaborated in Section 2.3) were indeed what motivated us to propose three metrics based on DFT calculations in Section 2.3. We found that despite UniMat and CDVAE have close performance in learning based evaluations, UniMat performs drastically better than CDVAE under DFT calculations (i.e., what considered to be near ground truth evaluation in material science).

Reviewer comment (Novelty):
There is another work that uses diffusion models for the same task on the same datasets [a]. Although [a] uses diffusion models in a different way compared to this work, it has similar or better performance in some cases.

Author response:
> Related work on diffusion model for materials.

Thank you for pointing out the related work. We have updated Table 1 with comparison to this work and the related work section. We note that out of 19 metrics being reported, UniMat performs better than DP-CDVAE on 16 metrics. We hypothesize that DP-CDVAE uses a separate VAE to predict lattice parameters and number of atoms, which limits its flexible in jointly modeling of lattice parameters, number of atoms, and atom locations.

Reviewer comment (Evaluation):
Since ICLR is an ML conference, the paper would benefit from explaining the different evaluation criteria and their importance in the material generation task. E.g. what are the property statistics exactly and do they have higher importance compared to validity?

Author response:
> Relative importance of evaluation metrics.

From our experience working with material scientists, the metrics in Table 1 can only serve as a proxy as opposed to reliable evaluation under the context of materials discovery, due to reasons listed in Section 2.3 (first paragraph). The property statistics are distributional distances (e.g., earth mover's distance) between property distributions (e.g., distribution of number of elements) in generated materials versus test materials. We have elaborated this more in the first paragraph of Section 3.1. If the property is predicted by another model (e.g., formation energy), the reliability of such evaluation is dependent on the quality of the other model, making the metric less reliable (compared to properties that don’t require prediction from another model, such as the number of elements). Not that property statistics are also proxy metrics, and the most reliable metrics should be DFT based.

Reviewer comment (Experiments):
The paper could be contrasted and compared against [a].

Author response:
> Related work on diffusion model for materials.

Thank you for pointing out the related work. We have updated Table 1 with comparison to this work and the related work section. We note that out of 19 metrics being reported, UniMat performs better than DP-CDVAE on 16 metrics. We hypothesize that DP-CDVAE uses a separate VAE to predict lattice parameters and number of atoms, which limits its flexible in jointly modeling of lattice parameters, number of atoms, and atom locations.