Paper: Representation-space diffusion models for generating periodic materials (ICLR-2024)

Reviewer comment (Novelty):
The technical contribution, while considerable, is not deep enough for a venue like ICLR. Although there is nothing wrong with combining existing methods, for this kind of contributions one expects a set of deeper insights or surprisingly better results than state of the art.

Author response:
We appreciate your recognition of the technical contributions we have made, even as we understand your perspective regarding the depth required for a venue like ICLR. We also agree that venue often demands a substantial improvement over the current state of the art. But In our research, we've demonstrated the feasibility of generating valid periodic structures with a method of diffusion in the representation space, a method which has never been tried before. Moreover, we've successfully managed to reconstruct these structures, transitioning them from the representation space to Cartesian space. Judging by the metrics of reconstruction and generation, we are confident about the competitiveness of our model's performance. We recognise that this is just a step in an ongoing journey of discovery and improvement in this field, but an important step towards opening a new paradigm of representation based generative model in the field of material discovery.

Reviewer comment (Writing):
The paper could have been better exposed. For example, it is not clear if the proposed concatenated representation is the idea of this paper or not. I assume it is, but have not seen the paper mention it explicitly. In general, mentioning an explicit list of contributions is recommended.

Author response:
We thank the author for this very important suggestion and acknowledge his recommendations by including an explicit list of our contributions in the Introduction section, with the heading Major contributions on Page 2.

Reviewer comment (Evaluation):
It is true that running density functional theory might be time consuming, but I think this is necessary for the evaluation. Obviously, the evaluation results based only on surrogate models cannot be judged clearly.

Author response:
The inclusion of density functional theory calculations will certainly provide a better platform for validation. However, as mentioned by the reviewer, this method is computationally very expensive and non-trivial to perform. More importantly, it detracts from our model's objectives, which, as previously mentioned, are geared towards pioneering a new paradigm of representation-based generative models in the field of material discovery.  
R2: We acknowledge the response by the reviewer and would like to humbly respond to this concern. The mentioned software PSI4 is not a suitable software for periodic systems, and DFT calculations for periodic crystals generally take more time than similarly sized molecular systems. We do not believe DFT evaluations on a small amount of structures within a limited computation budget would be meaningful or provide any additional insights beyond the existing metrics, especially as the prior work being compared against also do not have DFT. A more complete study of the structures with DFT would be more suitable for a subsequent study.

Reviewer comment (Novelty):
What is the motivation behind min-, max-pooling of the EAD representation? Is this known or the paper proposes it?

Author response:
The EAD descriptor is a local descriptor for each atom in the material. To make the overall descriptor for the material permutationally invariant, we pool over all atoms. This approach, which we propose in this work, has proven to be effective. It allows for efficient information retention without significant loss during reconstruction

Reviewer comment (Experiments):
Is Table 1 a fair evaluation? That means, do the competing methods in the table have a dedicated component for the reconstruction phase (like the method of Fung et al. which the paper relies on)?

Author response:
Table 1 measures the ability to reconstruct a material from its representation to the Cartesian positions. In the compared examples, these are latent representations which are obtained from a trainable model. In our case, our representation can be obtained directly from an analytical function. Nonetheless, the means of evaluation are identical, i.e. we are looking at the Cartesian positions of the reconstructed structure.  
R2: We acknowledge the suggestions by the reviewer by creating a detailed section A.4 highlighting the methods used in all respective works of Table 1.

Reviewer comment (Experiments):
Table 1: For generating this table, have the valid molecules first been separated for all methods? Or only for a few of them? If only for a few of them, which ones?

Author response:
The reviewer is correct in pointing at the separation of valid materials before the evaluation. As mentioned in our appendix section A.4 The Match rate is computed for materials which lie within the tolerance limits of ltol = fractional length tolerance, stol = site tolerance, atol = angle tolerance as per pymatgen's StructureMacher method. While the RMSE values are calculated over all successful matches only. Which is effectively calculating the fingerprint distance between a original structure and generated structure which have already passed the cut-off of match. This follows the identical procedure in previous works.  
R2: We acknowledge the suggestion and add the details in section A.5

Reviewer comment (Experiments):
In the sections of Introduction and Related Works, the generation methods are discussed in terms of representation-based generation and direct material generation. However, when the authors describe the baseline models in the Results section, the two categories are not explicitly mentioned, leading to ambiguity regarding the categorization of these models. It appears that the authors exclusively compared their methods with those of direct material generation. This raises questions about the extent to which the current representation embedding solely contributes to the improvement and to what extent the diffusion process is necessary for their proposed methodology.

Author response:
The reviewer correctly mentions the comparison of our model with direct material generation, as the current models have not implemented generative model in the representation space. And which brings the novelty to our approach. Nevertheless, we have included the work done by Uhrin (2021) [1] in section 2.1 of our paper, while the study does highlight about the possible use of generative models in the representation space but did not integrate these representations into a generative model, which somewhat limits their immediate applicability and hence no comparative analysis has been presented with this model.

Reviewer comment (Experiments):
I’m not familiar with the datasets the authors used, but I wonder how the test (or validation) data is divided from the training data. For example, the authors evaluate the PCA distribution analysis using the “training” data in Figure 5 according to the main text (page 7), which should be analyzed using the test data.

Author response:
Yes, the reviewer correctly observes the typing error on page 7 regarding figure 5 of our paper. All the three subfigures of figure 5 have been analysed on the testing dataset. We humbly acknowledge the mistake and have updated this in the submission.

Reviewer comment (Evaluation):
The authors mention “significant increase” and “significantly better” in the results section (page 7). Are these based on some statistical tests? If so, the authors should describe the details.

Author response:
Certainly, the significant increase in performance for the metrics mentioned in Table 1 are related to the Match rate (%) and RMSE (in the case of comparison for the MP dataset). The insightful suggestion on details for these metrics have been humbly acknowledged in the appendix section A.4 of the paper.

Reviewer comment (Presentation):
Figure 1’s terminologies are not consistent in the 4.1. The descriptions of Cell, Comp, and Structure would be better matched with those in the main text, at least in the caption of Figure 1. Besides, when the authors explain Section 4.1, referring to Figure 1 would be easier to understand.

Author response:
The insightful suggestion to improve Figure 1's caption for better understanding of the context has been acknowledged in the updated submission. The updated caption will indeed help the reviewers get better insight of the process.

Reviewer comment (Presentation):
At first, I thought that the hyperparameter of training was not described in the manuscript. Later, I learned it’s described in Appendix 3.4.4. Including a reference to Appendix 3.4.4 in the main text would help readers’ understanding.

Author response:
The suggestion of including the reference of Appendix with hyper-parameters information has been acknowledged. Moreover, the reference to Algo 2 in section 4.3 on Page 5 is indeed correct, as there we are mentioning about our iterative reconstruction process.

Reviewer comment (Reproducibility):
The source code is not available in the supplementary material. Sharing the code will contribute to future development in the field.

Author response:
Certainly, the source code for the model will benefit future development in the field, and we highly support this suggestion. However, aligning with the instructions of the initial submission, we have not yet included the link to our source code. We plan to make it publicly available as soon as the review process is finished.

Reviewer comment (Evaluation):
Let me confirm that the component distribution of generated and original samples in Figure 5 is computed from the same PCA loadings and these axes are directly comparable. I wonder why the PCA distribution of generated samples (Figure 5 left in particular) is more diverse than the original distribution.

Author response:
Yes, it is correct that the axes are directly comparable for the generated and the test samples. 

The reviewer is correct in noting the slightly higher diversity in the distribution of the generated structures. This is likely due to problems in the reconstruction step from representation to the Cartesian positions. As it is possible for the reconstruction to not fully converge (due to us choosing a fixed number of iterations), the partially reconstructed structures may not well represent the training distribution. We note that the number of these outliers is proportionally small, and can be easily screened out by removing structures with a high reconstruction loss.

Reviewer comment (Evaluation):
Could the authors explain more why the RMSE results are not improved much compared to those of the Match Rate?

Author response:
The reviewer correctly observes less improved RMSE results as compared to the Match Rate. As mentioned in our apendix section A.4 The Match rate is computed for materials which lie within the tolerance limits of ltol = fractional length tolerance, stol = site tolerance, atol = angle tolerance as per pymatgen's StructureMacher method. Meanwhile the RMSE values are calculated over all successful matches only. Which is effectively calculating the fingerprint distance between a original structure and generated structure which have already passed the cutt-off of match. Hence, even lower match rates can give certain structures which are closer in fingerprint distance, while higher match rate may give a higher RMSE value along. As can be seen in the case of CDVAE's C and MP RMSE columns.

Reviewer comment (Experiments):
It seems pretty clear this paper is far from finished. There's an idea, there's some early results, but this really just needs more work. It's impossible to tell at this stage if this idea works (it may, we just cannot tell).

Author response:
We disagree with this assessment as we have shown a complete end to end example of structure generation following this concept, and have provided the same evaluations and metrics used by all prior studies investigating structure generation. We also ask the reviewer to provide a more meaningful definition of if the idea "works." The problem of materials discovery (with generative modelling) is a challenging one, and it is difficult to expect that all problems associated with it can be resolved in a single study. What we have done here is shown the viability of generating valid periodic structures using a standard generative model, and reconstructing the generated structures from the representation space back to the Cartesian space. Based on the reconstruction and generation metrics, we believe we have been successful in this regard. We admit that there is still significant work which must be done in this area, but this requires a concerted effort in the community to develop better metrics, datasets, and validation with computational simulations.

Reviewer comment (Evaluation):
The evaluations are not very meaningful. The paper sets out to improve materials discovery through generative modeling. The goal of materials discovery is to find materials that are a) novel, b) thermodynamically stable, c) have interesting properties. The paper addresses only point a), and only to some extent. There isn't a single DFT calculation run to see if any of these materials would be stable. The word "stable" isn't even mentioned. That means it may be that all of these predicted structures could never exist in nature. Similarly, there's zero addressing of whether these (potentially unstable) materials have useful properties. Without addressing this, it's just impossible to assess if this is a good idea or not.

Author response:
We note that these evaluations have been used extensively in prior studies and allow us to provide a fair comparison between our method and these prior studies. There are inevitably limitations in these metrics , but it is not reasonable to discount a growing field and community in generative modelling for atomic structure for lack of better metrics.

Reviewer comment (Novelty):
The paper would benefit from discussing the relationship to Uhrin 2021, "Through the eyes of a descriptor: Constructing complete, invertible descriptions of atomic environments" https://arxiv.org/pdf/2104.09319.pdf -- which describes many of the core ideas discussed here.

Author response:
We acknowledge the suggestion by the reviewer in our updated submission and are thankful for the suggestion. Although, we would like to humbly suggest the following key areas where our work moves forward from Uhrin (2021) [1]

    - The paper does show the use of descriptors in representing structures and iteratively reconstructing back to structures, but has shown no instances of application to a generative model. As such it is impossible to evaluate if it is a successful concept or not in the context of generative modelling beyond simple speculation.
    - The authors propose a specific descriptor and iterative procedure for reconstruction which works in a similar concept, but our approach is valid for any differentiable descriptor, which goes beyond the specific descriptor proposed in the paper.

Reviewer comment (Reproducibility):
There lack many significant details in the presentation of obtaining coordinates from embedded atom density representations in Section 4.3. Authors are recommended to clarify the following details: (1) What is the formula of loss function $L$? (2) What are the gradient descent algorithm, step size, max_initializations, max_hops, and max_iterations used in experiments? (3) What are general range of the loss function values when optimization ends? It would be better to report the mean final loss function values averaged over all generated materials.

Author response:
We find a simple $L1$ Loss is sufficient for reconstruction, and this loss is used throughout the work.

   We are using a basin hopping algorithm with a local gradient descent optimization. The optimizer used is Adam.

   - The number of basin hopping trials is 6
   - The number of basin hops is 7
   - The basin hopping step size is a random number in the range \[-1, 1\]
   - The max number of iterations in gradient descent is 300

   The loss function values for the iterative reconstruction of our generated structures is usually around \[0.005, 0.05\] and the reconstruction for errors for valid structures is \( $\leq 0.01$ \). The details have now been added in appendix A.3.5.

Reviewer comment (Presentation):
Table 1 needs improvements. The column "Dataset" should be removed. Italic font type should not be used to the "P" under "RMSE" to keep consistency.

Author response:
The Table 1 has been updated as per the helpful recommendations of the reviewer.

Reviewer comment (Experiments):
Property optimization is important in real-world problems because real-world applications usually require to generate materials with some target chemical properties. However, it remains unknown how to do property optimization with the proposed model. Authors are encouraged to discuss how to property optimization with StructRepDiff, and explain whether the failure of achieving invertible mapping between material representations and atom coordinates has negative impacts on property optimization.

Author response:
The reviewer is absolutely correct about the importance of property optimisation in this domain of work. However, in the scope of this current work, we aimed to address the implementation and working of our overall method in non-conditionally generating periodic structures which the model is able to do as per our competitive performance from Table 2. We emphasize primarily on the viability of structure generation in our proposed representation space, which is completely novel compared to other current approaches. The inclusion of conditional generation will invariably also require the use of density functional theory calculations for validation, which is a computationally expensive task and non-trivial to do, and detracts from the primary focus of this paper. It is our opinion that evaluating the property prediction with another surrogate model, such as a GNN, is not sufficiently accurate to provide used metrics, nor can they be compared to prior studies in the literature which use different surrogate values. This remains an important problem which needs to be addressed by the community.

Property optimization can be performed in our framework by incorporating conditional generation into our diffusion model, without changing any other components of the process. Materials are then generated with the diffusion model conditional on a target property value, and reconstructed to the atomic structure as before. In the future, we would like to evaluate the effectiveness of this approach by performing conditional generation on a target band gap or formation energy, and performing DFT calculations on the structure with the identical settings as the training data from Materials Project. With respect to the impact of the "failure of achieving invertible mapping between material representations and atom coordinates" on property optimization, even though these representations do not have a closed-form solution for inverting to cartesian positions, our iterative reconstruction algorithm is nevertheless able to reconstruct structures very accurately, as measured by the reconstruction metrics in Table 1. Therefore, it is not anticipated to be a critical obstacle in performing property optimization with this approach, especially if the reconstruct is able to recover the positions exactly or near-exactly.

Reviewer comment (Novelty):
In addition to embedded atom density representations, are there any other ways for 3D structure representations? Authors are encouraged to give some alternative examples for embedded atom density representations.

Author response:
Yes, apart from embedded atom density representation, there exists a host of other representations which can be chosen, such as the weighted version of the classic atom centered symmetry function. As we note in this recent review (Chem. Rev. 2021, 121, 16, 9759–9815), there are many other potential options to choose which we do not exhaustively show here. We mainly chose the embedded atom density representation due to having slightly better reconstruction performance over the atom centered symmetry function in our experiments, and a faster evaluation speed. However, this approach is notable for being applicable to any differentiable representation.