Paper: Stoichiometry Representation Learning with Polymorphic Crystal Structures (ICLR-2024)

Reviewer comment (Novelty):
The novelty is limited, due to the fact that the proposed pipeline is a combination of previous technics.

Author response:
Thank you for recognizing that our work is pioneering work for crystal property prediction tasks without structural information! We are more than willing to clarify the reviewer’s concerns one by one.

**W1. Limited Novelty.**

As the reviewer pointed out, this paper is the first work that utilizes self-supervised learning to learn the representation of stoichiometry for various downstream tasks. While techniques in our paper are highly inspired by previous works, we believe that our contributions come from 1) identifying a novel and important problem, 2) suggesting a suitable method for the problem, and 3) providing insightful results that can progress the field with the method.

**Novelty and Importance of the Problem.**
Recently, the majority of machine learning (ML) models in materials science are built on top of the crystal structure that are computed based Density functional Theory (DFT). However, in real-world wet lab experiments, it is common situation where we only have compositional information (i.e., stoichiometry) without structural information as also mentioned by [reviewer rRHK](https://openreview.net/forum?id=yuGqe96t9O&noteId=B4csbUBOsR), due to the atomic-level rearrangement of the materials during the materials synthesis process (We have provided a more detailed explanation on this situation by addressing [W3](https://openreview.net/forum?id=yuGqe96t9O&noteId=efaOs4V5j0) in the following) [1, 2]. This missing structural information complicates the application of such structure-based ML models in the real-world materials discovery process. Therefore, one promising avenue for enhancing the practicality of ML in materials discovery lies in the prediction of materials property without structural information, i.e. relying solely on stoichiometry. However, as the reviewer mentioned in [W3](https://openreview.net/forum?id=yuGqe96t9O&noteId=efaOs4V5j0), it is also widely known that crystal structure heavily determines the properties of the materials. 


**Suitability of the Method.**
As a solution, we proposed PolySRL, which distills the knowledge of DFT-calculated structure into the stoichiometry encoders during the pre-training stage, and fine-tune or evaluate the model solely on stoichiometry. One straightforward approach would be contrasting the object with its 3D representation, which has shown promising results in organic molecules [3, 4]. However, our extensive experiments reveal that these methods do not adequately capture the complexities of inorganic materials, such as polymorphism. To this end, we propose to learn a probabilistic representation that can learn not only the appropriate representation of stoichiometry but uncertainty induced by polymorphism.

**Insightful Results.** Thanks to the suitability of methods for the task, we could make several insightful observations from experimental results. Briefly, uncertainty provided by models contains the information of stoichiometry regarding 1) a number of polymorphic structures, 2) computational challenges in calculating the accurate structure, and 3) similarity between polymorphic structures, all of which are common interest in materials science.

In conclusion, we fully agree with the reviewer that innovations in algorithms are crucial for the field's advancement. Concurrently, we also argue that a comprehensive investigation beyond novel algorithms, such as providing novel problems and insightful results, is also essential for advancing the field. Hence, we believe that PolySRL has contributed to the field's development by introducing problems that are more relevant to real-world applications, as well as by delivering meaningful results that have been long common interest in materials science.

[1] Predicting the band gaps of inorganic solids by machine learning, The journal of physical chemistry letters 2018.

[2] Predicting materials properties without crystal structure: Deep representation learning from stoichiometry, Nature communications 2020.

[3] 3d infomax improves gnns for molecular property prediction, ICML 2022.

[4] Pre-training molecular graph representation with 3d geometry, ICLR 2023.

Reviewer comment (Theory):
The global idea of predicting properties based on crystal compositional information is a little bit tricky, due to that crystal properties are largely determined by the structure. As you mentioned in your figure 1, diamond and graphite have totally different properties.

Author response:
**W3. The global idea of predicting properties solely based on compositional information is tricky.**

First of all, we appreciate the reviewer for raising an important discussion point! We fully understand the reviewer's comment that this topic is tricky, since most machine learning methods rely on structural data of crystalline materials. To further clarify the paper's contributions, we have added this topic to the discussion section in **Appendix H**.

As the reviewer mentioned, crystal property is highly determined by the structural information of the crystal. Most of the machine learning approaches for crystal property prediction utilize crystal structure information as an input, which is obtained through Density Functional Theory (DFT) calculations. However, most of the real-world wet lab experimental scenarios lack suitable structural information on the crystal due to the uncertainty of atomic arrangements [1, 2].

More specifically, during the synthesis process of materials, atomic-level rearrangement of the material occurs through the mixing of raw materials, heat treatment, and solvent reactions. Therefore, even if the crystal structure of the raw material is known, the synthesized material may have a new crystal structure due to thermodynamic uncertainty introduced by the synthesis operation, making it impossible to determine the crystal structure precisely.

Various chemical analysis techniques, such as X-ray diffraction (XRD) [3], have been developed to identify the crystal structure of synthesized materials, but due to cost and limitations in analytical accuracy, the crystal structure of synthesized materials is not typically the focus of analysis in actual chemical experiments.

In conclusion, a lack of knowledge about the crystal structure is a typical scenario in real-world wet laboratory experiments. Consequently, we argue that predicting properties using only compositional information aligns more closely with practical applications in the real-world material discovery process.

[1] Predicting the band gaps of inorganic solids by machine learning, The journal of physical chemistry letters 2018.

[2] Predicting materials properties without crystal structure: Deep representation learning from stoichiometry, Nature communications 2020.

[3] X-ray diffraction (XRD) techniques for materials characterization, Materials characterization using nondestructive evaluation (NDE) methods 2016.

**Final Remark.**

Again, thank you for recognizing our work is pioneering work for crystal property prediction tasks without structural information! We apologize for any confusion or ambiguity present in the initial submission. We hope our response has clarified the contents of the paper, and we are more than willing to engage in further discussion regarding any aspects that remain unclear.

Reviewer comment (Writing):
The keyword "stoichiometry" was not explained but the authors likely meant the chemical composition of a given material, more exactly ratios of weights of involved elements.

Author response:
**W1. Stoichiometry was not explained.**

We thank the reviewer for the suggestion that can definitely improve the presentation of our paper! Although we have provided the definition of stoichiometry in the footnote of the first page, we fully agree with the reviewer that its placement may not have been prominent. Therefore, we moved the definition of stoichiometry to the main introduction paragraph (colored in blue)!

Reviewer comment (Novelty):
The question “Is it possible for stoichiometry-based models to also capture the structural information of crystals?” has the trivial answer "no". Both diamond and graphite in Fig. 1c only a few lines before the question consist of pure carbon but have different crystal structures and vastly different properties, which is known even to school students.

Author response:
**W2. Possibility of stoichiometry-based models to capture the structure information of crystals.**

Thank you for pointing out our research question! As mentioned by the reviewer and depicted in Fig 1 (c) in the main manuscript, it is obvious that stoichiometry itself cannot capture the information about the structural information. However, thanks to the universal approximation theorem [1], we believe that deep neural networks possess the capacity to learn a function that correlates stoichiometry with its corresponding structural information.   Furthermore, a recent application of generative models to deduce stable crystal structures from given stoichiometries [2] provides empirical support that models based on stoichiometry can indeed infer the structural details of crystals.

[1] Multilayer feedforward networks are universal approximators, Neural networks 1989

[2] Crystal Structure Prediction by Joint Equivariant Diffusion, NeurIPS 2023

Reviewer comment (Writing):
The word "problem" appears once (in the quote below), though writing a rigorous problem statement might have helped the authors to understand the unresolved challenges.

Author response:
**W4. Problem Description.**

First of all, we would like to clarify our research problem statement. In materials science, it is a common situation where we cannot observe the exact structural information of crystalline materials during materials synthesis, since the atomic-level rearrangement of the material occurs through mixing of raw materials, heat treatment, and solvent reactions. However, the majority of machine learning models in materials science are built on top of the structural information, restricting their applicability in the real-world material discovery process. Therefore, in this paper, we aim to build a machine learning model that can predict the properties of materials given only stoichiometry.

Reviewer comment (Novelty):
If the problem is to find a complete invariant description of a periodic crystal, crystallographers solved this problem nearly 100 years ago by using Niggli's reduced cell of a lattice and then recording all atoms in so-called standard settings, as described in the book "TYPIX standardized data and crystal chemical characterization of inorganic structure types" by Parthé et al, which applies to all periodic crystals, not only inorganic.

Author response:
**W4-1. Find a complete invariance description of a periodic crystal?**

We agree with the reviewer that the challenge of identifying an invariant description of a periodic crystal was addressed almost a century ago, and also recognize that inductive biases related to invariances in the architectural design of deep neural networks for crystalline materials [1]. However, in this paper, we focus on learning the probabilistic representation of stoichiometry where the crystal structure remains unidentified, as opposed to seeking a comprehensive invariant description of a periodic crystal.

[1] Periodic graph transformers for crystal material property prediction, NeurIPS 2022

Reviewer comment (Novelty):
If the problem was to determine stoichiometry only from a crystal structure, this problem was resolved by the Crystal Isometry Principle, which states that any real periodic crystal (with all atomic types, hence stoichiometry) is uniquely determined (without any uncertainty) by the geometric structure of atoms (without chemical elements).

Author response:
**W4-2. Determining stoichiometry only from a crystal structure?**

We concur with the reviewer that, in theory, stoichiometry is uniquely determined by the geometric arrangement of atoms, as also described in our response to [reviewer rRHK](https://openreview.net/forum?id=yuGqe96t9O&noteId=D5aw44Ub3h).
However, in this paper, our objective is not to determine stoichiometry itself, but to learn the encoder that establishes a representation of stoichiometry where the crystal structure is not known.

Reviewer comment (Writing):
I would encourage the authors to improve the clarity of the last part of the paper, namely Section 5. While the quality is not poor, I did notice a decrease in the quality and clarity of this section with respect to the methods part of the paper, which I found very clear.

Author response:
We greatly appreciate the reviewer for pointing out the unclarified parts in our paper, as this will undoubtedly enhance the presentation of our research. In response to the reviewer's feedback, we have implemented the following revisions, which are highlighted in blue in the main manuscript:

- We have corrected the grammatical errors and typographical mistakes in the first paragraph of section 5.1
- We have increased the size of Figure 3 on Page 8
- We have increased the font size of Figure 4 on Page 9
- We have increased the size of Figure 5 on Page 9
- We have included more details on data splits in the third paragraph of Section 5.1

Additionally, we will ensure a thorough review of our paper to correct any grammatical errors throughout the document.

Reviewer comment (Writing):
The first paragraph of 5.1 contains a few typos or grammar mistakes.

Author response:
We greatly appreciate the reviewer for pointing out the unclarified parts in our paper, as this will undoubtedly enhance the presentation of our research. In response to the reviewer's feedback, we have implemented the following revisions, which are highlighted in blue in the main manuscript:

- We have corrected the grammatical errors and typographical mistakes in the first paragraph of section 5.1
- We have increased the size of Figure 3 on Page 8
- We have increased the font size of Figure 4 on Page 9
- We have increased the size of Figure 5 on Page 9
- We have included more details on data splits in the third paragraph of Section 5.1

Additionally, we will ensure a thorough review of our paper to correct any grammatical errors throughout the document.

Reviewer comment (Presentation):
The size of the figures in Section 5 is too small.

Author response:
We greatly appreciate the reviewer for pointing out the unclarified parts in our paper, as this will undoubtedly enhance the presentation of our research. In response to the reviewer's feedback, we have implemented the following revisions, which are highlighted in blue in the main manuscript:

- We have corrected the grammatical errors and typographical mistakes in the first paragraph of section 5.1
- We have increased the size of Figure 3 on Page 8
- We have increased the font size of Figure 4 on Page 9
- We have increased the size of Figure 5 on Page 9
- We have included more details on data splits in the third paragraph of Section 5.1

Additionally, we will ensure a thorough review of our paper to correct any grammatical errors throughout the document.

Reviewer comment (Writing):
An exception to the clarity in the methods section is Section 4.3, where the notation is a bit confusing. For instance, $\mathcal{P}^b$ is used here for the first time to refer to what I believe is the structural graph, previously referred to as $\mathcal{G}^b$.

Author response:
First of all, we apologize for any confusion caused by the use of the terms $\mathcal{P}^a$ and $m$. 
In fact, $\mathcal{P}^a$ is intended to represent the set of polymorphic crystal structures associated with the stoichiometry graph $\mathcal{G}^a$, as indicated in line 10 of Section 3.2. However, we fully agree with the reviewers that this notation might lead to ambiguity since superscript $b$ is consistently used for the structural graph. Therefore, to clarify our notation, we have decided to change the term $\mathcal{P}^a$  into $\mathcal{P}^{\mathcal{G}^a}$.
Moreover, we apologize for missing the precise definition of $m$, which is the indicator function of value 1 if $\mathcal{P}^{\mathcal{G}^a}$ is the set of polymorphic structures corresponding to $\mathcal{G}^a$ and 0 otherwise. We highlighted the modified term in blue color in the main manuscript.

Reviewer comment (Writing):
In Section 4.3, there is a question about whether the superscript should be $b$ instead of $a$ because it refers to the structural graph. The superscript of $z_p$ is $b$, which adds to the confusion.

Author response:
First of all, we apologize for any confusion caused by the use of the terms $\mathcal{P}^a$ and $m$. 
In fact, $\mathcal{P}^a$ is intended to represent the set of polymorphic crystal structures associated with the stoichiometry graph $\mathcal{G}^a$, as indicated in line 10 of Section 3.2. However, we fully agree with the reviewers that this notation might lead to ambiguity since superscript $b$ is consistently used for the structural graph. Therefore, to clarify our notation, we have decided to change the term $\mathcal{P}^a$  into $\mathcal{P}^{\mathcal{G}^a}$.
Moreover, we apologize for missing the precise definition of $m$, which is the indicator function of value 1 if $\mathcal{P}^{\mathcal{G}^a}$ is the set of polymorphic structures corresponding to $\mathcal{G}^a$ and 0 otherwise. We highlighted the modified term in blue color in the main manuscript.

Reviewer comment (Writing):
In Section 4.3, $m$ is used in the conditional probability without being defined.

Author response:
First of all, we apologize for any confusion caused by the use of the terms $\mathcal{P}^a$ and $m$. 
In fact, $\mathcal{P}^a$ is intended to represent the set of polymorphic crystal structures associated with the stoichiometry graph $\mathcal{G}^a$, as indicated in line 10 of Section 3.2. However, we fully agree with the reviewers that this notation might lead to ambiguity since superscript $b$ is consistently used for the structural graph. Therefore, to clarify our notation, we have decided to change the term $\mathcal{P}^a$  into $\mathcal{P}^{\mathcal{G}^a}$.
Moreover, we apologize for missing the precise definition of $m$, which is the indicator function of value 1 if $\mathcal{P}^{\mathcal{G}^a}$ is the set of polymorphic structures corresponding to $\mathcal{G}^a$ and 0 otherwise. We highlighted the modified term in blue color in the main manuscript.

Reviewer comment (Novelty):
Machine learning models that predict the structure given the stoichiometry can be reasonable or even better alternatives to richer representations of the stoichiometry as proposed here. A discussion of the pros and cons of these two approaches and ideally a review of crystal structure prediction methods would have been appreciated.

Author response:
Thank you for pointing out a great discussion point! In our opinion, here are the advantages and disadvantages of employing stoichiometry-based models versus crystal structure prediction (CSP) methods.

**Advantages of stoichiometry-based models**
- CSP methods firstly predict the structure of crystals, and then train another model for downstream tasks based on the predicted structure. Therefore, accurate crystal structure prediction is crucial, as the success of downstream tasks hinges on the initial accuracy of crystal structure prediction. However, it is also well-known that CSP methods encounter difficulties with complex systems, potentially hindering their efficacy in various downstream applications [1]. On the other hand, since stoichiometry-based models derive predictions directly from the elemental composition, they avoid the pitfalls associated with predicting the precise structure.
- Furthermore, we argue that models based on stoichiometry could be applicable to a broader range of materials, including doped and alloyed materials. Doping and alloying involve transformative processes that introduce a level of disorder into a conventional structure, making the precise prediction of the resulting material's structure exceedingly challenging for CSP methods [2]. On the other hand, stoichiometry-based models, which rely solely on the ratio of constituent atoms, offer greater versatility across a diverse spectrum of materials.
- Additionally, the computational expense of stoichiometry-based models is expected to be significantly lower than that of CSP models. Traditional CSP methods, particularly those reliant on Density Functional Theory (DFT), are known to be computationally demanding [3]. On the other hand, it is acknowledged that stoichiometry-based approaches circumvent the need for extensive computational resources, thereby offering a cost-effective alternative [4].

**Disadvantages of stoichiometry-based models**
- Given that the crystal structure is a key determinant of a material's properties, CSP models that can precisely predict crystal structure have the potential to surpass the performance of stoichiometry-based models.

[1] MLatticeABC: generic lattice constant prediction of crystal materials using machine learning. ACS omega 2021.

[2] Effects of doping on the crystal structure of poly (3-alkylthiophene), Journal of Materials Chemistry 2.9 1992.

[3] Density functional theory: a practical introduction, John Wiley & Sons 2022.

[4] Representations of materials for machine learning, Annual Review of Materials Research 53 2023.

Reviewer comment (Experiments):
Several times throughout the paper it is mentioned that there exists a one-to-many relationship between the stoichiometry and the possible structures. Does the structural graph contain information about the atoms species? I wonder if it is also the case, otherwise, that one structure actually corresponds to more than one stoichiometry.

Author response:
No, the structural graph in PolySRL does not include atom species information in the structural graph. Instead, as detailed in line 11 of Section 3.2, each node in the structural graph is associated with a learnable feature vector to make sure we utilize only structural information. Traditional machine learning methods that use structural graphs [1, 2] typically embed atom species information within the nodes of the graph. However, we intentionally omitted atom species information to ensure that our focus was solely on structural information. This methodology has also been validated in accurately estimating the mutual information between 2D molecule graphs and their corresponding 3D point clouds [3].

[1] Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties, Physical review letters 2018

[2] Atomistic line graph neural network for improved materials property predictions, npj Computational Materials 2021

[3] 3D Infomax improves GNNs for Molecular Property Prediction, ICML 2022

Reviewer comment (Evaluation):
Regarding the "collapsed dimensions" of the representation, I wonder whether it is actually due to the fact that some stoichiometries have no known polymorphs in the data sets and therefore their variance should in fact be zero. What do you think?

Author response:
Yes, we believe “collapsed dimension” is due to the fact that some stoichiometries have no known polymorphs in the data sets, and we posit that Figure 4 corroborates this understanding. As depicted in Figure 4, the predicted uncertainties for stoichiometries associated with a single possible structure (that is, those without known polymorphs in the dataset) exhibit the lowest levels of uncertainty compared to other groups. This suggests that stoichiometries with a single possible structure are likely to present more collapsed dimensions.