Paper: LLM-Prop: Predicting Physical And Electronic Properties of Crystalline Solids From Their Text Descriptions (ICLR-2024)

Reviewer comment (Experiments):
The authors mentioned there’s some related work that also used finetuned LLMs for crystal representation. These works collect crystal text descriptions based on Robocrystallographer too. However, the authors did not compare their dataset with theirs about the text content and text quality. So, the experiment is not solid.

Author response:
Unfortunately, they did not release their data sets which makes it impossible to compare our benchmark with the data they used. Since we did not have access to their train-test splits, we finetuned MatBERT (the model that they finetuned on their data)  on our benchmark so as to make a fair comparison with LLM-Prop.

Reviewer comment (Evaluation):
You have mentioned there’s some related work that also used finetuned LLMs for crystal representation. They collect crystal text descriptions based on Robocrystallographer too. Could you compare your dataset with theirs about the text content and text quality? Showing the advantage of your dataset is important for your contribution to data collection.

Author response:
Unfortunately, they did not release their data sets which makes it impossible to compare our benchmark with the data they used. Since we did not have access to their train-test splits, we finetuned MatBERT (the model that they finetuned on their data)  on our benchmark so as to make a fair comparison with LLM-Prop.

Reviewer comment (Experiments):
Section 5.1 “The possible reason for this improvement might be that LLMProp can easily access the most important information for volume prediction, e.g. space group information, from the text descriptions compared to GNNs.” Could you please make some more ablation about your text information to prove your statement? Also, this is important to measure the importance of different types of information.

Author response:
Dear Reviewer *vjeb*, we thank you again for the constructive feedback on our work. 

We would like to share the performance results of LLM-Prop when it does not see the space group information to support the claim of why having the information in text descriptions that is not present in structure favors text-based property prediction methods (e.g LLM-Prop) over structure-based ones (GNNs). As the results show in the table below (the continuation of Table 6 in the paper), when we remove the space group information from the input sequence we see the performance drop on predicting all properties, with the highest degradation being seen on Volume prediction. We will also update the paper accordingly.

| Model      | Band gap | Volume     | Is-gap-direct     |
| :---        |    :----:   |          :---: |           ---: |
| LLM-Prop (w/o space group info)      | 0.235       | 97.457  | 0.705|
| LLM-Prop   | **0.229**        | **42.259**| **0.857**|

We hope that we addressed your questions and we would be happy to respond to any other further concerns.

Reviewer comment (Experiments):
Could you please make a task description to show the importance of the two tasks (Band gap and volume) in the experiments? Many other properties are predicted in related works such as energy per atom and bulk modulus. Why did you choose these two tasks?

Author response:
Our motivation to focus on the physical properties (Band gap and Volume) was based on the fact that while these properties are very important they are understudied in many prior works probably because prior methods seem to not perform well on these two tasks.

Reviewer comment (Novelty):
Although the task of crystal property prediction is an interesting problem for AI for science, the innovation of methodology is not surprising. This paper only uses the language model to make the prediction which is less novelty.

Author response:
Although the idea of using LMs to make predictions is not novel, finetuning LLMs to achieve SOTA results on crystal property prediction using their descriptions compared to domain-specific models requires innovative approaches and to the best of our knowledge, LLM-Prop is the first method that is based on finetuning a non-domain-specific LLM to achieve that. Different techniques (see Table 6) that we combine together to finetune T5 encoder enable LLM-Prop to outperform a naively finetuned MatBERT despite having a three times fewer parameters than MatBERT which has been pretrained on domain-specific data. As the results show, we also improved over the GNN-based methods, achieving a significant difference in performance (see Table 3-5).

Reviewer comment (Novelty):
There are several recent works focusing on the text-rich graph where each node has text descriptions by jointly leveraging the GNN and LLM. These works are also related to the problem studied in this paper and should be discussed and compared.

Author response:
Although there are works that use GNN and LLM, for example [Zhao et al. 2023](https://openreview.net/forum?id=q0nmYciuuZN), we would like to remind the reviewer that these works are specifically implemented for tasks that are related to text-attributed graphs (for e.g. knowledge graphs and paper citation graphs) such as node classification. In those works each node is a text/document and nodes are interconnected in a graph format. Our work specifically deals with the task of crystal property prediction where each crystal is represented by a text description and does not require building a knowledge graph between different crystals, it directly predicts crystal properties using a finetuned LLM on a text description of that crystal. Although simple, our approach achieves state-of-the-art performance, outperforming even state-of-the-art GNN-based methods.

Reviewer comment (Writing):
What is the specific challenge in the problem of crystal property prediction?

Author response:
Crystals, unlike molecules, are rigid bodies. They have strong symmetries that are challenging to capture by even the current state-of-the-art GNNs ([Choudhary et al. 2021](https://www.nature.com/articles/s41524-021-00650-1)). Our work shows that working in the description space of these crystal structures is a promising way to predict the properties of crystals. The text descriptions that are the input to our proposed method already have information about symmetries such as space groups and by having a powerful pretrained language model like T5 that can easily understand text and extract useful information from the crystal descriptions, we get the crystal representation that is used to accurately predict its properties.

Reviewer comment (Novelty):
My top concern is the technical depth of the paper. For the method, this paper is an implementation for predicting crystal properties task with decoder part of T5. The discussion of choice of T5 is weak and Input Processing is also trivial.

Author response:
We did not use the decoder part of T5 but rather built on the encoder of T5 with a carefully designed finetuning approach. We refer the reviewer to the last paragraph of the Introduction section and section 3.2.1 and 3.2.2  for our detailed discussion of why we chose T5. We also refer the reviewer to section 5.2 and Table 6 that discuss the detailed contribution of the input processing to the overall performance of LLM-Prop. Please, note that the input processing is very crucial when handling scientific data. For instance this very recent work ([Golkar et al. 2023](https://arxiv.org/abs/2310.02989)) shows that encoding numbers in scientific text differently improves the LLMs performance substantially.

Reviewer comment (Experiments):
For the Data collection in Sec 3.1, I did not see the challenging part and discussion about the process of data collection including quality control. I am not sure about the quality of data at all.

Author response:
This is a good point. We will indeed add more details for this part in the revision. We will add details on how Robocrystallographer works, how long it took us to collect the structure data from the Materials Project database, and how to generate the descriptions with Robocrystallographer. We also add details on quality control where for instance we discarded crystals with shorter descriptions (less than 5 tokens), and statistics of the final datasets such as the average length of the descriptions, total number of tokens, average number of numerical values in each description, etc.