Element-Wise Formulation of Inorganic Retrosynthesis
Abstract
Synthesizing new inorganic functional materials is a practical goal of materials science.
While the advances in computational techniques accelerated the virtual design, the actual synthesis of predicted candidate materials still remain as an expensive and slow process.
While a few initial studies attempted to predict the synthesis routes for inorganic crystals, the existing models do not yield the uncertainty of the predictions and could produce thermodynamically unrealistic precursor chemicals.
Here, we propose an element-wise graph neural network to predict the inorganic synthesis recipes.
The trained model outperforms the popularity-based statistical baseline model for top - k exact match accuracy test, showing the validity of our approach for inorganic solid-state synthesis.
We further validate our model by the publication-year-split test, where the model trained based on the materials data until the year 2016 is shown to successfully predict the synthetic precursors for the materials synthesized after 2016.
The high correlation between the classification score and prediction accuracy suggests that the prediction score can be interpreted as a measure of uncertainty.
Introduction
Synthesizing new inorganic functional materials is a practical goal of materials science in various fields such as batteries, [1, 2, 3] (photo-)electrochemical catalyst,[4, 5] and solar cell[6] to name a few.
While the advances in computational power and electronic structure calculation methods helped to design new materials in a pace much faster than before,[7, 8, 9, 10] the actual synthesis of predicted candidate materials still remain as a slow process due to an empirical nature of synthesis.
Thus, to reduce the time and cost associated with failed syntheses, efforts to understand the chemistry of materials synthesizability have been attempted in literature.
Fedorovskiy et al.[11] and Ouyang et al.[12] suggested the use of heuristic rules to predict the materials synthesizability, e g., Goldschmidt’s tolerance factor for double halide perovskites or the stability rules for NASICON-structured materials.
Since these heuristic rules for synthetic accessibility are usually domain specific, several thermodynamic quantities (the energy above the convex hull and the decomposition enthalpies) obtained from electronic structure calculations have been widely used as a guideline to estimate synthesizability.[13, 14, 15, 16] More recently, data-driven machine learning (ML) models have been proposed to calculate the materials thermodynamics[17, 18, 19] or the synthesizability of materials based on the structural similarity.[20, 21] Figure
Beyond the synthetic feasibility predictions as briefly described above, a few studies attempted to further suggest synthesis routes for inorganic materials.
For example, based on thermodynamic parameters and some kinetic heuristics, favorable pathways for inorganic materials could be constructed.[22, 23]
Notably, several data-driven models have also been proposed to generate the precursors and synthetic conditions (e.g. heating temperature and time) to synthesize the target materials using the text-mined meta-datasets.[24, 25, 26, 27]
However, the outcomes from generative models used in the latter studies[27] can contain thermodynamically unstable precursors and do not generally inform the measure of uncertainty for predicted reactions, requiring an additional process by domain experts to screen or rank the generated reaction recipes.
This early stage of inorganic retrosynthetic reaction prediction can be contrasted with the organic synthesis planning where there are a number of template-free[28, 29, 30] and template-based[31, 32] models with promising prediction accuracy.
This difference suggests that the concepts used in the successful organic retrosynthesis models may be borrowed and adapted to address the inorganic retrosynthesis problem.
In particular, by noting that most solid-state inorganic syntheses are performed using a finite list of commercial precursors, we envision that the set of popular inorganic precursors used in literature can be seen as a “template" for inorganic solid-state synthesis, and a similar probability-based template selection model used in the organic retrosynthesis can be used in the inorganic synthesis planning.
This template-based recommendation would remove the possibility of yielding unrealistic precursor chemicals in some of the existing generative model-based inorganic retrosynthesis predictions.[27]
In this work, we introduce a template-based graph neural network for inorganic synthesis recipe prediction.
The model is trained to predict a set of precursors for inorganic crystals by ranking the sets of precursors as the probability scores.
Temperature for the solid-state reaction is another important parameter in actual experimental synthesis, that is affected by both the target crystals and detailed precursors chosen.
Thus, we additionally constructed a temperature prediction model that is sequentially-connected to the precursor set prediction model.
These two models combined then generate a set of precursors and temperature to produce a target solid compound.
Due to a high correlation between the prediction score and the prediction accuracy, the proposed model has a key advantage of quantifying uncertainty of the predictions.
Methods
Element-wise formulation of inorganic retrosynthesis
We formulate the retrosynthetic problems of inorganic materials by first dividing the chemical elements in the target product into two types: [Table] elements that have to be provided as reaction precursor (denoted as “source element”) and elements that can come from the reaction environments (denoted as “environmental element”).
After selecting the source elements from the given target inorganic compositions, proper anionic frameworks (denoted as “precursors templates”) have to be attached to each source element to complete the actual precursor compounds.
This formulation of the problem is summarized in Figure 1a.
To categorize the source and environmental elements, we examined the text-mined inorganic reaction database[33].
To that end, we assigned the metal groups (alkali, alkaline, transition, lanthanide, actinide, post transition), metalloid, phosphorus, selenium, and sulfur as the source elements, and the others as environmental elements from the inorganic retrosynthetic point of view.
Based on these definitions, we constructed the total 39 precursors templates (tabulated in Table 2) from the 11,122 curated inorganic retrosynthetic datasets.
The detailed procedures for the dataset selection and curation, and the precursor template extraction are described in the supplementary section.
Retrosynthetic model Based on these definitions, for a given target composition, the compound is encoded as Roost [34] representaiton, which is a 2D graph whose node features are obtained from a separate pretrained representation of inorganic compounds.
Once the representation is fed into the model, the inorganic retrosynthetic model predicts the precursors that can provide all source elements contained in the given target composition using the source element mask, as shown in Figure 1b.
The formulated source element mask enables the model to discriminate the source elements ( Li , La , and Zr ) information from the given compositions ( Li 7 La 3 Zr 2 O 12 ).
Each source element is separately used in the following precursor classifier which predicts the precursor in the formulated templates library.
By calculating the joint probability of a set of precursors determined for each source element, the precursor-sets (synthesis “recipe”) are finally predicted as a probability score which can be ranked.
The brief and detailed architecture of the proposed model, ElemwiseRetro , is described in Fig.
1 and the supplementary section, respectively.
Experiments
Precursor set prediction To demonstrate the ElemwiseRetro model performance, we calculated the top - k exact match accuracy for the test dataset.
Since the model might capture merely the popularity trend of literature-reported examples, as recently discussed for some organic retrosynthesis predictions,[35] we constructed the template-popularity-based model as a baseline comparison.
In this baseline, the prediction is made statistically based on the number of examples in which a particular template appears in the dataset.
The results for the top - k exact match tests are shown in Table 1.
The error bars correspond to the standard deviation of the 5 trained models.
The proposed ElemwiseRetro shows the promising 83 % top - 1 , and 97 % top - 5 accuracy, as compared to the popularity baseline model whose top - 1 and top - 5 accuracies are 54 % and 83 %, respectively.
Uncertainty estimation The reliability of the prediction is important to measure in order to prioritize and manage the cost of experimental environments.
We split the top - 1 exact match accuracy of precursors set prediction depending on their prediction scores.
As shown in Figure 2a, a positive correlation between the prediction score and the accuracy is clear.
This means that the predictions with higher classification scores can be considered as more reliable predictions.
Publication-year-split validation To further validate our model
, we performed the publicationyear-split test, for which we mined the published years of each data using the DOIs tagged in the inorganic reaction database (Figure 2c).
In this benchmark, instead of splitting the entire dataset between the training and test sets randomly as in the original ElemwiseRetro , we used the time sequence to split the dataset, the data before year 2016 for training ( ∼ 75%), and the data after 2016 for testing ( ∼ 25%).
As the accuracy results for this time split case is summarized in Table 1, both original ElemwiseRetro and ElemwiseRetro -TimeSplit yield the consistent model performance.
Furthermore, we split the top - 1 exact match accuracy depending on their classification scores (Figure 2b), and a positive correlation is still clear even though the test dataset was derived from the out of time domain.
This result clearly suggests that our model can be used to discover undiscovered inorganic materials in the future.
Synthetic temperature prediction
The synthetic temperature prediction model performance was investigated by the 2D parity heat map as shown in Figure 5.
The predicted temperatures from the model reproduces the real temperatures qualitatively with the mean absolute error (MAE) of 117 . 9 ◦ C , which outperforms the MAE ( ∼ 140 ◦ C ) of previous results.[25, 26] Nevertheless, a wide range of temperatures ( 300 ∼ 1600 ◦ C ) used to synthesize a target crystal with a limited number of data points is potentially contributing to a relatively large MAE observed here.
Conclusion
We proposed an element-wise template-based retrosynthetic model that enables a probabilistic prediction of precursors set for inorganic crystals and the corresponding synthetic temperatures.
Based on the concept of “source element” and “environmental element”, we derived a set of precursor templates of inorganic crystal compounds.
We demonstrated a promising model performance by the top - k exact match accuracy test.
The observed positive correlation between the classification score and the prediction accuracy also allows us to estimate the uncertainty of the predictions.
We further validated our model by the publication-year-split test, which suggests that our model has a possibility of covering up to the afterward time space where novel inorganic materials will be discovered.
While the current approach is the first and initial effort to use a probabilistic modeling for inorganic solid-state retrosynthesis predictions with uncertainty estimation, we expect that the concept of templates, source-element decomposition, and element-wise prediction proposed here can be a promising direction to further develop inorganic retrosynthetic models with improved performance. Acknowledgments and Disclosure of Funding We acknowledge the support from the Institute of Information & communications Technology
Planning & Evaluation (IITP) grant funded by the Korea government under grant No. 2021-002068 (Artificial Intelligence Innovation Hub) and the National Science Foundation of Korea (NRF2019M3D3A1A01069099 and NRF-2019M3D1A1079303).
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes] See Section 1. (b) Did you describe the limitations of your work? [Yes] See Section A.1 about the reaction coverage issue.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2. If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [Yes] See Sectionand A.1. (b) Did you include complete proofs of all theoretical results?
[Yes] See Section 3. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] This work is unpublished and ongoing process, yet.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section C.2.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Table 1.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section C.1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We cited [33] for the data curation and [34, 36, 37, 38, 39] for the model construction in References section (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A] A Method details A.1 Dataset preparation For preparing the dataset for train and test, we started with the inorganic synthesis-related dataset[33] which was text-mined from the literatures published after the year 2000.
The raw text-mined data contain some incomplete entries thus we further refined the data.
We removed the data with missing or incorrectly parsed elements and stoichiometry, which reduces the data size from 31,782 to 25,873.
Next, the entries with the inconsistency between the target crystal elements and the source elements in precursors were removed, resulting in 22,837 data.
The data are further trimmed by selecting data where only one source element is present for each precursor, which is the case for the most of the data (20,843) due to its affordability in real experimental synthesis for any type of inorganic synthesis.
The duplicate cases were removed (13,445), and few cases of the unstable or metastable precursors like Li {+
+} O 2 were also filtered out due to unmanageability in real synthetic environments.
For synthesis step with several experiment, the synthetic temperature was calculated by averaging.
The data with the synthesis temperature less than 300 ◦ C and more than 1600 ◦ C were removed, as they are outliers.
For multi-step reaction cases which have more than one heating step, we took the average temperature to represent the overall reaction.
Those with the high standard deviation data were removed.
In this study, several other reaction conditions (e.g. sequence of action verbs, type of mixing device, heating atmosphere, etc.) which might be up to each laboratorial standardized procedure would not be considered, incorporating the conditions is a topic of future work.
Through the aforementioned preprocessing, our final dataset size is 11,122 for the precursors set prediction from the targets and 7,541 for the synthetic temperature prediction.
The whole dataset was divided into training: validation: test (8: 1: 1) to separate test data from training process.
Figure 3 shows the coverage of the inorganic reaction data based on the formulated source elements and precursors templates.
The element coverage (Figure 3a) in the periodic table which makes up the target crystals and the reaction coverage (Figure 3b) depending on the target types in the inorganic reaction dataset were shown, which represents our inorganic reaction domain.
Although the total reaction coverage from our template-based approach is 74.6 %, which should be further developed, our formulated concepts still have a possibility to handle reactions involving most elements and the broad types of popular inorganic materials (e.g. oxide, composite, alloy, halide, and etc.).
A.2 Precursor templates extraction
In predicting the retrosynthetic precursors for given inorganic materials, we used source element-wise precursors templates to determine each types of precursor compounds.
After thoroughly investigating the whole 11,122 inorganic synthetic dataset which was curated by the abovementioned preprocessing, we obtained the list of the precursors templates (e.g. - CO 3 in Li 2 ( CO 3 ) , Na 2 ( CO 3 ) , and - OH in
Li ( OH ) , Al ( OH ) 3 ).
The full list of total 39 retrosynthetic precursor templates are shown in Table 2, which appeared at least once in the whole curated dataset.
Based on this precursor templates, our retrosynthetic model can predict each precursor per one source element within the pre-definedtemplate space.
[Table]
n - acetamide - ( NO 3 ) n O m ( OH ) l - oxy-nitrate-hydroxide - ( C 2 O 4 ) n - oxalate - -
B Model details B.1 Retrosynthetic model
We denote our retrosynthetic precursor prediction model as ElemwiseRetro , and the other for the synthetic temperature prediction.
The overall schematics of the two model architectures are illustrated in Figure 4.
In order to find the plausible set of precursors that could synthesize the target product, the composition of the inorganic target material was converted to a graph representation, referred to as Roost .[34]
The atomic feature vectors learned from the ElemNet [36] were embedded as the initial node states of the inorganic graph.
We then apply the message passing neural network (MPNN)[37] to the graph representation, which updates the initial atomic features by the surrounding environmental information.
Typically, after passing the MPNN, the pooling operation is used to gather the updated node vectors in order to obtain single inorganic descriptor as one-to-one mapping with target input.
Instead of using the pooling layer, however, we extracted the node vectors, which correspond to the source elements, from the updated atomic features to solve the retrosynthetic one-to-many (target-to-precursors) problem.
Then the source element descriptors were separately entered into the prediction network (element-wise prediction).
After the training with the precursor templates, the retrosynthetic model classified each source elements to infer their precursor template classes.
At the end of the classifier, probability score distributions of the precursor templates for each source elements were obtained by the SoftMax layer.
Using this individual probability, we can automatically compute the joint probability, resulted in the set of precursors outcome.
These probability concept enables the model to derive the most synthetically probable precursors for inorganic retrosynthesis by ranked as descending probability scores.
After predicting the set of precursors by ElemwiseRetro , both the target and precursors were inputted in the second model for predicting their synthetic temperature (Figure 4b).
The compositions of the target and precursors were converted to inorganic graph by aforementioned Roost .
To distinguish information between the target and precursors, the atomic nodes in the inorganic graph were only intra-connected within the target and precursor set, separately.
Therefore, the target (or precursor) atomic features were updated only from the surrounding target (or precursor) information.
After the
MPNN, the attention pooling layer was applied to extract the target and precursor descriptors from [Table] the updated target and precursor graphs, respectively.
Then the two descriptors were concatenated and fed into the regressor network to predict their synthetic temperature.
B.2 Model construction
The atomic feature vectors learned from the ElemNet [36] were embedded as the initial node vectors of the inorganic graph.
The atomic embedding dimension is 136, which is mapped to 63 dimensions by one linear layer.
The stoichiometric weight is concatenated to each mapped atomic vector, resulting that the initial node dimension is 64.
We used three MPNN layers to update the node features.
The three hidden layers of prediction network have 512/ 512/ 512 nodes.
At the end of the prediction network, SoftMax layer is used in ElemwiseRetro .
B.3 Model ablation study
To elucidate the crucial components of the model, we constructed ablation models. The schematic architectures for the ablation models were illustrated in Figure 6. To investigate the pooling effect that combines the updated atomic node features to one global descriptor, pooling layer was added to
ElemwiseRetro which denoted as the global aggregated prediction model (Global agg.), as shown in Figure 6a.
To further develop the source element-wise prediction model, the global descriptor is concatenated with the initial atomic features, the model of which is denoted as the source elementwise with global aggregated prediction model (Elem-wise w. GLA), as shown in Figure 6b.
All prediction networks for the ablation test were composed of identical GRU layers, comparing the effect of two different types of descriptor network.
The top-k exact match accuracy of precursors set prediction for each model was tabulated in 3.
Two source element-wise ablation models outperformed the conventional global aggregated model from top-1 to top-10, suggesting that the superior model performance was derived from the usage of element-wise descriptors to predict the set of precursors.
However, the model performance did not depend on the usage of the global factor.
This means that using the concept of source element-wise prediction is the important feature to predict the inorganic retrosynthetic reactions.
C Experimental details C.1 Implementation
For curating inorganic database, we used Pymatgen [38] library, which is an open-source Python library for materials analysis.
The model was constructed using Pytorch [39], the deep learning libraries.
All experiments were conducted under the machine, which has an Intel Core i9-12900K @ 3.20 GHz, 128 GB of RAM, and NVIDIA GeForce RTX 3090 GPU.
C.2 Training conditions
Train and validation dataset were used to train the two sequentially-connected models ( ElemwiseRetro and the synthetic temperature prediction model).
The learning rate for both models was 3e-4, weight decay coefficient was 1e-6, and the batch size was 128.
The cross entropy and robust L 1 loss functions were used to train these two models, respectively.
The weight parameters of the best validation loss during the training process (within 50 epoch) were used as the optimized model parameters.
The training curve of the trainset and the validation set for two models were shown in Figure 7.