Critical Temperature Prediction of Superconductors Based on Machine Learning: A Short Review
Abstract
Superconductors have a lot of promising potential applications in power transmission and power magnet development because of their special characteristics.
However, new superconductor discovery requires extensive trial-and-error experimentation, which is time-consuming and expensive.
The development of machine learning techniques makes it possible for identifying superconductors and predicting their critical temperature from the material’s proprieties.
This paper gives a short review of machine learning’s applications in superconductors’ critical temperature prediction.
Related datasets and different proposed methods are included.
And we also discussed the future research directions and opportunities in this ﬁeld
Introduction
The superconductor is a kind of material with a critical transition temperature T c below which the resistance drops to zero [1; 2].
It also has another unique property known as Meissner Effect [3] that it would eject the magnetic ﬁeld if was cooled below the critical temperature T c .
Much attention has been paid to superconductors for its potential application in achieving more efﬁcient electric power transmission and developing more powerful magnets for electric motors, energy storage, medical equipment and industrial separations [4].
Its clean energy-saving feature can deﬁnitely contribute to the global sustainable development goal.
Past decades witnessed that new superconductors with high critical temperature have been discovered [5; 6; 7; 8; 9; 10; 11; 12; 13], which encouraged people that the wider and cheaper application of superconductors is not a dream.
Critical temperature prediction is crucial for superconductors discovery as materials with high critical temperatures are what we may use economically in real applications.
Bardeen-Cooper-Schrieffer (BCS) theory [14] proposed in 1957, known as the ﬁrst microscopic theory of superconductivity, has explained the phenomenon in many materials.
However, it still meets difﬁculties in some unconventional superconductors.
Understanding the fundamental mechanism of high transition temperature thoroughly is even more challenging.
Finding new superconductors with high critical temperature prediction is time-consuming and expansive because of the difﬁculties in theoretical explanation. And only small fraction of candidate materials are superconductors [15; 16], requiring extensive trial-and-error experimentation [17].
Although the exact mechanism is still unknown, it is believed that the structures and some characteristics of the material like bond lengths, valency properties of ions, and the Coulomb coupling between electronic bands determines the conductive properties [18].
Data-driven methods allow learning from known superconductors and linking the characteristics of the material with its conductive properties and the critical temperature.
The rise of machine learning makes it possible for more efﬁcient superconductor discovery and more accurate critical temperature prediction, thus narrowing the search for superconducting materials with high critical temperatures.
This paper gives a short review of some existing works for machine learning’s application in the prediction of superconductors’ critical temperatures.
Both traditional machine learning-based methods and novel deep learning methods are mentioned.
This progress shows that the prospect of machine learning and deep learning in material discovery is broad.
Related Datasets and Databases
Data is the core part of machine learning for both training and evaluation.
In this section, we list some crucial datasets and databases of superconductors and material science.
NIST High Temp.
Superconducting Materials (HTS) Database [19] NIST High Temp.
Superconducting Materials (HTS)
Database contains evaluated thermal, mechanical, and superconducting property data for oxide superconductors.
It includes the compounds derived from the Y-Ba-Cu-O,
Bi-Sr-Ca-Cu-O, Tl-Sr-Ca-Cu-O, and La-Cu-O chemical families, as well as other variants of the cuprate and bismuthate materials that have superconductivity.
Information on physical characteristics such as density and crystal structures are provided.
Superconductivty Data Data Set [20] Superconductivity Data Data Set comes from the NIST High
Temp.
Superconducting Material Database.
Eighty-one features extracted from 21263 superconductors, along with the critical temperature and chemical formula, are provided.
The task is to predict the critical temperature based on the features extracted.
SuperCon [21] Created by the Japanese National Institute for Materials Science, SuperCon contains 16,413 superconductors and their corresponding critical temperatures.
DataS, DataK [22; 23] DataS and DataK are subsets of SuperCon.
DataS contains 6198 materials (3984 cuprates, 971 iron-based and 1243 other types), while DataK contains 13000 materials ( cuprates, 1142 iron-based and 5585 other types).
The Materials Project [24] Material Project is a core part of the Materials Genome Initiative, which contains information on 146,323 materials and 24,989 molecules currently.
Lattice structures, formation energy, and band gaps of crystals are provided.
NIST Inorganic Crystal Structure Database (ICSD)
[25] ICSD is an extensive database for completely identiﬁed inorganic crystal structures containing over 260,000 crystal structures, and around 12,000 new structures are added every year.
The ﬁrst record in this database can date back to 1913.
The Open Quantum Materials Database (OQMD) [26] Created by Chris Wolverton’s group at
Northwestern University, the OQMD database contains DFT calculated thermodynamic and structural properties of 1,022,603 materials.
The approximately 300,000 calculated structures are partly from the ICSD and partly from iterating over many chemistries for several simple prototypes.
Crystallography Open Database (COD) [27] COD is a collection of crystal structures of organic, inorganic, metal-organic compounds and minerals.
It contains approximately 37,000 inorganic compounds and alloys.
Critical Temperature Prediction
Critical temperature prediction is a regression problem to obtain T c value from proprieties of materials.
As shown in Figure 1, both traditional machine learning and deep learning methods have been applied in this task.
3.1 Traditional Machine Learning Methods
The general workﬂow of critical temperature prediction based on traditional machine learning methods is shown in Figure 2.
The core step in this workﬂow is feature extraction.
Standard features include elemental property statistics, electronic structure attributes like material’s atomic mass, density, ﬁrst ionization energy, atomic radius, electron afﬁnity, fusion heat, thermal conductivity and valence, as well as their mean, weighted mean, geometric mean, weighted geometric mean, entropy, entropy weighted, range, weighted range, standard deviation, and weighted standard deviation [28].
Besides, the chemical formula can also be used as a feature.
Based on the Periodic Table of elements, the chemical formula matrix can be built as input [29].
Common machine learning methods include Linear Regression, LASSO Regression [30], Ridge
Regression [31], Support Vector Regression (SVR) [32; 33; 34], Random Forest [35], Decision Tree [36], Elastic-net [37], XGBoost [38] and so on.
And these machine learning methods can be combined with classic intelligent optimization methods like Particle Swarm Optimization (PSO) [39; 40].
In this subsection, we will review existing works focusing on building more effective feature extraction methods and using different machine learning methods to regress critical temperature.
Zhang et al.
[41] proposed an RS-PSO-SVR prediction model, combining Rough Set (RS) theory [42], PSO, and SVR methods.
PSO is used to determine the critical parameters in SVR, including regularized constant C , and the kernel function parameter γ .
RS preprocessing algorithm is used to calculate the weight of each feature.
The vector of the distance between interacting layers ζ and the calculated spacing between interacting charges within layers (cid:96) is the input of the RS-PSO-SVR prediction model.
Back propagation neural network (BPNN) [43] is used as a baseline.
Similarly, Liu et al.
[44] proposed a PCA-PSO-SVR method, combining principle component analysis (PCA), PSO, and SVR methods.
The feature vectors are established by the PCA method, which calculates the eigenvalues of the covariance matrix of the dataset, and selects the determined number of top eigenvalues of all the eigenvalues.
Stanev et al.
[22] built a classiﬁcation ﬁrstly to separate materials into two distinct groups depending on whether T c is above or below a threshold temperature T sep .
Random Forest and its variant methods are used to predict T c .
The Materials Agnostic Platform for Informatics and Exploration (Magpie) [45] was employed to compute a set of attributes for each material, including elemental property statistics and electronic structure attributes.
Matsumoto et al.
[46] calculated the mean value, mean deviation, and standard deviation for each composition in element groups to build 53 descriptors as input features.
The machine learning method used is also Random Forest regression.
Xie et al.
[47] used low-dimensional descriptors of various measures of the phonon spectrum and the electron-phonon interaction, proposed by Allen and Dynes [48] for machine learning.
They got an optimal equation for critical temperature prediction, which led to some improvement from Allen and
Dynes’s ﬁt.
Roter et al.
[49] used Exponential Gaussian Process Elimination, Fine Tree, Boosted Tree, and a
Gaussian Support Vector Machine (SVM) for critical temperature regression.
The Bagged Tree method best predicted the values of T c .
The element-vectors input is the chemical composition matrix to represent chemical content.
The authors argued that predictors such as the number of valence electrons, electronegativity, covalent radius, electron afﬁnity, or the number of unﬁlled orbits are not directly relevant to superconductivity.
Gaikwad et al.
[29] used chemical formula from the atomic table directly as input and applied Random
Forest, Decision Tree, Bayes Model, Linear Regression, Decision Tree PCA, SVR, XGBoost, and
SVMRBF methods for regression.
Liu et al.
[50] applied the random forest regression, SVR, and artiﬁcial neural network regression methods were used.
The maximum mass difference ∆ M of atoms and the average atomic masse ¯ M were used as descriptors.
Moreover, a multi-step learning strategy was applied to solve problems of noisy data.
García-Nieto et al.
[28] used a hybrid regressive model combining the multivariate adaptive regression splines (MARS) approximation [51; 52] with the whale optimization algorithm (WOA) [53] for prediction.
The Ridge, Lasso, and Elastic-net regression models were used as baselines.
They used the mean, weighted mean, geometric mean, weighted geometric mean, entropy, entropy weighted, range, weighted range, standard deviation, and weighted standard deviation of material’s atomic mass, density, ﬁrst ionization energy, atomic radius, density, electron afﬁnity, fusion heat, thermal conductivity and valence as input features.
The proprieties used for prediction in Revathy et al.
’s
work [54] are mean atomic mass, geometric mean, atomic mass, entropy atomic mass, range atomic mass, standard atomic mass, Fie, mean density, electron afﬁnity, mean fusion heat, thermal conductivity and valence.
Machine learning methods include Linear Regression, Decision Tree Regressor, XGBoost, and Huber Regressor [55].
Zhang et al.
[56] developed the Gaussian process regression method, a nonparametric kernel-based probabilistic model, for doped Fe-based superconductor critical temperature prediction from structural and topological parameters.
And they also applied the Gaussian process regression model to a wider variety of superconductor families [18].
Zhang and Zhu et al.
[57] applied DScribe software [58] to build a SOAP descriptor, representing the local environment around a center atom by Gaussian-smeared neighbor atom positions made rotationally invariant [59], transforming atomic structural information database to input features of the machine learning models.
Revathy et al.
[60] utilized ﬁe, atomic mass, radius, density, electron afﬁnity, fusion heat, the valence electron, critical temperature, and thermal conductivity.
Random Forest Regressor, XGBoost
Regressor, Artiﬁcial Neural Networks, Support Vector Regressor, Decision Tree Regressor, Gradient
Boosting Regressor, AdaBoost Regressor, and Simple Linear Regressor are used for training and testing.
In their experiments, Random Forest Regressor achieved the best results.
3.2 Deep learning Methods
Past years have witnessed signiﬁcant progress in theories and applications of deep neural networks, which has emerged as a most heated area of machine learning research [61; 62].
Deep neural networks learn the representations needed for the tasks by composing enough processing layers to transform the representation at a lower level (starting with the raw input) into a representation at a higher and more abstract level [63], showing an excellent capability in understanding complex, high-dimensional data.
Most deep learning attempts at superconductors’ critical temperature prediction are based on CNNs or CNNs hybrid-based models.
Therefore, making the material a representation that can be input into CNNs is the ﬁrst step.
Unlike features in traditional machine learning, representations of material in deep learning are mainly based on chemical/molecular formulas.
Li et al.
[64] proposed a hybrid neural network (HNN) that combines a convolutional neural network (CNN) and long short-term memory neural network (LSTM).
Vector representations of 87 atoms were obtained by singular value decomposition (SVD) of the atomic environment matrix used Atom2vec methods [47] according to the order of the atoms in the chemical formula.
Dan et al.
[65] proposed a convolutional gradient boosting decision tree (ConvGBDT), which replaced replace the fully connected layer in regular CNNs by GBDT model for regression.
Statistical elemental properties in molecular formulas are used for material representation.
In Xiong et al.
’s superconductor critical temperature prediction experiments [66], based on Zheng et al.
’s method [67] shown in Figure 3, periodic table representation of materials was used with chemical information in the two-dimensional arrangement of elements as the input of CNNs.
Konno et al.
[23] also use a method called "reading the period table" to the representation of material also based on the representation of materials.
The original table is split into four tables corresponding to s , p , d , and f blocks, which show the orbital characteristics of the valence electrons.
The dimensions of the input representation of CNNs are 4 × 32 × 7.
Viatkin et al.
[68] used six networks for the model ensemble: an LSTM network, four CNNs, and one embedding ensemble of two CNNs.
The outputs are concatenated together and then ﬂattened before the fully connected layers.
3.3 Evaluation
Critical temperature prediction can be evaluated by commonly used evaluation metrics in regression as a regression task.
And various cross-validation methods are applied to make the evaluation results fairer and more convincing.
3.3.1 Evaluation Metrics
The commonly used evaluation metrics in critical temperature prediction are listed in Table 1, where x i is the actual value, y i is the predicted value, and n is the number of values.
3.3.2 Evaluation Methods
Cross-validation is a standard method in machine learning for convincing evaluation.
There are many different cross-validation methods.
k -fold cross-validation splits a dataset into k subsets, each of which is used as a test set, and the rest are used as training sets.
And the average cross-validation recognition accuracy of k times is used as the result.
Leave-one-out cross-validation(LOOCV) used each sample as a test set for a n samples dataset and used the remaining n − 1 samples as a training set each time.
The research of critical temperature prediction is to help new superconductor discovery, which may have signiﬁcantly different structures from the known ones.
Therefore, the explorative prediction capability of a machine learning method should be attached to great importance.
To better show such ability, some special evaluation methods designed for critical temperature prediction are proposed.
[Table]
The proposed methods are designed to destroy the dataset’s randomness.
This way, the testing set would be more different from the training set.
Meredig et al.
[69] argued the overestimation problem of traditional machine learning evaluation methods.
A leave-one-cluster-out cross-validation (LOCO CV) and a simple nearest-neighbor benchmark to show model performance were introduced.
k -means clustering is used for training and testing set splitting.
Each k cluster is used for validation, while the remaining clusters are used for training.
Xiong et al.
[66] also focused on building evaluation methods that can better reﬂect explorative prediction capability.
They proposed a k -fold- m -step forward cross-validation to evaluate machine learning algorithms for materials discovery.
The critical temperature prediction problem was also addressed.
In this method, samples are resorted by ascending/descending target property before the training/testing set split.
Conclusion and Discussion
Machine learning, including traditional and deep learning methods, is widely used in superconductors’ critical temperature prediction.
Feature selection/extraction and representation building are crucial in the learning process.
The performance in open datasets is inspirational for further application of machine learning in material properties prediction and superconductor discovery.
Current material representation based on periodic cannot reﬂect molecular/crystal connection and geometry.
Although there has been some research on the graph-based representation of the material, the application in superconductors’ critical temperature is limited.
Thus graph-based learning and geometric learning can be good future directions.