Critical Temperature Prediction of Superconductors Based on Machine Learning: A Short Review
Abstract
Superconductors have a lot of promising potential applications in power transmission and power magnet development because of their special characteristics.
However, new superconductor discovery requires extensive trial-and-error experimentation, which is time-consuming and expensive.
The development of machinelearning techniques makes it possible for identifying superconductors and predicting their critical temperature from the material’s proprieties.
This paper gives ashort review of machine learning’s application in superconductors’ critical temperature prediction.
Related datasets and different proposed methods are included.
Andwe also discussed the future research directions and opportunities in this ﬁeld
Introduction
The superconductor is a kind of material with a characteristic critical transition temperature T c belowwhich the resistance drops to zero [1; 2].
It also has another unique property known as Meissner
Effect [3] that it would eject the magnetic ﬁeld if was cooled below the transition temperature T c .
Much attention has been paid to superconductors for its potential application in achieving moreefﬁcient electric power transmission and developing more powerful magnets for electric motors,energy storage, medical equipment and industrial separations [4].
Its clean energy-saving feature candeﬁnitely contribute to the global sustainable development goal.
Past decades witnessed that newsuperconductors with high critical temperature have been discovered [5; 6; 7; 8; 9; 10; 11; 12; 13],which encouraged people that the wider and cheaper application of superconductors is not a dream.
Critical temperature prediction is crucial for superconductors discovery as materials with highcritical temperatures are what we can use widely and economically in real applications.
BardeenCooper-Schrieffer (BCS) theory [14] proposed in 1957, known as the ﬁrst microscopic theoryof superconductivity, has explained the phenomenon in many materials.
However, it still meetsdifﬁculties in some unconventional superconductors.
Understanding the fundamental mechanismof high transition temperature thoroughly is even more challenging.
Finding new superconductorswith high critical temperature prediction is time-consuming and expansive because of the difﬁcultiesin theoretical explanation and the small fraction of superconductors in candidate materials [15; 16],requiring extensive trial-and-error experimentation [17].
Although the exact mechanism is still unknown, it is believed that the structures and some characteristics of the material like bond lengths, valency properties of ions, and the Coulomb coupling betweenelectronic bands determines the conductive properties [18].
Data-driven methods allow learningfrom known superconductors and linking the characteristics of the material with its conductiveproperties and the critical temperature.
The rise of machine learning makes it possible for moreefﬁcient superconductor discovery and more accurate critical temperature prediction, thus narrowingthe search for superconducting materials with high critical temperatures.
This paper gives a short review of some existing works for machine learning’s application in theprediction of superconductors’ critical temperatures.
Both traditional machine learning-based methodsand novel deep learning methods are mentioned.
This progress shows that the prospect of machinelearning and deep learning in material discovery is broad.
Related Datasets and Databases
Data is the core part of machine learning for both training and evaluation.
In this section, we listsome crucial datasets and databases of superconductors and material science.
NIST High Temp.
Superconducting Materials (HTS) Database [19] NIST High Temp.
Superconducting Materials (HTS)
Database contains evaluated thermal, mechanical, and superconductingproperty data for oxide superconductors.
It includes the compounds derived from the Y-Ba-Cu-O,
Bi-Sr-Ca-Cu-O, Tl-Sr-Ca-Cu-O, and La-Cu-O chemical families, as well as other variants of thecuprate and bismuthate materials that have superconductivity.
Information on physical characteristicssuch as density and crystal structures are provided.
Superconductivty Data Data Set [20] Superconductivity Data Data Set comes from the NIST High
Temp.
Superconducting Material Database.
Eighty-one features extracted from 21263 superconductors, along with the critical temperature and chemical formula, are provided.
The task is to predictthe critical temperature based on the features extracted.
SuperCon [21] Created by the Japanese National Institute for Materials Science, SuperCon contains16,413 superconductors and their corresponding critical temperatures.
DataS, DataK [22; 23] DataS and DataK are subsets of SuperCon.
DataS contains 6198 materials(3984 cuprates, 971 iron-based and 1243 other types), while DataK contains 13000 materials (6267cuprates, 1142 iron-based and 5585 other types).
The Materials Project [24] Material Project is a core part of the Materials Genome Initiative,which contains information on 146,323 materials and 24,989 molecules currently.
Lattice structures,formation energy, and band gaps of crystals are provided.
NIST Inorganic Crystal Structure Database (ICSD)
[25] ICSD is an extensive database for completely identiﬁed inorganic crystal structures containing over 260,000 crystal structures, and around12,000 new structures are added every year.
The ﬁrst record in this database can date back to 1913.
The Open Quantum Materials Database (OQMD) [26] Created by Chris Wolverton’s group at
Northwestern University, the OQMD database contains DFT calculated thermodynamic and structuralproperties of 1,022,603 materials.
The approximately 300,000 calculated structures are partly fromthe ICSD and partly from iterating over many chemistries for several simple prototypes.
Crystallography Open Database (COD) [27] COD is a collection of crystal structures of organic,inorganic, metal-organic compounds and minerals.
It contains approximately 37,000 inorganiccompounds and alloys.
Critical Temperature Prediction
Critical temperature prediction is a regression problem to obtain T c value from proprieties of materials.
Both traditional machine learning and deep learning methods have been applied in this task.
3.1 Traditional Machine Learning Methods
The general workﬂow of critical temperature prediction based on traditional machine learning methodsis shown in Figure 1.
The core step in this workﬂow is feature extraction.
Standard features includeelemental property statistics, electronic structure attributes like material’s atomic mass, density, ﬁrstionization energy, atomic radius, density, electron afﬁnity, fusion heat, thermal conductivity andvalence, as well as their mean, weighted mean, geometric mean, weighted geometric mean, entropy,entropy weighted, range, weighted range, standard deviation, and weighted standard deviation [28].
Besides, the chemical formula can also be used as feature.
Based on the Periodic Table of elements,the chemical formula matrix [29] can be built as input.
Common machine learning methods include Linear Regression, LASSO Regression [30], Ridge
Regression [31], Support Vector Regression (SVR) [32; 33; 34], Random Forest [35], Decision Tree[36], Elastic-net [37], XGBoost [38] and so on.
And these machine learning methods can be usedcombined with classic intelligent optimization methods like Particle Swarm Optimization (PSO)[39; 40].
In this subsection, we will review some existing works that focus on building more effectivefeature extraction methods and use different machine learning methods to regress critical temperature.
Zhang et al.
[41] proposed an RS-PSO-SVR prediction model, combining Rough Set (RS) theory [42],
PSO and SVR methods.
PSO is used to determine the key parameters in SVR including regularizedconstant C , and the kernel function parameter γ .
RS preprocessing algorithm is used to calculate theweight of each feature.
The vector of the distance between interacting layers ζ and the calculatedspacing between interacting charges within layers (cid:96) are the input of the RS-PSO-SVR predictionmodel.
Back propagation neural network (BPNN) [43] is used as a baseline.
Similarly, Liu et al.
[44] proposed a PCA-PSO-SVR methods, combining principle componentanalysis (PCA), PSO and SVR methods.
The feature vectors are established by the PCA method,which calculated the eigenvalues of the covariance matrix of the dataset, and selected the determinednumber of top eigenvalues of all the eigenvalues.
Stanev et al.
[22] built a classiﬁcation ﬁrstly to separate materials into two distinct groups dependingon whether T c is above or below a threshold temperature T sep .
Random Forest and its variant methodsare used to predict T c .
The Materials Agnostic Platform for Informatics and Exploration (Magpie)[45] was employed to compute a set of attributes for each material, including elemental propertystatistics and electronic structure attributes.
Matsumoto et al.
[46] calculated the mean value, mean deviation, and standard deviation for eachcomposition in element groups to build 53 descriptors as input features.
The machine learning methodused is also Random Forest regression.
Xie et al.
[47] used low-dimensional descriptors of various measures of the phonon spectrum and theelectron-phonon interaction, proposed by Allen and Dynes [48] for machine learning.
They got anoptimal equation for critical temperature prediction, which led to some improvement from Allen and
Dynes’s ﬁt.
Roter et al.
[49] used Exponential Gaussian Process Elimination, Fine Tree, Boosted Tree, and a
Gaussian Support Vector Machine (SVM) for critical temperature regression.
The Bagged Treemethod best predicted the values of T c .
The element-vectors input is the chemical composition matrixto represent chemical content.
The authors argued that predictors such as the number of valenceelectrons, electronegativity, covalent radius, electron afﬁnity, or the number of unﬁlled orbits are notdirectly relevant for superconductivity.
Gaikwad et al.
[29] used chemical formula from the atomic table directly as input and applied Random
Forest, Decision Tree, Bayes Model, Linear Regression, Decision Tree PCA, SVR, XGBoost, and
SVMRBF methods for regression.
Liu et al.
[50] applied the random forest regression, SVR and the artiﬁcial neural network regressionmethods were used.
The maximum mass difference ∆ M of atoms and the average atomic masse ¯ Mwere used as a descriptor.
Moreover, a multi-step learning strategy was applied to solve problems ofnoisy data.
García-Nieto et al.
[28] used a hybrid regressive model combining the multivariate adaptive regressionsplines (MARS) approximation [51; 52] with the whale optimization algorithm (WOA) [53] forprediction.
The Ridge, Lasso, and Elastic-net regression models were used as baselines.
They usedthe mean, weighted mean, geometric mean, weighted geometric mean, entropy, entropy weighted,range, weighted range, standard deviation, and weighted standard deviation of material’s atomicmass, density, ﬁrst ionization energy, atomic radius, density, electron afﬁnity, fusion heat, thermalconductivity and valence as input features.
The proprieties used for prediction in Revathy et al.
’s
work [54] are mean atomic mas, geometricmean, atomic mass, entropy atomic mass, range atomic mass, standard atomic mass, Fie, meandensity, electron afﬁnity, mean fusion heat, thermal conductivity and valence.
Machine learningmethods include Linear Regression, Decision Tree Regressor, XGBoost, Huber Regressor [55].
Zhang et al.
[56] developed the Gaussian process regression method, a nonparametric kernel-basedprobabilistic model, for doped Fe-based superconductor critical temperature prediction from structuraland topological parameters.
And they also applied the Gaussian process regression model to a widervariety of superconductor families [18].
Zhang and Zhu et al.
[57] applied DScribe software [58] to build a SOAP descriptor, representingthe local environment around a center atom by Gaussian-smeared neighbor atom positions maderotationally invariant [59], transforming atomic structural information database to input features ofthe machine learning models.
Revathy et al.
[60] utilized ﬁe, atomic mass, radius, density, electron afﬁnity, fusion heat, thevalence electron, critical temperature, and thermal conductivity.
Random Forest Regressor, XGBoost
Regressor, Artiﬁcial Neural Networks, Support Vector Regressor, Decision Tree Regressor, Gradient
Boosting Regressor, AdaBoost Regressor, and Simple Linear Regressor are used for training andtesting.
In their experiments, Random Forest Regressor achieved the best results.
3.2 Deep learning Methods
Past years have witnessed signiﬁcant progress in theories and applications of deep neural networks,which has emerged as a most heated area of machine learning research [61; 62].
Deep neural networkslearn the representations needed for the tasks by composing enough processing layers to transformthe representation at a lower level (starting with the raw input) into a representation at a higher andmore abstract level [63], showing an excellent capability in understanding complex, high-dimensionaldata.
Most deep learning attempts at superconductors’ critical temperature prediction are based on CNNsor CNNs hybrid-based models.
Therefore, making the material as a representation that can be inputinto CNNs is the ﬁrst step.
Unlike features in traditional machine learning, representations of materialin deep learning are mainly based on chemical/molecular formulas.
Li et al.
[64] proposed a hybrid neural network (HNN) that combines a convolutional neural network(CNN) and long short-term memory neural network (LSTM).
Vector representations of 87 atoms wereobtained by singular value decomposition (SVD) of the atomic environment matrix used Atom2vecmethods [47] according to the order of the atoms in the chemical formula.
Dan et al.
[65] proposed a convolutional gradient boosting decision tree (ConvGBDT), whichreplaced replace the fully connected layer in regular CNNs by GBDT model for regression.
Statisticalelemental properties in molecular formulas are used for material representation.
In Xiong et al.
’s superconductor critical temperature prediction experiments [66], based on Zheng etal.
’s method [67] shown in Figure 3, periodic table representation of materials was used with chemicalinformation in the two-dimensional arrangement of elements as the input of CNNs.
Konno et al.
[23] also use a method called "reading the period table" to the representation ofmaterial also based on the representation of materials.
The original table is split into four tablescorresponding to s , p , d , and f blocks, which show the orbital characteristics of the valence electrons.
The dimensions of the input representation of CNNs are 4 × 32 × 7.
Viatkin et al.
[68] used six networks for the model ensemble: an LSTM network, four CNNs, andone embedding ensemble of two CNNs.
The outputs are concatenated together and then ﬂattenedbefore the fully connected layers.
3.3 Evaluation
Critical temperature prediction can be evaluated by commonly used evaluation metrics in regressionas a regression task.
And various cross-validation methods are applied to make the evaluation resultsfairer and more convincing.
3.3.1 Evaluation Metrics
The commonly used evaluation metrics in critical temperature prediction are listed in Table 1, wherex i is the actual value, y i is the predicted value, and n is the number of values.
3.3.2 Evaluation Methods
Cross-validation is a common method in machine learning for convincing evaluation.
There are manydifferent cross-validation methods.
k -fold cross-validation split a dataset into k subsets, each ofwhich is used as a test set, and the rest are used as training sets.
And the average cross-validationrecognition accuracy of k times is used as the result.
Leave-one-out cross-validation(LOOCV) usedeach sample as a test set alone for a n samples dataset, and used the remaining n − 1 samples as atraining set each time.
The research of critical temperature prediction is to help new superconductor discovery, whichmay have signiﬁcantly different structures from the known one.
Therefore, explorative predictioncapability of a machine learning method should be attached great importance to. To better show suchcapability, some special evaluation methods designed for critical temperature prediction are proposed. [Table]
Proposed methods are designed for destroyed the randomness of the dataset and resorted them.
Inthis way, the testing set would be more different from the training set.
Meredig et al.
[69] argued the overestimation problem of traditional machine learning evaluationmethods.
A leave-one-cluster-out cross-validation (LOCO CV) and a simple nearest-neighborbenchmark to show model performance were introduced.
In this method, k -means clustering is usedfor training and testing set splitting.
Each k cluster are used for validation while the remain clustersare used for training.
Xiong et al.
[66] also focused on building evaluation methods that can better reﬂect explorativeprediction capability.
They proposed a k -fold- m -step forward cross-validation to evaluate machinelearning algorithms for materials discovery.
The critical temperature prediction problem was alsoaddressed.
In this method, samples are resorted by ascending/descending target property before thetraining/testing set splitted.
Conclusion and Discussion
Machine learning, including traditional methods and deep learning methods, is widely used insuperconductors’ critical temperature prediction.
Feature selection/extraction and representationbuilding are crucial in the learning process.
The performance in open datasets is inspirational forfurther application of machine learning in material properties prediction.
Current material representation based on periodic cannot reﬂect molecular/crystal connection andgeometry.
Although there has been some research on the graph-based representation of the material,the application in superconductors’ critical temperature is limited.
Thus graph-based learning andgeometric learning can be good directions.