P IXEL C HEM : A R EPRESENTATION FOR P REDICTING M ATERIAL P ROPERTIES WITH N EURAL N ETWORK
Anonymous authors
Paper under double-blind review
A BSTRACT
In this work, we developed a new representation of the chemical information for the machine learning models, with beneﬁts from both the real space (R-space) and energy space (K-space).
Different from the previous symmetric matrix presentations, the charge transfer channel based on Pauling’s electronegativity is derived from the dependence on real space distance and orbitals for the heteroatomic structures.
This representation can work for the bulk materials as well as the low dimensional nanomaterials, and can map the R-space and K-space into the pixel space (P-space) by training and testing 130k structures.
P-space can well reproduce the R-space quantities within error 0 .
This new asymmetric matrix representation doubles the information storage than the previous symmetric representations.
This work provides a new dimension for the computational chemistry towards the machine learning architecture.
I NTRODUCTION
The most common way of representing material structures is using the coordinates of atoms together with the periodic unit cell vectors and categorized this information as unstructured data (i.e. cif ﬁles)(Lam Pham et al., 2017).
However, those unstructured data cannot put into neural network directly and need to be reformatted as matrices for the advanced GPU based neural network computational architecture, which is extremely robust in pixel processing.
As Scholkopf & Smola (2002) indicated, when using Kernel-based machine learning models, in order to reach a fast and accurate prediction of ﬁrst principle properties, a single Hilbert space of the atomistic system is required, in which regression is carried out.
In all above situation, an ideal representation needs to possess several essential characters like invariant, unique, continuous, general and efﬁcient etc.(Rupp, 2015;
Bartok et al., 2013; von Lilienfeld et al., 2015):
In addition, apart from the R-space information, other K-space information might be more important for the overall property of material, especially when the research target relates to the band gap or excitation.
Various kinds of representation have been developed and reported.
The Coulomb matrix representation (CM, Rupp (2015); Faber et al.
; Rupp et al.
(2012)) coded nuclear charge and atom displacement inside to predict atomization energies and formation energies.
Based on it, bag of bond (BOB)(Hansen et al., 2015) representation constructed an effective pairwise interatomic potential to strip down pair-wise variant of CM, which improved the predictive model remarkably.
Many-body tensor representation (MBTR, Huo & Rupp (2017)) is derived from 2 above and reached a signiﬁcant drop in error rate.
Symmetry functions(Behler, 2011; Eshet et al., 2010; 2012) is another method which focuses on the representation of local chemical environment of atoms and maps this representation to atomic energy.
Recently, based on Simpliﬁed molecular-input line-entry system (SMILES,
Weininger (1988)) representation, a new 3N-MCTS module based on Mote-Carlo tree search is proposed by Segler et al.
This module keeps only the structure-connection information and trained large dataset to provide supplementary instructions for retrosynthetic analysis.
In addition, representations like bonding-angular machine learning(Hansen et al., 2015) and others(Huang & von Lilienfeld, 2016; Schutt et al., 2014) have also been reported and tested.
However, most of these representations only satisfy those requirements partially, in which unique and continuous principles are often violated.
The most severe shortcoming for both CM and BoB is that they focus on predicting the extentive quantity,in R space like structural energy but left other intensive quantities in K space like band gap.
Though some reported researches also included other properties(Huang & von Lilienfeld, 2016), their error rates are way larger comparing with energy.
Overall, the R-space and K-space are still separated for this kind of representation.
The possible improvements could be made by endorsing more R/K space information to the targeted representations.
Recall that CM only coded atomic number together with distance, BoB added roughly designed bonds energy in compliment, both failed to represent most of the properties in K-space.
Therefore, deﬁning a new representation with enough physical identity from R/K space is a vital task for material informatics to ensure the uniqueness and continuity.
D EFINITION 2. P IXEL C HEM REPRESENTATION FOR FINITE STRUCTURES
Here in this work, we proposed representations with projected both R and K space chemical information into the pixel space (Pixel Chem).
We start from the property of different atoms, mainly binding strength and charge transfer ability.
To describe these, we ﬁrst introduce two new matrices,
C and B .
C , which mainly reﬂects the charge transfer ability, is based on electron afﬁnity energy and ionization energy of atoms.
Inspired by Mullikan Electronegativity(Mulliken, 1934), it is deﬁned as C i,j = ( IE i, : + E A : ,j ) / 2 .
As for binding strength, B i,j ( U Bond ) is imported, which represents the typical bond energy formed by two atoms.
Note that though B i,j = B j,i , C i,j (cid:54) =
C j,i , which means this matrix is not symmetric between diagonal.
Finally, we proposed a displacement matrix D i,j =
D j,i = exp( −| r i − r j | /a ) to represent the relative position of different atoms, where | r i − r j | represent the Euclidean distance between two atoms and a is the interaction strength coefﬁcient trained by machine learning methods.
To mix all the things up, we proposed a charge & energy matrix
E i,j which can be produced by multiplying D i,j with C Ti,j and B Ti,j as follows:
[Equation]
This operation ensures that for every element E (cid:48) in E i,j , there will be a product of the D (cid:48) in D i,j and B (cid:48) in B Ti,j corresponding to the same atom.
Therefore, for all the atoms, we have:
[Equation]
Atomic orbits information, including arrangement, position and number, is another important factor that highly inﬂuence the property of a structure.
In order to represent those information, we ﬁrst construct a 1 × 10 sized orbital charge vector
o i =
[ N (cid:80) core , N s , N px , N py . . .
N dyz , N dxz ] (core stands for all the core electrons) is created to represent (mainly) the electron conﬁguration of every atom in valance electron orbitals.
The binding energy B E of each orbital also needs to be considered to distinguish those orbitals in different chemical environments.
We therefore product the exp( − B orbit E ) of every orbital in o i to get an energy encoded orbital matrix o BE i , where:
[Equation]
To better represent the chemical environment of each atom, especially the different condition of each orbital, we hereby introduce two closely related methods, tight-binding method and linear combination of atomic orbitals (LCAO, Slater & Koster (1954)) method, to our representation model.
With the existence of atomic Hamiltonian, this matrix can precisely describe the interaction between two atoms orbital by orbital.
In detail, a 10 × 10 sized angular interaction matrix
A i,j , representing from all core electrons to d z orbital electrons, is implemented for all atoms in one speciﬁc structure, which formats as:
[Equation]
core ] , j [ (cid:80) core ])
Int ( i [ (cid:80) core ] , b [ s ]) · · · Int ( i [ (cid:80) core ] , j [ d x 2 − y 2 ])
Int ( i [ s ] , j [ (cid:80) core ])
Int ( i [ s ] , j [ s ]) · · ·
Int ( i [ s ] , j [ d x 2 − y 2 ]) .
Int ( i [ d x 2 − y 2 ] , j [ (cid:80) core ])
Int ( i [ d x 2 − y 2 ] , j [ s ]) · · ·
Int ( i [ d x 2 − y 2 ] , j [ d x 2 − y 2 ])   10 ×(4) which i [ m ] represent the m orbital of i atom.
Recalled 3, every element inside the matrix of atoms, shown as Int ( A [ orbit a ] , B [ orbit b ]) , represent the interaction strength between two orbitals which belong to two atoms respectively, it is yielded out as follows:
[Equation]
Which can be summarized as:
[Equation]
The coefﬁcient inside this formula is deﬁned by the position of two atoms, the direction of two target orbits and the energy of them.
To be speciﬁc, the calculation was primarily based on the
Slater-Koster Approximation(Hehre et al., 1972), which was designed to solve the overlapping area between two selected orbits.
Some orbits, such as p x , p y and
p z orbits for the same atom, are perpendicular to each other, which will result in 0 at the corresponding element.
The calculation detail could be found in supplementary notes.
In all, this angular interaction matrix expanded the previous matrix for 10 times, which encodes more physical meanings, especially the Hamiltonian of each atom, which could beneﬁt the learning efﬁciency and accuracy of our training model.
To combine all the angular matrices
A i,j with the previous E .
We product every element E i,j with its corresponding A i,j to get new element M i,j .
Note that new element M i,j is also a 10 ×matrix.
Therefore, M will be the ﬁnal Pixel Chem representation, denoted as:
[Equation]
Which can also be written as:
[Equation]
Moreover, when implementing Pixel Chem into organic structures, this representation could be simpliﬁed because most of the elements in organic structures are from the ﬁrst 3 periods which means they do not have d orbitals.
This simpliﬁcation results in a smaller matrix size and could handle the representation of organic molecules in a faster way.
When considering organic structures, hybridization, especially for carbon and nitrogen, the type of chemical bonds need to be considered.
We hereby implement an adjoint classiﬁer to estimate the chemical state for each carbon, which will return the bond type and hybridization state of carbon atoms.
Take sp3 as an example, instead of which proposed above, the A i,j for carbon atom will be transferred to: [Equation] [Equation]
Int ( i
[Equation]
pz ]
[Equation]
[ (cid:80) core ])
Int ( i [Equation] pz ]
[Equation] j [Equation] s ]
[Equation]
Int ( i [ pz ] , j [ pz ]) 
 5 ×(9)
Hereby the o BE i =
[ N (cid:80) core · exp( − B (cid:80) core E ) , N s · exp( − B sE ) , · · · , N pz · exp( − B pzE )] .
as 4 sp3 orbitals present the same property.
B sp3 E will also take the place of their initial values.
Figure 1 shows different orbital matrix for different hybridization state.
To illustrate this representation in a straightforward way, we visualized the pixel chem using matplotlib in Figure 2.
Each colored square represents a numerical value.
Black color stands for absolute 0, which also means there is no electron in the corresponding orbital.
Some of the offdiagonal and diagonal parts are carried out with the explanation, which are used to represent atoms properties (for diagonal parts) or interaction strength between atoms (for off-diagonal parts).
2. F ROM FINITE TO PERIODIC STRUCTURES
Apart from ﬁnite structures like fullerene, the bulk periodic structures like diamond and quasi 2D layered periodic structures such as graphite or multi-layer graphene could also be represented by
Pixel Chem representation, as shown in Figure 3 (a) to (c).
In this situation, we calculate the interaction including atoms belonging to itself and two adjacent primitive cells, − z and + z respectively.
For those adjacent cells, D i,j is the only difference and it was altered as D i,j = D j,i =
[Equation] ( c is the lattice parameter, k = {− 1 , 0 , 1 } for adjacent cells) to include all interaction parts.
Note that the D i,j in diagonal part will also be larger, which simply indicates whether this matrix stands for a periodic structure or a ﬁnite structure.
By measuring this part, this structure’s lattice constant, C , could be determined.
Moreover, this representation comes together with an adjacent cell interaction strength that could be measured directly from the matrix.
Generally, this methodology also works for periodic structures which have 0 to 2-dimensional periodicity.
In those cases, the adjacent cells in the corresponding dimension will be calculated in the same way.
Figure 3 lists 4 other different structures for a better view of the unity of Pixel Chem.
Among those 4 structures, all except C 60 are periodic and diamond has periodicity for all three dimensions.
This can be directly seen from the diagonal parts where the corresponding orbital interaction value of periodic dimensions will increase.
Note that for the bottom two isoelectronic structures, apart from the orbital difference, graphite is strictly symmetric across diagonal while BN is asymmetric because of the different charge transfer ability.
2. B ENEFITS
Comparing with other representations in previous works, Pixel Chem has enormous beneﬁts.
Firstly, it satisﬁed all the requirements indicated by Rupp (2015); Bartok et al.
; von Lilienfeld et al. (2015), which are (1) invariant, (2) unique, (3) continuous, (4) general, (5) fast and efﬁcient for calculation.
For the invariant property, the neural network we designed is order-uncorrelated.
Meanwhile, the variety of position and atom property of different structures could secure the uniqueness even for isomers.
For (3), though the representation is discrete, the interaction between every two atoms and their major orbitals are considered with no information gap.
For (4), this representation could handle both ﬁnite and periodic structures with only a little bit of difference in deﬁnition.
On the contrary, the CM and most of the previous modules could only handle ﬁnite structures.
For (5), compared with conventional methods, which typically needs more than 10 minutes for a structure with 20 atoms, our neural network could output the predicted value in less than 1 second if the model has been well trained.
Moreover,the matrix representation in the Pixel Chem is asymmetrically mirrored by the diagonals due to the charge transfer directions in heteroatomic structures, doubled of the information storage size compared with the previous matrix representation.
In conclusion, comparing with CM, though Pixel Chem has similar shape and arrangement, the information contains inside is more sufﬁcient as both orbital information, orbital angle, and charge transferability are encoded.
N ETWORK
What we designed is a Pixel Chemistry network (PCnet) which can learn a representation for the prediction of molecular properties such as energy and band gap.
The network reﬂects basic physical laws in molecule including invariance to atom indexing.
All the prediction for different properties is rotationally invariant.
3. A RCHITECTURE
The Figure 4 shows an overview of the architecture of PCnet, which is inspired by Deep Tensor
Neural Network (DTNN, Sch¨utt et al.
Interactions between atoms are modeled by three interaction blocks.
The ﬁnal prediction of the properties is obtained after atom-wise updates of the feature representation and pooling of the resulting atom-wise energy.
We will discuss the different components of the network in the following.
Molecular representation A molecule in a certain conformation can be described uniquely by a set of n atoms with data of atoms
A =
( A 1 , . . .
, A n ) and interaction between each two of all atoms.
We divided the interaction into upper part U
= ( U 1 , U 2 , . . .
, U n ) and lower part L
= ( L 1 , L 2 , . . .
, L n ) by the position in the matrix.
U i =
( U i, 1 , · · · , U i, ( i − 1) , U i, ( i +1) , · · · , U i,n ) represent all the upper part interaction between atom i and other atoms, L i =
( L i, 1 , · · · , L i, ( i − 1) , L i, ( i +1) , · · · , L i,n ) represent all the lower part interaction between atom i and other atoms.
Through the layers of PCnet, we use a tuple of features X l = ( x l 1 , . . .
, x ln ) to represent the atoms or interaction data, where x li ∈ R F with the number of atoms n , the current layer l , the number of features map F .
Atom-wise layer
A dense layer that is applied to the feature x li of atom i separately with learnable weight W l and learnable bias b l in layer l :
[Equation]
The atom-wise layer is designed to represent the combination of the origin feature map.
PCnet can work for molecules with any size since weights of atom-wise layers are shared across atoms.
Interaction
The interaction block is designed to update the representations of atoms based on the interaction with other atoms.
The interaction block uses a residual connection inspired by ResNet(He et al., 2016):
[Equation]
As shown in the interaction block in ﬁgure 4, the residual r li 1 and r li 2 are computed through two dense layers and a tanh non-linearity with lower and upper part interaction as input.
[Equation]
where γ and β are a learnable scale and a learnable shift parameter respectively.
(cid:15) is a small constant to prevent numerical instability.
The SwitchNorm is continuing with a tanh non-linearity.
We keep the number of feature map to be constant F = 150 throughout the whole network except the last atom-wise layer.
All atom-wise layers and dense layers are followed by a learnable PReLU non-linearity(Xu et al., 2015).
The last atom-wise layer is followed by a hyperbolic tangent nonlinearity.
The feature output from the last atom-wise layer represents the differences between the ground state energy and the energy contributed in the molecule, followed by a sum pooling layer which sum all atoms’ contributed energy to get the predicted energy.
3. TRAINING
We apply PCnet to the quantum chemistry dataset QM9(Ramakrishnan et al., 2014).
QM9 has 133885 molecules.
We remove 3423 unreasonable molecules with largest absolute cohesive energy, which may have N-N-N bonds in a single molecule.
The cohesive energy is deﬁned as the energy of molecule subtracts to the sum of atoms’ energies in the molecule.
Cohesive energy is useful inﬁltrating molecules by their stability.
This energy is usually negative, and because of the subtraction operation, it can show the part of the energy used to combine atoms together.
A molecule is considered more stable if less energy is used to combine atoms together.
We randomly chose 10000 molecules for validation, 10000 molecules for testing, and used remains for training.
For energy prediction, the model was trained using SGD with the ADAM optimizer(Kingma & Ba, 2014) with 32 molecules per mini-batch for 1 million steps.
We used 10 − 3 as the initial learning rate and an exponential learning rate decay with rate 0 .
96 every 2 epochs (220924 steps).
All targets were normalized to mean 0 and variance 1.
We minimized the mean absolute error between predicted energy ˆ E and true energy E .
The same method is also applied to predict other properties.
[Table]
[Table]
RESULT
In Table 1 we compare the performance of PCnet and the previous approaches.
These baselines include 5 different hand engineered molecular representations, which then get fed through a standard, off-the-shelf classiﬁer.
These input representations include the Coulomb Matrix (CM, Neese (2003)), BoB, Bonds Angles, Machine Learning (BAML, Huang & von Lilienfeld (2016)), Extended
Connectivity Fingerprints (ECPF4, Rogers & Hahn (2010)), and “Projected Histograms” (HDAD,
Faber et al.
(2017a))
representations.
In addition to these hand engineered features we include the
Molecular Graph Convolutions model (GC, Kearnes et al.
(2016)), the original GG-NN model(Li et al., 2015) trained with distance bins, DTNN and enn-s2s from Message Passing Neural Networks (MPNNs, Gilmer et al.
We can see from Table 1 that PCnet can achieve a low error bars (0.53) in the R-space U0 energy quantity due to the directional coulomb interaction in the representation without any additional load for imposing more geometric information than HDAD.
The K-space accuracy is already below the DFT (B3LYP, Faber et al. (2017b)) level errors.
But for other properties, the errors of PCnet prediction results are higher than some method, and PCnet needs further improvement.
C ONCLUSION
We proposed a new representation of material structures, named as Pixel Chem and designed a PCnet neural network to process this module.
We also introduced a new pixel space apart from R-space and
K-space in conventional chemistry representations.
Distinct from those representations which focus mainly on numerical approach, our module covers both ﬁnite and periodic systems.
In the future, this representation has a large prospect.
First, the combination of parameters can still be optimized to carry more information with less space.
Moreover, more physical and chemical properties, such as Young’s modulus and melting point, can be predicted using the same way.
Finally, it can be wildly implemented in other ﬁelds.
One example is computational biology, where it can be used to construct larger material structures.