A Binded VAE for Inorganic Material Generation
Abstract
Designing new industrial materials with desired properties can be very expensive and time consuming.
The main difﬁculty is to generate compounds that correspond to realistic materials.
Indeed, the description of compounds as vectors of components’ proportions is characterized by discrete features and a severe sparsity.
Furthermore, traditional generative model validation processes as visual veriﬁcation, FID and Inception scores are tailored for images and cannot then be used as such in this context.
To tackle these issues, we develop an original Binded-VAE model dedicated to the generation of discrete datasets with high sparsity.
We validate the model with novel metrics adapted to the problem of compounds generation.
We show on a real issue of rubber compound design that the proposed approach outperforms the standard generative models which opens new perspectives for material design optimization.
Introduction
The design of new drugs and materials with desired properties is an expensive and time consuming process.
It traditionally involved laborious trial-and-error procedures.
Evolutionary methods such as genetic algorithms [14, 1] accelerated the process, for instance by modifying molecules structures to obtain molecules with desired properties.
However, these methods need hand-crafted rules and constraints to avoid unrealistic generations, which might be hard or even impossible to express mathematically.
To bypass this issue, a good alternative are deep generative methods that are able to learn the data space distribution without the need of constraint deﬁnition.
In the past few years, deep generative models such as GANs [4] and VAEs [11] have been successfully applied to different tasks ranging from classiﬁcation to generation [9, 19] or even image translation [20].
However, as observed in our experiments (cf Section 3), these two models do not provide satisfactory solutions for the generation of new materials.
The difﬁculties encountered are the severe sparsity of the data combined with the diversity of components’ ratios from one compound to another.
Moreover, GANs and VAEs require their generations to be differentiable and thus fail to generate discrete compounds [5].
Their inability to directly generate discrete data appears to be an open problem.
Furthermore, unlike when generating images, visual inspection of each generated compound is more complex and requires an expert’s validation.
Finally, the traditional metrics used to evaluate a generative model are based on the internal features of a deep network trained on ImageNet [12] as the Inception network [18]; therefore, they cannot be used as such for material design.
To overcome all the above-mentioned issues, several works are carried out in the ﬁeld of new compound generation [15, 6, 13, 3].
For instance when working with GANs for inorganic material generation, the Negative Dice Loss [16] is adopted to enforce the discontinuity of the generations by enforcing the overlap between the real and the generated data [2].
Our work is in line with this research ﬁeld and proposes the two following main contributions:
• First, to deal with the sparsity and the diversity of components’ ratios, we introduce a Binded-VAE model (BVAE) which learns to jointly generate the raw data mask (a binary vector encoding whether or not a component is present in the compound) and the ratio of the components in the compound description (cf Section 2.1).
Second, we propose three metrics tailored to the problem of compound generation.
The ﬁrst one is an adaptation of the FID.
The second one is the KL divergence computed between the distributions of real compound properties and predictions of these properties for generated compounds.
The last one is a metric based on the agreement of "expert rules" (cf Section 2.2).
We show, on a real dataset of rubber compound, that the proposed BVAE provides outperforming results compared to the ones obtained with standard VAEs and GANs (cf Section 3).
Thus, the combination of our model with an optimization algorithm can be considered as a promising tool for industrials wishing to generate new compounds with desired properties.
Furthermore, the developed evaluation metric based on "expert rules" agreement appears to be a suitable validation process for speciﬁc issues as industrial ones.
We are convinced that this metric can easily be extended to other industrial problems as long as expert rules can be deﬁned on the dataset.
Methodology
We consider the problem of compound generation, where the learner has access to a sample of n real compounds with p components, and denote X ∈ R n × p + the corresponding design matrix.
Each compound x ∈ R p + is represented by a p multidimensional vector denoted by x = ( x 1 , ..., x p ) where x i ∈ R + is the ratio of the i th component in the compound such that (cid:80) ni
=1 x i = 1 .
In practice, each vector is very sparse and only a few x i are not null.
The goal of this work is to build a model able to simulate a dataset of new compounds (cid:101) X distributed in the same way than X . We denote (cid:101) x
= ( (cid:101) x 1 , ..., (cid:101) x p ) the multidimensional vector corresponding to (cid:101) X .
Because of the sparsity in the raw data X , we observe that a standard VAE fails to generate a compound with a few non-zero components and adequate ratios (cf section 3.2).
In order to overcome this problem, we decide to split the learning of the binary mask and the ratios into a Binded-VAE model (BVAE) as described in the next section.
2.1 BVAE model
The BVAE is composed of two chained Variational AutoEncoders (cf Figure 1).
The ﬁrst one, denoted "Mask-VAE" (VAE M ) receives as input a binary matrix M ∈ { 0 , 1 } n × p obtained by binarizing the real data X such that M ij = 1 if X ij > 0 and M ij = 0 elsewhere.
It generates a reconstructed binary mask (cid:102) M .
The second VAE referred as "Ratio-VAE" (VAE R ) is a Conditional VAE [17] that receives the concatenation of the real training data X and its corresponding reconstructed mask (cid:102)
M generated by the Mask-VAE.
It generates a reconstructed set of ratios (cid:101) R .
The ﬁnal reconstruction of X is then obtained by multiplying the mask (cid:102) M with the output of the Ratio-VAE such that (cid:101)
X
= (cid:102)
M × (cid:101) R .
The ﬁnal reconstruction (cid:101) X is differentiable so that the involved gradient can be calculated.
For the experiments in section 3 the generated data are truncated to the 4 th decimal place.
The loss of the Ratio-VAE is a classical VAE loss:
[Equation]
Where BCE is the Binary Cross-entropy deﬁned as:
[Equation]
and KL the Kullback-Leibler divergence.
µ x , σ x are the outputs of the VAE R encoder.
For the Mask-VAE, the overlap between the generated mask (cid:102) M and the binarized input M must be maximized.
To this purpose, inspired by measures of regions overlap in image segmentation [8], we add to the classical VAE loss, the Negative Dice loss (NegDICE) [16] which has been recently adopted for inverse design of inorganic materials [2].
The NegDICE loss between two vectors a, b ∈ [0 , 1] p with a = ( a 1 , ..., a p ) and b = ( b 1 , ..., b p ) is deﬁned as follows:
[Equation]
Therefore, the loss of the Mask-VAE is written as follows:
[Equation]
Finally, the total loss of the BVAE is:
[Equation]
With λ R , λ M , γ and β positive parameters selected with gridsearch.
2.2 Evaluation Metrics
As mentioned in Section 1, standard metrics used to evaluate generative models are tailored for images and thus are not suited for compound generation.
We then propose the followings metrics:
• Custom FID (FID*).
The FID is the most widespread score used to evaluate the quality of the images of a generative model [7] (cf equation 6).
The FID compares the ﬁrst two moments of the real and generated distributions in an embedding given by the penultimate layer of a pretrained deep network.
The FID score is then computed as follows:
[Equation]
With µ R , µ G and Σ R , Σ G the ﬁrst and the second moments of two distributions R, G .
The deep network usually used to compute the FID is the Inception network [18] trained on ImageNet
This network is trained on images and is then not suited
for our problem. Consequently we adapt the computation of the FID to our problem by using a fullyconnected neural network trained on the real compound dataset to predict properties of interest associated to each compound
We denote this custom metric FID*.
• KL divergence.
Following the same idea, we use the aforementioned predictor to predict the properties for the generated data.
We then compute the KL divergence between the predictor outputs of the generated data and the real distribution of the material properties.
• "Expert Rules Agreement" metric (ERA).
As our data represents compounds of real materials, they must verify speciﬁc rules given by experts as for instance, bounds on the total proportion of several components in the compound.
We then deﬁne the ERA metric as the percentage of generated data that match the expert rules.
Experiments
In our experiments, we compare different generative models trained on a real rubber compound dataset.
In this chapter, we start by presenting the experimental setup, then the dataset used for training the models, and lastly we present and discuss the results obtained.
3.1 Setup
Baselines.
To perform our experiments we use different generative models: a GAN, a Vanilla VAE, our BVAE model, and an "Unbinded-VAE" model built with a "Mask-VAE" and a "Ratios-VAE" trained successively and separately.
The models used in our experiments are composed of fully connected layers with the architectures given in Appendix A.1.
We use the Adam optimizer [10] for all trainings.
Hyper-parameters selection is performed through gridsearch.
The range for each hyper-parameter gridsearch are: [ 10 − 1 , 1 , 10] for λ R , λ M , γ and β (cf section 2.1), [32 , 64 , 128 , 256] for the latent dimensions and [10 − 3 , 10 − 2 , 10 − 1 ] for the learning rates.
The selected hyper-parameters are provided in Appendix A.1 Table 1.
Dataset.
To train the models, we use a dataset of rubber compound of size ∼ 12 k .
The dataset is made of 1D vectors of components ratios, with a size of 99 and with an average sparsity of ∼ 80% .
We train all the models for 200 epochs, with a batch size of 128.
The evaluation metrics presented in
Section 2.2 are computed at each epoch.
The predictor used to compute the FID* and KL scores is trained using a dataset containing the values of three properties of interest for each compound.
3.2 Results and Discussion
In this section, we discuss the experiments results.
We start by comparing the heatmaps of the generations and the real data; then, we compare the models with respect to the custom metrics introduced in section 2.2.
As mentioned in section 2.1, generations are truncated to the 4 th decimal place.
Data visualization.
Contrarily to image generation, inspecting each individual generated rubber compound would require an expert.
However, the presence of patterns in the data provides a visual means to verify the generations.
We propose to visualize the whole generated datasets as a 2D heatmap where zeros appear in dark and higher values are lightened.
As we can observe in Figure 2, this qualitative evaluation clearly shows that the BVAE generates a rubber compound dataset presenting much more realistic patterns than the one produced by the GAN.
This qualitative observation is conﬁrmed by quantitative metrics (cf Figure 3).
By using this kind of visual veriﬁcation, one can easily discriminate between models and quickly spot poor generations.
However, selecting the best generative model needs ﬁner evaluation metrics as the ones developed in the following.
FID* and
KL scores.
Figure 3 presents the evolution of FID* and KL score through epochs for the different generative models.
The two scores are computed on 10 000 generations.
The KL score is computed by averaging the KLs obtained for the three properties.
In addition we plot the test scores computed between two subsets of the real data, and the noise scores computed between the real data and a random Gaussian noise.
We can observe that the GAN performs poorly compared to the other models and is unstable during training, but sill has generally lower scores than the random noise.
We can also observe that the two metrics are coherent and give the same score ranking for all models.
We notice that the UnbindedVAE model has better scores in the ﬁrst epochs before being overrun by the BVAE, which is due to the fact that for the Unbinded-VAE, the mask generation is already pretrained.
The most notable point is the improved performance of the BVAE against the Unbinded-VAE.
This fact shows that the learning of the ratios by the Ratio-VAE has a positive inﬂuence on the learning of the mask by the Mask-VAE.
Indeed, some parts of the mask might be more important than others.
For instance if a component’s ratio is always high when this component is present in the compound, a good prediction of the mask for this component will be more valuable than a good mask prediction for a component that always appears with small ratios.
"Expert Rules Agreement" score.
Looking at the ERA score is even more instructive.
We observe that only the Unbinded-VAE and the BVAE models provide generated data that match all the rules that a rubber compound must at least verify to be considered plausible.
It means that the data produced by the GAN and the Vanilla VAE would not be considered as valid rubber compound by experts.
This highlights one limitation of the FID* and the KL divergence.
Indeed, according to these metrics
Vanilla VAE was close from the two others but as shown here, only 1% of the data generated can be considered as realistic.
Conclusion
The poor performances of standard generative models like GANs or VAEs applied to industrial material generations such as rubber compound, makes them unusable and not very popular amongst industrial experts.
This work proposes a VAE-based generative model able to overcome the difﬁculties encountered by standard generative models, and to generate plausible industrial compound-like data.
We believe that this work should contribute to the popularization of deep generative methods in the industry and demonstrates the potential that these methods can bring to this ﬁeld
Acknowledgments
Part of this research was funded by Manufacture Française des Pneumatiques Michelin and the
Industrial Data Analytics and Machine Learning chair of Centre Borelli from ENS Paris Saclay .
A Appendices A.1 Architectures and hyper-parameters
[Table] [Table]
[Table] [Table]