Multi-objective Deep Data Generation with Correlated Property Control
Anonymous Author(s)
AffiliationAddress email
Abstract
Developing deep generative models has been an emerging field due to the ability tomodel and generate complex data for various purposes, such as image synthesis andmolecular design.
However, the advancement of deep generative models is limitedby challenges to generate objects that possess multiple desired properties: 1) theexistence of complex correlation among real-world properties is common but hardto identify; 2) controlling individual property enforces an implicit partially controlof its correlated properties, which is difficult to model; 3) controlling multipleproperties under various manners simultaneously is hard and under-explored.
Weaddress these challenges by proposing a novel deep generative framework thatrecovers semantics and the correlation of properties through disentangled latentvectors.
The correlation is handled via an explainable mask pooling layer, andproperties are precisely retained by generated objects via the mutual dependencebetween latent vectors and properties.
Our generative model preserves propertiesof interest while handling correlation and conflicts of properties under a multiobjective optimization framework.
The experiments demonstrate our model’ssuperior performance in generating data with desired properties.
Introduction
Developing powerful deep generative models has been an emerging field due to its capability tomodel and generate high-dimensional complex data for various purposes, such as image synthesis[4, 25], molecular design [19, 39, 8], protein design [12, 13], co-authorship network analysis [6]and natural language generation [17, 27].
Extensive efforts have been spent on learning underlyinglow-dimensional representation and the generation process of high-dimensional data through deepgenerative models such as variational autoencoders (VAE) [22, 30], generative adversarial networks(GANs)
[9, 10], normalizing flows [34, 5] etc [40].
Particularly, enhancing the disentanglement andindependence of latent dimensions has been attracting the attention of the community [4, 36, 3, 29, 37,18], enabling controllable generation that generates data with desired properties by interpolating latentvariables [11, 24, 20, 32, 15, 7].
For instance, CSVAE transfers image attributes by correlating latentvariables with desired properties [23].
Semi-VAE pairs latent space with properties by minimizing themean-square-error (MSE) between latent variables and desired properties [26].
Property-controllable
VAE (PCVAE) synthesizes image objects with desired positions and scales [11] by enforcing themutual dependence between disentangled latent variables and properties.
Conditional Transformer
Language (CTRL) model generates text with task-specific style and contents [21].
Despite of the [Figure] rapid growth of research regarding property controllable generation in various domains, criticalchallenges still remain such as: 1) Difficulty in identifying property correlation .
Existing modelsfor data property control typically map each property to its exclusive latent variables.
Also, all thelatent variables are inherently enforced to be independent of each other.
Therefore such completedistentanglement among the consideration of properties disallows the model to characterize thecorrelation among properties and hence can only work for controlling properties that are independentof each other.
However, properties in real data objects are usually correlated (Figure 1 (a)).
Forexample, in a human face image, the face width has a correlation with eye size.
The color of thewild pepper has a correlation with the growth time (Figure 1 (a)).
The correlation of properties hasbeen under-explored, which largely impairs the effectiveness of generative models; 2) Controllingindividual property also enforces an implicit partial control of its correlated properties, whichis difficult to model.
For those correlated properties, controlling one of them will also constrainthe others into some subspace such as a hyperplane or even a non-convex set.
For example, whengenerating face images, if we constrain the width of the face is 100 pixels then the size of the eyes willbe constrained to a reasonable range; 3) Difficulty in simultaneously controlling multiple propertiesunder various manners .
The real-world application usually requires generated object to satisfymultiple constraints of properties simultaneously.
One may want to maximize a property’s value, fixanother property to a certain value, and constrain the third property within a range.
Therefore, the datageneration problem is entangled with and hardened by the multi-objective optimization goal, whichhas not been well explored.
For example, chemists may design a molecule that has specific potential,minimizes its toxicity and meanwhile possesses solubility within a range (Figure 1 (b)).
We overcomethese challenges by proposing a novel deep generative model, CorrVAE, that recovers semanticsand the correlation of properties via disentangled latent vectors.
The correlation is handled by anexplainable mask pooling layer, and properties are precisely retained by the generated data via themutual dependence between latent vectors and properties.
Our generative model preserves multipleproperties of interest while handling correlation and conflicts of properties under a multi-objectiveoptimization framework.
The contributions of this paper are summarized as follows:
• A novel deep generative model for multi-objective control of correlated properties.
Beyond disentangled representation learning, we aim at corresponding latent variables to target propertiesfor better interpretability and controllability.
The model is generic to different types of data suchas images and graphs, together with disentangled terms to obtain independent latent variables tojointly handle correlated properties.
• A correlated invertible mapping is proposed for mapping correlated real properties to latentindependent variables.
An interpretable mask pooling layer has been proposed to explicitly identify how the real-world properties are generated by the corresponding subsets of latent independentvariables.
The information of these latent variables will be aggregated and enforced to be mutually
dependent on the property via an invertible constraint.
• A multi-objective optimization framework is formulated for deep data generation problem .Corresponding
latent variables in the low-dimensional representation are optimized under multipleobjectives and constraints for property control purposes.
Our framework is generic to variousmultiple objectives such as optimizing a property value, constraining property values into a range,maximizing or minimizing a property value while maintaining the correlation among properties.• Extensive experiments are conducted on real-world datasets.
The proposed model can generatedata with multiple desired properties simultaneously in the generation process, demonstrating theeffectiveness of the model.
Moreover, our model shows superior accuracy of generated properties
against target properties compared with comparison models with multiple real-world datasets.
This paper firstly introduces the general framework of the proposed model.
Then we will discussthe details of the model, including the derivation of the overall objective, the mask pooling layer,and invertible mapping between latent space and correlated properties.
Lastly, we conduct thecomprehensive experiment to compare our model with existing methods.
Related works
2.1 Disentangled representation learning
Disentangled representation learning aims to encode information of high-dimensional complexdata into a low-dimensional space that consists of mutually independent variables to separate outindependent factors of variation of data distribution in the representation [36, 1, 3, 29, 16].
Becauseof the success of VAE and GANs as deep generative models [30, 14, 31], a few techniques havebeen developed as variations of VAE or GANs to achieve disentanglement of latent variables in the representation space.
For instance, β -VAE modifies the variational evidence lower bound (ELBO) byadding a hyperparameter β before the KL-divergence term to encourage disentangled latent variables[16].
Instead, cycle-consistent VAE was proposed for supervised disentanglement of specified andunspecified factors of variation using pairwise similarity labels [18].
On the other hand, InfoGANmaximizes the mutual information between the latent variable and the generated sample under theframework of GANs.
Disentangled Representation learning-Generative Adversarial Network (DRGAN) disentangles the representation with the pose of the face on the image through the pose codeprovided to the generator and pose estimation by the discriminator [36].
2.2 Property controllable deep generative models
Considerable efforts have been spent on developing deep generative models that generate data withdesired properties [11, 39, 15, 21, 20].
Techniques for property controllable generation include but arenot limited to 1) reinforcement learning (RL) approach to goal-directed data generation that preservestarget properties, such as Graph Convolutional Policy Network (GCPN) [39] and GraphAF
[35],and 2) mutual dependence between properties and latent variables to control generation process bymanipulating values of latent variables, such as PCVAE [11] and Conditional Subspace VAE (CSVAE)[23].
RL approach nevertheless suffers from the requirement of a large sample size for the trainingpurpose.
Moreover, all the above methods are unable to precisely capture complex correlation amongproperties in an explainable way, nor can they generate data that simultaneously satisfy multiplecorrelated targets of either values or ranges.
To fill the gap between the existing methods and the needfor controllable generation from various domains, we propose a novel controllable deep generativemodel that handles the correlation of properties via an explainable mask pooling layer and generatesdata with desired properties under a multi-objective optimization framework.
Problem formulation
Suppose we have a dataset D , in which each sample can be represented as x , along with y =
{ y 1 , y 2 , ..., y m } as m properties of x , which can be either correlated or independent with each other.
For instance, if the data is a molecule, properties can be molecular weight, polarity or solubility.
Wefurther assume that ( x, y ) is generated via some random processes from continuous latent variablesin ( w, z ) , where w controls the properties of interest in y and z controls all other aspects of x .
We aim to learn a generative model that generates ( x, y ) conditioning on ( w, z ) , where z is disentangled with w and variables in w are disentangled with each other to control either correlated orindependent properties.
Once the model is trained, the user can generate data with target valuesor ranges of properties via editing the corresponding elements in w .
For example, we may wantto generate a molecule with a specific value of the weight, and solubility within a range whileminimizing its toxicity by changing values of w that contribute to those properties.
This goal leadsto the following questions answered by our work: how to automatically identify the correlationamong properties, how to control individual property while enforcing an implicit partial control of itscorrelated properties, and how to control multiple properties simultaneously?
Proposed approach
[Figure]
w ′ .
The mutual dependence between w ′ and the corresponding property is enforced by the invertible constraint achieved by ResNet.
Lastly, the data can be generated from ( w, z ) via the object decoder.
In general, the proposed approach, CorrVAE, identifies the property correlation via a novel maskpooling layer.
It precisely retains properties via the constraint of mutual dependence between latentvectors and properties, and simultaneously controls multiple properties under a multi-objectiveoptimization framework.
The overall framework of the model is shown in Figure 2.
Specifically,the model contains two phases: (1) Learning phase encodes the information of properties via theproperty encoder and other information of the data via the object encoder.
As shown in Figure 2,the correlation information of properties is captured by a novel mask pooling layer.
The mutualdependence between latent variables and properties is enforced via a constraint applied to the learningobjective.
The data is generated via the object decoder in Figure 2; (2) Generation phase generatesdata with desired properties of specific values or within target ranges under the multi-objectiveoptimization framework.
In this section, we will introduce two phases in detail.
4.1 Learning phase4.1.1 Overall objective for disentangled learning on latent variables
The goal requires us to not only model the dependence between x and ( w, z ) for latent representation 141 learning and data generation, but also model the dependence between y and w for controlling theproperty.
We propose to achieve this by maximizing the joint log likelihood p ( x, y ) via its variational 143 lower bound.
Given an approximate posterior q ( z, w | x, y ) , we can use the Jensen’s equality to obtainthe variational lower bound of log p ( x, y ) as:
[Equation]
The joint likelihood log p ( x, y, w, z ) is further decomposed as log p ( x, y | z, w ) + log p ( z, w ) given
Two assumptions: (1) x and y are conditionally independent given w since w only captures informa- 147 tion from y ; (2) z is independent from w and y , equivalent to y ⊥ z | w .
This gives us x ⊥ y | ( w, z ) ,suggesting that log p ( x, y | w, z ) = log p ( x | w, z ) + log p ( y | w, z )
= log p ( x | w, z ) + log p ( y | w ) .
Consequently, we write the joint log-likelihood and maximize its lower bound:
[Equation]
where we define a next-level latent variable
w ′ i as the set of values in w that are independent with 151 each other and contribute to the i -th property to bridge the mapping w →
y while allowing propertycontrolling.
Each value w ′ i in w ′ = { w ′ 1 , w ′ 2 , ..., w ′ m } relates to each y i in y .
Since properties areindependent conditioning on w ′ , the decomposition of the third term holds in Eq.
The relationshipbetween y and w ′ will be further explained in Section 4.1.2.
Given q ϕ ( w, z | x, y ) =
q ϕ ( w, z | x ) =
q ϕ ( w | x ) · q ϕ ( z | x ) , we rewrite the joint probability in Eq.
(2) as the form of the Bayesian variational 156 inference as the first term of the learning objective:
[Equation]
Meanwhile, since the objective function in Eq.
(3) does not contribute to our assumption that z is 158 independent from w and y , and values in w are independent with each other, we decompose the
KL-divergence in Eq.
(3) and penalize the term:
[Equation]
where ρ 1 and ρ 2 are co-efficient hyper-parameters to penalize the two terms.
Details of the proof andderivation regarding the overall objective can be refereed in Appendix A.
4.1.2 Relating the properties and latent variables
To model the dependence between the correlated properties and the associated latent variables p ( y | w )in Eq (3) as well as to capture the correlation among properties, we propose to directly learn thespecific relationship between disentangled latent variables in w and properties y .
The correlationsamong y are also captured.
Specifically, we design a mask pooling layer achieved by a mask matrix M ∈ { 0 , 1 } l × m , where l is the dimension of the latent vector w .
M captures the way how w relatesto y , where M i,j = 1 denotes that w i relates to the j -th property y j , otherwise there is no relation.
Inthis way, two properties that relate to the same variable in w can be regarded as correlated.
The binaryelements in M are trained with the Gumbel Softmax function.
In implementation, the L 1 norm of themask matrix is also added to the objective to encourage the sparsity of M .
Next, given the learned mask matrix M , we model the mapping from w to y .
For properties y , we 173 can calculate the corresponding w ′ that aggregates the values in w that contribute to each property asw · J T ⊙ M , each column of which corresponds to the related latent variables in w to be aggregated to predict the corresponding y .
For each property y j in y , we aggregate all the information from itsrelated latent variable set in w into the next-level latent variable w ′ j (i.e., the j -th variable of w ′ ) viaan aggregation function h :
[Equation]
(5) where J is a vector with all values as one, ⊙ represents the element-wise multiplication and β is the 179 parameter of h .
Then the property y can be predicted using w ′ as:
[Equation]
(6) where f is the set of prediction functions with w ′
=
h ( w · J T ⊙ M ; β ) as the input and γ are theparameter which will be further explained in the next section.
Thus, we have built a one-to-onemapping between w ′ and y .
In addition, the correlation of y i and y j can be recovered if M T · i · M · j ̸ = 0 .
184 4.1.3 Invertible constraint for multiple-property control
As stated in the problem formulation, our proposed model aims to generate a data point x that retainsthe original property value requirement for the given properties.
The most straightforward way todo this is to model both the mutual dependence between each y i and its relevant latent variable setw ′
i .
However, this can incur double errors in this two-way mapping, since there exists a complexcorrelation among properties in y and there are many cases that M T · i · M · j ̸ = 0 .
To address it, wepropose an invertible function that mathematically ensures the exact recovery of bridging variablesw ′ given a group of desired properties y based on the following deduction.
As in Eq.
(6), the set of correlated properties y =
{ y 1 , y 2 , ..., y m } are correlated with the set of latentvariables w ′ =
{ w ′ 1 , w ′ 2 , ..., w ′ m } in a one-to-one mapping fashion.
Thus we assume that y can besampled from a multivariate Gaussian given w ′ as follows: [Equation] N [Equation]
..., y m )
[Equation]
w [Equation] w [Equation]
w
[Equation]
[Equation]
w
[Equation]
m }
[Equation]
Σ
[Equation]
s
[Equation]
t.
[Equation] [Equation] where Lip denotes to the Lipschitz − constant . Namely, to precisely control the properties y , 196 we learn a set of invertible functions f ( w ′ ; γ ) indicated in Eq 6 to model p γ ( y | w ′ ) .
γ is the set ofparameters in Eq 6.
The constraint enforces f ( w ′ ; γ )
[ j ] to be an invertible function to achieve mutualdependence between y j and w ′ j [2].
As a result, we have the third term of the objective function:
[Equation]
200 4.2 Multi-objective data generation phase
In this section, we introduce how to control the property of the generated data based on the well-trainedmodel.
Based on the problem formulation, we aim to generate data that holds the correlated propertiesy =
{ y 1 , y 2 , ..., y m } via w , where the properties need to meet a series of value requirements.
Weapproach the goal by firstly mapping properties y back to w ′ via the invertible function learned in Eq.(7), then optimizing w under a multi-objective optimization framework.
Finally, w is combined withz that controls other aspects of data to generate the data via the objective decoder (Figure 2).
To obtain the corresponding w ′ from properties y , conditioning on the fact that (1) w ′
=
h ( w · J T ⊙
M ; β ) where h is the aggregation function indicated in Eq (5); and (2) w ∼ p ( w ) , we optimize theconditional distribution of p γ ( y | w ′ ) by maximizing the probability that
y = ˆ y as follows:
[Equation]
where σ ij is the element of the i -th row and the j -th column of Σ in Eq.
ˆ y is the set of properties 211 of interest.
The above operations are based on the fact that y is a vector that contains continuousvariables.
Since Σ is unknown, we optimize Eq.
9 under the same condition by alternatively solving 213 the following problem based on the Theorem 4.1:
[Equation]
215 Theorem 4.1. Solution of Eq.
10 is also the solution to optimizing Eq. 9.
The proof of the theorem 4.1 is trivial and can be referred to Appendix B.
In this way, we can obtainw ′
directly from the invertible function f ( w ′ ; γ ) with ˆ y as the input.
Lastly, given the optimal w ′ , we aim to realize the requirements that are posed on the correlatedproperties y of the generated data, such as being specific values, lying in a range or reaching themaximum value.
To this end, we naturally formalize the process of searching the satisfied w ∗ as amulti-objective optimization framework.
The requirements can be defined as a range of properties iny , c i, 1 ≤ y i ≤
c i, 2 , while it can also be the exact values
when c i, 1 = c i, 2 .
These properties can bedivided into two groups based on the requirement of the target value: (1) Y v represents the set ofproperties that are required to be a specific value, namely, y i
=
c i, 1 ; and (2) Y r represents the setof properties, the value of which should lie in a range, namely,
c i, 1 ≤ y i ≤ c i, 2 .
Then the overalloptimization objective is formalized as:
[Equation]
where Y v and Y r are defined based on different applications.
The border of the range can evenbe set to be infinity or negative infinity.
When c i, 2 is set to be infinity, y i is maximized in Eq.
When c i, 1 is set to be negative infinity, y i is minimized in Eq.
The details of overall practical implementation of the aforementioned distributions to model thewhole learning and generation process are presented in Appendix C.
Experiments
[Equation] [Figure]
on the w 4 that controls both x position and x+y position ; (
d Traversing on the w 5 that controls both y position and x+y position ; ( e ) Traversing on the w 3 that simultaneously controls x position , y position and x+y position .
We evaluate the proposed and comparison models on two molecular datasets and two image datasets:1)
The Quaternary Ammonium Compound (QAC) dataset is a real dataset that contains 462quaternary ammonium compounds processed by the Minbiole Research Lab 1 .
An open-sourcecheminformatics and machine learning library were used to generate a number of properties orfeatures for each of the compounds, in which molecular weight and the logP value were used asdata properties in our experiments; 2) QM9 dataset is an enumeration of 134,000 stable organicmolecules with up to 9 heavy atoms [33].
Molecular weight and the logP values serve as the targetproperties for the comparison with the QAC dataset; 3) dSprites contains 737,280 total imagesregarding 2D shapes procedurally generated from 6 ground truth independent latent factors [28],in which shape , scale , x position and y position were employed in our experiments.
To constructcorrelated properties, we additionally formed and tested a new property, x+y positions by summingup x position with y position ; and 4) Pendulum dataset was originally synthesized to explore causality of the model [38].
Pendulum contains 7,308 images in total with 3 entities ( pendulum, light,shadow ) and 4 properties (( pendulum angle, light position ) →
( shadow position, shadow length )).
[Figure]
[Figure]
[Figure]
We compare the proposed model with three comparisonmodels that can be used for property controllable generation: 1) semi-VAE pairs the latent space with desiredproperties and minimizes MSE between latent variablesand desired properties during the training process [26]; 2)
CSVAE correlates the subset of latent variables with properties by minimizing the mutual information [23]; 3) PCVAE implements an invertible mapping between each pairof latent variables and desired properties, while it solelyrelies on the disentanglement assumption of the learnedlatent space and is short for capturing correlated properties[11].
Besides three comparison models, we consider twoother models adapted from CorrVAE for the ablation study:1)
CorrVAE-1 : we replace the mask pooling layer of theproposed model with the ground-truth mask that is manually obtained from the data; 2)
CorrVAE-2 : We replacethe MLP that maps w to w ′ with the simple linear regression to evaluate the significance of non-linear correlationamong properties.
The linear regression model employsthe w ′ as the response variable while the correspondingvalues in w as the independent variables.
5.3 Quantitative evaluation5.3.1 Evaluation metrics
In this section, we quantitatively evaluate the proposedmodel and comparison models on both molecular andimage datasets.
For image data, we evaluate the model by controlling three properties shape , size , xposition , y position and x+y position .
Molecule generation evaluation metrics .
We evaluate the performance of molecule generationvia three common evaluation metrics which focus on the validity, novelty and uniqueness of thegenerated molecules as follows:
Validity measures the percentage of valid molecules overall generated molecules.
Novelty measures the percentage of new molecules in the generated moleculesthat are not in the training set.
Uniqueness measures the percentage of the unique molecules overallgenerated molecules.
The results are shown in Appendix, Table 4.
Controllable molecule generation evaluation metrics .
To evaluate the performance of controllablegeneration on molecules, we evaluate the proposed model in generating molecules with desiredproperties.
Specifically, we evaluate the MSE between the molecular properties of the generatedmolecules and the expected molecular properties.
The results based on predicting three propertiessize , x position and x+y position are shown in Table [-
-]1.
Image property prediction evaluation metrics .
As an additional benefit of the proposed model, theprediction of image property from the latent space could be utilized as an image property predictor.
On the other side, the prediction performance of the predictor could reflect the quality of the model inlearning image properties.
We evaluate MSE between the ground truth image property value and thepredicted image property value.
The results based on predicting three properties size , x position andx+y position are shown in Table 1. We also quantitatively evaluate the quality of generated imagesvia the FID score, reconstruction error and negative log likelihood as shown in Appendix Table 5.
Disentanglement evaluation metrics .
We evaluate the disentanglement of latent variables fromdifferent perspectives.
First of all, we borrow the metric avgMI [26] to evaluate the overall performance of the proposed model and comparison models.
avgMI measures the mutual dependencebetween latent variables and properties, and is calculated by the Frobenius norm regarding the mutualinformation matrix and the ground-truth mask matrix: avgMI =
||
I ( w, y ) − ˜ M || 2 F , where I ( w, y )is the pairwise mutual information matrix between latent variables of w and properties.
˜ M is theground-truth mask matrix indicating of the contribution of latent variables to properties.
The resultsare show in Appendix Table 6.
Note that for CorrVAE, instead we calculate avgMI using I ( w ′ , y ) .
[Figure] [Figure]
5.3.2 Overall performance
We use avgMI to evaluate the overall performance of theproposed model and comparison models.
As shown in
Appendix Table 6, CorrVAE achieves aligned avgMI with PCVAE and semi-VAE in both dSprites and Pendulum [Figure] datasets, the value of which is rather small, suggesting that
CorrVAE achieves a strong mutual dependence betweenthe latent variables and the corresponding properties.
Bycontrast, CSVAE has a larger avgMI, which results fromthe fact that CSVAE bonds the property with all latentvariables rather than a single variable in w .
This mightaffect the efficiency of the model to enforce the mutualdependence between latent variables and properties.
We also visualize generated images of CorrVAEin Appendix Figure 1 and they all look similar with those in dSprites.
According to Appendix Table 4and Appendix Table 5, both molecule and image data are generated well by the proposed model since
CorrVAE achieves 100% validity and novelty on molecular generation and comparable reconstructionerror and FID values on image generation.
5.3.3 Property prediction evaluation
We evaluate the learning ability of the proposed model and comparison models by the MSE betweenpredicted properties and the true properties.
As shown in Table 1, CorrVAE achieves the lowestsize MSE of 0.0016 compared with comparison models, except CSVAE.
Nevertheless, CorrVAEhas a smaller MSE (0.0066) of x+y position than CSVAE (0.3563), showing the superior ability of
CorrVAE to handle correlated properties.
On Pendulum dataset, CorrVAE achieves the lowest MSEof pendulum angle (36.37) among all models except Semi-VAE (9.9455).
This might result from thefact that the correlation of properties in Pendulum dataset is much more complex than that in dSprites,which is hard for the mask layer to capture.
However, the performance of CorrVAE is also satisfyingregarding the MSE of light position and shadow position , compared with comparison models.
Notsurprisingly, as shown in Table 1, CorrVAE-1 achieves better performance than CorrVAE on thosecorrelated properties since it uses the ground-truth mask to control the correlation among properties,which is challenging for CorrVAE to learn.
Besides, the performance of CorrVAE-1 and CorrVAEare rather close on independent properties such as size for dSprites and pendulum angle as well asshadow position for Pendulum. This might be due to that size is independent with two other variablesin dSprites while pendulum angle and shadow position are independent conditioning on light positionand shadow position in Pendulum.
The independence can be well captured by CorrVAE (Figure 5and Appendix Figure 4) so that it has comparable performance on those variables with CorrVAE-1.
CorrVAE-2 has worse performance in both dSprites and Pendulum datasets, as shown in Table 1.
This is because that CorrVAE-2 models ˜ w to w ′ using simple linear regression, which cannot capturethe non-linear correlation among properties that might exist in the dataset.
5.3.4 Controllable generation quality
As shown in Appendix Table 7, for the MSE between generated and expected MolWeight comparedwith comparison models, CorrVAE achieves the lowest MSE of MolWeight(356701.5) and the MSEof logP(24.01) only larger than the Semi-VAE(15.13) on the QAC dataset; CorrVAE achieves acomparable MSE(logP: 2.75, MolWeight: 4476.54) on the QM9 dataset.
Meanwhile, CorrVAE has alower MSE between generated and expected logP than Semi-VAE.
We also evaluate the quality ofgenerated molecules based on the QAC and QM9 dataset, in which CorrVAE and all comparisonmodels have a satisfying performance (Appendix Table 4).
5.4 Qualitative evaluation
We qualitatively evaluate the ability of the property control of CorrVAE using the dSprites dataset.
We firstly interpret the mask pooling layer learned in the training process.
Then we visualize thechange of properties when manipulating corresponding latent variables in w .
Lastly, we visualize theimages that preserve target properties achieved based on section 4.2.
In addition, we pre-trained four predictors to extract properties, including shape , scale , x position , y position and x+y position , fromgenerated images.
All these models were pre-trained in the training set of CorrVAE with the MSEless than 0.001.
The generated images in Figure 3 and Figure 4 are annotated by their correspondingfeatures extracted by these pre-tained models.
Details regarding the structure of pre-trained modelsare explained in Appendix Table 3.
5.4.1 Interpreting mask matrix
We train CorrVAE using shape , scale and three correlated properties x position , y position and x+yposition while setting the dimension of w as 8.
As shown in Figure 5, eventually we obtained aninterpretable mask matrix indicating that w 1 only controls shape and w 2 only controls scale .
Thisis aligned with the fact that those two properties are independent with others in the data. We alsoobserved that w 3 simultaneously controls x position , y position and x+y position , indicating that theyare correlated since x+y position is generated from x position and y position . The correlation amongthose three variables is also captured by w 4 that simultaneously controls x position and x+y position ,and w 5 that simultaneously controls y position and x+y position . In addition, there is one singlevariable w 6 that only controls y position and another single variable w 8 that only controls x position .
5.4.2 Property control by manipulating latent variables
The mask matrix learned in the training process (Figure 5) enables the mask pooling layer of CorrVAE(Figure 2) to control properties accordingly.
Based on the mask matrix shown in Figure 5, as shownin Figure 3 (a), we traverse the value of w 1 within [ − 5 , 5] and the shape of the pattern changesaccordingly from ellipse to square.
Moreover, we traverse the value of w 2 that controls size ofthe object within [ − 5 , 5] and expectedly, the size of the pattern keep decreasing from 1.3877 to0.3406, as shown in Figure 3 (b). If we traverse on w 4 that controls both x position and x+y positionwithin [ − 5 , 5] , we find that the x position of the object moves from the left to the right while x+yposition changes accordingly (Figure 3 (c)). Similarly, when we traverse w 5 that controls both yposition and x+y position within [ − 5 , 5] , the y position moves from the bottom to the top (Figure 3(d)) while x+y position also changes accordingly.
We also evaluate the more complex setting bytraversing the value of w 3 within [ − 5 , 5] that simultaneously controls x position , y position andx+y position .
Not surprisingly, the position of the pattern changes in both horizontal and verticaldirections, corresponding to x+y position .
At the mean time, x position and y position changeaccordingly, as shown in Figure 3 (e).
We also showcase the whole batch (eight) of generated imagesin Appendix Figure 2 corresponding to each constraint of Figure 4.
All images for the same constraintlook similar, indicating the consistency and the replicability of our model.
5.4.3 Multi-objective generation
We evaluate the performance of CorrVAE for the multi-objective generation on the dSprites dataset.
The experiments are performed based on the model that controls five properties, shape , scale , xposition , y position and x+y position , and the mask matrix learned from the training process (Figure 5).
Four sets of property constraints are considered and the corresponding w ∗ ’s are obtained accordingto section 4.2.
Then the image is generated by the object decoder of CorrVAE from w ∗ .
Specifically,when the set of target properties is shape =square, size =0.9, x position =0.8, y position =0.8 and x+yposition =1.6, the location of the pattern leans roughly toward the lower right corner and has a shapeof square (Figure 4 (a)).
Besides, when the constraints of target properties are set as shape =square,size =0.9, x position =0.6 and y position ∈ [0 . 3 , 0 . 4] , we can visualize that the generated pattern has alarge size and is located towards the right hand side, as shown in Figure 4 (b). If we minimize the xposition and set the shape =ellipse, size =0.9 and y position =0.4, as shown in Figure 4 (c), a pattern ofan ellipse with large size but at the very top of the image is observed while the y position roughltaligns with the constraint.
Lastly, we decrease the size, minimize x position and maximize y position .
Based on Figure 4 (d), the generated pattern is located at the very lower left corner and has a muchsmaller size compared with other generated images.
In conclusion, the results show our model cangenerate objects with target properties based on the multi-objective optimization framework.
Allproperties including shape are roughly aligned with the constraints, as shown in Figure 4.
Conclusion
In this paper, we attempt to tackle several challenges in multi-objective data generation by proposinga novel deep generative model.
Firstly, we identify three challenges in generating data preservingtarget correlated properties.
Secondly, we propose CorrVAE that includes a mask pooling layer toidentify and control correlation among properties, and a multi-objective optimization framework togenerate data with desired properties.
Comprehensive experiments were conducted on real-worlddatasets and our model shows superior performance than comparison models.
Future work will bedone on testing other multi-objective optimiation techniques. Checklist For all authors... (a) Do the main
claims made
in
the
abstract
and introduction accurately
reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our work
does not have potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Please see the
code and instructions in the Supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Please see the implementation details in Supplemental material
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] Please see the implementation details in Supplemental
material
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] Please see the implementation
details in Supplemental material
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
Please refer to the implementation details in Supplemental material
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A] They are public or lab-prepared molecular data
or public image data that do not have personally identifiable information or offensive
content
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
We do not use crowdsourcing or conducted research with human
subjects in the work.
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
We do not use crowdsourcing or conducted
[Table]
research with human subjects in the work.
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A] We do not use crowdsourcing or conducted
[Table]
research with human subjects in the work.
[Table]