GraphCG: Unsupervised Discovery of Steerable Factors in Graphs
Abstract
Deep generative models have been extensively explored recently, especially for the graph data such as molecular graphs and point clouds.
Yet, much less investigation has been carried out on understanding the learned latent space of deep graph generative models.
Such understandings can open up a unified perspective and provide guidelines for essential tasks like controllable generation.
In this paper, we first examine the representation space of the recent deep generative model trained for graph data, observing that the learned representation space is not perfectly disentangled.
Based on this observation, we then propose an unsupervised method called GraphCG, which is model-agnostic and task-agnostic for discovering steerable factors in graph data.
Specifically, GraphCG learns the semantic-rich directions via maximizing the corresponding mutual information, where the edited graph along the same direction will possess certain steerable factors.
We conduct experiments on two types of graph data, molecular graphs and point clouds.
Both the quantitative and qualitative results show the effectiveness of GraphCG for discovering steerable factors.
The code will be public in the near future.
Introduction
Graph is a general format for many real-world data.
For instance, molecules can be modeled as a graph [10, 15] where the atoms and bonds are nodes and edges respectively.
Processing point clouds as graphs is also a popular strategy [53, 59], where points are viewed as nodes and edges based on nearest neighbors.
Many existing works on deep generative models (DGMs) focus on modeling the graph data and improving the synthesis quality.
However, understanding DGMs on graph and their learned representations has been much less explored, which may hinder the development of important applications like the controllable generation and the discovery of interpretable data structure.
The graph controllable generation task refers to modifying the steerable factors of graph so as to obtain graphs with desired properties [9, 43].
This is an important task in many applications, but traditional methods ( e.g. , manual editing) possess certain inherent limitations under certain circumstances.
A classic example is molecule editing, which aims at modifying the substructures of molecules [39] that may relate to some key tactics in drug discovery like functional group change [14] and scaffold hopping [2, 23].
This is a routine task in pharmaceutical company, yet, relying on domain experts for manual editing can be subjective or biased [9, 16].
Different to previous works, this paper aims to explore the unsupervised graph editing with DGMs.
It can act as a good complementary module to conventional methods and bring many crucial benefits: (1) It enables the efficient graph editing in the large-scale setting.
(2) It alleviates the requirements for extensive domain knowledge for factor change labeling.
It provides another perspective for editing preference, which reduces biases from the domain experts.
One core property relevant to unsupervised graph editing using DGMs is the disentanglement.
While there does not exist a widely-accepted definition of disentanglement, the key intuition [37] is that a disentangled representation should separate the distinct, informative, and steerable factors of variations in the data.
Thus, the controllable generation task would become trivial with the disentangled DGMs as the backbone.
Such a disentanglement assumption has been widely used in generative modeling on the image data, e.g. , β -VAE [20] learns disentangled representation by forcing the representation to be close to an isotropic unit Gaussian.
However, it may introduce extra constraints on the formulations and expressiveness of DGMs [20, 47, 11, 61].
For graph data, one crucial question remains: is the latent representation space learned from
DGMs on graph data disentangled?
For image generation, one finding [37] shows that without inductive bias, the latent space learned by VAEs is not guaranteed to be disentangled.
However, the disentanglement property of graph DGMs is much less explored.
In Sec. 3, we first study the latent space of DGMs on two typical graph data (molecular graphs and point clouds), and empirically illustrate that the learned space is not perfectly disentangled.
This observation then drives us to the second question: Given a not perfectly disentangled latent space, is there a flexible framework enabling the graph controllable generation in an unsupervised manner?
To tackle this, we propose a model-agnostic and task-agnostic framework called GraphCG for unsupervised controllable generation.
GraphCG has two main phases, as illustrated in Fig.
During the training phase (Fig. 1(a)), GraphCG starts with the assumption that the steerable directions can be learned by maximizing the mutual information (MI) among the semantic directions.
We formulate GraphCG with an energy-based model (EBM), which provides a large family of solutions.
Then for the test phase, with the learned semantic directions, we can carry out the editing task by moving along the direction with certain step sizes.
As the example illustrated in Fig. 1(b), the molecular structure (hydroxyl group) changes consistently along the editing sequence.
For evaluation, we visually verify the learned semantic directions on both types of data.
Further for the molecular graph, we propose a novel evaluation metric called sequence monotonic ratio (SMR) to measure the output sequences.
We summarize our contributions as follows: (1) We conduct an empirical study on the disentanglement property of three pretrained deep generative models (DGMs) on two types of graph data, molecular graphs and point clouds.
We find that the latent space of these pretrained graph DGMs is not perfectly disentangled.
(2) We propose a model-agnostic and task-agnostic method called GraphCG for the unsupervised graph controllable generation.
GraphCG aims at learning the semantic directions by maximizing their corresponding mutual information, and its outputs are sequences of graphs.
(3) We evaluate the proposed methods on two types of graph data, molecular graphs and point clouds.
The experimental results show the clear improvement (up to 3 times better in SMC) over the baselines.
Background and Problem Formulation
Graph and deep generative models (DGMs).
Each graph data (including nodes and edges) is denoted as x ∈ X , where X is the data space, and the DGMs are modeling the data distribution, i.e. , p ( x ) .
Our proposed graph editing method (GraphCG) is model-agnostic, so we also briefly introduce the mainstream DGMs as below.
Variational auto-encoder (VAE) [31, 20] measures a variational lower bound of p ( x ) by introducing a proposal distribution; flow-based model [8, 46] constructs revertible encoding functions such that the data distribution can be deterministically mapped to a prior distribution.
Notice that these mainstream DGMs, either explicitly or implicitly, contain an encoder ( f ( · ) ) and a decoder ( g ( · ) ) function as:
[Equation]
where z ∈ Z is the latent representation, Z is the latent space, and x ′ is the reconstructed output graph.
Notice that in the literature [51, 52], people also call latent representation as latent codes, and in what follows, we will be using these two terms interchangeably.
Semantic direction and step size.
In the latent space Z , we assume there exist D semantically meaningful direction vectors, i.e. , d i with i ∈ { 0 , 1 , . . .
, D − 1 } .
There is also a scalar variable, step size α , which controls the degree to edit the sampled data with desired steerable factors (as will be introduced below), and we follow the literature [51] on taking α ∈ [ − 3 , 3] .
Each direction corresponds to one or multiple factors, such that by editing the latent vector z with direction d i and step size α , the reconstructed graph will be augmented with the desired factors, leading to certain structural changes.
Steerable factors.
The steerable factors are attributes to the DGMs and they refer to the semantic information that we can explicitly learn from the pretrained DGMs.
In this work, we will be studying the steerable factors on the graph data, which is data- and task-specific.
Yet, there is one category of factors that is commonly shared among all the graph data: the structure information.
Concretely, the steerable factors can be the functional groups or fragments in molecular graphs and shapes or sizes in point clouds.
In Appendix C, we provide a comprehensive description of these steerable factors.
Problem formulation: graph controllable generation.
The goal here is: given a pretrained DGM ( i.e. , the encoder and decoder are fixed), we want to learn the most semantically rich directions ( d i ) in the latent space Z .
Then for each latent code z , with the i -th semantic direction and a step size α , we can get an edited latent vector ¯ z i,α and edited data ¯ x ′ , as:
[Equation]
where d i and h ( · ) are the edit direction and edit function that we want to learn.
We expect that ¯ z i,α can inherently possess certain steerable factors, which can then be reflected in the graph structure of ¯ x ′ .
Energy-based model (EBM).
EBM is a flexible framework for distribution modeling:
[Equation]
where E ( · ) is the energy function and A is the partition function.
In EBM, the bottleneck is the estimation of partition function A : it is commonly intractable due to the high cardinality of X .
Various methods have been proposed to handle this issue, including but not limited to contrastive divergence [21], noise-contrastive estimation [17, 4], and score matching [24, 54, 56].
Disentanglement of Latent Representation
In this section, we quantify the degree of disentanglement of the deep generative models (DGMs) for graph data.
In specific, we adopt six disentanglement scores, and the observed low disentanglement scores show that the attributes in latent space are not well disentangled.
The key intuition [37] behind disentanglement is that a disentangled representation space should separate the distinct, informative, and steerable factors of variations in the data.
In other words, each latent dimension of the disentangled representation correspond to one or multiple factors, and the change of the disentangled dimension can lead to the change in the factors accordingly.
This is an appealing attribute for the general controllable generation task, since the straightforward changes on the latent dimensions can result in the change of corresponding properties of the data.
Is the latent space learned in the graph deep generative models disentangled?
For image generation, one previous finding [37] shows that without inductive bias, the representation learned by VAEs is not perfectly disentangled.
We want to verify if this claim is also valid for the mainstream DGMs on graphs.
We conduct the following experiment for verification.
Steerable factors and experiments on disentanglement measure.
There has been a series of works [20, 30, 11, 47, 37] measuring the disentanglement of the latent space in DGMs.
The highlevel idea is that, given the latent representation from a pretrained DGM, we want to explore how predictive it is to certain steerable factors.
First we need to consider the steerable factors in graph.
The extraction of steerable factors requires the domain knowledge.
For instance, in molecular graphs, we can extract some special substructures named fragments or functional groups.
These substructures can be treated as steerable factors since they are the key components of the molecules and are closely related to certain molecular properties [50].
We use RDKit [33] to extract 9 most distinguishable fragments as steerable factors for disentanglement measurement.
For point clouds, we use PCL tool [48] to extract 75 VFH descriptors [49] as steerable factors, which depicts the geometries and viewpoints accordingly.
Then for the disentanglement measure, we consider six metrics on two data types with three backbone models, and results are are shown in Table 1.
All the metric values range from 0 to 1, and the higher the value, the more disentangled the DGM is.
So we can observe that generally, these graph DGMs are not perfectly disentangled
[Table]
[Table]
GraphCG Method
Following the formulation in Sec. 2, given a fixed pretrained graph generative model, the goal is to discover semantically meaningful directions where we can do interpolation in the latent space.
More concretely, we assume there are D semantic directions in the latent space Z , and each direction corresponds to multiple steerable factors.
Moving in the latent space along a single semantic direction with a step size α can result in the change of the certain steerable factors in the graph data accordingly, i.e. , the change of graph structures.
In Sec. 3, the experiment shows that the learned representation space in graph DGMs is not perfectly disentangled.
Without the disentanglement assumption, the graph controllable generation task becomes more challenging.
This naturally raises the next research question: given a not well-disentangled representation space, is there a flexible way to do the graph data editing?
The answer is positive.
We propose GraphCG, a flexible model-agnostic and task-agnostic framework to learn the semantic directions in an unsupervised manner.
It starts with the assumption that the latent representations edited with the same semantic direction and step size should possess similar information (corresponding to the factors) to certain degree, thus by maximizing the mutual information them, we can learn the most semantic-rich directions.
Then we formulate this editing task as a density estimation problem with the energy-based model (EBM).
As introduced in Sec. 2, EBM covers a broad range of solutions, and we further propose GraphCG-NCE by adopting the noise-contrastive estimation (NCE).
4.1 GraphCG with Mutual Information
The mutual information (MI) measures the non-linear dependency between variables.
To adapt it here, we assume that maximizing the MI between edited data points with different editing conditions (directions and step sizes) can maximize the shared information within each semantic direction and step size, while diversifying the semantic information among different conditions.
The pipeline is as follows.
We first sample two codes in the latent space, z u and z v .
Then we pick up the i -th semantic direction and one step size α to obtain the edited latent points in Z .
The corresponding points are as
[Equation]
Under our assumption, we expect that these two edited points can share certain information with respect to the steerable factors.
Thus, we want to maximize the MI between ¯ z ui,α and ¯ z vi,α , which is equivalent to estimating the following objective [36, 35]:
[Equation]
Till this step, we have transformed the graph data editing task into the summation of two conditional log-likelihoods estimation problem.
4.2 GraphCG with Energy-Based Model
Following Eq.
(5), maximizing the MI between I (cid:0)
¯ z ui,α ; ¯ z vi,α (cid:1) is equivalent to estimating the summation of two conditional log-likelihoods.
We then model them using the energy-based model (EBM).
Because these two views are in the mirroring direction, we may as well take one for illustration.
For example, for the first conditional log-likelihood, we can model it with EBM as:
[Equation]
where E ( · ) is the energy function, A ij is the intractable partition function, and f ( · ) is the negative energy.
The (negative) energy function can be quite flexible, and here we use the dot-product:
[Equation]
where h ( · ) is the editing function introduced in Eq.
Similarly for the other conditional loglikelihood term, and the objective becomes:
[Equation]
With Eq.
(8), we are able to learn the semantically meaningful direction vectors.
We name this unsupervised graph controllable generation framework as GraphCG.
In specific, GraphCG utilizes EBM for estimation, which yields a wide family of solutions.
Next we will introduce an intuitive solution.
[Figure]
4.3 GraphCG with Noise Contrastive Estimation
We solve Eq.
(8) using the noise contrastive estimation (NCE) [17].
The high-level idea of NCE is to introduce a noise distribution and transforms the density estimation into a binary classification problem: it aims at distinguishing if the data comes from the noise distribution or from the true distribution.
NCE for solving EBM has been widely discussed [55] and we adopt it as GraphCG-NCE:
[Equation]
where p data is the data distribution and p n is the noise distribution.
The latent vectors, z u , z v , are given, and the noise distribution is based on the semantic directions and step sizes.
Notice that the noise distribution can be very flexible, and we use a uniform random sampling in this paper.
The objective Eq.
(9) is for one latent code pair, and we will take the expectation of it over the dataset.
Besides, we would like to consider extra similarity and sparsity constraints as:
[Equation]
where sim ( · ) is the similarity function between two latent representations, and we take the dot product.
The similarity term is to minimize the similarity between semantic directions, and the sparsity term is to sparsify each semantic direction.
Both regularization terms target at making the learned semantic directions more sparse and diverse.
Thus, the final objective function is:
[Equation]
and c 1 , c 2 , c 3 are coefficients.
This pipeline is illustrated in Fig.
Positive and Negative Views.
GraphCG-NCE is essentially a contrastive learning method, i.e. , we define the positive and negative pairs, and the goal is to align the positives and contrast the negatives.
As described in Eq.
(9), the positive pairs ( z u , z v ) are latent codes moving with the same semantic direction and step size, while the negative pairs are the edited latent codes with different semantic directions and/or step sizes.
We consider two options in designing the positive views.
(1) Perturbation (GraphCG-P) is that for each latent variable z ∈ Z , we apply 2 perturbations ( e.g. , adding Gaussian noise) on z to get 2 perturbed latent codes as z u and z v respectively.
Random sampling (GraphCG-R) is that we randomly encode two data points from the empirical data distribution as z u and z v respectively.
For negative pairs, we simply sample latent codes with different directions and different step sizes.
Semantic Direction Modeling.
To model the semantic direction, we first randomly draw one direction basis vector for each direction, denoted as e i .
Then the corresponding direction vector is modeled using d i =
MLP ( e i ) , where MLP ( · ) is the multi-layer perceptron network.
Design of Editing Function.
Given the semantic direction and two views, the next task is to design the editing function in Eq.
Our proposed GraphCG is flexible, and the editing function is the key design in the energy function Eq.
Thus, we consider both the linear and non-linear cases as:
[Equation]
4.4 Implementations
[Table]
The training algorithm of GraphCGis in Algorithm 1, where the goal is to learn semantically meaningful direction vectors and an editing function in the latent space.
Then we will need to manually pick up the direction vectors corresponding to certain factors for editing with some post-training evaluation metrics.
Finally for the test phase, we can use the pretrained
DGM and selected direction for editing, as described in Eq.
The detailed algorithm is illustrated in Algorithm 2.
Next, we briefly discuss the differences to some other related methods.
[Table]
NCE and Contrastive Representation Learning.
GraphCG-NCE is using EBM-NCE, which is essentially a contrastive learning method, and another dominant contrastive loss is the
InfoNCE [42].
We list the main differences below.
(1) EBM-NCE is equivalent to InfoNCE, and both are essentially doing the same thing: align the positive pairs and contrast the negative pairs.
(2) EBM-NCE [19, 36] has been found to outperform InfoNCE on the general graph data.
(3) What we want to build up here is a flexible framework, as in EBM or GraphCG.
Specifically, EBM provides a more general framework by designing the energy functions.
Experiments 5.1 Graph Data: Molecular Graphs
Backbone DGMs.
We consider two state-of-the-art DGMs for molecular graph generation.
MoFlow [65] is a flow-based generative model on molecules which adopts an invertible mapping between the input molecular graphs and a latent prior.
HierVAE [27] is a hierarchical VAE model which encodes and decodes molecule atoms and motifs in a hierarchical manner.
Besides, the pretrained checkpoints are also provided, on ZINC250K [25] and ChEMBL [38] dataset respectively.
Editing Sequences and Anchor Molecule.
As discussed in Sec. 4, the output of the inference in
GraphCG is a sequence of edited molecules with the i -th direction, { ¯ x ′ } i .
We first randomly generate a molecule using the backbone DGMs, and we name such molecule as the anchor molecule, ¯ x ∗ .
Then we will take 21 step sizes from -3 to 3, with interval 0.3, to obtain a sequence of 21 molecules following Eq.
Note that the edited molecule with step size 0 under the linear editing function is the same as the anchor molecule, i.e. , ¯ x ∗ .
Change of Structure Factors and Evaluation Metrics.
We are interested in the change of the graph structure (the steerable factors) along the output sequence edited with the i -th direction.
To evaluate the structure change, we apply the Tanimoto similarity between each output molecule and the anchor molecule.
Besides, for the ease of evaluating the monotonicity, we utilize a transformation (on the
Tanimoto similarity) of output molecules with positive step sizes by taking the deduction from 2.
We call this calibrated Tanimoto similarity sequence (CTS).
Further, we propose a metric called Sequence
Monotonic Ratio (SMR), ϕ SMR ( · , γ, τ ) .
It measures the monotonic ratio of M generated sequences with two arguments: the diversity threshold γ constrains the minimum number of distinct molecules, and the ratio tolerance threshold τ controls the non-monotonic tolerance ratio along each sequence.
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Table] [Table]
Evaluating the Diversity of Semantic Directions.
To better illustrate that GraphCG is able to learn multiple directions with various semantic information, we also consider taking the average of top- K SMR w.r.t.
directions to reveal that all the best K directions are semantically meaningful, as in Eq.
Baselines.
For baselines, we consider four unsupervised editing methods.
The first is Random.
It randomly samples a normalized random vector in the latent space as the semantic direction.
The second one is Variance.
We analyze the variance on each dimension of the latent space, and select the highest one as the semantic direction.
The third one is SeFa [51].
The last one is DisCo [45].
It requires the backbone DGMs to be end-to-end and is infeasible
Quantitative results.
We take D = 10 to train GraphCG, and the optimal results on 100 sampled sequences are reported in Table 2.
We can observe that GraphCG can show consistently better structure change with both top-1 and top-3 directions.
Analysis on steerable factors in molecules: functional group change.
For visualization, we sample 8 molecular graph paths along 4 selected directions in Fig. 3, and the backbone DGM is HierVAE pretrained on ChEMBL.
The CTS holds good monotonic trend in these sequences.
Each direction shows certain unique changes in the steerable factors in molecules.
In Fig. 3(a) and Fig. 3(b), the number of halogen atoms and hydroxyl groups (in alcohols and phenols) in the molecules decrease from left to right, respectively.
In Fig. 3(c), the number of amides in the molecules increases along the path.
Because amides are polar functional groups, the topological polar surface area (tPSA) of the molecules also increase accordingly, which is a key molecular property for the prediction of drug transport properties, e.g. , permeability [13].
In Fig. 3(d), the flexible chain length, marked by the number of ethylene (CH 2 CH 2 ) units, increases from left to right.
Since the number of rotatable bonds (NRB) measures the molecular flexibility, it also increases accordingly [58].
5.2 Graph Data: Point Clouds
Backbone DGMs.
We consider one of the latest DGMs on point clouds, PointFlow [62].
It is using the normalizing flow model for estimating the 3D point clouds distribution.
Then we consider
PointFlow pretrained on three datasets in ShapeNet [3]: Airplane, Car, and Chair.
[Figure] [Figure]
Analysis on steerable factors in point clouds: shape change.
To train GraphCG, we take D =directions, and we sample 8 point cloud paths along 3 directions for visualization in Fig.
More results are in Appendix F.
It is observed that GraphCG can steer the shape of the point clouds, e.g. , the size of cars and the height of chair legs.
We also find it interesting that GraphCG can steer more finger-trained factors, like modifying the number jet engines of airplanes in Figs. 4(a) and 4(b).
Conclusion
In this work, we are interested in unsupervised graph editing.
It is a well-motivated task for various real-world applications, and we discuss two mainstream data types: molecular graphs and point clouds.
We start with exploring the latent space of mainstream deep generative models and propose
GraphCG, a model-agnostic and task-agnostic unsupervised method for graph data editing.
The key component of GraphCG is EBM, and we take the GraphCG-NCE as the solution for now.
For the future work, we may as well extend it to more advanced solutions like denoising diffusion model [22]
A Graph Data A.1 Molecules
Molecules can be naturally represented as the 2D molecular graphs, where atoms and bonds are nodes and edges in the graph, respectively.
For the recent years, graph representation learning has been extensively explored on the molecular graph [10, 15, 34, 63, 7].
Based on the graph representation of molecules, a variety of work have been done for molecule generation.
The state-of-the-art ones include MoFlow [65] and HierVAE [27].
A.2 Point Clouds
A point cloud is represented as a set of points, where each point P i is described by a vector of 3D Euclidean coordinates possibly with extra channels (e.g., colors, surface normals and returned laser intensity).
In 2017, Qi et.
al [44] designed a deep learning framework called PointNet that directly consumes unordered point sets as inputs and can be used for various tasks such as classification and segmentation.
For the generative models on point clouds, we consider one of the latest work, PointFlow [62].
B Related Work
Image editing and image controllable generation Many existing works on controllable generation with DGMs mainly focus on the image data.
With the assumption that the learned latent space already include rich semantic information [29, 26, 52, 18], the question then becomes how to identify semantic-rich directions from the latent space of DGMs.
Depending on whether or using supervised signals to discover the semantic directions, existing works can be divided into two settings: supervised and unsupervised .
The supervised setting relies on the supervised signals to learn the pre-defined semantic-rich directions.
For instance, InterfaceGAN [52] identifies the semantic directions in the latent space via linear models that recognize semantic boundary among data.
LACE [40] uses energy-based models to learn the joint distribution of data and properties ( i.e. , semantics) and formulate the sampling process as to solve an ordinary differential equation.
As supervised signals usually require domain knowledge and laborious annotations, latest works are more focused on the unsupervised setting, either model-specific or data-specific.
Specifically, GANSpace
[18] applies
PCA on the intermediate layers of GANs (instead of latent space) for learning the semantic-rich directions.
SeFa [51] exploits the pretrained GANs and extracts the semantic-rich directions by using PCA on the backbone layers.
Nevertheless, as both methods are specifically designed for StyleGAN [29], it is nontrivial to generalize them to other DGMs.
A more recent work DisCo [45] employs a different pipeline: it trains a new encoder after reconstruction, and maps the generated images to another latent space for editing.
However, training a new encoder introduces extra complexities.
Graph editing and graph controllable generation This is an emerging field with many downstream applications [9, 53].
However, existing works are mainly focusing on the supervised setting.
For example, conventional approaches, such as genetic algorithms (GAs), edit the molecule graphs in the data space via hand-crafted heuristics with the guidance of moleular property predictors.
More recent learning based methods perform the latent direction discovery, either by training a classifier on the latent space of DGMs on the graph data [27] or by learning to retrieve exemplar samples from a retrieval database for guidance [60].
Other works fine-tune a pre-trained graph DGM using the supervised property annotations as rewards, resulting
a controllable
DGM specifically for the considered task [41, 64].
To the best of our knowledge, our work is the first to explore the unsupervised graph editing in the unsupervised manner.
Besides, different from many previous approches that may only work for molecule graphs or point cloud graphs, our method is generic and thus can be applied to various graph modalities.
C Analysis Experiments on Disentanglement
In Sec. 3, we conduct three analysis experiments to conclude that the representation space is not perfectly disentangled in such the setting.
In this section, we provide more details and complementary information of these experiments.
C.1 Steerable Factors
As mentioned in Sec. 3, we consider the measuring the disentanglement with respect to three types of structured data: molecular graphs and point clouds.
Recall that we need to define steerable factors first, so as to measure the disentanglement.
[Table]
Molecular Graph For molecular graphs, we treat the important
[Table]
or
) as factors
[Table]
and they are extracted using RDKit.
We consider the existence of the as the factor labels:
• aliphatic hydroxyl groups.
• aliphatic hydroxyl groups excluding tert-OH.
• aromatic nitrogens.
• aromatic amines.
• Tertiary amines.
• Secondary amines.
• amides.
• ether oxygens (including phenoxy).
• nitriles.
Point Clouds For point clouds, we adopt the viewpoint feature histogram (VFH) [49] implemented in PCL [48].
There are 308 bins in total, where each bin is a histogram of the angles that viewpoint direction makes with each normal.
VFH has been widely used as point cloud descriptors, and here we binarize it into factors with:
• We collect the shared non-zero bins from all three datasets (Airplane, Car, and Chair), and ignore the bins where the values distribution are highly skewed.
This can give us 75 bins.
• Then for each of these selected bins, we use the median value as the threshold to generate the binary factor labels.
C.2 Disentanglement Measures
We follow the [37] on considering the following six disentanglement measures:
• β -VAE [20] evaluates the prediction accuracy of a linear classifier for the index of a fixed factor of variation.
• FactorVAE [30] addresses the limitations (i.e. corner case) of β -VAE via introducing a majority voting classifier on a different feature vector.
• MIG [5] measures the normalized difference between the highest and second highest mutual informa- tion between latent dimensions and each steerable factor.
• DCI [11] disentanglement score measures the average difference from one of the entropy of probability that a latent dimension is useful for predicting a steerable factor (computed by the relative importance score).
• Modularity [47] measures whether each latent dimension is dependent on at most one single steerable factor.
It computes the average normalized squared difference over the highest and second highest mutual information between each steerable factor and each latent dimension.
• SAP [32] calculates the R 2 score of the linear models trained to predict each steerable factor from each latent dimension.
Recall that all six disentanglement measures range from 0 to 1, and higher value means the corresponding space is more disentangled.
The results can be found in Table 3.
We can conclude that all the values are indeed low on all datasets and models, revealing that DGMs are not well-disentangled in general.
D Implementation and Experiment Details
Editing function.
For the editing function, we consider both linear (Eqs.
(16) and (17)) and non-linear (Eq.
(18)) cases as below, i.e. , for ¯ z i,α
= h
( z , d i , α ) :
¯ z i,α =
z + α · d i , d i =
norm
◦
Linear ( e i
[Equation]
, (16) ¯ z i,α =
z + α · d i , d i = sqrt ◦ norm ◦ ReLU ◦
Linear (
[Equation]
, (17) ¯ z i,α =
z
+ α · d i + norm ◦
Linear ◦ ReLU ◦
Linear ( z ⊕ d i ⊕ [ α ]) , d i = norm ◦
Linear ◦ ReLU ◦
Linear ( e i ) , (18) where ◦ means the composition of two functions.
Objective function.
The objective function is given as:
[Equation]
where L GraphCG-NCE is the MI estimation defined in Eq.
(9), L sim is the direction similarity defined in Eq.
(10), and L sparsity .
c 1 , c 2 and c 3 are three coefficients accordingly.
Hyperparameters.
We list the key hyperparameters in Table 4, and all the results are evaluated on 100 sampled sequences.
We also want to highlight that DisCo is an unstable baseline, in the sense that once we add more training data ( e.g. , from 100 to 500) or more training epochs ( e.g. , from 1 epoch to 5 epochs), the model will collapse, with the nan loss.
Thus, here we are reporting the most reasonable results for DisCo, i.e. , 100 training data with 1 epoch.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
E Results: Molecular Graph E.1 Evaluation Metrics
Change of Structure Factors and Calibrated Tanimoto Similarity (CTS).
We are interested in the change of the graph structure (the steerable factors) along the output sequence edited with the i -th direction.
To evaluate the structure change, we apply the Tanimoto similarity between each output molecule and the anchor molecule, as shown in Fig. 5(a).
Besides, for the ease of evaluating the monotonicity, we utilize a transformation (on the Tanimoto similarity) of output molecules with positive step size by taking the deduction from 2.
We call this calibrated Tanimoto similarity sequence (CTS), i.e. , { s (¯ x ′ ) } i , as shown in Fig. 5(b).
[Figure]
Sequence Monotonic Ratio (SMR) .
For evaluation, we propose a metric called Sequence Monotonic Ratio (SMR), ϕ SMR ( γ, τ ) i .
It measures the monotonic ratio of M generated sequences edited with the i -th direction.
It has two arguments: the diversity threshold γ constrains the minimum number of distinct molecules, and the tolerance threshold τ controls the non-monotonic tolerance ratio along each sequence.
In specific, for each learned semantic direction i , we will generate M sequences of edited molecules, and the calibrated Tanimoto similarity for each sequence is marked as { s (¯ x ′ ) } mi .
Then we can define the SMR on each direction as:
[Equation]
Evaluating the Diversity of Semantic Directions.
SMR can evaluate all the output sequences generated by one direction.
To better illustrate that GraphCG is able to learn multiple directions with various semantic information, we also consider taking the average of top- K SMR w.r.t.
directions to reveal that all the best K directions are semantically meaningful, as in Eq.
[Equation]
E.2 Results on Molecular Structures
Next we would like to show the comprehensive SMR on the CTS results with respect to different backbone models, as in Tables 5 and 6.
[Table]
17.3 Eq.(17) 35.
9.3 Eq.(18) 44.
[Table]
[Table] [Table]
[Table] [Table]
GraphCG-R
Eq.(16) 37.
13.3 Eq.(17) 35.
9.3 Eq.(18) 0.
[Table] [Table]
E.3 Results on Molecular Properties
By far, we have been mainly focusing on the structure change of the output sequence of molecules.
Yet, some works [50] also proved that certain key components of the molecules can be closely related to the molecular properties.
We summarize the properties into roughly three categories
[Table]
and the SMR on 8 properties with different γ , τ are listed below.
Physical properties, including water-octanol partition coefficient (LogP) [6], topological polar surface area (tPSA) [13]), and molecular weight (MW).
Drug-related molecular properties, including drug-likeness (QED) [1] and synthetic accessibility (SA) [12].
Biological properties, including three binding affinity tasks (DRD2, JNK3, GSK3 β ).
All the oracle labels are provided from the previous works [41, 28, 57].
[Table]
27.0 Eq.
[Table] [Table]
GraphCG-R
Eq.
28.0 Eq.
[Table] [Table]
[Table] [Table]
.0 Eq.
[Table] [Table]
HierVAEChEMBL
Random –
[Table] [Table]
Variance –
[Table]
SeFa –
[Table] [Table]
[Table] [Table]
[Table]
GraphCG-P
Eq.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
GraphCG-R
Eq.
[Table]
[Table] [Table]
[Table] [Table]
.0 Eq.
[Table] [Table]
[Table] [Table]
53.0 Eq.
[Table] [Table]
16.0 Variance – 24.
26.0 SeFa – 18.
GraphCG-P
Eq.
20.0 Eq.
54.0 Eq.
GraphCG-R
Eq.
16.0 Eq.
54.0 Eq.
[Table] [Table]
3.0 DisCo – 5.
GraphCG-P
Eq.
17.0 Eq.
13.0 Eq.
GraphCG-R
Eq.
12.0 Eq.
13.0 Eq.
HierVAEChEMBL
Random – 12.
15.0 Variance – 20.
21.0 SeFa – 18.
GraphCG-P
Eq.
19.0 Eq.
31.0 Eq.
[Table]
GraphCG-R
[Table] [Table]
[Table] [Table]
0.0 DisCo – 4.
GraphCG-P
Eq.
7.0 Eq.
5.0 Eq.
GraphCG-R
Eq.
4.0 Eq.
[Table] [Table]
5.0 Eq.
[Table] [Table]
[Table] [Table]
[Table] [Table]
HierVAEChEMBL
Random – 10.
[Table] [Table]
.0 Variance – 16.
18.0 SeFa – 18.
GraphCG-P
Eq.
17.0 Eq.
28.0 Eq.
[Table]
GraphCG-R
Eq.
[Table] [Table]
[Table] [Table]
[Table]
E.4 Visualization
For a more comprehensive visualization of the steerable factors in molecular graphs, we demonstrate 16 molecular graph paths along the 4 selected directions in Fig.
6, and the backbone DGM is
HierVAE pretrained on ChEMBL.
The CTS holds good monotonic trend in all these sequences.
Each direction shows certain unique changes in the molecular structures, i.e. , the steerable factors in molecules.
Some structural changes are reflected in molecular properties.
We expand all the details below.
In Fig.
6(a) and Fig.
6(b), the number of halogen atoms and hydroxyl groups (in alcohols and phenols) in the molecules decrease from left to right, respectively.
In Fig. 6(c), the number of amides in the molecules increases along the path.
As a result, the topological polar surface area (tPSA) of the molecules increase accordingly, which is a key molecular property for the prediction of drug transport properties, e.g. , permeability [13].
In Fig. 3(d), the flexible chain length, marked by the number of ethylene (CH 2 CH 2 ) units, increases from left to right.
Since the number of rotatable bonds (NRB) measures the molecular flexibility, it also increases accordingly [58].
[Figure]
how functional groups in the molecules can be viewed as the steerable factors as they change along the path, such as halogen atoms, hydroxyl groups and amides.
Fig.
6(d) illustrates the effect on the steerable factor on the length of flexible chains in the molecules.
Notebly, certain properties change together with molecular structures, like topological polar surface area (tPSA) and number of rotatable bonds (NRB).
F Results: Point Clouds
Here we compare two editing functions, i.e. , the key function design in GraphCG.
We provide the visualizations for the linear editing function in Appendix F.1, and non-linear editing function in Eq.
First we want to highlight that all the samples are generated randomly.
In the linear case, we can observe that the shape of the airplanes, cars, and chairs, are steerable using GraphCG.
We also find it interesting that GraphCG can steer more finger-trained factors, like modifying the airplane engines.
However, in the non-linear case, the diversity of the edited data is smaller.
This can been observed from the middle columns in Appendix F.2.
We will leave this for future exploration.
F.1 Linear Editing Function
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]