Original: G 3 : R EPRESENTATION L EARNING AND G ENERATION FOR G EOMETRIC G RAPHS
Revised: G 3 : R EPRESENTATION L EARNING AND G ENERATION FOR G EOMETRIC G RAPHS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: A geometric graph is a graph equipped with geometric information (i.e., node coordinates).
Revised: A geometric graph is a graph equipped with geometric information (i.e., node coordinates).

Original: A notable example is molecular graphs, where the combinatorial bonding is supplement with atomic coordinates that determine the three-dimensional structure.
Revised: A notable example is molecular graphs, where the combinatorial bonding is supplement with atomic coordinates that determine the three-dimensional structure.

Original: This work proposes a generative model for geometric graphs, capitalizing on the complementary information of structure and geometry to learn the underlying distribution.
Revised: This work proposes a generative model for geometric graphs, capitalizing on the complementary information of structure and geometry to learn the underlying distribution.

Original: The proposed model, Geometric Graph Generator (G 3 ), orchestrates graph neural networks and point cloud models in a nontrivial manner under an autoencoding framework.
Revised: The proposed model, Geometric Graph Generator (G 3 ), orchestrates graph neural networks and point cloud models in a nontrivial manner under an autoencoding framework.

Original: Additionally, we augment this framework with a normalizing ﬂow so that one can effectively sample from the otherwise intractable latent space.
Revised: Additionally, we augment this framework with a normalizing ﬂow so that one can effectively sample from the otherwise intractable latent space.

Original: G 3 can be used in computer-aided drug discovery, where seeking novel and optimal molecular structures is critical.
Revised: G 3 can be used in computer-aided drug discovery, where seeking novel and optimal molecular structures is critical.

Original: As a representation learning approach, the interaction of the graph structure and the geometric point cloud also improve signiﬁcantly the performance of downstream tasks, such as molecular property prediction.
Revised: As a representation learning approach, the interaction of the graph structure and the geometric point cloud also improve signiﬁcantly the performance of downstream tasks, such as molecular property prediction.

Original: We conduct a comprehensive set of experiments to demonstrate that Glearns more accurately the distribution of given molecules and helps identify novel molecules with better properties of interest.
Revised: We conduct a comprehensive set of experiments to demonstrate that Glearns more accurately the distribution of given molecules and helps identify novel molecules with better properties of interest.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Geometric deep learning refers to the development of deep learning techniques for data from nonEuclidean domains, such as graphs and manifolds (Bronstein et al., 2017).
Revised: Geometric deep learning refers to the development of deep learning techniques for data from nonEuclidean domains, such as graphs and manifolds (Bronstein et al., 2017).

Original: In recent years, graph neural networks (GNN) emerged to be a promising family of architectures that models the relational inductive bias ubiquitous to graph structured data (Battaglia et al., 2018;
Revised: In recent years, graph neural networks (GNN) emerged to be a promising family of architectures that models the relational inductive bias ubiquitous to graph structured data (Battaglia et al., 2018;

Original: Zhou et al., 2018).
Revised: Zhou et al., 2018).

Original: With the rise of deep generative modeling, various generative models have been adapted for graphs and proven to be effective in learning the underlying distribution implicitly deﬁned by a set of given graphs (Goodfellow et al., 2014; Kingma & Welling, 2013; Rezende & Mohamed, 2015; van den
Revised: With the rise of deep generative modeling, various generative models have been adapted for graphs and proven to be effective in learning the underlying distribution implicitly deﬁned by a set of given graphs (Goodfellow et al., 2014; Kingma & Welling, 2013; Rezende & Mohamed, 2015; van den

Original: Oord et al., 2016; De Cao & Kipf, 2018; You et al., 2018b; Jin et al., 2018; Shi et al., 2020).
Revised: Oord et al., 2016; De Cao & Kipf, 2018; You et al., 2018b; Jin et al., 2018; Shi et al., 2020).

Original: In this work, we consider a special type of graphs— geometric graphs —which are equipped with geometric information in addition to the combinatorial structure (Pach, 2013).
Revised: In this work, we consider a special type of graphs— geometric graphs —which are equipped with geometric information in addition to the combinatorial structure (Pach, 2013).

Original: In practice, this additional information corresponds to low-dimensional geometric structures and typically appears as node coordinates in R 2 or R 3 .
Revised: In practice, this additional information corresponds to low-dimensional geometric structures and typically appears as node coordinates in R 2 or R 3 .

Original: The coordinates ﬁnd broad uses in molecules (Van Aalten et al., 1996), meshes (Alliez et al., 2005), and graph drawing (Frishman & Tal, 2008).
Revised: The coordinates ﬁnd broad uses in molecules (Van Aalten et al., 1996), meshes (Alliez et al., 2005), and graph drawing (Frishman & Tal, 2008).

Original: This work aims to develop a generative model that captures both the combinatorial and the geometric characteristics exhibited in a given collection of geometric graphs.
Revised: This work aims to develop a generative model that captures both the combinatorial and the geometric characteristics exhibited in a given collection of geometric graphs.

Original: One driving application of geometric graphs is molecule generation.
Revised: One driving application of geometric graphs is molecule generation.

Original: In pharmacology, drug discovery is the process of identifying new candidate medications in the form of molecules.
Revised: In pharmacology, drug discovery is the process of identifying new candidate medications in the form of molecules.

Original: The process is known to be difﬁcult, costly, and time-consuming (Paul et al., 2010), because of the discrete nature of the search space and its vast size (estimated to contain at least 10 33 molecules)
Revised: The process is known to be difﬁcult, costly, and time-consuming (Paul et al., 2010), because of the discrete nature of the search space and its vast size (estimated to contain at least 10 33 molecules)

Original: (Polishchuk et al., 2013).
Revised: (Polishchuk et al., 2013).

Original: Recently, deep learning techniques have been developed to represent molecules as continuous vectors and apply continuous optimization (Jin et al., 2018) or reinforcement learning (You et al., 2018a) to conduct an efﬁcient search.
Revised: Recently, deep learning techniques have been developed to represent molecules as continuous vectors and apply continuous optimization (Jin et al., 2018) or reinforcement learning (You et al., 2018a) to conduct an efﬁcient search.

Original: In almost all existing work based on the molecular-graph approach, the graph is treated as a combinatorial object and the three-dimensional geometric information serves as node features only.
Revised: In almost all existing work based on the molecular-graph approach, the graph is treated as a combinatorial object and the three-dimensional geometric information serves as node features only.

Original: However, the atomic coordinates encode vital energy information of the molecule and they can be instrumental for the inference of structure and state.
Revised: However, the atomic coordinates encode vital energy information of the molecule and they can be instrumental for the inference of structure and state.

Original: Therefore, a proper modeling of the geometric information is paramount for a more accurate representation of the molecule in downstream tasks.
Revised: Therefore, a proper modeling of the geometric information is paramount for a more accurate representation of the molecule in downstream tasks.

Original: In this work, we develop an autoencoder for geometric graphs, orchestrating a GNN and a point cloud model in a nontrivial manner.
Revised: In this work, we develop an autoencoder for geometric graphs, orchestrating a GNN and a point cloud model in a nontrivial manner.

Original: Besides the straightforward use of a GNN (Duvenaud et al., 2015;
Revised: Besides the straightforward use of a GNN (Duvenaud et al., 2015;

Original: Kearnes et al., 2016; Gilmer et al., 2017) to encode the combinatorial structure, we treat the nodes with spatial coordinates as low-dimensional point clouds and use a point cloud model to encode the geometry.
Revised: Kearnes et al., 2016; Gilmer et al., 2017) to encode the combinatorial structure, we treat the nodes with spatial coordinates as low-dimensional point clouds and use a point cloud model to encode the geometry.

Original: Different from decoders merely based on graph structures in conventional graph autoencoders, on the other hand, we decode the graph structure by ﬁrst reconstructing the geometry.
Revised: Different from decoders merely based on graph structures in conventional graph autoencoders, on the other hand, we decode the graph structure by ﬁrst reconstructing the geometry.

Original: Speciﬁcally, we use a folding-based technique (Yang et al., 2018; Pang et al., 2021) to map a template of points to the correct geometry, from which the combinatorial structure is inferred.
Revised: Speciﬁcally, we use a folding-based technique (Yang et al., 2018; Pang et al., 2021) to map a template of points to the correct geometry, from which the combinatorial structure is inferred.

Original: This way, both sources of information are organically fused and processed.
Revised: This way, both sources of information are organically fused and processed.

Original: Furthermore, to fulﬁll the generative capability, we augment the autoencoder with a normalizing ﬂow (Kobyzev et al., 2019;
Revised: Furthermore, to fulﬁll the generative capability, we augment the autoencoder with a normalizing ﬂow (Kobyzev et al., 2019;

Original: Dinh et al., 2016; 2014), so that one is able to effectively sample from the otherwise intractable latent space.
Revised: Dinh et al., 2016; 2014), so that one is able to effectively sample from the otherwise intractable latent space.

Original: The proposed geometric graph generator, G 3 , addresses geometry that is rarely considered in past graph generative models.
Revised: The proposed geometric graph generator, G 3 , addresses geometry that is rarely considered in past graph generative models.

Original: A naive alternative to incorporate node coordinates is to treat them as part of features when consumed by a GNN.
Revised: A naive alternative to incorporate node coordinates is to treat them as part of features when consumed by a GNN.

Original: This approach, however, biases the geometry by the graph structure owing to local neighborhood aggregation—the global geometry may be compromised.
Revised: This approach, however, biases the geometry by the graph structure owing to local neighborhood aggregation—the global geometry may be compromised.

Original: For geometric graphs, the structure itself is combinatorial and discrete, while the geometry is continuous.
Revised: For geometric graphs, the structure itself is combinatorial and discrete, while the geometry is continuous.

Original: These two sources of information are often complementary and beneﬁt from collaborative processing.
Revised: These two sources of information are often complementary and beneﬁt from collaborative processing.

Original: Our strategy is to exploit both graph techniques and point cloud techniques to maximally retain information from the two sources.
Revised: Our strategy is to exploit both graph techniques and point cloud techniques to maximally retain information from the two sources.

Original: To evaluate the effectiveness of G 3 , we focus on molecule applications and use popular datasets (QM9 and ChEMBL) that contain atomic coordinates.
Revised: To evaluate the effectiveness of G 3 , we focus on molecule applications and use popular datasets (QM9 and ChEMBL) that contain atomic coordinates.

Original: We demonstrate that G 3 outperforms a number of representative graph generative models in generating novel and valid molecules and it also signiﬁcantly outperforms either a GNN or a point model alone for property prediction.
Revised: We demonstrate that G 3 outperforms a number of representative graph generative models in generating novel and valid molecules and it also signiﬁcantly outperforms either a GNN or a point model alone for property prediction.

Original: Appealingly,
Revised: Appealingly,

Original: G 3 decodes molecules substantially faster than do many popular models.
Revised: G 3 decodes molecules substantially faster than do many popular models.

Original: Furthermore, with the use of Bayesian optimization in concert with a trained model, we can identify better molecules under metrics of interest than do existing methods.
Revised: Furthermore, with the use of Bayesian optimization in concert with a trained model, we can identify better molecules under metrics of interest than do existing methods.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Generative models refer to a class of machine learning models that can learn the underlying distribution, implicitly deﬁned by a given set of data, and to sample from it.
Revised: Generative models refer to a class of machine learning models that can learn the underlying distribution, implicitly deﬁned by a given set of data, and to sample from it.

Original: Noteworthy generative models in the deep learning era include variational autoencoders (VAE) (Kingma & Welling, 2013), generative adversarial networks (GAN)
Revised: Noteworthy generative models in the deep learning era include variational autoencoders (VAE) (Kingma & Welling, 2013), generative adversarial networks (GAN)

Original: (Goodfellow et al., 2014) and normalizing ﬂows (NF) (Kobyzev et al., 2019;
Revised: (Goodfellow et al., 2014) and normalizing ﬂows (NF) (Kobyzev et al., 2019;

Original: Dinh et al., 2016; 2014).
Revised: Dinh et al., 2016; 2014).

Original: Among them, NF bears a unique advantage in estimating the density (likelihood).
Revised: Among them, NF bears a unique advantage in estimating the density (likelihood).

Original: For training, NF directly optimizes the likelihood, whereas VAE maximizes a lower bound of it (called the ELBO Kingma & Welling (2013)) and GAN minimizes the discrepancy between the input and the transformed noise distribution.
Revised: For training, NF directly optimizes the likelihood, whereas VAE maximizes a lower bound of it (called the ELBO Kingma & Welling (2013)) and GAN minimizes the discrepancy between the input and the transformed noise distribution.

Original: Under these generative models, a graph can be generated in a sequential or one-shot manner: the former samples nodes and edges incrementally while the latter samples a graph in its entirety.
Revised: Under these generative models, a graph can be generated in a sequential or one-shot manner: the former samples nodes and edges incrementally while the latter samples a graph in its entirety.

Original: Combining GAN and reinforcement learning, MolGAN (De Cao & Kipf, 2018) and GCPN (You et al., 2018a) generate molecules in a one-shot and sequential fashion, respectively.
Revised: Combining GAN and reinforcement learning, MolGAN (De Cao & Kipf, 2018) and GCPN (You et al., 2018a) generate molecules in a one-shot and sequential fashion, respectively.

Original: MolecularRNN (Popova et al., 2019) is an autoregressive model based on recurrent neural networks; it performs validity checks during the sequential generation process and rejects invalid structures.
Revised: MolecularRNN (Popova et al., 2019) is an autoregressive model based on recurrent neural networks; it performs validity checks during the sequential generation process and rejects invalid structures.

Original: JT-VAE (Jin et al., 2018) is a VAE-based tree model that constructs the molecular graph sequentially from sub-components.
Revised: JT-VAE (Jin et al., 2018) is a VAE-based tree model that constructs the molecular graph sequentially from sub-components.

Original: Popova et al.
Revised: Popova et al.

Original: (2019) point out that JT-VAE may suffer ambiguity in the conversion of the tree structure to a molecular graph, affecting property optimization.
Revised: (2019) point out that JT-VAE may suffer ambiguity in the conversion of the tree structure to a molecular graph, affecting property optimization.

Original: GraphNVP (Madhawa et al., 2019) and GraphAF (Shi et al., 2020) are recently proposed NF-based models that parameterize the mapping from a latent vector to a graph by using a ﬂow.
Revised: GraphNVP (Madhawa et al., 2019) and GraphAF (Shi et al., 2020) are recently proposed NF-based models that parameterize the mapping from a latent vector to a graph by using a ﬂow.

Original: The former employs one-shot sampling while the latter is sequential.
Revised: The former employs one-shot sampling while the latter is sequential.

Original: Molecules admit representations other than graphs.
Revised: Molecules admit representations other than graphs.

Original: An alternative is the simpliﬁed molecular-input line-entry system (SMILES)
Revised: An alternative is the simpliﬁed molecular-input line-entry system (SMILES)

Original: (Weininger, 1988), a string notation that universally describes molecular structures.
Revised: (Weininger, 1988), a string notation that universally describes molecular structures.

Original: Grammar VAE (GVAE) (Kusner et al., 2017) and syntax-directed VAE (SD-VAE) (Dai et al., 2018) are VAE-based models that process SMILES strings and apply grammatical rules to sequentially reconstruct the strings.
Revised: Grammar VAE (GVAE) (Kusner et al., 2017) and syntax-directed VAE (SD-VAE) (Dai et al., 2018) are VAE-based models that process SMILES strings and apply grammatical rules to sequentially reconstruct the strings.

Original: Another family of models, which are motivated by quantum chemistry principles, models molecules as point clouds and learns a physically robust representation to predict, e.g., molecular energies and equilibrium conformations.
Revised: Another family of models, which are motivated by quantum chemistry principles, models molecules as point clouds and learns a physically robust representation to predict, e.g., molecular energies and equilibrium conformations.

Original: Schütt et al.
Revised: Schütt et al.

Original: (2018) propose
Revised: (2018) propose

Original: SchNet, a network with continuous-ﬁlter convolution layers that model interactions between atoms and learn a representation that is invariant to translation and rotation.
Revised: SchNet, a network with continuous-ﬁlter convolution layers that model interactions between atoms and learn a representation that is invariant to translation and rotation.

Original: Using the same ﬁlters, Gebauer et al.
Revised: Using the same ﬁlters, Gebauer et al.

Original: (2019) devise Generative-SchNet (G-SchNet), which accurately models the target molecule distribution and generates novel 3D conformations that are relatively stable.
Revised: (2019) devise Generative-SchNet (G-SchNet), which accurately models the target molecule distribution and generates novel 3D conformations that are relatively stable.

Original: In addition to the graph structure, our work treats nodes as a point cloud, which is a collection of points in R 3 .
Revised: In addition to the graph structure, our work treats nodes as a point cloud, which is a collection of points in R 3 .

Original: Point clouds are a prominent subject in computer vision and computer graphics; they are obtained through, e.g., LIDAR scanning of object surfaces (Yang et al., 2018).
Revised: Point clouds are a prominent subject in computer vision and computer graphics; they are obtained through, e.g., LIDAR scanning of object surfaces (Yang et al., 2018).

Original: Deep learning with point clouds is faced with challenges in deﬁning the convolution operator (Bruna et al., 2013;
Revised: Deep learning with point clouds is faced with challenges in deﬁning the convolution operator (Bruna et al., 2013;

Original: Bronstein et al., 2017; Schonsheck et al., 2018; Jin et al., 2019).
Revised: Bronstein et al., 2017; Schonsheck et al., 2018; Jin et al., 2019).

Original: Volumetric CNN (Wu et al., 2015; Qi et al., 2016; Maturana & Scherer, 2015) applies convolution ﬁlters on voxels obtained from discretization of the three-dimensional space.
Revised: Volumetric CNN (Wu et al., 2015; Qi et al., 2016; Maturana & Scherer, 2015) applies convolution ﬁlters on voxels obtained from discretization of the three-dimensional space.

Original: Multiview CNN (Su et al., 2015; Qi et al., 2016) reduces point clouds to a collection of 2D images and applies gridded convolutions on them.
Revised: Multiview CNN (Su et al., 2015; Qi et al., 2016) reduces point clouds to a collection of 2D images and applies gridded convolutions on them.

Original: PointNet (Qi et al., 2017a;b) uses point-wise MLPs to featurize individual points, followed by a symmetric pooling that generates a permutation-invariant global description.
Revised: PointNet (Qi et al., 2017a;b) uses point-wise MLPs to featurize individual points, followed by a symmetric pooling that generates a permutation-invariant global description.

Original:  G EOMETRIC G RAPH G ENERATOR
Revised:  G EOMETRIC G RAPH G ENERATOR

Original: In this section, we present G 3 .
Revised: In this section, we present G 3 .

Original: The model is an autoencoder that contains interacting modules to process and reconstruct the geometry (i.e., node coordinates), the structure (i.e., edges and edge types), and additional features (e.g., node features).
Revised: The model is an autoencoder that contains interacting modules to process and reconstruct the geometry (i.e., node coordinates), the structure (i.e., edges and edge types), and additional features (e.g., node features).

Original: Such a sophisticated autoencoder allows forming a more accurate representation of the geometric graph and learning better the input distribution.
Revised: Such a sophisticated autoencoder allows forming a more accurate representation of the geometric graph and learning better the input distribution.

Original: A proper processing of the geometric and combinatorial information is crucial to the success of Gand downstream tasks.
Revised: A proper processing of the geometric and combinatorial information is crucial to the success of Gand downstream tasks.

Original: For example, an authentic representation of the node coordinates serves as the basis for decoding both the node and the edge types: the training hinges on an accurate registration between the reconstructed point cloud and the ground truth.
Revised: For example, an authentic representation of the node coordinates serves as the basis for decoding both the node and the edge types: the training hinges on an accurate registration between the reconstructed point cloud and the ground truth.

Original: The straightforward adaptation of GNNs for encoding the geometric information (as node features) results in poor coordinate reconstruction and worse generation quality overall.
Revised: The straightforward adaptation of GNNs for encoding the geometric information (as node features) results in poor coordinate reconstruction and worse generation quality overall.

Original: Moreover, one can naively reconstruct coordinates and node features with a single decoder.
Revised: Moreover, one can naively reconstruct coordinates and node features with a single decoder.

Original: However, mixing two modes of information together introduces unnecessary bias through, e.g., the difference in scales.
Revised: However, mixing two modes of information together introduces unnecessary bias through, e.g., the difference in scales.

Original: Hence, separate but dependent decoders for features and geometry render a more accurate reconstruction.
Revised: Hence, separate but dependent decoders for features and geometry render a more accurate reconstruction.

Original: 3. N OTATION
Revised: 3. N OTATION

Original: Let n be the number of nodes, d f be the number of node features, d e be the number of edge types, and d c be the dimension of the node coordinates (typically two or three).
Revised: Let n be the number of nodes, d f be the number of node features, d e be the number of edge types, and d c be the dimension of the node coordinates (typically two or three).

Original: We denote a geometric graph as
Revised: We denote a geometric graph as

Original: G =
Revised: G =

Original: ( A, X ) , where A ∈ { 0 , 1 } n × n × d e is the adjacency tensor and X =
Revised: ( A, X ) , where A ∈ { 0 , 1 } n × n × d e is the adjacency tensor and X =

Original: [ C, F ] ∈ R n × ( d c + d f ) is the node matrix, where each note vector x i
Revised: [ C, F ] ∈ R n × ( d c + d f ) is the node matrix, where each note vector x i

Original: =
Revised: =

Original: [ c i , f i ] is a concatenation of coordinates c i ∈ R 1 × d c and features f i ∈ R 1 × d f .
Revised: [ c i , f i ] is a concatenation of coordinates c i ∈ R 1 × d c and features f i ∈ R 1 × d f .

Original: For molecules, the node features are one-hot encoding of the atom type, and the edge types correspond to bond types.
Revised: For molecules, the node features are one-hot encoding of the atom type, and the edge types correspond to bond types.

Original: 3. E NCODER
Revised: 3. E NCODER

Original: The purpose of the encoder is to use a latent vector to represent both the geometry and the structure of G .
Revised: The purpose of the encoder is to use a latent vector to represent both the geometry and the structure of G .

Original: It contains two modules: a point encoder that maps the point cloud C to a geometry descriptor z p ; and a graph encoder that takes A and F as input and maps them to a structure descriptor z g .
Revised: It contains two modules: a point encoder that maps the point cloud C to a geometry descriptor z p ; and a graph encoder that takes A and F as input and maps them to a structure descriptor z g .

Original: Both descriptors are concatenated to form the latent representation z
Revised: Both descriptors are concatenated to form the latent representation z

Original: =
Revised: =

Original: [ z p , z g ] ∈ R d z .
Revised: [ z p , z g ] ∈ R d z .

Original: A usual graph encoder processes the structure information A and node features F by using a GNN.
Revised: A usual graph encoder processes the structure information A and node features F by using a GNN.

Original: When the coordinate information C is available, one could have treated C as part of the node features and feed them into the GNN, indiscriminately of F .
Revised: When the coordinate information C is available, one could have treated C as part of the node features and feed them into the GNN, indiscriminately of F .

Original: However, the geometry information of a graph is often complementary to the structure.
Revised: However, the geometry information of a graph is often complementary to the structure.

Original: Hence, a drawback of such an approach is that one heavily relies on the local neighborhoods from graph structures to parameterize the geometry, resulting in a strong bias toward the structure.
Revised: Hence, a drawback of such an approach is that one heavily relies on the local neighborhoods from graph structures to parameterize the geometry, resulting in a strong bias toward the structure.

Original: A notable feature of our design is that we separate the geometry from the structure when encoding a geometric graph; we will show an ablation study supporting this design subsequently.
Revised: A notable feature of our design is that we separate the geometry from the structure when encoding a geometric graph; we will show an ablation study supporting this design subsequently.

Original: This idea offers a means to process all node information, including features F and coordinates C , independently of the combinatorial structure, as an additional parameterization over the pure use of a
Revised: This idea offers a means to process all node information, including features F and coordinates C , independently of the combinatorial structure, as an additional parameterization over the pure use of a

Original: GNN.
Revised: GNN.

Original: In what follows, we describe the two encoder modules and defer details of the speciﬁc network architecture in the appendix.
Revised: In what follows, we describe the two encoder modules and defer details of the speciﬁc network architecture in the appendix.

Original: Point Encoder.
Revised: Point Encoder.

Original: In this module, we treat each row of C as a point and use a point cloud architecture to obtain the geometry descriptor z p .
Revised: In this module, we treat each row of C as a point and use a point cloud architecture to obtain the geometry descriptor z p .

Original: First, a multilayer perceptron (MLP) is applied point-wise to the point cloud to amplify the feature dimensions.
Revised: First, a multilayer perceptron (MLP) is applied point-wise to the point cloud to amplify the feature dimensions.

Original: The point-wise features are then passed through a series of multi-headed attention blocks (Vaswani et al., 2017) to further featurize the points.
Revised: The point-wise features are then passed through a series of multi-headed attention blocks (Vaswani et al., 2017) to further featurize the points.

Original: These attention-based blocks are more effective than simple point-wise MLPs used in PointNet (Qi et al., 2017a) as they are able to leverage global information, i.e. features on all other points, to produce rich and expressive point-wise features.
Revised: These attention-based blocks are more effective than simple point-wise MLPs used in PointNet (Qi et al., 2017a) as they are able to leverage global information, i.e. features on all other points, to produce rich and expressive point-wise features.

Original: At last, we use a global max-pooling over the points followed by an additional MLP to yield the geometry descriptor z p .
Revised: At last, we use a global max-pooling over the points followed by an additional MLP to yield the geometry descriptor z p .

Original: Graph Encoder.
Revised: Graph Encoder.

Original: This module takes the graph adjacency tensor and the node feature matrix as inputs and outputs a structure descriptor z g .
Revised: This module takes the graph adjacency tensor and the node feature matrix as inputs and outputs a structure descriptor z g .

Original: Virtually all GNNs serve the purpose; some are even specialized to molecules (e.g., Duvenaud et al.
Revised: Virtually all GNNs serve the purpose; some are even specialized to molecules (e.g., Duvenaud et al.

Original: ; Kearnes et al.
Revised: ; Kearnes et al.

Original: ; Gilmer et al.
Revised: ; Gilmer et al.

Original: Our empirical investigation ﬁnds that GINE, Graph Isomorphism Network with Edge features (Xu et al., 2018), obtains a good and stable performance for our purpose.
Revised: Our empirical investigation ﬁnds that GINE, Graph Isomorphism Network with Edge features (Xu et al., 2018), obtains a good and stable performance for our purpose.

Original: 3. D ECODER
Revised: 3. D ECODER

Original: Decoding a graph is substantially harder than encoding one.
Revised: Decoding a graph is substantially harder than encoding one.

Original: A common idea is to employ a sequence model and decode nodes and edges one by one Popova et al.
Revised: A common idea is to employ a sequence model and decode nodes and edges one by one Popova et al.

Original: Training such a model is increasingly more challenging for long sequences (i.e., large graphs).
Revised: Training such a model is increasingly more challenging for long sequences (i.e., large graphs).

Original: On the other hand, approaches decoding the adjacency tensor also face scalability challenges owing to the O ( n 2 ) degrees of freedom Samanta et al.
Revised: On the other hand, approaches decoding the adjacency tensor also face scalability challenges owing to the O ( n 2 ) degrees of freedom Samanta et al.

Original: In this work, we exploit the unique opportunity offered by a geometric graph to decode ﬁrst the geometry, then the point-wise features, and lastly the structure.
Revised: In this work, we exploit the unique opportunity offered by a geometric graph to decode ﬁrst the geometry, then the point-wise features, and lastly the structure.

Original: Speciﬁcally, we propose a three-step decoding process that decodes a graph G (cid:48)
Revised: Speciﬁcally, we propose a three-step decoding process that decodes a graph G (cid:48)

Original: =
Revised: =

Original: ( A (cid:48) , X (cid:48) ) from z , where X (cid:48) =
Revised: ( A (cid:48) , X (cid:48) ) from z , where X (cid:48) =

Original: [ C (cid:48) , F (cid:48) ] .
Revised: [ C (cid:48) , F (cid:48) ] .

Original: The ﬁrst step is to decode C (cid:48) , the coordinates for the point cloud, which involves using the latent representation z to morph a template of points to the correct geometry.
Revised: The ﬁrst step is to decode C (cid:48) , the coordinates for the point cloud, which involves using the latent representation z to morph a template of points to the correct geometry.

Original: By isolating the decoding of coordinates, we can leverage specialized architectures to achieve accurate reconstructions of the geometry, which is crucial for the subsequent decoding of F (cid:48) and A (cid:48) .
Revised: By isolating the decoding of coordinates, we can leverage specialized architectures to achieve accurate reconstructions of the geometry, which is crucial for the subsequent decoding of F (cid:48) and A (cid:48) .

Original: Secondly, the reconstructed geometry C (cid:48) is recycled along with the latent code z to predict the point-wise features F (cid:48) .
Revised: Secondly, the reconstructed geometry C (cid:48) is recycled along with the latent code z to predict the point-wise features F (cid:48) .

Original: Finally, X (cid:48)
Revised: Finally, X (cid:48)

Original: =
Revised: =

Original: [ C (cid:48) , F (cid:48) ] is used to infer the graph structure A (cid:48) through a link predictor.
Revised: [ C (cid:48) , F (cid:48) ] is used to infer the graph structure A (cid:48) through a link predictor.

Original: This procedure sidesteps the scalability challenge of both the sequential generation approach and the adjacency decoding approach.
Revised: This procedure sidesteps the scalability challenge of both the sequential generation approach and the adjacency decoding approach.

Original: In what follows, we describe each component of decoder modules and defer details of the speciﬁc network architecture in the appendix.
Revised: In what follows, we describe each component of decoder modules and defer details of the speciﬁc network architecture in the appendix.

Original: Coordinate Decoder
Revised: Coordinate Decoder

Original: The idea of decoding a point cloud from a latent vector z is to successively fold a template of points until it forms the desired geometry.
Revised: The idea of decoding a point cloud from a latent vector z is to successively fold a template of points until it forms the desired geometry.

Original: Our coordinate decoder is inspired by
Revised: Our coordinate decoder is inspired by

Original: TearingNet (Pang et al., 2021), which is a reﬁnement over FoldingNet (Yang et al., 2018; Tao, 2020), a popular archetypal architecture for point cloud reconstruction.
Revised: TearingNet (Pang et al., 2021), which is a reﬁnement over FoldingNet (Yang et al., 2018; Tao, 2020), a popular archetypal architecture for point cloud reconstruction.

Original: Note that the template design is application-dependent; an effective template is guided by the geometry to be modeled.
Revised: Note that the template design is application-dependent; an effective template is guided by the geometry to be modeled.

Original: For example, a spherical template with roughly equidistant points is more suitable for the three-dimensional structure of molecules.
Revised: For example, a spherical template with roughly equidistant points is more suitable for the three-dimensional structure of molecules.

Original: This design is different from the original TearingNet architecture that uses a regular 2D grid to model surfaces.
Revised: This design is different from the original TearingNet architecture that uses a regular 2D grid to model surfaces.

Original: Feature Decoder To decode the point-wise features F (cid:48) , we apply the folding operations with the decoded coordinates C (cid:48) from the aforementioned coordinate decoder as the starting template.
Revised: Feature Decoder To decode the point-wise features F (cid:48) , we apply the folding operations with the decoded coordinates C (cid:48) from the aforementioned coordinate decoder as the starting template.

Original: The latent code z is replicated and concatenated to C (cid:48) , which is then forwarded through point-wise MLPs to complete one set of folding.
Revised: The latent code z is replicated and concatenated to C (cid:48) , which is then forwarded through point-wise MLPs to complete one set of folding.

Original: We repeat two sets of folding to arrive at the decoded features.
Revised: We repeat two sets of folding to arrive at the decoded features.

Original: The architecture of the feature decoder is illustrated in Figure 1.
Revised: The architecture of the feature decoder is illustrated in Figure 1.

Original: Link Predictor (Graph Decoder)
Revised: Link Predictor (Graph Decoder)

Original: The link predictor determines the existence of an edge (and its type if so) for every pair of decoded nodes.
Revised: The link predictor determines the existence of an edge (and its type if so) for every pair of decoded nodes.

Original: A typical link predictor takes the inner product between two nodes to indicate the strength of connection (Trouillon et al., 2016).
Revised: A typical link predictor takes the inner product between two nodes to indicate the strength of connection (Trouillon et al., 2016).

Original: However, such a link predictor is oblivious to the geometry because a large inner product in 3D does not necessarily indicate closeness.
Revised: However, such a link predictor is oblivious to the geometry because a large inner product in 3D does not necessarily indicate closeness.

Original: Rather, the displacement of two points, their respective neighborhoods, and their neighbors’ features may all play a role in determining the existence and type of an edge.
Revised: Rather, the displacement of two points, their respective neighborhoods, and their neighbors’ features may all play a role in determining the existence and type of an edge.

Original: For example, in molecules, the bond between a pair of atoms depends on their distance as well as the neighboring atoms.
Revised: For example, in molecules, the bond between a pair of atoms depends on their distance as well as the neighboring atoms.

Original: Therefore, we augment the features of a decoded node x i from [ c i , f i ] to ˆ x i as
Revised: Therefore, we augment the features of a decoded node x i from [ c i , f i ] to ˆ x i as

Original: [Equation]
Revised: [Equation]

Original: where n 1 , ..., n k − 1 are the node indices of the sorted k -nearest neighbors of x i ; f i ∈
Revised: where n 1 , ..., n k − 1 are the node indices of the sorted k -nearest neighbors of x i ; f i ∈

Original: R d f and c i ∈ R d c are decoded features and coordinates, respectively; and the absolute sign indicates element-wise absolute values.
Revised: R d f and c i ∈ R d c are decoded features and coordinates, respectively; and the absolute sign indicates element-wise absolute values.

Original: We then use these augmented features as well as the coordinate-wise displacement |
Revised: We then use these augmented features as well as the coordinate-wise displacement |

Original: c i − c
Revised: c i − c

Original: j | to deﬁne the link predictor.
Revised: j | to deﬁne the link predictor.

Original: Formally,
Revised: Formally,

Original: [Equation]
Revised: [Equation]

Original: We highlight that our link predictor works remarkably well as a standalone module for molecules, which can be characterized as a procedure for inferring inter-atomic bonds based on molecular conformation and atom types.
Revised: We highlight that our link predictor works remarkably well as a standalone module for molecules, which can be characterized as a procedure for inferring inter-atomic bonds based on molecular conformation and atom types.

Original: To the best of our knowledge, there are no reliable public software available for this task, especially for large molecules with noise in coordinates or atom types.
Revised: To the best of our knowledge, there are no reliable public software available for this task, especially for large molecules with noise in coordinates or atom types.

Original: Therefore, our proposed link predictor has its own value other than being a part of G 3 .
Revised: Therefore, our proposed link predictor has its own value other than being a part of G 3 .

Revised: We refer appendix G for illustrating the outstanding performance of the standalone link predictor.

Original: 3. L ATENT S PACE S AMPLING : N ORMALIZING F LOW
Revised: 3. L ATENT S PACE S AMPLING : N ORMALIZING F LOW

Original: To reliably sample from G 3 ’s latent space, we make use of a normalizing ﬂow model (Kobyzev et al., 2019), which consists of a sequence of invertible transformations that map between an easily sampled base distribution (e.g., standard normal) and an unknown distribution (e.g., the latent distribution of our autoencoder).
Revised: To reliably sample from G 3 ’s latent space, we make use of a normalizing ﬂow model (Kobyzev et al., 2019), which consists of a sequence of invertible transformations that map between an easily sampled base distribution (e.g., standard normal) and an unknown distribution (e.g., the latent distribution of our autoencoder).

Original: The densities of the two distributions are related by the Jacobian of the transformations, so that log-likelihood of the unknown distribution can be computed and optimized for training.
Revised: The densities of the two distributions are related by the Jacobian of the transformations, so that log-likelihood of the unknown distribution can be computed and optimized for training.

Original: In practice, we ﬁnd that RealNVP (Dinh et al., 2016) to be a simple yet effective model for our needs.
Revised: In practice, we ﬁnd that RealNVP (Dinh et al., 2016) to be a simple yet effective model for our needs.

Original: After the training is ﬁnished, samples from the base distribution are forwarded through the normalizing ﬂow and subsequently decoded into the generated molecules.
Revised: After the training is ﬁnished, samples from the base distribution are forwarded through the normalizing ﬂow and subsequently decoded into the generated molecules.

Original: 3. O VERALL M ODEL AND T RAINING L OSS
Revised: 3. O VERALL M ODEL AND T RAINING L OSS

Original: The overall architecture of G 3 is outlined in Figure 2, which summarizes all the components described in the preceding subsections.
Revised: The overall architecture of G 3 is outlined in Figure 2, which summarizes all the components described in the preceding subsections.

Original: To train this model, we use a loss consisting of multiple terms with a tunable weighting scheme: L = λ 1 · L C + λ 2 · L F + λ 3 · L E + λ 4 · L NF + λ 5 · L R .
Revised: To train this model, we use a loss consisting of multiple terms with a tunable weighting scheme: L = λ 1 · L C + λ 2 · L F + λ 3 · L E + λ 4 · L NF + λ 5 · L R .

Original: Node coordinate loss L C .
Revised: Node coordinate loss L C .

Original: In this loss, we use only the coordinate part C contained in the node matrix X .
Revised: In this loss, we use only the coordinate part C contained in the node matrix X .

Original: In our experiments, we use the Sliced Wasserstein-2 distance (SW2) as our loss for robust point cloud reconstructions (Rabin et al., 2011).
Revised: In our experiments, we use the Sliced Wasserstein-2 distance (SW2) as our loss for robust point cloud reconstructions (Rabin et al., 2011).

Original: Note that in the calculation of L C , every point c ∈ C is identiﬁed with the nearest match c (cid:48) ∈ C (cid:48) and vice versa.
Revised: Note that in the calculation of L C , every point c ∈ C is identiﬁed with the nearest match c (cid:48) ∈ C (cid:48) and vice versa.

Original: This correspondence allows us to feasibly deﬁne the following losses L F and L E with differentiability.
Revised: This correspondence allows us to feasibly deﬁne the following losses L F and L E with differentiability.

Original: We remark that another commonly used loss for point-cloud matching, Chamfer distance (Yang et al., 2018), can also be used here.
Revised: We remark that another commonly used loss for point-cloud matching, Chamfer distance (Yang et al., 2018), can also be used here.

Original: Node feature loss L F .
Revised: Node feature loss L F .

Original: Since node correspondence is established, the difference between F and F (cid:48) can be meaningfully quantiﬁed.
Revised: Since node correspondence is established, the difference between F and F (cid:48) can be meaningfully quantiﬁed.

Original: When the rows are one-hot vectors, a cross-entropy loss is suitable; otherwise, mean absolute error (MAE) or other losses may be appropriate.
Revised: When the rows are one-hot vectors, a cross-entropy loss is suitable; otherwise, mean absolute error (MAE) or other losses may be appropriate.

Original: Edge loss L E .
Revised: Edge loss L E .

Original: The edge loss quantiﬁes whether edge types are correctly inferred.
Revised: The edge loss quantiﬁes whether edge types are correctly inferred.

Original: Hence, a crossentropy loss between A and A (cid:48) is natural, where node correspondence is needed.
Revised: Hence, a crossentropy loss between A and A (cid:48) is natural, where node correspondence is needed.

Original: Normalizing ﬂow loss L NF .
Revised: Normalizing ﬂow loss L NF .

Original: The normalizing ﬂow model is equipped with a standard negative log-likelihood loss (Dinh et al., 2016).
Revised: The normalizing ﬂow model is equipped with a standard negative log-likelihood loss (Dinh et al., 2016).

Original: Additional loss L R .
Revised: Additional loss L R .

Original: This term enforces structural and other chemical constraints innate to the speciﬁc application.
Revised: This term enforces structural and other chemical constraints innate to the speciﬁc application.

Original: Some constraints may be easily violated even if the decoding of nodes and edges is reasonably accurate (as long as error exists, however small).
Revised: Some constraints may be easily violated even if the decoding of nodes and edges is reasonably accurate (as long as error exists, however small).

Original: Hence, a regularization that penalizes the violation of the constraints helps improve semantic validity.
Revised: Hence, a regularization that penalizes the violation of the constraints helps improve semantic validity.

Original: For example, for molecules, one incorrect decoding of a carbon atom to oxygen is enough to create an invalid molecule, because the valence of oxygen is much smaller.
Revised: For example, for molecules, one incorrect decoding of a carbon atom to oxygen is enough to create an invalid molecule, because the valence of oxygen is much smaller.

Original: In this case, we follow the work by Ma et al.
Revised: In this case, we follow the work by Ma et al.

Original: and include a soft valence regularization term L R to encourage higher validity of the generated molecules.
Revised: and include a soft valence regularization term L R to encourage higher validity of the generated molecules.

Original:  E XPERIMENTS 4. D ATASETS AND B ASELINE M ODELS
Revised:  E XPERIMENTS 4. D ATASETS AND B ASELINE M ODELS

Original: We perform experiments on two benchmark datasets that come with atomic coordinate information:
Revised: We perform experiments on two benchmark datasets that come with atomic coordinate information:

Original: QM9 (Ruddigkeit et al., 2012; Ramakrishnan et al., 2014) and ChEMBL (Mendez et al., 2019; Davies et al., 2015).
Revised: QM9 (Ruddigkeit et al., 2012; Ramakrishnan et al., 2014) and ChEMBL (Mendez et al., 2019; Davies et al., 2015).

Original: The QM9 dataset contains 134k molecules with 4 atom types and up to 9 heavy atoms per molecule.
Revised: The QM9 dataset contains 134k molecules with 4 atom types and up to 9 heavy atoms per molecule.

Original: The ChEMBL dataset contains 1.5 million molecules, and the largest one has hundreds of atoms.
Revised: The ChEMBL dataset contains 1.5 million molecules, and the largest one has hundreds of atoms.

Original: For feasible experimentation, we take a subset of it that contains 250k molecules with 19 atom types and up to 39 heavy atoms per molecule.
Revised: For feasible experimentation, we take a subset of it that contains 250k molecules with 19 atom types and up to 39 heavy atoms per molecule.

Original: We name this new dataset ChEMBL250k.
Revised: We name this new dataset ChEMBL250k.

Original: Details of dataset proprocessing are provided in the appendix.
Revised: Details of dataset proprocessing are provided in the appendix.

Original: We compare G 3 with several representative architectures that feature a variety of generative approaches.
Revised: We compare G 3 with several representative architectures that feature a variety of generative approaches.

Original: SDVAE (Dai et al., 2018) is a syntax-directed VAE model that processes SMILES representation of molecules and generates syntactically correct ones.
Revised: SDVAE (Dai et al., 2018) is a syntax-directed VAE model that processes SMILES representation of molecules and generates syntactically correct ones.

Original: MolGAN (De Cao & Kipf, 2018) combines GAN and reinforcement learning to generate molecules with optimized properties.
Revised: MolGAN (De Cao & Kipf, 2018) combines GAN and reinforcement learning to generate molecules with optimized properties.

Original: JT-VAE (Jin et al., 2018) is a VAE-based model that assembles chemical substructures, boasting high validity of the generated molecules.
Revised: JT-VAE (Jin et al., 2018) is a VAE-based model that assembles chemical substructures, boasting high validity of the generated molecules.

Original: GraphNVP (Madhawa et al., 2019) is a ﬂow-based model with a one-shot generation procedure.
Revised: GraphNVP (Madhawa et al., 2019) is a ﬂow-based model with a one-shot generation procedure.

Original: GraphAF
Revised: GraphAF

Original: (Shi et al., 2020) is another ﬂow-based autoregressive model that advertises high ﬂexibility in density modeling.
Revised: (Shi et al., 2020) is another ﬂow-based autoregressive model that advertises high ﬂexibility in density modeling.

Original: G-SchNet (Gebauer et al., 2019) is a quantum-chemistry-inspired model that learns accurate conformations and sequentially generates molecules close to the trained distribution.
Revised: G-SchNet (Gebauer et al., 2019) is a quantum-chemistry-inspired model that learns accurate conformations and sequentially generates molecules close to the trained distribution.

Original: 4. G ENERATION Q UALITY
Revised: 4. G ENERATION Q UALITY

Original: Metrics and processing.
Revised: Metrics and processing.

Original: To evaluate the generative capability of different models, we measure the validity, uniqueness, and novelty (V, U, N) on their generated molecules.
Revised: To evaluate the generative capability of different models, we measure the validity, uniqueness, and novelty (V, U, N) on their generated molecules.

Original: Validity is the percentage of molecules that pass RDKit’s sanitization check.
Revised: Validity is the percentage of molecules that pass RDKit’s sanitization check.

Original: Uniqueness is the portion of valid molecules that are distinct.
Revised: Uniqueness is the portion of valid molecules that are distinct.

Original: Novelty is the portion of valid molecules that do not appear in the training set.
Revised: Novelty is the portion of valid molecules that do not appear in the training set.

Original: In addition, we include metrics computed between the test and the generated molecules via MOSES (Polykovskiy et al., 2020): Frechet ChemNet Distance (FCD), which measures the distance between the two sets according to ChemNet’s penultimate layer’s activation; Fragment similarity (FRAG) measures the similarity in fragments present in both sets of molecules; Internal Diversity (IntDiv2) assesses the diversity within the generated molecules; LogP, QED, and Weight are the 1D Wasserstein distances between the two sets on different chemical properties: LogP measures the lipophilicity of a compound (Bhal, 2007); QED is the quantitative estimate of druglike-ness (Bickerton et al., 2012);
Revised: In addition, we include metrics computed between the test and the generated molecules via MOSES (Polykovskiy et al., 2020): Frechet ChemNet Distance (FCD), which measures the distance between the two sets according to ChemNet’s penultimate layer’s activation; Fragment similarity (FRAG) measures the similarity in fragments present in both sets of molecules; Internal Diversity (IntDiv2) assesses the diversity within the generated molecules; LogP, QED, and Weight are the 1D Wasserstein distances between the two sets on different chemical properties: LogP measures the lipophilicity of a compound (Bhal, 2007); QED is the quantitative estimate of druglike-ness (Bickerton et al., 2012);

Original: Weight is the sum of atomic weights in a molecule.
Revised: Weight is the sum of atomic weights in a molecule.

Original: All scores are computed on 10k generated molecules.
Revised: All scores are computed on 10k generated molecules.

Original: In our experiments, G 3 sometimes generates disconnected molecules, for which we treat the largest connected component as the ﬁnal generated molecule (breaking ties arbitrarily) for evaluation.
Revised: In our experiments, G 3 sometimes generates disconnected molecules, for which we treat the largest connected component as the ﬁnal generated molecule (breaking ties arbitrarily) for evaluation.

Original: Additionally, we observe the normalizing ﬂow cannot capture the autoencoder’s latent distribution perfectly when trained on ChEMBL250k.
Revised: Additionally, we observe the normalizing ﬂow cannot capture the autoencoder’s latent distribution perfectly when trained on ChEMBL250k.

Original: Therefore, we interpolate between samples from the normalizing ﬂow and the nearest encoded training data before decoding.
Revised: Therefore, we interpolate between samples from the normalizing ﬂow and the nearest encoded training data before decoding.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Results and discussions.
Revised: Results and discussions.

Original: Results reported in Tables 1 and 2 invite a few observations.
Revised: Results reported in Tables 1 and 2 invite a few observations.

Original: Overall, we see that G 3 has strong performances across all metrics in both QM9 and ChEMBL250k.
Revised: Overall, we see that G 3 has strong performances across all metrics in both QM9 and ChEMBL250k.

Original: Notably,
Revised: Notably,

Original: G 3 ’s low FCD score indicates accurate modeling of the training distribution, while a high novelty rate means G 3 has not simply memorized the training set.
Revised: G 3 ’s low FCD score indicates accurate modeling of the training distribution, while a high novelty rate means G 3 has not simply memorized the training set.

Original: This is important because a model can always reproduce the training set to get perfect scores on e.g. FCD.
Revised: This is important because a model can always reproduce the training set to get perfect scores on e.g. FCD.

Original: In contrast, although JT-VAE has better FCD and LogP scores on QM9, its low novelty is characteristic of overﬁtting.
Revised: In contrast, although JT-VAE has better FCD and LogP scores on QM9, its low novelty is characteristic of overﬁtting.

Original: In addition, G 3 ’s validity scores are consistently the highest among the models with no hard valency checks (the others have 100% validity), and G 3 ’s high uniqueness and molecular diversity suggest it is able to avoid mode collapse, which is evident in MolGAN.
Revised: In addition, G 3 ’s validity scores are consistently the highest among the models with no hard valency checks (the others have 100% validity), and G 3 ’s high uniqueness and molecular diversity suggest it is able to avoid mode collapse, which is evident in MolGAN.

Original: Furthermore, G 3 ’s robust performance in LogP, QED, and Weight implies that G 3 ’s generated molecules are not only faithful to the training set in terms of (sub)structure, but also with respect to various chemical properties.
Revised: Furthermore, G 3 ’s robust performance in LogP, QED, and Weight implies that G 3 ’s generated molecules are not only faithful to the training set in terms of (sub)structure, but also with respect to various chemical properties.

Original: Efﬁciency.
Revised: Efﬁciency.

Original: G 3 ’s performance on large molecular datasets is especially impressive considering its one-shot generation approach and is far superior to other models in the same camp (GraphNVP,
Revised: G 3 ’s performance on large molecular datasets is especially impressive considering its one-shot generation approach and is far superior to other models in the same camp (GraphNVP,

Original: MolGAN).
Revised: MolGAN).

Original: Comparing to the best sequential models, e.g. JT-VAE, G 3 can generate molecules as good in quality in a much faster fashion, thanks to its one-shot nature.
Revised: Comparing to the best sequential models, e.g. JT-VAE, G 3 can generate molecules as good in quality in a much faster fashion, thanks to its one-shot nature.

Original: When benchmarked on a machine with i7-8700K CPU, GeForce 1080 RTX GPU, and 16GB RAM, GraphAF and JT-VAE took 4804.36s and 2642.97s to generate 10k molecules when trained on ChEMBL250k, respectively.
Revised: When benchmarked on a machine with i7-8700K CPU, GeForce 1080 RTX GPU, and 16GB RAM, GraphAF and JT-VAE took 4804.36s and 2642.97s to generate 10k molecules when trained on ChEMBL250k, respectively.

Original: In contrast, G 3 took 47.79s, which is approximately 55 times faster than JT-VAE and 100 times faster than GraphAF.
Revised: In contrast, G 3 took 47.79s, which is approximately 55 times faster than JT-VAE and 100 times faster than GraphAF.

Original: As discussed, the inefﬁciency of sequential generation models is mainly due to the challenge of parallelization.
Revised: As discussed, the inefﬁciency of sequential generation models is mainly due to the challenge of parallelization.

Original: Latent space interpolation.
Revised: Latent space interpolation.

Original: Lastly, we present examples of G 3 ’s generated molecules as well as their local latent space in Figure 3.
Revised: Lastly, we present examples of G 3 ’s generated molecules as well as their local latent space in Figure 3.

Original: The surrounding molecules are decoded along two random orthogonal axes starting from the center molecule highlighted by the blue box.
Revised: The surrounding molecules are decoded along two random orthogonal axes starting from the center molecule highlighted by the blue box.

Original: The transition between molecules are gradual and chemically sound, which illustrates G 3 ’s great capability to smoothly parameterize discrete molecules in its latent space.
Revised: The transition between molecules are gradual and chemically sound, which illustrates G 3 ’s great capability to smoothly parameterize discrete molecules in its latent space.

Original: More qualitative examples can be found in the appendix.
Revised: More qualitative examples can be found in the appendix.

Original: 4. A BLATION S TUDY AND P ROPERTY P REDICTION
Revised: 4. A BLATION S TUDY AND P ROPERTY P REDICTION

Original: To process the complementary geometry and structure information, we use a pair of point encoder and graph encoder for G 3 .
Revised: To process the complementary geometry and structure information, we use a pair of point encoder and graph encoder for G 3 .

Original: A natural question asks if two encoders are really necessary.
Revised: A natural question asks if two encoders are really necessary.

Original: Here, we conduct an ablation study and compare G 3 with two simpler models.
Revised: Here, we conduct an ablation study and compare G 3 with two simpler models.

Original: The ﬁrst one uses only the point encoder and removes the graph encoder.
Revised: The ﬁrst one uses only the point encoder and removes the graph encoder.

Original: Because of a lack of graph structure encoding, the edge decoding quality entirely depends on node information.
Revised: Because of a lack of graph structure encoding, the edge decoding quality entirely depends on node information.

Original: The second simpliﬁed model is to remove the point encoder but retain the graph encoder.
Revised: The second simpliﬁed model is to remove the point encoder but retain the graph encoder.

Original: Since the graph encoder takes coordinates as part of the input, the geometry information is still encoded, albeit through a GNN module that was not designed to handle geometry in the ﬁrst place.
Revised: Since the graph encoder takes coordinates as part of the input, the geometry information is still encoded, albeit through a GNN module that was not designed to handle geometry in the ﬁrst place.

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: We train the models on QM9 until convergence using the same hyperparameters and latent dimensions.
Revised: We train the models on QM9 until convergence using the same hyperparameters and latent dimensions.

Original: It is clear that both ablated models are inferior to G 3  in nearly all molecular metrics,  which attests to the efﬁcacy of  G 3 ’s overall architecture.
Revised: From Table 3, G 3 ’s encoder achieves the best results in two-thirds of the metrics, and the remaining third is split between the other approaches. We further highlight the combined encoder’s convincing lead in FCD, which is the best single indicator of the quality of generated molecules in the absence of overﬁtting. As a result, we ﬁnd G 3 ’s combined approach to be clearly favored.

Original: To further indicate that the combination of geometry and graph information used in G 3 learn a meaningful latent representation for the molecules, we show downstream tasks for predicting chemical properties.
Revised: To further indicate that the combination of geometry and graph information used in G 3 learn a meaningful latent representation for the molecules, we show downstream tasks for predicting chemical properties.

Original: In the subsequent experiment, we show that a G 3 encoder pretrained in a unsupervised manner can be used to predict chemical properties of QM9 molecules with an additional MLP to ﬁt desired properties.
Revised: In the subsequent experiment, we show that a G 3 encoder pretrained in a unsupervised manner can be used to predict chemical properties of QM9 molecules with an additional MLP to ﬁt [Table]

Original:  Alternatively, one can also train G 3 ’s encoder in a supervised manner to predict the desired properties, which yield better results.
Revised: desired properties. Alternatively, one can also train G 3 ’s encoder in a supervised manner to predict the desired properties, which yield better results.

Original: The pretrained models are ﬁrst trained on the QM[Table] dataset in an unsupervised manner for 80 epochs.
Revised: The pretrained models are ﬁrst trained on the QM dataset in an unsupervised manner for 80 epochs.

Original: The weights on the encoders are then frozen, and an additional MLP is trained on the encoded vectors until convergence.
Revised: The weights on the encoders are then frozen, and an additional MLP is trained on the encoded vectors until convergence.

Original: The supervised models are trained on the QM9 dataset until convergence.
Revised: The supervised models are trained on the QM9 dataset until convergence.

Original: We predict four properties that come with the QM9 dataset: Homo-Lumo energy (H-L), internal energy, free energy, and heat capacity.
Revised: We predict four properties that come with the QM9 dataset: Homo-Lumo energy (H-L), internal energy, free energy, and heat capacity.

Original: In all cases, G 3 ’s graph-point encoder signiﬁcantly outperforms the standalone graph encoder used in section 3.2.
Revised: In all cases, G 3 ’s graph-point encoder signiﬁcantly outperforms the standalone graph encoder used in section 3.2.

Original: This further substantiates G 3 ’s ability to leverage the geometric and combinatorial features in a geometric graph more effectively by separating the graph and the point encoder.
Revised: This further substantiates G 3 ’s ability to leverage the geometric and combinatorial features in a geometric graph more effectively by separating the graph and the point encoder.

Original: 4. P ROPERTY O PTIMIZATION
Revised: 4. P ROPERTY O PTIMIZATION

Original: One use of generative models with latent spaces is to search for better molecules under metrics of interest.
Revised: One use of generative models with latent spaces is to search for better molecules under metrics of interest.

Original: Following the approach outlined by Kusner et al.
Revised: Following the approach outlined by Kusner et al.

Original: , we apply Bayesian optimization (BO) on the G 3 model pretrained from Table 1 and 2 to optimize two metrics: penalized logP and
Revised: , we apply Bayesian optimization (BO) on the G 3 model pretrained from Table 1 and 2 to optimize two metrics: penalized logP and

Original: QED.
Revised: QED.

Original: Penalized logP is logP less the synthetic accessibility score and the number of long cycles (Bhal, 2007), and QED measures how likely a compound is to be a drug (Bickerton et al., 2012).
Revised: Penalized logP is logP less the synthetic accessibility score and the number of long cycles (Bhal, 2007), and QED measures how likely a compound is to be a drug (Bickerton et al., 2012)

Original: [Table]
Revised: [Table]

Original: The results for QM9 and ChEMBL250k are reported in Table 5.
Revised: The results for QM9 and ChEMBL250k are reported in Table 5.

Original: One sees that G 3 consistently ﬁnds the best molecule under both metrics.
Revised: One sees that G 3 consistently ﬁnds the best molecule under both metrics.

Original: As is observed, the penalized logP metric favors carbon chains while the QED metric favors more complex structures, which is consistent with common knowledge of molecular structures.

Original: We remark that BO works better for low dimensional search spaces (Frazier, 2018).
Revised: We remark that BO works better for low dimensional search spaces (Frazier, 2018).

Original: The latent spaces of SDVAE and JT-VAE are of a similar nature to that of G [Table] but GraphAF must use a latent space of the same dimension as the input space.
Revised: The latent spaces of SDVAE and JT-VAE are of a similar nature to that of G 3 , but GraphAF must use a latent space of the same dimension as the input space.

Original: Hence, BO may not be the best method for GraphAF.
Revised: Hence, BO may not be the best method for GraphAF.

Original: Indeed, Shi et al.
Revised: Indeed, Shi et al.

Original: (2020) employ reinforcement learning to chase for optimal metric values and the reported results therein are higher than those reported here.
Revised: (2020) employs reinforcement learning to chase for optimal metric values, which gives more promising results on ZINC250k.

Original: However, for fairness of comparison, we opt to use BO for all methods.
Revised: However, for fairness of comparison, we opt to use BO for all methods.

Original: More details are provided in the appendix.
Revised: More details are provided in the appendix.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have presented a generative model G 3 for geometric graphs that are equipped with additional geometry information.
Revised: We have presented a generative model G 3 for geometric graphs that are equipped with additional geometry information.

Original: Geometric graphs appear in many practical applications, a notable one being molecules, whose three-dimensional structure is rarely captured by existing graph generative models.
Revised: Geometric graphs appear in many practical applications, a notable one being molecules, whose three-dimensional structure is rarely captured by existing graph generative models.

Original: Our model is an autoencoder that employs both GNN architectures and point cloud architectures to handle the complementary structure and geometry information.
Revised: Our model is an autoencoder that employs both GNN architectures and point cloud architectures to handle the complementary structure and geometry information.

Original: Moreover, a normalizing ﬂow is inserted into the latent space so that one can effectively sample new graphs.
Revised: Moreover, a normalizing ﬂow is inserted into the latent space so that one can effectively sample new graphs.

Original: Experiment results show that G 3 is able to more accurately learn the distribution of given graphs and better predict molecular properties.
Revised: Experiment results show that G 3 is able to more accurately learn the distribution of given graphs and better predict molecular properties.

Original: It helps discover novel molecules with better properties than do many representative graph generative models.
Revised: It helps discover novel molecules with better properties than do many representative graph generative models.

Original: A E XPERIMENTAL D ETAILS
Revised: A E XPERIMENTAL D ETAILS

Original: Datasets We preprocess each molecule by centering it based on the 3D coordinates and aligning its orientation with the principal axes via PCA.
Revised: Datasets We preprocess each molecule by centering it based on the 3D coordinates and aligning its orientation with the principal axes via PCA.

Original: The molecules in the datasets have different number of atoms.
Revised: The molecules in the datasets have different number of atoms.

Original: Hence, we pad each graph with “phantom nodes” to reach the maximum molecule size in the dataset.
Revised: Hence, we pad each graph with “phantom nodes” to reach the maximum molecule size in the dataset.

Original: The phantom nodes are indicated by one of the node features, and we also treat nonexistence of edges as an edge type.
Revised: The phantom nodes are indicated by one of the node features, and we also treat nonexistence of edges as an edge type.

Original: Importantly, the phantom nodes need artiﬁcial coordinates.
Revised: Importantly, the phantom nodes need artiﬁcial coordinates.

Original: We cannot naively allocate all the phantom nodes to the origin, because overlapping coordinates cause non-unique correspondences in the point-cloud registration, making the learning of atom and edge types impossible.
Revised: We cannot naively allocate all the phantom nodes to the origin, because overlapping coordinates cause non-unique correspondences in the point-cloud registration, making the learning of atom and edge types impossible.

Original: In practice, we use an extra coordinate dimension and assign the phantom coordinates for the i -th phantom node as (0 , 0 , i ) or (0 , 0 , 0 , i ) if the original point cloud is in 2D or 3D, respectively.
Revised: In practice, we use an extra coordinate dimension and assign the phantom coordinates for the i -th phantom node as (0 , 0 , i ) or (0 , 0 , 0 , i ) if the original point cloud is in 2D or 3D, respectively.

Original: In addition, we make a distinction between each atom type and its aromatic counterpart whenever possible.
Revised: In addition, we make a distinction between each atom type and its aromatic counterpart whenever possible.

Original: For example, a carbon atom may be part of an aromatic ring, in which case we mark its atom type as aromatic carbon to distinguish it from an ordinary carbon atom.
Revised: For example, a carbon atom may be part of an aromatic ring, in which case we mark its atom type as aromatic carbon to distinguish it from an ordinary carbon atom.

Original: This facilitates the reconstruction of ring structures in the link predictor, as it can predict aromatic bonds easily by checking if the atom types are aromatic.
Revised: This facilitates the reconstruction of ring structures in the link predictor, as it can predict aromatic bonds easily by checking if the atom types are aromatic.

Original: Note that another popularly used benchmark, ZINC250k
Revised: Note that another popularly used benchmark, ZINC250k

Original: (Sterling & Irwin, 2015), does not contain coordinate information and hence we omit it.
Revised: (Sterling & Irwin, 2015), does not contain coordinate information and hence we omit it.

Original: However, the scales of ChEMBL250k and ZINC250k are rather close, and their results unsurprisingly share similar trends when cross-checked against the performance of baseline models.
Revised: However, the scales of ChEMBL250k and ZINC250k are rather close, and their results unsurprisingly share similar trends when cross-checked against the performance of baseline models.

Original: Models For G 3 , we use the same architecture and identical number of layers and hidden dimensions on both QM9 and ChEMBL250k, without separate ﬁne tuning.
Revised: Models For G 3 , we use the same architecture and identical number of layers and hidden dimensions on both QM9 and ChEMBL250k, without separate ﬁne tuning.

Original: Such a practice helps corroborate robustness of the model under different data size and complexity.
Revised: Such a practice helps corroborate robustness of the model under different data size and complexity.

Original: The latent dimension used for
Revised: The latent dimension used for

Original: QM9 is [ d p , d g ] =
Revised: QM9 is [ d p , d g ] =

Original: [10 , 30] , and [35 , 25] for ChEMBL250k.
Revised: [10 , 30] , and [35 , 25] for ChEMBL250k.

Original: The additional graph latent dimensions is required to accurately decode the diverse atom types present in ChEMBL250k, which is more than twice than that in QM9.
Revised: The additional graph latent dimensions is required to accurately decode the diverse atom types present in ChEMBL250k, which is more than twice than that in QM9.

Original: [Table]
Revised: [Table]

Original: We did not check G-SchNet’s performance on ChEMBL250k for two reasons.
Revised: We did not check G-SchNet’s performance on ChEMBL250k for two reasons.

Original: First, G-SchNet excels at reconstructing and generating conformers, which are the 3D structures of molecules that are crucial for various computations in quantum chemistry.
Revised: First, G-SchNet excels at reconstructing and generating conformers, which are the 3D structures of molecules that are crucial for various computations in quantum chemistry.

Original: The coordinates of ChEMBL250k, however, come from molecular graphs and are all in 2D. Second, G-SchNet’s sequential generation process utilizes all previously placed atoms to predict the position of the new atom at each step, which scales poorly with respect to the size of molecules.
Revised: The coordinates of ChEMBL250k, however, come from molecular graphs and are all in 2D. Second, G-SchNet’s sequential generation process utilizes all previously placed atoms to predict the position of the new atom at each step, which scales poorly with respect to the size of molecules.

Original: As many ChEMBL250k molecules have more thanatoms after adding Hydrogens, the computational resource required to train a G-SchNet model on
Revised: As many ChEMBL250k molecules have more thanatoms after adding Hydrogens, the computational resource required to train a G-SchNet model on

Original: ChEMBL250k is likely to be prohibitively high.
Revised: ChEMBL250k is likely to be prohibitively high.

Original: All baseline results are obtained by running the implementation provided by the authors, with default hyperparameters.
Revised: All baseline results are obtained by running the implementation provided by the authors, with default hyperparameters.

Original: The baseline results generally match the reported ones from the original papers.
Revised: The baseline results generally match the reported ones from the original papers.

Original: One discrepancy occurs on GraphAF with QM9, which is caused by the disabling of a ﬁlter that ignores small molecules.
Revised: One discrepancy occurs on GraphAF with QM9, which is caused by the disabling of a ﬁlter that ignores small molecules.

Original: Turning off the ﬁlter amounts to a more fair comparison because small molecules are still sensible results.
Revised: Turning off the ﬁlter amounts to a more fair comparison because small molecules are still sensible results.

Original: The V, U, N metrics for MolGAN on QM9 are also somewhat off, which is likely a symptom of GAN’s training instability.
Revised: The V, U, N metrics for MolGAN on QM9 are also somewhat off, which is likely a symptom of GAN’s training instability.

Original: Code G 3 is implemented with the Pytorch framework.
Revised: Code G 3 is implemented with the Pytorch framework.

Original: All relevant code as well as the trained models will be made available upon publication.
Revised: All relevant code as well as the trained models will be made available upon publication.

Original: B A RCHITECTURE D ETAILS
Revised: B A RCHITECTURE D ETAILS

Original: In this section, we specify the architecture used to produce the reported experiment results.
Revised: In this section, we specify the architecture used to produce the reported experiment results.

Original: Let
Revised: Let

Original: MLP ( n, h, o, l ) denote a (point-wise) multi-layer perceptron that has input dimension n , hidden dimension h , output dimension o , and number of hidden layers l .
Revised: MLP ( n, h, o, l ) denote a (point-wise) multi-layer perceptron that has input dimension n , hidden dimension h , output dimension o , and number of hidden layers l .

Original: Let MHA ( n, h ) be the multiheaded attention block with embedding dimension n and h heads.
Revised: Let MHA ( n, h ) be the multiheaded attention block with embedding dimension n and h heads.

Original: The implementation of multiheaded attention is borrowed from Paszke et al.
Revised: The implementation of multiheaded attention is borrowed from Paszke et al.

Original: Point Encoder As illustrated in Figure 4, the G 3 point encoder is :
Revised: Point Encoder As illustrated in Figure 4, the G 3 point encoder is :

Original: MLP ( n, 256 , 768 , 1) → MHA (256 , 3) → MHA (256 , 3) → GlobalMaxPooling → MLP (768 , 256 , d p , 1)
Revised: MLP ( n, 256 , 768 , 1) → MHA (256 , 3) → MHA (256 , 3) → GlobalMaxPooling → MLP (768 , 256 , d p , 1)

Original: Graph Encoder Let GINE ( h, l ) denote a GINE kernel (Xu et al., 2018) with embedding dimension h and l hidden layers in the MLP.
Revised: Graph Encoder Let GINE ( h, l ) denote a GINE kernel (Xu et al., 2018) with embedding dimension h and l hidden layers in the MLP.

Original: Let Set2Set be a Set2Set global pooling layer (Vinyals et al., 2015).
Revised: Let Set2Set be a Set2Set global pooling layer (Vinyals et al., 2015).

Original: We ﬁrst use linear layers to embed the node and edge features to 256 dimensional vectors.
Revised: We ﬁrst use linear layers to embed the node and edge features to 256 dimensional vectors.

Original: The graph encoder is
Revised: The graph encoder is

Original: GINE (256 , 3) → GINE (256 , 3) → GINE (256 , 3) → Set2Set → MLP (512 , 256 , d g , 2)
Revised: GINE (256 , 3) → GINE (256 , 3) → GINE (256 , 3) → Set2Set → MLP (512 , 256 , d g , 2)

Original: A Batchnorm (Ioffe & Szegedy, 2015) layer is applied after every GINE convolution.
Revised: A Batchnorm (Ioffe & Szegedy, 2015) layer is applied after every GINE convolution.

Original: Coordinate Decoder This module is inspired by (Pang et al., 2021).
Revised: Coordinate Decoder This module is inspired by (Pang et al., 2021).

Original: Let U be the template with the i -th row denoded as u i , z
Revised: Let U be the template with the i -th row denoded as u i , z

Original: =
Revised: =

Original: [ z p , z g ] be the latent vector, and Y (cid:48) be the output with the i -th row denoded as y (cid:48) i .
Revised: [ z p , z g ] be the latent vector, and Y (cid:48) be the output with the i -th row denoded as y (cid:48) i .

Original: Deﬁne the FoldingNet F , where F ( U, z )
Revised: Deﬁne the FoldingNet F , where F ( U, z )

Original: =
Revised: =

Original: Y (cid:48) .
Revised: Y (cid:48) .

Original: It performs the following operation for each point i :
Revised: It performs the following operation for each point i :

Original: [Equation]
Revised: [Equation]

Original: Deﬁne the TearingNet T , where T ( U, X, z ) =
Revised: Deﬁne the TearingNet T , where T ( U, X, z ) =

Original: Y (cid:48) .
Revised: Y (cid:48) .

Original: It performs the following operation for each point i :
Revised: It performs the following operation for each point i :

Original: [Equation]
Revised: [Equation]

Original: The coordinate decoder does the following operations:
Revised: The coordinate decoder does the following operations:

Original: [Equation]
Revised: [Equation]

Original: Where C (cid:48) with the i -th row as c (cid:48)
Revised: Where C (cid:48) with the i -th row as c (cid:48)

Original: i are the decoded coordinates.
Revised: i are the decoded coordinates.

Original: The MLPs all have two hidden layers with hidden dimensions 1024.
Revised: The MLPs all have two hidden layers with hidden dimensions 1024.

Original: Feature Decoder Let C (cid:48) be the decoded coordinates with the i -th row denoded as c (cid:48)
Revised: Feature Decoder Let C (cid:48) be the decoded coordinates with the i -th row denoded as c (cid:48)

Original: i , and z =
Revised: i , and z =

Original: [ z p , z g ] be the latent vector.
Revised: [ z p , z g ] be the latent vector.

Original: Using the same FoldingNet notation as above, the feature decoder does the following:
Revised: Using the same FoldingNet notation as above, the feature decoder does the following:

Original: [Equation]
Revised: [Equation]

Original: Where F (cid:48) with the i -th row as f (cid:48)
Revised: Where F (cid:48) with the i -th row as f (cid:48)

Original: i are the decoded features.
Revised: i are the decoded features.

Original: The MLPs all have two hidden layers with hidden dimensions 1024.
Revised: The MLPs all have two hidden layers with hidden dimensions 1024.

Original: Link Predictor Let d x =
Revised: Link Predictor Let d x =

Original: ( d a + d c ) · k ) − d c .
Revised: ( d a + d c ) · k ) − d c .

Original: The MLP inside the link predictor is
Revised: The MLP inside the link predictor is

Original: MLP ( d x , 1024 , d e , 3 ) .
Revised: MLP ( d x , 1024 , d e , 3 ) .

Original: C H YPERPARAMETERS
Revised: C H YPERPARAMETERS

Original: Experiment on generation quality G 3 : For QM9, the latent vector z =
Revised: Experiment on generation quality G 3 : For QM9, the latent vector z =

Original: [ z p , z g ] has dimension 10 + 30 and the loss weighting is ( λ 1 , λ 2 , λ 3 , λ 4 , λ 5 ) =
Revised: [ z p , z g ] has dimension 10 + 30 and the loss weighting is ( λ 1 , λ 2 , λ 3 , λ 4 , λ 5 ) =

Original: (10 , 1 , 1 , 10 − 3 , 10) .
Revised: (10 , 1 , 1 , 10 − 3 , 10) .

Original: For ChEMBL250k, the latent dimension is 35 + 25 and the loss weighting is (10 , 0 .
Revised: For ChEMBL250k, the latent dimension is 35 + 25 and the loss weighting is (10 , 0 .

Original: 1 , 9 , 10 − 4 , 5) .
Revised: 1 , 9 , 10 − 4 , 5) .

Original: GraphAF:
Revised: GraphAF:

Original: In this model, there is an option to ﬁlter out generated molecules with fewer than m atoms.
Revised: In this model, there is an option to ﬁlter out generated molecules with fewer than m atoms.

Original: Since no other models use ﬁltering, we set m = 0 for a fair comparison.
Revised: Since no other models use ﬁltering, we set m = 0 for a fair comparison.

Original: Because larger molecules are more likely to be novel and unique, the scores reported by the original authors are higher than those we obtain.
Revised: Because larger molecules are more likely to be novel and unique, the scores reported by the original authors are higher than those we obtain.

Original: Moreover, we have tried best efforts to tune hyperparameters for QM9 as the authors did not release their setting.
Revised: Moreover, we have tried best efforts to tune hyperparameters for QM9 as the authors did not release their setting.

Original: Other models: We use the default hyperparameters released by the authors for QM9.
Revised: Other models: We use the default hyperparameters released by the authors for QM9.

Original: For
Revised: For

Original: ChEMBL250k, we borrow those used for ZINC250k.
Revised: ChEMBL250k, we borrow those used for ZINC250k.

Original: We ﬁnd this to be a reasonable approach because ZINC250k and ChEMBL250k have the same number of molecules, and the maximum number of atoms per molecule is also quite close: 38 and 39, respectively.
Revised: We ﬁnd this to be a reasonable approach because ZINC250k and ChEMBL250k have the same number of molecules, and the maximum number of atoms per molecule is also quite close: 38 and 39, respectively.

Original: As such, it is expected that the resulting performance on these two datasets is similar, which is indeed the case.
Revised: As such, it is expected that the resulting performance on these two datasets is similar, which is indeed the case.

Original: Experiment on property optimization In each iteration of the Bayesian optimization, a Gaussian process is used to ﬁt the explored data, and a batch of new candidates are suggested based on the use of an acquisition function.
Revised: Experiment on property optimization In each iteration of the Bayesian optimization, a Gaussian process is used to ﬁt the explored data, and a batch of new candidates are suggested based on the use of an acquisition function.

Original: For all models, we use training molecules as the initial set of points and perform ﬁve iterations of Bayesian optimization, wherein the acquisition function is EI (expected improvement).
Revised: For all models, we use training molecules as the initial set of points and perform ﬁve iterations of Bayesian optimization, wherein the acquisition function is EI (expected improvement).

Original: In each iteration, 60 candidates are generated.
Revised: In each iteration, 60 candidates are generated.

Original: The implementation is provided by Kusner et al.
Revised: The implementation is provided by Kusner et al.

Original: (2017) via a custom build of Theano.
Revised: (2017) via a custom build of Theano.

Original: To conduct
Revised: To conduct

Original: Bayesian optimization on the baseline models, we use the default hyperparameters released by the authors whenever available, and follow the practice above otherwise.
Revised: Bayesian optimization on the baseline models, we use the default hyperparameters released by the authors whenever available, and follow the practice above otherwise.

Original: D L OSS D ETAILS
Revised: D L OSS D ETAILS

Original: For completeness, in what follows we give the detailed expressions for all the training losses.
Revised: For completeness, in what follows we give the detailed expressions for all the training losses.

Original: They are generally straightforward or have been deﬁned in the literature.
Revised: They are generally straightforward or have been deﬁned in the literature.

Original: D. C OORDINATE L OSS
Revised: D. C OORDINATE L OSS

Original: The Wasserstein distance is intimately connected with the study of Optimal Transport.
Revised: The Wasserstein distance is intimately connected with the study of Optimal Transport.

Original: Let (Ω , F ) be a measurable space, and P (Ω) be the set of all probability measures on (Ω , F ) .
Revised: Let (Ω , F ) be a measurable space, and P (Ω) be the set of all probability measures on (Ω , F ) .

Original: Deﬁne two probability measures µ, ν ∈ P 2 (Ω) , where P 2 (Ω) = { µ ∈ P (Ω) : (cid:82) Ω (cid:107) x (cid:107) 2 µ ( dx ) < ∞} .
Revised: Deﬁne two probability measures µ, ν ∈ P 2 (Ω) , where P 2 (Ω) = { µ ∈ P (Ω) : (cid:82) Ω (cid:107) x (cid:107) 2 µ ( dx ) < ∞} .

Original: The Wassersteindistance W 2 ( µ, ν ) is deﬁned as:
Revised: The Wassersteindistance W 2 ( µ, ν ) is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: Where C ( µ, ν ) is the set of all couplings between µ, ν that satisﬁes: γ ( A × Ω) =
Revised: Where C ( µ, ν ) is the set of all couplings between µ, ν that satisﬁes: γ ( A × Ω) =

Original: µ ( A ) , ∀ A ∈ F , γ (Ω × B ) = µ ( B ) , ∀ B ∈ F , and F is the sigma algebra associated with Ω .
Revised: µ ( A ) , ∀ A ∈ F , γ (Ω × B ) = µ ( B ) , ∀ B ∈ F , and F is the sigma algebra associated with Ω .

Original: It is known that W 2 ( µ, ν ) deﬁnes a metric on P (Ω) (Ambrosio et al., 2008).
Revised: It is known that W 2 ( µ, ν ) deﬁnes a metric on P (Ω) (Ambrosio et al., 2008).

Original: In general, W 2 ( µ, ν ) is difﬁcult to compute, especially in high dimensions.
Revised: In general, W 2 ( µ, ν ) is difﬁcult to compute, especially in high dimensions.

Original: However, it has a simple form in one-dimension:
Revised: However, it has a simple form in one-dimension:

Original: [Equation]
Revised: [Equation]

Original: Where F µ , F ν denote the cumulative distribution functions associated with µ, ν , respectively.
Revised: Where F µ , F ν denote the cumulative distribution functions associated with µ, ν , respectively.

Original: The simple one-dimensional form of W 2 ( µ, ν ) inspires the usage of sliced Wasserstein distances, which is the expected value of the projected one-dimensional Wasserstein distance.
Revised: The simple one-dimensional form of W 2 ( µ, ν ) inspires the usage of sliced Wasserstein distance , which is the expected value of the projected one-dimensional Wasserstein distance.

Original: Deﬁne the push-forward operation ∗ as: f ∗ µ ( A ) = µ ( f − 1 ( A )) , ∀ A ∈ F , and the projection map as Θ( x ) =
Revised: Deﬁne the push-forward operation ∗ as: f ∗ µ ( A ) = µ ( f − 1 ( A )) , ∀ A ∈ F , and the projection map as Θ( x ) =

Original: (cid:104) θ, x (cid:105) for a normalized direction θ .
Revised: (cid:104) θ, x (cid:105) for a normalized direction θ .

Original: Formally, the sliced Wasserstein 2 distance SW 2 ( µ, ν ) in d -dimension is:
Revised: Formally, the sliced Wasserstein-2 distance SW 2 ( µ, ν ) in d -dimension is:

Original: [Equation]
Revised: [Equation]

Original: where dθ is the uniform measure on S d − 1 .
Revised: where dθ is the uniform measure on S d − 1 .

Original: Bonnotte (2013) has shown that SW 2( µ, ν ) also deﬁnes a metric on P (Ω) , and it induces the same topology as W 2( µ, ν ) when Ω is compact.
Revised: Bonnotte (2013) has shown that SW 2 ( µ, ν ) also deﬁnes a metric on P (Ω) , and it induces the same topology as W 2 ( µ, ν ) when Ω is compact.

Original: In practice, we compute SW 2( µ, ν ) by approximating the integral via Monte Carlo sampling directions on the sphere.
Revised: In practice, we compute SW 2 ( µ, ν ) by approximating the integral via Monte Carlo sampling directions on the sphere.

Original: We then compute the projection of the point clouds onto the sampled directions, sort the one-dimensional projections, and average their differences.
Revised: We then compute the projection of the point clouds onto the sampled directions, sort the one-dimensional projections, and average their differences.

Original: Through experimentation, we found that 200directions is an appropriate number of directions to sample.D.  [Equation]
Revised: Through experimentation, we found that 200 is an appropriate number of directions to sample.

Original:  ODE 
Revised: D. N ODE F EATURE L OSS L F AND E DGE L OSS L E

Original: [Equation] EATURE L OSS L F AND E DGE L OSS L E

Original: Let C ∈ R n × d c be the original point cloud, where each row c i :
Revised: Let C ∈ R n × d c be the original point cloud, where each row c i :

Original: = C i, : gives the coordinates of a point.
Revised: = C i, : gives the coordinates of a point.

Original: Similarly, let
Revised: Similarly, let

Original: C (cid:48) ∈
Revised: C (cid:48) ∈

Original: R n × d c be the reconstructed point cloud with c (cid:48) i :
Revised: R n × d c be the reconstructed point cloud with c (cid:48) i :

Original: =
Revised: =

Original: C (cid:48) i, : .
Revised: C (cid:48) i, : .

Original: Deﬁne the permutation matrix σ as:
Revised: Deﬁne the permutation matrix σ as:

Original: [Equation]
Revised: [Equation]

Original: Let F, F be the original and the reconstructed node feature matrices, respectively.
Revised: Let F, F be the original and the reconstructed node feature matrices, respectively.

Original: The node feature loss is deﬁned as:
Revised: The node feature loss is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where F (cid:48)
Revised: where F (cid:48)

Original: =
Revised: =

Original: σF is the reconstructed node feature matrix that is aligned with F ; w N is the weighting vector for each coordinate; and H ( x, y, w ) is the weighted cross entropy between probability vectors x and y :
Revised: σF is the reconstructed node feature matrix that is aligned with F ; w N is the weighting vector for each coordinate; and H ( x, y, w ) is the weighted cross entropy between probability vectors x and y :

Original: [Equation]
Revised: [Equation]

Original: Let A, A be the original and the reconstructed adjacency tensors, respectively.
Revised: Let A, A be the original and the reconstructed adjacency tensors, respectively.

Original: The edge loss is deﬁned as:
Revised: The edge loss is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where A (cid:48) is the reconstructed adjacency tensor that is aligned with A , satisfying A (cid:48) : , : ,k = σA : , : ,k σ T for k = 1 , 2 , ..., d e ; and w E is the weighting vector for the edge classes.
Revised: where A (cid:48) is the reconstructed adjacency tensor that is aligned with A , satisfying A (cid:48) : , : ,k = σA : , : ,k σ T for k = 1 , 2 , ..., d e ; and w E is the weighting vector for the edge classes.

Original: Note the exclusion of j = i in the inner summation; self-loops are not counted.
Revised: Note the exclusion of j = i in the inner summation; self-loops are not counted.

Original: We use the following weighting for the cross entropies:
Revised: We use the following weighting for the cross entropies:

Original: [Equation]
Revised: [Equation]

Original: where D is a probability vector that gives the empirical distribution of classes computed from data; and the operations (cid:11) and ln are element-wise.
Revised: where D is a probability vector that gives the empirical distribution of classes computed from data; and the operations (cid:11) and ln are element-wise.

Original: D. N ORMALIZNG F LOW L OSS L NF
Revised: D. N ORMALIZNG F LOW L OSS L NF

Original: Let z ∈ R D be a random vector with density
Revised: Let z ∈ R D be a random vector with density

Original: p z ( z ) and let x be a vector resulting from a sequence of
Revised: p z ( z ) and let x be a vector resulting from a sequence of

Original: K invertible transformations from z :
Revised: K invertible transformations from z :

Original: [Equation]
Revised: [Equation]

Original: Deﬁne z 

Revised: Deﬁne z

Original: = z , z i =
Revised: = z , z i =

Original: f i ( z i − 1 ) for i = 1 , ..., K , and z K = x .
Revised: f i ( z i − 1 ) for i = 1 , ..., K , and z K = x .

Original: By the change-of-variable formula, the density of x can be expressed as
Revised: By the change-of-variable formula, the density of x can be expressed as

Original: [Equation]
Revised: [Equation]

Original: If z is standard normal
Revised: If z is standard normal

Original: [Equation]
Revised: [Equation]

Original: then
Revised: then

Original: [Equation]
Revised: [Equation]

Original: In our work
Revised: In our work

Original: [Equation]
Revised: [Equation]

Original: we use RealNVP Dinh et al.
Revised: we use RealNVP Dinh et al.

Original: (2016) to parameterize the mappings.
Revised: (2016) to parameterize the mappings.

Original: Speciﬁcally, a mapping f ( u ) = v is deﬁned as
Revised: Speciﬁcally, a mapping f ( u ) = v is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where d = 12 D .
Revised: where d = 12 D .

Original: One easily calculates that the Jacobian of f is
Revised: One easily calculates that the Jacobian of f is

Original: [Equation]
Revised: [Equation]

Original: Therefore, det
Revised: Therefore, det

Original: [Equation]
Revised: [Equation]

Original: Substituting equation 2 and equation 3 into equation 1, we obtain the negative log-likelilhood loss
Revised: Substituting equation 2 and equation 3 into equation 1, we obtain the negative log-likelilhood loss

Original: [Equation]
Revised: [Equation]

Original: D. A DDITIONAL L OSS L R
Revised: D. A DDITIONAL L OSS L R

Original: We follow the approach outlined in Ma et al.
Revised: We follow the approach outlined in Ma et al.

Original: (2018) to introduce a soft valence regularization to penalize the excess of valence capacity for each atom, thereby encouraging the decoder to output chemically valid molecules.
Revised: (2018) to introduce a soft valence regularization to penalize the excess of valence capacity for each atom, thereby encouraging the decoder to output chemically valid molecules.

Original: In particular, deﬁne L R = (cid:80) ni
Revised: In particular, deﬁne L R = (cid:80) ni

Original: =1 σ ( V ( i ) − U ( i )) for each molecule, where σ is the ReLU function and U ( i ) and V ( i ) are the valence and capacity of atom i , respectively.
Revised: =1 σ ( V ( i ) − U ( i )) for each molecule, where σ is the ReLU function and U ( i ) and V ( i ) are the valence and capacity of atom i , respectively.

Original: The soft (thus differentiable) version is:
Revised: The soft (thus differentiable) version is:

Original: [Equation]
Revised: [Equation]

Original: where F (cid:48) and A (cid:48) are the reconstructed and aligned node feature matrix and adjacency tensor, respectively.
Revised: where F (cid:48) and A (cid:48) are the reconstructed and aligned node feature matrix and adjacency tensor, respectively.

Original: Here, u ( r ) is the valence of node type r , determined from data; and h ( k ) is the capacity for edge type k .
Revised: Here, u ( r ) is the valence of node type r , determined from data; and h ( k ) is the capacity for edge type k .

Original: Speciﬁcally, h ( k )
Revised: Speciﬁcally, h ( k )

Original: = 1 , 2 , 3 , 1 . 5 for single, double, triple, and aromatic bonds, respectively.
Revised: = 1 , 2 , 3 , 1 . 5 for single, double, triple, and aromatic bonds, respectively.

Original: Moreover, u ( r ) and h ( k ) are both 0 for phantom nodes and edges.
Revised: Moreover, u ( r ) and h ( k ) are both 0 for phantom nodes and edges.

Original: E G 3 P ROPERTIES
Revised: E G 3 P ROPERTIES

Original: A typical pitfall for many graph-based models is the ambiguity under node permutations.
Revised: A typical pitfall for many graph-based models is the ambiguity under node permutations.

Original: Reordering the nodes does not change the graph itself, but the latent representation of the graph may change if not properly modeled.
Revised: Reordering the nodes does not change the graph itself, but the latent representation of the graph may change if not properly modeled.

Original: Moreover, a single latent vector may not be able to decode an adjacency tensor with an arbitrary node permutation.
Revised: Moreover, a single latent vector may not be able to decode an adjacency tensor with an arbitrary node permutation.

Original: Thanks to the geometry, our model G 3 is able to identify node correspondence and maintain desirable permutation and translation invariance.
Revised: Thanks to the geometry, our model G 3 is able to identify node correspondence and maintain desirable permutation and translation invariance.

Original: Proposition E.1.
Revised: Proposition E.1.

Original: The G 3 encoder and link predictor are invariant to input node permutations as well as translations on the coordinates.
Revised: The G 3 encoder and link predictor are invariant to input node permutations as well as translations on the coordinates.

Original: Proof.
Revised: Proof.

Original: The G 3 encoder is composed of a point encoder and a graph encoder.
Revised: The G 3 encoder is composed of a point encoder and a graph encoder.

Original: The point encoder is invariant to node permutations because the MLP is weight sharing for all points and the global pooling is permutation invariant.
Revised: The point encoder is invariant to node permutations because the MLP is weight sharing for all points and the global pooling is permutation invariant.

Original: The graph encoder is also invariant to node permutations because the graph convolution layers of MoNet are.
Revised: The graph encoder is also invariant to node permutations because the graph convolution layers of MoNet are.

Original: The overall latent vector is a concatenation of the outputs of these two encoders and thus is also invariant to node permutations.
Revised: The overall latent vector is a concatenation of the outputs of these two encoders and thus is also invariant to node permutations.

Original: Additionally, the point clouds are preprocessed so that each is centered at the origin, invariant to translations.
Revised: Additionally, the point clouds are preprocessed so that each is centered at the origin, invariant to translations.

Original: The link predictor takes pairwise nodes as input and is therefore invariant to node permutations.
Revised: The link predictor takes pairwise nodes as input and is therefore invariant to node permutations.

Original: In addition, node coordinates are used always in the form of displacements.
Revised: In addition, node coordinates are used always in the form of displacements.

Original: Hence, the link prediction is always unchanged regardless of translation.
Revised: Hence, the link prediction is always unchanged regardless of translation.

Original: F Q UALITATIVE E XAMPLES
Revised: F Q UALITATIVE E XAMPLES

Original: In Figure 5, we show the generated examples for QM9 as well as results of latent space interpolation.
Revised: In Figure 5, we show the generated examples for QM9 as well as results of latent space interpolation.

Original: They correspond to the results reported in Table 1.
Revised: They correspond to the results reported in Table 1.

Original: The generated samples and interpolation results for ChEMBL250k are in Figure 6, which correspond to the results reported in Table 2. 
Revised: The generated samples and interpolation results for ChEMBL250k are in Figure 6, which correspond to the results reported in Table 2. [Figure]

Revised: G E XPERIMENTS ON THE S TANDALONE L INK P REDICTOR

Original: For each dataset, the top subﬁgure gives randomly generated molecules and the bottom subﬁgure visualizes the local latent space around a training molecule.
Revised: In this section [Table] we empirically demonstrate the effectiveness of our link predictor as a standalone model. The task is to predict the edge types for all edges in the graph given the ground truth node types and coordinates for all nodes.

Revised: We train the link predictor on QM9 and ChEMBL250k until convergence, then report the accuracy and the F-1 score on the test set.

Revised: The classiﬁcation reports are computed by using scikit-learn, where the "Z ERO " type corresponds to the non-existence of a bond.

Revised: In addition, we provide visualized comparisons between the ground truth molecules and the reconstructed molecules with ground truth nodes and predicted edges

Revised: [Table]

Original: The surrounding new molecules are decoded along two random orthogonal axes starting from the center molecule highlighted by the blue box.
Revised: [Table] We remark that while link prediction is pairwise on nodes [Table] the inclusion of neighbor information allows it to faithfully reconstruct complex structures like aromatic rings.

Revised: Overall, G 3 ’s link predictor attains near perfect aggregate F-1 score and accuracy on both QM9 and ChEMBL250k.

Revised: It therefore serves as a viable standalone model for bond prediction in molecules

Original: [Figure]
Revised: [Table]

Revised: H V ISUALIZATION OF G EOMETRIC F EATURES

Revised: In this section, we provide qualitative examples for the learned geometric features in the point encoder.

Revised: Similar to Wang et al.

Revised: (2021), we present the channel-wise activation before the global pooling step in the point encoder of G 3 pretrained on ChEMBL250k.

Revised: Intuitively, the difference in the level of activation reveals the geometric primitives that the neurons focus on.

Revised: In general, we observe that G 3 ’s point encoder can identify aromatic rings in molecules based purely on geometry; i.e. no bond information is provided.

Revised: For example, Figure 9 shows that the 63rd neuron identiﬁes rings to the right end of the molecules (see the red atoms).

Revised: In addition, different neurons can partition a molecule into meaningful sub-components.

Revised: In Figure 10, we show neurons that are excited by the ﬁve-member ring on the left, the benzene ring in the middle, and the benzene ring on the right, respectively.

Revised: In Figure 11, the top ﬁgure presents a neuron that identiﬁes two rings on the right as a group, whereas the bottom ﬁgures showcase more discerning neurons that distinguish the two rings separately.