Original: T ENSOR VAE: A DIRECT GENERATIVE MODEL FOR MOLECULAR CONFORMATION GENERATION DRIVEN BY NOVEL FEATURE ENGINEERING
Revised: T ENSOR VAE: A DIRECT GENERATIVE MODEL FOR MOLECULAR CONFORMATION GENERATION DRIVEN BY NOVEL FEATURE ENGINEERING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Efficient generation of 3D conformations of a molecule from its 2D graph is a key challenge in in-silico drug discovery.
Revised: Efficient generation of 3D conformations of a molecule from its 2D graph is a key challenge in in-silico drug discovery.

Original: Deep learning (DL) based generative modeling has recently become a potent tool to tackling this challenge.
Revised: Deep learning (DL) based generative modeling has recently become a potent tool to tackling this challenge.

Original: However, many existing DL-based methods are either indirect-leveraging inter-atomic distances or direct-but requiring numerous sampling steps to generate conformations.
Revised: However, many existing DL-based methods are either indirect-leveraging inter-atomic distances or direct-but requiring numerous sampling steps to generate conformations.

Original: In this work, we propose a simple model abbreviated TensorVAE capable of generating conformations directly from a 2D molecular graph in a single step.
Revised: In this work, we propose a simple model abbreviated TensorVAE capable of generating conformations directly from a 2D molecular graph in a single step.

Original: The main novelty of the proposed method is focused on feature engineering .
Revised: The main novelty of the proposed method is focused on feature engineering .

Original: We develop a novel encoding and feature extraction mechanism relying solely on standard convolution operation to generate token-like feature vector for each atom.
Revised: We develop a novel encoding and feature extraction mechanism relying solely on standard convolution operation to generate token-like feature vector for each atom.

Original: These feature vectors are then transformed through standard transformer encoders under a conditional Variational Auto Encoder framework for learning to generate conformations directly.
Revised: These feature vectors are then transformed through standard transformer encoders under a conditional Variational Auto Encoder framework for learning to generate conformations directly.

Original: We show through experiments on two benchmark datasets that with intuitive feature engineering, a relatively simple and standard model can provide promising generative capability rivalling recent state-of-the-art models employing more sophisticated and specialized generative architecture.
Revised: We show through experiments on two benchmark datasets that with intuitive feature engineering, a relatively simple and standard model can provide promising generative capability rivalling recent state-of-the-art models employing more sophisticated and specialized generative architecture.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent advance in deep learning has enabled significant progress in computational drug design (Chen et al., 2018).
Revised: Recent advance in deep learning has enabled significant progress in computational drug design (Chen et al., 2018).

Original: Particularly, capable graph-based generative models have been proposed to generate valid 2D graph representation of novel drug-like molecules (Honda et al., 2019; Mahmood et al., 2021; Yu & Yu, 2022), and there is an increasing interest on extending these methods to generating 3D molecular structures which are essential for structured-based drug discovery (Li et al., 2021; Simm et al., 2021; Gebauer et al., 2022).
Revised: Particularly, capable graph-based generative models have been proposed to generate valid 2D graph representation of novel drug-like molecules (Honda et al., 2019; Mahmood et al., 2021; Yu & Yu, 2022), and there is an increasing interest on extending these methods to generating 3D molecular structures which are essential for structured-based drug discovery (Li et al., 2021; Simm et al., 2021; Gebauer et al., 2022).

Original: A stable 3D structure or conformation of a molecule is specified by the 3D Cartesian coordinates of all its atoms.
Revised: A stable 3D structure or conformation of a molecule is specified by the 3D Cartesian coordinates of all its atoms.

Original: Traditional molecular dynamics or statistical mechanic driven Monte Carlo methods are computationally expensive, making them unviable for generating 3d molecular structures at scale (Hawkins, 2017).
Revised: Traditional molecular dynamics or statistical mechanic driven Monte Carlo methods are computationally expensive, making them unviable for generating 3d molecular structures at scale (Hawkins, 2017).

Original: In this regard, deep learning(DL)-based generative methods have become an attractive alternative.
Revised: In this regard, deep learning(DL)-based generative methods have become an attractive alternative.

Original: DL-based generative methods may be broadly classified into three categories: distance-based, reconstruction-based, and direct methods.
Revised: DL-based generative methods may be broadly classified into three categories: distance-based, reconstruction-based, and direct methods.

Original: The main goal of distance-based methods is learning a probability distribution over the inter-atomic distances.
Revised: The main goal of distance-based methods is learning a probability distribution over the inter-atomic distances.

Original: During inference, distance matrices are sampled from the learned distribution and converted to valid 3D conformations through postprocessing algorithms.
Revised: During inference, distance matrices are sampled from the learned distribution and converted to valid 3D conformations through postprocessing algorithms.

Original: Two representative methods of this category include GraphDG (Simm &
Revised: Two representative methods of this category include GraphDG (Simm &

Original: Hern´andez-Lobato, 2019) and CGCF (Xu et al., 2021a).
Revised: Hern´andez-Lobato, 2019) and CGCF (Xu et al., 2021a).

Original: An advantage of modeling distance is its roto-translation invariance which is an important inductive bias for molecular geometry modeling (K¨ohler et al., 2020).
Revised: An advantage of modeling distance is its roto-translation invariance–an important inductive bias for molecular geometry modeling (K¨ohler et al., 2020).

Original: Additional virtual edges and their distances between 2 nd and 3 rd neighbors are often introduced to constrain bond angles and dihedral angles crucial to generating a valid conformation.
Revised: Additional virtual edges and their distances between 2 nd and 3 rd neighbors are often introduced to constrain bond angles and dihedral angles crucial to generating a valid conformation.

Original: However, they may still be indequate to capture structural relationship between distant atoms in a long-sequence molecule.
Revised: However, Luo et al. (2021) have argued that these additional bonds are still inadequate to capture structural relationship between distant atoms.

Original:  DGSM (Luo et al., 2021) attempted alleviating to some extent this issue by adding higher-order virtual bonds between atoms in an expanded neighborhood region.
Revised: To alleviate this issue, DGSM (Luo et al., 2021) proposed to add higher-order virtual bonds between atoms in an expanded neighborhood region.

Original: Another weakness of the distance-based methods is the error accumulation problem; random noise in the predicted distance can be exaggerated by an Euclidean Distance Geometry algorithm, leading to generation of inaccurate conformations (Xu et al., 2022; 2021b).
Revised: Another weakness of the distance-based methods is the error accumulation problem; random noise in the predicted distance can be exaggerated by an Euclidean Distance Geometry algorithm, leading to generation of inaccurate conformations (Xu et al., 2022; 2021b).

Original: To address the above issues, reconstruction-based methods directly model a distribution over 3D coordinates.
Revised: To address the above weaknesses, reconstruction-based methods directly model a distribution over 3D coordinates.

Original: Their main idea is to reconstruct valid conformations from distorted coordinates.
Revised: Their main idea is to reconstruct valid conformations from distorted coordinates.

Original: GeoDiff (Xu et al., 2022) and Uni-Mol (Zhou et al., 2022) are pioneering studies in this respect.
Revised: GeoDiff (Xu et al., 2022) and Uni-Mol (Zhou et al., 2022) are pioneering studies in this respect.

Original: Though sharing similar idea, they differ in the process of transforming corrupted coordinates to stable conformations.
Revised: Though sharing similar idea, they differ in the process of transforming corrupted coordinates to stable conformations.

Original: While GeoDiff adapted a reverse diffusion process (Sohl-Dickstein et al., 2015), Uni-Mol treated conformation reconstruction as an optimization problem.
Revised: While GeoDiff adapted a reverse diffusion process (Sohl-Dickstein et al., 2015), Uni-Mol treated conformation reconstruction as an optimization problem.

Original: Despite their promising performance, both methods require design of task-specific and complex coordinate transformation method.
Revised: Despite their promising performance, both methods require design of task-specific and complex coordinate transformation method.

Original: This is to ensure the transformation is roto-transition or SE(3)-equivariant.
Revised: This is to ensure the transformation is roto-transition or SE(3)-equivariant.

Original: To achieve this, GeoDiff proposed a specialized SE(3)-equivariant Markov transition kernel.
Revised: To achieve this, GeoDiff proposed a specialized SE(3)-equivariant Markov transition kernel.

Original: On the other hand, Uni-Mol accomplished the same by combining a task-specific adaption of transformer (Vaswani et al., 2017)
Revised: On the other hand, Uni-Mol accomplished the same by combining a task-specific adaption of transformer (Vaswani et al., 2017)

Original: called Evoformer (Jumper et al., 2021) with another specialized equivariant graph neural network (Satorras et al., 2021).
Revised: called Evoformer (Jumper et al., 2021) with another specialized equivariant graph neural network (Satorras et al., 2021).

Original: Furthermore, GeoDiff requires numerous diffusing steps to attain satisfactory generative performance which can be time consuming.
Revised: Furthermore, GeoDiff requires numerous diffusing steps to attain satisfactory generative performance which can be time consuming.

Original: CVGAE (Mansimov et al., 2019) and DMCG (Zhu et al., 2022) have attempted to resolve the generative efficiency issue by developing models that can produce a valid conformation directly from a 2D molecular graph in a single sampling step.
Revised: CVGAE (Mansimov et al., 2019) and DMCG (Zhu et al., 2022) have attempted to resolve the generative efficiency issue by developing models that can produce a valid conformation directly from a 2D molecular graph in a single sampling step.

Original: Regrettably, the performance of CVGAE is significantly worse than its distance-based counterparts mainly due to the use of inferior graph neural network for information aggregation (Zhu et al., 2022).
Revised: Regrettably, the performance of CVGAE is significantly worse than its distance-based counterparts mainly due to the use of inferior graph neural network for information aggregation (Zhu et al., 2022).

Original: DMCG aimed to improve the performance of its predecessor by using a more sophisticated graph neural network and a loss function invariant to symmetric permutation of molecular substructures.
Revised: DMCG aimed to improve the performance of its predecessor by using a more sophisticated graph neural network and a loss function invariant to symmetric permutation of molecular substructures.

Original: Although DMCG achieved superior performance, acquiring such loss function requires enumerating all permutations of a molecular graph, which can become computationally expensive for long-sequence molecules.
Revised: Although DMCG achieved superior performance, acquiring such loss function requires enumerating all permutations of a molecular graph, which can become computationally expensive for long-sequence molecules.

Original: Regardless of their category, a common recipe of success for these models can be distilled to developing model architecture with ever increasing sophistication and complexity.
Revised: Regardless of their category, a common recipe of success for these models can be distilled to developing model architecture with ever increasing sophistication and complexity.

Original: There is little attention on input feature engineering.
Revised: There is little attention on input feature engineering.

Original: In this work, we forgo building specialized model architectures but instead focus on intuitive input feature engineering.
Revised: In this work, we forgo building specialized model architectures but instead focus on intuitive input feature engineering.

Original: We propose to encode a molecular graph using a fully connected and symmetric tensor.
Revised: We propose to encode a molecular graph using a fully connected and symmetric tensor.

Original: For preliminary information aggregation, we run a rectangle kernel filter through the tensor in a 1D convolution manner.
Revised: For preliminary information aggregation, we run a rectangle kernel filter through the tensor in a 1D convolution manner.

Original: This operation has a profound implication; with a filter size of 3, the information from two immediate neighbors as well as all their connected atoms can be aggregated onto the focal atom in a single operation.
Revised: This operation has a profound implication; with a filter size of 3, the information from two immediate neighbors as well as all their connected atoms can be aggregated onto the focal atom in a single operation.

Original: It also generates tokenlike feature vector per atom which can be directly consumed by a standard transformer encoder for further information aggregation.
Revised: It also generates tokenlike feature vector per atom which can be directly consumed by a standard transformer encoder for further information aggregation.

Original: The generative framework follows the standard conditional variational auto encoder (CVAE) setup.
Revised: The generative framework follows the standard conditional variational auto encoder (CVAE) setup.

Original: We start with building two input tensors with one encoding only the 2D molecular graph and the other also encoding 3D coordinate and distance.
Revised: We start with building two input tensors with one encoding only the 2D molecular graph and the other also encoding 3D coordinate and distance.

Original: Both tensors go through the same feature engineering step and the generated feature vectors are fed through two separate transformer encoders.
Revised: Both tensors go through the same feature engineering step and the generated feature vectors are fed through two separate transformer encoders.

Original: The output of these two encoders are then combined in an intuitive way to form the input for another transformer encoder for generating conformation directly.
Revised: The output of these two encoders are then combined in an intuitive way to form the input for another transformer encoder for generating conformation directly.

Original: The complete generative model is abbreviated as TensorVAE.
Revised: The complete generative model is abbreviated as TensorVAE.

Original: In summary, the proposed method has three main advantages.
Revised: In summary, the proposed method has three main advantages.

Original: (1) Direct and Efficient , generating conformation direclty from a 2D molecular graph in a single step.
Revised: (1) Direct and Efficient , generating conformation direclty from a 2D molecular graph in a single step.

Original: (2) Simple , not requiring tasksepecific design of neural network architecture, relying only on simple convolution and off-the-shelf transformer architecture; (3) Easy to implement , no custom module required as both PyTorch and
Revised: (2) Simple , not requiring tasksepecific design of neural network architecture, relying only on simple convolution and off-the-shelf transformer architecture; (3) Easy to implement , no custom module required as both PyTorch and

Original: TensorFlow offer ready-to-use convolution and transformer implementation.
Revised: TensorFlow offer ready-to-use convolution and transformer implementation.

Original: These advantages translate directly to excellent practicality of the TensorVAE method.
Revised: These advantages translate directly to excellent practicality of the TensorVAE method.

Original: We demonstrate through extensive experiments on two benchmark datasets that the proposed TensorVAE, despite its simplicity, can perform competitively against 18 recent state-of-the-art methods for conformation generation and molecular property prediction.
Revised: We demonstrate through extensive experiments on two benchmark datasets that the proposed TensorVAE, despite its simplicity, can perform competitively against 18 recent state-of-the-art methods for conformation generation and molecular property prediction.

Original:  M ETHOD 2. P RELIMINARIES
Revised:  M ETHOD 2. P RELIMINARIES

Original: Problem Definition.
Revised: Problem Definition.

Original: We formulate molecular conformation generation as a conditional generation task.
Revised: We formulate molecular conformation generation as a conditional generation task.

Original: Given a set of molecular graphs G and their corresponding i.i.d conformations R , the goal is to train a generative model that approximates the Boltzman distribution, and from which a valid conformation conditioned on a molecular graph can be easily sampled in a single step.
Revised: Given a set of molecular graphs G and their corresponding i.i.d conformations R , the goal is to train a generative model that approximates the Boltzman distribution, and from which a valid conformation conditioned on a molecular graph can be easily sampled in a single step.

Original: Story Line.
Revised: Story Line.

Original: In the ensuing sections, we breakdown the formulation of the proposed method in three novel ideas.
Revised: In the ensuing sections, we breakdown the formulation of the proposed method in three novel ideas.

Original: We first introduce how the tensor encoding method is derived.
Revised: We first introduce how the tensor encoding method is derived.

Original: Based on the tensor input, we propose a “naive” model that applies a convolutional neural network directly on the tensor to generate distance matrix, from which conformation are obtained through an Euclidean Distance
Revised: Based on the tensor input, we propose a “naive” model that applies a convolutional neural network directly on the tensor to generate distance matrix, from which conformation are obtained through an Euclidean Distance

Original: Geometry algorithm.
Revised: Geometry algorithm.

Original: We also show that such model can already provide comparable performance to several advanced distance-based methods.
Revised: We also show that such model can already provide comparable performance to several advanced distance-based methods.

Original: Subsequently, we demonstrate how token-like feature vector can be generated from the input tensor by using 1D convolution operation.
Revised: Subsequently, we demonstrate how token-like feature vector can be generated from the input tensor by using 1D convolution operation.

Original: Finally, we elaborate on how to combine all the components together under a conditional Variational Auto Encoder framework to arrive at the final generative model.
Revised: Finally, we elaborate on how to combine all the components together under a conditional Variational Auto Encoder framework to arrive at the final generative model.

Original: 2. I NPUT TENSOR GRAPH
Revised: 2. I NPUT TENSOR GRAPH

Original: Graph neural network (GNN) is a popular feature extraction backbone for DL-based molecular conformation generation.
Revised: Graph neural network (GNN) is a popular feature extraction backbone for DL-based molecular conformation generation.

Original: The input of GNN is composed of three components, including atom feature vectors, edge feature vectors and an adjacency matrix.
Revised: The input of GNN is composed of three components, including atom features, edge features and an adjacency matrix.

Original: Atom and edge feature normally pass through separate embedding step before being fed to GNN.
Revised: Atom and edge features normally pass through separate embedding step before being fed to GNN.

Original: Adjacency matrix is then used to determine neighboring atoms for layer-wise information aggregation.
Revised: Adjacency matrix is then used to determine neighboring atoms for layer-wise information aggregation.

Original: Although bond features are aggregated onto atom features and vice versa, these two features are maintained separately throughout the message passing layers (Gilmer et al., 2017;
Revised: Although bond features are aggregated onto atom features and vice versa, these two features are maintained separately throughout the message passing layers (Gilmer et al., 2017;

Original: Satorras et al., 2021).
Revised: Satorras et al., 2021).

Original: Instead of having separate inputs, our first simple idea is to combine them into a single input.
Revised: Instead of having separate inputs, our first simple idea is to combine them into a single input.

Original: Specifically, we add an additional dimension to the adjacency matrix, making it a tensor, similar to that used in computer vision task.
Revised: Specifically, we add an additional dimension to the adjacency matrix, making it a tensor, similar to that used in computer vision task.

Original: The diagonal section of the tensor holds the atom features.
Revised: The diagonal section of the tensor holds the atom features.

Original: v s C d v v v v d C s v v v
Revised: [Figure] s C d v v v v [Figure]

Original: v s C d d C s v v v
Revised: C s v [Figure] v

Original: C d v v v s s v v v d C 00010 0.
Revised: [Figure]

Original: [-Atom type encoding Atom charge encoding 
-]
Revised: [-Atom type encoding Atom charge encoding 
-]v

Original: [Figure]
Revised: [Figure]

Original: We consider three types of atom features comprising atom type, charge and chirality.
Revised: We consider three types of atom features comprising atom type, charge and chirality.

Original: Each feature is onehot encoded and are stacked together to form a single atom feature vector.
Revised: Each feature is onehot encoded and are stacked together to form a single atom feature vector.

Original: There are two variants of the atom feature vector corresponding to two input tensors for the two encoders of the CVAE: an encoder conditioned only on graph and the other conditioned on both graph and coordinates.
Revised: There are two variants of the atom feature vector corresponding to two input tensors for the two encoders of the CVAE: an encoder conditioned only on graph and the other conditioned on both graph and coordinates.

Original: For the latter, every atom feature vector has three additional channels incorporating the 3D coordinate of the respective atom.
Revised: For the latter, every atom feature vector has three additional channels incorporating the 3D coordinate of the respective atom.

Original: The off-diagonal section holds the bond features.
Revised: The off-diagonal section holds the bond features.

Original: The considered bond features are bond type, bond stereochemistry type, associated ring size and normalized bond length.
Revised: The considered bond features are bond type, bond stereochemistry type, associated ring size and normalized bond length.

Original: A virtual bond is also included in the bond type.
Revised: A virtual bond is also included in the bond type.

Original: The normalized bond length is calculated as edge length (1 for direct neighbor, 2 for 2 nd neighbor, etc.) divided by the longest chain length.
Revised: The normalized bond length is calculated as edge length (1 for direct neighbor, 2 for 2 nd neighbor, etc.) divided by the longest chain length.

Original: It is worth noting that all high-order virtual bonds share the same virtual bond type; they only differ in their normalized bond length.
Revised: It is worth noting that all high-order virtual bonds share the same virtual bond type; they only differ in their normalized bond length.

Original: To construct bond feature vector, we first sum the atom feature vectors of the related atoms.
Revised: To construct bond feature vector, we first sum the atom feature vectors of the related atoms.

Original: This new vector is then stacked with one-hot encoded bond type vector, normalized bond length, and one-hot encoded ring size vector to become the bond feature vector.
Revised: This new vector is then stacked with one-hot encoded bond type vector, normalized bond length, and one-hot encoded ring size vector to become the bond feature vector.

Original: Similar to the atom feature vector, there are also two variants of the bond feature vector.
Revised: Similar to the atom feature vector, there are also two variants of the bond feature vector.

Original: For the encoder conditioned only on graph, coordinate channles are excluded from all bond feature vectors.
Revised: For the encoder conditioned only on graph, coordinate channles are excluded from all bond feature vectors.

Original: For the other encoder, to match the size of the atom feature vector, every bond feature vector has three more coordinate channels filled with 0s, and an additional distance channel holding the Euclidean distance between two connected atoms.
Revised: For the other encoder, to match the size of the atom feature vector, every bond feature vector has three more coordinate channels filled with 0s, and an additional distance channel holding the Euclidean distance between two connected atoms.

Original: This bond feature vector is obtained for all atom pairs, making the proposed tensor fully connected and symmtric.
Revised: This bond feature vector is obtained for all atom pairs, making the proposed tensor fully connected and symmtric.

Original: An example input tensor graph of benzene ring is illustrated in Fig.1.
Revised: An example input tensor graph of benzene ring is illustrated in Fig.1.

Original: Having obtained the tensor representation, a blatant and perhaps naive way of building a generative model is to apply a convolutional neural network directly on top of the tensor, and train it to predict a distribution over the inter-atomic distances.
Revised: Having obtained the tensor representation, a blatant and perhaps naive way of building a generative model is to apply a convolutional neural network directly on top of the tensor, and train it to predict a distribution over the inter-atomic distances.

Original: We utilize a standard UNet (Ronneberger et al., 2015) structure to map the input tensor to a probability distribution over a distance matrix containing all pair-wise Euclidean distances.
Revised: We utilize a standard UNet (Ronneberger et al., 2015) structure to map the input tensor to a probability distribution over a distance matrix containing all pair-wise Euclidean distances.

Original: Distance matrices are then sampled and converted to valid conformations following the same method presented in GraphDG (Simm & Hern´andez-Lobato, 2019).
Revised: Distance matrices are then sampled and converted to valid conformations following the same method presented in GraphDG (Simm & Hern´andez-Lobato, 2019).

Original: More details of this naive model can be found in Sec.
Revised: More details of this naive model can be found in Sec.

Original: A.1.
Revised: A.1.

Original: Despite its naive nature, this model achieves a mean coverage (COV) score of 51 .
Revised: Despite its naive nature, this model achieves a mean coverage (COV) score of 51 .

Original: 95% and a mean matching (MAT) score of 1 .
Revised: 95% and a mean matching (MAT) score of 1 .

Original: 3822˚A on the GEOM-Drugs dataset, already comparable to several more complex distance-based baselines, as shown in Sec.3.2.
Revised: 3822˚A on the GEOM-Drugs dataset, already comparable to several more complex distance-based baselines, as shown in Sec.3.2.

Original: However, there are two major issues to this approach.
Revised: However, there are two major issues to this approach.

Original: First, with a small kernel size ( 3 × 3 used in the UNet), it takes many convolution layers to achieve information aggregation between atoms that are far apart; it does not take full advantage of high-order bonds already made available in the input tensor.
Revised: First, with a small kernel size ( 3 × 3 used in the UNet), it takes many convolution layers to achieve information aggregation between atoms that are far apart; it does not take full advantage of high-order bonds already made available in the input tensor.

Original: Secondly, the output size grows quadratically with the number of atoms, as compared to only linear growth in reconstruction-based or direct generation methods.
Revised: Secondly, the output size grows quadratically with the number of atoms, as compared to only linear growth in reconstruction-based or direct generation methods.

Original: The solution to the first issue is rather simple, obtained by increasing the kernel size to expand its “field of view”.
Revised: The solution to the first issue is rather simple, obtained by increasing the kernel size to expand its “field of view”.

Original: On the other hand, solving the second issue requires elevating the naive two-step generative model to a direct one.
Revised: On the other hand, solving the second issue requires elevating the naive two-step generative model to a direct one.

Original: 2. E XTENDED KERNEL AND A TTENTION M ECHANISM
Revised: 2. E XTENDED KERNEL AND A TTENTION M ECHANISM

Original: We observe that every row or column of the proposed tensor contains the complete information of a focal atom and all of its connected atoms (by both chemical and virtual bond).
Revised: We observe that every row or column of the proposed tensor contains the complete information of a focal atom and all of its connected atoms (by both chemical and virtual bond).

Original: This motivates our second main idea which is to extend the length of the kernel to the length of the tensor graph while keeping the width unaltered.
Revised: This motivates our second main idea which is to extend the length of the kernel to the length of the tensor graph while keeping the width unaltered.

Original: This idea has a profound implication; information from the immediate neighbors, all their connected atoms, and all the bond features can be aggregated onto the focal atom in a single convolution operation.
Revised: This idea has a profound implication; information from the immediate neighbors, all their connected atoms, and all the bond features can be aggregated onto the focal atom in a single convolution operation.

Original: In contrast, achieving the same aggregation may require many layers of propagation for the naive model and other GNN-based models.
Revised: In contrast, achieving the same aggregation may require many layers of propagation for the naive model and other GNN-based models.

Original: A direct consequence of this modification is that only 1D convolution is permitted.
Revised: A direct consequence of this modification is that only 1D convolution is permitted.

Original: With multiple kernels being applied simultaneously, each stride of these kernels generates a feature vector for a single atom.
Revised: With multiple kernels being applied simultaneously, each stride of these kernels generates a feature vector for a single atom.

Original: An illustration of the 1D convolution operation is shown in Fig.2.
Revised: An illustration of the 1D convolution operation is shown in Fig.2.

Original: [Figure]
Revised: [Figure]

Original: We further observe that the generated feature vectors resemble the token-like feature vectors used in language modeling.
Revised: We further observe that the generated feature vectors resemble the token-like feature vectors used in language modeling.

Original: This observation combined with the proven success of attention mechanism in other related work leads to the selection of transformer architecture as the backbone of our generative model.
Revised: This observation combined with the proven success of attention mechanism in other related work leads to the selection of transformer architecture as the backbone of our generative model.

Original: A significant advantage of using transformer’s self attention mechanism is, similar to the extended kernel, it enables information aggregation from and for all atoms.
Revised: A significant advantage of using transformer’s self attention mechanism is, similar to the extended kernel, it enables information aggregation from and for all atoms.

Original: It also eliminates the need of maintaining separate atom and bond features at each step of feature transformation.
Revised: It also eliminates the need of maintaining separate atom and bond features at each step of feature transformation.

Original: 2. P UTTING EVERYTHING TOGETHER
Revised: 2. P UTTING EVERYTHING TOGETHER

Original: Conditional variational AutoEncoder framework.
Revised: Conditional variational AutoEncoder framework.

Original: We aim at obtaining a generative model p θ ( R | G ) that approximates the Boltzmann distribution through Maximum Likelihood Estimation.
Revised: We aim at obtaining a generative model p θ ( R | G ) that approximates the Boltzmann distribution through Maximum Likelihood Estimation.

Original: Particularly, given a set of molecular graphs G and their respective ground-truth conformations R , we wish to maximize the following objective.
Revised: Particularly, given a set of molecular graphs G and their respective ground-truth conformations R , we wish to maximize the following objective.

Original: [Equation]
Revised: [Equation]

Original: A molecular graph can have many random conformations.
Revised: A molecular graph can have many random conformations.

Original: We assume this randomness is driven by a latent process governed by a random variable z ∼ p ( z ) , where p ( z ) is a known distribution e.g. a standard normal distribution.
Revised: We assume this randomness is driven by a latent process governed by a random variable z ∼ p ( z ) , where p ( z ) is a known distribution e.g. a standard normal distribution.

Original: As p θ ( R | z, G ) is often modeled by a complex function e.g. a deep neural network, evaluation of the integral in Eq.1 is intractable.
Revised: As p θ ( R | z, G ) is often modeled by a complex function e.g. a deep neural network, evaluation of the integral in Eq.1 is intractable.

Original: Instead, we resort to the same techniques proposed in the original VAE (Kingma & Welling, 2013) to establish a tractable lower bound for Eq.1.
Revised: Instead, we resort to the same techniques proposed in the original VAE (Kingma & Welling, 2013) to establish a tractable lower bound for Eq.1.

Original: [Equation]
Revised: [Equation]

Original: where D KL is the Kullback-Leibler divergence and q w ( z | R, G ) is a variational approximation of the true posterior p ( z | R, G ) .
Revised: where D KL is the Kullback-Leibler divergence and q w ( z | R, G ) is a variational approximation of the true posterior p ( z | R, G ) .

Original: We assume p ( z ) =
Revised: We assume p ( z ) =

Original: N (0 , I ) and q w ( z | R, G ) a diagonal Gaussian distribution whose means and standard deviations are modeled by a transformer encoder.
Revised: N (0 , I ) and q w ( z | R, G ) a diagonal Gaussian distribution whose means and standard deviations are modeled by a transformer encoder.

Original: The input of this transformer encoder is the proposed tensor containing both the coordinate and distance information.
Revised: The input of this transformer encoder is the proposed tensor containing both the coordinate and distance information.

Original: We denote this tensor the GDR tensor.
Revised: We denote this tensor the GDR tensor.

Original: On the other hand, p θ ( R | z, G ) is further decomposed into two parts: a decoder p θ 2 ( R | z, σ θ 1 ( G )) for predicting conformation directly and another encoder σ θ 1 ( G ) .
Revised: On the other hand, p θ ( R | z, G ) is further decomposed into two parts: a decoder p θ 2 ( R | z, σ θ 1 ( G )) for predicting conformation directly and another encoder σ θ 1 ( G ) .

Original: The input tensor for σ θ 1 ( G ) is absent of coordinate and distance information, and is therefore denoted the G tensor.
Revised: The input tensor for σ θ 1 ( G ) is absent of coordinate and distance information, and is therefore denoted the G tensor.

Original: Both encoders share the same standard transformer encoder structure.
Revised: Both encoders share the same standard transformer encoder structure.

Original: However, there is a minor modification to the transformer structure for the decoder.
Revised: However, there is a minor modification to the transformer structure for the decoder.

Original: Specifically, the Query, Key matrices for the first multi-head attention layer are computed based on the output vectors of σ θ 1 ( G ) , and the Value matrices come directly from the reparameterization of the output of q w ( z | R, G ) , as z = µ w + Σ w ϵ , where µ w and Σ w are the predicted mean and standard deviation respectively.
Revised: Specifically, the Query, Key matrices for the first multi-head attention layer are computed based on the output vectors of σ θ 1 ( G ) , and the Value matrices come directly from the reparameterization of the output of q w ( z | R, G ) , as z = µ w + Σ w ϵ , where µ w and Σ w are the predicted mean and standard deviation respectively.

Original: ϵ is sampled from N (0 , I ) .
Revised: ϵ is sampled from N (0 , I ) .

Original: We present the complete picture of how the two encoders and the decoder are arranged in a conditional variational Auto Encoder framework in Fig.3a.
Revised: We present the complete picture of how the two encoders and the decoder are arranged in a conditional variational Auto Encoder framework in Fig.3a.

Original: It is worth noting that this is also the same CVAE framework adopted in CVGAE (Mansimov et al., 2019),
Revised: It is worth noting that this is also the same CVAE framework adopted in CVGAE (Mansimov et al., 2019),

Original: ConfVAE (Xu et al., 2021b), and DMCG (Zhu et al., 2022).
Revised: ConfVAE (Xu et al., 2021b), and DMCG (Zhu et al., 2022).

Original: We also show the illustration of the modified multi-head attention in Fig.3b.
Revised: We also show the illustration of the modified multi-head attention in Fig.3b.

Original: Intuition behind the modified attention.
Revised: Intuition behind the modified attention.

Original: There are multiple ways to join together the output of the two encoders to form the input to the final decoder.
Revised: There are multiple ways to join together the output of the two encoders to form the input to the final decoder.

Original: Popular methods include stacking or addition.
Revised: Popular methods include stacking or addition.

Original: We tried both these methods with unsatisfactory performance.
Revised: We tried both these methods with unsatisfactory performance.

Original: We notice that, due to direct stacking or addition of the sampled output of q w onto the output of σ θ 1 , attention weights computed in the first layer of the decoder are easily overwhelmed by random noise of the sampled values, and become almost indiscernible 1 .
Revised: We notice that, due to direct stacking or addition of the sampled output of q w onto the output of σ θ 1 , attention weights computed in the first layer of the decoder are easily overwhelmed by random noise of the sampled values, and become almost indiscernible 1 .

Original: This leads to ineffective information aggregation which is then further cascaded through the remaining attention layers.
Revised: This leads to ineffective information aggregation which is then further cascaded through the remaining attention layers.

Original: Intuitively, in the first attention layer, the attention weights dictating how much influence an atom exerts on the other should predominantly be determined by the graph structure, and remain stable for the same molecule.
Revised: Intuitively, in the first attention layer, the attention weights dictating how much influence an atom exerts on the other should predominantly be determined by the graph structure, and remain stable for the same molecule.

Original: Further, attention weights are computed by Query and Key matrices.
Revised: Further, attention weights are computed by Query and Key matrices.

Original: Therefore, these two matrices should stay stable for the same graph.
Revised: Therefore, these two matrices should stay stable for the same graph.

Original: This motivates our third and final main idea ; that is, we compute Query and
Revised: This motivates our third and final main idea ; that is, we compute Query and

Original: Key matrices only from the output (cid:8) h L 1 , ..., h LN (cid:9) of σ θ 1 , and attribute the variation in conformation to the Value matrices which are directly sampled from { z 1 , ..., z N } ∼ q w .
Revised: Key matrices only from the output (cid:8) h L 1 , ..., h LN (cid:9) of σ θ 1 , and attribute the variation in conformation to the Value matrices which are directly sampled from { z 1 , ..., z N } ∼ q w .

Original: The resultant information aggregation is much more meaningful and each output vector corresponds to an individual atom carries distinct features, facilitating information aggregation of the ensuing attention layers.
Revised: The resultant information aggregation is much more meaningful and each output vector corresponds to an individual atom carries distinct features, facilitating information aggregation of the ensuing attention layers.

Original: Permutation and roto-transition invariant loss.
Revised: Permutation and roto-transition invariant loss.

Original: An important feature of using the sliding extended kernel is that the ordering of the generated feature tokens follows exactly the ordering of the atoms.
Revised: An important feature of using the sliding extended kernel is that the ordering of the generated feature tokens follows exactly the ordering of the atoms.

Original: The same permutation order is also preserved through all the transformer components.
Revised: The same permutation order is also preserved through all the transformer components.

Original: In addition, following ConfVAE (Xu et al., 2021b), we formulate the reconstruction loss as.
Revised: In addition, following ConfVAE (Xu et al., 2021b), we formulate the reconstruction loss as.

Original: [Equation]
Revised: [Equation]

Original: Multi-head Attention
Revised: Multi-head Attention

Original: Add & Norm
Revised: Add & Norm

Original: Feed Forward
Revised: Feed Forward

Original: Add & Norm
Revised: Add & Norm

Original: Lx
Revised: Lx

Original: G tensor
Revised: G tensor

Original: GDR tensor
Revised: GDR tensor

Original: L x trainformer encoder blocks zzz N
Revised: L x trainformer encoder blocks zzz N

Original: Modified Multi-head Attention
Revised: Modified Multi-head Attention

Original: Add & Norm
Revised: Add & Norm

Original: Feed Forward
Revised: Feed Forward

Original: Add & Norm
Revised: Add & Norm

Original: L-1 x trainformer encoder block
Revised: L-1 x trainformer encoder block

Original: R R
Revised: R R

Original: R N
Revised: R N

Original: KL Regularization loss
Revised: KL Regularization loss

Original: Reconstruction loss
Revised: Reconstruction loss

Original: Kabsch alignment (
Revised: Kabsch alignment (

Original: [Figure]
Revised: [Figure]

Original: framework
Revised: framework

Original: Self-attentionSelf-attention Scaled Dot-product Attention
Revised: Self-attentionSelf-attention Scaled Dot-product Attention

Original: Linear
Revised: Linear

Original: Linear
Revised: Linear

Original: Linear qkqkq N k N
Revised: Linear qkqkq N k N

Original: K Q V zzz N
Revised: K Q V zzz N

Original: Split
Revised: Split

Original: Concat
Revised: Concat

Original: Linear (b) Modified multi-head attention where A ( · ) is an alignment function aligning the predicted conformation ˆ R onto the reference conformation R .
Revised: Linear (b) Modified multi-head attention where A ( · ) is an alignment function aligning the predicted conformation ˆ R onto the reference conformation R .

Original: The sum aggregation of per atom errors and the permutation preserving property of the feature extraction process make Eq.3 invariant to permutation of atom order.
Revised: The sum aggregation of per atom errors and the permutation preserving property of the feature extraction process make Eq.3 invariant to permutation of atom order.

Original: Furthermore, we choose Kabsch algorithm as the alignment method which translates and rotates the predicted conformation onto its corresponding ground-truth before loss computation.
Revised: Furthermore, we choose Kabsch algorithm as the alignment method which translates and rotates the predicted conformation onto its corresponding ground-truth before loss computation.

Original: This makes the reconstruction loss roto-translation invariant.
Revised: This makes the reconstruction loss roto-translation invariant.

Original: Finally, the KL-loss component D KL [ q w ( z | R, G ) || p ( z )] does not involve any coordinate.
Revised: Finally, the KL-loss component D KL [ q w ( z | R, G ) || p ( z )] does not involve any coordinate.

Original: Therefore, the objective function defined in Eq.2 is both permutation and roto-translation invariant.
Revised: Therefore, the objective function defined in Eq.2 is both permutation and roto-translation invariant.

Original: Direct conformation generation at inference time.
Revised: Direct conformation generation at inference time.

Original: To generate a single conformation, we first construct the G tensor of a molecular graph and obtain a single latent sample { z 1 , ...
Revised: To generate a single conformation, we first construct the G tensor of a molecular graph and obtain a single latent sample { z 1 , ...

Original: z N } from a standard diagonal Gaussian distribution.
Revised: z N } from a standard diagonal Gaussian distribution.

Original: The G tensor is passed through σ θ 1 encoder to produce (cid:8) h L 1 , ..., h LN (cid:9) which is then combined with the latent sample via the modified multi-head attention mechanism.
Revised: The G tensor is passed through σ θ 1 encoder to produce (cid:8) h L 1 , ..., h LN (cid:9) which is then combined with the latent sample via the modified multi-head attention mechanism.

Original: The output of this modified attention layer further goes through L − 1 standard attention layers to be transformed to the final conformation.
Revised: The output of this modified attention layer further goes through L − 1 standard attention layers to be transformed to the final conformation.

Original: The entire generation process depends only on a 2D molecular graph, and requires a single sampling step and a single pass of the TensorVAE model.
Revised: The entire generation process depends only on a 2D molecular graph, and requires a single sampling step and a single pass of the TensorVAE model.

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: In this section, we first elaborate on the implementation details of the TensorVAE model including determining the size of  input tensor, network architecture and how the entire framework is trained end-to-end.
Revised: In this section, we first elaborate on the implementation details of the TensorVAE model including determining the size of the input tensors, network architecture and how the entire framework is trained end-to-end.

Original: We then present conformation generation experiment results of the proposed TensorVAE on two benchmark data-sets, GEOM-QM9 and GEOM-Drugs.
Revised: We then present conformation generation experiment results of the proposed TensorVAE on two benchmark data-sets, GEOM-QM9 and GEOM-Drugs.

Original: These results are compared to those of 11 state-of-the-art baselines.
Revised: These results are compared to those of 11 state-of-the-art baselines.

Original: In addition to conformation generation, we present further experiment on molecular property prediction in Sec.
Revised: In addition to conformation generation, we present further experiment on molecular property prediction in Sec.

Original: A.5, where the performance of the proposed method is compared against 7 more state-of-the-art baselines on the MolecularNet benchmark (Wu et al., 2018).
Revised: A.5, where the performance of the proposed method is compared against 7 more state-of-the-art baselines on the MolecularNet benchmark (Wu et al., 2018).

Original: 3. E XPERIMENT SETUP
Revised: 3. E XPERIMENT SETUP

Original: Dataset.
Revised: Dataset.

Original: Following existing work (Luo et al., 2021; Shi et al., 2021; Xu et al.
Revised: Following existing work (Luo et al., 2021; Shi et al., 2021; Xu et al.

Original: , 2021b;a; 2022; Zhou et al., 2022), we utilize the GEOM data-set for evaluating the performance of the proposed TensorVAE.
Revised: , 2021b;a; 2022; Zhou et al., 2022), we utilize the GEOM data-set for evaluating the performance of the proposed TensorVAE.

Original: GEOM contains 37 million energy and statistical weight annotated molecular conformations corresponding to 450,000 molecules (Axelrod & G´omez-Bombarelli, 2022).
Revised: GEOM contains 37 million energy and statistical weight annotated molecular conformations corresponding to 450,000 molecules (Axelrod & G´omez-Bombarelli, 2022).

Original: This dataset is further divided into two constituent datasets, Drugs and QM9.
Revised: This dataset is further divided into two constituent datasets, Drugs and QM9.

Original: The Drugs dataset covers 317,000 mediansized molecules averaging 44.4 number of atoms.
Revised: The Drugs dataset covers 317,000 mediansized molecules averaging 44.4 number of atoms.

Original: The QM9 dataset contains 133,000 smaller molecules averaging only 18 atoms.
Revised: The QM9 dataset contains 133,000 smaller molecules averaging only 18 atoms.

Original: We randomly select 40000 molecules from each dataset to form the training set.
Revised: We randomly select 40000 molecules from each dataset to form the training set.

Original: For each molecule, we choose the top 52 most likely  conformations.
Revised: For each molecule, we choose the top 5 most likely 2 conformations.

Original: This results in 200,000 training conformations for each train set.
Revised: This results in 200,000 training conformations for each train set.

Original: For validation set, we randomly sample 2,500 conformations for both Drugs and QM9 experiments.
Revised: For validation set, we randomly sample 2,500 conformations for both Drugs and QM9 experiments.

Original: Finally, for testing, following (Shi et al., 2021; Xu et al., 2022), we randomly select 200 molecules each with more than 50 and less thanconformations from QM9, and another 200 with more than 50 and less than 100  conformations from Drugs 3 .
Revised: Finally, for testing, following (Shi et al., 2021; Xu et al., 2022), we randomly select 200 molecules each with more than 50 and less thanannotated conformations from QM9, and another 200 with more than 50 and less than 100 annotated conformations from Drugs 3 .

Original: Determining input tensor graph size.
Revised: Determining input tensor graph size.

Original: We conduct a basic data analysis on the entire Drugs dataset to determine the 98 .
Revised: We conduct a basic data analysis on the entire Drugs dataset to determine the 98 .

Original: 5 th percentile of the number of atoms to be 69, and the percentage of molecules having more than 69 atoms and with more than 50 but less than 100 conformations is only 0 .
Revised: 5 th percentile of the number of atoms to be 69, and the percentage of molecules having more than 69 atoms and with more than 50 but less than 100 conformations is only 0 .

Original: Accordingly, we set the size of the input tensor to 69 × 69 for Drugs.
Revised: Accordingly, we set the size of the input tensor to 69 × 69 for Drugs experiment.

Original: On the other hand, we use the maximum number of atoms 30 for QM9.
Revised: On the other hand, we use the maximum number of atoms 30 for QM9 experiment.

Original: The channel features for the input tensor include atom types, atom charge, atom chirality, bond type, bond stereo-chemistry and bond in-ring size.
Revised: The channel features for the input tensor include atom types, atom charge, atom chirality, bond type, bond stereo-chemistry and bond in-ring size.

Original: For the GDR tensor, we also include 3D coordinate channel and the computed distance channel.
Revised: For the GDR tensor, we also include 3D coordinate channels and the computed distance channel.

Original: The resulting channel depth is 50 for GDR tensor and 46 for G tensor.
Revised: The resulting channel depth is 50 for GDR tensor and 46 for G tensor.

Original: The detailed information of these features and their encoding method is listed in Sec.
Revised: The detailed information of these features and their encoding method is listed in Sec.

Original: A.2.
Revised: A.2.

Original: Implementation details.
Revised: Implementation details.

Original: We implement the proposed TensorVAE using Tensorflow 2.3.1.
Revised: We implement the proposed TensorVAE using Tensorflow 2.3.1.

Original: All three transformer encoders of TensorVAE follow the standard Tensorflow implementation in https: //www.tensorflow.org
Revised: All three transformer encoders of TensorVAE follow the standard Tensorflow implementation in https: //www.tensorflow.org

Original: /text/tutorials/transformer .
Revised: /text/tutorials/transformer .

Original: All of them have 4 layers, 8 heads and a latent dimension of 256.
Revised: All of them have 4 layers, 8 heads and a latent dimension of 256.

Original: Both QM9 and Drugs experiments share the same network architecture and hyper-parameter configuration.
Revised: Both QM9 and Drugs experiments share the same network architecture and hyper-parameter configuration.

Original: Training is conducted on a single Tesla V
Revised: Training is conducted on a single Tesla V

Original: GPU.
Revised: GPU.

Original: We follow a similar learning rate schedule, shown by Eq.3 of the original Transformer paper (Vaswani et al., 2017) but with d model
Revised: We follow a similar learning rate schedule, shown by Eq.3 of the original Transformer paper (Vaswani et al., 2017) but with d model

Original: = 9612 .
Revised: = 9612 .

Original: This results in a maximum learning rate of 1 . 6 e − 4 .
Revised: This results in a maximum learning rate of 1 . 6 e − 4 .

Original: To tackle the notorious issue of KL vanishing (Fu et al., 2019), we set a minimum KL weight of 1 e − 4 and double it every 62 .
Revised: To tackle the notorious issue of KL vanishing (Fu et al., 2019), we set a minimum KL weight of 1 e − 4 and double it every 62 .

Original: 5 e 3 iterations until a maximum weight of 0.0256 is reached.
Revised: 5 e 3 iterations until a maximum weight of 0.0256 is reached.

Original: We present some interesting observations of the training/validation curve corresponding to this setup in Sec.
Revised: We present some interesting observations of the training/validation curve corresponding to this setup in Sec.

Original: A.3.
Revised: A.3.

Original: For both experiments, the TensorVAE is trained for 1 e 6 iterations with a batch size of 128.
Revised: For both experiments, the TensorVAE is trained for 1 e 6 iterations with a batch size of 128.

Original: The implementation details of NaiveUNet is explained in Sec.
Revised: The implementation details of NaiveUNet is explained in Sec.

Original: A.
Revised: A.

Original: Evaluation metrics.
Revised: Evaluation metrics.

Original: We adopt the widely accepted coverage score (COV) and matching score (MAT) (Shi et al., 2021) to evaluate the performance of the proposed TensorVAE model.
Revised: We adopt the widely accepted coverage score (COV) and matching score (MAT) (Shi et al., 2021) to evaluate the performance of the proposed TensorVAE model.

Original: These two scores are computed as.
Revised: These two scores are computed as.

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where C g is the set of generated conformations and C r is the corresponding reference set.
Revised: where C g is the set of generated conformations and C r is the corresponding reference set.

Original: The size of C g is twice of that of C r , as for every molecule, we follow (Xu et al., 2022) to generate twice the number of conformations as that of reference conformations.
Revised: The size of C g is twice of that of C r , as for every molecule, we follow (Xu et al., 2022) to generate twice the number of conformations as that of reference conformations.

Original: δ is a predefined threshold and is set to 0 .
Revised: δ is a predefined threshold and is set to 0 .

Original: 5˚A for QM9 and 1 .
Revised: 5˚A for QM9 and 1 .

Original: 25˚A for Drugs respectively (Shi et al., 2021) .
Revised: 25˚A for Drugs respectively (Shi et al., 2021) .

Original: RMSD stands for the root-mean-square deviation between R and ˆ R , and is computed using the GetBestRMS method in the RDKit (Riniker & Landrum, 2015) package.
Revised: RMSD stands for the root-mean-square deviation between R and ˆ R , and is computed using the GetBestRMS method in the RDKit (Riniker & Landrum, 2015) package.

Original: While COV score measures the ability of a model in generating diverse conformations to cover all reference conformations, MAT score measures how well the generated conformations match the ground-truth.
Revised: While COV score measures the ability of a model in generating diverse conformations to cover all reference conformations, MAT score measures how well the generated conformations match the ground-truth.

Original: A good generative model should have a high COV score and a low MAT score.
Revised: A good generative model should have a high COV score and a low MAT score.

Original: Baselines .
Revised: Baselines .

Original: We compare the performance of the proposed TensorVAE model to those of 1 classical
Revised: We compare the performance of the proposed TensorVAE model to those of 1 classical

Original: RDKit method; 5 distance-based methods including GraphDG, CGCF, ConfVAE, ConfGF and
Revised: RDKit method; 5 distance-based methods including GraphDG, CGCF, ConfVAE, ConfGF and

Original: DGSM; 2 reconstruction-based methods including GeoDiff and Uni-Mol; 4 direct methods including
Revised: DGSM; 2 reconstruction-based methods including GeoDiff and Uni-Mol; 4 direct methods including

Original: CVGAE, GeoMol, DMCG and its symmetric permutation variant.
Revised: CVGAE, GeoMol, DMCG and its symmetric permutation variant.

Original: The detailed information of the molecular property prediction baselines are presented in Sec.
Revised: The detailed information of the molecular property prediction baselines are presented in Sec.

Original: A.
Revised: A.

Original:  R ESULTS AND D ISCUSSION
Revised:  R ESULTS AND D ISCUSSION

Original: The COV and MAT scores for all compared methods on both QM9 and Drugs datasets are presented in Tab.1.
Revised: The COV and MAT scores for all compared methods on both QM9 and Drugs datasets are presented in Tab.1.

Original: All experiment results are obtained from the same test data generation configuration introduced in (Shi et al., 2021).
Revised: All experiment results are obtained from the same test data generation configuration introduced in (Shi et al., 2021).

Original: [Table]
Revised: [Table]

Original: In general, distance-based methods except for DGSM and ConfGF have relatively poor performance as compared to that of the classic RDKit.
Revised: In general, distance-based methods except for DGSM and ConfGF have relatively poor performance as compared to that of the classic RDKit.

Original: It has been argued that the the performance of RDkit is facilitated by an additional empirical force field (FF) (Halgren, 1996) optimization.
Revised: It has been argued that the the performance of RDkit is facilitated by an additional empirical force field (FF) (Halgren, 1996) optimization.

Original: Subsequently,
Revised: Subsequently,

Original: CGCF and ConfVAE showed that with FF optimization, they can outperform RDKit.
Revised: CGCF and ConfVAE showed that with FF optimization, they can outperform RDkit .

Original: We also show the performance of the proposed TensorVAE with FF optimization compared to 5 other methods also employing FF optimization in Tab.3 in Sec.
Revised: We also show the performance of the proposed TensorVAE with FF optimization compared to 5 other methods also employing FF optimization in Tab.3 in Sec.

Original: A.4, where TensorVAE outperforms all of them with a significant margin.
Revised: A.4, where TensorVAE outperforms all of them with a significant margin.

Original: However, this additional step further introduces complexity to the already complex two-stage generative model.
Revised: However, this additional step further introduces complexity to the already complex two-stage generative model.

Original: ConfGF attempted to rectify this weakness by simulating a pseudo gradient-based force field.
Revised: ConfGF attempted to rectify this weakness by simulating a pseudo gradient-based force field.

Original: Such force field can be utilized in the annealed Langevin dynamics sampling to sequentially guide atom positions to a valid conformation.
Revised: Such force field can be utilized in the annealed Langevin dynamics sampling to sequentially guide atom positions to a valid conformation.

Original: DGSM further improves the performance of ConfGF by using a dynamically constructed graph structure that is able to model long range atom interaction.
Revised: DGSM further improves the performance of ConfGF by using a dynamically constructed graph structure that is able to model long range atom interaction.

Original: Despite being posed as a direct method, both DGSM and ConfGF still need to compute atom distance as an intermediate step.
Revised: Despite being posed as a direct method, both DGSM and ConfGF still need to compute atom distance as an intermediate step.

Original: Noticeably, DGSM also requires dynamically changing the graph structure for every sampling step.
Revised: Noticeably, DGSM also requires dynamically changing the graph structure for every sampling step.

Original: As shown in Tab.1, DGSM and ConGF outperform RDKit by a significant margin in both experiments.
Revised: As shown in Tab.1, DGSM and ConGF outperform RDKit by a significant margin in both experiments.

Original: Nevertheless, their main weakness lies in the fact they they require numerous sampling steps to attain desirable performance.
Revised: Nevertheless, their main weakness lies in the fact they they require numerous sampling steps to attain desirable performance.

Original: It has been reported in the ensuing work (Xu et al., 2022; Zhu et al., 2022), that it took ConfGF approximately 8500 seconds to fully decode 200 QM9 molecules and a staggering 11500 seconds for decoding
Revised: It has been reported in the ensuing work (Xu et al., 2022; Zhu et al., 2022), that it took ConfGF approximately 8500 seconds to fully decode 200 QM9 molecules and a staggering 11500 seconds for decoding

Original: Drugs molecules.
Revised: Drugs molecules.

Original: In constrast, TensorVAE takes only 62 seconds using a single Xeon 8163 CPU core to decode 200 QM9 molecules, and 128 seconds for 200 Drug molecules .
Revised: In constrast, TensorVAE takes only 62 seconds using a single Xeon 8163 CPU core to decode 200 QM9 molecules, and 128 seconds for 200 Drug molecules .

Original: A main goal of GeoDiff is to be free of the dependence on distance.
Revised: A main goal of GeoDiff is to be free of the dependence on distance.

Original: To achieve this, transformations are directly applied to 3D conformation, through which, a random or distorted conformation can be sequentially denoised to a valid conformation.
Revised: To achieve this, transformations are directly applied to 3D conformation, through which, a random or distorted conformation can be sequentially denoised to a valid conformation.

Original: An essential requirement for such transformation is that it needs to be SE(3) equivariant.
Revised: An essential requirement for such transformation is that it needs to be SE(3) equivariant.

Original: This necessitates design of sophisticated equivariant transition kernel.
Revised: This necessitates design of sophisticated equivariant transition kernel.

Original: In spite of its claim of not involving computing distance as an intermediate step, the equivariant graph field network (Satorras et al., 2021) used still relies on a distance-like quantity (cid:13)(cid:13)
Revised: In spite of its claim of not involving computing distance as an intermediate step, the equivariant graph field network (Satorras et al., 2021) used still relies on a distance-like quantity (cid:13)(cid:13)

Original: x Li − x Lj (cid:13)(cid:13) 2 for every step of transformation.
Revised: x Li − x Lj (cid:13)(cid:13) 2 for every step of transformation.

Original: GeoDiff produces promising performance on both datasets.
Revised: GeoDiff produces promising performance on both datasets.

Original: Unfortunately, it needs numerous diffusion steps ( T = 5000 ) for generating conformations, requiring approximately the same amount of decoding time as ConfGF (Xu et al., 2022).
Revised: Unfortunately, it needs numerous diffusion steps ( T = 5000 ) for generating conformations, requiring approximately the same amount of decoding time as ConfGF (Xu et al., 2022).

Original: Uni-Mol circumvents this issue by reformulating molecular generation problem as an optimization problem.
Revised: Uni-Mol circumvents this issue by reformulating molecular generation problem as an optimization problem.

Original: The structure of Uni-Mol is inspired by the Evoformer proposed in Alphafold (Jumper et al., 2021) which also considers “atom-pair” interaction.
Revised: The structure of Uni-Mol is inspired by the Evoformer proposed in Alphafold (Jumper et al., 2021) which also considers “atom-pair” interaction.

Original: This requires maintaining a pair interaction matrix ( N × N ) through every attention layer.
Revised: This requires maintaining a pair interaction matrix ( N × N ) through every attention layer.

Original: In addition, Uni-Mol has a significantly larger transformer structure as compared to ours, featuring 15 attention layers, 64 attention heads and a latent dimension size of 2048.
Revised: In addition, Uni-Mol has a significantly larger transformer structure as compared to ours, featuring 15 attention layers, 64 attention heads and a latent dimension size of 2048.

Original: The Uni-Mol model is also first pretrained on a much larger dataset ( 19 e 6 molecules, each with 10 conformations), and then fine-tuned on the GEOM dataset for conformation generation.
Revised: The Uni-Mol model is also first pretrained on a much larger dataset ( 19 e 6 molecules, each with 10 conformations), and then fine-tuned on the GEOM dataset for conformation generation.

Original: Despite using a much smaller and also standard transformer model without pretraining, the proposed TensorVAE offers comparable performance to that of Uni-Mol, yielding better COV score for the Drugs dataset.
Revised: Despite using a much smaller and also standard transformer model without pretraining, the proposed TensorVAE offers comparable performance to that of Uni-Mol, yielding better COV score for the Drugs dataset.

Original: Further, we show that the proposed model outperforms
Revised: Further, we show that the proposed model outperforms

Original: Uni-Mol on the molecular property prediction task in Sec.
Revised: Uni-Mol on the molecular property prediction task in Sec.

Original: A.5 without any pretraining, validating the effectiveness of the input feature engineering.
Revised: A.5 without any pretraining, validating the effectiveness of the input feature engineering.

Original: CVGAE is the first method proposed to generate molecular conformation directly from a 2D molecular graph.
Revised: CVGAE is the first method proposed to generate molecular conformation directly from a 2D molecular graph.

Original: However, it yields the worst performance among the compared methods.
Revised: However, it yields the worst performance among the compared methods.

Original: DMCG attempted to revitalize the same framework by adapting a more advanced graph neural network structure combining GATv2 (Brody et al., 2021) with GN block (Battaglia et al., 2018).
Revised: DMCG attempted to revitalize the same framework by adapting a more advanced graph neural network structure combining GATv2 (Brody et al., 2021) with GN block (Battaglia et al., 2018).

Original: Noticeably, similar to Uni-Mol, it also maintains a N × N bond feature matrix throughout all the layers.
Revised: Noticeably, similar to Uni-Mol, it also maintains a N × N bond feature matrix throughout all the layers.

Original: In constrast, we only used a standard transformer encoder architecture.
Revised: In constrast, we only used a standard transformer encoder architecture.

Original: Another major modification proposed in DMCG is introducing the permutation invariance of symmetric molecular substructures to the RMSD loss.
Revised: Another major modification proposed in DMCG is introducing the permutation invariance of symmetric molecular substructures to the RMSD loss.

Original: Achieving this invariance requires enumerating all possible permutations which can become expensive for large molecules.
Revised: Achieving this invariance requires enumerating all possible permutations which can become expensive for large molecules.

Original: DMCG achieved the best COV score on drugs dataset; nevertheless, its variant using the same simple Kabsch aligned loss function as ours performs significantly worse on both Drugs and QM9 dataset.
Revised: DMCG achieved the best COV score on drugs dataset; nevertheless, its variant using the same simple Kabsch aligned loss function as ours performs significantly worse on both Drugs and QM9 dataset.

Original: This indicates that the proposed TensorVAE has  better feature extraction capability, eliminating the need of maintaining an additional N × N feature matrix and a more complex loss function for achieving  comparable performance.
Revised: This indicates that the proposed TensorVAE has a better feature extraction capability, eliminating the need of maintaining an additional N × N feature matrix and a more complex loss function for achieving a comparable performance.

Original: Finally, some samples of the TensorVAE generated conformations are shown in Sec.
Revised: Finally, some samples of the TensorVAE generated conformations are shown in Sec.

Original: A.
Revised: A.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We develop TensorVAE, a simple yet powerful model that is able to generate 3D conformation directly from a 2D molecular graph.
Revised: We develop TensorVAE, a simple yet powerful model that is able to generate 3D conformation directly from a 2D molecular graph.

Original: Unlike many existing work focusing on designing complex neural network structure, we focus on developing novel input feature engineering techniques.
Revised: Unlike many existing work focusing on designing complex neural network structure, we focus on developing novel input feature engineering techniques.

Original: We decompose these techniques into three main ideas, and explain how one idea naturally evolves to the next.
Revised: We decompose these techniques into three main ideas, and explain how one idea naturally evolves to the next.

Original: We first propose a tensor representation of a molecular graph.
Revised: We first propose a tensor representation of a molecular graph.

Original: Then, we demonstrate that sliding a rectangle kernel through this tensor in an 1D convolution manner can achieve complete information aggregation.
Revised: Then, we demonstrate that sliding a rectangle kernel through this tensor in an 1D convolution manner can achieve complete information aggregation.

Original: Finally, we present the complete CVAE-based framework featuring 2 transformerbased encoders and another transformer-based decoder, and propose a novel modification to the first multi-head attention layer of the decoder to enable sensible integration of the output of the other two encoders.
Revised: Finally, we present the complete CVAE-based framework featuring 2 transformerbased encoders and another transformer -based decoder, and propose a novel modification to the first multi-head attention layer of the decoder to enable sensible integration of the output of the other two encoders.

Original: We show through extensive experiments that with sensible feature engineering, simple and standard model architecture can provide competitive performance compared to 18 recent stateof-the-art models.
Revised: We show through extensive experiments that with intuitive feature engineering, simple and standard model architecture can provide competitive performance compared to 18 recent stateof-the-art models.

Original: For future work, we plan to extend our method and methodology to tackling the challenging protein structure prediction.
Revised: For future work, we plan to extend our method and methodology to tackling the challenging protein structure prediction.

Original:  R EPRODUCIBILITY STATEMENT
Revised:  R EPRODUCIBILITY STATEMENT

Original: We did not introduce nor have used any task-specific neural network archiecture.
Revised: We did not introduce nor have used any task-specific neural network archiecture.

Original: The results presented in this study can be straightforwardly reproduced using publically available datasets and ready-to-use implementation of convolution operation and Transformer from either PyTorch or
Revised: The results presented in this study can be straightforwardly reproduced using publically available datasets and ready-to-use implementation of convolution operation and Transformer from either PyTorch or

Original: TensorFlow .
Revised: TensorFlow .

Original: Specifically, we use the same transformer implementation found in https: //www.tensorflow.org
Revised: Specifically, we use the same transformer implementation found in https: //www.tensorflow.org

Original: /text/tutorials/transformer .
Revised: /text/tutorials/transformer .

Original: To compute RMSD, we first zero-center both the predicted and the ground-truth conformations.
Revised: To compute RMSD, we first zero-center both the predicted and the ground-truth conformations.

Original: Then, we obtain the optimal rotation matrix using the Kabsch algorithm implementation found in https://en.
Revised: Then, we obtain the optimal rotation matrix using the Kabsch algorithm implementation found in https://en.

Original: wikipedia.org/wiki/Kabsch_algorithm .
Revised: wikipedia.org/wiki/Kabsch_algorithm .

Original: It is straightforward to implement this algorithm in a python function.
Revised: It is straightforward to implement this algorithm in a python function.

Original: Please note that if TensorFlow is used, this python function needs to be wrapped inside a tf.py function and then followed by tf.stop gradient to prevent gradient computation.
Revised: Please note that if TensorFlow is used, this python function needs to be wrapped inside a tf.py function and then followed by tf.stop gradient to prevent gradient update.

Original: Finally, we rotate the predicted conformation onto its ground-truth for calculating the RMSD.
Revised: Finally, we rotate the predicted conformation onto its ground-truth for calculating the RMSD.

Original: The above details together with the already presented hyper-parameter setting should suffice to reproduce our results.
Revised: The above details together with the already presented hyper-parameter setting should suffice to reproduce our results.

Original: A. A TOM AND BOND FEATURES
Revised: A. A TOM AND BOND FEATURES

Original: We list the atom features and bond features together with the encoding method used to construct the proposed tensor in Tab.2.
Revised: We list the atom features and bond features together with the encoding method used to construct the proposed tensor in Tab.2.

Original: [Table]
Revised: [Table]

Original: A. I NTERESTING BEHAVIOUR OF TRAINING AND VALIDATION CURVE
Revised: A. T RAINING AND VALIDATION CURVE

Original: We present the train and validation plots for KL and reconstruction loss based on Drugs dataset in Fig.5a and Fig.5b, respectively.
Revised: We present the train and validation plots for KL and reconstruction loss based on Drugs dataset in Fig.5a and Fig.5b, respectively.

Original: Both plots are based on an initial KL weight of 1 e − 4 doubling every 62 .
Revised: Both plots are based on an initial KL weight of 1 e − 4 doubling every 62 .

Original: 5 k iterations (40 epochs).
Revised: 5 k iterations (40 epochs).

Original: While KL validation loss reached 18.29 after 1 e 6 iterations ( epochs), the reconstruction/RMSD loss reached 0 .
Revised: While KL validation loss reached 18.29 after 1 e 6 iterations ( epochs), the reconstruction/RMSD loss reached 0 .

Original: 64˚A at the end of training.
Revised: 64˚A at the end of training.

Original: During the firstepochs of training, model learning focused on reducing the KL loss due to it is orders of magnitude larger than the RMSD loss.
Revised: During the firstepochs of training, model learning focused on reducing the KL loss due to it is orders of magnitude larger than the RMSD loss.

Original: We were expecting this trend to continue for a while until both losses converge roughly in the same range.
Revised: We were expecting this trend to continue for a while until both losses converge roughly in the same range.

Original: However, much to our surprise, the model seemed to find a way to drastically reduce RMSD loss much earlier by leveraging the information from the GDR encoder; it learned to ”cheat” by directly reversing coordinate information embedded in the output of GDR encoder back to the original conformation.
Revised: However, much to our surprise, the model seemed to find a way to drastically reduce RMSD loss much earlier by leveraging the information from the GDR encoder; it learned to ”cheat” by directly reversing coordinate information embedded in the output of GDR encoder back to the original conformation.

Original: The RMSD loss dropped to as low as 0 .
Revised: The RMSD loss dropped to as low as 0 .

Original: 08˚A .
Revised: 08˚A .

Original: On the other hand, the KL loss climbed to almost 800, signaling signifcant divergence from standard normal distribution.
Revised: On the other hand, the KL loss climbed to almost 800, signaling signifcant divergence from standard normal distribution.

Original: At this stage, output of the GDR encoder contains informative features of the original 3D coordinates.
Revised: At this stage, output of the GDR encoder contains informative features of the original 3D coordinates.

Original: With the KL loss weight increasing, it becomes more difficult for the model to cheat since training is forcing the output of GDR encoder to conform to a standard uninformative
Revised: With the KL loss weight increasing, it becomes more difficult for the model to cheat since training is forcing the output of GDR encoder to conform to a standard uninformative

Original: Gaussian distribution.
Revised: Gaussian distribution.

Original: The KL loss started to drop while the RMSD loss remained steady, indicating increasing reliance on the output of G encoder for reconstructing the conformation.
Revised: The KL loss started to drop while the RMSD loss remained steady, indicating increasing reliance on the output of G encoder for reconstructing the conformation.

Original: As the output of
Revised: As the output of

Original: GDR encoder becomes less informative, the model learned to rely almost entirely on the aggregated feature from the G encoder to decode conformation.
Revised: GDR encoder becomes less informative, the model learned to rely almost entirely on the aggregated feature from the G encoder to decode conformation.

Original: We attempted to initiate the training with a much larger initial KL weight ( 1 e − 2 ) to prevent ”cheating” from begining.
Revised: We attempted to initiate the training with a much larger initial KL weight ( 1 e − 2 ) to prevent ”cheating” from begining.

Original: However, this quickly led to the notorious KL vanishing issue (Fu et al., 2019).
Revised: However, this quickly led to the notorious KL vanishing issue (Fu et al., 2019).

Original: We figure that ”cheating” is actually beneficial in that it reduces learning difficulty particularly for the decoder; its weights are tuned on easy training task, simply reversing what GDR encoder has done.
Revised: We figure that ”cheating” is actually beneficial in that it reduces learning difficulty particularly for the decoder; its weights are tuned on easy training task, simply reversing what GDR encoder has done.

Original: In other words, the tuned weights of the decoder already hold crucial information on how to decode highly informative input features.
Revised: In other words, the tuned weights of the decoder already hold crucial information on how to decode highly informative input features.

Original: As KL weight increases, model learning shifts to make the output of G encoder more informative.
Revised: As KL weight increases, model learning shifts to make the output of G encoder more informative.

Original: Also, this maybe an easier learning task as the RMSD loss is already very low (back-propagation of this loss contributes little to weight update); instead, model learning primarily focuses on optimizing the KL loss.
Revised: Also, this maybe an easier learning task as the RMSD loss is already very low (back-propagation of this loss contributes little to weight update); instead, model learning primarily focuses on optimizing the KL loss.

Original: This two-stage iterative loss optimization is much easier than optimizing both losses simultaneously throughout the training process.
Revised: This two-stage iterative loss optimization is much easier than optimizing both losses simultaneously throughout the training process.

Original: [Figure]
Revised: [Figure]

Original: A. G ENERATION PERFORMANCE ON D RUGS DATASET WITH F ORCE F IELD O PTIMIZATION
Revised: A. G ENERATION PERFORMANCE ON D RUGS DATASET WITH F ORCE F IELD O PTIMIZATION

Original: [Table]
Revised: [Table]

Original: A. M OELCULAR PROPERTY PREDICTION RESULTS
Revised: A. M OELCULAR PROPERTY PREDICTION RESULTS

Original: We conduct further experiment on molecular property prediction to demonstrate the effectiveness of the proposed feature engineering method.
Revised: We conduct further experiment on molecular property prediction to demonstrate the effectiveness of the proposed feature engineering method.

Original: Following Uni-Mol
Revised: Following Uni-Mol

Original: (Zhou et al., 2022) and GEM (Fang et al., 2022), we report property prediction result on the MolecularNet QM9 regression task ( https://moleculenet.org/datasets-1 ).
Revised: (Zhou et al., 2022) and GEM (Fang et al., 2022), we report property prediction result on the MolecularNet QM9 regression task ( https://moleculenet.org/datasets-1 ).

Original: The goal of this task is to estimate homo , lumo , and homo-lumo gap properties of molecules in the QM9 dataset based on their molecular structure.
Revised: The goal of this task is to estimate homo , lumo , and homo-lumo gap properties of molecules in the QM9 dataset based on their molecular structure.

Original: This task is different from the conformation generation experiment in that both 3D coordinates and 2D molecular graph of a molecule are supplied as input to a prediction model.
Revised: This task is different from the conformation generation experiment in that both 3D coordinates and 2D molecular graph of a molecule are supplied as input to a prediction model.

Original: The accuarcy of property prediction depends on how well a model can extract and aggregate such input information among atoms.
Revised: The accuarcy of property prediction depends on how well a model can extract and aggregate such input information among atoms.

Original: Similar to Uni-Mol, we adapt the proposed GDR encoder to this regression task by changing its prediction head.
Revised: Similar to Uni-Mol, we adapt the proposed GDR encoder to this regression task by changing its prediction head.

Original: Specifically, we use the same GDR transformer encoder structure as presented in
Revised: Specifically, we use the same GDR transformer encoder structure as presented in

Original: Fig.3 and add an additional mean pooling layer, which is then followed by a linear layer for property prediction.
Revised: Fig.3 and add an additional mean pooling layer, which is then followed by a linear layer for property prediction.

Original: To obtain training data, we follow the same data train-val-test split ( 8 : 1 : 1 ) in Uni-Mol and GEM and standardize the output property data.
Revised: To obtain training data, we follow the same data train-val-test split ( 8 : 1 : 1 ) in Uni-Mol and GEM and standardize the output property data.

Original: This results in 107748 train samples, 11967 val samples and 13306 test samples.
Revised: This results in 107748 train samples, 11967 val samples and 13306 test samples.

Original: We train the adapted model for 200 epochs with a batch size of 128.
Revised: We train the adapted model for 200 epochs with a batch size of 128.

Original: The learning rate schedule is the same as TensorVAE.
Revised: The learning rate schedule is the same as TensorVAE.

Original: We report the mean average error(MAE) over all the test samples.
Revised: We report the mean average error(MAE) over all the test samples.

Original: The result of the adapted model is compared to those of 7 other models including;
Revised: The result of the adapted model is compared to those of 7 other models including;

Original: • D-MPNN (Yang et al., 2019), AttentiveFP (Xiong et al., 2019) and GEM which are GNN based models without pretraining; • N-Gram (Liu et al., 2019), PretrainingGNN (Hu et al., 2019) and GROVER (Rong et al., 2020) with pretraining.
Revised: • D-MPNN (Yang et al., 2019), AttentiveFP (Xiong et al., 2019) and GEM which are GNN based models without pretraining; • N-Gram (Liu et al., 2019), PretrainingGNN (Hu et al., 2019) and GROVER (Rong et al., 2020) with pretraining.

Original: In particular, GROVER integrates GNN into a Transformer architecture, and there are two variants with different model capacity, GROVER base and GROVER large ; • and finally, Uni-Mol featuring a modified Transformer with a linear prediction head.
Revised: In particular, GROVER integrates GNN into a Transformer architecture, and there are two variants with different model capacity, GROVER base and GROVER large ; • and finally, Uni-Mol featuring a modified Transformer with a linear prediction head.

Original: The MAE for all compared methods are summaried in Tab.4.
Revised: The MAE for all compared methods are summaried in Tab.4.

Original: The proposed method outperforms all compared methods.
Revised: The proposed method outperforms all compared methods.

Original: Noticeably, the current state of the art Uni-Mol achieves its superior performance using a larger and modified transformer archiecture
Revised: Noticeably, the current state of the art Uni-Mol achieves its superior performance using a larger and modified transformer archiecture

Original: [Table]
Revised: [Table]

Original: and with pretraining on a large dataset containing 19M molecules.
Revised: and with pretraining on a large dataset containing 19 e 6 molecules.

Original: In constrast, the propsoed TensorVAE outperforms Uni-Mol with a stock and smaller transformer, and more importantly without any form of pretraining.
Revised: In constrast, the propsoed TensorVAE outperforms Uni-Mol with a stock and smaller transformer, and more importantly without any form of pretraining.

Original: Additionally, the Uni-Mol variant using the same Transformer encoder structure as ours, excluding the N × N pair-type interaction performs worse than its Evoformer counterpart.
Revised: Additionally, the Uni-Mol variant using the same Transformer encoder structure as ours, excluding the N × N pair-type interaction performs worse than its Evoformer counterpart.

Original: This experiment demonstrates that the proposed feature engineering method is very effective at global information aggregation
Revised: This experiment demonstrates that the proposed feature engineering method is very effective at global information aggregation

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A.
Revised: A.

Original: S AMPLES OF GENERATED CONFORMATIONS
Revised: S AMPLES OF GENERATED CONFORMATIONS

Original: Ground Truth
Revised: Ground Truth

Original: Generated
Revised: Generated

Original: Ground Truth
Revised: Ground Truth

Original: Generated
Revised: Generated

Original: Ground Truth
Revised: Ground Truth

Original: Generated
Revised: Generated

Original: Ground Truth
Revised: Ground Truth

Original: Generated
Revised: Generated