M OLECULAR G EOMETRY P RETRAINING WITH SE(3)-I NVARIANT D ENOISING D ISTANCE M ATCHING
Anonymous authors
Paper under double-blind review
A BSTRACT
Pretraining molecular representations is critical in a variety of applications for drug and material discovery due to the limited number of labeled molecules, yet most existing work focuses on pretraining on 2D molecular graphs.
The power of pretraining on 3D geometric structures, however, has been less explored.
This is owning to the difficulty of finding a sufficient proxy task that can empower the pretraining to effectively extract essential features from the geometric structures.
Motivated by the dynamic nature of 3D molecules, where the continuous motion of a molecule in the 3D Euclidean space forms a smooth potential energy surface, we propose a 3D coordinate denoising pretraining framework to model such an energy landscape.
Leveraging an SE(3)-invariant score matching method, we propose
GeoSSL in which the coordinate denoising proxy task is effectively boiled down to denoising the pairwise atomic distances in a molecule.
Our comprehensive experiments confirm the effectiveness and robustness of our proposed method
I NTRODUCTION
Learning effective molecular representations is critical in a variety of tasks in drug and material discovery, such as molecular property prediction [13, 19, 20, 70], de novo molecular design and optimization [6, 49, 73], and retrosynthesis and reaction planning [21, 48, 60].
Recent work based on graph neural networks (GNNs) [19] has shown superior performance thanks to the simplicity and effectiveness of GNNs in modeling graph-structured data.
However, the problem remains challenging due to the limited number of labeled molecules as it is in general expensive and time-consuming to label molecules, which usually requires expensive physics simulations or wet-lab experiments.
As a result, recently, there is growing interest in developing pretraining or self-supervised learning methods for learning molecular representations by leveraging the huge amount of unlabeled molecule data [27, 34, 59, 71].
These methods have shown superior performance on many tasks, especially when the number of labeled molecules is insufficient.
One limitation of these approaches is, however, that they represent molecules as topological graphs and molecular representations are learned through pretraining 2D topological structures ( i.e. , based on the covalent bonds).
But intrinsically, for molecules, a more natural representation is based on their 3D geometric structures, which largely determine the corresponding physical and chemical properties.
Indeed, recent work [19, 35] has empirically verified the importance of applying 3D geometric information for molecular property prediction tasks.
Therefore, a more promising direction is to pretrain molecular representations based on their 3D geometric structures, which is the main focus of this paper.
The main challenge for molecule geometric pretraining arises from discovering an effective proxy task to empower the pretraining to extract essential features from the 3D geometric structures.
Our proxy task is motivated by the following observations.
Studies [44] have shown that molecules are not static but in a continuous motion in the 3D Euclidean space, forming a potential energy surface (PES).
As shown in Figure 1, it is desirable to study the molecule in the local minima of the PES, called conformer .
However, such stable state conformer often comes with different noises for the following reasons.
First, the statistical and systematic errors on conformation estimation are unavoidable [10].
Second, it has been well-acknowledged that a conformer can have vibrations around the local minima in PES.
Such characteristics of the molecular geometry motivate us to attempt to denoise the molecular coordinates around the local minima, to mimic the computation errors and conformation vibration within the corresponding local region.
The denoising goal is to learn molecular representations which are robust to such noises and effectively capture the energy surface around the local minima.
To achieve the aforementioned goal, we propose GeoSSL, an SE(3)- invariant denoising distance matching pretraining algorithm.
In a nutshell, to capture the smooth energy surface around the local minima, we aim to maximize the mutual information (MI) between a given stable geometry and its perturbed version ( i.e. , gand g 2 in Figure 1).
In practice, it is difficult to directly maximize the mutual information between two random variables.
Thus, we propose to maximize a lower-bound of the above mutual information, which in turn amounts to denoising a geometric structure.
Moreover, directly denoising such noisy coordinates, nevertheless, remains challenging because one may need to effectively constrain the pairwise atomic distances while changing the atomic coordinates.
To cope with this obstacle, we further leverage an SE(3)-invariant score matching method to successfully transform the coordinate denoising desire to the denoising of pairwise atomic distances, which then can be effectively computed.
In other words, our pretraining proxy task, namely mutual information maximization, effectively boils down to achieving an intuitive learning objective: denoising a molecule’s pairwise atomic distances.
We empirically verify, using 22 downstream geometric molecular prediction tasks, that our method outperforms nine pretraining baselines.
Our main contributions are summarized as follows.
(1) We propose a novel coordinate denoising method for molecular geometry pretraining, which to the best of our knowledge is the first to only utilize 3D molecular data for pretraining 1 .
(2) To overcome the challenge of attaining the coordinate denoising objective, we introduce an SE(3)-invariant score matching strategy to successfully transform such objective into the denoising of pairwise atomic distances, which can be effectively computed.
We empirically demonstrate the effectiveness and robustness of our proposed method, GeoSSL.
R ELATED W ORK 2. E QUIVARIANT G EOMETRIC M OLECULE R EPRESENTATION L EARNING
Geometric representation learning.
Recently, 3D geometric representation learning has been widely explored in the machine learning community, including but not limited to 3D point clouds [7, 40, 51, 63], N-body particle [41, 43], and 3D molecular conformation [5, 30, 31, 37, 46, 47, 52], amongst many others.
The learned representation should satisfy the physical constraints, e.g. , it should be equivariant to the transition on the Euclidean space.
Such constraints can be depicted with the group symmetry as introduced below.
SE(3)-invariant energy.
Constrained by the physical nature of 3D data, a key principle we need to follow is to learn an SE(3)-equivariant representation function.
The SE(3) is the special Euclidean group consisting of rigid transformations in the 3D Cartesian space, where the transformations include all the combinations of translations and rotations.
Namely, the learned representation should be equivariant to translations and rotations for molecule geometries.
We also note that for some specific tasks like molecular chirality [1], the representation needlessly satisfy the reflection equivariance.
For more rigorous discussion, please check [16, 18, 61].
In this work, we will design an SE(3)-invariant energy function based on an SE(3)-equivariant representation backbone model.
2. S ELF -S UPERVISED L EARNING FOR M OLECULE R EPRESENTATION L EARNING
In general, there are two categories of self-supervised learning (SSL) [36, 38, 67, 68]: contrastive and generative, and the main difference is if the supervised signals are constructed in an inter-data or intra-data manner.
Contrastive SSL extracts two views from the data and designs the supervised signals by detecting if the sampled view pairs are from the same data, and generative SSL learns structural information by reconstructing partial information from the data itself.
2D molecular graph (topology) self-supervised learning.
One of the mainstream research lines for molecule pretraining is on the 2D molecular graph.
It treats the molecules as 2D graphs, where atoms and bonds are nodes and edges respectively.
It then carries out a pretraining task by either detecting if the two augmentations ( e.g. , neighborhood extraction, node dropping, edge dropping, etc) correspond to the same molecular graph [27, 59, 71] or if the representation can successfully reconstruct certain substructures of the molecular graphs [27, 28, 34].
3D molecular graph (geometry) self-supervised learning.
Self-supervised learning for 3D molecular graphs is still underexplored.
Existing methods leverage both 2D and 3D molecular graphs to improve the representation learning [15, 35].
For example, ChemRL-GEM [15] encodes both the 2D topology and the 3D geometry information of molecules to enhance the graph representation.
GraphMVP [35] makes use of the contrastiveness of the 2D and 3D views of a molecule to form informative graph representation for 2D downstream tasks.
Unlike the above approaches, our method uses 3D geometry data only and aims specifically at 3D downstream applications.
We note that there is a parallel work [72], which is also under review; Appendix E provides a detailed comparison with this work, highlighting the fact that the parallel work is indeed a special case of our GeoSSL.
P RELIMINARIES
Molecular geometry graph.
Molecules can be naturally featured in a geometric formulation, i.e. , all the atoms are spatially located in 3D Euclidean space.
Note that the covalent bonds are added heuristically by expert rules, so they are only applicable in 2D topology graph not 3D geometry graph.
Besides, atoms are not static, but in a continual motion along a potential energy surface [2].
The 3D structures at the local minima on this surface are named conformer , as shown in Figure 1.
Conformers at such equilibrium state possess nice properties and we would like to model them during pretraining.
Geometric neural network.
We denote each conformer as g =
( X, R ) .
Here X ∈ R n × d is the atom attribute matrix and R ∈ R n × 3 is the atom 3D-coordinate matrix, where n is the number of atoms and d is the feature dimension.
The representations for the i -th node and whole molecule are:
[Equation]
where T is the transformation function like atom masking, and READOUT is the readout function.
In this work, we take the mean over all the node representations as the readout function.
Energy-based model and denoising score matching.
Energy-based model (EBM) is a flexible and powerful tool for modeling data distribution.
It has the form of Gibbs distribution as p θ ( x ) = exp( − E ( x )) /A , where p θ ( x ) is the model distribution and A denotes the normalization constant.
The computation of such probability is intractable due to the high cardinality of the data space.
Also, great progresses have been recently made in solving this intractable function, including contrastive divergence [11], noise contrastive estimation [24], and score matching (SM) [29, 56, 57].
For example,
SM solves this by first introducing the so-called score, the gradient of the log-likelihood with respect to the data, and then matching the model score with the data score using Fisher divergence.
This approach has been further improved by combining SM with denoising auto-encoding, forming the promising denoising score matching (DSM) strategy [65].
In this work, we will explore the potential of leveraging DSM for molecule geometry representation learning.
We aim at utilizing the pairwise distance information, one of the most fundamental factors in the geometric data.
Problem setup.
Our goal here is to apply a self-supervised pretraining algorithm on a large molecular geometric dataset, and adapt the pretrained representation for fine-tuning on geometric downstream tasks.
For both the pretraining and downstream tasks, only the 3D geometric information is available, and our solution is agnostic in terms of the backbone geometric neural network.
M ETHOD
This section introduces our proposed GeoSSL.
We start with exploring the coordinate perturbation for molecular data in Section 4.1.
Then we propose a coordinate-aware MI maximization formula and turn it into a coordinate denoising problem in Section 4.2.
Nevertheless, the coordinate denoising is non-trivial, since it requires the geometric data reconstruction, and we adopt the score matching for estimation, as introduced in Section 4.3.
The ultimate training objective is discussed in Section 4.4.
4. C OORDINATE P ERTURBATION FOR G EOMETRIC D ATA
The mainstream self-supervised learning community designs the pretraining task by defining multiple views from the data, and these views share common information to some degree.
Thus, by designing generative or contrastive task to maximize the mutual information (MI) between these views, the pretrained representation can encode certain key information.
This will make the representation more robust and can be more generalizable to downstream tasks.
In our work, we propose GeoSSL, an SE(3)- invariant self-supervised learning (SSL) method for molecule geometric representation learning.
The 3D geometric information, or the atomic coordinates are critical to molecular properties.
We carry out an additional ablation study to verify this in Appendix B. Then based on this acknowledgement, we propose a geometry perturbation, which adds small noises to the atom coordinates.
For notation, following Section 3, we define the original geometry graph and an augmented geometry graph as two views, denoted as g 1 =
( X 1 , R 1 ) and g
=
( X 2 , R 2 ) respectively.
The augmented geometry graph can be seen as a coordinate perturbation to the original graph with the same atom types, i.e. ,
X
= X 1 and R 2 =
R 1 + ϵ , where ϵ is drawn from a normal distribution.
4. C OORDINATE D ENOISING WITH M UTUAL I NFORMATION M AXIMIZATION
The two views defined above share certain common information.
By maximizing the mutual information (MI) between them, we expect that the learned representation can better capture the geometric information and is robust to noises and thus can generalize well to the target downstream tasks.
To maximize the MI, we turn to maximizing the following lower bound on the two geometry views:
[Equation]
In Equation (2), we transform the MI maximization problem into maximizing the summation of two conditional log-likelihoods.
In addition, these two conditional log-likelihoods are in the mirroring direction, and such symmetry can reveal certain nice properties, e.g. , it highlights the equal importance and uncertainty of the two views and can lead to a more robust representation of the geometry.
To solve Equation (2), we introduce the energy-based model (EBM) for estimation.
EBM has been acknowledged as a flexible framework for its powerful usage in modeling distribution over highlystructured data, like molecules [26, 33].
To adapt it for MI maximization in our setting, the lower bound can be turned into:
[Equation]
where the f ( · ) are the negative of energy functions, and A R 1 | g 2 and A R 2 | g 1 are the intractable partition functions.
The first equation in Equation (3) results from that the two views share the same atom types.
This equation can be treated as denoising the atom coordinates of one view from the geometry of the other view.
In the following, we will explore how to use the score matching for solving the above EBM, and further to transform the coordinate-aware mutual information maximization to the denoising distance matching as the final objective.
4. F ROM C OORDINATE D ENOISING TO D ISTANCE D ENOISING : G EO SSL
Before going into details, first we would like to briefly discuss the denoising score matching (DSM).
DSM has three main advantages that inspire us to apply it for solving the coordinate-aware MI.
The DSM solution has a nice formulation, such that the final objective function can be simplified with an intuitive explanation: GeoSSL can be seen as solving the denoising pairwise distance at multiple noise levels.
The score defined in geometric data can be viewed as a coordinate-based pseudoforce.
Such pseudo-force can play an important role for the corresponding geometric representation learning.
In terms of the MI maximization, existing methods like InfoNCE [64], EBM-NCE and
Representation Reconstruction [35] map the data to the representation space for either inter-data contrastive learning or intra-data reconstruction.
This operation can avoid the decoding design issue for highly-structured data [12], yet the trade-off is losing the data-inherent information by certain degree.
In other words, the data-level reconstruction task ( e.g. , DSM) is expected to lead to a more robust representation.
Thus, by combining the above three points, we adopt DSM to our problem and propose GeoSSL.
We expect that it is able to learn an expressive geometric representation function by solving coordinate-aware SSL task.
In addition, the two terms in Equation (3) are in the mirroring direction.
Thus in what follows, we may as well adopt a proxy task that these two directions can be calculated separately, and we take one direction for illustration, e.g. , log exp( f ( R 1 , g 2 ))
A R 1 | g.
4.3. D ENOISING D ISTANCE M ATCHING
Score.
The score is defined as the gradient of the log-likelihood w.r.t.
the data, i.e. , the atom coordinates in our case.
Because the normalization function is a constant w.r.t.
the data, it will disappear during the score calculation.
To adapt it into our setting, the score is obtained as the gradient of the negative energy function w.r.t.
the atom coordinates, as:
[Equation]
If we assume that the learned optimal energy function, i.e. , f ( · ) , possesses certain physical or chemical information, then the score in Equation (4) can be viewed as a special form of the pseudo-force.
This may require more domain-specific knowledge, and we leave this for future exploration.
Score decomposition: from coordinates to distances.
Through back-propagation [50], the score on atom coordinates can be further decomposed into the scores attached to pairwise distances:
[Equation]
where r 1 ,i is the i -th coordinate in g 1 , d 1 ,ij denotes the pairwise distance between the i -th and j -th nodes in g 1 , and s ( d 1 , g 2 ) ij ≜ ∂f ( R 1 , g 2 ) ∂d 1 ,ij .
Such decomposition has a nice intuition from the pseudoforce perspective: the pseudo-force on each atom can be further decomposed as the summation of pseudo-forces attached to the pairwise distances between this atom and all its neighbors.
Note that here the pairwise atoms are connected in the 3D Euclidean space, not by the covalent bonds.
Denoising distance matching (DDM).
Then we adopt the denoising score matching (DSM) [65] to our task.
To be more concrete, we take the Gaussian kernel as the perturbed noise distribution on each pairwise distance, i.e. , q σ ( ˜ d 1 | g 2 ) =
E p data ( d 1 | g 2 )
[ q σ ( ˜ d 1 | d 1 )] , where σ is the deviation in Gaussian perturbation.
One main advantage of using the Gaussian kernel is that the following gradient of conditional log-likelihood has a closed-form formulation: ∇ ˜ d 1 log q σ ( ˜ d 1 | d 1 , g 2 ) =
( d 1 − ˜ d 1 ) /σ 2 , and the objective function of DSM is to train a score network to match it.
This trick was first introduced in [65], and has been widely utilized in the deep generative modeling tasks [54, 55].
To adapt to our setting, this is essentially saying that we want to train a “distance network”, i.e. , s θ ( ˜ d 1 | g 2 ) , to match the distance perturbation, or we can say it aims at matching the pseudo-force with the pairwise distances from the pseudo-force aspect.
By taking the Fisher divergence as the discrepancy metric and the trick mentioned above, the estimation objective can be simplified to
[Equation]
For more detailed derivations, please refer to Appendix C. In this section, we turn the coordinateaware MI maximization problem into distance perturbation matching problem, which is equivalent to denoising distance matching.
The corresponding pipeline is illustrated in Figure 2.
4.3. SE(3)-I NVARIANT D ISTANCE N ETWORK M ODELING
To solve Equation (6), the key step is to design an SE(3)-invariant distance network.
For modeling, we take an SE(3)-equivariant 3D geometric graph neural network as the geometric representation backbone model, i.e. , the h ( · ) .
Following the notations in Section 3 and g 2 modeling, we can have h
[Equation]
i =
3D-GNN
[Equation]
for the atom-level and atom pairwise-level representation respectively.
Based on the pairwise representation, we define the distance network as:
[Equation]
where ⊕ is the concatenation and MLP is the multi-layer perceptron.
Because the backbone model satisfies SE(3)-equivariance and the distance network in Equation (8) is based on the pairwise distance and SE(3)-equivariant representation, our proposed GeoSSL pretraining is SE(3)-invariant [18].
4. U LTIMATE O BJECTIVE
With the above distance network modeling, we can formulate the ultimate objective function.
We further adopt the following four training tricks from [35, 54, 55] to stabilize the score matching training process.
(1) We carry out the distance denoising at L -level of noises.
(2) We add a weighting coefficient λ ( σ )
= σ β for each noise level, where β acts as the annealing factor.
(3) We scale the score network by a factor of 1 /σ .
(4) We sample the exactly same atoms from the two geometry views with a masking ratio r .
Finally, the ultimate objective function for GeoSSL is as follows:
[Equation]
L
L (cid:88) l = σ βl E p data ( d 2 | g 1 ) E q ( ˜ d 2 | d 2 , g 1 ) (cid:104)(cid:13)(cid:13)(cid:13)
s θ
( ˜ d 2 , g 1 ) σ l − d 2 − ˜ dσ 2 l (cid:13)(cid:13)(cid:13) (cid:105) .
[Table]
The algorithm is in Algorithm 1.
Comparison with score matching in generative modeling.
We note that score matching has been widely used for generative modeling task.
One of the main drawbacks in the generative setting is the long mixing time for MCMC sampling.
However, our work aims at representation learning, so the sampling issue won’t affect our task.
We further note that there also exists a series of work exploring the score matching for conformation generation [50].
However, their scores or pseudo-forces are attached to the 2D topology (the covalent bonds), while our work is based on the pairwise distances from the geometric coordinates only.
E XPERIMENTS
In this section, we compare our method with nine 3D geometric pretraining baselines, including one randomly-initialized, one supervised, and seven self-supervised approaches.
For the downstream tasks, we adopt 22 tasks covering quantum mechanics prediction, force prediction, and binding affinity prediction.
We provide all the experiment details and ablation studies in Appendix D.
5. B ACKBONE M ODELS
Our proposed GeoSSL is model-agnostic, and here we evaluate our method using one of the state-ofthe-art geometric graph neural networks, PaiNN [47].
We carry out the exactly same experiments on another backbone model, SchNet [45], and results are in in Appendix D due to space limit.
PaiNN [47] is a follow-up work of SchNet [45].
It addresses the limitation of rotational equivariance in SchNet by embracing rotational invariance, attaining a more expressive 3D geometric model.
Other backbone models.
First we want to highlight that what we propose is a general solution and is agnostic to the backbone 3D geometric models.
And in addition to the PaiNN, we want to acknowledge that there are more recent progress in this research line, including but not limited to [5, 16, 16, 31, 37, 43, 52].
Yet, they may require large computation resources and may be infeasible ( e.g. , out of GPU memory) in our setting.
The decision is made by considering the model performance, computation efficiency, and memory cost.
For more benchmark results and detailed comparisons of the 3D geometric models, please check Appendix A.
5. B ASELINES AND P RETRAINING D ATASET
Pretraining dataset.
The PubChemQC database is a large-scale database with around 4M molecules with 3D geometries, and it calculates both the ground-state and excited-state 3D geometries using DFT (density functional theory).
Due to the high computational cost, only several thousand molecules can be processed every day, and this dataset takes years of efforts in total.
Following this, Molecule3D [69] takes the ground-state geometries from PubChemQC and transforms the data formats into a deep learning-friendly way.
It also parses essential quantum properties for each molecule, including energies of the highest occupied molecular orbital (HOMO) and the lowest occupied molecular orbital (LUMO), the energy gap between HOMO-LUMO, and the total energy.
For our molecular geometry pretraining, we take a subset of 1M molecules with 3D geometries from Molecule3D.
Self-supervised learning pretraining baselines.
We first consider the four coordinate-MI-unaware
SSL methods: (1) Type Prediction is to predict the atom type of masked atoms; (2) Distance
Prediction aims to predict the pairwise distances among atoms; (3) Angle Prediction is to predict the angle among triplet atoms, i.e. , the bond angle prediction; (4) 3D InfoGraph adopts the contrastive learning paradigm by taking the node-graph pair from the same molecule geometry as positive and negative otherwise.
Next, following the coordinate-aware MI maximization framework introduced in Equation (2), we include two contrastive and one generative SSL baselines.
(5) InfoNCE [64] and (6) EBM-NCE [35] are the two widely-used contrastive learning loss functions, where the goal is to simultaneously align the positive views and contrast the negative views.
Finally, (7) Representation
Reconstruction (RR) [35] is a generative SSL that is proxy to maximize the MI.
It is a more general form of non-contrastive SSL methods like BOYL [22] and SimSiam [8], and the goal is to reconstruct each view from its counterpart in the representation space.
Following this, our proposed GeoSSL can be classified as generative SSL, yet it aims at denoising the pairwise distances instead.
Supervised pretraining baseline.
We also compare our method with a supervised pretraining baseline.
As aforementioned, the large-scale pretraining dataset uses the DFT to calculate the energy, and extracts the most stable conformers with the lowest energies, which reveal the most fundamental properties of molecules in the 3D Euclidean space.
Thus, such energies can be naturally adopted as the supervised signals, and we take this as a supervised pretraining baseline.
5. D OWNSTREAM T ASKS ON Q UANTUM M ECHANICS AND F ORCE P REDICTION QM
[42] is a dataset of 134K molecules consisting of 9 heavy atoms.
It includes 12 tasks that are related to the quantum properties.
For example, U0 and U298 are the internal energies at 0K at 0K and 298.15K respectively, and U298 and G298 are the other two energies that can be transferred from
[Table] [Table]
H298 respectively.
The other 8 tasks are quantum mechanics related to the DFT process.
MD17 [9] is a dataset on molecular dynamics simulation.
It includes eight tasks, corresponding to eight organic molecules, and each task includes the molecule positions along the potential energy surface (PES), as shown in Figure 1.
The goal is to predict the energy-conserving interatomic forces for each atom in each molecule position.
We follow the literature [30, 37, 46, 47] of using 1K for training and 1K for validation, while the test set (from 48K to 991K) is much larger.
The results on QM9 and MD17 are displayed in Tables 1 and 2 respectively.
From Tables 1 and 2, we can observe that most the pretraining baselines tested perform on par with or even worse than the randomly-initialized baseline.
The top performing baseline is the representation reconstruction method (RR), which optimizes the coordinate-aware MI; it outperforms the other baselines on 5 out of 12 tasks in QM9 and 6 out of 8 tasks in MD17.
This implies the potential of applying generative SSL for maximizing this coordinate-aware MI.
Promisingly, our proposed GeoSSL achieves consistently improved performance on all the 12 tasks in QM9 and 8 tasks in MD17.
All these observations empirically verify the effectiveness of the distance denoising in GeoSSL, which models the most determinant factor in molecule geometric data.
5. D OWNSTREAM T ASKS ON B INDING A FFINITY P REDICTION
Atom3D [62] is a recently published dataset.
It gathers several core tasks for 3D molecules, including binding affinity.
The binding affinity prediction is to measure the strength of binding interaction between a small molecule to the target protein.
Here we will model both the small molecule and protein with their 3D atom coordinates provided.
We follow Atom3D in data preprocessing and data splitting.
For more detailed discussions and statistics, please check Appendix D.
During the binding process, there is a cavity in a protein that can potentially possess suitable properties for binding a small molecule (ligand), and it is termed a pocking [58].
Because of the large volume of the protein, we follow [62] by only taking the binding pocket, where there are no more thanatoms for each molecule and protein pair.
To be more concrete, we consider two binding affinity tasks.
(1) The first task is ligand binding affinity (LBA).
It is gathered from [66] and the task is to predict the binding affinity strength between a small molecule and a protein pocket.
The second task is ligand efficacy prediction (LEP).
We have a molecule bounded to pockets, and the goal is to detect if the same molecule has a higher binding affinity with one pocket comparing to the other one.
Results in Table 3 illustrate that, for the LBA task, two pretraining baseline methods fail to generalize to LBA (the loss gets too large), and all the other pretraining baselines cannot beat the randomlyinitialized baseline.
For the LEP task, the supervised and two contrastive learning pretraining
[Table]
baselines stand out for both ROC and PR metrics.
Meaningfully, for both tasks, GeoSSL is able to achieve promising improvement, revealing that modeling the local region around conformer with distance denoising can also benefit for binding affinity downstream tasks.
5. D ISCUSSION : C ONNECTION WITH M ULTI - TASK P RETRAINING
In the above experiments, we test multiple self-supervised and supervised pretraining tasks separately.
Yet, all these pretraining methods are not contradict, but could be complementary instead.
Existing work has successfully shown the effect of combining them in various ways.
For example, [27] shows that jointly doing supervised and self-supervised pretraining can augment the pretrained representation.
[35, 53] prove that contrastive and generative SSL pretraining methods can be learned simultaneously as a multi-task pretraining.
In addition, in terms of the molecule-specific pretraining, [35] empirically verifies that 2D topology and 3D geometry views can share certain information, and maximizing their mutual information together with 2D topology SSL for pretraining is beneficial.
With these insights, we would like to claim that all of these points are worth exploration in the future, especially in the line of pretraining for molecular geometry.
Because pretraining datasets often come with multiple quantum properties and the 2D molecular topology can be obtained heuristically.
Yet as the first step to explore the self-supervised learning using only the 3D geometric data ( i.e. , without covalent bonds), our study here would like to leave multi-task pretraining for future exploration.
C ONCLUSIONS AND F UTURE D IRECTIONS
We proposed a novel coordinate denoising method, coined GeoSSL, for molecular geometry pretraining.
GeoSSL leverages an SE(3)-invariant score matching strategy, under the energy-based model (EBM) framework, to successfully decompose its coordinate denoising objective into the denoising of pairwise atomic distances in a molecule, which then can be effectively computed and directly target the determinant factors in molecular geometric data.
We empirically verified the effectiveness and robustness of our method, showing its superior performance to nine state-of-the-art pretraining baselines on 22 benchmarking geometric molecular property prediction and binding affinity tasks.
Our work here opens up venues for multiple promising directions.
First from the machine learning perspective, we propose a general pipeline on using EBM for MI maximization on geometric data pretraining.
Yet, there are more explorations on the success of EBM, like GFlowNet [3], and it would be interesting to explore how to combine it with molecular geometric data along this systematic path.
In addition, GeoSSL does not utilize the 2D structure ( i.e. , covalent bonds for molecules), and it would be desirable to consider how to utilize the distance denoising together with the 2D topology information.
In terms of applications, our proposed GeoSSL is a general framework, and it can be naturally applied to other geometric data, such as point clouds and protein pretraining.
In addition, our current goal is to perform denoising in the local region, yet it would be interesting to explore larger regions.
From this aspect, the denoising can be viewed as recovering the molecular dynamics trajectory, and we would explore how generalizable this pretrained representation is to the downstream tasks.
E THICS S TATEMENT
We authors acknowledge that we have read and commit to adhering to the ICLR Code of Ethics.
R EPRODUCIBILITY S TATEMENT
To ensure the reproducibility of the empirical results, we provide the implementation details (hyperparameters, dataset statistics, etc.)
in Section 5 and appendix D, and publically share our source code through this anonymous link.
Besides, the complete derivations of equations and clear explanations are given in Section 4 and appendix C
A B ENCHMARKS AND R ELATED W ORK A. G EOMETRIC N EURAL N ETWORKS
Recently, geometric neural networks have been actively proposed, including SchNet [46], TFN [16],
DimeNet++ [30], SE(3)-Trans [16], EGNN [43], SEGNN [5], SphereNet [37], SpinConv [52],
PaiNN [47], and GemNet [31].
We reproduce most of them on the QM9 dataset as shown in Table 4.
Among this, we would like to highlight two models: SchNet and PaiNN.
SchNet [45] is composed of the following key steps:
[Equation]
where K is the number of hidden layers, and
[Equation]
is the continuous-filter convolution layer, enabling the modeling of continuous coordinates of atoms.
PaiNN [47] is an improved work of SchNet [45].
It addresses the limitation of rotational equivariance in SchNet by embracing rotational invariance, attaining a more expressive SE(3)-equivariant neural network model.
A. B ENCHMARK ON QM
Current work is using different optimization strategies and different data split (in terms of the splitting size).
Originally there are 133,885 molecules in QM9, where 3,054 are filtered out, leading to 130, molecules.
During the benchmark, we find that:
• The performance on QM9 is very robust to either using (1) 110K for training, 10K for val, 10,831 for test or using (2) 100K for training, 13,083 for val and 17,748 for test.
• The optimization, especially the learning rate scheduler is very critical.
During the benchmarking, we find that using cosine annealing learning rate schedule [39] is generally the most robust.
For more detailed discussion on QM
[Table]
please refer to Appendix D. We show the benchmark results on QM9 in Table 4.
[Table]
A. R ELATED W ORK
We acknowledge that there is a parallel work called Protein Tertiary SSL (PTSSL) [23] working on the geometric self-supervised learning.
Yet, there are some fundamental differences between theirs and ours, as listed below: (1) Key notion on pseudo-force.
PTSSL directly applies the denoised score matching method into protein tertiary structures, yet our focus is on how the notion of pseudo-force can come into the play, which possess better generalization ability.
(2) Task setting.
PTSSL works on protein and utilize both the 2D and 3D information, and our work is purely working on the 3D geometric information.
(3) Technical novelty.
PTSSL designs the DSM objective for
SSL, and what we propose is a systematic tool: using energy-based model and score matching to solve the geometric SSL problem opens a new venue in this field.
(4) Objective.
PTSSL directly designs one objective function, which is denoising from one view to the other.
Ours starts from the lower bound of MI, which is symmetric in terms of the denoising directions.
We believe that such symmetry are treating the two views equally, and can better reveal the mutual concept, making the pre-trained representation more robust to the position augmentations.
(5) Empirical baseline.
PTSSL lacks the comparisons with other pre-training methods, while we compare with 7 SOTA pre-training methods, especially those driven by maximizing the MI with the same augmentations.
Without such comparisons, it is hard to tell the effectiveness of the pseudo-force matching for geometric data.
Score network.
Last but not least, the score network designed in PTSSL does not satisfy the SE(3) equivariant property.
B A N E XAMPLE O N T HE I MPORTANCE OF A TOM C OORDINATES
First it has been widely acknowledged [14] that the atom positions or molecule shapes are important factors to the quantum properties.
Here we carry out an evidence example to empirically verify this.
The goal here is to make predictions on 12 quantum properties in QM9.
The molecule geometric data includes two main components as input features: the atom types and atom coordinates.
Other key information can be inferred accordingly, including the pairwise distances and torsion angles.
We consider corruption on each of the component to empirically test their importance accordingly.
• Atom type corruption.
There are in total 118 types of atom types, and the standard embedding option is to apply the one-hot encoding.
In the corruption case, we replace all the atom types with a hold-out index, i.e. , index 119. • Atom coordinate corruption.
Originally QM9 includes atom coordinates that are in the stable state, and now we replace them with the coordinates generated with MMFF [25] from RDKit [32].
[Table]
We take SchNet and PaiNN as the backbone 3D GNN models, and the results are in Table 5.
We can observe that (1) Both corruption examples lead to performance decrease.
(2) The atom coordinate corruption may lead to more severe performance decrease than the atom type corruption.
To put this into another way is that, when we corrupt the atom types with the same hold-out type, it is equivalently to removing the atom type information.
Thus, this can be viewed as using the equilibrium atom coordinates alone, and the property prediction is comparatively robust.
This observation can also be supported from the domain perspective.
According to the valence bond theory, the atom type information can be implicitly and roughly inferred from the atom coordinates.
Therefore, by combining all the above observations and analysis, one can draw the conclusion that, for molecule geometry data, the atom coordinates reveal more fundamental information for representation learning .
C M UTUAL I NFORMATION M AXIMIZATION WITH E NERGY -B ASED M ODEL
In this section, we will give a detailed discussion on the mutual information (MI) maximization with the energy-based model (EBM).
First we can get a lower bound of MI.
Assuming that there exist (possibly negative) constants a and b such that a ≤ H ( X ) and b ≤ H ( Y ) , i.e. , the lower bounds to the (differential) entropies, then we have:
[Equation]
where the loss L MI is defined as:
[Equation]
Empirically, we use energy-based models to model the distributions.
The existence of a and b can be understood as the requirements that the two distributions ( p x , p y ) are not collapsed.
Notice that to keep consistent with the notations in Section 3, we will be using g 1 and g 2 as the two variables.
Then the goal is equivalent to optimizing the following equation:
[Equation] [Equation]
Thus, we transform the MI maximization problem into maximizing the summation of two conditional log-likelihoods.
Such objective function opens a wider venue for estimating MI, e.g. , using the EBM to estimate Equation (14).
Adaptation to Geometric Data
The 3D geometric information, or the atomic coordinates are critical to molecular properties.
Then based on this, we propose a geometry perturbation, which adds small noises to the atom coordinates.
This geometry perturbation possess certain motivations from both domain and machine learning perspectives.
(1) From the practical experiment perspective, the statistical and systematic errors [10] on conformation estimation are unavoidable.
Coordinate perturbation is a natural way to enable learning representations robust to such noises.
(2) From the domain aspect, molecules are not static but in a continuous motion in the 3D Euclidean space, and we can obtain a potential energy surface accordingly.
We are interested in modeling the conformer, i.e. , the 3D coordinates with the lowest energy.
However, even the conformer at the lowest energy point can have vibrations, and coordinate perturbation can better capture such movement yet with the same order of magnitude on energies.
(3) As will be illustrated later, our proposed method can be simplified as denoising atomic distance matching.
(4) Leveraging coordinate perturbation for model regularization has also been empirically verified its effectiveness for supervised molecule geometric representation learning [20].
Such characteristics of the molecular geometry motivate us to apply the coordinate perturbation.
If we take each of the two views as adding noise to the coordinates from the other view, then the objective in Equation (14) essentially states that we want to conduct coordinate denoising, as shown in Figure 3.
Yet, this is not a trivial task due to the complicated geometric space ( e.g. , 3D coordinates) reconstruction.
C. A N EBM FRAMEWORK FOR MI ESTIMATION
The lower bound in Equation (14) is composed of two conditional log-likelihood terms, and then we model the conditional likelihood with EBM.
This gives us:
[Equation] [Equation]
where f g 1 ( g 1 , g 2 ) = − E ( g 1 | g 2 ) and f g 2 ( g 2 , g 1 ) = − E ( g 2 | g 1 ) are the negative energy functions, and A g 1 | g 2 and A g 2 | g 1 are the corresponding partition functions.
The energy functions can be flexibly defined, thus the bottleneck here is the intractable partition function due to the high cardinality.
To solve this, existing methods include noise-contrastive estimation (NCE) [24] and score matching (SM) [56, 57], and we will describe how to apply them for MI maximization.
C. EBM-NCE FOR MI ESTIMATION
Under the EBM framework, if we solve Equation (15) with Noise-Contrastive Estimation (NCE) [24], the final objective is termed EBM-NCE, as:
[Equation]
All the detailed derivations can be found in [24].
Specifically, EBM-NCE is equivalent to the JensenShannon estimation for MI, while the mathematical intuitions and derivation processes are different.
Besides, it also belongs to the contrastive SSL venue.
That is, it aims at aligning the positive pairs and contrasting the negative pairs.
[Figure]
In this subsection, we will be focusing on the geometric data like molecular geometry.
Recall that we have two views: g 1 and g 2 , and the goal is to maximize the lower bound of the mutual information in Equation (14).
Because the two views share the same atomic features, it can be reduced to:
[Equation]
where the f ( · ) are the negative of energy functions, and A R 1 | g 2 and A R 2 | g 1 are the intractable partition functions.
The first equation in Equation (17) results from that the two views share the same atom types.
This equation can be treated as denoising the atom coordinates of one view from the geometry of the other view.
In the following, we will explore how to use the score matching for solving EBM, and further transform the coordinate-aware mutual information maximization to the denoising distance matching (GeoSSL) as the final objective.
Score Definition
The two terms in Equation (3) are in the mirroring direction.
Thus in what follows, we may as well adopt a proxy task that these two directions can calculated separately, and take one direction for illustration, e.g. , log exp( f ( R 1 , g 2 ))
A R 1 | g. The score is defined as the gradient of the log-likelihood w.r.t.
the data, i.e. , the atom coordinates in our case.
Because the normalization function is a constant w.r.t.
the data, it will disappear during the score calculation.
To adapt it into our setting, the score is obtained as the gradient of the negative energy function w.r.t.
the atom coordinates, as:
[Equation]
If we assume that the learned optimal energy function, i.e. , f ( · ) , possesses certain physical or chemical information, then the score in Equation (18) can be viewed as a special form of the pseudo-force.
This may require more domain-specific knowledge, and we leave this for future exploration.
Score Decomposition:
From Coordinates To Distances Through back-propagation [50], the score on atom coordinates can be further decomposed into the scores attached to pairwise distances:
[Equation]
19) where s ( d 1 , g 2 ) ij ≜ ∂f ( R 1 , g 2 ) ∂d 1 ,ij .
Such decomposition has a nice underlying intuition from the pseudoforce perspective: the pseudo-force on each atom can be further decomposed as the summation of pseudo-forces on the pairwise distances starting from this atom.
Note that here the pairwise atoms are connected in the 3D Euclidean space, not by the covalent-bonding.
Denoising Distance Matching (DDM)
Then we adopt the denoising score matching (DSM) [65] to our task.
To be more concrete, we take the Gaussian kernel as the perturbed noise distribution on each pairwise distance, i.e. , q σ ( ˜ d 1 | g 2 ) =
E p data ( d 1 | g 2 )
[ q σ ( ˜ d 1 | d 1 )] , where σ is the deviation in Gaussian perturbation.
One main advantage of using the Gaussian kernel is that the following gradient of conditional log-likelihood has a closed-form formulation: ∇ ˜ d 1 log q σ ( ˜ d
[Equation]
d 1 , g 2 ) =
( d 1 − ˜ d 1 ) /σ 2 , and the goal of DSM is to train a score network to match it.
This trick was first introduced in [65], and has been widely utilized in the score matching applications [54, 55].
To adapt this into our setting, this is essentially saying that we want to train a “distance network”, i.e. , s θ ( ˜ d 1 | g 2 ) , to match the distance perturbation, or we can say it aims at matching the pseudo-force with the pairwise distances from another aspect.
By taking the Fisher divergence as the discrepancy metric and the trick mentioned above, the estimation s θ ( ˜ d 1 , g 2 )
≈ ∇ ˜
d 1 log q ( ˜ d 1 | d 1 , g 2 ) can be simplified to the following:
[Equation]
Final objective.
We adopt the following four model training tricks from [35, 54, 55] to stabilize the score matching training process.
(1) We carry out the distance denoising at L -level of noises.
We add a weighting coefficient λ ( σ )
= σ β for each noise level, where β is the annealing factor.
We scale the score network by a factor of 1 /σ .
(4) We sample the exactly same atoms from the two geometry views with masking ratio r .
Finally, by considering the two directions and all the above tricks, the objective function becomes the follows:
[Equation]
[Table]
(cid:13)(cid:13)
[Table]
C. D ISCUSSIONS
Using the energy-based model (EBM) to solve MI maximization can open a novel venue, especially for high-structured data like molecular geometry.
To solve EBM, existing methods include noise-contrastive estimation (NCE)
[24], score matching (SM) [57], etc.
To put this under the MI maximization setting, EBM-NCE is essentially a contrastive learning method, where the goal is to align the positive pairs and contrast the negative pairs simultaneously.
While EBM-SM or GeoSSL is a generative self-supervised learning (SSL) on distance denoising, and it is especially appealing in the field for geometric data representation learning.
Score matching can be smoothly adopted to 3D geometric setting.
Because scores are defined as gradients of the energy function with respect to the atom positions, it can be thought of a form of pseudo-forces.
Following this, GeoSSL can be viewed as a pseudo-force matching, which is more natural to the molecular structures.
However, further understanding of this requires more domain knowledge in understanding or designing of the energy function.
This is beyond the score of this paper, and we would like to leave it for future exploration.
Recently, there have been a certain works [35] proving that 3D geometric information is useful for 2D topology.
Here we want to conjecture that the reverse direction is also meaningful: 2D topology can be also useful for 3D representation.
This may not seem reasonable from the domain perspective, since 2D topology can be heuristically obtained from the 3D geometry, i.e. , all the 2D information is redundant to 3D geometry.
However, from the machine learning theory perspective [4, 17], this is still helpful in reducing the sample complexity.
From a higher level perspective, we want to explicitly point out that such gap between machine learning and scientific domain has been widely existed, and it would be an interesting direction for further exploration.
D E XPERIMENTS
In this section, we would like to discuss the experiment details of our work.
The main structure is as follows:
• In Appendix D.1, we introduce the computation resources.
• In Appendices D.2 to D.4, we introduce the downstream datasets.
– Notice that because the performance of QM9 and MD17 is quite stable after fixing the seed ( e.g. , 42), we we will not run cross-validation.
This also follows the main literature [37, 46, 47].
– Yet, for LBA & LEP, these two datasets are quite small and are very sensitive to the data splitting, so we pick up 5 seeds (12, 22, 32, 42, and 52) and run cross validation on them.
• In Appendix D.5, we list the key hyperparameters for all the pretraining baselines and GeoSSL.
• In Appendix D.6, we show the empirical results using SchNet as the backbone model.
D. C OMPUTATIONAL R ESOURCES
We have around 20 V100 GPU cards for computation at an internal cluster.
Each job can be finished within 3-24 hours (each job takes one single GPU card).
D. D ATASET : QM
QM9 [42] is a dataset of 134K molecules consisting of 9 heavy atoms.
It includes 12 tasks that are related to the quantum properties.
For example, U0 and U298 are the internal energies at 0K at 0K and 298.15K respectively, and U298 and G298 are the other two energies that can be transferred from H298 respectively.
The other 8 tasks are quantum mechanics related to the DFT process.
We follow [46] in preprocessing the dataset (including unit transformation for each task).
Current work is using different data split (in terms of the splitting size).
Originally there are 133, molecules in QM9, where 3,054 are filtered out, leading to 130,831 molecules.
During the benchmark, we find that the performance on QM9 is very robust to either using (1) 110K for training, 10K for val, 10,831 for test or using (2) 100K for training, 13,083 for val and 17,748 for test.
In this paper, we are using option (1).
D. D ATASET : MD
MD17 [9] is a dataset on molecular dynamics simulation.
It includes eight tasks, corresponding to eight organic molecules, and each task includes the molecule positions along the potential energy surface (PES), as shown in Figure 1.
The goal is to predict the energy-conserving interatomic forces for each atom at each molecule position.
We list some basic statistics in Table 6.
We follow [37, 47] in preprocessing the dataset (including unit transformation for each task).
[Table]
D. D ATASET : LBA & LEP
Atom3D [62] is a newly published dataset.
It gathers several core tasks for 3D molecules, including binding affinity.
The binding affinity prediction is to measure the strength of binding interaction between a small molecule to the target protein.
Here we will model both the small molecule and large molecule (protein) with their 3D atom coordinates provided.
During the binding process, there is a cavity in a protein that can potentially possess suitable properties for binding a small molecule (ligand), and it is termed a pocking [58].
Because of the large volume
[Table]
of protein, we follow [62] by only taking the binding pocket, where there are no more than 600 atoms for each molecule and protein pair.
To be more concrete, we consider two binding affinity tasks.
The first task is ligand binding affinity (LBA).
It is gathered from [66] and the task is to predict the binding affinity strength between a small molecule and a protein pocket.
The second task is ligand efficacy prediction (LEP).
We have a molecule bounded to pockets, and the goal is to detect if the same molecule has a higher binding affinity with one pocket comparing to the other one.
We list some basic statistics in Table
[Table]
D. H YPERPARAMETER S PECIFICATION
We list all the detailed hyperparameters in this subsection.
For all the methods, we use the same optimization strategy, i
[Table]
e. , with learning rate as 5e-4 and cosine annealing learning rate schedule [39].
The other hyperparameters for each pretraining method are listed in Table 8.
For the other hyperparameters, we are using the default hyperparameters, as attached in the codes.
[Table]
D. S CH N ET AS B ACKBONE M ODEL
We want to highlight that some backbone models ( e.g. , DimeNet++ and SphereNet) may perform better or on par with the PaiNN, as shown in Table 4.
Yet they will be out of GPU memory.
Thus, considering all (including the model performance, computation efficiency, and memory cost) together, we adopt PaiNN as the backbone model in the main paper.
In this section, we carry out experiments using SchNet as the backbone model.
We follow the same process as in Section 5, i.e. , we compare our method with one randomly-initialized and seven pretraining baselines.
The results on QM9, MD17, LBA and LEP are in Tables 9 to 11 accordingly.
From these three tables, we can observe that in general, GeoSSL can reach the most optimal results, yielding 21 best performance in 22 downstream tasks, and can reach comparative performance on the remaining task (within top 2 model).
This can largely support the effectiveness of our proposed method, GeoSSL.
In addition, we also want to mention that a lot of pretraining tasks show the negative transfer issue.
Comparing to the results in Section 5, we conjecture that this is related to the task (both pretraining and downstream tasks) and the backbone model.
Yet, this is beyond the scope of our work, and we would like to leave this as a future direction.
[Table] [Table]
[Table]
E A BLATION S TUDIES E. T HE E FFECT OF A NNEALING F ACTOR IN G EO SSL
Among all the hyperparameters (see Table 8) for GeoSSL
[Table]
we find that annealing factor is one of the most sensitive ones.
Annealing factor β is applied on the weighting coefficient λ ( σ ) = σ β .
In this section
[Table]
we carry out an ablation study to verify this by pretraining GeoSSL with annealing factors at five different scales.
[Table]
As can be observed in Table
[Table]
the models are more stable with smaller annealing values ( e.g. , 0. and 0.05).
With large annealing values, the model performance can degrade drastically.
E. T HE E FFECT ON THE N UMBER OF N OISE L AYERS IN G EO SSL
Another important hyperparameter listed in Table 8 is the number of noise layers, L .
Here we conduct an ablation study on it, and the results are shown in Table 13.
[Table]
In Table
[Table]
we can observe that in general, GeoSSL can attain better performance with more denoising layers.
This is in fact consistent with that in vision applications [57].
Promisingly, even with smaller L ( e.g. , L = 1 ), GeoSSL can still achieve a modest improvement to some extent.
F S TRONG M ODEL R OBUSTNESS WITH R ANDOM S EEDS
To further illustrate that our proposed GeoSSL is robust and insensitive to certain random seeds
[Table]
we further provide the downstream results with more random seeds.
We list the key details as follows:
• Dataset.
We conduct downstream experiments with random seeds on two datasets: QM and MD17.
• Backbone models.
We run two backbone models: PaiNN in Appendix F and SchNet in Appendix F. • Seeds.
Up till now, for both the main tables (Tables 1, 2, 9 and 10) and ablation studies (in Appendix E), we use a fixed seed 42.
In this section, we provide results with two additional seeds 22 and 32.
• Baselines.
We here compare against the most optimal baselines: random initialization (without any pretraining), distance prediction, representation reconstruction (RR)
[Table]
and EBM-NCE.
• Reported results.
We report both the mean and standard deviation with seeds 22, 32, and 42 for all the experiments.
F. P AI NN
Here we take the PaiNN as the backbone model.
The results on QM9 and MD17 are reported in Tables 14 and 15 respectively.
Such empirical results match with the main result in Tables 1 and
[Table]
and they do verify that our proposed GeoSSL is indeed learning a more robust representation.
[Table]
[Figure]
[Table]
[Table]
F. S CH N ET
Here we take the SchNet as the backbone model.
The results on QM9 and MD17 are reported in Tables 16 and 17 respectively. Such empirical results match with the main result in Tables 9 and 10, and they do verify that our proposed GeoSSL is indeed learning a more robust representation. [Table] [Figure]
[Table]
G C OMPARISON WITH A P ARALLE W ORK
We note that there is a parallel work introduced in [72], which also explores the effect of denoising for geometric data pretraining.
That work is very different from GeoSSL and we here summarize the main differences as follows:
• The parallel work as presented in [72] is similar to that of denoising score matching (DSM) introduced in [65], i.e. , with only one layer of denoising in score matching.
On the contrary, our model has multiple denoising layers, which is much closer to the NCSN [54], where the number of noise layers has been proven to be important to the effectiveness of the denoising score matching models.
We here also empirically verify the above analysis.
That is, we present the experimental results in Table 13, where L = 1 is equivalent to the method in [72].
We can observe that with layer number L = 1 (namely the third row of the table), the performance does increase in some cases, which matches with the observation in [72].
Nevertheless, the results in Table 13 clearly indicate that with larger L , the model can attain further error reduction and improve model robustness.
• Theoretically, the work in [72] specifically aims at the application task of representation learning in geometric pretraining, through a straightforward adaption of denoising score matching from vision.
In contrast, our GeoSSL approach indeed provides a very general framework that leverages energy based model (EBM) for mutual information (MI) maximization for geometric data pretraining.
As such, GeoSSL can be easily replaced by other EBM models such as the GFlowNet network [3] to better capture the multi-mode distributions in geometric data during pretraining (please see Section 6 for more discussion).