Regularized Molecular Conformation Fields
Anonymous Author(s)
AffiliationAddress email
Abstract
Predicting energetically favorable 3-dimensional conformations of organicmolecules from molecular graph plays a fundamental role in computer-aideddrug discovery research.
However, effectively exploring the high-dimensionalconformation space to identify (meta) stable conformers is anything but trivial.
Inthis work, we introduce R MCF , a novel framework to generate a diverse set of lowenergy molecular conformations through sampling from a regularized molecularconformation field.
We develop a data-driven molecular segmentation algorithmto automatically partition each molecule into several structural building blocks toreduce the modeling degrees of freedom.
Then, we employ a Markov Random
Field to learn the joint probability distribution of fragment configurations and interfragment dihedral angles, which enables us to sample from different low-energyregions of a conformation space.
Our model constantly outperforms state-of-the-artmodels for the conformation generation task on the GEOM-Drugs dataset.
We attribute the success of R MCF to modeling in a regularized feature space and learninga global fragment configuration distribution for effective sampling.
The proposedmethod could be generalized to deal with larger biomolecular systems.
Introduction
The spatial arrangement of atoms within a molecule, also known as molecular conformation, determines the molecular physico-chemical properties, which plays an essential role in downstreamcomputer-aided drug discovery tasks.
However, the high-dimensional conformation space spannedby all atomic degrees of freedom (DoF) makes it a great challenge to identify the local minima ofthe associated potential energy surface (PES) of a molecule.
Recently, many machine learning (ML)models have achieved remarkable success in molecular conformation generation tasks [Mansimovet al., 2019, Simm and Hernandez-Lobato, 2020, Xu et al., 2021a, Ganea et al., 2021, Xu et al.
, 2022],exhibiting orders of magnitude faster conformation prediction speed than traditional computationalsimulation approaches.
This makes ML generative models a powerful tool for high-throughputscreening in drug discovery, especially large molecules.
To fully exploit the power of ML in molecular conformation generation, we need to tackle a few keychallenges.
First, a molecule exhibits invariance under SE(3) transformations (i.e., translation androtation) in 3-dimensional (3D) Euclidean space.
In other words, each molecular conformation isuniquely defined up to rigid motion.
Hence a single molecule can adopt an infinite set of possibleposes.
Second, molecules exhibit a variety of dynamics under ambient conditions (e.g., bond rotation,vibration, etc.), leading to a possibly complex PES landscape of high dimension.
This makesit particularly challenging for ML models to identify local minima within this space to generateenergetically favorable conformations effectively.
Some recently developed ML models tried to solve these problems from different perspectives.
For instance, GraphDG[Simm and Hernandez-Lobato, 2020] and CGCF[Xu et al., 2021a] useddistance geometry as invariant features for molecular representation learning.
The drawbacks of such representations are that these invariant variables are potentially redundant and exhibit mutualdependency, which may cause numerical instability during training.
One can also design equivariantnetworks to hard-code the invariance/equivariance condition into the model[Satorras et al., 2021, Xuet al., 2022].
These models circumvent using intermediate invariant features, but instead directly learnfrom the spatial arrangement of atoms.
However, some studies[Cohen et al., 2018, Li et al., 2021]have suggested that the specialized equivariant layers may cause some loss in the expressivenessof the neural network.
On the other hand, GeoMol[Ganea et al., 2021] predicts a local structurefor each atom, followed by assembling these atom-based building blocks to form the molecularconformation.
These models have difficulty dealing with cyclic graphs (e.g., benzyl group) and maylead to unreasonable conformation predictions.
Dihedral angle
P o t en t i a l ene r g y
HH
HH HH
HH
HH HH
HH
HH HH
In the meantime, we realize that the high-dimensional
PES could be described in a reduced basis by considering only a few significant DoF that contribute to thelow-energy conformations.
Specifically, consider thewell-known example of the potential energy diagramof ethane molecule (Figure 1), while the moleculeexhibits a total of 18 DoF, we could capture the mostsignificant dynamics using a single variable, i.e., therotation w.r.t.
the carbon-carbon bond, quantified bythe H-C-C-H dihedral angle.
Other DoF in the ethanemolecule, e.g., the stretching and bending of otherbonds, do not have significant impact on the potentialenergy landscape, where they can be treated as smallperturbations to the molecular conformation withina potential well.
In other words, to serve the purposeof effective sampling local minima for low-energyconformations, we could reduce the dimension of theconformation space by tracking only a few significant DoF, as long as they adequately shape the corresponding energy landscape (e.g., the “ ω ” shape in Figure 1).
It then follows naturally to generate new conformersby sampling from the low-dimensional conformation space, which is physically meaningful.
This paper presents Regularized Molecular Conformation Fields (R MCF ), a novel framework for 3Dmolecular conformation generation.
The novelty of this work lies in (1) developing a data-drivenmolecular segmentation algorithm to partition each molecule into fragments with low internal flexibility, which serves as a regularization term for our framework, and (2) employing a Markov Random
Field (MRF[Murphy, 2012]) to capture the joint probability distribution of intra-fragment configurations and inter-fragment dihedral angles.
Our work is partly inspired by the Ising model[Cipra,1987] for simulating quantum spin systems, where we make an analogy between the spin state (i.e.,spin up or down) and the configuration of each molecular fragment (e.g., chair or boat conformationof cyclohexane).
The introduction of molecular fragments effectively reduces the dimension of theconformation space, where we only keep the most significant components (i.e., fragment configuration and inter-fragment dihedral angle) for conformation generation.
This molecular segmentationserves as a regularization term for our framework, which is data-driven and has the effect of reducingfeature dimensions.
Then, we employ MRF as a generalized setting of the Ising model (with morespin states and more coupling terms) to capture the relationship between adjacent fragments andmodel the uncertainty of conformations.
By estimating the parameters of the MRF, we can obtain anenergy surface with different conformations.
Therefore, we can later sample from the low-energyregion of PES, which is implicitly learned by the MRF, to obtain energetically favorable molecularconformations.
We demonstrate the effectiveness of R MCF using the GEOM-Drugs dataset[Axelrod and GomezBombarelli, 2022], where results show that R MCF significantly outperforms state-of-the-art MLmodels.
Specifically, our model can generate a diverse set of low-energy conformations located atdistinct local minima of the underlying PES.
We attribute the success of our model to the automaticconstruction of low-dimensional conformation space and learning a joint probability distribution using MRF to achieve effective sampling of new conformations.
We argue that capturing the governing dynamics of molecular systems could significantly improve the performance of learning models forgenerative purposes.
Related Work
Recently, various machine learning models have been proposed for molecular conformation generation.
CVGAE [Mansimov et al., 2019] first used a variational autoencoder (VAE) model to generateconformation with atomic coordinates.
The model suffers from multi-modality problems due toinvariance under SE(3) transformations.
Some studies tried to address this problem using two maintypes of approaches.
The first type of approach use intermediate invariant features, such as atomicdistance, to encode the system, then leverage geometric algorithms to solve a set of atomic coordinatesthat match the invariant quantities.
For example, GraphDG[Simm and Hernandez-Lobato, 2020] and CGCF [Xu et al., 2021a] proposed to predict the distance matrix by VAE and Flow, respectively, andsolve the geometry through the Distance Geometry (DG) method [Liberti et al., 2014].
However,these invariant features are potentially redundant and could exhibit mutual dependency, which makesthese methods numerically unstable and could predict unreasonable conformations.
Some studiessuggest that inconsistencies between training and test are responsible for the poor performance ofthe models.
ConfGF[Shi et al., 2021] passed the gradient of loss to coordinates and ConfVAE [Xuet al., 2021b] used a bilevel optimization to alleviate the inconsistencies.
GeoMol[Ganea et al. , 2021]defined another set of invariants, which are bond lengths, bond angles, and dihedral angles.
Geomolgenerated multiple conformations by adding noise to the input, which did not learn a global energyfunction from which to sample the low-energy conformations.
Another type of approach applies equivariant networks or kernels, such that after the input is rotatedand translated, the output can be transformed accordingly.
Satorras et al.
[2021] proposed anequivariant normalizing flow, E(n)-flow, and Xu et al.
[2022] proposed GeoDiff, which is a diffusionmodel [Song and Ermon, 2019] with equivariant Markov kernels.
Equivariant methods circumventintermediate invariants and model the coordinates directly, but require the design of specializedequivariant layers, which some studies have suggested would lose the expressive power of thenetwork.
In addtion, some software for conformation generation are widely used in biochemical research.
For instance, RDKit[Riniker and Landrum, 2015] is a popular open-source software generate use ETKDG distance geometry, and OMEGA is a commercial software which assembles the fragmentswith knowledge-based rules to generate conformations.
R MCF : Generating in Regularized Conformation Space
In this section, we present the regularized molecular conformation field R MCF in detail.
Generally,
R MCF learns the energy landscape of molecular conformations by modeling a joint distribution offragment states and dihedral angles, from which we can perform inference or draw diverse samples. We will first give a brief overview of R MCF in Section 3.1. Then we construct and parameterizethe R MCF in Section 3.2 and Section 3.3.
Later in Section 3.4 and Section 3.5, we describe how toestimate the parameters and draw samples from the R MCF . Finally, we assemble the fragments into amolecule in Section 3.6.
3.1 Formalization
We formally describe the pipeline of our method.
Given a molecular graph G , we segment it into fragments and construct a regularized molecular conformation field G . Then, we infer a configuration set X including the fragment state and the dihedral state from the learned joint probability distribution using MRF. Finally, we assemble the 3D fragments with the predicted inter-fragment dihedral angles to generate the conformation R .
[Equation]
A regularized conformation field G = ( F, D, E ) is an undirected graph formed by a collection offragment vertices F
= ( f 1 , f 2 , · · · , f N F ) , a collection of dihedral vertices D
=
( d 1 , d 2 , · · · , d N D ) , [Figure] RMCF
Assembling
Conformation and a collection of edges E = ( e 1 , e 2 , · · · , e N E ) ⊂
F × D , where N F , N D , N E are the size of F , D , E , respectively [Wainwright et al., 2008].
Each edge consists of a pair of vertices f ∈ F andd ∈ D .
We associate with each vertex f ∈ F a random variable X f taking values in a set of possiblefragment variants, each vertex d ∈ D a random variable X d taking values in R .
Such a graph is acollection of distributions that factorize as: [Equation] where Z is the normalizing factor, C is the maximal cliques of a graph.
A configuration X =
( X f , X d ) of the R MCF contains a state set of fragment nodes
X
f = { x f i | i =1 , 2 , · · · , N F } and a state set of dihedral nodes X d = { x d i | i
= 1 , 2 , · · · , N d } .
3.2 Graph Construction with Least DoF Principle
To reduce the DoF of the conformation space, we determine which atoms and bonds on the moleculargraph to pack into fragments.
As a general rule of thumb, we should minimize the intra -fragment
DoF while capturing the significant dynamic modes using inter -fragment dihedral angles. To that end,we design a data-driven fragment segmentation algorithm.
Let { f 1 , f 2 , ..., f n } be a graph partitionof a molecular graph G , f i is one of the corresponding sub-graphs, also called molecular fragments.
First, we use existing functional group segmentation methods such as BRICS [Liu et al., 2017] topre-cut all molecular graphs and their 3d conformations.
We keep all rings but remove all the sidechains on them.
After obtaining these molecular fragments, a sufficient number of 3D fragments c are sampled foreach 2D fragment and clustered using root-mean-square-deviation (RMSD), which measures theaverage distance between atoms.
The set of cluster centroids are used as a vocabulary V ( f ) torepresent various 3D states of the 2D fragment f .
We defined the measurement of the internal DoF asthe maximum RMSD between all the 3D fragment pairs in the vocabulary V ( f ) .
The best segmentation P ∗ should exhibit the least sum of the internal DoF on the resulting fragmentset. We employ graph dynamic programming to search for the optimal solution.
[Equation]
Then we can get the fragment set F = ( f 1 , f 2 , · · · , f N F ) and insert the dihedral vertices betweenconnected fragments to construct the R MCF G on the molecular graph G .
3.3 Parameterization with GNN
Given the R MCF G constructed from a molecular graph G , we parameterize it with graph neuralnetworks.
Let e f and e d denote the fragment embedding and the dihedral embedding (which is a“dummy” embedding as the network input), we compute the contextualized representations r by:
[Equation]
We can also obtain an oracle configuration X =
( S, Φ) of the R MCF G based on the conformation
R , which is segmented to ( r 1 , r 2 , · · · , r N F ) .
For a given fragment f , we pick the state r s as groundtruth which has the minimal RMSD to the real fragment r i .
[Equation]
[Figure]
The dihedral angle φ is defined as the angle between half-planes of twoconnected 3d fragments.
Since the dihedral angle depends on the set ofthe atoms involved in the calculation, for the same type of fragments, weshould keep the same set of atoms to calculate.
For a system where two fragments contain four consecutively-bondedatoms, two half-planes intersect on a rotatable bond.
The angle betweenthem is the dihedral angle .
If the connected points are sequentiallynumbered and located at positions p 1 , p 2 , p 3 , p 4 and the correspondingbond vectors are defined as u 1 =
p 2 − p 1 , u 2 = p 3 − p 2 , u 3 = p 4 − p 3 .
Then we have:
[Equation]
In practice, we quantize the angles into a number of evenly divided bins to discretize the continuousangle values. 3.4 Parameter Estimation
[Figure]
The parameter estimation in R MCF is intractable, especially when it is loopy [Murphy, 2012].
Piecewise learning has proven efficient in training graphical models [Sutton and McCallum, 2009, 2012,
Lin et al., 2016, Qu et al., 2022], which minimizes a tractable upper bound on the exact log partitionfunction [Sutton and McCallum, 2012].
We train the parameters of each edge independently, and thelearned parameters from this local training approach can be used for global inference purposes.
Concretely, for a pair of nodes in the field e = ( f, d ) ∈ G , let | f | and | g | denote the cardinalityof the state space, y f ∈ I and y d ∈
I indicate the true state from an index set.
We calculate therepresentations of them, r f ∈ R D and r d ∈ R D from the graph neural networks.
The node-wisenegative log-likelihood is defined as:
[Equation]
where out( · ) : R D → R | f | is an operator parameterized by a linear layer.
The edge-wise negativelog-likelihood is defined as:
[Equation]
where E f ∈ R | f |× D and E d ∈ R | d |× D are the representations of a set of node states,
W e ∈ R D
× D is a matrix carrying the global information between two nodes.
In this study, we use a neural network to parameterize the matrix [Sun et al., 2019]:
[Equation]
To sum up, the complete training objective is:
[Equation]
3.5 Inference & Sampling
In this section, we give a brief overview of how we perform inference and sampling in R MCF .
Inference If a R MCF is acyclic (linear- or tree-structured), the max-a-posterior decoding can bedone via dynamic programming [Forney, 1973].
Cases are a bit complicated when loops occur in the
R MCF .
We adopt a more generalized algorithm, namely loopy belief propagation (LBP)
[Murphyet al., 2013], to overcome such obstacles.
We refer the reader to textbooks for more details about the
LBP algorithm
[Murphy, 2012].
Sampling Markov Chain Monte Carlo is popular in sampling from graphical models [Andrieu et al.
We employs an simple variant of it, Gibbs sampling [George and McCulloch, 1993], to drawsamples from R MCF .
Specifically, we resample each fragment or dihedral individually, keeping allother nodes fixed in each iteration [Murphy, 2012].
Clustering Upon the Markov chain achieves detailed balance, we can draw 1 , 000 samples randomlyfrom it.
However, these samples can be governed by a few conformations, which means most of themare similar.
We measure the distance between two configurations X (1) and X (2) by:
[Equation]
8) where ∥ · ∥ 2 is the Euclidean norm measuring the difference between dihedrals, H is the Hammingdistance [Hamming, 1950] counting the number of different fragment states.
We further run K-meansclustering [MacQueen et al., 1967] based on the pair-wise distance to partition all samples into afixed number of clusters. Require: an R MCF G , the number of conformations N C , the number of sampling iterations N S
Ensure: a set S C containing N C conformations S ← {} initialize a configuration X randomly for i ∈ [1 , 2 , · · · , N S ] do for n ∈ F ∪ D do U ← { u | ( n, u ) ∈ E ▷ x u is fixed in this iteration
P ← softmax (cid:8)(cid:80) u ∈ U E u [ x u ] W un E T n + out( r n ) (cid:9) ▷ the value can be pre-computed x n ∼ Cat( | n | , P ) end for add X to S end for compute the pairwise distance K with Eq. S C ← run clustering with k = N C over S based on K return S C 3.6 Assembling
𝑛 θ θ 𝑢 !
𝑢 " 𝑣 ! 𝑣 " 𝑣 !
# 𝑣 "# 𝜑′ 𝑅 𝑅′ 𝑡
Once we have predicted the dihedral angle φ between twofragments, we need to compute the transformation matrix Tto align the two fragments in arbitrary position and make theirdihedral angle equal to the given value φ and repeat these stepsuntil the whole conformation is assembled.
Given two fragments with six points, located at p 1 , p 2 , p 3 , p 4 ,p 5 , p 6 , the bond vectors are u 1 = p 2 − p 1 , u 2 = p 3 − p 2 , v 1 =p 5 − p 4 , v 2 = p 6 − p 5 .
u 2 and v 1 are the same bond afterassembling, so we have | u 2 | = | v 1 | .
First we need to rotate the second fragment around some axisso that u 2 and v 1 will be parallel.
this axis is the unit normalvector n towards the plane of u 2 and v 1 , and the rotate angle isequal to the cross angle θ .
[Equation]
By using the Rodrigues’ rotation formula, we can obtain the rotation matrix with unit normal vectorn and rotation angle θ .
Next, we rotate the second fragment around u 2 to match with the targetdihedral angle.
Finally we align the anchor points by calculating the final transformation matrix T asa composite matrix of two rotations and one translation.
[Equation]
Experiment
We now demonstrate the effectiveness of R MCF on the conformation generation task for drug-likemolecules.
4.1 Dataset and Baseline
Following previous work on conformation generation, we benchmark our model performance usingthe GEOM-Drugs dataset, which contains mid-sized organic molecules with high quality conformations.
We use the same test set as that in GeoDiff, where the remaining molecules are used for trainingand validation with a 9:1 ratio.
The final training/validation/test set contains 271,539/30,171/1,034molecules, respectively.
Our molecular segmentation algorithm eventually returned a vocabulary of 9,081 types 2D fragmentsand 30,408 types of 3D fragments for the GEOM-Drugs dataset.
As for the angular discretization ofdihedral angles, the 360 degree interval is evenly divided into 72 bins.
We adopt the Message-Passing Neural Network (MPNN)
[Gilmer et al., 2017] as the framework for implementing our graph neuralnetwork.
Other implementation details are provided in Appendix A.
We compare our R MCF model with seven recently developed ML models.
The performance of CVGAE[Mansimov et al., 2019], GraphDG[Simm and Hernandez-Lobato, 2020], CGCF[Xu et al. ,2021a], and ConfVAE[Xu et al., 2021b], ConfGF[Shi et al., 2021] are tabulated as reported in Shiet al.
[2021] and Xu et al.
[2021b], and we show the metrics of GeoMol[Ganea et al., 2021] and
GeoDiff[Xu et al., 2022] as reported in Xu et al.
We also compare our model with open-sourcesoftware RDKit and commercial software OMEGA (as reported in Ganea et al.
4.2 Evaluation Metrics
To quantitatively measure the quality and diversity of generated molecular conformations, we use the same evaluation metrics as defined in Ganea et al.
[2021] and Xu et al.
Let S g and S r denote the set of generated and reference conformations, respectively.
The coverage score ( COV ) and matching score ( MAT ) following the conventional Recall measurement can be defined as:
[Equation]
where the threshold δ in coverage score is set as 1 . 25 Å for the GEOM-Drugs dataset in our work.
The corresponding prediction precision metrics, i.e., COV-P and MAT-P , are defined in a similar [Table] manner, where the set of generated and reference conformations are swapped in the above definition.
We set S g as twice the size of S r for each molecule for fair comparison with previous work [Ganeaet al., 2021, Xu et al., 2022].
The precision-associated metrics focus more on generating accurateconformations that match those in the reference dataset, while the recall-associated metrics emphasizethe structural diversity of generated conformations.
4.3 Results and Discussions
The main results are presented in Table 1.
We generated conformations by sampling on the R MCFwith two strategies: (1) directly sample a specific number of conformations (i.e., S g ), and (2) firstgenerate a sufficient number of conformations (we generate 10,000 conformations in this work), thencluster them into S g clusters and sample one conformation from each cluster.
The model performanceassociated with the above two strategies are named RMCF-R and RMCF-C, respectively.
As shownin the table, R MCF significantly outperforms all other models in all metrics except for the COV-Rmetric, where our model performance is comparable to that of GeoDiff.
In particular, for COV-Pand MAT-R metrics, our model shows a significant performance gain compared to others, whichmeans R MCF can generate more accurate and high quality conformations.
In addition, we find thatthe clustering-after-sampling strategy leads to a boost in the structural diversity, while keeping a goodquality (i.e., energetically favorable conformations) of the generated conformations.
Since we employ a discretized treatment for the continuous dihedral angle as well as the fragment 3Dconformation, the model performance will inevitably have some discrepancy between the predictedand ground truth conformations.
To that end, we investigate the upper and lower bound of our modelperformance to have a better understanding of its predicting power, as shown in Table 2.
Specifically,we discretize the ground truth conformations using the molecular segmentation algorithm to obtain thefragment configurations (“gold X f ”) and inter-fragment dihedral angles (“gold X d ”), then evaluate thecorresponding metrics.
On the other hand, we randomly sample some X f and X d , whose evaluationresults should indicate the performance lower bound using our fragment representation.
Surprisingly,we find that using randomly sampled X f and X d , we can still outperform some previous models.
Thisis an advantage of using molecular fragment as building blocks for conformation generation, since webypass the need to generate many insignificant variables which may degrade the model performance.
We also learn that having accurate dihedral angle predictions (i.e., X d ) is more significant thangood fragment configuration predictions (i.e., X f ).
These experimental results support our claimthat capturing a few significant DoF within a molecule is adequate for generating a diverse set oflow-energy conformations, while other DoF could be safely ignored during modeling.
At last, we showcase the generated conformations of two example molecules in Figure 5.
Foreach molecule, we take the first three predicted conformations, and align the non-rigid parts forvisualization purposes.
From Figure 5(a), we observe a diverse set of conformations mainly drivenby the rotation of two single bonds without causing too much steric effect.
For the amide bondregions, the model correctly predicts the corresponding dihedral angles to form planar conjugate [Table] [Table]
[Table] [Figure] systems. Meanwhile, for Figure 5(b) we see a large planar conjugate system on the left side and acyclooctane ring on the right.
Again, the model accurately captures the conjugate system and onlymakes variations in the cyclooctane conformations.
Interestingly, the segmentation algorithm tendsto only preserve the cyclic groups (e.g., benzyl and furan groups), and even cuts through the amidebonds and other acyclic conjugate systems.
This behavior is a direct consequence of balancing thevocabulary size and the variety of local chemical environment.
We show that our model performs reasonable segmentation of drug-like molecules into small functional groups with limited internal DoF, and could correctly predict planar conjugate systems.
Therefore, we believe that our generative process is essentially sampling from the local-minima of thelearned potential energy surface by R MCF , where those non-essential DoF which contribute to localstructural perturbations are omitted in our modeling framework.
We argue that both the data-drivensegmentation and MRF modeling of the joint probability distribution are essential for the success ofour model.
Conclusion
We introduce R MCF , a novel framework for 3D molecular conformation generation.
Our modelis physics-motivated, with the central idea to effectively model the joint probability distribution ofgoverning dynamical modes in a reduced conformation space to achieve energetically favorableconformation generation.
Experimental results show that R MCF outperforms state-of-the-art modelson the GEOM-Drugs dataset to predict a diverse set of conformations located at distinct local minimaof the corresponding molecular potential energy surface.
Our methodology can be naturally extendedto larger biomolecular systems, e.g., proteins, whose conformation prediction is a significant topic inthe biological research community.
We will address this challenge in our future work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[No](c) Did you discuss any potential negative societal impacts of your work?
[No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[No] without
theoretical results
(b) Did you include complete proofs of all theoretical results?
[No] without theoretical
results
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Appendix and Section 4(c)
Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[No] The results are stable
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[No] No existing assets
(b) Did you mention the license of the assets?
[No] No assets
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
No assets
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[No] Open-source data
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No] The data without personally identifiable
information or offensive content
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[No](b) Did you describe any potential participant risks, with links to Institutional ReviewBoard (IRB) approvals, if applicable?
[No]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[No]