Paper: Learning Over Molecular Conformer Ensembles: Datasets and Benchmarks (ICLR-2024)

Reviewer comment (Experiments):
It is recommended to add discussions about [2] as [2] proposes a molecular conformer ensemble learning module named ConfDSS. Also, it is recommended to add it as a baseline if it can be applied to the task in MARCEL.

Author response:
Thank you for bringing this paper to our attention! ConfDSS integrates 2D and 3D GNN models for learning from conformer ensembles. This approach is indeed more complex compared to our current ensemble learning models (Mean, DeepSets, and Attention). Acknowledging the significance of ConfDSS in the context of our work, we will include a detailed discussion about it in the final version of our paper.

Reviewer comment (Experiments):
In Table 2, which molecular conformers are used as inputs to 3D graph neural network models?

Author response:
For both training and inference on Drug-75K, Kraken, and EE datasets, all the single-conformer 3D models operate on the **lowest-energy conformer of each conformer ensemble**, which has the largest Boltzmann weight. Since imprecise conformers from Open Babel are encoded for the BDE task, we use a fixed, randomly sampled conformer for each unbound- and bound-catalyst during training and inference. It is worth noting that using the lowest-energy conformer during the test stage is actually somewhat unrealistic: without extensive geometric optimization across the entire conformer space, it is often not possible to precisely determine the lowest energy conformer, particularly for large flexible molecules relevant to the drug discovery and computational chemistry community.

Reviewer comment (Novelty):
Since all four datasets prepared are secondary data from referenced primary data, it's unclear how challenging it would be for researchers to prepare them on their own. While the paper mentions quality control and the removal of redundancies, it's unclear whether there is any original information added to this study.

Author response:
A: Thank you for your constructive reviews. First of all, we would like to clarify that historically it is not yet shown that conformer ensembles could improve performance on real-world molecular tasks. Therefore, our benchmark should consist of well-defined, simulated tasks without additional confounding variables to ensure a fair comparison with existing baseline models, and to assist in the development of new models that can more effectively make use of extra structural information contained in conformer ensembles.

Secondly, regarding the added information to the datasets, the Drugs-75K dataset requires significant computational effort. The original GEOM-Drugs dataset was constructed using semi-empirical Density Functional Theory (DFT) methods, which is less accurate than full DFT. To curate the Drugs-75K subset, we generate the conformer ensembles with Auto3D and compute their corresponding energies using AIMNET-NSE. This process took approximately 600 CPU hours on an AMD EPYC 7763 server.

Last, the datasets compiled in MARCEL have been reorganized and standardized to form a unified benchmark, requiring substantial effort as well. We have developed a Python interface for easy access of these datasets into PyTorch, facilitating their use and extension by the research community.

Reviewer comment (Experiments):
While Drugs-75K, a subset of the GEOM-Drugs [27], Kraken [33], EE [34], BDE [36] all have associated citations, indicating they are secondary data, it was unclear whether the presented datasets are a simple curated version of these primary data, or if any original information was generated in this study. If there is any original information, it would be very helpful to be clarified.

Author response:
A: Thank you for your constructive reviews. First of all, we would like to clarify that historically it is not yet shown that conformer ensembles could improve performance on real-world molecular tasks. Therefore, our benchmark should consist of well-defined, simulated tasks without additional confounding variables to ensure a fair comparison with existing baseline models, and to assist in the development of new models that can more effectively make use of extra structural information contained in conformer ensembles.

Secondly, regarding the added information to the datasets, the Drugs-75K dataset requires significant computational effort. The original GEOM-Drugs dataset was constructed using semi-empirical Density Functional Theory (DFT) methods, which is less accurate than full DFT. To curate the Drugs-75K subset, we generate the conformer ensembles with Auto3D and compute their corresponding energies using AIMNET-NSE. This process took approximately 600 CPU hours on an AMD EPYC 7763 server.

Last, the datasets compiled in MARCEL have been reorganized and standardized to form a unified benchmark, requiring substantial effort as well. We have developed a Python interface for easy access of these datasets into PyTorch, facilitating their use and extension by the research community.

Reviewer comment (Experiments):
Regarding 'Dataset preparation' in the Supplementary Material, why are different methods used to generate conformers depending on the data? (Auto3D for Drugs-75k, Q2MM for EE, Open Babel with DTF?) Is this point not problematic for benchmarking several methods?

Author response:
A: Thank you for your careful reviews! In our benchmark, the choice of method for generating conformers was carefully considered to best suit the specific characteristics of each dataset. For example, Auto3D was chosen for Drugs-75K because of its efficiency in generating high-quality conformers. Similarly, Q2MM was employed for the EE dataset due to its ability to accurately generate Transition State Force Fields (TSFFs). It is important to note that these tools are not only well-established but have also been extensively validated within the chemistry community. It is also important to note that many real-world tasks in the chemistry community employ different conformer generation workflows, and hence our benchmark properly reflects this diversity of conformer generation strategies.

Also, all baseline models are evaluated using the same set of datasets. This ensures a fair and direct comparison across different methods. The diversity in conformer generation does not detract from the comparability of results but rather enriches the benchmark by covering a broader spectrum of realistic scenarios in molecular machine learning.

Reviewer comment (Experiments):
Is the BDE task also about predicting the Boltzmann-averaged value?

Author response:
A: Thank you for your valuable feedback regarding our paper. You raised an important question about whether the Binding Energy Difference (BDE) task in our study also involves predicting the Boltzmann-averaged value. Indeed, a Boltzmann-averaged BDE would more holistically capture physical reality, as it would account for the full distribution of thermodynamically-accessible conformers under experimental conditions. However, in our case, the BDE dataset was computed with a simpler approach to manage the computational cost of DFT calculations.  In the BDE dataset, binding energy difference is computed from single-point energy calculations (at a high level of DFT) between the two lowest-energy conformations of the unbound/bound complexes. These lowest-energy structures (one for each of the unbound and bound structures) were identified following a conformer search with OpenBabel. The lowest energy conformers then underwent further geometry optimization with DFT at the B3LYP/3-21G and B3LYP/def2-SVP levels, followed by a single-point energy calculation at the more expensive def2-TZVP level. The regression label is the (single-point) energy difference between the unbound and bound structures. The input conformers encoded by our models, however, are the original OpenBabel conformers prior to conformer selection and geometry optimization. Please note that the BDE is still a function (albeit indirect) of this conformer ensemble, and thus nicely fits within the MARCEL benchmark. The use of OpenBabel conformers as input to the models more realistically represents the real-world scenario during inference in which expensive DFT calculations are not practical.

We will revise the relevant sections to ensure this distinction is explicitly stated and understood, thus preventing any potential confusion.