Paper: 3D Molecular Pretraining via Localized Geometric Generation (ICLR-2024)

Reviewer comment (Novelty):
Although the current method is not directly inspired by this work, the RL reconstruction of molecular geometries from 3D fragments in Flam-Shepherd et al (https://arxiv.org/abs/2202.00658) relates closely to the core inspiration of this method and might warrant discussion in the intro; in contrast, some of the discussion of the atom-bond inconsistency problem is potentially possible to skip as it doesn't add meaningful insights. The reviewer asks whether the authors thought of extending their tetrahedral segmentation approach to use a similar fragment-based approach instead.

Author response:
We appreciate you highlighting the promising fragment-based reconstruction approach by Flam-Shepherd et al. As you suggested, we will cite this relevant work in the introduction and discuss extensions of our tetrahedron segmentation to incorporate larger functional fragments as building blocks. This could enable more complex biologically meaningful reconstruction objectives in our future work.

Reviewer comment (Evaluation):
Can the authors comment on the disparity of the approach of their validation scores on moleculeNet vs those on the large-scale challenge?

Author response:
**Response:** For PCQM4M-v2, we would like to argue that existing top approaches incorporate sophisticated graph encoding priors, while we apply a pure transformer. Nevertheless, the benefits of our pretraining approach can be demonstrated by the performance increase over the non-pretrained TokenGT model.  The primary contribution of this work is to give a glimpse at how proper selection of semantic units impacts 3D molecular pretraining, and we believe a further introduction of graph inductive bias will further improve our result in table2.

Reviewer comment (Experiments):
The ablation study is not helpful: the perturbation of the model is too limited, and Table 3 suggests strongly that the parameters are actually not optimal.

Author response:
**Response:** Thank you for the feedback requesting more extensive ablation experiments and analysis, and we have revised the abaltion study section in our paper. In table 3, we show that excessive noise in 3D structure denoising can lead to training divergence and detrimental impacts and validate the setting of our perturbation strategy. In table 4, we validate the benefits of structural pretraining by comparing against a naive random atom-level denoising baseline with equivalent 36% masking.

Reviewer comment (Reproducibility):
What exactly constitutes the input for TokenGT-3D? Is it the original molecules, or do you utilize each local structure segmentation after masking and perturbation? Or is it the masked and perturbed local structure segmentations of a single molecule, or something else entirely?

Author response:
During pretraining, TokenGT-3D still take node and edge tokens, i.e. the original molecules as the input, as is shown in Figure 3 on page 4. The attribute mask and coordinate noise at perturbation stage are implemented token-wise during perturbation. Afterwards, the model takes both clean and perturbed tokens of the original molecule as the input; the segmentation on molecular conformations does not function as a 3D tokenizer.

Reviewer comment (Reproducibility):
Could you elaborate on how the local structures are reconstructed? What serves as the input for this process: a single embedding from the TokenGT-3D output, or a collection of embeddings from local structure segmentations within a single molecule?

Author response:
Thanks for your feedback. To better illustrate the local structure reconstruction procedure, we have revised the paper to reposition Algorithm 1 within the methodology section. As is previously stated in Q1, the model takes both clean and perturbed tokens as input, producing an embedding $\mathbf{Emb}$ in shape $[n+m,dim]$. During pretraining, we utilize three separate mask indicators -- $M_\text{center}, M_\text{edge},M_\text{leaf}$ -- for center atoms, edges and leaf atoms within the whole molecule. For reconstruction, the central atom embeddings, $\mathbf{Emb}[M_\text{center}]$, are fed into a specialized head to predict global orientation. Likewise, edge embeddings $\mathbf{Emb}[M_\text{edge}]$ and leaf node embeddings $\mathbf{Emb}[M_\text{leaf}]$are input into dedicated modules for reconstructing edge lengths and angles, respectively.