Paper: Latent 3D Graph Diffusion (ICLR-2024)

Reviewer comment (Writing):
What's the difference between a one-shot AE and a Cascaded AE? The paper claims that "a one-shot AE embeds and reconstructs molecule data, evaluating both structure topology and geometry simultaneously." How does a Cascaded AE differ? Are topology and geometry independent from each other in a Cascaded AE?

Author response:
- The reviewer understands correctly that the topological and geometric AEs encode and sequentially decode, independent from each other in the cascaded version, and in the one-shot version the two AEs jointly encode/decode. Please also refer to Appendices B.1 & B.2 for more information.

Reviewer comment (Writing):
I would recommend adding an algorithm section to improve clarity. Based on my current understanding, a graph is initially encoded by an encoder in the latent space, then it undergoes diffusion in the latent space, and finally, it is transformed back to the graph space. Please correct me if I'm mistaken. Could you also clarify which diffusion model is used for the diffusion process in the latent space?

Author response:
- The reviewer understands our pipeline correctly.  
- Following the suggestion, we add the detailed algorithm sections (Algorithms 1 & 2) to describe our latent 3D graph diffusion, beyond the overview Figures 2 & 3. They describe the processes of (conditional) in/equivariant latent encoding, latent diffusion, and sampling.  
- The adopted diffusion model is DDPM (https://arxiv.org/abs/2006.11239).

Reviewer comment (Evaluation):
The paper asserts that "a latent space should possess (i) low dimensionality, (ii) low reconstruction error, and (iii) preserve group symmetry." How do you ensure that the autoencoder can meet these objectives? Have you conducted any analysis in this regard?

Author response:
- The factors of (i) low dimensionality and (iii) preserving group symmetry are by design in the AE architecture, which is guaranteed.  
- The factor of (ii) low reconstruction error is achieved by building the 3D graph AE in a cascaded manner, as one of our contributions described in Section 3.1. The numerical comparison of different AEs is shown in Tables 1 & 11.

Reviewer comment (Experiments):
For the objective of unconditional 3D molecular generation, I noticed the absence of results from "Ours-GSSL". Including these would be crucial to validate the effectiveness of GCL.

Author response:
- We provide additional results in Tables 15 & 16, on applying GSSL for unconditional generation and conditional generation with less training data.
- Based on our experiments, we observed that the benefit of GSSL is mainly on:
    - OOD scenarios when the GSSL prior is aligned with the condition semantics, as we stated in Section 4.2;
    - Unconditional or ID scenarios when training data are insufficient. This is consistent with the role of GSSL in discriminative modeling (https://arxiv.org/abs/2010.13902).
- For the OOD scenarios, though the improvement does not appear for all datasets, we know when it is improved: Specifically, we can measure the semantics-alignment by computing the homogeneity ratio as detailed in Appendix E.2. This indicates a future potential to design semantics-specific GSSL tasks for generative modeling.

Reviewer comment (Evaluation):
Regarding the unconditional 3D molecule generation, the 'molsta' metric, which gauges molecular integrity, seems paramount. From the data presented, your findings exhibit significant variances when juxtaposed against the previously introduced 3D latent diffusion model, GeoLDM. Additionally, the AtomSta and MolSta metrics for the Drug dataset appear to be omitted.

Author response:
- Our method is different from GeoLDM. We summarize the difference across different methods in Table 10, mainly lying in two perspectives: (i) latent dimension and (ii) the modeling of topological and geometric features.
    - Latent dimension (LD). Assuming the feature dimension of data is $N \times (D+3)$. GeoLDM, diffusing at the latent space of nodes is of LD $N \times (D’+3)$. Guided by our theory in Section 3.2, we chase a lower dimensional latent space by compressing the node factor $N$, that our method at the other end of the spectrum of the minimum LD $D’$.
    - Modeling of topological and geometric features. EDM, GeoLDM and our one-shot model jointly model the topological and geometric features for all $N$ nodes. However, our one-shot model cannot manage to reconstruct the feature per node for its overly low LD, that the node factor $N$ is compressed. Guided by our theory in Section 3.2, we develop the cascaded model, achieving satisfactory reconstruction performance while maintaining symmetry and sufficiently low LD.
- The above aspects lead to different results: GeoLDM performs better in generating geometrically stable molecules, and our method performs better in generating topologically valid molecules. Both of them are important in real-world applications.
- The evaluation is standardized as in EDM and GeoLDM, and we do report AtomSta for Drugs. Following EDM and GeoLDM, the MolSta metric is less informative to be evaluated in Drugs, since the ground-truth molecules in Drugs have 86.5% atom-level and nearly 0% molecule-level stability, due to which contain larger and more complex structures, creating errors during bond type prediction based on pair-wise atom types and distances.

Reviewer comment (Evaluation):
In the context of conditional generation based on (invariant) quantum properties, it's essential to evaluate both ID and OOD. Yet, the "Ours-GSSL" performance doesn’t exhibit a noteworthy distinction compared to "ours", "Random", and similar methods. In some instances, it even underperforms. This might cast a shadow on the robustness of conclusions articulated in Section 4.2, especially point (v).

Author response:
- We provide additional results in Tables 15 & 16, on applying GSSL for unconditional generation and conditional generation with less training data.
- Based on our experiments, we observed that the benefit of GSSL is mainly on:
    - OOD scenarios when the GSSL prior is aligned with the condition semantics, as we stated in Section 4.2;
    - Unconditional or ID scenarios when training data are insufficient. This is consistent with the role of GSSL in discriminative modeling (https://arxiv.org/abs/2010.13902).
- For the OOD scenarios, though the improvement does not appear for all datasets, we know when it is improved: Specifically, we can measure the semantics-alignment by computing the homogeneity ratio as detailed in Appendix E.2. This indicates a future potential to design semantics-specific GSSL tasks for generative modeling.

Reviewer comment (Evaluation):
Concerning the experiments related to conditional generation associated with (equivariant) protein targets, I'd recommend broadening the evaluation scope by incorporating metrics such as the Vina Score and Vina Dock in your findings. Furthermore, beyond Targetdiff, newer techniques like DiffSBDD[1] and DecompDiff[2] have emerged. Comparing your approach with these could provide a more holistic view. As per the metrics currently displayed, the method delineated in your manuscript doesn't seem to lead the pack.

Author response:
- We provide a more comprehensive evaluation including metrics of Vina Score, Vina Dock, and baselines of DiffSBDD and DecompDiff in Table 6.
- Overall, there is no single method that outperforms across all metrics. Our method is very competitive to achieve the best on 4 out of 8 metrics.
- Our method in general performs well in generating small molecules that are more drug-like, easier to synthesize, and more diverse compared to state-of-the-arts.
- Although our method underperforms in the population evaluation of Vina Score, Vina Dock, it significantly outperforms others on Vina Score (top 10%), following the DiffSBDD evaluation. This indicates that by modeling the distribution in the more compact and semantic-aware 3D graph latent space, it owns the potential to generate very potent small-molecule binders to the protein target. This would be very useful in real-world drug discovery to screen the effective drug candidates.

Reviewer comment (Presentation):
In Figure 4, there appear to be some omission errors; the circled references seem to be missing.

Author response:
- We thank the reviewers for comments. We have added the reference and will further refine the structure of Appendix A for better presentation.

Reviewer comment (Experiments):
The design of latent diffusion appears to be complex. Have you considered simplifying it by merely applying KL divergence to $z^{{0}}$ and deriving a VAE-like model using the encoder and decoder as proposed in this paper? An ablation study should be added here.

Author response:
- We provide the suggested ablation study in Tables 12-14.
- We observed that the diffusion model outperforms VAE here, and agreed with the findings in many fields, e.g. https://arxiv.org/abs/2011.13456.

Reviewer comment (Experiments):
Given your assertion that your model "produces superior 3D molecules faster" owing to latent diffusion, could you perhaps contrast it with GeoLDM, a previously introduced 3D latent diffusion model?

Author response:
- We thank the reviewer for the comment. We supplement the GeoLDM running time result in Table 4.

Reviewer comment (Experiments):
Is it possible to conduct a more in-depth evaluation and comparative study of experiments related to the generation of conditions related to equivariant protein targets?

Author response:
- We provide a more comprehensive evaluation including metrics of Vina Score, Vina Dock, and baselines of DiffSBDD and DecompDiff in Table 6.
- Overall, there is no single method that outperforms across all metrics. Our method is very competitive to achieve the best on 4 out of 8 metrics.
- Our method in general performs well in generating small molecules that are more drug-like, easier to synthesize, and more diverse compared to state-of-the-arts.
- Although our method underperforms in the population evaluation of Vina Score, Vina Dock, it significantly outperforms others on Vina Score (top 10%), following the DiffSBDD evaluation. This indicates that by modeling the distribution in the more compact and semantic-aware 3D graph latent space, it owns the potential to generate very potent small-molecule binders to the protein target. This would be very useful in real-world drug discovery to screen the effective drug candidates.

Reviewer comment (Novelty):
The novelty of the work is somewhat limited. It takes an existing 3D molecule generation setup with diffusion models and transforms it into latent diffusion, which is a known concept in general. Some latent 3D models do exist (e.g., GEOLDM), and the work uses mostly off-the-shelf architectures.

Author response:
- **Our work is not a direct extension of latent diffusion into 3D graphs.** We aim at generating 3D graphs of symmetry-group equivariance through diffusion. In other words, the quality of the generated graphs of topology and geometry is invariant / equivariant to a set of transformations (permutation group and Lie group E(3) of rotations and translations in our study).  This aim has deep roots in 3D molecular generation as demonstrated in our numerical experiments for desired quantum properties (invariant to E(3) perturbation of the 3D molecules) or desired protein binding (equivariant to E(3) perturbation of the 3D molecules); and it has broader application scopes for 3D graphs of symmetry-group equivariance in general.
- **Contribution (i).** We provided theoretical analysis (Section 3.2) on what space should diffusion generative models operate on for generating 3D graphs of non-Euclidean structure and group-symmetry equivariance.  We addressed group-symmetry equivariance while deriving model performance bounds.  Specifically, Proposition 1 gave the upper bound of the diffusion model performances (as measured in total variation distance) operating in the original 3D graph domain ($D’$-dimensional); whereas Proposition 2 gave the upper bound in the latent space ($D’’$-dimensional). The latter led to a better upper bound of the diffusion model performances if reconstruction errors are low and the latent dimension is low, which motivates our approach of latent diffusion.
- **Contribution (ii).** In pursuing the latent space as guided by our theoretical, motivational analyses, we constructed a 3D graph autoencoder of the following properties: (1) permutation and E(3) invariant and (2) sufficiently low reconstruction errors despite low latent dimension (these two present nontrivial trade-offs), which is under-explored.  We accomplished this by combining 2D graph autoencoder (for topology features) and point cloud autoencoder (for geometry features given topologies) in a cascade manner (Section 3.1).
- **Contribution (iii).** To improve the out-of-domain (OOD) generation, we further incorporate graph self-supervised learning into learning the 3D graph autoencoder and the resulting latent space for diffusion (Section 3.1).  Our self-supervised latent diffusion has been largely unexplored for 3D graphs of group-symmetry equivariance.  Our numerical experiments demonstrated its effectiveness in improving OOD performances, which is a critical barrier for designing molecules with much improved properties or new properties.
- **Contribution (iv).** We further extend our work into conditional generation of 3D graphs, which stems from practical applications and presents a new challenge.  Namely the new challenge is to maintain conditional equivariance, that is, the generated 3D graphs should be E(3)-invariant to the property condition and E(3)-equivariant to the protein binder’s structure condition. We addressed this challenge by introducing conditional equivariant geometric autoencoder in our latent diffusion pipeline (Section 3.3).
- **Difference from GeoLDM.** We summarize the difference across different methods in Table 10, mainly lying in two perspectives: (i) latent dimension and (ii) the modeling of topological and geometric features.
    - Latent dimension (LD). Assuming the feature dimension of data is $N \times (D+3)$. At  one end of the spectrum, EDM diffusing in the original space is of the maximum LD $N \times (D+3)$. The followup work, GeoLDM, diffusing in the latent space of nodes is of LD $N \times (D’+3)$. Guided by our theory in Section 3.2, we chase an even lower dimensional latent space by compressing the node factor $N$, that our method at the other end of the spectrum of the minimum LD $D’$.
    - Modeling of topological and geometric features. EDM, GeoLDM and our implemented one-shot model jointly model the topological and geometric features for all $N$ nodes. However, our one-shot model cannot manage to reconstruct the feature per node for its overly low latent dimension, that the node factor $N$ is compressed. Guided by our theory in Section 3.2, we develop the cascaded model, achieving satisfactory reconstruction performance while maintaining symmetry and sufficiently low LD.

Reviewer comment (Experiments):
One of the main contributions of this work is the cascaded autoencoder, and the main motivation for why it is needed stems from the fact that a one-shot AE effectively failed to train. It is strange that the one-shot AE fails to train completely, as there are 3D molecule generative models that generate 2D and 3D graph features jointly (e.g., MIDI or the example referenced in the paragraph below). These usually re-weight the losses for 2D and 3D terms, but they do work. A more detailed analysis is needed on why the one-shot AE doesn't work, including potentially an ablation on the one-shot AE architecture and losses. While it is understandable that a one-shot AE could work worse than the proposed cascaded AE with teacher forcing, the fact that it essentially performs like a random initialization raises concerns about whether it was tested sufficiently carefully.

Author response:
- We provide the re-weighting results in Table 11.
- We tested one-shot AE extensively. We have conducted very extensive experiments on tuning the one-shot AE, including loss re-weighting, architecture variation, and hyperparameter optimization. The training does not converge and the reconstruction looks random, which turned us to the cascaded manner.
- The potential reason might lie in the dimensionality of our latent space, as described in the Q1 answer, that our latent space dimensionality is overly low, compressing the node factor $N$. This leads to the challenge of optimizing one-shot AE.

Reviewer comment (Evaluation):
In section 3.1 it is stated that graph matching is used for the AE training loss to ensure permutation and SE3 invariance. This can be computationally expensive. More details are needed on how fast the AE training is, both in terms of wall time and asymptotically (in O notation). Without this information, comparing the training time of the proposed method versus EDM is complicated. As understood, Table 4 does not account for AE training time (even though AE can be trained once, it still requires training).

Author response:
- We train both topology and geometry AEs on one NVIDIA A100 GPU for 2 days. The latent diffusion training time depends on the datasets, e.g. in Geom-Drugs it takes ~4 hours compared to 5.5 days of EDM with the same epochs.
- Also noting that AEs are pretrained for once and repetitively utilized in varied scenarios e.g. conditional generation on varied properties. This is very important since effective and efficient multi-property optimization is highly demanded in the real-world drug discovery process (https://www.nature.com/articles/nrd1086).
- It is difficult to directly compare the asymptotic complexity. While both AE and EDM are composed of building blocks of invariant/equivalent representation neural networks, it would lead to similar asymptotic complexity if they are constructed with the same building blocks.

Reviewer comment (Experiments):
Since the AE is trained on a much larger dataset, it might be fair to also pre-train the EDM on general molecule generation. It has been shown that pretraining helps molecule generation (https://arxiv.org/pdf/2309.17296.pdf). While this paper is recent, it would still make more sense to compare against such a properly tuned state-of-the-art molecular diffusion setup instead of vanilla EDM, which is an older baseline (the first ever diffusion model for 3D molecule generation).

Author response:
- We thank the reviewer for bringing this interesting paper, though the numbers in the paper are not directly comparable to ours for its different setting, and we are currently unable to reproduce the carefully designed and complicated algorithm without their codebase public.
- The conclusion in the mentioned paper differs from us on pretraining. The mentioned paper concludes that the benefit of pretraining is in the small-data regime: “Interestingly, the fine-tuned model shares similar (best) scores with trained from scratch on 100% of the data” (Section 6). Our latent diffusion improves the generation quality in the large-data regime (100% of the data).
- We have implemented our diffusion model pretraining experiment in the larger PubChem3D database to validate the conclusion. The diffusion model pretraining is very time-consuming (compared to simply pretraining AE), expected to take more than 12 days of the rebuttal window. We will include the part of the experiment in our later revision.

Reviewer comment (Experiments):
MIDI, which is cited numerous times in the paper, is not compared against in the experiments. This is notable because MIDI performs significantly better than EDM in certain metrics, and EDM is used as the main baseline here.

Author response:
- The reported results in MiDi were evaluated in a different setting as stated in Section 5.2, which we did not include before.
- We re-evaluated MiDi in the EDM setting and supplemented the result in Table 2.

Reviewer comment (Experiments):
In section 3.2 Setup, the authors say that connectivity is commonly determined later based on domain rules and cite MIDI among others. While it is true that such domain-based rules are used to recover connectivity in, for example, EDM, in MIDI connectivity is modeled explicitly.

Author response:
- We are sorry for the confusion. The reviewer is correct that MiDi is mistakenly cited at this place. We removed it in the paper.

Reviewer comment (Reproducibility):
According to this paper, I did not clearly catch how the authors project the 3D molecules into the 2D latent space. The technical details are missing. It naively mentioned that the authors borrow the architectures from previous works and such simple comment is not self-contained, in my opinion.

Author response:
- We are sincerely grateful to the reviewer’s feedback. We are committed to improving our clarity, and also would like to point out some aspects missed by the reviewers.
- We add the detailed algorithm sections (Algorithms 1 & 2) to describe our latent 3D graph diffusion, beyond the overview Figures 2 & 3. They describe the processes of (conditional) in/equivariant latent encoding, latent diffusion, and sampling.
- More specifically, the details of invariant latent encoding are further presented in Section 3.1.
- The details of conditional in/equivariant latent encoding are presented in Section 3.3.
Regarding latent encoding architectures, we supplement more detailed descriptions of architecture design in Appendix B.2 & C.1
- The details of diffusion model are presented Section 2.

Reviewer comment (Theory):
Why is the latent space typically important to the graph generation? While several recent studies utilize the latent space for the representation of diffusion-based generative models, I could not find any theoretical analysis behind the bridge between _the necessity of the latent space_ and _the properties of the 3D graph representation, typically for the 3D molecules_. For me, this appears as a naive extension of latent diffusion models into graph representation.

Author response:
- **Our work is not a naive extension of latent diffusion into 3D graphs.** We aim to generate 3D graphs of symmetry-group equivariance through diffusion. In other words, the quality of the generated graphs of topology and geometry is invariant / equivariant to a set of transformations (permutation group and Lie group E(3) of rotations and translations in our study).  This aim has deep roots in 3D molecular generation as demonstrated in our numerical experiments for desired quantum properties (invariant to E(3) perturbation of the 3D molecules) or desired protein binding (equivariant to E(3) perturbation of the 3D molecules); and it has broader application scopes for 3D graphs of symmetry-group equivariance in general.
- Thus, directly related to the reviewer’s question, **one contribution of our work is to demonstrate the necessity of building the 3D graph latent space** to achieve a potential generation quality. We provided theoretical analysis (Section 3.2) on what space should diffusion generative models operate on for generating 3D graphs of non-Euclidean structure and group-symmetry equivariance.  We addressed group-symmetry equivariance while deriving model performance bounds. 
    - Specifically, Proposition 1 gave the upper bound of the diffusion model performances (as measured in total variation distance) operating in the original 3D graph domain ($D’$-dimensional);
    - whereas Proposition 2 gave the upper bound in the latent space ($D’’$-dimensional). The latter led to a better upper bound of the diffusion model performances if reconstruction errors are low and the latent dimension is low, which motivates our approach of latent diffusion.
- **Another contribution is to illustrate how to construct such 3D graph latent space (autoencoder), guided by our theoretical, motivational analyses**, of the following properties: (1) permutation and E(3) invariant and (2) sufficiently low reconstruction errors despite low latent dimension (these two present nontrivial trade-offs), which is under-explored.  We accomplished this by combining 2D graph autoencoder (for topology features) and point cloud autoencoder (for geometry features given topologies) in a cascade manner (Section 3.1).

Reviewer comment (Theory):
Can the authors exactly prove the equation in Sec 3 of this manuscript?  
_3D Graph Diffusion Performance <= Latent Space Reconstruction Quality + Symmetry Preservation * Data Dimensionality_

Author response:
- The plain language equation is the informal expression of Proposition 2, which is proved in Appendix A.

Reviewer comment (Experiments):
If the main contributions of this paper are about the analysis of the latent space in the 3D graph representation, the authors should have provided clear results or an ablation study about this contribution. For instance, using the same baseline model from (Hoogeboom et al., 2022), the authors could slightly modify the architecture while maintaining the number of parameters for fair comparison. However, I could not find such kind of analysis or experiment.

Author response:
- The contribution of 3D graph latent spaces was demonstrated in our results, and the confusion may arise from a lack of a more explicit summary in our presentation. We added them in the paper. Following our answers in Q3:
    - Lower LD latent space, better generation quality: Shifting from EDM and GeoLDM by reducing LD per node, better generation quality is achieved.
    - Overly low LD latent space challenges its quality (in reconstruction) for generation: Shifting from GeoLDM to our one-shot model by further reducing the node factor in LD, it is difficult to achieve sufficient low reconstruction errors for generation.
    - More qualified latent space (in reconstruction), better generation quality: Shifting from our one-shot model to cascaded model, we achieve state-of-the-art generation results by building the 3D graph AE with low reconstruction errors.
    - More “well-structured” latent space (in semantics), better OOD generation quality: Shifting from our cascaded model to self-supervised cascaded model, we achieve better OOD conditional generation results when the homogeneity metric is higher.
- We also completely agree with the reviewer’s opinion of ablation studies, though the suggested one of slight variation from EDM is difficult to implement in the short period since our pipeline is not trivially extended, starting from EDM.
- We follow the suggestion to provide the result when the 3D latent space is omitted in Table 17, leading to a worse generation quality. Since the 2D latent space is the prerequisite for 3D in the cascaded model, it demonstrates the necessity of both AEs in our pipeline.

Reviewer comment (Theory):
While the authors describe full of equations with comments, the concept in Proposition 1 is highly straightforward. For instance, in the manuscript, the authors said: '__Proposition 1.__ _Performance bound of 3D graph diffusion is related to feature dimensionality_'. In my opinion, simply increasing the network capacity by adding more layers or increasing the channel length with lots of training data would surely result in performance gains for diffusion models. I cannot catch the authors' intention from this proposition.

Author response:
- We are sorry for the confusion. The feature dimensionality in Proposition 1 refers to the feature dimensionality of data, which does not relate to the network capacity. We revised the comment for better clarity. Please also see our last answer for the intention behind Proposition 1.

Reviewer comment (Experiments):
What if the authors intentionally omit the latent space encoding? Instead of using 2D-3D autoencoders, what happens if the network only consists of the 3D autoencoders? How much gain could we obtain if we adopt the latent space encoding?

Author response:
- The contribution of 3D graph latent spaces was demonstrated in our results, and the confusion may arise from a lack of a more explicit summary in our presentation. We added them in the paper. Following our answers in Q3:
    - Lower LD latent space, better generation quality: Shifting from EDM and GeoLDM by reducing LD per node, better generation quality is achieved.
    - Overly low LD latent space challenges its quality (in reconstruction) for generation: Shifting from GeoLDM to our one-shot model by further reducing the node factor in LD, it is difficult to achieve sufficient low reconstruction errors for generation.
    - More qualified latent space (in reconstruction), better generation quality: Shifting from our one-shot model to cascaded model, we achieve state-of-the-art generation results by building the 3D graph AE with low reconstruction errors.
    - More “well-structured” latent space (in semantics), better OOD generation quality: Shifting from our cascaded model to self-supervised cascaded model, we achieve better OOD conditional generation results when the homogeneity metric is higher.
- We also completely agree with the reviewer’s opinion of ablation studies, though the suggested one of slight variation from EDM is difficult to implement in the short period since our pipeline is not trivially extended, starting from EDM.
- We follow the suggestion to provide the result when the 3D latent space is omitted in Table 17, leading to a worse generation quality. Since the 2D latent space is the prerequisite for 3D in the cascaded model, it demonstrates the necessity of both AEs in our pipeline.

Reviewer comment (Evaluation):
Limited generalization capability in comparison.

Author response:
**Q1.** Limited generalization capability in comparison. How to improve generalization capability?

**Answer.**
- We provide additional results in Tables 15 & 16, on applying GSSL for unconditional generation and conditional generation with less training data.
- Based on our experiments, we observed that the benefit of GSSL is mainly on:
    - OOD scenarios when the GSSL prior is aligned with the condition semantics, as we stated in Section 4.2;
    - Unconditional or ID scenarios when training data are insufficient. This is consistent with the role of GSSL in discriminative modeling (https://arxiv.org/abs/2010.13902).
- Our results indicate a future potential to design semantics-specific GSSL tasks for generative modeling.

Reviewer comment (Experiments):
How to improve generalization capability?

Author response:
**Q1.** Limited generalization capability in comparison. How to improve generalization capability?

**Answer.**
- We provide additional results in Tables 15 & 16, on applying GSSL for unconditional generation and conditional generation with less training data.
- Based on our experiments, we observed that the benefit of GSSL is mainly on:
    - OOD scenarios when the GSSL prior is aligned with the condition semantics, as we stated in Section 4.2;
    - Unconditional or ID scenarios when training data are insufficient. This is consistent with the role of GSSL in discriminative modeling (https://arxiv.org/abs/2010.13902).
- Our results indicate a future potential to design semantics-specific GSSL tasks for generative modeling.