Paper: GDL-DS: A Benchmark for Geometric Deep Learning under Distribution Shifts (ICLR-2024)

Reviewer comment (Novelty):
How do the findings of this study compare with earlier research on CV and NLP tasks concerning distribution shifts?

Author response:
Firstly, we’d like to appreciate earlier OOD works on CV and NLP tasks, many of which have shed valuable insights into our research. We have meticulously connected our multiple conclusions with the existing OOD work in CV and NLP and put details in the appendix. The details are as follows.

- In Sec. 4.2, we observe that fine-tuning can sometimes result in negative effects when the labeled OOD data is quite limited. **This is consistent with catastrophic forgetting in [1]. To mitigate this issue, the strategy of surgical fine-tuning raised in [2]** is a potential solution for this problem.

- In Sec. 4.2, we observe that multiple OOD generalization methods in our No-Info level find it hard to provide significant improvement across various applications. We can find **similar observations in existing CV/NLP works, including [3], [4]**. Actually, this could be even **more severe in GDL compared to CV tasks**, considering the intricate nature of irregularity and geometric prior (information on the structure space and symmetry properties like invariance or equivariance) inherent in geometric data.   

- In Sec. 4.3, we conclude that “For the OOD generalization methods to learn robust representations, the more informatively the groups obtained by splitting the source domain $S$ indicate the distribution shift, the better performance these methods may achieve.” This is **consistent with previous CV works like [5]**, which revealed the importance of appropriate subgroup partitioning for invariant learning. And we note that **[6] proposed to leverage** additional auxiliary variables for enhancing OOD generalization. This may also shed insights into future research on scientific GDL. 

Moreover, certain shifts in scientific GDL are different from common shifts in CV/NLP, which leads to challenges when addressing these shifts with methods initially proposed for CV/NLP tasks. Here are some examples in our work:

- Shifts related to graph/geometric structure: For instance, the I-conditional shift can be interpreted as a size shift (the number of nodes in data), where the model trained in data with smaller sizes is to generalize to data with larger sizes. We observed that the methods designed for CV struggle to perform well. In addition, the features in GDL additionally incorporate the geometric structure. The methods in CV work on feature representations without explicitly handling the change in geometric structure, which may result in suboptimal solutions. 

- Fidelity shift, which indicates the change of causal correlation between $X_c$ and Y, corresponds to a significant challenge in material property prediction. However, we rarely encounter the shift in the causal relationship in CV, so most methods in our benchmark find it hard to handle such a shift, except the pertaining-finetuning strategy. 

The above two points of **difference indicate the necessity to develop OOD methods specifically designed for GDL**.
Overall, We thank the reviewer’s suggestion and the comparisons made above would enhance the connection between our findings with earlier research in CV/NLP domains and shed more insights for future research.

[1] Kirkpatrick, James, et al. "Overcoming catastrophic forgetting in neural networks." Proceedings of the national academy of sciences 114.13 (2017): 3521-3526.

[2] Lee, Yoonho, et al. "Surgical Fine-Tuning Improves Adaptation to Distribution Shifts." ICLR, 2022.

[3] Gulrajani, Ishaan, and David Lopez-Paz. "In Search of Lost Domain Generalization." ICLR, 2020.

[4] Koh, Pang Wei, et al. "Wilds: A benchmark of in-the-wild distribution shifts." ICML, 2021.

[5] Creager, Elliot, Jörn-Henrik Jacobsen, and Richard Zemel. "Environment inference for invariant learning." ICML, 2021.

[6] Lin, Yong, et al. "ZIN: When and How to Learn Invariance Without Environment Partition?." NeurIPS, 2022.

Reviewer comment (Experiments):
What is the rationale behind the choice of the considered GL backbones? Would incorporating more diverse GDL backbones or learning algorithms significantly alter the conclusions drawn from this study?

Author response:
We consider choosing the three GDL backbones from the following 2 aspects.

Firstly, they are representative in different categories of GDL backbones. In our selected backbones, DGCNN is an early work of 3D-CNN designed for processing point cloud data; Point Transformer is a Transformer-based GDL architecture; EGNN is an E(n)-Equivariant Graph Neural Networks.

Secondly, they have been widely used for various scenarios of scientific GDL, such as [7], [8], and [9]. We are aware that there are several wonderful works regarding GDL network architecture designed for some specific scientific task, but we did not choose them considering that they are not easily applicable to diverse scientific domains. 

Overall, we found that the order of the results is basically consistent in different backbones, and our chosen baselines tend to span a variety of learning mechanisms, so we believe that the overall conclusions would not be significantly altered by adding more backbones or algorithms.

[7] Qu, Huilin, and Loukas Gouskos. "Jet tagging via particle clouds." Physical Review D 101.5 (2020): 056019.

[8] Atz, Kenneth, Francesca Grisoni, and Gisbert Schneider. "Geometric deep learning on molecular representations." Nature Machine Intelligence 3.12 (2021): 1023-1032.

[9] Gagliardi, Luca, et al. "SHREC 2022: protein–ligand binding site recognition." Computers & Graphics 107 (2022): 20-31.

Reviewer comment (Writing):
The use of critical terms should be better considered. Concept drift is a well-established term in the study of causality and distribution shift. As defined in [1], which the authors also cited, the only constraint for concept drift is "changes in $p(y|X)$". To avoid any confusions to readers, this conventional definition should be followed without modifications like $P(Y|X_c)$. Similarly for the definition of covariate shift. If the authors attempt to define a more specific kind of shift, another term should be used.

Author response:
**1). Use of terms: “The use of critical terms should be better considered.”**

We thank the reviewer for raising this confusion based on our paper presentation. We are not aiming to modify the existing well-established terms. Instead, we would like to further explain and extend the common definition of these well-established terms under the context of our selected application datasets. We have clarified in our refined draft by first following the well-established definitions of these distribution shifts and then highlighting that we further extend the definition to what we present in the paper given our assumption.

Reviewer comment (Theory):
The statement that $X$ consists of two disjoint parts and $X_i ⊥Y |X_c$ does not hold. An easy violation would be $X_c →Y →X_i$. Intrinsically, $Y$ is often a property of the input and therefore $X$ cannot be divided into two disjoint parts that are causal/independent, but there would exist a part of $X$ that is statistically associated with $Y$ while non-causal to $Y$. A classic example is the PIIF and FIIF causal modeling, such as the analysis in [2].

Author response:
- 2.1 	soundness of causal statements: “The statement that $X$ consists of two disjoint parts and $X_i \bot Y|X_c $ does not hold” 

This review confuses “the settings this work is targeting” and “other possible settings that this work is not to cover”. Note that such conditional independence just describes the settings approximately indicated by our datasets. We agree that there exist other types of dependence, while they are out of the scope of this work because we did not find interesting scientific GDL datasets that reflect those types of dependence. Moreover, limiting the scope properly allows for more in-depth study without losing much generality.

First, such conditional independence is derived from real-world scientific application and holds in our established scenarios. Take our track Dataset in particle physics as an example. The entire point cloud can be split into 2 parts, namely the signal part used to determine whether there is a specific signal we are looking for (determine the label), and the background part, which itself is independent of the label given the signal part. This exactly follows the conditional independence assumption as $X_i \bot Y|X_c$. The other two datasets also comply with the assumption that the label is determined by only a part of the data entry, with another part being statistically correlated but conditionally independent of the label given the causal part. Therefore, we respectfully disagree with the argument that “X cannot be divided into two disjoint parts that are causal/independent”, which is at least properly reflected by the datasets considered in this work. 

Second, this setting of conditional independence is of a broad range of research interest to the machine learning community. For instance, we can find similar formulations in DIR [1], Invariant Rationalization [2], etc. Take DIR for example, Equation 1 in this paper indicates the conditional-independence property of the oracle rationale, where $S$ is “statistically associated with $Y$ but non-causal to $Y$”, and is independent of $Y$ given the causal part $C$.

In a word, the example $X_c\to Y\to X_i$ raised by the reviewer exists, but it is beyond the scope of this work, especially when we did not find a scientific GDL dataset that obeyed such a setting. We are happy to enrich our benchmark in the future if the reviewer may suggest some scientific GDL datasets that reflect the settings the reviewer would like to cover.  

[1] Wu, Yingxin, et al. "Discovering Invariant Rationales for Graph Neural Networks." ICML, 2021.

[2] Chang, Shiyu, et al. "Invariant rationalization." ICML, 2020.

Reviewer comment (Theory):
Following the above point, even for $X → Y$, $P(Y|X)P(X) = P(Y|X_c)P(X)$ does not hold. For $X → Y$, there can be a case where $P_S(Y |X_i)\neq P_T (Y |X_i)$, which will also result in a "conditional shift". It is also included by the definition of concept shift. Constraining $Y → X$ does not seem like a necessity for conditional shift.

Author response:
- 2.2	Involving data generation process when characterizing shift: “Constraining $Y\to X$ does not seem like a necessity for conditional shift”

We understand the reviewer’s comment that the data generation process $X \to Y$ or $Y \to X$ is not a necessity to characterize a type of distribution shift. However, given a data generation process ($X \to Y$ or $Y \to X$), it becomes more clear to tell by domain experts which type of distribution shift a dataset satisfies. For instance, pairing the generating process $X\to Y$ with $P_S(Y|X_c)\ne P_T(Y|X_c)$ perfectly describes the mechanism of fidelity shift, an important real-world scenario in molecular dynamic simulations in materials science. And plus pairing $Y\to X$ with $P_S(X|Y)\ne P_T(X|Y)$ perfectly aligns with the proposed scenarios of particle physics, as $Y$ indicates the type of particle decay while $X$ is the particle hit data generated from this type of decay.

As the reviewer argued, the data generation process $X \to Y$ may also satisfy conditional shift $P_S(X|Y)\ne P_T(X|Y)$ (in theory), but it may also satisfy $P_S(X|Y)=P_T(X|Y)$. There is not a simple way to tell whether the conditional shift exists and how large such the conditional shift could be. There is possibly $P_S(Y|X_i)\ne P_T(Y|X_i)$ that can also derive $P_S(X|Y)\ne P_T(X|Y)$ but the datasets considered in this work do not have $P_S(Y|X_i)\ne P_T(Y|X_i)$. Instead, for the data generated via process $Y \to X$, it is easy to tell the existence of $P_S(X|Y)\ne P_T(X|Y)$ and the degree of such a shift by analyzing how $X$ is generated given $Y$.  Our way of categorization is more succinct and aligns well with practical understandings. 

Moreover, the seminal work in ML [3] that initiated the study of conditional shift also incorporates the data generation process $Y\to X$. Our work follows this tradition.

[3] Zhang, Kun, et al. "Domain adaptation under target and conditional shift." ICML, 2013.

Reviewer comment (Novelty):
Contribution overclaimed and related works not well addressed. In the comparison with existing benchmarks, the authors claim no existing OOD benchmarks consider conditional shift, which is not true. OoD-Bench, GDS, and GOOD all include the Cmnist dataset, which is clearly conditional shift. GOOD also constructed conditional shift for each dataset. Also, though benchmarks like WILDS do not use test labeled/unlabeled data for algorithm learning, these OOD info are available. Therefore, Table 1 has multiple overclaiming issues, and the authors should treat existing works properly.

Author response:
**3).	Contribution overclaimed**
- 3.1	“CMNIST dataset in OoD-Bench, GDS, and GOOD is clearly conditional shift.”

The reviewer raises that the Cmnist Dataset follows a conditional shift while we missed discussing it in our literature review. We appreciate the reviewer for this insightful comment . After revisiting the papers, we found they satisfied the concept shift definition as a change in $P(Y|X)$. When constructing Table 1, we didn’t mark them as “concept shift” since their mechanism is different from what we formalized as the “concept shift” in our work. We are sorry for being too specific in Table 1 and missed this point. In the revised manuscript, we revised Table 1 and clarified the missing contributions of previous works. In the following response, we will detail how the shift in CMNIST correlates to our work.

All these works mentioned in this comment did not explicitly define the shift as “conditional shift”. From the explicit formulations established in OoD-Bench[4] and GOOD[5], it corresponds to concept shift, characterized by the change of $P(Y|X)$. So we tend to treat it as “concept shift” rather than “conditional shift” to avoid confusion.
Also, it is important to note that this concept shift in GOOD has **a different mechanism than the one formalized in our study**: In our work, concept shift particularly denotes the change in causal correlation between $X_c$ and $Y$. In contrast, in GOOD's context, concept shift corresponds to the change in statistical rather than causal correlation, such as correlations between color and digit in CMNIST.

Overall, we have added more comparison & clarification in the refined draft and tend not to make confusion. We thank the reviewer again for pointing out this.

- 3.2	OOD info is available in WILDS

We thank the reviewer for pointing out the confusion. In Table 1, the column “Available OOD Info” actually means how previous work leveraged the learning algorithms corresponding to the three OOD-Info levels. WILDS did not use test labeled/unlabeled data for algorithm learning.
We have made further clarification in Table 1 in the refined draft.

[4] Ye, Nanyang, et al. "Ood-bench: Quantifying and understanding two dimensions of out-of-distribution generalization." CVPR, 2022.

[5] Gui, Shurui, et al. "Good: A graph out-of-distribution benchmark." NeurIPS, 2022.

[6] Wiles, Olivia, et al. "A Fine-Grained Analysis on Distribution Shift." ICLR, 2021.

Reviewer comment (Experiments):
Experimental setting not fair. Some methods are trained solely on the Train-ID dataset, while DA algorithms are trained on both Train-ID and OOD input data, and TLs also learn labeled Train-OOD data. This does not seem like a fair setting since different methods are trained on even different numbers of data samples. Given that the analyses are conducted based on comparing all these methods together, a fair evaluation setting is certainly needed.

Author response:
**4). The unfairness of the model training process**

This comment proposed by the reviewer precisely pertains to a key aspect of our contributions & motivation, while unfortunately causing some confusion on the reviewer’s side. We are sorry for the confusion and here are some clarifications.

We **intentionally** incorporate different levels of OOD information into the training process to investigate how training with different OOD information helps with model generalization. Comparing OOD methods stand-alone is not our goal. Instead, we are to compare different combinations of OOD methods with a certain level of OOD data info.  Such comparisons uncover useful insights into handling scientific application OOD challenges: The insights can guide researchers in selecting the appropriate category of methodology when having different levels of available OOD information under a specific shift. Researchers may even decide whether it is valuable to collect some unlabeled or labeled data from the testing domain by trading off between the cost of collecting such data and the expected performance gain given such extra OOD data info told by this work. Moreover, we claim we are fair in the evaluation process by fairly evaluating the boost of model generalization across the three settings on the same validation and testing dataset, which is well addressed in Sec. 4.1 of our paper.

Reviewer comment (Experiments):
Baselines out-of-date. These years many new OOD methods including new sota have been proposed. The benchmark should include more recent methods as baselines. For learning algorithms the sota methods on the Wilds leaderboard should be considered. For graph OOD methods, many recent methods can easily outperform DIR. Also, geometric methods specifically developed for scientific tasks should be considered.

Author response:
- 5.1	Baselines out of date

Firstly, the goal of the paper is not to find the SOTA methods. We select algorithms that are well-known and span a broad range of learning strategies under different levels of OOD info, including 1) vanilla ERM, 2) invariant learning, 3) data augmentation, 4) subgroup robustness, 5) causal inference, 6) information bottleneck, 7) domain invariance and adaptation (adversarial training & distance regularization), 8) Pre-training and Fine-tuning. Our selection of algorithms includes various fundamental learning principles in the OOD field, and newly proposed methods are also generally based on the aforementioned principles.

We are happy to include more methods to compare if those methods introduce principles to deal with OOD challenges that have not been covered by the above already included principles.

Reviewer comment (Experiments):
The benchmark includes only 3 datasets. Though more than one shift is identified for each dataset, this number seems a bit few for a benchmark. Given that the datasets are not newly collected, possibly more discussions on contributions like curating 3D coordinate can make up for the overall contribution.

Author response:
- 5.2	Lack in the number and originality of datasets

We respectfully argue that the review substantially underestimates our contribution. We have 3 scientific domains and 6 datasets in total, including Track-Pileup, Track-Signal, QMOF, DrugOOD-3D-Assay, DrugOOD-3D-Size, DrugOOD-3D-Scaffold, which can be seen in Table 2. We respectfully disagree with just treating one scientific domain as a dataset, otherwise DrugOOD [7], a wonderful benchmark in AI-aided Drug Discovery, would have had only a single dataset. Moreover, the two datasets in particle physics, i.e, Track-Pileup and Track-Signal, are newly created by ours. They are two fundamentally different application tasks in particle physics, of which each has thousands of researchers focused all over the world.   

[7] Ji, Yuanfeng, et al. "DrugOOD: Out-of-Distribution (OOD) Dataset Curator and Benchmark for AI-aided Drug Discovery--A Focus on Affinity Prediction Problems with Noise Annotations." arXiv preprint, 2022.