Paper: HYPO: Hyperspherical Out-Of-Distribution Generalization (ICLR-2024)

Reviewer comment (Theory):
Although Theorem 5.1 provides insights into the upper bound of generalization variation, it does not conclusively demonstrate the superiority of the proposed method or loss, since the theorem directly assumes that the variation can be optimized to a small value under the proposed loss, i.e., $\frac{1}{N}\sum_j\mu_{c(j)}^T z_j\ge 1-\varepsilon$. If one were to substitute an alternative loss, such as changing the prototype to another sample within the same class (e.g., employing the SupCon loss) or directly using PCL's loss, it would also yield a generalization bound. Consequently, the question arises: How can we establish that the proposed loss is indeed superior, provably?

Author response:
We would like to clarify that our goal is not to theoretically establish HYPO's superiority over SupCon and PCL. Our primary contribution is to provide the **first theoretical justification** of prototype-based learning algorithms for OOD generalization. This is also acknowledged by Reviewer W1ps, who commented:

> _"...several learning methods have previously been proposed that utilize hyperspherical embeddings. But this paper is the first to provide a theoretical justification angled at OOD generalization."_

Unlike PCL, our learning framework HYPO is motivated and guided by theory (Section 3.1), and provably guaranteed to reduce OOD generalization error (Thm 3.1 & Thm 5.1). Indeed, there may exist several alternative proxy-based learning algorithms such as PCL. However, it remains unclear if PCL's performance can be rigorously interpreted and bounded. We believe that our theoretical framework takes an important first step towards understanding domain generalization via hyperspherical learning formally, paving the way for more effective and theoretically grounded approaches in the field.

Empirically, unlike PCL which relies on SWAD optimization, HYPO is able to achieve strong performance by simple SGD optimization and the performance can be further improved with SWAD. Compared to PCL, HYPO achieves superior performance of an average accuracy of 89% on PACS, further demonstrating its effectiveness in practice.

Reviewer comment (Theory):
Theorem 5.1 does not account for the influence of inter-class separation, a key aspect that this paper seeks to enhance through the second term in loss eq. (5). I notice that in Ye et al (2021)'s Theorem 4.1, function O(.) also depends on additional factors beyond just the variation.

Author response:
We address the effect of inter-class separation in our theoretical analysis presented in **Appendix J**, which links to the second term in our loss eq. (5). Additionally, for the discussion on regularity conditions in Thm 4.1, please see our response to Reviewer W1ps.

Reviewer comment (Theory):
Since prototypes $\mu_i$ are updated in an EMA manner, it's worth noting that the second term in eq. (5) will not generate a gradient for $h$. Consequently, the second term of the loss becomes devoid of meaning.

Author response:
The separation term's primary purpose is to maintain distinctiveness between class prototypes in the embedding space. While the separation term may not directly affect the gradients of $\mathbf{z}$, it indirectly influences how $\mathbf{z}$ is learned. By keeping the prototypes separate, the variation term has to adapt $\mathbf{z}$ in a way that aligns with these well-separated prototypes, leading to better OOD generalization. As detailed in Algorithm 1 in Appendix A, in this work, we choose to update the prototypes $\boldsymbol{\mu}_i$ via EMA, which leads to stable updates of prototypes. Alternatively, one can set class prototypes as learnable parameters and update prototypes via gradients. We compare the performance of our method with EMA update and learnable prototypes (LP) in Appendix H (P24). We observe our method with EMA achieves superior performance compared to LP, which empirically verifies the effectiveness of EMA-style update of prototypes in practice.

Reviewer comment (Novelty):
The idea is quite straightforward and shares many similarities with proxy-based contrastive learning methods. Is there any additional insight that I might have overlooked?

Author response:
We provide a comparison in the paragraph *Relation to PCL [1]* on page 7. Here we would like to highlight notable distinctions and provide further clarifications:

- [**Theoretical grounding**] PCL offers no theoretical insights, while our loss HYPO is guided by theory. We provide a formal theoretical justification that our method reduces intra-class variation which is essential to bounding OOD generalization error (Section 5). 
- [**Statistical interpretation via vMF**] While PCL and our loss share high-level intuitions, our loss formulation can be rigorously interpreted as shaping vMF distributions of hyperspherical embeddings (Section 4.2). In contrast, it remains unclear if PCL can be statistically interpreted.
- [**Inter-class separation**] HYPO explicitly promotes inter-class separation via the prototype-to-prototype loss, whereas PCL does not. This loss is directly motivated by and grounded in the theoretical framework by Ye et al. 
- [**Strong performance w/o specialized optimization technique SWAD**] Unlike PCL which relies on SWAD, a dense and overfit-aware stochastic weight sampling strategy for OOD generalization, HYPO is able to achieve strong performance by simple SGD optimization (Avg Acc 88.0\% without SWAD) and the performance can be further improved with SWAD.

Reviewer comment (Evaluation):
The empirical improvements appear to be marginal, as indicated by the data in Tables 1 and 2.

Author response:
We appreciate Reviewer kMBq's scrutiny of our empirical results. We believe, however, that the improvements we demonstrate are both statistically significant and practically meaningful, particularly when considering the challenging nature of the OOD generalization problem and the competitive landscape of the field. As Reviewer W1ps commended, our empirical evaluation is extensive, where HYPO consistently outperforms a wide range of baseline methods on challenging OOD benchmarks. In particular, HYPO demonstrates an overall 1.2% improvement on average (Table 1), which is substantial compared to the relative improvement of competitive baselines. This achievement is also underscored by the 89% accuracy (w. SWAD) reported in Table 2, reflecting a notable advance in a highly competitive area. 

In light of HYPO's consistently demonstrated superiority, its straightforward and efficient implementation, and theoretical foundations, we believe that HYPO represents a notable advancement in the field.

Reviewer comment (Theory):
The theoretical result shown gives a bound on the intra-class variation. This is a useful component of producing an OOD generalization bound, but it is not sufficient by itself. The results in Ye et al. require some regularity conditions that depend on the distribution over the learned representations --- this is difficult to compute in this case. From my point of view, the theoretical results in this paper provide a strong intuition for the success of the method but have not yet been demonstrated to produce a tractable OOD generalization bound.

Author response:
> *C1. Clarification on the significance of Thm 5.1*

We would like to clarify that in Thm 5.1, the impact of sample-to-prototype alignment on the intra-class variation can be explicitly characterized via the first term $ε^{1\over 3}$ in the upper bound.  In particular, $ε^{1\over 3}$ measures how sample embeddings are aligned with their class prototypes on the hyperspherical space (as we have $\frac{1}{N} \sum_{j=1}^N \boldsymbol{\mu}_{c(j)}^{\top} \mathbf{z}_j \geq 1-\epsilon$), which is optimized by our proposed loss.  This Theorem implies that improved alignment (an illustration is shown in Figure 4(b)) leads to smaller upper bound of the intra-class variation $\mathcal{V}^{\text{sup}}$, a key term to upper bound the OOD generalization error by Thm 3.1. We provide empirical verifications below which further verify that the core assumption in our theoretical framework is satisfied in practice via our loss.

Reviewer comment (Theory):
How strong is the assumption that the samples are aligned? Intuitively, the intra-class variation measures how much the features vary across environments for a single class. The alignment assumption is an assumption over all of the training data in the available environments. Consequently, Theorem 5.1 consists of a term that depends on epsilon, and a generalization term that (intuitively) describes generalization to the unavailable environments. I think it would be more valuable if one could show that the alignment assumption is satisfied by reducing the training loss directly, bringing the result more in line with typical PAC generalization bounds.

Author response:
> *C1. Clarification on the significance of Thm 5.1*

We would like to clarify that in Thm 5.1, the impact of sample-to-prototype alignment on the intra-class variation can be explicitly characterized via the first term $ε^{1\over 3}$ in the upper bound.  In particular, $ε^{1\over 3}$ measures how sample embeddings are aligned with their class prototypes on the hyperspherical space (as we have $\frac{1}{N} \sum_{j=1}^N \boldsymbol{\mu}_{c(j)}^{\top} \mathbf{z}_j \geq 1-\epsilon$), which is optimized by our proposed loss.  This Theorem implies that improved alignment (an illustration is shown in Figure 4(b)) leads to smaller upper bound of the intra-class variation $\mathcal{V}^{\text{sup}}$, a key term to upper bound the OOD generalization error by Thm 3.1. We provide empirical verifications below which further verify that the core assumption in our theoretical framework is satisfied in practice via our loss.

Reviewer comment (Theory):
Ye et al. provide a specialized result for linear models (Theorem 4.2 in their work). I see this as a justification that the theoretical framework can be realized by some model. However, in the present work, it is unclear whether the vMF distribution can satisfy the regularity conditions for some choice of environment distribution(s). In other words, how do we know that the OOD generalization bounds can actually be computed for the choice of model used?

Author response:
> *Q3. Ye et al. provide a specialized result for linear models (Theorem 4.2 in their work). I see this as a justification that the theoretical framework can be realized by some model...How do we know that the OOD generalization bounds can actually be computed for the choice of model used?*

Great question! As specified in P9, we perform classification by identifying the closest class prototype: $\hat y = \text{argmax}_{c \in[C]} f_c$ where $f_c= \mathbf{z}^\top\boldsymbol{\mu}_c$ for embedding $\mathbf{z}$.  This is equivalent to the linear top model as in Ye et al. (without the bias term), as we can reformulate as $f_c= (W\mathbf{z})_c$, where the $c$-th row of $W$ is $\boldsymbol{\mu}_c$. In this case, we have a linear convergence rate without the regularity conditions in Thm 4.1 (as shown in P7 in Ye et al.). In a more general case, the generalization bound in Thm 4.1 (pasted below for easy reference) depends on additional factors
$$\operatorname{err}(f) \leq O\left(s\left(\mathcal{V}\_\rho^{\text {sup }}\left(h, \mathcal{E}\_{\text {avail }}\right)\right)^{\frac{\alpha^2}{(\alpha+d)^2}}\right) $$

such as $\alpha$ and $d$, where $d$ denotes the dimension of feature embeddings, and $\alpha$ is closely correlated to the density concentration. Such factors mainly affect the convergence rate of the error upper bound. The Theorem shows that, for any model, the generalization gap depends largely on the model’s variation captured by $\mathcal{V}^{\text {sup}}$. As HYPO aims to reduce the intra-class variation (verified in the previous response), it effectively reduces the OOD generalization bound.

Reviewer comment (Novelty):
[2] is another reference that explores a contrastive metric learning approach for hyperspherical embeddings. The goal here is not to do OOD generalization, but the algorithm is modestly similar.

Author response:
> *C1. Typos and citations* 

Thanks for pointing out! We have fixed the typo in proof of Thm 5.1 from "at last" to "at least" and have added citation for [2] in the updated version.

Reviewer comment (Presentation):
It would be nice if Table 1 were sorted by ascending average accuracy.

Author response:
> *C3. It would be nice if Table 1 were sorted by ascending average accuracy*

Thanks for the suggestion! We have adjusted the order of algorithms in Table 1 for clarity and ease of comparison.

Reviewer comment (Theory):
I struggle to see how Theorem 5.1 connects back to the proposed loss function. From Theorem 3.1 we know that $\nu^{\text{sup}}$ serves as an upper bound to the OOD error, and then Theorem 5.1 in-turn provides an upper bound for $\nu^{\text{sup}}$ in terms of the Rademacher complexity and some additive constants. Which term here is the loss trying to minimise here? The Rademacher complexity is over any $\sigma_i$, so its sign has nothing to do with the true labels. I don’t see how the developed loss would encourage to minimise this quantity. It’s also a worst-case bound in terms of the hypothesis $h$, so again I don’t see how that could be minimised. I hope the authors can elaborate on this connection.

Author response:
> *Q1: How does Thm 5.1 connect back to the proposed loss function?*

In Thm 5.1, we can see that the upper bound consists of three factors: (1) the optimization error $ε^{1\over 3}$, (2) the Rademacher complexity of the given neural network, and (3) the estimation error which goes down as the number of samples $N$ increases. While (2) and (3) are standard, the critical term, (1) $ε^{1\over 3}$, measures how sample embeddings are aligned with their class prototypes on the hyperspherical space (as we have $\frac{1}{N} \sum_{j=1}^N \boldsymbol{\mu}_{c(j)}^{\top} \mathbf{z}_j \geq 1-\epsilon$), which is optimized by our proposed loss. 

Our Theorem implies that improved alignment (as illustrated in Figure 4(b)) leads to smaller upper bound of the intra-class variation $\mathcal{V}^{\text{sup}}$, a key term to upper bound the OOD generalization error by Thm 3.1.

Reviewer comment (Experiments):
The CIFAR10-C results look strong but only naive ERM is provided as a baseline. How does the approach fair against more specialised algorithms. I don’t expect this novel approach to be state-of-the-art but it would be nice to know where it stands among more modern algorithms.

Author response:
> *Q2: How does our loss compare with more specialized algorithms on CIFAR10-C?*  

Thanks for the suggestion! Here we compare our loss (HYPO) with very recent algorithms: EQRM (NeurIPS'22) [3] and SharpDRO (SOTA from CVPR'23) [4], on the CIFAR10-C dataset (Gaussian noise). All three methods are trained on CIFAR-10 using ResNet-18, consistent with the architecture used in our main paper. The results are presented below. 

| Method | OOD Acc. (%) |
|-------|-------|
| EQRM       | 77.06 |
| SharpDRO   | 81.61 |
| HYPO (ours)| 85.21 |

> Q2. How does PCL perform?

We train with PCL loss on CIFAR-10 and evaluate on CIFAR-10-C (same test dataset as ours), the OOD accuracy is 80.68\%. 

| Method | OOD Acc. (%) |
|-------|-------|
| EQRM       | 77.06 |
| SharpDRO   | 81.61 |
| PCL        | 80.68 |
| HYPO (ours)| **85.21** |

Reviewer comment (Experiments):
While the authors do compare against [1], I think the paper would benefit from a more in-depth comparison of the two losses. I’m also a bit confused as to why the results of [1] are not reported in Table 1 but only in a separate ablation in Table 2. Could the authors clarify this?

Author response:
> *Q3: A more in-depth comparison with the PCL loss*

We provide a comparison in the paragraph *Relation to PCL [1]* on page 7. Here we would like to highlight notable distinctions and provide further clarifications:
- [**Theoretical grounding**] PCL offers no theoretical insights, while our loss HYPO is guided by theory. We provide a formal theoretical justification that our method reduces intra-class variation which is essential to bounding OOD generalization error (Section 5). 
- [**Statistical interpretation via vMF**] While PCL and our loss share high-level intuitions, our loss formulation can be rigorously interpreted as shaping vMF distributions of hyperspherical embeddings (Section 4.2). In contrast, it remains unclear if PCL can be statistically interpreted.
- [**Strong performance w/o specialized optimization technique SWAD**] Unlike PCL which relies on SWAD [2], a dense and overfit-aware stochastic weight sampling strategy for OOD generalization, HYPO is able to achieve strong performance by simple SGD optimization (Avg Acc 88.0\% without SWAD) and the performance can be further improved with SWAD.

> *Q4: Why the comparison of HYPO vs. PCL is in not included in Table 1?*

Table 1 intends to contrast the effect of different loss functions **under the standard SGD optimization**. We exclude PCL from Table 1 since the performance reported in the original PCL paper (Table 3) is implicitly based on SWAD. 

Due to the relevance of two losses and the choice of optimization strategies, we believe it is worthwhile to provide a detailed comparison of HYPO vs. PCL under four optimization strategies: PCL w/o SWAD, HYPO w/o SWAD, PCL w/ SWAD, and PCL w/ SWAD. Therefore, we summarize the results in a separate table (Table 2), as opposed to including them in Table 1 for better clarity. As shown in the table, compared to PCL, HYPO achieves superior performance with (Avg Acc 88.7\% $\rightarrow$ 89\%) and without SWAD (Avg Acc 86.3\% $\rightarrow$ 88\%), which further demonstrates its effectiveness and generality. 

Table 2 in the main paper is pasted here for easy reference:

| Model | Art painting | Cartoon | Photo | Sketch | Average Acc. (%)|
|-----|-----|-----|-----|-----|-----|
| PCL w/o SWAD  | 88.0 | 78.8 | 98.1 | 80.3 | 86.3 |
| HYPO w/o SWAD (Ours) | 87.2 | 82.3 | 98.0 | 84.5 | 88.0 |
| PCL w/ SWAD   | 90.2 | 83.9 | 98.1 | 82.6 | 88.7 |
| HYPO w/ SWAD (Ours) | 90.5 | 84.6 | 97.7 | 83.2 | 89.0 |

Reviewer comment (Theory):
I think Equation (5) has some typos, shouldn’t the embedding $z_i$ also depend on the environment $e$, i.e. $z_i^e$? If one interprets this equation “literally” you would be summing over the same $z_i$ over and over again.

Author response:
> *Q5: Typos*

Thanks for pointing out! We have fixed this typo in the updated manuscript by adding a superscript that explicitly denotes the environment ($\mathbf{z}_i → \mathbf{z}_i^e$) to avoid potential misunderstanding.