Paper: Extending to New Domains without Visual and Textual Oracles (ICLR-2024)

Reviewer comment (Novelty):
A noteworthy concern arises from the marginal improvement over the most competitive baseline, LADS, as evidenced by Table 1 (0.78% improvement on CUB, 0.92% improvement on DomainNet). This minor improvement raises doubts about the merit and significance of the proposed method, especially considering the striking similarities to existing work, such as CGAP. The paper's originality and contribution require further clarification and validation to convincingly demonstrate its value in the domain of single-domain generalization.

Author response:
## Q2. Results

As we mentioned in the first paragraph in Section 5.4, Tab. 1 shows the results under the LADS setting, where exact test domain names are given and LADS is expected to perform well. Tab. 2 is our setting and the interesting part, where exact test domain names are not given. Without exact test domains, the performance gap between LADS and ours is more obvious: **2.86% (OOD)** and **2.17% (AVG)** in CUB-Paintings, and **2.15% (OOD)** and **2.05% (AVG) in** DomainNet. Note that we achieve better results without any augmentation networks nor training during stage-1, while LADS requires training an augmentation network for each given domain prompt. We are significantly more efficient. Besides, our method requires less time in the stage-2 training (four times faster with “invar.” mode in tab. 2). Our model is also more robust across different sets of domain prompts (Appendix E and F show more experiments and details), while LADS only works well when the exact test domain names are given, which is impractical. *We kindly remind that, as mentioned in the paper, out-of-domain (OOD) performance is our major metric as we focus on the domain generalization task, where test domain accuracy is the major metric, like existing works [1,5,6]. To summarize, our model achieves obvious results on our practical setting (Tab. 2) over computing baselines (their performances are very close to each other while we achieve a more obvious improvement)* **without** *training during stage-1 and less time during stage-2*  *with “invar.” mode:*

| Method  | OOD (CUB-Paintings) | OOD (DomainNet) | Average (CUB-Paintings) | Average (DomainNet) |
| ------- | ------------------- | --------------- | ----------------------- | ------------------- |
| WiSE-LP | 64.80               | 93.68           | 73.27                   | 94.44               |
| LADS    | 64.85               | 94.65           | 74.99                   | 94.97               |
| Ours    | **67.71**           | **96.70**       | **77.16**               | **96.18**           |

Efficiency is also one of our key improvements.

**Reference**

[1] Vidit, Vidit, Martin Engilberge, and Mathieu Salzmann. "CLIP the Gap: A Single Domain Generalization Approach for Object Detection." CVPR, 2023.

[2] Liang et al., Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. NeurIPS 2022

[3] Patashnik et al., Styleclip: Text-driven manipulation of stylegan imagery, ICCV 2021

[4] Gal et al., “Stylegan-nada: Clip-guided domain adaptation of image generators,” ACM Transactions on Graphics (TOG)

[5] Li et al., A Simple Feature Augmentation for Domain Generalization, ICCV 2021

[6] Lisa et al., Using Language to Extend to Unseen Domains, ICLR 2023

Reviewer comment (Theory):
In **Training-free Augmentation with Modality Direction** (on page 5) the proof of z exists in the CLIP's normalized embedding space is trivial to get.

Author response:
We agree that obtaining a solution for $z$ is not hard mathematically. 
Actually, our intention is not to showcase our skills of solving the equations, but rather to explore the conditions under which a valid solution is available in the normalized embedding space (i.e., training-free feasibility) (Eq. 17 in Appendix B), and its relationship with the pre-trained vision-language (VL) model (Lemma 1).
We are more interested in $\lambda_{1}$ in Eq. 17 (Appendix B), which determines if the solved $z$ is a valid solution in the embedding space.
The exploration is critical because it theoretically demonstrates that, if aligning with the global direction (Eq. 21), the condition for the existence of a valid $z$ is more stringent, requiring the cosine similarity of any negative image-text pair is always less than that of the positive pair. This implies that the pre-trained VL model should be perfect (Appendix C). On the other hand, if aligning with our modality direction, the condition for the existence of $z$ is more relaxed: once the embeddings are clustered per modality in the VL model’s embedding space, a solution is readily available. Fortunately, (Liang et al.) [1] validate that the condition is generally satisfied with a VL model trained with a contrastive loss.
Our ablation studies in Tab. 3 also validate this point: aligning modality direction proves to be more effective:

| Augmentation Method                 | Average   | ID        | OOD       |
| ----------------------------------- | --------- | --------- | --------- |
| LADS                                | 74.99     | 85.33     | 64.85     |
| Training-free + Align Global Dir.   | 74.67     | 85.21     | 64.12     |
| Training-free + Align Modality Dir. | **77.16** | **86.61** | **67.71** |

Reviewer comment (Evaluation):
In Table 1 and Table 2, the performance of the proposed method on CUB and DomainNet does not stand out from the baselines with a significant margin.

Author response:
Tab. 1 presents our results under the LADS setting, where the **exact** test domain names are given. The main results are presented in Tab. 2 in our text-driven domain generalization setting, which is of particular interest to us, where exact test domain names are not given. 

It is important to note that it is not an easy task. Under our setting, most competing CLIP-based baselines including WiSE-LP and LADS exhibit very close performance, making further improvements challenging. However, we continue to distinguish ourselves from them (Tab. 2) in AVG and OOD scores:

| Method  | OOD (CUB-Paintings) | OOD (DomainNet) | Average (CUB-Paintings) | Average (DomainNet) |
| ------- | ------------------- | --------------- | ----------------------- | ------------------- |
| WiSE-LP | 64.80               | 93.68           | 73.27                   | 94.44               |
| LADS    | 64.85               | 94.65           | 74.99                   | 94.97               |
| Ours    | **67.71**           | **96.70**       | **77.16**               | **96.18**           |

Notably, this distinction is achieved **without** the need for any parametric augmentation functions and **without** any training in the first phase. Our work contributes significantly to time efficiency, a crucial aspect beyond accuracy improvement. *We kindly remind that, as mentioned in the paper, out-of-domain (OOD) performance is our major metric as we focus on the domain generalization task, where test domain accuracy is the major metric, like existing works [2,3,4]. To summarize, our model achieves obvious results on our practical setting (Tab. 2) over computing baselines (their performances are very close to each other while we achieve a more obvious improvement)* **without** *training during stage-1 and less time during stage-2*  *with “invar.” mode.

Reviewer comment (Experiments):
Ablation Studies are limited.

Author response:
We have multiple ablation studies on: (1) augmentation modes and (2) invariant modes in Tab. 3 to show the effectiveness of our augmentation method. We also have ablations on choices of backbone vision-language models in Tab. 11, and on different sources of prompt templates in Tab. 10 and Tab. 11, where we use pre-defined templates such as ImageNet Templates, as well as prompts from different LLMs (e.g., ChatGPT, CPT4, New Bing, and Bard), to demonstrate our robustness across different sets of prompts. 

Our existing ablation studies answer most interesting questions. If you believe there is any particular study that we’re missing, we’re more than happy to include.

We add one table regarding your two questions below. We train our model on DomainNet with: ① a mix of augmented data and original data. ② augmented data. ③ original data.


| Model                            | Average   | ID        | OOD       |
| -------------------------------- | --------- | --------- | --------- |
| ① original data + augmented data | **96.18** | **95.61** | **96.70** |
| ② augmented data only            | 95.60     | 95.21     | 95.99     |
| ③ original data only             | 94.39     | 95.03     | 93.75     |

We will discuss it in detail in the response to your questions below.

Reviewer comment (Theory):
Why can the linear probe trained on the domain-invariant feature and original feature be directly applied to the target feature (without any augmentation)?

Author response:
The two questions can be answered by a single statement: The two questions you mentioned are in fact standard pipelines for data augmentation based domain generalization methods [2,3,4], which are also adopted by LADS [4]. As our TEAM model is also a data augmentation based method, we simply follow existing works.

We explain it in detail below. Let's quickly recap our methodology first: We have two model versions: TEAM-full and TEAM-invariant. In TEAM-full, $k$ crafted domain names are provided. We utilize a training-free method to obtain augmented features for $k$ provided domains, which are then **mixed** with the source domain for training on (k+1) domains during stage-2. The evaluation is conducted directly on test domains images without prior image augmentation to a specific domain. **This follows the standard approach in data augmentation based domain generalization (DG) methods [2,3,4], which LADS also follows:** 

**(Point. 1 / Q1) Trained model is applied to test domain data without any augmentation.** 

**(Point. 2 / Q2) Augmented data is mixed with original data for training.**

In the "invariant" version, augmentation is not performed for $k$ domains. Instead, augmentation is performed for a single "invariant" domain. **The key is, it is still a data augmentation method**, which means the obtained invariant embeddings are still “augmented” data, just like any augmented data for $k$ domains in TEAM-full. The difference from TEAM-full is that, here we set $k$ = 1 (i.e., our so-called "invariant domain").

Therefore, the training and evaluation process are the same with TEAM-full and LADS, i.e., the two bolded points we mentioned above.

In addition, we have done ablation studies on DomainNet regarding your question: we train our model on DomainNet with: ① a mix of augmented data and original data. ② augmented data only. ③ original data only.

| Model                            | Average   | ID        | OOD       |
| -------------------------------- | --------- | --------- | --------- |
| ① original data + augmented data | **96.18** | **95.61** | **96.70** |
| ② augmented data only            | 95.60     | 95.21     | 95.99     |
| ③ original data only             | 94.39     | 95.03     | 93.75     |

Reviewer comment (Experiments):
In Stage 2, why not only train the classifier only on the augmented embeddings?

Author response:
The two questions can be answered by a single statement: The two questions you mentioned are in fact standard pipelines for data augmentation based domain generalization methods [2,3,4], which are also adopted by LADS [4]. As our TEAM model is also a data augmentation based method, we simply follow existing works.

We explain it in detail below. Let's quickly recap our methodology first: We have two model versions: TEAM-full and TEAM-invariant. In TEAM-full, $k$ crafted domain names are provided. We utilize a training-free method to obtain augmented features for $k$ provided domains, which are then **mixed** with the source domain for training on (k+1) domains during stage-2. The evaluation is conducted directly on test domains images without prior image augmentation to a specific domain. **This follows the standard approach in data augmentation based domain generalization (DG) methods [2,3,4], which LADS also follows:** 

**(Point. 1 / Q1) Trained model is applied to test domain data without any augmentation.** 

**(Point. 2 / Q2) Augmented data is mixed with original data for training.**

In the "invariant" version, augmentation is not performed for $k$ domains. Instead, augmentation is performed for a single "invariant" domain. **The key is, it is still a data augmentation method**, which means the obtained invariant embeddings are still “augmented” data, just like any augmented data for $k$ domains in TEAM-full. The difference from TEAM-full is that, here we set $k$ = 1 (i.e., our so-called "invariant domain").

Therefore, the training and evaluation process are the same with TEAM-full and LADS, i.e., the two bolded points we mentioned above.

In addition, we have done ablation studies on DomainNet regarding your question: we train our model on DomainNet with: ① a mix of augmented data and original data. ② augmented data only. ③ original data only.

| Model                            | Average   | ID        | OOD       |
| -------------------------------- | --------- | --------- | --------- |
| ① original data + augmented data | **96.18** | **95.61** | **96.70** |
| ② augmented data only            | 95.60     | 95.21     | 95.99     |
| ③ original data only             | 94.39     | 95.03     | 93.75     |

Reviewer comment (Experiments):
The authors might consider evaluating their proposed method on a larger dataset, such as WILDS, which provides extensive and diverse real-world data from various domains.

Author response:
Thanks for this insightful suggestion, which points out a critical future direction for this line of research. As we also discussed in Section 6: Limitations, WILDS (Koh et al.) remains a challenge. As our text-driven augmentation requires text descriptions of target domains, it is particularly challenging for complicated natural domain shifts that are hard to capture and express solely through language. We are interested in this problem and will continue to explore it.

Reviewer comment (Presentation):
Could you please explain the meaning of "1x" for Time in several tables?

Author response:
Sure! Sorry for the confusion of our symbols. "$n$ x" indicates that the stage-2 training time of a particular model is $n$ times that of the LADS baseline. In other words, we consider the training time of LADS as the reference unit time.

For instance, in the following example: 

| Model         | Average | ID    | OOD   | Training-free (Stage-1) | Time (Stage-2) |
| ------------- | ------- | ----- | ----- | ----------------------- | -------------- |
| LADS          | 74.99   | 85.33 | 64.85 | ✗                       | 1x             |
| TEAM-*invar.* | 74.67   | 85.21 | 64.12 | ✓                       | 0.23x          |

When the model is TEAM-*invar.*, we have $0.23$x because the training time of TEAM-*invar.* is 0.23 times that of LADS. Therefore, when the model is LADS, we have $1$x because the training time of LADS is the same as that of itself.

Reviewer comment (Theory):
While the authors have successfully developed a method based on the findings of Liang et al. and Zhang et al., the paper itself lacks a sufficient explanation of the embedding space in the context of domain generalization. Although this might not be a critical flaw in the paper, providing theoretical or empirical analyses in this regard could enhance its overall impact and inspiration.

Author response:
Thanks for the valuable advice. We are also interested to see the geometry of clip embeddings space from the perspective of the training-free augmentation in our domain generalization context. More specifically, we would like to see if the assumptions in Appendix A hold well. We show the following modality gap geometry metrics. 

The distance between the modality clusters in the CLIP embedding space is referred to as the modality gap. The instance-level modality gap $g$ is defined as the difference between image and text embeddings for a single pair. The class-level modality gap is defined as $g_{c}$, where for a given class $c$, we calculate the expectation of $g$ over all instances of class $c$ **across different domains**. Finally, we calculate the domain-level modality gap $g_{d}$, where given a specific domain, we calculate the expectation of $g$ over all instances of domain $d$ **across different classes**.

Formally, given an instance pair ($x$, $y$), $g = x - y$, where $x$ and $y$ are CLIP features of an image and a text. We also have:

$g_{c} = x_c - y_c$, where $x_c = E_d[ x_c^d ]$ and  $y_c = E_d[ y_c^d ]$.  Note that $E_d[ x_c^d ]$ means the expectation of all instances of a given class $c$ across all **domains**. 

and 

$g_{d} = x_d - y_d$, where $x_d = E_c[ x_d^c ]$ and  $y_d = E_c[ y_d^c ]$.  Note that $E_c[ x_d^c ]$ means the expectation of all instances of a given domain $d$ across all **classes**. 

We hope that the modality gap $g_{c} $ grants the assumptions we have in Appendix A. Intuitively, if satisfied, given data from one source domain, we can perform our training-free augmentation as  $g_{c} $ is valid across different domains.

More specifically, the two assumptions are:

1. *The modality gap between corresponding image and text embeddings can be approximated by a constant vector, particularly given a class across different domains. This is verified by computing distributions over ||$g$|| (magnitude) and $cos(g, E_g[g])$ (direction).*

2. *The modality gap is approximately orthogonal to the span of image embeddings and text embeddings, and image embeddings and text embeddings have a nearly zero mean in the subspace orthogonal to the modality gap. This is verified by computing distributions over $cos(x - E_x[x], E_g[g])$ (orthogonality) and $E_x[x - x^T g’g’]_i$ (center), where $g’ = E_g[g] / || E_g[g] ||$ and $i \in [d]$. The subscript $i$ denotes indexing the $i$-th dimension of the vector.*

The results are listed:

| VL Model - Dataset   | Magnitude - *Instance* | Magnitude - *Class* | Magnitude - *Domain* | Direction - *Instance* | Direction - *Class* | Direction - *Domain* | Orthogonality     | Center           |
| -------------------- | ---------------------- | ------------------- | -------------------- | ---------------------- | ------------------- | -------------------- | ----------------- | ---------------- |
| CLIP - DomainNet     | 1.32 $\pm$ 0.034       | 1.19 $\pm$ 0.027    | 1.32 $\pm$ 0.029     | 0.76 $\pm$ 0.051       | 0.89 $\pm$ 0.023    | 0.77 $\pm$ 0.049     | 0.01 $\pm$  0.107 | 0.00 $\pm$ 0.024 |
| CLIP - CUB Paintings | 1.29 $\pm$ 0.035       | 1.18 $\pm$ 0.029    | 1.30 $\pm$ 0.038     | 0.70 $\pm$ 0.051       | 0.88 $\pm$ 0.029    | 0.74 $\pm$ 0.045     | 0.00 $\pm$  0.095 | 0.00 $\pm$ 0.026 |

We can observe that, the orthogonality and center are all close to 0 with small STD, which means assumption 2 is well satisfied. For magnitude and direction, the class-level modality gap (**across domains**) better satisfies assumption 1, as we have higher scores for direction on both datasets. It satisfies our expectation that, the class-level modality gap $g_{c}$ better grants the assumptions. It indicates that, given data from one source domain, we can perform our training-free augmentation because the modality gap $g_{c} $ of a given class $c$ is valid across different domains.

Reviewer comment (Experiments):
As the authors mentioned in Figure 2 (right), there might be cases where equation 4 is not satisfied. I am curious if the CLIP features always meet the conditions of equation 4. Could the authors conduct empirical analysis to offer an intuitive understanding of when it fails?

Author response:
It is an interesting question. Let's illustrate it with an example, where the source domain is "painting" and target domain is "photo". Given an image-text pair from source data, say, (apple_painting.jpg, "a painting of an apple"), and a text description from target domain: "a photo of an apple", intuitively, equation 4 means: in the CLIP embedding space, "a painting of an apple" is closer to  "a photo of an apple"  than to apple_painting.jpg. In other words, embeddings of the **same** modality should be closer to each other than embeddings of the same semantic meaning but different modality. This is what we expect.

Fortunately, Liang et al. reveal that, when a vision-language model is pre-trained with a contrastive loss, the embeddings are clustered per modality, which naturally guarantees what we expect. **And in practice, we find it always holds.** It is reasonable because the CLIP model is well-trained with 400M data, during which time the embeddings are very well clustered per modality. 

You may refer to Fig. 1b (https://arxiv.org/pdf/2203.02053.pdf). Intuitively, if there is a failure case, you could imagine a blue point appearing among the red points, which indicates a text embedding is closer to one or some image embeddings than other text embeddings. In our practice with the pre-trained CLIP model, it never happens.

Liang et al. further explain why they are so separated from each other from three aspects: cone effect, initializations, and contrastive learning objective.