Original: D. B RIDGE
Revised: E. B RIDGE

Original: In the bridge setting (see Appx.
Revised: In the bridge setting (see Appx.

Original: E.2), we aimed to bridge the gap between our theoretical setup to the practical setup.
Revised: F.2), we aimed to bridge the gap between our theoretical setup to the practical setup.

Original: The main differences from the scientiﬁc setups are that the empirical generalization gap is considered and the average-case source predictor is used, as detailed below:
Revised: The main differences from the scientiﬁc setups are that the empirical generalization gap is considered and the average-case source predictor is used, as detailed below:

Original: Incorporating empirical generalization In practice, empirical-generalization gap should also be considered besides the source-target generalization gap.
Revised: Incorporating empirical generalization In practice, empirical-generalization gap should also be considered besides the source-target generalization gap.

Original: Thus, we randomly split the PACS dataset to 80% training and 20% validation splits for each domain.
Revised: Thus, we randomly split the PACS dataset to 80% training and 20% validation splits for each domain.

Original: The training splits were used to train both the encoder and the source predictor, and the validation splits were used for encoder and source predictor selection as well as evaluation on target domains.
Revised: The training splits were used to train both the encoder and the source predictor, and the validation splits were used for encoder and source predictor selection as well as evaluation on target domains.

Original: We used the ResNet-50 model as the encoder and initialized it from ImageNet pretrained model.
Revised: We used the ResNet-50 model as the encoder and initialized it from ImageNet pretrained model.

Original: The encoder was trained to a maximum of 50 epochs with a 1e-5 weight decay, using the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 5e-5, a batch size of 112 (28 for each domain), and a cosine learning rate decay schedule.
Revised: The encoder was trained to a maximum of 50 epochs with a 1e-5 weight decay, using the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 5e-5, a batch size of 112 (28 for each domain), and a cosine learning rate decay schedule.

Original: Using average-case source predictor Instead of approximating the worst-case source predictor in the scientiﬁc setting, we considered the average-case 2 source predictor which is closer to the common practice.
Revised: Using average-case source predictor Instead of approximating the worst-case source predictor in the scientiﬁc setting, we considered the average-case 3 source predictor which is closer to the common practice.

Original: Speciﬁcally, we freezed the encoder and trained a SVM classiﬁer with L2 regularization on the source training split.
Revised: Speciﬁcally, we freezed the encoder and trained a SVM classiﬁer with L2 regularization on the source training split.

Original: The regularization parameter was tuned over {1e-4, 1e-3, 1e-2, 1e-1, 1, 1e1, 1e2, 1e3} with the source validation accuracy.
Revised: The regularization parameter was tuned over {1e-4, 1e-3, 1e-2, 1e-1, 1, 1e1, 1e2, 1e3} with the source validation accuracy.

Original: Next, we provide other experimental details for reproducibility:
Revised: Next, we provide other experimental details for reproducibility:

Original: Selection of λ
Revised: Selection of λ

Original: For all different setups considered in bridge settings, the CAD bottleneck was used and the λ was tuned over {1e-3, 1e-2, 1e-1, 1, 1e1} independently for each.
Revised: For all different setups considered in bridge settings, the CAD bottleneck was used and the λ was tuned over {1e-3, 1e-2, 1e-1, 1, 1e1} independently for each.

Original: D. D OMAIN B ED
Revised: E. D OMAIN B ED

Original: Datasets We used non-MNIST datasets on DomainBed that were non-synthetic, including VLCS (Fang et al., 2013), PACS (Li et al., 2017), OfﬁceHome (Venkateswara et al., 2017), TerraIncognita (Beery et al., 2018), and DomainNet (Peng et al., 2019).
Revised: Datasets We used non-MNIST datasets on DomainBed that were non-synthetic, including VLCS (Fang et al., 2013), PACS (Li et al., 2017), OfﬁceHome (Venkateswara et al., 2017), TerraIncognita (Beery et al., 2018), and DomainNet (Peng et al., 2019).

Original: For each dataset, we split it to 80%/20% training/validation set according to DomainBed.
Revised: For each dataset, we split it to 80%/20% training/validation set according to DomainBed.

Original: SSL-based models & Training For all models based on pretrained SSL models (either CLIP-based or DINO-based) with ﬁnetuning in this experiment
Revised: SSL-based models & Training For all models based on pretrained SSL models (either CLIP-based or DINO-based) with ﬁnetuning in this experiment

Original: , we freezed the pretrained SSL model and added on top a 1-layer MLP with hidden size 1024, and residual connection.
Revised: , we freezed the pretrained SSL model and added on top a 1-layer MLP with hidden size 1024, and residual connection.

Original: We used CLIP ResNet- (CLIP S) to obtain the best possible fair comparison with baselines from DomainBed, and CLIP
Revised: We used CLIP ResNet- (CLIP S) to obtain the best possible fair comparison with baselines from DomainBed, and CLIP

Original: ViT-B/32 (CLIP L) to achieve the best results.
Revised: ViT-B/32 (CLIP L) to achieve the best results.

Original: Note that the ResNet-50 model of CLIP S was modiﬁed as described in Radford et al.
Revised: Note that the ResNet-50 model of CLIP S was modiﬁed as described in Radford et al.

Original: and contained 38M parameters (more than 23M of the original CLIP).
Revised: and contained 38M parameters (more than 23M of the original CLIP).

Original: The model was trained to 300 epochs for DomainNet and 50 epochs on other datasets (an epoch is deﬁned as a single pass over the smallest domain according to DomainBed).
Revised: The model was trained to 300 epochs for DomainNet and 50 epochs on other datasets (an epoch is deﬁned as a single pass over the smallest domain according to DomainBed).

Original: No data augmentation was used and the temperature for scaling the logits in CAD was ﬁxed to 0.05.
Revised: No data augmentation was used and the temperature for scaling the logits in CAD was ﬁxed to 0.05.

Original: We used the Adam optimizer with a 1e-5 weight dacay, and a cosine learning rate decay schedule.
Revised: We used the Adam optimizer with a 1e-5 weight dacay, and a cosine learning rate decay schedule.

Original: The hyperparameter search space is:
Revised: The hyperparameter search space is:

Original: • Learning rate: discrete set {1e-4, 3e-4, 1e-3, 3e-3} • Batch size: discrete set {128, 256, 512} for DomainNet and OfﬁceHome, and {64, 128, 256} for other datasets • MLP dropout: discrete set {0., 0.1, 0.5} • Learning rate warmup: discrete set {True, False}
Revised: • Learning rate: discrete set {1e-4, 3e-4, 1e-3, 3e-3} • Batch size: discrete set {128, 256, 512} for DomainNet and OfﬁceHome, and {64, 128, 256} for other datasets • MLP dropout: discrete set {0., 0.1, 0.5} • Learning rate warmup: discrete set {True, False}

Original: End-to-end models & Training In Table 1, we also included an end-to-end trained model without any pretrained SSL models.
Revised: End-to-end models & Training In Table 1, we also included an end-to-end trained model without any pretrained SSL models.

Original: We used exactly the same model architecture (the original ResNet50, initialized from ImageNet pretrained model), training procedure and evaluation protocal as baselines on DomainBed.
Revised: We used exactly the same model architecture (the original ResNet50, initialized from ImageNet pretrained model), training procedure and evaluation protocal as baselines on DomainBed.

Original: Importantly, the linear classiﬁer was jointly trained with the encoder, and no reﬁtting was applied.
Revised: Importantly, the linear classiﬁer was jointly trained with the encoder, and no reﬁtting was applied.

Original: The model was trained to a maximum of 5000 steps on each dataset, and data augmentations were applied.
Revised: The model was trained to a maximum of 5000 steps on each dataset, and data augmentations were applied.

Original: The Adam optimizer was used without any particular learning rate schedule.
Revised: The Adam optimizer was used without any particular learning rate schedule.

Original: The hyperparameter search space is (same as DomainBed except we added the temperature):
Revised: The hyperparameter search space is (same as DomainBed except we added the temperature):

Original: • Learning rate: log-uniform over [1e-5, 1e-3.5] • Batch size: log-uniform over [8, 64] for DomainNet, and [8, 2 5 . 5 ] for other datasets • MLP dropout: discrete set {0., 0.1, 0.5} • Weight decay: log-uniform over [1e-6, 1e-2] • Temperature: discrete set {0.05, 0.1}
Revised: • Learning rate: log-uniform over [1e-5, 1e-3.5] • Batch size: log-uniform over [8, 64] for DomainNet, and [8, 2 5 . 5 ] for other datasets • MLP dropout: discrete set {0., 0.1, 0.5} • Weight decay: log-uniform over [1e-6, 1e-2] • Temperature: discrete set {0.05, 0.1}

Original: Linear Probe Evaluation
Revised: Linear Probe Evaluation

Original: In all the experiments except for the end-to-end training setup, we always followed the procedure of two-stage training, where we ﬁrst trained the encoder with speciﬁed objectives, and then reﬁt the classiﬁer.
Revised: In all the experiments except for the end-to-end training setup, we always followed the procedure of two-stage training, where we ﬁrst trained the encoder with speciﬁed objectives, and then reﬁt the classiﬁer.

Original: For datasets except DomainNet, we ﬁtted the SVM classiﬁer and tuned the regularization parameter over {1e-4, 1e-3, 1e-2, 1e-1, 1, 1e1, 1e2, 1e3} with source validation selection.
Revised: For datasets except DomainNet, we ﬁtted the SVM classiﬁer and tuned the regularization parameter over {1e-4, 1e-3, 1e-2, 1e-1, 1, 1e1, 1e2, 1e3} with source validation selection.

Original: Since DomainNet was too large and SVM cannot ﬁt it efﬁciently, we used the logistic regression classiﬁer which was trained with a batch size 512, the Adam optimizer with a learning rate 5e-4 and early stopping.
Revised: Since DomainNet was too large and SVM cannot ﬁt it efﬁciently, we used the logistic regression classiﬁer which was trained with a batch size 512, the Adam optimizer with a learning rate 5e-4 and early stopping.

Original: Note that an alternative was to just use the linear head ﬁtted when training the representor (as we used CE loss with source labels), and we found this could work better than reﬁtting since the classiﬁer was less overﬁtted to the source domain.
Revised: Note that an alternative was to just use the linear head ﬁtted when training the representor (as we used CE loss with source labels), and we found this could work better than reﬁtting since the classiﬁer was less overﬁtted to the source domain.

Original: However, we didn’t do that since we wanted to stick to the representation learning protocol with two-stage training.
Revised: However, we didn’t do that since we wanted to stick to the representation learning protocol with two-stage training.

Original: We did that in our end-to-end training setup since we wanted it to be compeletely comparable to baselines on DomainBed (which did not do reﬁtting).
Revised: We did that in our end-to-end training setup since we wanted it to be compeletely comparable to baselines on DomainBed (which did not do reﬁtting).

Original: Selection of λ
Revised: Selection of λ

Original: In our experiments, we treated λ as a special hyperparamter.
Revised: In our experiments, we treated λ as a special hyperparamter.

Original: For each model, weselected λ on the PACS dataset, and then used the same λ value for all other datasets, because our bottleneck was fairly robust to the choice of λ . 
Revised: For each model, we used the same λ selected on PACS on all datasets except DomainNet, because our bottleneck is fairly robust to the choice of λ . For the large-scale DomainNet dataset, we selected its λ individually.

Original: The λ values chosen for SSL-based models (i.e., CLIP
Revised: The λ values chosen for each model were:

Original: S, CLIP L, DINO) and end-to-end ResNet-50 were 1e-1  and 1e-5, respectively.
Revised: • CLIP S: 1 on DomainNet and 1e-2 on other datasets • CLIP L: 1e-1 on DomainNet and 1e-2 on other datasets • DINO: 1e-1 on all datasets • End-to-end ResNet-50: 1e-5 on all datasets

Original: D. LAION
Revised: E. LAION

Original: Model We used the CLIP L model (i.e., CLIP ViT-B/32) with an additional network on top for ﬁnetuning.
Revised: Model We used the CLIP L model (i.e., CLIP ViT-B/32) with an additional network on top for ﬁnetuning.

Original: The additional network were two blocks of 2-layer MLP, each with hidden size 2048, pre-activation batch normalization, residual connection, and dropout probability 0.1.
Revised: The additional network were two blocks of 2-layer MLP, each with hidden size 2048, pre-activation batch normalization, residual connection, and dropout probability 0.1.

Original: Note that the original CLIP L model was frozen and only the additional network was trained.
Revised: Note that the original CLIP L model was frozen and only the additional network was trained.

Original: Dataset We used the LAION-400M dataset which contained 400 million image-text pairs for training.
Revised: Dataset We used the LAION-400M dataset which contained 400 million image-text pairs for training.

Original: Though the dataset might not be as clean as the original CLIP training data (as evidenced by our experimental results), it was the largest publicly available image-text-pair dataset that we could get access to.
Revised: Though the dataset might not be as clean as the original CLIP training data (as evidenced by our experimental results), it was the largest publicly available image-text-pair dataset that we could get access to.

Original: As we froze the CLIP L model and only did ﬁnetuning for  model.
Revised: As we froze the CLIP L model and only did ﬁnetuning, we used the 1TB preprocessed embeddings provided by LAION-400M 4 . No further preprocessing was applied. Training We used the image-text contrastive loss as introduced in Radford et al. (2021) for training model.

Original: The temperature was learnable which was initialized as 0.07 and clipped with a minimum 0.01.
Revised: The temperature was learnable which was initialized as 0.07 and clipped with a minimum 0.01.

Original: The model was trained for 1 epoch using the Adam optimizer with a batch size of 16384 and a cosine learning rate decay schedule.
Revised: The model was trained for 1 epoch using the Adam optimizer with a batch size of 16384 and a cosine learning rate decay schedule.

Original: The learning rate was tuned over the set {3e-5, 1e-4, 3e-4, 1e-3, 3e-3, 1e-2} and the λ value for the Ent bottleneck was tuned over {1e-3, 1e-2, 1e-1, 1, 1e1}.
Revised: The learning rate was tuned over the set {3e-5, 1e-4, 3e-4, 1e-3, 3e-3, 1e-2} and the λ value for the Ent bottleneck was tuned over {1e-3, 1e-2, 1e-1, 1, 1e1}.

Original: Evaluation For the evaluation on the ImageNet-related datasets, we followed a similar procedure in
Revised: Evaluation For the evaluation on the ImageNet-related datasets, we followed a similar procedure in

Original: Radford et al.
Revised: Radford et al.

Original: (2021), where a linear classiﬁer was ﬁtted on ImageNet using the model representations and evaluated on 7 natural distribution shift datasets.
Revised: (2021), where a linear classiﬁer was ﬁtted on ImageNet using the model representations and evaluated on 7 natural distribution shift datasets.

Original: In particular, we ﬁtted a logistic regression classiﬁer with 1e-5 L2 regularization on ImageNet training set which was trained with a batch size 512, the Adam optimizer with a learning rate 3e-4 and early stopping.
Revised: In particular, we ﬁtted a logistic regression classiﬁer with 1e-5 L2 regularization on ImageNet training set which was trained with a batch size 512, the Adam optimizer with a learning rate 3e-4 and early stopping.

Original: Note that this was different from Radford et al.
Revised: Note that this was different from Radford et al.

Original: (2021), where a logistic regression classiﬁer was ﬁtted using full-batch data with decent hyperparameter tuning, due to our computational budget.
Revised: (2021), where a logistic regression classiﬁer was ﬁtted using full-batch data with decent hyperparameter tuning, due to our computational budget.

Original: For evaluation on natural distribution shift datasets, we followed Taori et al.
Revised: For evaluation on natural distribution shift datasets, we followed Taori et al.

Original: and used their released testbed 4 .
Revised: and used their released testbed 5 .

Original: The evaluation datasets and their abbreviations used in Table 2 were: ImageNetV2 (IN-V2, Recht et al., 2019), ImageNet-Sketch (IN-S, Wang et al., 2019), Youtube-BB (YT-BB, Shankar et al., 2019),
Revised: The evaluation datasets and their abbreviations used in Table 2 were: ImageNetV2 (IN-V2, Recht et al., 2019), ImageNet-Sketch (IN-S, Wang et al., 2019), Youtube-BB (YT-BB, Shankar et al., 2019),

Original: ImageNet-Vid (IN-Vid, Shankar et al., 2019), ObjectNet (Barbu et al., 2019), ImageNet Adversarial (IN-A, Hendrycks et al., 2021), and ImageNet Rendition (IN-R Hendrycks
Revised: ImageNet-Vid (IN-Vid, Shankar et al., 2019), ObjectNet (Barbu et al., 2019), ImageNet Adversarial (IN-A, Hendrycks et al., 2021), and ImageNet Rendition (IN-R Hendrycks

Original: et al., 2020).
Revised: et al., 2020).

Revised: https://github.com/modestyachts/imagenet-testbed

Original: E A DDITIONAL E XPERIMENTAL R ESULTS E. S CIENTIFIC
Revised: F A DDITIONAL E XPERIMENTAL R ESULTS F. S CIENTIFIC

Revised: [Figure]

Original: What’s the effect of λ for different objectives on the worst-case DG performance?
Revised: What’s the effect of λ for different objectives on the worst-case DG performance?

Original: In Fig.
Revised: In Fig.

Original: 5, the worst-case target log likelihood versus λ values for different objectives is shown.
Revised: 5, the worst-case target log likelihood versus λ values for different objectives is shown.

Original: We found that
Revised: We found that

Original: Ent is much more sensitive to the choice of λ than CAD, which was part of the reason why we used the latter in most of our experiments.
Revised: Ent is much more sensitive to the choice of λ than CAD, which was part of the reason why we used the latter in most of our experiments.

Original: Note that for SupCon-Ent with small λ values, it was worse than SupCon-Base because of the discretization introduced by the Ent bottleneck, which we veriﬁed by observing that setting λ = 0 lead to similar results.[Figure]
Revised: Note that for SupCon-Ent with small λ values, it was worse than SupCon-Base because of the discretization introduced by the Ent bottleneck, which we veriﬁed by observing that setting λ = 0 lead to similar results.

Original:  RIDGE
Revised: F. B RIDGE

Original: The scientiﬁc setup is closer to our theory than what we do in practice in that worst-case predictor was considered and empirical generalization gap was ignored.
Revised: The scientiﬁc setup is closer to our theory than what we do in practice in that worst-case predictor was considered and empirical generalization gap was ignored.

Original: Here we bridged these gaps with a more practical setup.
Revised: Here we bridged these gaps with a more practical setup.

Original: In particular, we split the PACS dataset to training and validation splits for each domain and considered the setting: the representor trains the encoder on all-domain training splits with a validation loss selection; the learner trains the SVM predictor (average-case) on the source training split which is selected over the source validation split, and evaluates on the validation splits of other target domains.
Revised: In particular, we split the PACS dataset to training and validation splits for each domain and considered the setting: the representor trains the encoder on all-domain training splits with a validation loss selection; the learner trains the SVM predictor (average-case) on the source training split which is selected over the source validation split, and evaluates on the validation splits of other target domains.

Original: The target validation accuracy averaged over all (source, target) setups was reported.
Revised: The target validation accuracy averaged over all (source, target) setups was reported.

Original: For simplicity, we will use CE to denote the objective with the cross-entropy loss that uses labels to minimize R [ Y | Z ] , and SupCon for the contrastive loss that uses supervised augmentations to minimize H
Revised: For simplicity, we will use CE to denote the objective with the cross-entropy loss that uses labels to minimize R [ Y | Z ] , and SupCon for the contrastive loss that uses supervised augmentations to minimize H

Original: [ A | Z ] .
Revised: [ A | Z ] .

Original: We will use CAD in following experiments unless otherwise speciﬁed (chosen with initial experiments).
Revised: We will use CAD in following experiments unless otherwise speciﬁed (chosen with initial experiments).

Original: Details in Appx.
Revised: Details in Appx.

Original: D.2.
Revised: E.2.

Original: [Table]
Revised: [Table]

Original: Does domain bottleneck improve the average-case DG performance?
Revised: Does domain bottleneck improve the average-case DG performance?

Original: Though our theory focuses on the worst-case DG, we empirically showed that adding bottlenecks to enforce support match can also improve the average-case DG performance by comparing CE-Base and CE-CAD in Table 3.
Revised: Though our theory focuses on the worst-case DG, we empirically showed that adding bottlenecks to enforce support match can also improve the average-case DG performance by comparing CE-Base and CE-CAD in Table 3.

Original: What if the representor only has access to source domains?
Revised: What if the representor only has access to source domains?

Original: Similar to what we did in the scientiﬁc setting, we considered the setup where one single domain is speciﬁed as the target domain and excluded from the training set of the representor and used for evaluation with source predictors trained on other domains.
Revised: Similar to what we did in the scientiﬁc setting, we considered the setup where one single domain is speciﬁed as the target domain and excluded from the training set of the representor and used for evaluation with source predictors trained on other domains.

Original: This is denoted as CE+CAD (partial domains) in Table 3, which is much worse then CE-CAD.
Revised: This is denoted as CE+CAD (partial domains) in Table 3, which is much worse then CE-CAD.

Original: This shows the necessity of getting access to target domain information for DG.
Revised: This shows the necessity of getting access to target domain information for DG.

Original: What if the representor only has access to domain-agnostic augmentations?
Revised: What if the representor only has access to domain-agnostic augmentations?

Original: In Table 3, we also compared SupCon-CAD which used supervised augmentations through the labels with CECAD and they achieved the same performance.
Revised: In Table 3, we also compared SupCon-CAD which used supervised augmentations through the labels with CECAD and they achieved the same performance.

Original: This shows that the representor can still learn good representations without labels but only domain-agnostic augmentations in practice.
Revised: This shows that the representor can still learn good representations without labels but only domain-agnostic augmentations in practice.

Original: Can we use standard augmentations?
Revised: Can we use standard augmentations?

Original: In Fig.
Revised: In Fig.

Original: 2, we point out that standard augmentations are not domain-agnostic and thus not suitable for SSL with our objectives.
Revised: 2, we point out that standard augmentations are not domain-agnostic and thus not suitable for SSL with our objectives.

Original: We empirically showed this by using augmentations of SimCLR (see Appx.
Revised: We empirically showed this by using augmentations of SimCLR (see Appx.

Original: D.1 for details) with our objectives (SimCLR-CAD).
Revised: E.1 for details) with our objectives (SimCLR-CAD).

Original: In
Revised: In

Original: Table 3, we indeed observed that using standard augmentations performed much worse than using desired augmentations (SupCon-CAD).
Revised: Table 3, we indeed observed that using standard augmentations performed much worse than using desired augmentations (SupCon-CAD).

Original: How do augmentations matter?
Revised: How do augmentations matter?

Original: Besides investigating the ‘Supervised’ augmentations (SupConCAD) and ‘Standard’ augmentations (SimCLR-CAD) above, we also compared other three augmentations as in the scientiﬁc section.
Revised: Besides investigating the ‘Supervised’ augmentations (SupConCAD) and ‘Standard’ augmentations (SimCLR-CAD) above, we also compared other three augmentations as in the scientiﬁc section.

Original: Speciﬁcally, we considered the ‘SingleDom’, ‘IntraDom’, and ‘ApproxDA’ augmentations.
Revised: Speciﬁcally, we considered the ‘SingleDom’, ‘IntraDom’, and ‘ApproxDA’ augmentations.

Original: As shown in Table 3, SupCon-CAD (SingleDom) and (ApproxDA) maintained the DG performance but SupCon-CAD (IntraDom) was slightly worse (0.5 accuracy drop).
Revised: As shown in Table 3, SupCon-CAD (SingleDom) and (ApproxDA) maintained the DG performance but SupCon-CAD (IntraDom) was slightly worse (0.5 accuracy drop).

Original: We assumed the small gap was due to the speciﬁc dataset that we used (PACS).
Revised: We assumed the small gap was due to the speciﬁc dataset that we used (PACS).

Original: We did the same analysis on VLCS, and SupCon-CAD with ‘Supervised’, ‘SingleDom’, and ‘IntraDom’ augmentations gave 84.7 ± 0.4, 83.2 ± 0.3, and 77.5 ± 2.3, respectively.
Revised: We did the same analysis on VLCS, and SupCon-CAD with ‘Supervised’, ‘SingleDom’, and ‘IntraDom’ augmentations gave 84.7 ± 0.4, 83.2 ± 0.3, and 77.5 ± 2.3, respectively.

Original: This shows the importance of using domain-agnostic augmentations in practice.
Revised: This shows the importance of using domain-agnostic augmentations in practice.

Original: Do standard augmentations affect source performance?
Revised: Do standard augmentations affect source performance?

Original: Previously, we showed that using standard augmentations hurt the DG performance measured by the average target accuracy.
Revised: Previously, we showed that using standard augmentations hurt the DG performance measured by the average target accuracy.

Original: It is natural to ask whether using standard augmentations also hurt the source performance since we should also be interested in the ‘effective robustness’ (Taori et al., 2020).
Revised: It is natural to ask whether using standard augmentations also hurt the source performance since we should also be interested in the ‘effective robustness’ (Taori et al., 2020).

Original: Thus we also reported the average source accuracy of SupCon-CAD and SimCLR-CAD which were 96.9 ± 0.2 and 90.1 ± 0.2, respectively.
Revised: Thus we also reported the average source accuracy of SupCon-CAD and SimCLR-CAD which were 96.9 ± 0.2 and 90.1 ± 0.2, respectively.

Original: The source performance using standard augmentations was indeed worse, but if we consider the source-target gap which was 0.2 for SupCon-CAD and 28.4 for SimCLR-CAD, which still veriﬁed that the non-domain-agnostic standard augmentations were harder to force support match.
Revised: The source performance using standard augmentations was indeed worse, but if we consider the source-target gap which was 0.2 for SupCon-CAD and 28.4 for SimCLR-CAD, which still veriﬁed that the non-domain-agnostic standard augmentations were harder to force support match.

Original: To be even more convincing, we did the same analysis on VLCS, and the average source accuracy of
Revised: To be even more convincing, we did the same analysis on VLCS, and the average source accuracy of

Original: SupCon-CAD and SimCLR-CAD were 86.6 ± 0.1 and 84.6 ± 0.5 which were fairly close, but the average target accuracy were 84.7 ± 0.4 and 57.5 ± 1.7, respectively.[Table]
Revised: SupCon-CAD and SimCLR-CAD were 86.6 ± 0.1 and 84.6 ± 0.5 which were fairly close, but the average target accuracy were 84.7 ± 0.4 and 57.5 ± 1.7, respectively.

Original: What is the impact of CLIP pretraining? To ensure that our gains are not only due to a novel CAD bottleneck, but the synergy between enforcing support constraint and using desired SSL models, we investigated CAD using the standard DomainBed protocol denoted as CAD in the table. It shows that CAD on its own performs similarly with DomainBed baselines (see Table 4 for a full comparison).
Revised: F. D OMAIN B ED

Original: Full result of Table 

Revised: Full result of Table

Original: We included the full result of Table 1 with all baselines on DomainBed as in
Revised: We included the full result of Table 1 with all baselines on DomainBed as in

Original: Table 4.
Revised: Table 4.

Original: We considered most representative baselines from DomainBed, most of which considered learning invariant representations or optimal classiﬁers across domains.
Revised: We considered most representative baselines from DomainBed, most of which considered learning invariant representations or optimal classiﬁers across domains.

Original: Speciﬁcally, we included
Revised: Speciﬁcally, we included

Original: IRM (Arjovsky et al., 2019), GroupDRO (Sagawa et al., 2019), Mixup (Yan et al., 2020), CORAL (Sun & Saenko, 2016), MMD (Li et al., 2018a), DANN (Ganin et al., 2016), CDANN (Li et al., 2018b), and VREx (Krueger et al., 2021).
Revised: IRM (Arjovsky et al., 2019), GroupDRO (Sagawa et al., 2019), Mixup (Yan et al., 2020), CORAL (Sun & Saenko, 2016), MMD (Li et al., 2018a), DANN (Ganin et al., 2016), CDANN (Li et al., 2018b), and VREx (Krueger et al., 2021).

Original: We also included the result pretrained CLIP S model with a zero-shot classiﬁer using text representations (CLIP S Zero Shot), which demonstrated better DG performance than CLIP S with linear probe.
Revised: We also included the result pretrained CLIP S model with a zero-shot classiﬁer using text representations (CLIP S Zero Shot), which demonstrated better DG performance than CLIP S with linear probe.

Original: But we observed that it was outperformed by our CLIP
Revised: But we observed that it was outperformed by our CLIP

Original: S + CAD.
Revised: S + CAD.

Revised: What is the impact of CLIP pretraining?

Revised: To ensure that our gains are not only due to a novel

Revised: CAD bottleneck, but the synergy between enforcing support constraint and using desired SSL models, we investigated CAD using the standard DomainBed protocol denoted as CAD in the table.

Revised: It shows that CAD on its own performs similarly with DomainBed baselines (see Table 4 for a full comparison).

Revised: [Table]

Original: Why ‘oracle’ selection?
Revised: Why ‘oracle’ selection?

Original: In the main body, we provided the results with ‘oracle selection’ which was the closest to our theory among the model selection methods in DomainBed (in the sense that we needed target domain information to achieve IDG).
Revised: In the main body, we provided the results with ‘oracle selection’ which was the closest to our theory among the model selection methods in DomainBed (in the sense that we needed target domain information to achieve IDG).

Original: Here, we also provided results with ‘source validation’ selection in Table 5.
Revised: Here, we also provided results with ‘source validation’ selection in Table 5.

Original: Source validation selection relies on the assumption that source and target data follow similar distributions (Gulrajani & Lopez-Paz, 2021) thus source and target accuracy are highly correlated, which is not really true in practice.
Revised: Source validation selection relies on the assumption that source and target data follow similar distributions (Gulrajani & Lopez-Paz, 2021) thus source and target accuracy are highly correlated, which is not really true in practice.

Original: We found some issues with source validation selection results:
Revised: We found some issues with source validation selection results:

Original: • The selected model with the highest source validation accuracy tends to overﬁt the source domain, thus possibly leads to worse performance on the target domain.
Revised: • The selected model with the highest source validation accuracy tends to overﬁt the source domain, thus possibly leads to worse performance on the target domain.

Original: This can be probed by the fact that the ﬁnetuned CLIP models (CLIP + Base or CLIP + CAD) were generally worse than the original CLIP model; • Selecting model with source validation accuracy tends to diminish the effect of bottlenecks.
Revised: This can be probed by the fact that the ﬁnetuned CLIP models (CLIP + Base or CLIP + CAD) were generally worse than the original CLIP model; • Selecting model with source validation accuracy tends to diminish the effect of bottlenecks.

Original: This can be seen by the fact that the gap between CLIP + Base and CLIP + CAD of source validation selection is much smaller than that of oracle selection;[Table] • The source accuracy is not a good indicator of target accuracy thus its result has a larger variance.
Revised: This can be seen by the fact that the gap between CLIP + Base and CLIP + CAD of source validation selection is much smaller than that of oracle selection; • The source accuracy is not a good indicator of target accuracy thus its result has a larger variance.

Original: E. LAION
Revised: F. LAION

Original: Evaluation results on DomainBed We included the evaluation results of trained models on DomainBed in Table [Table] where we followed exactly the same linear evaluation protocal discussed in
Revised: Evaluation results on DomainBed We included the evaluation results of trained models on DomainBed in Table 6, where we followed exactly the same linear evaluation protocal discussed in

Original: Appx.
Revised: Appx.

Original: D.3.
Revised: E.3.

Original: We observed similar results asTable 2: the CLIP L model trained with the Ent bottleneck on LAION (LAION + Ent) outperformed the one without ((LAION + Base) on all DomainBed datasets, but slightly underperformed the original CLIP L model (which might be due to quality of the LAION-400M dataset).
Revised: We observed similar results as

Original: [Table]
Revised: [Table]

Revised: [Table]