Paper: Towards domain-invariant Self-Supervised Learning with Batch Styles Standardization (ICLR-2024)

Reviewer comment (Experiments):
Inadequate ablation experiments. In Section 5.2, the authors do not show the ablation effect of the original SimCLR. Table 4 demonstrates the effectiveness of FA and BSS compared to baseline SimCLR. However, it is not reflected in Section 5.2. So it does not show that FA and BSS are effective compared to original SimCLR.

Author response:
> **In Section 5.2, the authors do not show the ablation effect of the original SimCLR.**

We have conducted the same ablation experiments as detailed in Section 5.2 specifically for a SimCLR with standard image data augmentation. The results of these additional experiments have been included to the existing plots on Figure 2. In examining the experiment on average domain purity varying with the number of nearest neighbors (Figure 2a), standard SimCLR exhibits only slightly higher average domain purity than SimCLR with FA, but significantly higher than SimCLR with Batch Style Standardization (BSS) (higher domain purity is worse). This observation suggests that FA does not significantly enhance domain-invariance compared to BSS. Similarly, in Figure 2b, it is noticeable that standard SimCLR results in negatives dissimilar to the anchors. SimCLR with FA produces negatives more similar to the anchors but not to the same extent as SimCLR with BSS. Finally, in terms of performance concerning batch size, standard SimCLR yields considerably lower performances compared to SimCLR with FA or with BSS.

Reviewer comment (Theory):
Perhaps there is a limitation to BSS. Assuming that domain labels are not available, there is no guarantee that a number of randomly selected images belong to different domains. As the author said, "Finally, after applying the inverse Fourier transform to the different modified Fourier transforms, the style of the randomly chosen image is transferred to all images, effectively standardizing/harmonizing the style." In the worst case, it is possible that a number of randomly selected images all belong to the same domain (same style), and then their magnitude spectra may be similar. If the magnitude spectra of these images are still used to augment all of the images, it may indeed create harder negative samples (because of the similar magnitude spectra). However, it would also create the problem of more similarity between pairs of positive samples, potentially forcing the network to capture more similar style information than domain invariant information in the positive sample pairs.

Author response:
> **Perhaps there is a limitation to BSS. Assuming that domain labels are not available, there is no guarantee that a number of randomly selected images belong to different domains. [...] it may indeed create harder negative samples. However, it would also create the problem of more similarity between pairs of positive samples, potentially forcing the network to capture more similar style information than domain invariant information in the positive sample pairs.**

Indeed, in the worst-case scenario, we could sample a batch of $N$ examples from a single domain (same style). Applying BSS successively $V$ times on this batch would result in
$V$ batches of examples, each with its own style (as depicted in Fig 1c). As you correctly mentioned, because all examples initially came from the same domain (same style), we should end up with a single style for all examples, creating harder negatives but simpler positives. In contrastive learning, the loss tries at the same time to bring the positive pairs closer and to drive away the negative ones: since in this pathological case, all examples have the same style, the style cannot be used to do both at the same time. It should also be noted that on top of the style standardization, geometric augmentation and batch-wise color augmentations are applied (see section 3.2.2, ablation in section 5.1, table 4). This mitigates the problem since the $V$ positives have undergone different color augmentations.

Reviewer comment (Writing):
What does "spurious correlations" mean? Is it the possibility that in SSL, when repelling negative samples, the main basis for error may be the difference in style?

Author response:
> **What does "spurious correlations" mean?**

In general, correlations or features are said to be spurious when they predict the target label without possessing a causal relationship with it. In the context of SSL, spurious features would be typically associated with non-causal attributes such as style (color, texture, etc.), while the target variable would be the ones related to the SSL task objectives. As an example, in SimCLR, when diverse styles coexist among the positive and negative examples, “style features” may be used to solve the contrastive task.

Reviewer comment (Experiments):
In section 4.2, "We did not use ImageNet transfer learning, except on DomainNet to allow fair comparisons with prior UDG works." Does it mean that this paper uses an ImageNet pre-trained model when conducting experiments on the DomainNet dataset?

Author response:
> **In section 4.2, "We did not use ImageNet transfer learning, except on DomainNet to allow fair comparisons with prior UDG works." Does it mean that this paper uses an ImageNet pre-trained model when conducting experiments on the DomainNet dataset?**

 Using Transfer Learning / pretrained models in DG/UDG is a common practice in the community but we think it is misguided. Often the pre-training dataset, like ImageNet for example, can include one or more of the target domains  (e.g: photo for PACS). We have tried to limit as much as possible the usage of transfer learning in our experiments and only used it for the DomainNet dataset.

Reviewer comment (Experiments):
What is the performance of the ERM (empirical risk minimization) on those benchmarks? People have observed ERM being a very strong baseline when it comes to domain generalization settings [1, 2].

Author response:
Performances of the ERM baseline have been included in Tables 1 and 2. In our setting, ERM has a low performance since it is the only method that does not  benefit from the SSL pretraining step while the amount of labeled data used for finetuning is too small to make the model generalize well.

Reviewer comment (Experiments):
As a small fraction of labeled data is always used, why not try some good semi-supervised learning methods such as FixMatch [3] or AdaMatch [4] (which also deals with domain shift)? And there is also contrastive-based semi-supervised learning such as CoMatch [5]. Since the goal is to improve the performance on unseen target domains, what is the advantage of using a framework of unsupervised pretraining + finetuning?

Author response:
This is a fair question: Indeed, in practical scenarios aimed at producing a model with robust generalization to unseen domains, a dilemma can arise regarding two primary learning strategies:
1. Unsupervised Pretraining handling domain shift (UDG) + Finetuning
2. Semi-Supervised Learning, such as FixMatch, combined with Domain Generalization techniques to handle domain shift. (Note: AdaMatch works in a Unsupervised Domain Adaptation setting not Domain Generalization)
In practice, these strategies are not mutually exclusive: Combining both approaches would allow one to harness the strengths of each, potentially achieving optimal performance on unseen domains. Consequently, the suggested approach would be to (1) perform unsupervised pretraining addressing domain-shift (UDG) and subsequently (2) fine-tune the model using Semi-Supervised Learning in conjunction with Domain Generalization techniques.

However, within the context of UDG, where the primary focus is evaluating the method's capacity to learn domain-invariant features, the typical workflow involves: (1) unsupervised training on source domains, (2) finetuning on source domains via linear probing, and (3) evaluation on unseen target domains. That is why the community follows this workflow.

Reviewer comment (Experiments):
A related remark would be: what is the SOTA methodology when it comes to domain shift? In practice, one may easily resort to some VLMs (e.g. CLIP and its variants) pre-trained on large-scale image-text pair data when facing domain shift problems. As CLIP models have shown very good performance on samples under distribution shift, especially in image classification, I wonder to what extent can the problem be solved by them already. I think it makes more sense to develop techniques on top of these strong baselines, as it is very likely the method or the performance improvement on small-scale datasets or non-SOTA models does not transfer/scale well. It would be much more interesting to see if the proposed BSS still holds the same performance gap on top of CLIP. That being said, this remark is not a criticism of the authors who follow the common practice. But it would be still great if the authors could share their thoughts on this point.

Author response:
It's indeed a very insightful remark. Vision-language models like CLIP have demonstrated impressive zero-shot capabilities in in-distribution evaluation scenarios and even in open-world evaluation scenarios. We acknowledge that VLMs have made strides in solving domain-shift problems and may serve as a strong foundation for developing new techniques for out-of-distribution (OOD) generalization.

However, there are certain considerations and potential challenges when relying solely on VLMs for domain-shift problems:

1. VLMs often necessitate a substantial amount of labeled data for pretraining, and pretrained image and text encoders might not be well-suited for specific task data. For instance, in the histopathology field, existing pretrained VLM like CLIP does not generalize well while histopathology available paired data (images, diagnoses) are too scarce to train a new VLM from scratch [1].
2. While fine-tuning these models can yield significant performance gains on a given data distribution, it has been observed to potentially diminish out-of-distribution generalization ability [2, 3]. Hence, careful fine-tuning is essential to preserve their generalization capability in OOD evaluation scenarios.
3. On top of that, fair evaluation of their OOD capabilities are hard because VLMs are often trained on very large datasets that include many unknown domains. This makes it difficult to be sure that the target domains have not been seen during training.

As suggested, it would be interesting to investigate whether BSS can enhance domain-invariance and improve performances on top of VLMs, thereby addressing the challenge of loss of OOD generalization during fine-tuning.

References:
* [1] Lu et al. "Towards a visual-language foundation model for computational pathology.", arXiv preprint 2023.
* [2] Wortsman et al. "Robust fine-tuning of zero-shot models.", CVPR 2022
* [3] Goyal, Sachin, et al. "Finetune like you pretrain: Improved finetuning of zero-shot vision models.”, CVPR 2023.

Reviewer comment (Presentation):
To be more self-contained, it would be great if the authors could also introduce the contrastive-based UDG methods before Section 3.2.2.

Author response:
Contrastive-based UDG methods have been introduced before Section 3.2.2 (related works about UDG).

Reviewer comment (Experiments):
What are the source domains for PACS pretraining?

Author response:
For the experiments on PACS, we followed the common practice in the UDG community employing a leave-one-out domain evaluation. In Table 1, we only specifies the target domain for evaluation which implies that all other domains serve as pretraining.

Reviewer comment (Novelty):
The major weakness is novelty. Given that Fourier-based methods have been extensively studied in existing DG literatures, the direct adoption of Fourier features is not entirely novel. Plus, the idea of transforming the styles in batch is deeply related to Mixstyle [Zhou et al., ICLR21] but authors did not compare or discuss the difference.

Author response:
While MixStyle also transform the styles in the batch to improve out-of-domain generalization, significant differences exist between the proposed BSS and MixStyle:
1. The objectives of MixStyle and BSS diverge: MixStyle seeks to alter the style of examples, introducing a broad style variability at the features-level, whereas BSS strives to eliminate the style variability by enforcing a single style at the image-level (style augmentation vs style standardization).
2. To alter the style of examples at the features-level, MixStyle mixes the features maps` statistics (mean, std)  with those of random examples. In contrast, BSS normalizes the style of all examples in a batch by replacing all amplitudes with those of a randomly chosen image, resulting in a batch with a single style. In practice, MixStyle could be extended to normalize the style of examples by swapping all the statistics of a batch with those of a randomly chosen image and used within BSS to replace the proposed Fourier-based normalization.
3. MixStyle relies on the assumption that style information of examples can be captured by the examples’ statistics (mean, std) of bottom layers features in CNN [1]. This assumption is not directly transferable to all architecture like Vision Transformers for example. This makes MixStyle specific to the CNN architecture, whereas BSS operating at the image-level is architecture agnostic.

References:
* [1] Huang et al.. "Arbitrary style transfer in real-time with adaptive instance normalization.", ICCV 2017

Reviewer comment (Writing):
Section 3.2, i.e., the BSS part, is hard to understand. Authors should do their best to better present this part.

Author response:
We have tried to be as clear as possible by providing a description of the exact process of how BSS is performed (Section 3.2.2 - 2nd paragraph), a Figure illustrating how BSS is performed (Figure 1) and pseudo-code along with a Pytorch implementation of BSS. We will try to improve the explanation but it would help if you could explain which specific aspect you find unclear.

Reviewer comment (Experiments):
Comparision approaches are not enough: authors should compare with existing Fourier methods to validate their effectiveness.

Author response:
All results from Table 1, 2 and 3 compare to existing UDG methods based on FA (DiMAE, CycleMAE (new)). Comparison to a standard FA strategy is also available for all experiments. If you can think of more pertinent comparisons, could you please point them out?

Reviewer comment (Theory):
There lacks theoretical support of why such BSS approach can succeed in learning domain-invariant representations.

Author response:
In the UDG community, it is common practice to present motivations, intuitions, and empirical validations through performances or experiments (DARLING, BRaD, DiMAE, CycleMAE). Producing meaningful theoretical support for DG deep learning methods is hard:  strong assumptions regarding the data, such as a modelization of semantic/style separation, would be required and in practice, such assumptions never hold on real datasets. Moreover, our non-reliance on domain labels makes it even harder.