Paper: Views Can Be Deceiving: Improved SSL Through Feature Space Augmentation (ICLR-2024)

Reviewer comment (Evaluation):
It's unclear how the correlation in Table 1 is computed. It seems spurious attributes and core features are not defined for real data.

Author response:
We originally discussed how the connectivity error rates in Table 1 are computed in Appendix E. We have added a short summary of this procedure to Section 4.2. To summarize, we utilize the spurious attributes given as metadata in each of the datasets, along with the label (i.e. the core attribute), to partition the dataset into different groups. We then compute the connectivity terms by training classifiers to distinguish augmented data from each combination of the two groups and reporting their error rates.

Reviewer comment (Theory):
The theoretical foundation leans heavily on Spectral Contrastive Learning, yet it is not used in the experimental section. It begs the question of its relevance or applicability, and whether the empirical results are well justified by theory.

Author response:
HaoChen et al. (2021) [1] shows that spectral contrastive learning achieves similar empirical performance to other contrastive learning methods, but enables easier theoretical analysis. Theoretically, the spectral contrastive loss is similar to many popular contrastive losses. For instance, the contrastive loss in SimCLR can be re-written as an expression similar to the spectral contrastive loss. We have added a short clarification to Appendix C.

Reviewer comment (Reproducibility):
Which training parameters were utilized in the LaterTVG method?

Author response:
The training parameters used for Late-TVG are identical to the SSL-Base model. The Late-TVG encoder parameters are modified via pruning during training. We have clarified this in Appendix D.2.

Reviewer comment (Writing):
How do the authors identify spurious features? The paper does not provide clear definitions distinguishing between spurious attributes and core features.

Author response:
The spurious features are provided as metadata in each dataset we use, thus are known apriori (e.g. background in Waterbirds). We do not attempt to discover unknown spurious correlations. We have added text to clarify this assumption in Appendix D.1.

Reviewer comment (Theory):
The intuition / reasoning behind the choice to prune the later layers to forget the spurious feature is not well-explained / discussed sufficiently.

Author response:
We have clarified the paragraph on "Intuition for lateTVG" at the end of Section 5.1 in the updated version. To summarize, 

1. Existing SSL augmentations cause undesirable invariances in the representation space, which we demonstrate in the result that the spurious connectivity is greater than the invariant connectivity. This means that images with the same spurious feature are more likely to be mapped closer to each other in the representation space.

2. One approach would be to modify the augmentations in the image space, but here, we take the approach of augmenting the representation space.

3. Prior works [3, 4] have shown that, for a supervised network, spurious features are encoded in the final layers, and that perturbing the final layers is effective in removing the spurious feature. Furthermore, pruning has been shown to be more effective on minority examples [5].

4. If this intuition also holds for an SSL model, then the pruned network (which encodes invariant features more than spurious ones) provides a view of the samples in the representation space that do not include the spurious feature. In this case, the main encoder is forced to use the core feature to map representations to these views.

Reviewer comment (Novelty):
The problem of spurious correlations in SSL is actually equivalent to **feature suppression** studied first here: https://proceedings.neurips.cc/paper/2021/hash/628f16b29939d1b060af49f66ae0f7f8-Abstract.html. It will be useful to discuss the equivalence of this problem as well as other relevant literature.

Author response:
We had cited the suggested work in the related work section when discussing representation learning under dataset imbalances and shortcuts (Chen et al., 2021). We are happy to cite any other suggestions for additional relevant prior work in a future revision.

Reviewer comment (Reproducibility):
In supervised learning, the models trained on Waterbirds etc. are initialized from ImageNet pretrained weights. Is that the case here as well?

Author response:
We use randomly initialized networks for our experiments, so spurious cues from ImageNet pretraining [e.g. 1] would not impact our evaluations. We explain this decision further in “The role of Initialization” paragraph at the end of Section 4.1.

Reviewer comment (Experiments):
Experiments on some larger scale datasets provided in packages such as WILDS (https://wilds.stanford.edu/) or SpuCo (https://spuco.readthedocs.io/en/latest/) can strengthen the empirical success of the method.

Author response:
We thank the reviewer for the suggestion. We found that the only dataset in WILDS with explicit spurious correlation is the CivilComments, which is a natural language dataset, and is outside the scope of this work. The other WILDS datasets do not appear to contain strong spurious cues to the best of our knowledge. SpuCo is certainly a relevant dataset, and we have cited it in our work. We hope to conduct experiments on this dataset for a potential camera-ready, but did not prioritize this as it includes subsets that are similar to our current datasets, such as waterbirds and CMNIST. 

Following the suggestion from Reviewer H1ZX, we have conducted experiments on Hard ImageNet [2], which have been added to Appendix F.1., and which we reproduce below:


|              | None (&#8593;) | Gray (&#8595;) | Gray BBox (&#8595;) | Tile (&#8595;) |
|--------------|----------------|----------------|--------------------|----------------|
| SSL-Base     |   79.5       |     61.6        |        53.5         |       58.1      |
| SSL-Late-TVG |      78.0        |    59.5          |        51.1          |     52.1           |

We train an SSL model on the full Hard ImageNet train set and a linear classifier on the balanced subset (obtained by spurious rankings), then evaluate the classifier on the original test split and three additional splits with stronger color, shape, and texture spurious cues. This tests the model's ability to learn useful representations without relying on spurious correlations, which should result in lower accuracy on the manipulated splits, which we observe in the above table.

Reviewer comment (Novelty):
The paper is missing references to some important works on spurious feature learning: 1. Salient ImageNet: How to discover spurious features in deep learning? 2. WILDS: A Benchmark of in-the-Wild Distribution Shifts.

Author response:
Thanks for pointing out the missing references, we have cited these papers in the revised draft.

Reviewer comment (Experiments):
Most of the results given in the paper are using smaller datasets. I believe the analysis of Section 4 could have been carried out using a large number of publicly available SSL trained models. I would have liked to see some results using models trained on large datasets.

Author response:
We appreciate the suggestions of the three datasets. We note that WILDS does not contain strong spurious correlations except for the natural language dataset CivilComments. The image datasets in WILDS exhibit natural distribution shifts, but do not have the type of spurious correlations studied in our work to the best of our knowledge. FOCUS is definitely an interesting dataset to build upon, but for initial analysis on a larger scale dataset, it is unclear which attribute we should have chosen as the spurious attribute. We can add results on this dataset to a future version of this paper. 

Per your suggestion, we have added experiments on Hard ImageNet in Appendix F.1 to demonstrate our approach on a larger dataset. For the analysis in Section 4, we focused on datasets that contain the spurious attribute as a metadata, since our goal was to provide an in-depth empirical evaluation of spurious feature learning in SSL. 

We train an SSL model on the full Hard ImageNet train set and a linear classifier on the balanced subset (obtained by spurious rankings), then evaluate the classifier on the original test split and three additional splits with stronger spurious cues. This tests the model's ability to learn useful representations without relying on spurious correlations, which should result in lower accuracy on the manipulated splits.


The summarized result table for Hard ImageNet is as follows:


|              | None (&#8593;) | Gray (&#8595;) | Gray BBox (&#8595;) | Tile (&#8595;) |
|--------------|----------------|----------------|--------------------|----------------|
| SSL-Base     |   79.5       |     61.6        |        53.5         |       58.1      |
| SSL-Late-TVG |      78.0        |    59.5          |        51.1          |     52.1           |



As shown in this table, Late-TVG reduces accuracy on three Gray, Gray BBox, and Tile splits, meaning that the downstream classifier trained on Late-TVG representations relies less on the spurious (i.e. non-object) features. We have added more details and results on this experiment to Appendix F.1, and will include it in the revised main paper. 

We believe the added experiments on Hard ImageNet help address your concern about evaluation on larger datasets, and we hope to add the other large-scale dataset in a following version of this paper.

Reviewer comment (Experiments):
Most of the results seem to be on relatively simpler datasets.

Author response:
We appreciate the suggestions of the three datasets. We note that WILDS does not contain strong spurious correlations except for the natural language dataset CivilComments. The image datasets in WILDS exhibit natural distribution shifts, but do not have the type of spurious correlations studied in our work to the best of our knowledge. FOCUS is definitely an interesting dataset to build upon, but for initial analysis on a larger scale dataset, it is unclear which attribute we should have chosen as the spurious attribute. We can add results on this dataset to a future version of this paper. 

Per your suggestion, we have added experiments on Hard ImageNet in Appendix F.1 to demonstrate our approach on a larger dataset. For the analysis in Section 4, we focused on datasets that contain the spurious attribute as a metadata, since our goal was to provide an in-depth empirical evaluation of spurious feature learning in SSL. 

We train an SSL model on the full Hard ImageNet train set and a linear classifier on the balanced subset (obtained by spurious rankings), then evaluate the classifier on the original test split and three additional splits with stronger spurious cues. This tests the model's ability to learn useful representations without relying on spurious correlations, which should result in lower accuracy on the manipulated splits.


The summarized result table for Hard ImageNet is as follows:


|              | None (&#8593;) | Gray (&#8595;) | Gray BBox (&#8595;) | Tile (&#8595;) |
|--------------|----------------|----------------|--------------------|----------------|
| SSL-Base     |   79.5       |     61.6        |        53.5         |       58.1      |
| SSL-Late-TVG |      78.0        |    59.5          |        51.1          |     52.1           |



As shown in this table, Late-TVG reduces accuracy on three Gray, Gray BBox, and Tile splits, meaning that the downstream classifier trained on Late-TVG representations relies less on the spurious (i.e. non-object) features. We have added more details and results on this experiment to Appendix F.1, and will include it in the revised main paper. 

We believe the added experiments on Hard ImageNet help address your concern about evaluation on larger datasets, and we hope to add the other large-scale dataset in a following version of this paper.

Reviewer comment (Experiments):
Why is the analysis of Section 4 limited to simpler datasets and no results are provided for the datasets discussed above?

Author response:
We appreciate the suggestions of the three datasets. We note that WILDS does not contain strong spurious correlations except for the natural language dataset CivilComments. The image datasets in WILDS exhibit natural distribution shifts, but do not have the type of spurious correlations studied in our work to the best of our knowledge. FOCUS is definitely an interesting dataset to build upon, but for initial analysis on a larger scale dataset, it is unclear which attribute we should have chosen as the spurious attribute. We can add results on this dataset to a future version of this paper. 

Per your suggestion, we have added experiments on Hard ImageNet in Appendix F.1 to demonstrate our approach on a larger dataset. For the analysis in Section 4, we focused on datasets that contain the spurious attribute as a metadata, since our goal was to provide an in-depth empirical evaluation of spurious feature learning in SSL. 

We train an SSL model on the full Hard ImageNet train set and a linear classifier on the balanced subset (obtained by spurious rankings), then evaluate the classifier on the original test split and three additional splits with stronger spurious cues. This tests the model's ability to learn useful representations without relying on spurious correlations, which should result in lower accuracy on the manipulated splits.


The summarized result table for Hard ImageNet is as follows:


|              | None (&#8593;) | Gray (&#8595;) | Gray BBox (&#8595;) | Tile (&#8595;) |
|--------------|----------------|----------------|--------------------|----------------|
| SSL-Base     |   79.5       |     61.6        |        53.5         |       58.1      |
| SSL-Late-TVG |      78.0        |    59.5          |        51.1          |     52.1           |



As shown in this table, Late-TVG reduces accuracy on three Gray, Gray BBox, and Tile splits, meaning that the downstream classifier trained on Late-TVG representations relies less on the spurious (i.e. non-object) features. We have added more details and results on this experiment to Appendix F.1, and will include it in the revised main paper. 

We believe the added experiments on Hard ImageNet help address your concern about evaluation on larger datasets, and we hope to add the other large-scale dataset in a following version of this paper.

Reviewer comment (Experiments):
**Scalability and Efficiency:** The paper introduces the LATE TVG method, which involves regularizing later layers of the encoder via pruning. While this approach is novel, the scalability and computational efficiency of the method in large-scale settings are not thoroughly addressed. Pruning, especially in deeper layers, can be computationally intensive and may not scale well with very deep networks or extremely large datasets. **Actionable Insight:** Future work could focus on optimizing the pruning process to enhance scalability and efficiency. This might involve investigating more computationally efficient pruning techniques or adapting the method to work with sparse neural networks.

Author response:
> W1, Q1. Scalability and Efficiency

The SSL-LateTVG model updates the same number of parameters as SSL-Base during training, with the forward pass keeping both the original and pruned encoder (See Algorithm 1 in Appendix B). The pruning operation is cost O(n) where n is the number of parameters. So any FLOPs used for the extra pruning mechanism will be very small compared to a single forward pass. 

In summary, the training parameters are identical between SSL-Base and LateTVG, with the LateTVG encoder modified via pruning. For the backward pass, only the original encoder and other SSL parameters like the projection head are updated. We have also clarified this computational process in Appendix B, and parameters in Appendix D.2. 

We provide runtimes on the MetaShift dataset below as an example:

SimSiam: SSL-Base: 4h 26m, SSL-LateTVG: 5h 14m

SimCLR: SSL-Base: 4h 39m SSL-LateTVG: 4h 50m

Reviewer comment (Experiments):
**Domain Generalization:** Although the paper demonstrates the effectiveness of LATE TVG across several benchmarks, it primarily focuses on visual representation learning. The generalizability of the method to other domains or types of data, such as text or audio, is not explored. **Actionable Insight:** To strengthen the paper's contributions, additional experiments could be conducted to evaluate the method's performance on non-visual datasets. This would provide a clearer understanding of the method's applicability across different domains and data modalities.

Author response:
> W2. Domain Generalization

We appreciate the reviewer raising this point. We focus on the image modality in this work because SSL methods we use as base implementations are designed for images. We agree that evaluating our approach on non-visual modalities is an important direction for future work.

Reviewer comment (Experiments):
**Comparison with State-of-the-Art:** While the paper shows that LATE TVG outperforms traditional methods, it does not provide an extensive comparison with the latest state-of-the-art methods in SSL that address similar challenges. Without this context, it's difficult to gauge the relative progress made. **Actionable Insight:** The authors could enhance the paper by including a more comprehensive comparison with state-of-the-art methods. This might involve reproducing results from recent papers or directly comparing the proposed method against other cutting-edge techniques in SSL.

Author response:
> W3. Comparison with State-of-the-Art

To the best of our knowledge, this is the first work evaluating visual self-supervised representation learning on datasets containing known spurious correlations. If the reviewer could suggest recent SSL papers addressing similar challenges, we would be happy to include comparisons.

Reviewer comment (Experiments):
**Clarification on Scalability and Efficiency:** Question: Could you provide more details on the computational efficiency and scalability of the LATE TVG method, especially when applied to very large datasets or extremely deep networks? Suggestion: It would be beneficial if the authors could include a section discussing the computational complexity of their method, possibly comparing it with other SSL methods in terms of training time and resource utilization.

Author response:
> W1, Q1. Scalability and Efficiency

The SSL-LateTVG model updates the same number of parameters as SSL-Base during training, with the forward pass keeping both the original and pruned encoder (See Algorithm 1 in Appendix B). The pruning operation is cost O(n) where n is the number of parameters. So any FLOPs used for the extra pruning mechanism will be very small compared to a single forward pass. 

In summary, the training parameters are identical between SSL-Base and LateTVG, with the LateTVG encoder modified via pruning. For the backward pass, only the original encoder and other SSL parameters like the projection head are updated. We have also clarified this computational process in Appendix B, and parameters in Appendix D.2. 

We provide runtimes on the MetaShift dataset below as an example:

SimSiam: SSL-Base: 4h 26m, SSL-LateTVG: 5h 14m

SimCLR: SSL-Base: 4h 39m SSL-LateTVG: 4h 50m

Reviewer comment (Experiments):
**Long-Term Effects of Pruning:** Question: Pruning, especially in later layers, might have long-term effects on the learning capabilities of the network. Have you investigated how the pruning process affects the network's ability to learn new tasks or adapt to new data over time? Suggestion: Providing insights or conducting experiments on the long-term effects of pruning could offer a more nuanced understanding of the method's robustness and adaptability.

Author response:
> Q2. Long-Term Effects of Pruning

While we agree that this is an interesting direction, our focus is on mitigating spurious correlations for self-supervised representation learning itself. We leave investigations into pruning's effects on adapting to new data or tasks to future work.

Reviewer comment (Experiments):
**Robustness to Adversarial Attacks:** Question: How robust is the LATE TVG method to adversarial attacks, given that it focuses on mitigating spurious correlations? Suggestion: Including experiments or discussions on the method's robustness to adversarial examples could highlight another dimension of its efficacy, particularly in security-sensitive applications.

Author response:
> Q3. Robustness to Adversarial Attacks

We believe that evaluating adversarial robustness is orthogonal to the problem studied in this paper. Our method targets removing spurious correlations, not increasing robustness to adversarial examples. While an interesting direction, analyzing LateTVG's robustness to adversarial attacks is out of scope for this work. We leave this to future work and maintain our focus on addressing spurious correlations in self-supervised learning.