Paper: When Does Bias Transfer in Transfer Learning? (ICLR-2024)

Reviewer comment (Novelty):
The novelty and contribution of their findings is limited. Previous works [1][2] already investigated such aspects of transfer learning, and some findings in this submission (particularly the bias transfer phenomenon in the scenario (2) and (3)) can be implied from their results.

Author response:
**W1, W3. Novelty and Related Work:** Neyshabur et. al focuses on understanding whether low-level statistics or feature reuse is transferred during transfer learning. However, they do not investigate the problem of bias transfer: where features from the pre-training data (that are not used during downstream training) can persist during downstream deployment. Shafahi et al focus on adversarial robustness: in our paper, we focus on biases as features (and in particular, emphasize natural biases which might appear in standard. 

Lubana et. al is a recent work (ICML 2023) which performs mechanistic analysis. As part of their analysis, they perform fine-tuning from a backdoored CIFAR-10 to a clean CIFAR-10, and find that fine-tuning may not eliminate spurious cues. In our work, we perform a larger study that fully investigates the landscape of bias transfer across a wide range of tasks for different fine-tuning regimes. We further discuss potential mitigation strategies, such as target dataset debiasing and weight decay. Finally, we investigate bias transfer beyond synthetic biases, and find evidence of such transfer in natural datasets such as ImageNet.

We thank the reviewer for bringing these to our attention and have added them to the related work of the final version.

Reviewer comment (Novelty):
Discussions on previous works is not enough. The most related works [1][2] are not cited and not discussed. In relation to transferrability of backdoor attacks, I think [3] is one of very related works, but is not discussed. I recommend the authors to survey their previous works and make clear the novelty and contribution of this paper.

Author response:
**W1, W3. Novelty and Related Work:** Neyshabur et. al focuses on understanding whether low-level statistics or feature reuse is transferred during transfer learning. However, they do not investigate the problem of bias transfer: where features from the pre-training data (that are not used during downstream training) can persist during downstream deployment. Shafahi et al focus on adversarial robustness: in our paper, we focus on biases as features (and in particular, emphasize natural biases which might appear in standard. 

Lubana et. al is a recent work (ICML 2023) which performs mechanistic analysis. As part of their analysis, they perform fine-tuning from a backdoored CIFAR-10 to a clean CIFAR-10, and find that fine-tuning may not eliminate spurious cues. In our work, we perform a larger study that fully investigates the landscape of bias transfer across a wide range of tasks for different fine-tuning regimes. We further discuss potential mitigation strategies, such as target dataset debiasing and weight decay. Finally, we investigate bias transfer beyond synthetic biases, and find evidence of such transfer in natural datasets such as ImageNet.

We thank the reviewer for bringing these to our attention and have added them to the related work of the final version.

Reviewer comment (Writing):
What is the definition of biases in this paper? It should be specified first of all to discuss "bias" transfer.

Author response:
**W2. Defining Bias:** We define bias as a feature that the model relies on but is not causally linked with the target class. In particular, we focus on biases that the transfer learning model would not have relied on if it had been trained from scratch. We have added this definition to Section 2 in our revision.

Reviewer comment (Novelty):
Novelty is a concern for this paper: all the observations in the paper are expected and not surprising. For example, isnt it obvious that the full network transfer learning setting will be less affected by the source biases than the fixed one?

Author response:
**W1. Novelty:** We want to point out that the main message of our paper is that *spurious features which are present in the pre-training data and completely absent in the downstream task can unexpectedly pop up during downstream deployment*. The rest of our paper (comparing fixed-feature to full-network fine tuning, varying weight decay, looking at different downstream tasks) is all in an effort to fully characterize this phenomenon, but they are not meant to be “surprising” in and of themselves.

Reviewer comment (Novelty):
Wang et al [1] suggest manipulating the finetuning data to reduce the biases. No suggestion is proposed by the authors.

Author response:
**W3. Manipulating the target dataset:** In our overall framing, we consider the case where the target dataset is balanced (e.g., does not have the bias) but the source dataset is skewed. In this case, bias transfer occurs, even though the bias is not present in the downstream dataset. Furthermore, in Section 3.2 and 4.2, we consider actively debiasing the dataset to counter the bias. However, we find that debiasing the dataset only partially mitigates bias transfer (especially in the fixed transfer case).

Reviewer comment (Experiments):
For the synthetic bias case, what is termed as backdoor attack is simply adding a spurious correlation synthetcially to the dataset to increase/induce bias into it.

Author response:
**W4. Backdoor attack is just inserting a feature.** We agree that backdoors are just features, which is why we believe it is a useful starting point for studying bias transfer (since biases are also features). We use the standard conception of backdoor attacks found in the data poisoning literature, see e.g., [1, 2].

Reviewer comment (Writing):
What happens when the latter is balanced?

Author response:
**Q2. What happens when the latter is balanced?** As we show in Sections 3.2 and 4.2 of our paper, even if the downstream dataset is balanced or unbiased, bias can still transfer! This means that ML developers must pay attention to the pre-training models/data, and cannot rely on solely downstream balancing.

Reviewer comment (Novelty):
The work is rather incremental to recent work in the literature, especially the work by Wang and Russakovsky [1]. I missed a reference to that work. The novelty would be more obvious e.g. had the authors demonstrated their results beyond the vision modality. See this recent survey for an overview of closely-related pieces of work, where a proper comparison would help highlight the novelty of the presented work https://arxiv.org/abs/2310.17626

Author response:
**W1. Wang et. al:** We thank the reviewers for bringing this to our attention. Wang et. al investigates gender biases in the case where the spurious feature is present in both the pre-training dataset and the fine-tuning dataset. They then consider actively changing the fine-tuning dataset to counter the correlation in the pre-training data (this is equivalent to the “debiasing” scheme in our paper). In our work, we find that (especially in the fixed feature case) a large portion of the dataset would need to be debiased in order to mitigate bias transfer. As you mention, this level of downstream debiasing may not be feasible.

Our work further highlights a different risk: spurious features which are present in the pre-training data and completely absent in the downstream task can unexpectedly pop up during downstream deployment. These types of “backdoor” features are especially pernicious, since a practitioner might not be aware that their pre-training model has such features. We fully investigate the landscape of this phenomenon, across different types of biases (synthetic and natural), architectures, and fine-tuning regimes (fixed-feature vs. full-network). While we discuss potential mitigation strategies (e.g., weight decay), we arrive at a very different conclusion than Wang et. al: practitioners cannot solely depend on post-hoc strategies to deal with bias transfer. As a result, we believe it is critical to prevent such biases at their source by more carefully choosing our pre-trained data in the first place.

Reviewer comment (Experiments):
Full-network fine-tuning obviously has a better chance of reducing the bias in the pre-trained backbone, compared with a frozen backbone (where the bias mainly exists) + a linear head.

Author response:
**W2a. Full-network vs fixed-feature effectiveness is obvious.** We agree with the reviewer that fixed-feature fine-tuning being more prone to bias transfer than full-network fine-tuning is not surprising. Our goal here is to fully investigate the discrepancy between these two fine-tuning strategies, particularly in light of the fact that fixed-feature fine-tuning is becoming increasingly popular in the age of large pre-trained (foundation) models. We thus view quantifying and ablating the gap between the two as a valuable contribution.

More generally, the main finding of our paper is that *spurious features which are present in the pre-training data and completely absent in the downstream task can unexpectedly pop up during downstream deployment*. The remainder of our paper (including the comparison of  fixed-feature to full-network fine tuning) is all in an effort to fully characterize this phenomenon, but is not meant to be “surprising” in and of itself.

Reviewer comment (Experiments):
The experiments about weight decay do not explain why it is helpful. Is it generally the case that regularization helps mitigate the bias? Is there something specific to weight decay that helps reduce the bias? What about other regularization strategies?

Author response:
**W2b. Weight Decay:** In appendix A.5, we discuss a theoretical justification for why weight decay can help mitigate the bias. Specifically, we consider the logistic regression example from Section 2. The gradient of the logistic loss with L2 regularization becomes


$$\nabla \ell_{\mathbf{w}} (\mathbf{x}_i, y_i) = (\sigma(\mathbf{w}^\top \mathbf{x}_i) - y_i)\cdot \mathbf{x}_i + \lambda\mathbf{w}=(\sigma (\mathbf{w}_S^\top \mathbf{x}_i) - y_i)\cdot \mathbf{x}_i + \lambda(\mathbf{w}_S +\mathbf{w}\_{S'})$$

Where $\mathbf{w}\_S$ and $\mathbf{w}\_{S’}$ are the projections of $\mathbf{w}$ onto the span of the target datapoints ($S$) and its complementary subspace ($S’$). The gradient (as in Section 2) restricts the space of updates to those in $S$. However, due to regularization, $\mathbf{w}_{S’}$ is driven to 0, so any planted bias in $S’$ collapses with regularization.

- Any $\mathbf{w}$ can be written as $\mathbf{w}\_S + \mathbf{w\}_S'$ where $\mathbf{w}\_S $ is $\mathbf{w}$ projected into the span of the training examples $S$ and $\mathbf{w\}_S'$ is $\mathbf{w}$ projected onto the associated null space $S'$. Then  $$\mathbf{w}^\top x_i = (\mathbf{w}\_S + \mathbf{w}\_{S'})^\top x_i = \mathbf{w}_S^\top x_i$$
because $\mathbf{w}\_{S'}^\top x_i  = 0$ (since $S'$ is the null space of the span of the training examples).
- Note that 
$$||\mathbf{w}\_S + \mathbf{w}\_{S'}||_2^2 = \mathbf{w}\_S^\top \mathbf{w}\_S + 2\mathbf{w}\_S^\top \mathbf{w}\_{S'} + \mathbf{w}\_{S'}^\top\mathbf{w}\_{S'} = ||\mathbf{w}\_S||_2^2 + || \mathbf{w}\_{S'}||_2^2$$
since $\mathbf{w}\_S^\top \mathbf{w}\_{S'}  = 0$. Since $ \mathbf{w}\_{S'}$ does not appear in the logistic loss, any solution $\mathbf{w}$ where the component $||\mathbf{w}\_{S'}||_2^2 > 0$ will be suboptimal. The problem is convex, so SGD will lead to an optimal solution which means $||\mathbf{w}\_{S'}||_2^2 = 0$.
- We have just added an experiment in the uploaded revision of Appendix A.5 which considers a toy logistic regression example. We show that the component of the initial weight vector projected onto the null space of the training examples (i.e., $||\mathbf{w}\_{S''}||$) degrades to 0 with weight decay (but is unchanged without weight decay). Measuring $||\mathbf{w}\_S||$ and $||\mathbf{w}_{S'}||$ for our neural networks in the main experiments is much more involved (one initial strategy might be to study the SVD of the weights with weight decay). We would be happy to pursue this for the final version.

Reviewer comment (Experiments):
Modifying the target dataset to counter the bias seems helpful but it is not obvious how it can be done in the general case (e.g. beyond balancing the sample in different subgroups or reintroducing the backdoor attacks in the target dataset at random).

Author response:
**W2c. Debiasing the target dataset**: We fully agree with the reviewer here—in fact, we believe that (a) according to our paper, debiasing the target dataset does not completely eliminate bias transfer, and (b) “fully” debiasing a dataset is often intractable, as the reviewer notes. This trend holds more generally: while several mitigation strategies can help reduce bias transfer, post-hoc approaches typically do not fully remove it (and often are not infeasible without sacrificing something performance). We thus believe it is critical to prevent such biases at their source by more carefully choosing our pre-trained data in the first place.

Reviewer comment (Reproducibility):
The authors mention that they "find that weight decay does not reduce bias transfer in the fixed feature transfer learning regime, where the weights of the pretrained model are frozen.". How is weight decay applied to frozen weights?

Author response:
**Q2. Weight Decay in the Fixed Feature Regime:**
In this setting, we apply regularization to the linear layer only.

Reviewer comment (Experiments):
It would be useful to know how sensitive these conclusions are to fine-tuning hyper-parameters such as learning rate, momentum, and epochs.

Author response:
**Impact of learning rate:** We replicated our backdoor experiments in Section 3 with varying learning rates (Appendix A.8 in the revision). Similar to weight decay, we find that increasing the learning rate does not mitigate bias transfer in the fixed-feature setting (ASR actually goes up) but does help in the full-network setting. However, increasing learning rate is a fairly invasive intervention, and often has adverse effects on model performance.

Reviewer comment (Experiments):
Did you do experiments looking at the initial learning rate for fine-tuning and how that affects the amount of bias transfer? I would expect higher learning rates would lead to lower bias transfer.

Author response:
**Impact of learning rate:** We replicated our backdoor experiments in Section 3 with varying learning rates (Appendix A.8 in the revision). Similar to weight decay, we find that increasing the learning rate does not mitigate bias transfer in the fixed-feature setting (ASR actually goes up) but does help in the full-network setting. However, increasing learning rate is a fairly invasive intervention, and often has adverse effects on model performance.

Reviewer comment (Experiments):
Do you think these conclusions would hold for other pretrained models like SimCLR and CLIP?

Author response:
**SimCLR and CLIP:** While we focused on supervised pre-training models in our paper, we would expect biases from SimCLR and CLIP to similarly transfer. Indeed, there has been a large body of work indicating that CLIP contains several concerning biases, especially concerning race and gender [1, 2]. Understanding how these types of biases manifest in fine-tuned models is an interesting avenue for future work.

[1] https://arxiv.org/abs/2108.02818

[2] https://arxiv.org/abs/2210.14562