Paper: CLIP the Bias: How Useful is Balancing Data in Multimodal Learning? (ICLR-2024)

Reviewer comment (Presentation):
Regarding Figure 4: At first glance, without referring to the captions, all the color bars appear identical. My initial interpretation was that the results were largely uniform across the board.

Author response:
In Figure 4, both “balanced” and “proxies” have lower AB than the baseline in most of the experiments, with $p$-values of less than $10^{-6}$. The error bars are for the three random seeds used for each experiment, while the $p$-values are calculated across all experiments (e.g. seeds, model size, representation length).

Reviewer comment (Evaluation):
I find myself somewhat perplexed by the conclusions drawn from the study's results. This confusion does not necessarily point to a flaw in the research but suggests that further clarification might be beneficial. I recommend referring to the detailed queries I've raised in the question section.

Author response:
**Answers to Questions**  
* We believe our findings are more nuanced. For example, it might turn out that training on large datasets helps CLIP learn a more accurate representation of sensitive attributes and labels, thereby separating them. In our experiments, however, we answer that question by showing that a bigger dataset doesn’t fix the bias. Also, it was not clear how effective fine-tuning would be for AB and RB. Our experiments provide a detailed answer: RB is sensitive to the *last* distribution it sees on the data so fine-tuning is as effective as training on balanced data from the start, but in AB the effectiveness of fine-tuning is more gradual and depends on the length of its duration. We also study the impact of using proxies as well; e.g. proxies help RB but hurt AB, and we offer an explanation. We also study in detail the impact on the model's quality, and provide a detailed explanation in Appendix A.6 for why classification and retrieval metrics are impacted differently. We hope this clarifies our contribution.

Reviewer comment (Novelty):
In Section 4.1, the authors discuss various strategies, including adjusting the training set sizes and fine-tuning with or without dataset intervention. While these strategies are standard in model training, the authors' approach underscores a well-known principle: merely increasing the size of a 'corrupted' dataset—say, by tenfold—will not address inherent issues. This scenario is a classic case of "garbage in, garbage out." Furthermore, the practice of fine-tuning a model on a specific dataset inevitably alters some pre-existing weights, adapting the model to new data characteristics. Consequently, it is not surprising that models fine-tuned on intervened sets exhibit improved bias metrics. In summary of this question: What's new here?

Author response:
**Answers to Questions**  
* We believe our findings are more nuanced. For example, it might turn out that training on large datasets helps CLIP learn a more accurate representation of sensitive attributes and labels, thereby separating them. In our experiments, however, we answer that question by showing that a bigger dataset doesn’t fix the bias. Also, it was not clear how effective fine-tuning would be for AB and RB. Our experiments provide a detailed answer: RB is sensitive to the *last* distribution it sees on the data so fine-tuning is as effective as training on balanced data from the start, but in AB the effectiveness of fine-tuning is more gradual and depends on the length of its duration. We also study the impact of using proxies as well; e.g. proxies help RB but hurt AB, and we offer an explanation. We also study in detail the impact on the model's quality, and provide a detailed explanation in Appendix A.6 for why classification and retrieval metrics are impacted differently. We hope this clarifies our contribution.

Reviewer comment (Experiments):
This study suggests that balancing our training datasets can mitigate bias. However, it also implies that researchers must painstakingly identify what constitutes bias and determine the features requiring adjustment. I am particularly interested in the authors' insights on how their proposed framework accommodates additional features present in real-world datasets.

Author response:
**Answers to Questions**  
* We agree with your assessment. We hope that since our proposed algorithm is designed to be adaptable to various sensitive attributes and labels by relaxing many assumptions (e.g. attributes and labels can overlap), it is flexible enough to incorporate different scenarios that may be pertinent in real-world applications.

Reviewer comment (Experiments):
My concern extends to the interrelated nature of certain features, as highlighted in the occupation versus gender discussion in Section 4.2. While it is inappropriate to rely on stereotypes for inferring gender, assuming a 50/50 gender split across all professions disregards real-world disparities. It remains unclear how the proposed methods address "association bias." A link to “U.S. BUREAU OF LABOR STATISTICS” https://www.bls.gov/cps/cpsaat11.htm

Author response:
**Answers to Questions**  
* We acknowledge the complexity of this issue. As you mentioned, real-world disparities can continue to have an influence on the model even after balancing the data, and we observe evidence for this in our experiments. For example, AB is not removed entirely even after decorrelating perceived gender with occupation. But, we believe that understanding the extent of how effective balancing the data is for multimodal learning, such as CLIP, would be a step in the right direction.

Reviewer comment (Experiments):
The number of sensitive attributes in the experiments is limited to only gender and occupation.

Author response:
Addressing your concerns regarding the limited number of sensitive attributes in our experiments, we acknowledge that our current focus has been primarily on perceived gender and occupation. This decision was intentional. First, we did not have skin tone annotations in the downstream evaluation datasets. For example, FairFace and UTKFace use race/ethnicity, not skin tone. Second, our goal was to determine how biases in the data (both RB and AB) transfer to their corresponding biases in the model. We believe that using perceived gender and occupation would still provide useful insights that can illuminate this relationship.

Reviewer comment (Experiments):
Further experiments on proposed data balancing algorithm is lacking in the main text.

Author response:
About the experiments of the data balancing algorithm, those comparisons with classical preprocessing methods are available in Appendix A.2.  We prefer to keep them in the appendix since the primary scope of our work is to study the impact of balancing the data on CLIP models.

Reviewer comment (Reproducibility):
How exactly the de-correlation between sensitive attributes and proxies is implemented?

Author response:
* We treat proxies as labels, so they are included in the set of variables $\textbf{y}_r$ in Equation 3. The algorithm identifies the set of weights that satisfy all constraints, including the correlation constraints between the sensitive attributes and proxies. For instance, we remove any correlation between the proxy “computer” and perceived gender.

Reviewer comment (Evaluation):
In AB experiment, why adding proxies has inconsistent performance?

Author response:
* This phenomenon might be attributed to the complex interactions between different variables in the dataset. In [1], for example, the authors find that many factors influence the model’s prediction of sensitive attributes, such as gender, including non-obvious ones such as beliefs and communion. We will elaborate on this in the revised version of the paper. This is why we do statistical analysis to confirm the significance of our findings.

Reviewer comment (Reproducibility):
In data balancing algorithm, how to intuitively interpret $\alpha$ and $\beta$? and how $s$ is determined, is it calculating from the dataset over sensitive attributes?

Author response:
* $\alpha$ and $\beta$ are Lagrange multipliers associated with the constraints $q\le Q$ and $q\ge 0$ respectively (in our experiments $Q=1$). One way to interpret Lagrange multipliers is using the “sensitivity analysis” method [2]. In brief terms, a large value of either $\alpha$ or $\beta$ means that the training example has a bigger “influence” on the final solution (set of weights). More precisely, a large value of $\alpha$ means that the training example will have a bigger impact on the optimal objective function if we allow weights to be larger than $Q$. We will elaborate on this in the revised version of the paper. On the other hand, $s$ is a binary indicator for the sensitive attribute. For example, if we have two attributes for perceived gender, then $s=(1, 0)$ for perceived women and $s=(0, 1)$ for perceived men.

Reviewer comment (Experiments):
While AB is relatively easy to mitigate, RB seems much more difficult to remove. In this regard, I would suggest the authors to shed more light on possible reasons and solutions. For example, I assume data augmentation shall be a promising workaround, and encourage the authors to explore more.

Author response:
Thank you for your thoughtful evaluation of our paper and for your insightful questions and suggestions. We appreciate your recognition of the strengths of the paper, including the motivation, clarity of presentation, and the comprehensive experimental results we provided. Your feedback is invaluable in enhancing our work.

Addressing your question regarding the mitigation of representation bias (RB) versus association bias (AB), we suspect that the reason AB is easier to mitigate is because of its closed-vocabulary nature. RB, on the other hand, deals with an open-vocabulary problem so it is harder to mitigate. Nevertheless, we still observe a positive message. As shown in Figure 2: (1) balancing the sensitive attributes generally helps, and (2) using proxies make balancing more effective. However, these do not eliminate RB entirely as acknowledged in the paper. We will elaborate on this more in the revised version.

Your suggestion of developing a suitable augmentation strategy that might help RB is well-taken. Another open research question is to identify (from the statistics of the data) the minimal set of proxies that can have the biggest impact on RB. We plan to explore these directions in future work.

Reviewer comment (Theory):
To me, the representation and association biases respectively correspond to the distribution shift of marginal output probability p(y) and conditional output probability p(y|x). What about another widely-discussed distribution shift in domain adaptation/generalization literature, i.e., marginal input probability p(x)? Can it also be a significant bias issue in large multimodal models?

Author response:
Thank you for your thoughtful evaluation of our paper and for your insightful questions and suggestions. We appreciate your recognition of the strengths of the paper, including the motivation, clarity of presentation, and the comprehensive experimental results we provided. Your feedback is invaluable in enhancing our work.

About viewing AB and RB as distribution shifts, this is an interesting angle to view both problems. In your notation, $y$ would be the sensitive attribute and $x$ would be the label (e.g. occupation or proxy) if we understand it correctly. So, a shift in $p(y)$ would correspond to RB (balancing → uniform distribution) white a shift in $p(y|x)$ corresponds to AB (balancing → $p(y|x)=p(y)$, i.e. independence). It’s unclear, however, if a shift in $p(x)$ (i.e. changing the distribution of labels) would correspond to a notion of fairness/bias. We would appreciate it if you could elaborate on this.