Paper: FairerCLIP: Debiasing CLIP's Zero-Shot Predictions using Functions in RKHSs (ICLR-2024)

Reviewer comment (Experiments):
The authors did not compare their results with unimodal baselines, i.e., techniques that debias only the image/text representations of the VLM.

Author response:
We compared our method with various baselines, both with and without ground truth $Y$. Among those, Orth-Cali (Chuang et al., 2023) only debiases the VLM’s text representations, and Contrastive Adapter (Zhang & Ré, 2022) debiases only the VLM’s image representations.

Moreover, FairVLM itself can be adapted to debias only one of the modalities by setting $\tau_I$ or $\tau_T$ to zero. In either case, we will have a closed-form solution for the encoder parameters in a single shot and will not need to perform any alternating optimization. 

Our motivation for debiasing text and image encoders stems from prior studies demonstrating bias in both image encoders [R1-R3] and text encoders [R4-R8]. Subsequently, to achieve our goal of proposing a method that can perform under different settings and conditions of debiasing, we made FairVLM controllable so that it can easily be adapted to an unimodal debiasing setting by only setting $\tau_I$ or $\tau_T$ to zero.

Reviewer comment (Theory):
While the authors argue that RKHS has nice universal approximation properties, it's unclear how and why they aid in debiasing the original representations.

Author response:
The key to the effectiveness of FairVLM is debiasing with HSIC type measure of dependence, which measures non-linear dependence. HSIC is typically estimated through RKHS. And, as we explain on page 4, under the Choice of Encoder paragraph, RKHS lends itself to closed-form solutions for the optimization problem, which is vital when optimizing competing objectives. SGD-type algorithms are not well suited in such cases, as is common in GANs and other adversarial learning methods. We repeat our description in the paper below for completeness:

“Our choice of RKHS is motivated by several reasons. Debiasing is inherently an optimization problem with multiple competing objectives. In such cases, optimization is the primary bottleneck rather than model expressivity. This was also observed in Sadeghi et al. (2022). The closed-form solution afforded by our approach mitigates the optimization challenges (Sec. 4.3 and App. A.6).”

Moreover, in Section 3.1 and Figure 4, we provide a geometric illustration of how FairVLM works. Using a universal RKHS like RBF kernel maps the data into infinite dimensional space where all attributes are perhaps amenable to disentangling through linear mappings, thus allowing us to estimate the correct direction of $Y$ and $S$ labels. In this space, a linear correlation can capture all types of dependencies, i.e., all linear and non-linear dependencies.

Reviewer comment (Evaluation):
While one of core features of VLM is zero-shot classification, the suggested method still requires parameter tuning (RBF kernel parameter) and stacking test data, which could be a limitation of the method.

Author response:
****A1**** Please recall that we consider two scenarios. One where we have ground truth $Y$ labels and the other without $Y$ labels. In both cases, ground truth $S$ labels are not available. 

In both cases, we have a train-test split where we learn on the train split and evaluate on the test split. So, in both cases, there is no stacking of test data, and we can evaluate a single test sample at a time. We stack the train data in both cases and select the RBF kernel parameter through cross-validation. This experimental design choice was motivated by supervised baselines, which train with ground truth $Y$. We do not perform any online updates for FairVLM.

The reviewer’s comment suggests an alternative scenario where one learns FairVLM at test time and applies the learned model to the same test samples. In this scenario, yes, we have to stack the test data. However, even in this scenario, it is possible to pre-train FairVLM on a separate set of training samples (only images, without labels) obtained through other means and apply it to the test sample without training. For a completely novel task where samples for training (only images, without labels) cannot be obtained, we will have to stack test data for learning FairVLM. And, as we demonstrate, FairVLM can learn from limited data (e.g., CFD dataset with 597 samples).

Reviewer comment (Experiments):
Can this method be extended to a single point debiasing? (i.e. a single point inference for online prediction?)

Author response:
****A2**** Yes, as we explained in response to the previous comment, we do not train at test time but instead train on a separate dataset. So at test time, we can debias a single point.

Reviewer comment (Evaluation):
Table 3 CFD results look interesting! Does it imply FairVLM has its strength when the number of training samples is limited? How do other zero-shot methods in CFD dataset?

Author response:
****A4**** Our motivation for conducting this experiment was to evaluate FairVLM and other baselines in an extremely challenging setting. As mentioned in the paper, the CFD experiment has two characteristics that make it difficult.

1. The number of training samples is minimal (597 samples), which might not be sufficient for models with a large number of parameters.
2. The performance of the zero-shot classifier for this task suggests that the image and text features from CLIP are not aligned with each other for the worst group, leading to incorrect predictions for $\hat{S}$ and $\hat{Y}$. In response to the second question, we now evaluated the zero-shot prediction performance of 108 publicly available models from the OpenCLIP repository to verify if CLIP is an outlier. Below, we present the results of a select few models in Table R1. The results clearly indicate that this task is challenging for most of the VLMs, as reflected in their low average and worst group accuracy. The results for the entire list of 108 models are too long to include in the paper. So, we will make them available along with the code in a CSV file.

****************Table R3: CFD results for other VLMs****************

| OpenCLIP Models | #Parameters (Millions) | Pretrained Dataset | WG | Avg | Gap |
| --- | --- | --- | --- | --- | --- |
| ViT-SO400M-14-SigLIP-384 | 877.96 | WebLI | 10.81 | 50.83 | 40.02 |
| ViT-bigG-14-CLIPA-336 | 2,517.76 | DataComp1B | 0 | 55.0 | 55.0 |
| ViT-SO400M-14-SigLIP | 877.36 | WebLI | 0 | 55.0 | 55.0 |
| ViT-H-14-CLIPA-336 | 968.64 | DataComp1B | 11.0 | 54.17 | 43.17 |
| convnext_xxlarge | 1,200.58 | LAION2B | 34.88 | 57.5 | 17.14 |
| EVA01-g-14 | 1,136.44 | LAION400M | 2.33 | 55.0 | 52.67 |
| RN101 | 119.69 | OpenAI | 0 | 52.5 | 52.5 |
| RN50 | 102.01 | OpenAI | 0 | 50.83 | 50.83 |

Reviewer comment (Theory):
Is there any convergence guarantee for Algorithm 1 (FairVLM Training Without Labels)? Also, I am wondering if there can be a failure mode that the errors in the initialization step propagate further in iteration steps.

Author response:
****A5**** There is no theoretical convergence guarantee for Algorithm 1 apart from the closed-form solvers in each iteration. But empirically, we observed that the algorithm converges to local optima across all our experiments. For tasks where $Y$ labels are available, FairVLM typically converged in just one iteration and saturated its best performance. In the case w/o ground truth $Y$ labels, we found that it converged in less than five iterations when its best performance was also observed. In this case, the extra iterations might be since we also update the target attribute’s pseudo-labels in each iteration.

Across all the experiments presented in the paper, we did not notice the algorithm fail, at least not catastrophically, as the other approaches did in the CFD experiment. Even in the CFD dataset case, where the error in the initial zero-shot predictions is very high, we did not observe FairVLM fail as the results demonstrate. However, our observations do not preclude possible failures, either due to the scenario described by the reviewer or otherwise. We did not encounter any failures in *our* experiments.

Reviewer comment (Novelty):
Possibly related works  
- Chen, A. S., Lee, Y., Setlur, A., Levine, S., & Finn, C. (2023). Project and Probe: Sample-Efficient Domain Adaptation by Interpolating Orthogonal Features. *arXiv preprint arXiv:2302.05441*.  
- Adila, D., Shin, C., Cai, L., & Sala, F. (2023). Zero-Shot Robustification of Zero-Shot Models With Foundation Models. *arXiv preprint arXiv:2309.04344*.  
- An, B., Zhu, S., Panaitescu-Liess, M. A., Mummadi, C. K., & Huang, F. (2023, July). More Context, Less Distraction: Improving Zero-Shot Inference of CLIP by Inferring and Describing Spurious Features. In *Workshop on Efficient Systems for Foundation Models@ ICML2023*.

Author response:
****A6**** Thank you for the suggestions. These papers seek to mitigate only spurious correlations and focus on domain generalization or robustness. (1) uses partial labels while we consider full or no labels. (2) does not use labels for debiasing, but performance is lower than some of the baselines we already compare against. (3) is a training-free debiasing approach. We have added them to the updated paper at appropriate places.

Reviewer comment (Evaluation):
The experiment results on the datasets (w/ labels) are not good enough.

Author response:
As pointed out by the reviewer as a strength, one of the main contributions of FairVLM is its ability to adapt to different settings to mitigate various types of biases. In this paper, we have conducted several experiments on FairVLM, including mitigating spurious correlation, mitigating intrinsic dependencies, and conducting experiments on datasets with limited training samples and both w/ and w/o ground-truth labels. In all of these settings, FairVLM either outperforms other baselines, e.g.,

- In Table 2, WG and Avg. increased for all experiments (except CelebA on CLIP ResNet-50) in w/o label settings,
- Chicago Face Dataset (CFD) in Table 3 (right)
- Reduce MaxSkew fairness metric for the FairFace dataset in Table 3 (left).
- Reduce the fairness metric EOD on the CelebA dataset in Table 1.
- Improve the WG accuracy and decrease the Gap for experiments on CLIP ViT-L/14 in Table 2.

Or it is second best in other experiments except for CelebA for CLIP ResNet-50 in w/ ground-truth labels.

Furthermore, FairVLM enjoys computational advantages for training (see Table 4). On the Waterbirds dataset, FairVLM is 40 times more efficient than Contrastive Adapter, which has some accuracy advantages over FairVLM in two of the experiments on CLIP ResNet-50 and is five times more efficient than the other baselines. On the CelebA dataset that contains more samples, FairVLM is more than ten times faster than the model with the best accuracy, DRF (Upsample), and about 100 times faster than the Contrastive Adapter, which is the second-best method in terms of accuracy in that specific setting. It is worth mentioning that since the FairVLM employs kernel methods in its encoders, it can be scaled to medium-sized datasets using Random Fourier Features [R1], while the other baselines are not efficient even for the relatively small-sized datasets (CelebA and Waterbirds), we evaluated.

In conclusion, based on the information provided above, we maintain a robust belief that **suboptimal outcomes against specialized solutions occur in only two out of thirteen experiments.** These results amply demonstrate the overall value and effectiveness of FairVLM's contributions.

Reviewer comment (Novelty):
The pare of the method is too complex.

Author response:
Our solution employs kernel methods, which are part of the standard machine learning toolbox. Fig 4 in Section 3.1 provides a geometric illustration of FairVLM. Moreover, Fig. 2 provides an overview of the training and inference phases of the proposed method, and Fig. 3 explains our training algorithm as a figure.

Reviewer comment (Writing):
The terminology "VLM" is misused in this paper. VLM literally includes a vast area of models trained with vision and language, such as visual-question answering (VQA) models, vision-language pre-training (VLP) models with cross-attention transformers (e.g., ViLBERT [R2], ViLT [R3], Align [R4], VinVL [R5], ALBEF [R6], BLIP [R7]), multi-modal generation models (e.g., dall-e 1, 2 and 3, stable diffusion, dreambooth), and recent language-model combined vision models (e.g., BLIP2 [R8], Fromage [R9], GPT-4). However, the scope of this paper is very narrow compared to the entire VLM family.

Author response:
****A1**** We thank the reviewer for detailing the generality of the term “VLM.” We narrowed the focus of our presentation to “debiasing CLIP embeddings for zero-shot predictions.” We renamed our approach to FairerCLIP and rephrased all instances of VLM with CLIP or zero-shot predictions with CLIP, as appropriate. We also changed the paper's title to “FairerCLIP: Mitigating Bias in CLIP Representations for Zero-Shot Prediction.”

**To avoid confusing the other reviewers, we continue to use the term “FairVLM” in the rebuttal.**

Reviewer comment (Novelty):
The title "FairVLM" and the terminology "VLM" are overclaimed and can mislead readers about the paper’s focus, which should be toned down and made more specific. This paper only targets a feature refinement method on top of the frozen CLIP model, and the comparison methods are also based on frozen CLIP encoders. This is still a narrow topic within "de-biasing CLIP zero-shot prediction," especially since many works focus on fine-tuning strategies [R10-12], whereas this paper relies solely on pre-trained feature encoders, potentially weakening its contribution.

Author response:
****A2**** We have renamed our proposed method changed the title, and toned down our contributions and focus to “debiasing CLIP embeddings for zero-shot predictions.” We hope that our changes address the reviewer’s concerns.

While our approach operates on a frozen CLIP model, some of the baselines we compare against perform fine-tuning on at least one layer in CLIP if not all.

- **ERM Linear Probe** fine-tunes the last linear layer of CLIP.
- **ERM Adapter** adds new linear layers and fine-tunes them
- **DFR** fine-tunes the last layer of the CLIP

Despite not fine-tuning CLIP, FairVLM outperforms those that fine-tune CLIP, reiterating its effectiveness.

Reviewer comment (Novelty):
There are a number of missing related works that should be discussed, including those already mentioned in [R1-12]. For example, HSIC is a popular approach in de-biasing studies [R13, R14], where R13 directly optimizes the HSIC between features and sensitive attribute labels (similar to Dep(Z, Y) in the paper), and R14 optimizes the HSIC between biased and target features to avoid using sensitive labels. Extending to RHKS, work using MMD to achieve fairness has also been done [R15]. Many HSIC-based regularization methods relevant to this paper are omitted, and more citations should be added.

Author response:
****A4**** Thank you for providing such a comprehensive list of suggestions for related work; we really appreciate it. We added the three suggested papers (R13-R15) to the related discussion on page 3 of the paper in the **Choice of Dependence Measure** paragraph. There are potentially other methods that use HSIC as a regularizer. However, due to page limits for the main paper, we could not add more of them at this time.

Reviewer comment (Reproducibility):
The paper lacks clarity in explaining implementation details. It is unclear how the "encoder" works—although it is presumed that encoder parameter $\Theta$ is a linear projection and an RBF kernel is used for HSIC computation, this is not specified. There is no description or value given for $r$ (which becomes the dimensionality of the generated representation), and no details are provided for RKHS hyperparameters (e.g., projection layer dimensionality, kernel hyperparameters). Similarly, choices of learning hyperparameters such as the number of iterations and batch size are not specified. This omission makes it impossible or extremely difficult to reproduce the results. Overall, the method and implementation details are very hard to understand.

Author response:
**A6** For clarity, we report the details below and point to the parts of the paper where they are described.

$Z := [Z_1, Z_2, \dots, Z_r] \text{ where } Z_j = f_j(X), f_j \in \mathcal{H}_X \text{ } \forall j=\{1,2,\dots,r\}$ (this is in Equation 5): $Z$ is the generated representation

$\mathbf{f}(X) = \mathbf{\Theta}[k_x(x_1, X), \dots, k_x(x_n, X)]^T, \text{ where } \mathbf{\Theta} \in \mathbb{R}^{r\times n}$, $k_x(\cdot,\cdot)$ is the kernel and $n$ is the number of training samples. (this is in Theorem 2): Here $\textbf{f}$ is the encoder which is a linear projection on the RBF kernel.

$\text{Dep}(Z, S) :=\sum_{j=1}^r \sum_{\beta \in \mathcal U_S } \mathbb{C}\text{ov}^2\left(Z_j, \beta(S)\right)$ (this is in Equation 1): This is the definition of the simplified version of HSIC.

$\text{Dep}(Z, S) := \frac{1}{n^2}\left\|   \mathbf{\Theta}  \mathbf{K}_X  \mathbf{H}  \mathbf{L}_S \right\|^2_F$ (this is in Equation 2): If we plug $Z$ from the second equation into the third equation we will get this compact formula for calculating Dep between the representation and the sensitive attribute.

Here $Z \in \mathbb{R}^r$ is the debiased representation, and $ \mathbf{\Theta}$ are the parameters to learn.

Reviewer comment (Presentation):
The paper has a critical problem in its writing and presentation, with many missing related works and implementation details. The initial version is improper to be published as an ICLR paper due to these issues, but the concerns are mainly about writing and presentation quality rather than technical soundness. Clarification of all method implementation and training details is needed in a revised version, and significant effort will be required to improve presentation. If these concerns are resolved in revision, the review score could be updated.

Author response:
******A8****** The implementation details were present in Appendix A.4. We have updated it with more information and added a table (Table 6) for clarity. Upon acceptance of the paper, we will also release our source code for reproducing all experimental results presented in the paper. We have updated the manuscript to address the reviewer’s concerns by adding more related works and narrowing the claims of the paper. We also addressed the reviewer's comments on fine-tuning methods. Please let us know if there are any unresolved comments or further concerns.