Paper: Debias your VLM with Counterfactuals: A Unified Approach (ICLR-2024)

Reviewer comment (Novelty):
One of the primary weaknesses of the paper is its novelty in terms of the main framework. In order to infuse fairness, Agarwal et al. [1] introduced a triplet-based objective that maximizes the agreement between the original graph and its counterfactual views. Given that CVLD follows suit and incorporates a similar framework, the novelty is limited.

Author response:
Please see point “**Novelty**” in general comments. We appreciate the additional reference of NIFTY (Agarwal et al., 2021) which is conceptually similar and will acknowledge the work in the updated version, although we argue that the scope of the two works differ substantially: while NIFTY aims to fair and robust GNNs with a *known* underlying causal graph, our work involves vision-language transformer models on raw images and text as inputs, with the underlying graph *unknown* and simulated by image-text editing models. The primary contribution of CVLD is that it demonstrates the effectiveness of image-text editing models in debiasing VLMs across multiple downstream tasks with no architectural changes, which has not been demonstrated in prior work to our knowledge.

Reviewer comment (Evaluation):
In most cases, the counterfactual image seems noisy and is non-reflective of the counterfactual protected attribute (e.g., in Fig. 3, we don't observe women riding the rowboat). In such cases, are the counterfactual just some noisy version of the original image? How do we attribute the debiasing to a protected attribute if the quality of the counterfactuals is not good?

Author response:
Refer to “**Quality of counterfactual editing**” in general response. More examples of intervened images are provided in Fig. 7 and 8, highlighting the capabilities of editing models. Quantitative measures of edit quality are also included. We agree that the noise and imperfections in the counterfactual edits are unavoidable, but would like to emphasize that CVLD has shown its effectiveness for improving model fairness on standard metrics across classification, retrieval and captioning. Such improvements cannot be achieved simply by adding random noise to training images.

Reviewer comment (Experiments):
The framework is a data-extensive approach, i.e., for debiasing, it needs a counterfactual version of each image-text pair and expensive fine-tuning of VLMs for debiasing.

Author response:
See point “**Computational cost/overhead of CVLD**” in general response. It should also be noted that the experiments are deliberately conducted in a low-resource regime where we considered few-shot classification and fine-tuned retrieval/captioning on as low as 1% of COCO, which greatly reduces the cost from adapting to the full training set. CVLD is found to be effective across different scales of training set size.

Reviewer comment (Novelty):
One of the central claims of the work is that the proposed approach is a unified way to mitigate bias in VLMs across multiple tasks, as per the following evidence: In the title (Debias your VLM with Counterfactuals: A **Unified** Approach, bold text by myself). Also throughout text: e.g. In Section 2 "[...] we focus on a task-agnostic fairness framework for VLMs, unifying the study of bias across different tasks and domains." Claiming that the proposed approach is unified and task-agnostic seemed reasonable until Section 4. However, after reading through the details of how fine-tuning with synthetic data should be carried out for the three considered tasks, it seems to me that CVLD practical instantiation takes a very different format from task to task, rendering it a specific and not-unified framework for debiasing.

Author response:
## Unified approach.
We acknowledge that the framing of CVLD as “universal” and “task-agnostic” may have been misleading, since they refer to the data debiasing process, while the subsequent model adaptation still involves “a simple regularized training objective and minimal modifications to the adaptation pipeline” for each task, as mentioned in the Intro section. However, in defense of the title of the paper, we argue that CVLD, as a data-centric approach, has the distinctive advantage over model-centric mitigation methods in that it greatly reduces the effort to tailor the training pipeline to downstream tasks. The adaptation procedures in Sec. 4 can be distilled into two modes: For classification and captioning, the training is merely augmented by counterfactual examples with a weight parameter ($\lambda$), with no change to model computation at all. For retrieval, the contrastive loss itself is modified to account for counterfactual candidates in each batch, which reduces to simply modifying the optimization targets during training. The internals of the VLM is again unchanged, as illustrated in Fig. 3, meaning no overhead in deployment over a vanilla VLM (more discussion in “Computational cost/overhead of CVLD” in general comments). In contrast, model debiasing approaches tend to utilize additional modules, multiple training stages, and/or sophisticated loss functions, making them less trivial to adapt to a wide range of tasks. While there was not enough time to conduct the experiments, we believe that edited datasets produced by CVLD may also benefit tasks like detection/segmentation on COCO, as we find the edits to preserve the scene layout and background objects with high probability (Fig. 7 in appendix). In general, we believe that while not truly universal, CVLD is a significant step forward in minimizing the efforts needed to debias foundation VLMs on different downstream tasks.

Reviewer comment (Experiments):
One of the key parts of the introduced approach is the counterfactual data generation. However, the authors did not mention at any part of the manuscript details of the evaluation of the data generation pipeline. Moreover, it is not clear how the quality of generated counterfactuals could affect the performance of CVLD. Moreover, other fine-grained aspects such as *how much synthetic data is needed* and how the number of synthetic samples used at training time affects performance were not addressed in the manuscript, making it difficult to judge to what extent this framework would generalize to other scenarios where it might be difficult to generate high quality counterfactuals.

Author response:
## Evaluation of the data generation pipeline & relation to debiasing performance.
Refer “**Quality of counterfactual editing**” in general comments. We appreciate the suggestion to include additional evaluation metrics for the counterfactual intervention. It is difficult to conclude how the edit quality metrics relate to the eventual debiasing performance, as the metrics rely on pre-trained CLIP which itself may be biased.
However, the current version of CVLD performs sample selection based on visual similarity to the input image, and we observed in preliminary experiments that this strategy produces slightly better debiasing results than uniformly sampling edits at random, suggesting that poor quality images deviating from the input data may be detrimental.

In general, we believe that the ability of CVLD is bounded by the capabilities of the editing models, and that future study may be needed to establish a definitive link between edit quality and debiasing performance. However, as demonstrated in “Evaluation beyond gender bias” in general comments, we observe positive sign that the existing editing models are powerful enough for many commonly studied forms of bias in vision-language.

Reviewer comment (Evaluation):
Some parts of the text do not seem to reflect the actual insights that can be extracted from the results. For example, in the introduction, the authors mentioned that CVLD "demonstrates striking effectiveness for the most studied problems in the bias literature" while it is not clear from the results that the CVLD demonstrates **striking** effectiveness, neither there are references to support the statement that the considered problems in this work are the most studied ones in the bias literature.

Author response:
## Experimental results.
We would like to clarify that all evaluation datasets and metrics in this work are well-established standards in the literature. These include DEO for classification on CelebA (Ramaswamy et al., 2021), Bias and MaxSkew for retrieval on FairFace (Berg et al., 2022), BiasAmp and LIC for captioning on COCO (Hirota et al., 2022). For all fairness metrics lower is better, as is now indicated in the main tables of the paper. The fact that CVLD not only reduces model bias by large margins without sacrificing in-distribution performance, but also achieves so through simple image-text editing and no task-specific model debiasing, is particularly striking to us. 

In Tab. 1, we showed results of CVLD in both 16- and 256-shot settings, and compared them to naive BLIP using the same number of shots. In the first section of the table, BLIP-PT is not adapted to CelebA dataset while BLIP and RN-50 are fine-tuned on the full dataset, making them not directly comparable to the few-shot results below. Best results of each section of the table are highlighted in bold.

Reviewer comment (Evaluation):
The experimental results are a bit confusing and hard to parse. The employed metrics in the evaluation are not standard in the literature and it is not clear whether an improvement is observed when a metric increases or decreases. Moreover, it is not clear why some methods were grouped in different parts of the tables (e.g. in Table 1 it is not clear why both CVLDs are in different sections from BLIP-PT and the ResNet-50, aren't they directly comparable?). On a similar note, it is not clear how the different bolded numbers in the tables represent and how they should be compared against each other.

Author response:
## Experimental results.
We would like to clarify that all evaluation datasets and metrics in this work are well-established standards in the literature. These include DEO for classification on CelebA (Ramaswamy et al., 2021), Bias and MaxSkew for retrieval on FairFace (Berg et al., 2022), BiasAmp and LIC for captioning on COCO (Hirota et al., 2022). For all fairness metrics lower is better, as is now indicated in the main tables of the paper. The fact that CVLD not only reduces model bias by large margins without sacrificing in-distribution performance, but also achieves so through simple image-text editing and no task-specific model debiasing, is particularly striking to us. 

In Tab. 1, we showed results of CVLD in both 16- and 256-shot settings, and compared them to naive BLIP using the same number of shots. In the first section of the table, BLIP-PT is not adapted to CelebA dataset while BLIP and RN-50 are fine-tuned on the full dataset, making them not directly comparable to the few-shot results below. Best results of each section of the table are highlighted in bold.

Reviewer comment (Theory):
How can all the three different approaches to fine-tune VLMs with counterfactual data be seen as a unified framework? Also, how would this generalize to other tasks such as, for example, counting and object detection?

Author response:
## Unified approach.
We acknowledge that the framing of CVLD as “universal” and “task-agnostic” may have been misleading, since they refer to the data debiasing process, while the subsequent model adaptation still involves “a simple regularized training objective and minimal modifications to the adaptation pipeline” for each task, as mentioned in the Intro section. However, in defense of the title of the paper, we argue that CVLD, as a data-centric approach, has the distinctive advantage over model-centric mitigation methods in that it greatly reduces the effort to tailor the training pipeline to downstream tasks. The adaptation procedures in Sec. 4 can be distilled into two modes: For classification and captioning, the training is merely augmented by counterfactual examples with a weight parameter ($\lambda$), with no change to model computation at all. For retrieval, the contrastive loss itself is modified to account for counterfactual candidates in each batch, which reduces to simply modifying the optimization targets during training. The internals of the VLM is again unchanged, as illustrated in Fig. 3, meaning no overhead in deployment over a vanilla VLM (more discussion in “Computational cost/overhead of CVLD” in general comments). In contrast, model debiasing approaches tend to utilize additional modules, multiple training stages, and/or sophisticated loss functions, making them less trivial to adapt to a wide range of tasks. While there was not enough time to conduct the experiments, we believe that edited datasets produced by CVLD may also benefit tasks like detection/segmentation on COCO, as we find the edits to preserve the scene layout and background objects with high probability (Fig. 7 in appendix). In general, we believe that while not truly universal, CVLD is a significant step forward in minimizing the efforts needed to debias foundation VLMs on different downstream tasks.

Reviewer comment (Evaluation):
How is CVLD performance affected by counterfactual data generation quality? How did the authors assess the quality of generated data in order to know whether it was "good enough" to be employed for bias mitigation?

Author response:
## Evaluation of the data generation pipeline & relation to debiasing performance.
Refer “**Quality of counterfactual editing**” in general comments. We appreciate the suggestion to include additional evaluation metrics for the counterfactual intervention. It is difficult to conclude how the edit quality metrics relate to the eventual debiasing performance, as the metrics rely on pre-trained CLIP which itself may be biased.
However, the current version of CVLD performs sample selection based on visual similarity to the input image, and we observed in preliminary experiments that this strategy produces slightly better debiasing results than uniformly sampling edits at random, suggesting that poor quality images deviating from the input data may be detrimental.

In general, we believe that the ability of CVLD is bounded by the capabilities of the editing models, and that future study may be needed to establish a definitive link between edit quality and debiasing performance. However, as demonstrated in “Evaluation beyond gender bias” in general comments, we observe positive sign that the existing editing models are powerful enough for many commonly studied forms of bias in vision-language.

Reviewer comment (Experiments):
How is the performance of CVLD affected by the choice of the lambda hyperparameter?

Author response:
## Choice of the lambda hyperparameter.
For classification on CelebA, an ablation on choices of $\lambda$ is included in Fig. 4, and relevant discussions in Sec. 5.2. We include below the ablation results on captioning and retrieval tasks, both using COCO 1%.

- Retrieval
  | $\lambda$      | 0 (baseline) | 0.1  | 0.3  | 1    |
  | --------------- | ------------ | ---- | ---- | ---- |
  | MaxSkew$^{128}$ ↓ | .281         | .224 | .222 | .257 |
  | Rec@1 ↑         | 59.7         | 59.5 | 59.4 | 59.2 |
  

- Captioning
  | $\lambda$ | 0 (baseline) | 0.1   | 0.3    | 1      |
  | ---------- | ------------ | ----- | ------ | ------ |
  | BiasAmp ↓  | 0.91         | 0.25  | \-0.42 | \-1.18 |
  | CIDEr ↑    | 130.0        | 130.5 | 130.4  | 128.8  |  

It can be seen that $\lambda = 0.3$ achieves a nice balance between fairness and performance for retrieval and captioning.

Reviewer comment (Novelty):
Lack of novelty: The paper simply uses some state-of-the-art LLM to generate counterfactual text, and null text inversion/InstructPix2Pix to generate the counterfactual images.

Author response:
Please see point “**Novelty**” in general comments. CVLD showcases the effectiveness of image-text editing models in debiasing VLMs across multiple downstream tasks with no architectural changes, which has not been demonstrated in prior work. We consider the simplicity of the proposed framework a strength, as it does not require tweaking the internal model architecture or sophisticated training procedures, making the method more practical for tuning and deploying models at scale. It also makes the method complementary to model debiasing approaches, which may further benefit from the availability of debiased data.

Reviewer comment (Experiments):
The paper only covers gender biases - no experiments on other biases like racial/age. Biases may exist even in non-social cases (like the water-land bias in the popular Waterbirds dataset). This has not been explored.

Author response:
Please see “**Evaluation beyond gender bias**” in general response. Additional results on *background* bias on Waterbirds and *racial* bias on COCO are included, demonstrating that CVLD is generally applicable to various forms of biases. Our understanding is that CVLD is potentially useful for all settings where 1) bias can be described in natural language, and 2) corresponding causal interventions (e.g., change gender/race of subjects, or replace the background) can be reliably performed by the image-text editing models.

Reviewer comment (Experiments):
The paper advocates generating counterfactuals for bias mitigation. However, not many sample examples are shown even in the supplementary.

Author response:
As mentioned in “**Quality of counterfactual editing**” in general response, we have added qualitative results of generated images to the appendix of the paper, as well as a set of quantitative evaluation for edit quality.

Reviewer comment (Theory):
Not all biases (for example, models are seen to learn various spurious correlations like camels can only be present in deserts, airplanes can only be in the sky, etc) are quantifiable like gender. Is generating counterfactuals the solutions for those kinds of biases too?

Author response:
Please see “**Evaluation beyond gender bias**” in general response. Additional results on *background* bias on Waterbirds and *racial* bias on COCO are included, demonstrating that CVLD is generally applicable to various forms of biases. Our understanding is that CVLD is potentially useful for all settings where 1) bias can be described in natural language, and 2) corresponding causal interventions (e.g., change gender/race of subjects, or replace the background) can be reliably performed by the image-text editing models.

Reviewer comment (Experiments):
What if multiple biases are present at once? Like gender and race together. Can this method of generating counterfactuals handle such scenarios?

Author response:
Simultaneous mitigation of multiple biases is a great suggestion. One simple way is to extend CVLD with multiple edits, one for each protected attribute, which presents a challenge to the compositionality of editing models. However, as we intend for CVLD to become a baseline in data-centric VLM debiasing, we believe that the study of multiple bias attributes lies beyond the scope of this work and should be left for follow-up research.

Reviewer comment (Experiments):
**Limited Empirical Evaluation.** The proposed method has been evaluated almost exclusively on a specific type of bias---the gender bias. This is a very severe limitation for a paper which frames itself as targeting general bias in VLMs; the paper exemplifies racial bias multiple times in the text. If the authors are exclusively targeting the gender bias, a significant portion of this paper should be re-written to clarify this point.

Author response:
## Limited empirical evaluation.
Please see “**Evaluation beyond gender bias**” in general response. In short, while we experimented primarily with gender bias, CVLD is by no means exclusive to gender editing, and is as capable as the ability of the image-text editing models to alter protected attributes under text prompt. We show additional results on 1) background bias on Waterbirds, and 2) racial bias on COCO, to demonstrate the potential of the method.

Reviewer comment (Evaluation):
**Relies on external models, which may be prone to other types of bias.** The debiasing procedure of this paper relies on the generative/editing capabilities of existing models (e.g., prompt-to-prompt editing). This is a vulnerability in terms of a bias, because such edited images may be prone to other types of biases that may be difficult to detect (see, e.g., Bias-to-Text by Kim et al. (2023)). I wonder if authors could demonstrate any "robustness" of the proposed paradigm to the potential biases hidden in the LLMs or prompts.

Author response:
## Bias from editing models.
Thanks for the insightful suggestion. As discussed in “**Quality of counterfactual editing**” in general comments, it is true that the external editing models may be biased themselves, although very few studies have attempted to study this bias in a quantitative manner. Therefore, we resort to standard fairness benchmarks in classification, retrieval and captioning in the main paper for evaluating the debiasing quality. The fact that CVLD reduces model bias across dataset and tasks can be seen as strong evidence that the potential bias of external models is not enough to offset their effectiveness to remove spurious bias in the training data.

Although it is difficult to directly study the robustness of CVLD to the bias of LLMs/diffusion models, there are steps taken to reduce the potential impact. For example, by sampling multiple edits per image and keeping those most similar to the original, we minimize the chance of inadvertently altering image regions not causally related to the protected attribute (e.g., if some edits cause the color of the clothes or background objects to be changed while others preserve the original colors, the latter would be accepted).

Reviewer comment (Presentation):
**(minor) Clarity.** Figure 1 is not very informative and difficult to parse what the figure is trying to say. What the sketch part is trying to say is unclear (perhaps more details in the caption will be better). Also, it took me some time to notice that "M -> F" means male -> female. The "lock" figures are somewhat difficult to tell whether they are locked or unlocked (maybe use frozen <-> fire analogy, like many other papers, or use additional color cues?).

Author response:
## Improve clarity of figures.
Thanks for the suggestion. We have revised Fig. 1 and 3 to improve the clarity of modules and editing operations. We will continue to improve the readability of the paper.