Paper: Finetuning Text-to-Image Diffusion Models for Fairness (ICLR-2024)

Reviewer comment (Evaluation):
This paper kind of mixing fairness and bias and uses both terms interchangeably, especially for the experiment evaluation part, they define the metric for bias by themself which reduces the credential of the evaluation.

Author response:
We apologize for the confusion arising from our uncareful use of the terms fairness and bias. They indeed have different meanings in our paper. We consider fairness as the distributional alignment with a user-defined target distribution, a flexible notion. Since our main experiments use a perfectly balanced target distribution, we define the metric bias to measure how far the generated images are from the perfectly balanced target distribution. When evaluating fairness for age, where we consider 75% young and 25% old as the target distribution, we directly report the percentage of age=old. We added clarification in the second paragraph  of section 5.1, highlighted in blue.

We note that demographic parity (DP) and equal opportunity (EO) are defined for prediction problems, where every individual has a protected attribute $S$, a true label $Y$, and a prediction $\hat{Y}$ (chapter 3 of [1]). For T2I generative models, every generated image has only a protected attribute $S$. Therefore, DP and EO do not exactly apply here. In a way, the fairness problem in T2I generative models is simpler than that in prediction problems. Our objective is simply to control the generated images’ marginal distribution in terms of $S$ towards a target distribution.

We understand the reviewer has concern over whether the bias metric is truthful. In response, we have added plots of gender and racial representations by each prompt in Appendix Fig. A.2, A.3, A.4, and A.5, specifically at page15 & 16. These results provide a more detailed analysis than those presented in Tab. 1.

Reviewer comment (Evaluation):
I wonder if any other metrics from other research papers have been used for evaluation.

Author response:
We apologize for the confusion arising from our uncareful use of the terms fairness and bias. They indeed have different meanings in our paper. We consider fairness as the distributional alignment with a user-defined target distribution, a flexible notion. Since our main experiments use a perfectly balanced target distribution, we define the metric bias to measure how far the generated images are from the perfectly balanced target distribution. When evaluating fairness for age, where we consider 75% young and 25% old as the target distribution, we directly report the percentage of age=old. We added clarification in the second paragraph  of section 5.1, highlighted in blue.

We note that demographic parity (DP) and equal opportunity (EO) are defined for prediction problems, where every individual has a protected attribute $S$, a true label $Y$, and a prediction $\hat{Y}$ (chapter 3 of [1]). For T2I generative models, every generated image has only a protected attribute $S$. Therefore, DP and EO do not exactly apply here. In a way, the fairness problem in T2I generative models is simpler than that in prediction problems. Our objective is simply to control the generated images’ marginal distribution in terms of $S$ towards a target distribution.

We understand the reviewer has concern over whether the bias metric is truthful. In response, we have added plots of gender and racial representations by each prompt in Appendix Fig. A.2, A.3, A.4, and A.5, specifically at page15 & 16. These results provide a more detailed analysis than those presented in Tab. 1.

Reviewer comment (Evaluation):
Is it possible to use well-defined fairness metrics like demographic parity/ equal opportunity, etc?

Author response:
We apologize for the confusion arising from our uncareful use of the terms fairness and bias. They indeed have different meanings in our paper. We consider fairness as the distributional alignment with a user-defined target distribution, a flexible notion. Since our main experiments use a perfectly balanced target distribution, we define the metric bias to measure how far the generated images are from the perfectly balanced target distribution. When evaluating fairness for age, where we consider 75% young and 25% old as the target distribution, we directly report the percentage of age=old. We added clarification in the second paragraph  of section 5.1, highlighted in blue.

We note that demographic parity (DP) and equal opportunity (EO) are defined for prediction problems, where every individual has a protected attribute $S$, a true label $Y$, and a prediction $\hat{Y}$ (chapter 3 of [1]). For T2I generative models, every generated image has only a protected attribute $S$. Therefore, DP and EO do not exactly apply here. In a way, the fairness problem in T2I generative models is simpler than that in prediction problems. Our objective is simply to control the generated images’ marginal distribution in terms of $S$ towards a target distribution.

We understand the reviewer has concern over whether the bias metric is truthful. In response, we have added plots of gender and racial representations by each prompt in Appendix Fig. A.2, A.3, A.4, and A.5, specifically at page15 & 16. These results provide a more detailed analysis than those presented in Tab. 1.

Reviewer comment (Writing):
For the equation 4, what is the u?

Author response:
We have significantly revised Section 4.1 to enhance the clarity of our presentation. Eq. 4 in the initial paper corresponds to Eq. 4 and 5 in the updated paper. $\\{\boldsymbol{u}_1,\cdots,\boldsymbol{u}_N\\}$ are $N$ iid samples drawn from the target distribution $\mathcal{D}$. Say we use the one-hot vectors [0,1] and [1,0] to denote male and female, and consider a balanced target distribution over these two classes. Then approximately half of $\\{\boldsymbol{u}_1,\cdots,\boldsymbol{u}_N\\}$ will be the vector [0,1] and the other half will be [1,0]. We construct $\\{\boldsymbol{u}_1,\cdots,\boldsymbol{u}_N\\}$ in this way so that it represents the target distribution and the optimal transport finds the most efficient modification of the current images towards the target distribution.

Reviewer comment (Experiments):
As for the alignment loss, have you tried some other metrics other than the optimal transport loss? And why did you choose the optimal transport metric?

Author response:
We naturally arrived at the optimal transport (OT) formulation and didn’t experiment with other losses. We start by considering the following problem: suppose we have a set of 10 images, composed of 2 female and 8 male images, and our goal is to reconfigure this set to contain an equal number of female and male images, specifically 5 of each. The key question is determining which gender class every image should be reconfigured to. Additionally, it is essential to minimise alterations to the images when striving to attain an equal distribution of 5 female and 5 male images. This is naturally the OT (or linear assignment) problem, identifying which class every image should be transported to while minimising the transport distance. Finally, the OT problem considered here is low dimensional (2 for gender, 4 for race, and 8 for gender$\times$race) and therefore efficient to solve.

We sample $\\{\boldsymbol{u}_1,\cdots,\boldsymbol{u}_N\\}$ from the target distribution and compute the expectation of OT because it (1) acknowledges even a perfectly balanced distribution can still produce unbalanced finite samples, (2) takes into account the sensitivity of OT, and (3) produce a confidence measure for the target class assignment.

Reviewer comment (Evaluation):
CLIP is not without its own issues; see e.g. Shtedritski et al., Wang et al., Wolfe et al., or Zhang and Ré. Could the authors discuss how this might affect the efficacy of their proposed method?

Author response:
Thank you for pointing out our oversight regarding CLIP's own biases. We added acknowledgement in Section 4.1, page 4, highlighted in blue. We did not observe significant negative impact on our debiasing method arising from CLIP and DINO’s own social impacts.  But we did observe DINO’s ViT architecture that splits an image into fixed-size patches can cause faces to appear rectangular in finetuning during our initial experiments. This observation is one of the factors that led us to use both CLIP and DINO. 

The inherent image-image biases of CLIP and DINO, as opposed to text-image biases, could potentially affect our debiasing methods, given that we use CLIP and DINO to minimize image-image similarities. These biases might vary from the text-image biases documented in Shtedritski et al. [1], Wang et al. [2], Wolfe et al. [3], and Zhang and Ré [4]. We hypothesise that CLIP could be more biased than DINO in our context because CLIP is trained with text supervision while DINO is trained using image self-supervision. CLIP's training process, which aligns image features with text captions, is likely to introduce social biases. Our use of both CLIP and DINO might help mitigate the impact of these biases from CLIP.

Reviewer comment (Reproducibility):
How do the authors implement using different lambda values for different regions of the same image, as they describe at the end of section 4.1? ("We use a smaller weight for the non-face region ... and the smallest weight for the face region.") L_img is per image, not per pixel, no?

Author response:
$\mathcal{L}\_{\textrm{img}}$ can differ between the detected face region and non-face region in the same image, and therefore is per pixel. We implement $\mathcal{L}\_{\textrm{img}}$  by adding a gradient manipulation layer right before inputting the generated image $\boldsymbol{x}^{(i)}$ to CLIP & DINO. By scaling the gradients at the level of the generated image $\boldsymbol{x}^{(i)}$, we can effectively implement a per-pixel $\mathcal{L}_{\textrm{img}}$.

Reviewer comment (Experiments):
I was confused about the training of the gender and race classifiers for the evaluation. Were separate classifiers trained for the training and the evaluation stage? If yes, why? Or is this maybe some kind of unintended text duplication? ("The gender and race classifiers used for the evaluation loss are trained on the CelebA and FairFace datasets. ... Evaluation. We train new gender and race classifiers using CelebA and FairFace.")

Author response:
We trained separate gender and race classifiers for evaluation. We trained them for twice longer iterations, using the same architecture and training data. We do so in an effort to make the evaluation fairer. The debiasing method should not overfit to the classifier used during training and generates images that the training classifier views as fair but not the separately trained evaluation classifier. We added clarification in the revised paper.

Reviewer comment (Reproducibility):
It was unclear to me which of (soft prompt, text encoder, U-Net) were actually finetuned for all of the main experiments in section 5.1?

Author response:
For all of the main experiments, except explicitly stated otherwise, we finetune a LoRA adaptor with rank 50 applied on the text encoder. We added clarification in the first paragraph of Section 5.1, page 6, in the revised paper.

Reviewer comment (Evaluation):
Can the authors think of (and comment on) any drawbacks of applying their method at scale and for many fairness-relevant properties simultaneously? Could there be any negative consequences (e.g., in terms of image quality or biases) when e.g. generating very different images (bears, tables, groups of people, ... )? Recent observations by Qi et al. could also be interesting to discuss in this regard.

Author response:
Thank you for pointing us to consider the potential negative impacts of our debiasing finetuning. In response, we added an evaluation and discussion of the debiased SD’ generations for general prompts in Appendix page 17-22. We identify the debiased SD can occasionally and to a mild extent decrease the naturalness and smoothness of the generated images. Additionally, there are limitations in the generalizability of the debiasing effect. As illustrated in Figure A.11 on page 23, for the prompt "A beautiful painting of woman by Mandy Jurgens, Trending on artstation", the debiased SD increases the representation of Asian women, but does not similarly enhance the representation of Black and Indian women. We refer the reviewer to the more detailed discussion at Appendix Section A.5, specifically at page 14 & 17, highlighted in blue.

Reviewer comment (Writing):
The optimal transport approach in Eq. (4) was a bit confusing to me at first because it differs from the usual setting in which the optimal transport scheme between two *distributions* is considered. By contrast, the authors consider an expectation over the optimal transport schemes between two *vectors* here. I believe everything is actually described correctly, but maybe there is a way to make this section even easier / more intuitive for readers to grasp? (Sorry, possibly not very helpful.)

Author response:
Your observation is correct: we consider the optimal transport (OT), or the linear assignment problem, between two sets of vectors here, followed by computing the expectation of OT w.r.t. randomness of the target set of vectors $\\{\boldsymbol{u}\_1,\cdots,\boldsymbol{u}\_N\\}$. We are particularly grateful that you pointing out our presentation of Equation 4 was not clear. We have thoroughly revised Section 4.1 to enhance its clarity. Eq. (4) in the initial paper corresponds to Eq. (4) and (5) in the revised paper.

Reviewer comment (Writing):
It is a little bit confusing that y denotes both the generated target labels (normal font) as well as the images generated by the frozen model (boldface).

Author response:
We have changed $\boldsymbol{y}^{(i)}$ to $\boldsymbol{o}^{(i)}$ to denote the original images generated by the frozen model.

Reviewer comment (Novelty):
The authors might want to consider citing Lester et al. regarding soft prompt tuning?

Author response:
Thank you for highlighting relevant previous research on soft prompt tuning. We have acknowledged it in the second-to-last paragraph of the introduction, highlighted in blue.

Reviewer comment (Evaluation):
It is unclear whether fine-tuning with the fairness loss affects the quality and diversity of the generated images on general prompts.

Author response:
***W2: It is unclear whether fine-tuning with the fairness loss affects the quality and diversity of the generated images on general prompts.***

We have incorporated an evaluation of general prompts in the revised paper, on Appendix Section A.5, page 14 & 17~22. In short, the quality of the generated images remains largely unchanged but we do identify occasional and relatively minor reduction in naturalness and smoothness of the generated images. We observed no reduction in diversity of the generated images in general. We observed increased diversity for the prompt “beautiful painting of a woman by Mandy Jurgens, trending on ArtStation”, even though the model was debiased for templated occupational prompts and never for the term "woman". We refer the reviewer to the more detailed discussion at Appendix page 14 & 17, highlighted in blue. It analyzes the quality, diversity, and fairness of images generated from general prompts.

Reviewer comment (Evaluation):
Although experiments in the paper show better performance than baseline methods, it is unclear how expensive the fine-tuning is compared with the baseline methods.

Author response:
***W3: Although experiments in the paper show better performance than baseline methods, it is unclear how expensive the fine-tuning is compared with the baseline methods.***

Debiasing SD may require a one-time training cost or additional compute at every inference time. Our fine-tuning method generally requires more training compute than baselines. But it effectively scales to a large number of prompts and does not incur additional inference compute. In the second paragraph of the introduction (highlighted in blue), we recognized existing methods are more lightweight.

In terms of specific comparisons, Ethical Intervention is a prompting method and does not incur additional training or inference compute.  Based on our experiments, DebiasVL requires around 1 GPU-hour (using an A100) training. UCE requires around 24 GPU-hours to debias 50 prompts. But we find the required compute time increases drastically when we increase the number of prompts. Concept Algebra does not require training but doubles the inference cost, which can be undesirable. Our method requires 8 GPUs * 48 hours for debiasing 1000 prompts but does not increase inference compute.

We have included the training losses plot in Appendix Figure A.1, page 14. This plot indicates that although we train for 10k iterations for best performance, the finetuning achieves debiasing quite early, within 1k iterations, and can potentially be early stopped at 5k iterations. Therefore, the computational requirements for our method might not be as extensive as reported.

Reviewer comment (Reproducibility):
In Equation (4), how to obtain the expectation in practice?

Author response:
***Q1: In Equation (4), how to obtain the expectation in practice?***

We have significantly revised Section 4.1 to improve its clarity. Eq. 4 in the initial paper corresponds to Eq. 4 and 5 in the revised paper. In general, the expectation is approximated by repeatedly drawing $\\{\boldsymbol{u}\_1,\cdots,\boldsymbol{u}\_N\\}$ from the target distribution $\mathcal{D}^{N}$, computing the optimal transport from $\\{\boldsymbol{p}\_1,\cdots,\boldsymbol{p}\_N\\}$ to $\\{\boldsymbol{u}\_1,\cdots,\boldsymbol{u}\_N\\}$, and computing the average of these OTs. When the number of classes $K$ as well as $N$ is small, we can enumerate all possible samples from $\mathcal{D}^{N}$, compute OT for all of them, and analytically compute the expectation by weighing different OTs according to the target distribution.

Reviewer comment (Evaluation):
In Table 2, on unseen prompts, how good is the proposed method compared with the baselines? Does the fine-tuning have an overfitting problem?

Author response:
***Q2: In Table 2, on unseen prompts, how good is the proposed method compared with the baselines? Does the fine-tuning have an overfitting problem?***

We have included baselines comparisons in Table 2, page 7, highlighted in blue. We generally do not find overfitting, as demonstrated by the additional evaluation on general prompts (Appendix Section A.5, page 14 & 17~22). We did identify some negative impacts on image quality for the debiased SD, including occasional reduced naturalness and smoothness in the generated images. We discuss these in Appendix page 14 & 17, highlighted in blue.