Paper: Causal analysis of social bias in CLIP (ICLR-2024)

Reviewer comment (Novelty):
The novelty of the paper may be limited, as there is already abundant literature studying correlations between image and attribute embeddings in CLIP (many already cited in the related work). The contributions of this study are 1) using attributes from social psychology instead of a self-defined list of words, and 2) claiming causality instead of correlation. The first contribution may not be enough by itself, whereas the second contribution is challenged in the following point.

Author response:
**W1**: The reviewer's observation that our paper employs ``attributes from social psychology instead of a self-defined list of words" may not fully acknowledge the depth and scientific rigor associated with using psychometrically validated scales; We refer to [Furr (2021)](https://books.google.ch/books?hl=en&lr=&id=xto9EAAAQBAJ&oi=fnd&pg=PT14&dq=Psychometrics:+An+Introduction&ots=FeGG0pXykH&sig=qeu28wVVW03Ji9G1Hzv3H3_t2SE&redir_esc=y#v=onepage&q=Psychometrics%3A%20An%20Introduction&f=false) for an introduction to the field of Psychometrics. Moreover, the widespread adoption of the SCM and ABC models underscore their practical relevance in measuring stereotype perception. Thus, our decision to utilize validated scales in our study is not a mere preference for a different list of words but a deliberate choice to enhance the contribution of our research to the field.

Reviewer comment (Theory):
The paper argues that the analysis conducted on the CausalFace dataset, in contrast to FairFace, implies causality and not only correlation because the elements/confounders in CausalFace can be controlled by the generation process. However, it could be argued there is not enough evidence to assume there are no confounders in CausalFace just because it has been generated synthetically.

Author response:
**W2**: Yes, it is important to check for biases, artifacts, and confounds. If the referee has specific concerns, we will be the first ones to want to hear about them. The reviewer asks for "evidence of absence of confounders". Unfortunately, that is typically impossible to come by. However, the evidence for the presence of confounders is precious. We selected the Liang (2023) dataset precisely because it was carefully controlled. They verified that attribute manipulations were equally effective across different demographics using the ratings of five annotators per image (Fig. 3, Liang 2023). They checked for visible image artifacts (Sec 4.1) and excluded all images containing artifacts from their dataset. We have added to the manuscript a sentence discussing this (please also see response to referee 3zN2). Are we sure that all possible confounds and biases have been eliminated? Of course not. That would be foolish. In the history of science, there are many examples of important experiments where confounds and biases were hammered out year after year by generations of scientists. The crucial point is adopting the scientific method, which is based on experimentation, and experiments may be refined over time. We argue that measurements of bias in AI models now can, and thus must, move beyond correlations. Generative methods offer a practical way forward toward controlled experiments and causal conclusions. Thanks to progress in generative AI and constructive scrutiny from an engaged community of scholars, we expect our measurements to become more accurate with time.

Reviewer comment (Experiments):
Why only use 3 groups (Asian, Black, White) from FairFace instead of all the data?

Author response:
**Q2**: Our study compares FairFace to CausalFace; The latter however only includes three races: Asian, White, and Black. Some researchers might be interested in analyzing social perceptions across all FairFace races without comparing them to CausalFace. We will open-source our code, which allows those researchers to adapt our experiments to all racial categories in FairFace.

Reviewer comment (Writing):
I think the title should be a little more precise - specifically this is around "social bias in CLIP for face images" or something, it's not studying all areas of bias.

Author response:
**W1**: You are inviting us to make our title more specific and we agree:  *"A causal analysis of social bias towards faces in CLIP."*

Reviewer comment (Presentation):
Figure B.1 was unclear to me - I'm not sure how to read the point about within vs cross valence similarities off this plot.

Author response:
**W2**: We agree with the reviewer that the figure caption (now Figure C.3)  was unclear. The interpretation of the figure was moved to the appendix and a mathematical definition was added: *"Specifically, let $a^+_1$ and $a^+_2$ be two text embeddings of attributes of the positive valence agency dimension, and $a^-_1$ and $a^-_2$ be text embeddings of attributes of the negative valence agency dimension. The within-valence similarity is given by calculating the cosine similarities $ \cos{(a^+_i, a^+_j)}$ for all positive attribute pairs \( (i,j) \). The cross-valence similarity is determined by calculating $ \cos{(a^+_i, a^-_j)}$ for all mixed attribute pairs \( (i,j) \)."*

Reviewer comment (Evaluation):
Sec 4.2: I get a little confused about a few of the evaluation procedures here. For instance, I think "mean cosine similarities" could be explained a bit more - I can guess what it might be but don't know for sure. Also I think markedness is not explained so clearly: not clear what "relative preference frequency" is or a "neutral prompt" - again I could guess but should be written out.

Author response:
**W3a**: We agree that mean cosine similarities were not defined sufficiently. For clarification, we moved Eq. 1 from the appendix to the main paper (see. Sec. 3.1.). In addition, we added further explanation in the results section when discussing Table 1, which reads as follows: *"Mean cosine similarities are calculated as the mean of cosine similarities between an image category and all attribute categories of SCM and the positive attribute categories of ABC (see Equation (1))"*.  

**W3b**: We added appendix A.3 to further explain and mathematically define markedness.

Reviewer comment (Experiments):
I think the point about positive correlation between positive and negative terms is a good one, I think this could be shown more strongly with a "control group" of words: is it that some images are just correlated to all words? words of a certain type?

Author response:
**W4**: We show the positive correlation of positive and negative valence attributes in Figure 2. The high correlation is however only visually assessed. We agree with the reviewer that we should calculate a correlation coefficient. We did so for each dimension (agency, belief, communion) and for each intersecting group separately. We plot and interpret a heatmap of correlations in Figure C.4.

Reviewer comment (Writing):
The word "intensity" is used without definition - what does this mean?

Author response:
**W5**: Indeed, it was not obvious what the term "intensity" means in the context or study. Therefore, we added the following explanation when it was first introduced: *"In this context, we use valence as the direction of change in cosine similarity (increasing or decreasing) and intensity as the magnitude or absolute difference in change."*

Reviewer comment (Reproducibility):
Sec 3.4: unclear how these thresholds are determined or what they really mean (e.g. 0.7 for age).

Author response:
**SN2**: To maintain consistency with Liang (2023), we kept the original scales for smiling (-2.5 to 4) and age (0.8 to 4.4), as shown in Figures D.2 and D.3. For smiling, the non-equidistant levels range from -2.5 to 4. We selected a threshold of 0.7 to distinguish between sufficiently different smiling faces, deliberately excluding the 0.5 increments. A similar approach was applied to age. To avoid confusing a future reader, we introduced the following footnote: *"The effect of the thresholds can be better understood by looking at the scale of Figures D.2 and D.3"*

Reviewer comment (Presentation):
What is the x-axis in Fig 3a?

Author response:
**SN3**: To avoid redundancy between the title and the x-axis label in Figure 3a, the x-axes are unlabeled. To clarify, we have added to the figure caption: *'The three panels depict the variations in smiling, lighting, and pose on the x-axis.'*

Reviewer comment (Evaluation):
Fig 3b and commentary at end of 4.4: if I'm reading this correctly, most groups’ negative associations decrease as smiles intensify - it's stated here that this is unusual for black males.

Author response:
**SN4**: We agree that we should have been more precise with the use of language at this point. The patterns of all gender-race intersections look similar, with one exception: Male-blacks. Only for this group does smiling lead to an increase in positive social perception. The interpretation needs to be more nuanced. There is a subtle but existing difference between stating a "glass is half full" versus "a glass is half empty". Similarly, there is a subtle but existing difference between describing someone with less negative terms or more positive terms. Black males are the only group that is described predominantly with more positive terms as the smile increases. We changed the corresponding description in the paper to: *"For "belief", only black males deviate from the pattern consistently observed for the remaining subgroups."*

Reviewer comment (Theory):
I think the observation that "the widely held belief that age- and gender-induced variations are strong factors needs to be reconsidered" is misguided (end of 4.5 and 5). The authors seem to believe that social factors like gender receive special attention since they are causes of particularly strong deviations - I think this misses an important point. It isn't that these factors necessarily cause the largest deviations (of course other visibly salient factors will matter), it's that these factors are important for socially determined reasons, and therefore measuring model impacts and mitigating disparities/harms where they exist matters more.

Author response:
**Q1**: We appreciate the reviewer's insightful feedback on our discussion of variations in legally protected versus non-protected variables. We acknowledge that the original phrasing might have led to misunderstandings. We concur with the reviewer that gender, race, and age are socially significant factors, meriting the focused research they receive. Their importance indeed underscores our call for greater precision in measurement and a thorough examination of potential confounds in statistical analyses. To clarify our position, we have revised the statement at the end of section 4.4: *"Our observations suggest that non-protected attributes are important and should be taken into consideration to obtain a clear measurement of the socially relevant age- and gender-induced variations."*

Reviewer comment (Experiments):
The first limitation is that the authors use a single dataset only in their analysis, which is FairFace. There are other datasets such as UTK Face and CelebA that can be included to support the argument further. It's not clear if the conclusions in FairFace would continue to hold so showing that they hold in other datasets would strengthen the argument. In addition, MIAP (https://paperswithcode.com/dataset/miap) is quite different from FairFace in that it collects images in natural settings (not just face images) so I would expect those confounding factors to be even more prominent in that dataset. This would be a useful message to point out.

Author response:
We appreciate the reviewer's suggestion regarding the selection of face datasets for our analysis. In response, we have thoroughly evaluated each of the proposed datasets and would like to share our rationale behind their consideration: MIAP does not have face crops and, therefore, would introduce a significant amount of noise from contextual elements. These are additional confounding factors that could result in visible differences compared to FairFace. It is not the main goal of our paper to control for these types of confounds; instead, we focus on face-related confounds. MIAP would, therefore, not directly support our findings, and thus, we decided to focus on face-cropped datasets. CelebA meets our criteria but we do not consider it ideal for assessing biases in race, gender, or age groups for the following reason: It is likely that celebrities occur in the training data of CLIP. Therefore, we would assume that embeddings are overly specific to individuals rather than representative of broader demographic groups. Unlike FairFace, which includes only a limited selection of celebrities, CelebA is comprised entirely of celebrity individuals. We recognize UTKFaces as a viable alternative to FairFace for our analysis. We are in the process of extending our study to include this dataset and plan to integrate these results before the end of the rebuttal period.

Reviewer comment (Experiments):
The authors focus on sentiment related attributes, such as warmth, communion, competence, etc. It would make sense for these attributes to be impacted by confounders, such as pose and smiling. But, there are other equally important association biases, such as relating gender with occupation, that may not be as sensitive to those issues. These are not studied in the paper.

Author response:
The reviewer hypothesizes that cosine similarities between warmth/competence-related text prompts and images might fluctuate more significantly due to varying smile/pose levels than those between job type-related prompts and images with similarly varying smiling/pose levels. This is an interesting hypothesis and we do not investigate the relation of smiling/pose to occupation in generative AI and we are not aware of literature that does so. However, research investigating humans suggests that facial attributes do systematically vary with occupation: Facial appearance matters for leader selection [(Stocker 2016)](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0159950). More specifically, the width-to-height ratio seems to distinguish CEOs [(Lewis 2013)](https://www.sciencedirect.com/science/article/abs/pii/S0191886912000049) . 
Furthermore, men and women smiled more in response to a low-status job than to a high-status job [(Vrugt 2002)](https://onlinelibrary.wiley.com/doi/10.1002/ejsp.99)--suggesting that smiling may be a more significant predictor of job type than gender. More generally, smiling has been found to differ across age and gender [(Morse 2010](https://www.tandfonline.com/doi/abs/10.1080/00223980.1982.9915318),
[LaFrance 2003](https://pubmed.ncbi.nlm.nih.gov/12696842/),[Otta 1998](https://pubmed.ncbi.nlm.nih.gov/9923165/),
[Doff 1999](https://link.springer.com/article/10.1007/BF03395325),
[DeSantis 2005)](https://pubmed.ncbi.nlm.nih.gov/16342596/). Therefore, we assume that occupation is also sensitive to smiling/pose. 
Consequently, there is no clear answer to the question of which outcome metric is more sensitive to varying levels of smiling/pose. 
This is an interesting question to ask therefore, we add the following paragraph to the discussion: *"Our study uses social perception as the primary outcome metric, revealing that smiling and pose significantly affect this measure. Research in human contexts shows that smiling in particular, influences not only social but also job-related perceptions. Future research could focus on practical metrics such as occupation, investigating how varying degrees of smiling and pose impact these aspects in generative AI."*

Reviewer comment (Writing):
What does "neutral lightning" mean? It seems from Figure 3 that neutral lightning may lie at the middle of the scale, which would reveal a pattern that is different from random noise.

Author response:
**Q1**: We could not locate the exact use of 'neutral lighting' in our text, could you please point it out to us? Figure C.5 illustrates varying lighting levels, with the leftmost column (-1) representing the 'default lighting' that is also used in all other images that are not modified along the lighting dimensions. Lighting values 0-6 describe specific lighting from different directions. We welcome the reviewer's insight on how these variations might introduce systematic variation rather than random noise, as we have yet to find a fitting interpretation.

Reviewer comment (Experiments):
When comparing the impact of "smiling" in Figure 3.b, why did you choose to compare -1.5 with 3.0 instead of going for the extreme ends; i.e. -2.5 with 4.0?

Author response:
**Q2**: We have corrected the error in the legend. Upon review, it's evident that the plot includes all smiling levels from -2.5 to 4.0, as can be confirmed by counting the markers of various sizes.

Reviewer comment (Evaluation):
In Section 4.4, the authors claim that "black males" deviate from the expected pattern in Figure 3.b because they are "the only subgroup showing amplified positive associations with intensified smiles." I'm curious to know why the authors think this is unexpected? Shouldn't the fact that the other groups don't exhibit this be the unexpected pattern?

Author response:
**Q3**: We agree that we should have been more precise with the use of language at this point. The patterns of all gender-race intersections look similar, with one exception: Male-blacks. Only for this group does smiling lead to an increase in positive social perception. The interpretation needs to be more nuanced. There is a subtle but existing difference between stating a "glass is half full" versus "a glass is half empty". Similarly, there is a subtle but existing difference between describing someone with less negative terms or more positive terms. Black males are the only group that is described predominantly with more positive terms as the smile increases. We changed the corresponding description in the paper to: *"For ``belief'', only black males deviate from the pattern consistently observed for the remaining subgroups."*

Reviewer comment (Reproducibility):
Which specific prompts did the authors use to create various levels of "pose," "lightning", and "smile"?

Author response:
**W2b**: We do not generate synthetic images ourselves but the dataset was pre-generated and made available to us by Liang (2023).
Smiling, lighting, and pose are dimensions generated through latent space traversals using a GAN to create face images. The specific manifestations of these dimensions in the images are illustrated in Appendix D. Upon reviewing our paper's section on image datasets, we acknowledge a potential source of confusion. Initially, we mentioned "we generate variations," which we have now revised to *"we sample variations"* to better reflect our methodology.

Reviewer comment (Writing):
**Clarity:** I found the paper little difficult to follow especially in the sections 3.2 in which the authors describe about theoretical frameworks of social phycology. The authors can provide more context about these frameworks and how are they related in the proposed study in the form of examples, figures etc.

Author response:
**W1**: To address the reviewer's concerns concerning clarity, we have made the following revisions: Firstly, we revised section (formerly) 3.2/ now 3.1 and additionally include mathematical notation. Additionally, we improved the caption of Figure B1, which now conveys our methodology more transparently. Secondly, recognizing the constraints of paper length, we have introduced Appendix A.4, which contains a detailed discussion of social psychology's view on stereotypes, and add additional references. This should provide an in-depth background for readers with a specific interest in the theoretical underpinnings of our study.  We also made concise changes to section 2.2 to logically connected both models by stating: 
*"A more recent framework, the ABC Model, Koch(2016), proposes beliefs as alternative central dimension, and subdivides the Warmth and Competence differently into two categories."* Furthermore, we introduce both theories by stating that *"social psychology has long been recognized for delineating the primary dimensions of beliefs about stereotypes."* Thirdly, we justified the integration of social psychology frameworks within our study as follows in 2.2: *"Given the reflection of human-like biases in Generative AI, our study employs two leading theoretical frameworks from social psychology—long recognized for quantifying cognitive biases—to measure stereotypes systematically"*. In all sections, we have striven for consistency in terminology, using "dimensions" and "attributes" to describe the components of the SCM and ABC models consistently. We hope these changes address your concerns, and we invite your further feedback to refine our manuscript.

Reviewer comment (Experiments):
**Unaddressed questions about biases in synthetic datasets:** The authors didn't include discussion about potential limitations in the synthetic faces created using GANs.

Author response:
**W2**: Thank you for pointing this out. Yes, we should have commented on this. We have included the following statement in the discussion: *"Generative models, in particular GANs and Diffusion Models, can now generate face images that look realistic to human observers, and one can effectively control some of the facial attributes. There are three caveats. First, manipulating one attribute may induce unwanted changes in other attributes;  Balakrishnan (2021) show how to address this issue systematically, and it should be checked by visual inspection, as was done for the dataset we use (Liang 2023). Second, it is possible that some physiognomies are generated more frequently than others. E.g., within the European group, it is possible that more Mediterranean-looking and fewer Scandinavian-looking physiognomies are generated. If this were a concern for a study, one would have to come up with a fine-grained definition of race and control this attribute directly, as our dataset does for the main racial groups. Third, it is possible that facial attribute manipulations are more or less effective for some or the other demographic groups. For the dataset we use, this was excluded by direct verification with human annotators (Liang 2023, Fig. 3.). Generative methods and techniques to validate their output are progressing fast. Thus, the experimental method we advocate, based on carefully controlled synthetic data, will become an increasingly attractive option for researchers."*

Reviewer comment (Experiments):
Can these findings be applicable to other vision-language models?

Author response:
**Q1**: One could expect that bias stems from training data (i.e., text-image pairs) rather than the model architecture. Therefore, we would expect that our findings apply to other models if they are trained on similar datasets. We present our method primarily as a tool for investigating biases in such models. Following the reviewer's question, we have updated the discussion section to emphasize the applicability of our approach to future research on a broader range of vision-language models.

Reviewer comment (Evaluation):
Is there any reason to authors not using bias metrics like max-skewness or NDKL?

Author response:
**Q2**: Besides WEAT, we considered max-skewness and NDKL as alternative bias metrics but eventually omitted these results. Responding to the reviewer's interest, we are now reintegrating these findings into the paper and will present them before the rebuttal period concludes.  

**Q2**: We welcomed the question proposed by reviewer SWyU on including additional bias metrics. The bias metrics proposed by [Geyik et al. (2019)](https://arxiv.org/abs/1905.01989), namely Skew@k, MaxSkew@k, and NDKL, provide conclusions similar to the initially proposed ones. We have moved the explanation and evaluation of these metrics to Appendix B1. 
We believe that including additional bias metrics significantly enhances the robustness of our paper, and we are hopeful that the reviewer will share this perspective.