Paper: An Investigation of Representation and Allocation Harms in Contrastive Learning (ICLR-2024)

Reviewer comment (Experiments):
The main weakness I can find from this paper is the lack of a large-scale study. SSL, and CL in particular, are most effective when used on large amounts of data. The controlled studies in this paper allow us to analyze the existing representation harms. However, it can be seen that these values are generally much lower for BiasBios than for CIFAR10.

Author response:
**However, it can be seen that these values are generally much lower for BiasBios than for CIFAR10.** We would like to emphasize that the results from the CIFAR10 and BiasBios datasets are not directly comparable, as they are based on slightly different metrics. The metric for the CIFAR10 dataset is specific to our *controlled study* design, where we measure the degree to which the representations of two classes have collapsed with each other due to under-representation by comparing representations learned using balanced data to those learned using imbalanced data. In our experiments with the BiasBios dataset, we wish to demonstrate the potential harmfulness of collapse in a real data scenario where *underrepresentation occurs naturally*, which requires a slightly different measure of collapse. Specifically, we measure the ratio of the similarity of samples with the same gender and different occupations to the similarity of samples with different gender and the same occupation. This measure allows us to notice harmful cases of collapse, such as female surgeons being more similar to female dentists than to male surgeons.

Reviewer comment (Experiments):
The difference in results between BiasBios and CIFAR10 could be due to the different domain (text). A study on ImageNet (larger number of both samples and classes) could potentially disentangle this confound.

Author response:
**Regarding ImageNet:** Thank you for the suggestion. We started looking into such an experiment, but realized that the `imagenet-1k` dataset used for training publicly available CL models is indeed pretty balanced. Most classes have 1300 samples, with the least frequent class being `black-and-tan_coonhound` with 732 samples. Therefore, it is not suitable for studying the effect of underrepresentation. Artificially subsampling the data and training our own CL model as we did on CIFAR10 is unfortunately computationally too challenging for our computational resources.  
**On ImageNet:** As an alternative to ImageNet, we are considering an experiment with the CIFAR100 dataset, which has 100 classes, while being easier to experiment with in terms of data size given our computational resources. This dataset has reasonably large classes, allowing more diversity in the dataset than CIFAR10, similarly to what you suggested in the review. We intend to subsample one class from each of the superclasses and investigate whether the effects representation and allocation harms are as severe as in CIFAR10.

Reviewer comment (Writing):
There needs to be an intuitive definition of allocative harms and representation harms when they are first mentioned in the intro, which matches the precise definition in section 2.

Author response:
**Intuitive definition of allocative harms and representation harms:** 
We added an intuitive definition and an example of allocative harms in the first paragraph of the Introduction. *Allocation harm* refers to the demographic disparities in resource allocations, and this harm is easily traced back to the rate of misclassification of the underlying resource allocation mechanism. In our eq. (4.2) we use this difference miclassification rate to quantify the allocation harm caused by under-representation. 


We also extended the second paragraph of the Introduction with an example of *representation harm* to facilitate the intuition behind it. In the third paragraph and in Figure 1 we also present an example using the CIFAR10 dataset, where an undersampling of the `automobile` images causes their representations to cluster with those of `trucks`. In Section 2.1.2, we measure this clustering by calculating the ratio of the average cosine distances between the representations of two groups when one of them is under-represented to the average distance between the same two groups when they are class-balanced.

Reviewer comment (Presentation):
Figure text is generally too small to see clearly.

Author response:
**Enlarge texts in figures:** Thank you for the suggestion. We are updating the figures with larger text and shall update our draft with modified figures shortly.

Reviewer comment (Theory):
Why does this metric for representation harm make sense? It seems like an important decision critical for the rest of the paper, so needs better justification.

Author response:
**Justification of the representation harm metric:** We reiterate that the key idea behind measuring representation harm is to measure *collapse* between groups in the data, while relevant measures of collapse, the definitions of groups, and the kinds of groups between which collapse is most problematic may vary across domains and applications.

Reviewer comment (Presentation):
The examples used in section 2 are not very useful - sure automobiles and trucks could collapse but is it the worst thing - are there more compelling real-world examples to illustrate these problems?

Author response:
**Examples used in Section 2 are not very useful:** The collapse between `automobile` and `truck` representations (in the third paragraph of Section 1 and Section 2.1) is an example to illustrate the idea of representation harm. For a more realistic example, we refer to the Wall Street Journal article titled "Google Mistakenly Tags Black People as 'Gorillas,' Showing Limits of Algorithms" [3]. We have included this example in the second paragraph in the Introduction and in the first paragraph of Section 2.1.2. Additionally, we provide realistic examples in our experiments with the BiasBios dataset in Section 2.2.2, where the representations of under-represented female surgeons are closer to the representation of female dentists than they are to male surgeons. Similar harm is also observed between the `attorney` and `paralegal` professions. For a more detailed discussion, see our last paragraph in Section 2.2.2 named **Results**.

Reviewer comment (Evaluation):
Why are the metrics for representation harm in 2.1.2 and 2.2.2 different? This seems weird. Even if the exact distance (e.g., cosine vs l2 for image or word embeddings) are different - why are the metrics different?

Author response:
**Different representation harm metrics in Sections 2.1.2 and 2.2.2:** Although the exact quantification of representation harm metric may depend on specific instances of under-representation, the main idea behind the metrics are similar: they measure the collapse of the representations of an under-represented group with semantically similar groups. In our experiments with the CIFAR10 dataset in Section 2.1 we *control the undersampling* for each class and wish to demonstrate that the collapse grows/emerges when the underrepresentation occurs. In our experiments with the BiasBios dataset, we wish to demonstrate the potential harmfulness of collapse in a real data scenario where *underrepresentation occurs naturally*, which requires a slightly different measure of collapse. Specifically, we measure the ratio of the similarity of samples with the same gender and different occupations to the similarity of samples with different gender and the same occupation. This measure allows us to notice harmful cases of collapse, such as female surgeons being more similar to female dentists than to male surgeons.

Reviewer comment (Experiments):
The empirical analysis is conducted only on CIFAR10 and BIASBIOS datasets - I would have liked to see more to further strengthen the results in the paper, such as including celebA which is quite standard for studying biases in vision models, or perhaps even with results on a CLIP model for results on image-text models.

Author response:
**Empirical experiments beyond the CIFAR10 and BIASBIOS datasets:** We believe that the experiments conducted with the two datasets, the synthetic dataset, and the theoretical analysis were sufficient to demonstrate that under-representation can lead to representation and allocation harm in contrastive learning settings. In addition to various CL methods, such as SimCLR, SimSiam, and SimCSE, we have also included the *boosted contrastive learning* method [2] (in Section 5) to further strengthen our experimental results. Although we acknowledge that additional experiments would be beneficial, the current paper already consists of nine pages of main text and 11 pages of supplementary material, not including references.

Reviewer comment (Presentation):
The text in the figures is too small for easy readability and needs enlargement for better clarity.

Author response:
**Enlarge texts in figures:** Thank you for the suggestion. We have updated some of the figures in the main draft with a larger text and shall update our other figures (in the main and supplementary draft) shortly. If there are any figures remaining that you find difficult to read, let us know which ones and we will try to improve their readability.

Reviewer comment (Evaluation):
A more thorough justification is needed for the chosen metric of representation harm, given its critical importance to the paper's analysis.

Author response:
**Justification of the representation harm metric:** We reiterate that the key idea behind measuring representation harm is to measure *collapse* between groups in the data, while relevant measures of collapse, the definitions of groups, and the kinds of groups between which collapse is most problematic may vary across domains and applications.

Reviewer comment (Experiments):
The necessity for different data setups in Sections 5 and 6, particularly the choice of causal mediation analysis for Section 6, lacks proper motivation and explanation.

Author response:
**Different data setups in Sections 5 and 6:** We assume that this was a typo, and you meant Sections 3 and 4. If so, then we point out that the synthetic experiments in Section 3 are used to motivate and interpret our theoretical analysis (Section 3.2) and to draw its connections with our empirical studies in Section 2. On the other hand, the causal mediation analysis in Section 4 shows that in CIFAR10 the downstream allocation harm in the classification task is partly caused by the representation harm.  

**Motivation for causal framework:** As we explain in the second line of the first paragraph of Section 4 and in Section 4.1, causal mediation analysis [1] is the statistically principled way to dissect total allocation harm (**TE**) and tease out the part caused by representation harm through the natural indirect effect (**NIE**).

Reviewer comment (Experiments):
The paper's primary limitation is the absence of a comprehensive large-scale study, especially in the realm of self-supervised learning (SSL).

Author response:
**Absence of a comprehensive large-scale study:** We believe that the experiments conducted with the two datasets, the synthetic dataset, and the theoretical analysis were sufficient to demonstrate that under-representation can lead to representation and allocation harm in contrastive learning settings. In addition to various CL methods, such as SimCLR, SimSiam, and SimCSE, we have also included the *boosted contrastive learning* method [2] (in Section 5) to further strengthen our experimental results. Although we acknowledge that additional experiments would be beneficial, the current paper already consists of nine pages of main text and 11 pages of supplementary material, not including references.