Paper: DySTreSS: Dynamically Scaled Temperature in Self-Supervised Contrastive Learning (ICLR-2024)

Reviewer comment (Writing):
The paper lacks explanations of key concepts. For example, to understand this paper, it is important to understand what FN, PN, and TP mean, but the paper does not kindly define each of them.

Author response:
We thank the reviewer for this question. We would like to bring the attention of the reviewer to Sec. 3 (below Eqn. 1), where we have already defined True Positive (TP), False Negative (FN) and True Negative (TN) pairs. In the context of contrastive learning, every sample belongs to a class of its own, since we do not have any ground truth label information. In the pre-training stage, we can only say if a pair is positive or negative. However, when dealing with all these pairs, we can argue that there are samples which may belong to the same ground truth class but could be in a negative pair. These pairs are False Negative (FN) pairs. Similarly, there are pairs which contain samples obtained by different augmentations of the same sample, and we term these pairs as True Positive (TP) Pairs. Lastly, pairs containing samples which may belong to different ground truth classes are termed as True Negative (TN) pairs.

Reviewer comment (Writing):
There are some coarse parts of the notation. For example, the sample index is written as a superscript in equation 5, but not in other equations. In addition, p^(i↓j)+p^(ij↓) seems to be a typo of p^(i↓j)+p^(j↓i). In addition, ∂L/(∂s_ij )>0=δ in equation 7 should be changed to ∂L/(∂s_ij )=δ>0.

Author response:
Thanks for the suggestion. We have made the necessary changes in the manuscript.

Reviewer comment (Theory):
The theoretical analysis seems a bit strange. First, the proposition contains information about the slope of the temperature function, but this is not a mathematically proven fact, but just about the idea of the proposed algorithm. In addition, the author's claim that ∂L/(∂s_ij ) in equation 8 is always negative is suspicious.

Author response:
We prove the proposition in Appendix C, where we derive the nature of the slope of the temperature function in detail. On plotting the derived temperature function in Equation 14, we plot the curve in Fig. 8, and we can observe the nature of the temperature function from that figure itself. 

Furthermore, we would like to draw the attention of the reviewer to the fact that for the negative pairs $\partial L/(\partial s_{ij} )$ is a “non-negative” quantity.  We had also mentioned it in the manuscript. In fact, the reviewer has also kindly suggested a correction to the order of ‘>0’ and ‘=$\delta$’ in the previous comment for the same.

Reviewer comment (Experiments):
Originally, only one temperature hyperparameter needs to be tuned for existing contrastive learning, but the proposed DySTress has more hyperparameters (τ_min,τ_max,Δ_s,k) as a result, and it can be seen from Table 6 that the performance of DySTress is sensitive to the values of hyperparameters.

Author response:
Thanks to the reviewer for pointing this out. From Table 6, it can be seen how the performance varies with the parameters $\tau_{min}$ and $\tau_{max}$. However, we have already provided an intuitive explanation why this happened in Sec. 4.2, that is, the effect of temperature on the local and global structures in the feature space. Table 6 only proves the statements made in Sec. 4.2. The general approach we took when applying DySTreSS on top of SimCLR, is that we set $\tau_{max}$ to the optimal temperature for SimCLR and tuned only $\tau_{min}$ to understand how far we can decrease $\tau_{min}$ before performance starts to degrade. 

The hyper-parameter $\Delta s$ is to study and get an idea about the optimal temperature function, as we have previously stated in Sec. 4.3, that the boundary between false negative pairs and true negative pairs is not hard. However, we see in Table 7, that the performance is not sensitive to the value of $\Delta s$, and may be ignored or not tuned. The hyper-parameter k is to maintain the minimum and maximum temperature value $\tau_{min}$ and $\tau_{max}$, respectively, hence, can be ignored as a hyper-parameter, as it does not require tuning and is a byproduct of using $\Delta s$.

Reviewer comment (Experiments):
Why is the temperature scaling function used in the main text different from the temperature scaling function obtained from the ODE solution in the appendix? And why was the function in Algorithm 1 chosen as the temperature scaling function among several candidates? Are there any comparison experiments with other temperature scaling functions?

Author response:
Due to the presence of logarithm in the denominator, logarithm can assume value tending to zero. This will lead to the value of $\tau_{ij}$ tending to infinity.Due to the possibility of numerical instability, we opted for different forms of temperature function but with the properties that we have mentioned in the paper, that is, negative and positive slope in the negative and positive half of the cosine similarity vs temperature plane. We experimented with different types of functions, like linear and exponential. However we observed that we obtained similar performance with the cosine function. The accuracies are given in the table below, and will be added in the appendix of the manuscript. The corresponding codes will also be added in the github repository.

Table 1.
| Function Type | CIFAR10 | CIFAR100 |
|---|---|---|
|Cosine|85.85|56.57|
|Linear|85.74|56.78|
|Exponential|85.81|56.47|

However, using the exponential function involves setting up two parameters to maintain the maximum and minimum temperature at $\tau_{max}$ and $\tau_{min}$, respectively, which needs to be manually tuned and verified. Whereas, it is easier for the cosine function to maintain the same. Although the linear function has no such extra parameters, both the linear and exponential function will be non-differentiable at $s_{ij} = 0.0$ (Equation 11). Hence, we argue that use of the cosine function is a better choice.

Reviewer comment (Experiments):
In the Experimental Results section, the proposed method seems to have been carefully tuned for optimal performance with respect to the hyperparameters (Δ_s,k, etc.). Were the hyperparameters for the other baseline methods also tuned in a fair manner?

Author response:
Yes, the results for the other baseline methods used in the manuscript were the best that can be obtained using the lightly-ai library that we used for all our experiments. The same values are reported in the official website of lightly-ai. For the ImageNet experiments, we used the original hyper-parameters from the paper and implemented then using the lightly-ai library.

Reviewer comment (Experiments):
Cosine similarity is high for hard negative samples, so it makes sense to give them a high temperature to reduce their weight. However, why do we need to increase the temperature for easy negative samples with low cosine similarity? I think a monotonically increasing temperature function would make more sense. Are there any experiments with this?

Author response:
Primarily, our motivation of using the cosine function as the temperature function comes from the derived form in Appendix C. However, we respect the suggestion from the reviewer and ran our proposed framework with a monotonically increasing temperature function. 

Like easy negative samples, we also consider the possibility of samples that form hard false negative pairs (samples in negative pairs belonging to the same ground truth classes). In the pre-training stage, we can’t have access to the ground truth labels, hence cannot say for certain the range of cosine similarities for false negative pairs in the pre-training stage where the weights are initialized randomly. Also, we wouldn't want the samples to drift farther away from each other and hence, we keep a higher temperature towards low cosine similarity. 

We have included the results in Table 2 below, for monotonically increasing functions in the table below (as well as in the Appendix of the manuscript). We see that the results are almost similar to the reported results, which does in fact give a crude idea about the range of cosine similarity of the hard false negative pairs resulting from augmentations applied to positive pairs. It also proves that the easy true negative pairs do not contribute or have little effect on representation learning. This conclusion agrees with the hypothesis stated in [1].

We have included the above discussion in Appendix H in the revised manuscript.

Table 2. Comparison of performance on CIFAR datasets when using proposed cosine function and monotonic function as the temperature function in the DySTreSS framework.

| Type of Cosine | CIFAR10 | CIFAR100 |
|---|---|---|
| Proposed Cosine | 85.85 | 56.57 |
| Monotonic Cosine | 84.57 | 55.61 |

---

References:

[1] Feng Wang and Huaping Liu. Understanding the behavior of contrastive loss, CVPR 2021

Reviewer comment (Novelty):
As stated in the summary, due to the limitation of negative-required methods (batch size sensitive), and the mainstream of CL is the negative-free framework (DINO, iBOT), the significance of exploring NCE loss is somewhat limited.

Author response:
Thanks to the reviewer for this comment. The frameworks like DINO [2] and iBOT [7] fall under the canopy of instance-based non-contrastive self-supervised learning frameworks, and utilizes self-distillation with no labels. Although the mainstream focus of CL has shifted to non-contrastive methods, however, the role of temperature in contrastive learning is an intriguing phenomenon and still not explored fully. We can see works like iSogCLR [5], Kukleva et al. [6], DCL [8], DirectCLR [9], etc. where the authors have explored the self-supervised contrastive learning setting in recent years (2022-23). In this work, we also intend to study the effect of temperature in contrastive learning (InfoNCE loss based frameworks) and improve the same. We agree that the contrastive learning frameworks come with its limitations. However, we believe that there is more to the function of temperature hyper-parameter in the InfoNCE loss, than what meets the eye. Carefully tuning the temperature hyper-parameter can greatly change the representation learning capability of the network as evident from the results.

Reviewer comment (Experiments):
The improvements are limited, which can be obtained by switching hyperparameters (e.g., lr, wd). Could the author provide the mean and variance over 5 times running?

Author response:
For the CIFAR10 dataset, we ran the proposed method with 5 different random seeds. We obtained a mean accuracy of 85.854 with a standard deviation of 0.0621. For the CIFAR100 dataset, we obtained a mean accuracy of 56.568 with a standard deviation of 0.215.

Reviewer comment (Experiments):
The accuracies reported in Table 2 are too low. For example, Barlow Twins with 100 epoch pretraining can actually achieve 67+ top-1 accuracy. However, only 62.9 is reported in Table 2. Perhaps the author should tune the hyperparameters to make the proposed method comparable.

Author response:
We used the lightly-ai library for all our experiments. We ran all our experiments on 24GB NVIDIA A5000 GPU for ImageNet1K, with batch size 256 and FP16 precision. The hyper-parameters for Barlow Twins used were the same as used in the paper, consequently resulting in the reported result. The result can also be seen on the lightly-ai benchmark website. We in fact tried to investigate the performance of Barlow Twins, but obtained the same result as reported there. On the contrary, the reported accuracy for Barlow Twins in the manuscript of Zero-CL is 67.7 for a batch size of 1024.

Reviewer comment (Experiments):
Missing references. The authors miss some negative-free methods (e.g., Zero-CL, W-MSE, DINO, ARB) and should compare with them in the main results (as they also provide results on ResNets). I recommend the author discuss the advantages and disadvantages of the proposed methods and these negative-free methods. 

References:  
[1] Zhang S, Zhu F, Yan J, et al. Zero-cl: Instance and feature decorrelation for negative-free symmetric contrastive learning[C]//International Conference on Learning Representations. 2021.  
[2] Caron M, Touvron H, Misra I, et al. Emerging properties in self-supervised vision transformers[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 9650-9660.  
[3] Ermolov A, Siarohin A, Sangineto E, et al. Whitening for self-supervised representation learning[C]//International Conference on Machine Learning. PMLR, 2021: 3015-3024.  
[4] Zhang S, Qiu L, Zhu F, et al. Align representations with base: A new approach to self-supervised learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 16600-16609.

Author response:
Thanks for pointing this out and we have considered these now. We have now compared and added a discussion about the advantages and disadvantages of the above mentioned methods in Sec. 2 (Related Works). 

In the table below, we have added the performance of DINO and WMSE on CIFAR datasets only with a batch size of 128 and pre-training of 200 epochs.

Table 1. Comparison of the the proposed method with DINO and WMSE on CIFAR datasets
| Methods | CIFAR10 | CIFAR100 |
|---|---|---|
| DINO | 84.02 | 46.79 |
| WMSE | 85.52 | 52.72 |
| DySTreSS | 85.68 | 56.57 |

We also ran an experiment on the ImageNet100 dataset for 400 epochs with a batch size of 128, as done in [4]. We observed that it outperformed DINO [2], WMSE [3], ZeroCL [1], and ARB [4] by a satisfactory margin. The evidence is presented in Table 2 below.

Table 2. Comparison of the proposed method with DINO, WMSE, Zero-CL, and ARB on the ImageNet100 dataset (Accuracy for WMSE is taken from [1] and [4])
| Methods | Projector Hidden Layer Dimension | Top-1 Linear Eval. Accuracy (%) |
|---|---|---|
| DINO [2] | 2048 |74.84 |
| WMSE [3] | 256 | 69.06 |
| ZeroCL [1] | 2048 | 79.26 |
| ZeroFCL [1] | 2048 | 79.32 |
| ARB [4] | 2048 | 79.48 |
| DySTreSS| 2048 | 81.24 |

The proposed work attempts to study the effect of temperature hyper-parameter in contrastive learning methods and improves the same. The advantage of our proposed method is that it improves the performance of contrastive learning frameworks by tuning the temperature hyper-parameter as a function of cosine similarity. It also outperforms several contrastive learning methods, with the same compute resource and time requirement. Tuning the temperature hyper-parameter for each pair allows the framework to give more or less importance based on the cosine similarity of the samples in the pair. While the infoNCE loss itself takes care of the global semantics, tuning the temperature as a function of cosine similarity, allows the framework to take the local semantics into consideration as well.

However, methods like Zero-CL[1], W-MSE[3] are said to be negative-free contrastive learning frameworks. Although ZeroCL is a negative free method, it does use information from non-positive pairs to compute batch and dimension-wise statistics. W-MSE uses a whitening strategy. W-MSE also uses information from samples in non-positive pairs to compute batch statistics and computing the Whitening matrix with a complexity same as computing a cosine similarity matrix. Furthermore, Cholesky decomposition is done on the Whitening matrix with a complexity of $O(k^3 + Mk^2)$. However, our proposed method does not incur any such extra complexity. 

Methods like DINO [2], iBOT [5] use knowledge distillation and require two separate networks and an extra parameter update step for the teacher network. However, our method does not use any separate network and also doesn't use an extra parameter update step. However, knowledge distillation based methods, like DINO and iBOT have shown to outperform contemporary frameworks on benchmark datasets.

ARB [4] proposes a Nearest Orthonormal Basis based optimization objective. The authors claim that the objective complexity is O(d), but to take care of the Non-full rank matrix case, the authors use a pseudo-orthonormal base, which involves computing a spectral decomposition of the correlation matrix. This particular computational complexity is absent in our framework.

Reviewer comment (Evaluation):
Although terms TP, TN, FN might be well-known, I think it needs to be carefully explained and presented when you are using it under your circumstance. For example, it is unclear to me what is the exact definition of FN pairs in images. How do we measure it? When the representation are not formed, how do we effectively decide what is TP, TN, FN pairs without making mistakes?

Author response:
We thank the reviewer for this question. We would like to bring the attention of the reviewer to Sec. 3 (below Eqn. 1), where we have already defined True Positive (TP), False Negative (FN) and True Negative (TN) pairs. In the context of contrastive learning, every sample belongs to a class of its own, since we do not have any ground truth label information. In the pre-training stage, we can only say if a pair is positive or negative [2, 3, 4]. However, when dealing with all these pairs, we can argue that there are samples which may belong to the same ground truth class but could be in a negative pair. These pairs are False Negative (FN) pairs. Similarly, there are pairs which contain samples obtained by different augmentations of the same sample, and we term these pairs as True Positive (TP) Pairs. Lastly, pairs containing samples which may belong to different ground truth classes are termed as True Negative (TN) pairs. Only in the evaluation stage, where we can have access to the ground truth label, can we say for sure, which pairs belong to any of these three categories.

Reviewer comment (Theory):
For Figure 1(b), why would the ideal convergence only have few points? To me, Figure 1(c) also looks ideal? The paper explains as "the ideal global structure should consist of N closed subser with Minimum intraclass scattering and maximized interclas distance". However, class itself, it a very vague and "human-biased" term. For example, in imagenet, there are many classes that look very like each other, in this case, would it be ideal to have minimum intraclass scattering? In that case, would it be what's known as the neural collapse in supervised training? So, would that be an "ideal" representation we truly need? I think the motivation here needs to be very carefully explained and justified.

Author response:
We do agree that the definition of class is subjective. However, many classical algorithms consider that different classes should have higher interclass scattering, whereas samples within a defined class should have minimum intraclass scattering. The term “intraclass scattering” refers to scattering between samples belonging to the same class. On the other hand, the term “Inter-class distance” refers to the distance between the clusters of different classes. Minimum intraclass scattering is possible only if a trained model is able to correctly classify every sample in a class and map every sample of a particular class very close to each other, which is crucial to achieve human-like performance. Also, maximum interclass distance is possible to achieve if the model is able to map samples from different class farther away from each other.

It is obvious that in datasets like ImageNet, and even in small-scale datasets like CIFAR10 or CIFAR100, there are samples from classes which are difficult to differentiate. However, if a model is unable to differentiate between different classes which are very similar to each other, then it can result in a sort of collapse, which is the opposite of the scenario we have mentioned in our motivation. However, the purpose of the illustration was to depict an ideal scenario any learning model aims to achieve after pre-training. Hence, the statement “Minimum intraclass scattering and maximized interclass distance” actually depicts an ideal end state scenario, where the classes are linearly separable and the samples with the same class are mapped close to each other.

However, we want to emphasize that our proposed method is independent of such assumptions and the figures are just given to discuss about feature representation.

Reviewer comment (Reproducibility):
In Algo1, s_ij is defined as the cosine sim of the pair (x_i, x_j). If x_i and x_j are images, do we flatten them and compute the cosine sim? Or do we compute the cosine similarity differently? in the latent space?

Author response:
After passing the images $x_i$ and $x_j$ through the ResNet encoder and MLP projector, we obtain the feature vectors $z_i$ and $z_j$, respectively. We flatten the vectors to have dimensions $1 \times D$ and L2-normalize the vectors $z_i$ and $z_j$. To calculate the cosine similarity, we simply take the dot product of $z_i$ and $z_j$. So, as the reviewer has correctly pointed out, the cosine similarity is calculated in the latent space.

Reviewer comment (Evaluation):
If we go back to the motivation, would the proposing method giving us better representation? Visually? Is the method learning representation with more intra distance and less inter distance?

Author response:
From the tables presented below, we can take a look at the overall uniformity and inter-class uniformity values for SimCLR, MoCov2, DCLW, DCL, and DySTreSS. Overall uniformity takes all the samples in the feature space and is calculated as defined in [1]. We define another term, Inter-class uniformity which is defined by Eqn. 19 in Appendix E. 

Table 1. Overall Uniformity and Inter-Class Uniformity values for SimCLR, MoCov2, DCL, DCLW, DySTreSS for CIFAR10 dataset.
| CIFAR10 | | | |
|----------|-----------|-----------------|----------|
| Method | Unifomity | InterClass Unif | Accuracy |
| SimCLR [3] | -2.2135 | -0.5194 | 83.65 |
| MoCov2 [5]| -2.1929 | -0.5094 | 83.67 |
| DCLW [6]| -2.5482 | -0.5349 | 84.02 |
| DCL [6]| -2.7717 | -0.5524 | 84.47 |
| DySTreSS | -2.8430 | -0.5797 | 85.83 |

Table 2. Overall Uniformity and Inter-Class Uniformity values for SimCLR, MoCov2, DCL, DCLW, DySTreSS for CIFAR100 dataset.
| CIFAR100 | | | |
|----------|-----------|-----------------|----------|
| Method | Unifomity | InterClass Unif | Accuracy |
| SimCLR [3]| -2.2556 | -0.5246 | 52.32 |
| MoCov2 [5]| -2.2202 | -0.5364 | 54.01 |
| DCLW [6]| -2.6865 | -0.6405 | 55.87 |
| DCL [6]| -2.8314 | -0.6792 | 56.02 |
| DySTreSS | -2.8169 | -0.7657 | 56.72 |

Table 3. Overall Uniformity and Inter-Class Uniformity values for SimCLR, MoCov2, DCL, DCLW, DySTreSS for ImageNet1K dataset.
| ImageNet1K | | | |
|----------|-----------|-----------------|----------|
| Method | Unifomity | InterClass Unif | Accuracy |
| SimCLR [3]| -2.9084 | -0.8739 | 63.2 |
| DCLW [6]| -3.0137 | -0.9544 | 64.2 |
| DCL [6]| -3.0786 | -1.0053 | 65.1 |
| DySTreSS | -3.0597 | -1.0015 | 65.2 |

We can see that the uniformity and the inter-class uniformity values decreases, which indicates that the inter-cluster distance increases from SimCLR to DySTreSS, evident from the increasing accuracy values, as well. Lower inter-cluster uniformity means better separability between clusters. Hence, we can argue that the proposed model learns more separable classes, that is, more inter-class distance.

Reviewer comment (Evaluation):
Does the method incur extra training cost? If so, how does it compare with the gain in performance? Because computing cosine similarity constantly could be a non-trivial increase in terms of computation (resource and time).

Author response:
No, the proposed method does not incur extra training costs for computing cosine similarity constantly, when compared to existing methods like SimCLR. It is computing the same cosine similarity matrix as done in many other contrastive SSL methods, e.g. SimCLR, MoCo, DCL.

Reviewer comment (Theory):
The design of Algorithm 1 in the paper is merely based on certain rules and lacks theoretical underpinnings.

Author response:
Thanks for your comment. Please note that our paper is not merely based on certain rules. The criteria (1)-(4) as mentioned in Sec. 4.3 arises from the intuitive explanations of the effect of temperature on the local and global structures as mentioned in Sec. 4.2. However, the temperature function in Algorithm 1 is not based on those criteria, but it is a result of the derived temperature function that comes from Equation 14 in Appendix C. As can be seen in Sec. 4.3, the equations 6 and 7 arise as a natural consequence of differentiating the InfoNCE loss with respect to the cosine similarity of the positive pair $s_{ii+}$ and negative pairs $s_{ij}$. Considering that the temperature hyper-parameter $\tau$ is a function of cosine similarity. We take no other assumption in our work. In Proposition 1, we prove that the nature of the slope arises as a consequence of solving the ODE resulting from Equations 6 and 7. Only after that, we propose a temperature function in Algorithm 1. In Appendix C, we have derived the mathematical formulation of the temperature function, and provided necessary insights from the equations as mentioned in Equation 14 and 15 of Appendix C.  This  shows that our proposition is not based on certain rules and has a theoretical background.

Reviewer comment (Experiments):
While the paper mentions that adjusting the temperature coefficient can improve feature distribution, corresponding results are not presented in the experimental section.

Author response:
Thanks to the reviewer for pointing this out. As a metric of the feature distribution, we present the overall uniformity and inter-class uniformity values for SimCLR, DCL, DCLW, MoCov2 and DySTreSS (proposed) on CIFAR10, CIFAR100 and ImageNet1K datasets. Uniformity values denote how well the samples are distributed over the feature space [3]. We have included the tables for the same in the reply to your Q2, and in Table 11 of Appendix E of the manuscript as well. In all the datasets, our proposed method shows lowest overall uniformity and inter-class uniformity, which indicate that the proposed method learns more separable clusters than the SOTA methods.

Reviewer comment (Theory):
On what basis is the cosine function used in Algorithm 1? Are there any theoretical results to support this choice?

Author response:
We thank the reviewer for asking this insightful question. Please note that the cosine function used in Algorithm 1 has theoretical support and is discussed as follows. We would like to draw the attention of the reviewer to the fact that we have shown a derivation of the nature of the slope of the temperature function in the negative and positive half of the cosine-similarity vs. temperature plane as shown in Fig. 8 of Appendix C. The derivation starts with the basic postulate that the term $\frac{\partial \mathcal{L}}{\partial s_{ij}}$ should be positive when $s_{ij}$ is the cosine similarity of the samples in a negative pair. Whereas, for the positive pairs, the term $\frac{\partial \mathcal{L}}{\partial s_{ij}}$ should be negative as with the decrease in the loss, the cosine similarity between the samples in a positive pair increases. By solving the resulting differential equation, we get an expression for the temperature function given by Eqn. 14. Plotting the function in Fig. 8, we get an idea about the slopes of the temperature function. Any function following these criteria can be used as a temperature function.

We have also added now results by using different temperature functions (cosine, linear and exponential), in the Appendix G of the revised manuscript which show that using any function which satisfy the conditions in Proposition 1 (and subsequently proved in Appendix C), that is, positive and negative slope in the positive and negative half of the temperature vs. cosine-similarity plane, yields satisfactory performance. As cosine function follows this proposition and is differentiable everywhere, we have selected it for all our experiments.

Reviewer comment (Experiments):
Is there experimental evidence to support the claim that adjusting the temperature parameter can improve feature representation?

Author response:
Thanks for the comment. We would like to mention that there is experimental evidence and it is discussed as follows. In the tables given below, we have provided overall uniformity and inter-class uniformity values for 5 contrastive learning algorithms on CIFAR10, CIFAR100 and ImageNet1K datasets to support the claim.

Table 1. Overall Uniformity and Inter-Class Uniformity values for SimCLR, MoCov2, DCL, DCLW, DySTreSS for CIFAR10 dataset.
| CIFAR10 | | | |
|----------|-----------|-----------------|----------|
| Method | Unifomity | InterClass Unif | Accuracy |
| SimCLR | -2.2135 | -0.5194 | 83.65 |
| MoCov2 | -2.1929 | -0.5094 | 83.67 |
| DCLW | -2.5482 | -0.5349 | 84.02 |
| DCL | -2.7717 | -0.5524 | 84.47 |
| DySTreSS | -2.8430 | -0.5797 | 85.83 |

Table 2. Overall Uniformity and Inter-Class Uniformity values for SimCLR, MoCov2, DCL, DCLW, DySTreSS for CIFAR100 dataset.
| CIFAR100 | | | |
|----------|-----------|-----------------|----------|
| Method | Unifomity | InterClass Unif | Accuracy |
| SimCLR | -2.2556 | -0.5246 | 52.32 |
| MoCov2 | -2.2202 | -0.5364 | 54.01 |
| DCLW | -2.6865 | -0.6405 | 55.87 |
| DCL | -2.8314 | -0.6792 | 56.02 |
| DySTreSS | -2.8169 | -0.7657 | 56.72 |

Table 3. Overall Uniformity and Inter-Class Uniformity values for SimCLR, MoCov2, DCL, DCLW, DySTreSS for ImageNet1K dataset.
| ImageNet1K | | | |
|----------|-----------|-----------------|----------|
| Method | Unifomity | InterClass Unif | Accuracy |
| SimCLR | -2.9084 | -0.8739 | 63.2 |
| DCLW | -3.0137 | -0.9544 | 64.2 |
| DCL | -3.0786 | -1.0053 | 65.1 |
| DySTreSS | -3.0597 | -1.0015 | 65.2 |

From the above tables, we can see 2 quantities, overall uniformity and inter-class uniformity. Overall uniformity takes all the samples in the feature space and is calculated as defined in [1]. We define another term, Inter-class uniformity which is defined by Eqn. 19 in Appendix E.  

A lower uniformity value indicates that the samples are more spread out than a model with a higher uniformity value. The inter-class uniformity metric value indicates how farther apart the class centroids are from one another. Hence, a lower inter-class uniformity value indicates that the clusters are more separated than a mode with a higher value. We see that the overall uniformity values are the lowest for our proposed method. In fact, we can see a trend in the uniformity values with the accuracy for different models as well. In our case, the inter-class uniformity metric is the lowest, which means that the class centroids are farther away than the other models, and is an indicator of how easily separable the clusters are for our proposed method. Thus, we can say that the proposed framework improves feature representations, and hence, learns more separable clusters, resulting in higher 200-NN accuracy.

Reviewer comment (Experiments):
Why wasn't a comparison made with the method proposed by Kukleva et al. [1]?

Author response:
Thanks for your suggestion and based on your suggestion we have made this comparison now and it is discussed as follows. To check the performance of our proposed framework in comparison to Kukleva et al. [1], where the authors trained their models for 2000 epochs on Long-Tailed datasets. For fair comparison we also trained our proposed framework for the same number of epochs  (2000 epochs) on CIFAR10-LT, CIFAR100-LT. In the table below we report the 10-NN accuracy (as per Kukleva et al.) of both the datasets. For the experiments on ImageNet100-Lt, we used a batch size of 256 on a 24GB NVIDIA A5000 GPU. From the results it can be seen that our proposed framework outperforms Kukleva et al. on long tailed datasets.

Table 4. Comaprison of performance on benchmark long-tailed datasets with Kukleva et al. [1]
|  | CIFAR10-LT | CIFAR100-LT | ImageNet100-LT |
|---|---|---|---|
| Kukleva et al. [1] | 62.91 | 30.20 | 45.3 (repro.) |
| DySTreSS | 64.98 | 31.71 | 46.1 |

We have added the above comparison in our revised manuscript.

Reviewer comment (Novelty):
There was a recent paper [2] in ICML that utilized individualized temperature parameters to optimize the contrastive learning loss. How do these two papers differ?

Author response:
In this ICML paper [2], the authors attempt to implement distributionally robust optimization (DRO) on an existing algorithm SogCLR, which leads to temperature individualization. However, it requires the use of a temperature hyper-parameter $\tau_i$ corresponding to each anchor sample, and it is updated at each iteration. In our opinion, this no longer keeps the temperature as a hyper-parameter and converts it into another optimizable parameter. The authors also claim that by doing so, the global semantics is taken into account during pre-training. 

However, in our case, the temperature hyper-parameter is not dependent on each anchor sample. Rather, we devise a temperature function which gives a suitable non-learnable temperature for each pair of samples in contrastive learning. 


Thus, the two papers differ significantly.

We have included the above discussion in the “Related Works” section of the revised manuscript.

To compare our method with this ICML paper [2], we ran our proposed method for 400 epochs, following the configuration in [2], and the results are reported below.

Table 5. Comparion with iSogCLR [2] on CIFAR datasets
|| CIFAR10 | CIFAR100 |
|---|---|---|
|iSogCLR [2] | 89.24 | 63.82 |
|DySTreSS | 90.34 | 63.94 |