Paper: Probabilistic Self-supervised Representation Learning via Scoring Rules Minimization (ICLR-2024)

Reviewer comment (Presentation):
There are some typos in the article, affecting the overall readability of the paper. In Table 3, should "PN" be replaced with "PL"? The experimental setups in the first and last rows of Table 3 are exactly the same. Is this an error?

Author response:
>* We thank the reviewer for pointing out the typos and we have updated our manuscript and incorporated these suggestions.

Reviewer comment (Writing):
In the last sentence of the third paragraph in Chapter 4, should "$\mu $" and "$\sigma $" have the superscript "i"? As per my understanding, "$\mu $" and "$\sigma $" vary for each data point.

Author response:
>* We thank the reviewer for pointing out the typos and we have updated our manuscript and incorporated these suggestions.

Reviewer comment (Theory):
The association between the probabilistic method and the prevention of dimensional collapse still requires further elaboration and demonstration. In the second paragraph of Chapter 1, the authors give two possible causes for the dimensional collapse. Is the author's solution inspired by either of the two causes or did it address either of the two factors? I think this needs to be further explained.

Author response:
>* Thank you for asking this question, we address collapsing representation without overparametrization and heavy data augmentation instead we push the boundaries of self-supervised learning by embracing the richness of probabilistic models. By adopting a probabilistic framework, our proposed method encourages the neural network to assimilate a comprehensive distribution of representations, rather than being constrained to a singular, fixed representation.

Reviewer comment (Experiments):
I suggest a performance comparison with various baseline methods for the same feature dimensions.

Author response:
>* We have the experiments for different embedding sizes in Figure 3-d for 100 epochs, and we also add a new result with the same embedding size as DINO.  We summarize the knn results in the following Table:
| Methods | embedding in the loss |knn|
|----------:|------:| -----:|
| iBOT | 8192 | 71.4|
| Our method  | 8000 |71.2|
| DINO | 65536 | 69.7|
| Our method | 65536 |69.9|

Reviewer comment (Novelty):
The network structure of the proposed method is identical to BYOL. What do you think is the most important difference between these two approaches?

Author response:
>* We employ a similar architecture to DINO (ViT-S) for both the encoder and the projection head, incorporating three linear layers with Gelu activation. Notably, our projection head does not include batch normalization. We then add a prediction layer similar to BYOL. Even without a batch normalization layer, we use a higher representation of 16k compared to 4096 for BYOL. However, we have shown that the performance does not suffer drastically at lower dimensions. Then, we have two layers for mean value and standard deviation. Our loss function is the proper scoring rule loss based on energy score L1 loss, and BYOL uses MSE loss. So we use a prediction layer like BYOL in terms of architecture, but we don't have batch normalization and use a different encoder and projection head, and we use another loss function based on a probabilistic approach. So, our method is different from BYOL, and we also showed that batch normalization and the prediction layer are not necessary components to prevent the collapse scenario, unlike BYOL.

Reviewer comment (Theory):
How do you ensure $q_\theta$ is not an identity network w.r.t $\mu$ with $\sigma = 0$ since the dimensionality of $z_\theta$ and $t_\theta$

Author response:
*Thanks for pointing out this issue. To avoid this case, the $q_{\theta}$ contains two linear layers with GELU as nonlinearity and uses ReLU nonlinearity between $q_{\theta}$ and $\mu$, so $q_{\theta}$ is a nonlinear function between $t_{\theta}$ and $\mu$ in case $\sigma$ is zero.

Reviewer comment (Experiments):
Since your scoring function is a variation of $L_2$ loss between $z_i$ and $z_{\xi}$, can you show that the variational encoding actually makes a difference by perturbing $t_\theta$ by a Gaussian noise of $\sigma = \epsilon$

Author response:
*The hyperparameter ${\lambda}$ is the determining factor for the effect of the perturbation. Thus, a lower value of ${\lambda}$ shows a more significant effect on the second-term portion of the loss, which is dominated by the injected Gaussian noise. Figure 3-a shows the effect of ${\lambda}$, lower ${\lambda}$ shows better performance compared to a higher value of ${\lambda}$. This means that injecting Gaussian noise improves the performance of the model. However, we adjust ${\lambda}$ to keep the loss function positive, which is essential for the proper scoring rule. Therefore, there is a limit to the lower value of ${\lambda}$, depending on the case.