Paper: Self-supervised Representation Learning from Random Data Projectors (ICLR-2024)

Reviewer comment (Evaluation):
Despite the good performance of the experiments, what has been learned from the latent space using the scheme introduced in this paper, even intuitively?

Author response:
While we do not have a complete theoretical explanation for the success of learning from randomness, we can offer you some indication of an interesting direction. Random orthogonal projections on high dimensional spaces have been studied in mathematics and found to have interesting properties. One result, known as the Johnson-Lindenstrauss lemma [A], says that a set of points on a high-dimensional Euclidean space can be embedded into a lower-dimensional space such that distances are preserved. It has been observed that distances are still well-preserved when the embedding is a random orthogonal projection. This rigorous mathematical setting loosely mimics the practical setting of our paper - where high-dimensional data is projected into a low-dimensional space via a randomly initialized network. While the network does not produce an orthogonal embedding, it may still approximately preserve the distance between datapoints. This information can then be leveraged by the representation learner. We anticipate that the research community will be interested in exploring more theoretical explanations after publication.

Reviewer comment (Experiments):
It says in the paper that $g^{(k)}(x)$ uses the same architecture design of $f_{\theta}$. Even with random initialization, it may still follow a certain distribution family. How would changing this architecture affect the learning?

Author response:
There is literature (e.g. [B]) discussing what distributions are produced by randomly initialized networks, so we certainly agree with your point. We did not have space or capacity to give a full analysis of the types of distributions produced by each of the networks we used, but we hope that members of the community will find that direction interesting to pursue.

We found it sensible to use random projectors that share a similar architecture to the representation model since researchers have already determined classes of network architectures that have good inductive biases for extracting useful features. This information dictates the choice of representation model, and we reuse that information in choosing the projector architecture. In principle one could use any projector, but in preliminary experiments we found that matching inductive biases was beneficial.

Reviewer comment (Experiments):
The paper only exams the accuracy on the classification task for frozen representations. Nonetheless, a good representation could be used for various purposes, i.e., manipulation of each dimension in the latent space for generating new data, understanding the essential dynamics of the system in physical models and time series data. How could this strategy be applied to scenarios beyond classification?

Author response:
As with any SSRL model, training is done without reference to any specific downstream task. Evaluation is commonly performed on classification tasks since class labels are widely available, but it would be possible to predict any kind of label, such as segmentations or depth in image data. On tabular datasets which were our main focus, there are not many examples of downstream tasks other than classification for which standardized datasets are publicly available. However, please see our general comment for a new experiment on transfer learning.

Reviewer comment (Experiments):
Although the fact that choosing good augmentations usually requires domain knowledge, in many of the applications mentioned, such knowledge could be partially obtained using physical intuition. By comparing with models under random corruptions doesn't seem to be a fair comparison.

Author response:
We agree that with time and effort, it would be possible to produce useful semantic-preserving augmentations for most datasets, however our focus is on circumventing that process entirely. The advantage of LFR is that one does not need to put any effort into choosing good augmentations. In your last sentence we believe you are referring to one of the baseline methods, SCARF (Bahri et al., 2022), reviewed in Section 4.2. We believe that comparing to SCARF as one of the baselines is fair, since the random corruptions are exactly what (Bahri et al., 2022) proposed to use. We note that this is the only baseline that uses random corruptions. DACL for instance uses mixup for augmentations, while STab uses stochastic regularizations of networks.

Reviewer comment (Experiments):
The experimental section does not include any medium/large scale dataset, which obfuscates the benefit of the method for any SSL task.

Author response:
**W1: The experimental section does not include any medium/large scale dataset.**

We disagree that there are no large-scale datasets used in our experiments. Please see Table 3 in Appendix C.1 for a summary of datasets and their sizes. We point out that MIMIC-III has a training set with over 2.5M datapoints, and HEPMASS consists of 7M. Meanwhile, our medical image dataset Kvasir consists of images at 80x100 resolution which is a fair bit larger than something like CIFAR10.

In the general comment, we have added experiments on two additional datasets - CIFAR10 for the natural images domain, and a fault diagnosis dataset from (Eldele et al. 2021).

Reviewer comment (Experiments):
I found a transfer learning experiment lacking in the results.

Author response:
**W1/Q1/S1: Perform transfer learning experiments to support the main hypothesis of the work.**

We appreciate this suggestion and that you think such results would “strongly reinforce the claim of the paper”. We originally did not include transfer learning experiments because they are not straightforward in the tabular and time-series domain. However, since it was a common request, we did find a suitable setting from prior literature and applied LFR to it. The results are discussed in the general comment.

Reviewer comment (Evaluation):
More discussion about the pros/cons of using random projections would be valuable.

Author response:
**W3/Q2: More discussion about the pros/cons of using random projections would be valuable.**

We can provide some additional thoughts on the pros and cons of learning from random projections.

While we motivate our work with reference to domains where semantic-preserving augmentations are difficult to craft, there certainly are other domains where they do exist (e.g. natural images). In those cases, LFR does not make use of the expert domain knowledge that is available. This could lead to suboptimal performance, as indicated by our new results on CIFAR-10 in the general comment. One way to improve LFR would be through incorporating this domain knowledge when available.

LFR also requires initializing many networks for random projectors, but then discards many of them when selecting for diversity. This could be seen as wasteful, and the method could be improved if there were a good way to initialize more diverse networks to begin with, but we note that these are small networks, and that the entire process takes a negligible computational time compared to SSRL (3.01s to initialize and select for diversity, vs 849.21s to train). 

In terms of pros, we have tried to highlight these throughout the paper. LFR can be used on any domain or modality. LFR has better performance across tasks where semantic-preserving augmentations are not well-developed, including important domains like medical images (Table 2). LFR can be more computationally efficient than methods like SimCLR and SimSiam that use multiple passes through the large encoder and CPU-based augmentations (see general comment).

Reviewer comment (Experiments):
The main intuition of this method is to create random downstream tasks that capture different aspects of $z$. Therefore, I would suggest adding a transfer learning experiment, in which an encoder $f_\theta$ is learnt on a source dataset, and such features are transferred to other real downstream tasks. It would be really interesting to see a gain in such setting, wrt. other SOTA methods. I believe this would strongly reinforce the claim of the paper.

Author response:
**W1/Q1/S1: Perform transfer learning experiments to support the main hypothesis of the work.**

We appreciate this suggestion and that you think such results would “strongly reinforce the claim of the paper”. We originally did not include transfer learning experiments because they are not straightforward in the tabular and time-series domain. However, since it was a common request, we did find a suitable setting from prior literature and applied LFR to it. The results are discussed in the general comment.

Reviewer comment (Experiments):
I wonder if random projections are enough to capture the diversity inherent in complex datasets. I think the paper would benefit from a discussion on the pros and cons of using random projections. Note that the datasets shown in the paper are considered small-scale in the community, which limit the understanding on how effective LFR is for _any_ dataset. Larger, or more diverse, datasets would be extremely valuable.

Author response:
**W3/Q2: More discussion about the pros/cons of using random projections would be valuable.**

We can provide some additional thoughts on the pros and cons of learning from random projections.

While we motivate our work with reference to domains where semantic-preserving augmentations are difficult to craft, there certainly are other domains where they do exist (e.g. natural images). In those cases, LFR does not make use of the expert domain knowledge that is available. This could lead to suboptimal performance, as indicated by our new results on CIFAR-10 in the general comment. One way to improve LFR would be through incorporating this domain knowledge when available.

LFR also requires initializing many networks for random projectors, but then discards many of them when selecting for diversity. This could be seen as wasteful, and the method could be improved if there were a good way to initialize more diverse networks to begin with, but we note that these are small networks, and that the entire process takes a negligible computational time compared to SSRL (3.01s to initialize and select for diversity, vs 849.21s to train). 

In terms of pros, we have tried to highlight these throughout the paper. LFR can be used on any domain or modality. LFR has better performance across tasks where semantic-preserving augmentations are not well-developed, including important domains like medical images (Table 2). LFR can be more computationally efficient than methods like SimCLR and SimSiam that use multiple passes through the large encoder and CPU-based augmentations (see general comment).

Reviewer comment (Experiments):
I suggest evaluating LFR vs. computer vision methods in a different scenario than medical images. It is possible that SimCLR would perform better on CIFAR-10 or ImageNet, nevertheless that experiment would show how well LFR is performing against methods tailored for such scenarios. This would also help to understand how well random projections "emulate" the use of well-chosen augmentations. I think a natural images dataset is required for this paper for acceptance.

Author response:
**Q3/S2:  Evaluate in a setting where classic methods perform at their best. A natural images dataset is required for this paper for acceptance.**

This was a common request amongst reviewers, so we have provided discussion and experimental results on CIFAR10 in the general comment. In summary, LFR (using no augmentations) does not outperform SimCLR (using the full suite of augmentations) on natural image datasets. The same can be said for several domain-agnostic approaches including DIET and DACL.

Reviewer comment (Experiments):
A batch size of 256 is strongly detrimental for a contrastive approach such as SimCLR, for example. Small batch sizes harm performance for such methods. For the reader to fully grasp the scenarios where LFR is suitable, I suggest comparing with SimCLR in settings where the latter is known to excel.

Author response:
**Q5: A batch size of 256 is strongly detrimental for a contrastive approach. I suggest comparing with SimCLR in settings where the latter is known to excel.**

We agree that contrastive methods have been shown to perform well at very large batch sizes. We see this as a negative of such approaches, as it increases the computational requirements of running such methods. Based on Figure 4, LFR does well at reasonably small batch sizes. However, we take your point, so for the new CIFAR10 experiments in the general comment we did increase the batch sizes to 512.

Reviewer comment (Experiments):
More experiments would be required for the method to be rigorously evaluated; otherwise, there is doubt about whether random projections are valuable for small datasets only.

Author response:
**W3/Q2: More discussion about the pros/cons of using random projections would be valuable.**

We can provide some additional thoughts on the pros and cons of learning from random projections.

While we motivate our work with reference to domains where semantic-preserving augmentations are difficult to craft, there certainly are other domains where they do exist (e.g. natural images). In those cases, LFR does not make use of the expert domain knowledge that is available. This could lead to suboptimal performance, as indicated by our new results on CIFAR-10 in the general comment. One way to improve LFR would be through incorporating this domain knowledge when available.

LFR also requires initializing many networks for random projectors, but then discards many of them when selecting for diversity. This could be seen as wasteful, and the method could be improved if there were a good way to initialize more diverse networks to begin with, but we note that these are small networks, and that the entire process takes a negligible computational time compared to SSRL (3.01s to initialize and select for diversity, vs 849.21s to train). 

In terms of pros, we have tried to highlight these throughout the paper. LFR can be used on any domain or modality. LFR has better performance across tasks where semantic-preserving augmentations are not well-developed, including important domains like medical images (Table 2). LFR can be more computationally efficient than methods like SimCLR and SimSiam that use multiple passes through the large encoder and CPU-based augmentations (see general comment).

Reviewer comment (Experiments):
Perform transfer learning experiments to support the main hypothesis of the work (random projections allow learning different aspects of $z$, thus making $z$ more generally applicable).

Author response:
**W1/Q1/S1: Perform transfer learning experiments to support the main hypothesis of the work.**

We appreciate this suggestion and that you think such results would “strongly reinforce the claim of the paper”. We originally did not include transfer learning experiments because they are not straightforward in the tabular and time-series domain. However, since it was a common request, we did find a suitable setting from prior literature and applied LFR to it. The results are discussed in the general comment.

Reviewer comment (Experiments):
It would be interesting for the reader to see how the training behaves with joint training; I suggest adding some of these plots in the Appendix.

Author response:
**S3: It would be interesting to see how joint training behaves.**
 
We did experiment with joint training, with an ablation on Kvasir shown in Figure 4. This ablation helps to justify why we used an EM-like algorithm rather than the simpler joint training idea.

Reviewer comment (Experiments):
The main weakness for me is inability to compare with the existing method on well-known datasets in computer vision tasks. We have 3 x time series, 3 x tabular, and 1 x image - where Kvasir is not commonly used dataset. Not sure if the results are not picking the datasets that show good results. Why not presents the results on ImageNet, and if computationally not possible - cifar100.

Author response:
**W1: The main weakness for me is inability to compare with the existing method on well-known datasets in computer vision tasks. We have 3 x time series, 3 x tabular, and 1 x image - where Kvasir is not commonly used dataset.**

We must disagree on one aspect - Kvasir is in fact widely used for benchmarking applications of ML to healthcare with over 1000 combined citations [Pogorelov et al. 2017] [A]. We used it as an example of the differences between domains (within the image modality), and how natural image augmentations are not necessarily semantic-preserving for medical images. This is a core motivation of our paper - moving towards domain-agnostic SSRL methods - and was not picked based on results.

As to your interest in seeing results on natural image data as well, and based on multiple requests from reviewers, we have rerun our experimental pipeline on CIFAR10. The results and discussion are in the general comment. In summary, LFR (using no augmentations) does not outperform SimCLR (using the full suite of augmentations) on natural image datasets. The same can be said for several domain-agnostic approaches including DIET and DACL.

Reviewer comment (Experiments):
We don't know the final computational overhead of the initialization and comparison to any SSL method. In the appendix we have total time spent for a particular dataset (e.g. Kvasir V100 1095 GPU hours for all experiments). Would be better to know the comparison between SimCLR/SimSiam vs LFR.

Author response:
**W3: We don't know the final computational overhead of the initialization and comparison to any SSL method. In the appendix we have total time spent for a particular dataset (e.g. Kvasir V100 1095 GPU hours for all experiments). Would be better to know the comparison between SimCLR/SimSiam vs LFR.**

The initialization and selection of random projectors is a minor part of overall training time with LFR. As a concrete example, on the Income dataset training LFR for 100 epochs took 849.21s on our machine, while the DPP selection took 3.01s. 

In terms of model training, SimCLR took around 9 hours on the Kvasir dataset, while LFR completed in just 2 hours on the same machine. Please see the general comment for a discussion of this result and our analysis of computational costs.

Reviewer comment (Theory):
Lack of more theoretical explanation why it should work? What random projectors can be used? etc.

Author response:
**W5: Lack of more theoretical explanation why it should work? What random projectors can be used? Etc.**

We found it sensible to use random projectors that share a similar architecture to the representation model. For each modality, researchers have already determined classes of network architectures that have good inductive biases for extracting useful features. This information dictates the choice of representation model, and we reuse that information in choosing the projector architecture. In principle one could use any projector, but in preliminary experiments we found that matching inductive biases was beneficial. While we do not have a complete theoretical explanation for the success of learning from randomness, we can offer you some indication of an interesting direction. We anticipate that the research community will be interested in exploring theoretical explanations after publication.

Random orthogonal projections on high dimensional spaces have been studied in mathematics and found to have interesting properties. One result, known as the Johnson-Lindenstrauss lemma [B], says that a set of points on a high-dimensional Euclidean space can be embedded into a lower-dimensional space such that distances are preserved. It has been observed that distances are still well-preserved when the embedding is a random orthogonal projection. This rigorous mathematical setting loosely mimics the practical setting of our paper - where high-dimensional data is projected into a low-dimensional space via a randomly initialized network. While the network does not produce an orthogonal embedding, it may still approximately preserve the distance between datapoints. This information can then be leveraged by the representation learner.

We do not claim this to be a sufficient theoretical explanation, but we hope to investigate it in future work and look forward to other explanations that the community can propose.

Reviewer comment (Experiments):
How the method perform without using heavy SSL (SimCLR) data augmentations?

Author response:
**Q1:  How the method perform without using heavy SSL (SimCLR) data augmentations?**

We believe you are asking how LFR would perform if it was used *with* heavy data augmentations, since our method does not use any augmentations to begin with. It could be possible to perform data augmentations before sending a datapoint through our encoder and random projectors. Such augmentations could potentially increase the diversity of data seen by the model, but it is important to note that our method is not contrastive in the same way as, for example SimCLR. In SimCLR the augmentations produce two views of a datapoint which are deemed to be semantically the same, and the representations of the views are pushed together. Augmenting data in LFR would not have the same meaning. In effect, the random projectors of LFR are producing multiple views, and the representation model must encode enough information for the predictor heads to match the views. Whether data is augmented or not before being projected does not fundamentally change this process.

Since we designed and tested our method for domains where semantic-preserving augmentations are not known or easy to create, we did not conduct a test that uses heavy data augmentations.

Reviewer comment (Experiments):
One of the issues I have with this paper is that despite the use of several datasets used to evaluate LFR, the commonly used ones such as CIFAR-10/100 are not among them. While I understand that LFR does not aim to improve performance on these datasets (since natural image augmentations already perform very well) it would be interesting to examine those to compare as well. It would be useful to know if LFR is better/worse than optimized augmentations, such as those used in SimCLR.

Author response:
**W1: While LFR does not aim to improve performance on datasets like CIFAR10, it would be useful to know if LFR is better/worse than optimized augmentations, such as those used in SimCLR.**

This was a common sentiment amongst reviewers, so we have provided discussion and experimental results on CIFAR10 in the general comment. In summary, LFR (using no augmentations) does not outperform SimCLR (using the full suite of augmentations) on natural image datasets. The same can be said for several domain-agnostic approaches including DIET and DACL.

Reviewer comment (Experiments):
I think it would be interesting to see some ablations on the distance metric used for training. Right now, the authors use Barlow Twins as the metric, but it would be interesting to perform ablations with e.g. MSE or Contrastive losses for this (although I must note that the authors do make an argument for this design decision in the paper).

Author response:
**W2.1: It would be interesting to perform ablations with e.g. MSE or Contrastive losses.**

Since multiple reviewers brought up this point, we have addressed it in the general rebuttal by running experiments using LFR with InfoNCE instead of Barlow Twins. In addition to the arguments for our design decision in the paper, our batch-wise Barlow Twins loss did give a slight improvement over InfoNCE for downstream performance.

Reviewer comment (Experiments):
I think it would also be interesting to see the transferability of the trained models across different datasets. I would be interested in knowing if LFR leads to learning representations that are good for the particular dataset, or good for the chosen modality in general.

Author response:
**W2.2: It would also be interesting to see the transferability of the trained models across different datasets.** 

We originally did not include transfer learning experiments because they are not straightforward in the tabular and time-series domain. However, since it was a common request, we did find a suitable setting from prior literature and applied LFR to it. The results are discussed in the general comment.

Reviewer comment (Experiments):
It is good to see that experiments include various domains including time series, tabular, and image, but they seem to be relatively small and not commonly used for benchmarking machine learning models. For example, Kvasir is a medical image dataset, which is different from the widely used "natural" image datasets; it should be categorized differently from natural image datasets. Authors may want to refer to [Lee et al.] and [Wu et al.] to find commonly used datasets to provide the general applicability to various domains and scalability of the proposed method.

Author response:
We respectfully disagree on this point. Table 3 in Appendix C.1 summarizes all the datasets we used, which includes MIMIC-III at over 2.5M training examples, and HEPMASS at 7M. Compared with common datasets like CIFAR-10, these datasets are very large scale. The smaller datasets, HAR and Epilepsy, were chosen because they have previously been used for benchmarking time-series SSRL methods (e.g. Eledele et al. 2021), while Income and Theorem have been used for tabular benchmarking (e.g. Hajiramenzanali et al. 2022). In fact these datasets are commonly used: the paper introducing MIMIC-III has over 6000 citations [A], Kvasir has over 1000 combined [Pogorelov et al. 2017] [B], HAR over 2000 [Anguita et al. 2013], Epilepsy over 3000 [Andrzejak 2001], and Income over 2000 [Kohavi 1996]. As a side note, we did explicitly call out that Kvasir was a medical image dataset (not natural image) in Table 2 and Section 4.1.

You also mentioned two domain-agnostic works with more suggestions for other datasets to consider. [Lee et al.] uses image, speech, and tabular datasets, while [Wu et al.] experiment on images, point clouds, and audio. As can be seen, there is a lack of consistency in which modalities are tested for domain-agnostic work, and unfortunately we are not able to introduce several new modalities in the discussion period. However, to partially accommodate your requests, we have repeated our experimental setup on CIFAR-10 as a representative natural image dataset with the results shown in the general comment. The DABS benchmark does appear useful, and we will investigate it for the future.

[A] Johnson et al., “MIMIC-III, a freely accessible critical care database”, Nature Scientific Data 2016.
[B] Jha et al., “Kvasir-SEG: A Segmented Polyp Dataset”, International Conference on Multimedia Modeling 2020.

Reviewer comment (Theory):
While the authors claim that the optimization strategy for the proposed method is EM, but it is not clear how the proposed alternating optimization is related with EM by looking at the formulation. I think the transition from Eq. (2) to Eq. (3--4) requires more explanation supported with some math.

Author response:
Consider the MLE objective in Equation 2. As variable $\mathbf{z}_i$'s distribution could be intractable, we introduce a manageable distribution $q(\mathbf{z}_i)$ such that

$\sum_i\sum_k \log \int_{\mathbf{z}_i}  p\left(\mathbf{y}_i^{(k)}\Bigm| \mathbf{z}_i\right) p\left(\mathbf{z}_i\mid \mathbf{x}_i\right)   d\mathbf{z}_i = $

$ \sum_i\sum_k \log \int_{\mathbf{z}_i} q(\mathbf{z}_i)\frac{ p\left(\mathbf{y}_i^{(k)}\Bigm| \mathbf{z}_i\right) p\left(\mathbf{z}_i\mid \mathbf{x}_i\right) }{q(\mathbf{z}_i)}d\mathbf{z}_i $

Using Jensen's inequality, the lower bound of the objective is therefore
$\sum_i\sum_k \log \int_{\mathbf{z}_i} q(\mathbf{z}_i)\frac{ p\left(\mathbf{y}_i^{(k)}\Bigm| \mathbf{z}_i\right) p\left(\mathbf{z}_i\mid \mathbf{x}_i\right) }{q(\mathbf{z}_i)}d\mathbf{z}_i \geq$

$\sum_i\sum_k \int_{\mathbf{z}_i} q(\mathbf{z}_i)\log\frac{ p\left(\mathbf{y}_i^{(k)}\Bigm| \mathbf{z}_i\right) p\left(\mathbf{z}_i\mid \mathbf{x}_i\right) }{q(\mathbf{z}_i)}d\mathbf{z}_i.$

Hence, to maximize the variational lower-bound of the above equation with respect to the proposed distribution $q(\mathbf{z}_i)$, we note its optimal solution of $q(\mathbf{z}_i)$ is simply letting the equation hold (as in the classic EM algorithm) such that

$\sum_i\sum_k \log \int_{\mathbf{z}_i} q(\mathbf{z}_i)\frac{ p\left(\mathbf{y}_i^{(k)}\Bigm| \mathbf{z}_i\right) p\left(\mathbf{z}_i\mid \mathbf{x}_i\right) }{q(\mathbf{z}_i)}d\mathbf{z}_i =$

$\sum_i\sum_k \int_{\mathbf{z}_i} q(\mathbf{z}_i)\log\frac{ p\left(\mathbf{y}_i^{(k)}\Bigm| \mathbf{z}_i\right) p\left(\mathbf{z}_i\mid \mathbf{x}_i\right) }{q(\mathbf{z}_i)}d\mathbf{z}_i,$

which, in turn, requires the following equation to hold
$\frac{ p\left(\mathbf{y}_i^{(k)}\Bigm| \mathbf{z}_i\right) p\left(\mathbf{z}_i\mid \mathbf{x}_i\right) }{q(\mathbf{z}_i)} = C$, where $C$ is an constant. By transforming this equation a bit, the optimal solution of $q(\mathbf{z}_i)$ is

$q(\mathbf{z}_i) = \frac{ p\left(\mathbf{y}_i^{(k)}\Bigm| \mathbf{z}_i\right)}{ p\left(\mathbf{y}_i^{(k)}\Bigm| \mathbf{x}_i\right)} p\left(\mathbf{z}_i\mid \mathbf{x}_i\right).$

As both $p\left(\mathbf{y}_i^{(k)}\Bigm| \mathbf{z}_i\right)$ and $p\left(\mathbf{y}_i^{(k)}\Bigm| \mathbf{x}_i\right)$ 

are delta distributions with probability 1 conditioned on deterministic functions modelled by $g^{(k)}$ and $h^{(k)}_\phi$ respectively, the optimal solution of $q(\mathbf{z}_i)$ is simply $p(\mathbf{z}_i|\mathbf{x}_i)$ given the conditions are satisfied. In other words, we need $p(\mathbf{z}_i|\mathbf{x}_i)$ that can let $\mathbf{y}_i^{(k)}$ become 

$h^{(k)}_{\phi}(\mathbf{z}_i) $ for all $k$. Thus, the optimization is essentially an EM algorithm where we optimize $\theta$ and $\phi$ alternatively to gradually increase the conditional likelihood of Eq 2.

Reviewer comment (Experiments):
The claim around batch-wise barlow twins that MSE is preferred over cross-entropy/contrastive/triplet losses is not justified. Isn't the batch-wise barlow twins loss just a kind of contrastive loss, in that it contrasts all samples within the batch? Note that the original contrastive loss (not the InfoNCE variation) also computes the MSE loss. An ablation study with different type of losses might also be helpful.

Author response:
Fortunately, the core method of LFR, using random projections of data as a pre-training task, can be used with many different losses. We advocated for the batch-wise Barlow Twins loss because of its ability to learn disentangled representations [Zbontar et al. 2021]. We agree that Barlow Twins could be viewed as a contrastive loss, and did not claim otherwise in the paper, and also agree that a comparison could provide more insight. To this end, we used the InfoNCE loss for two datasets to show that batch-wise Barlow Twins leads to empirical improvements - our new results are discussed in the general rebuttal.

Reviewer comment (Theory):
The criterion for diversity encouragement requires more intuition. It is hard to imagine what is going on when optimizing the proposed learning objective. Also, what is the computational cost for the NP-hard objective function?

Author response:
Although the problem is NP-hard in theory, we only need to solve an instance of it with finite size. Compared to the cost of training an encoder, the diversity selection procedure using the Fast Determinantal Point Process (Chen et al. 2018) is an almost negligible contribution to the total training time. As a concrete example, on the Income dataset training LFR for 100 epochs took 849.21s on our machine, while the DPP selection took 3.01s.

Reviewer comment (Experiments):
The comparison might not be fair as the proposed method requires more computational cost to encode input with multiple random projectors and predictors compared to other baselines. The computational cost should be matched for a fair comparison and reported.

Author response:
Upon examining the details of each training method, it becomes clear that LFR actually requires *less* computation than alternatives like SimCLR and SimSiam. Please see our general comment for a full discussion on this point.

Many SSRL methods incorporate small predictor heads, so LFR is not unique in this regard. Meanwhile, the random projectors that LFR introduces are not a source of ongoing computational cost; since they are fixed at the start of training we perform a single forward pass over the entire dataset and reuse these results throughout training.

Reviewer comment (Experiments):
After discussion with authors, I feel that the experimental results are not sufficient to support the claim that they cover "a wide range of representation learning tasks that span diverse modalities and real-world applications." Initially their experiments covered time series, medical image, and tabular domains, and the additional results in the natural image domain show that their method is not effective for natural images, compared to other baselines. Authors are encouraged to explicitly limit the scope to the domains they experimented in the title/abstract/intro.

Author response:
Thank you for clarifying your points and updating your position.

We maintain that our claim is accurate and not overreaching, as we have shown that the method is effective on various domains within three modalities. We did not claim that the method can match augmentation-heavy approaches on the natural images domain, and have provided clear evidence to the contrary. We did use datasets from multiple prior works in SSRL, just not from the three specific papers you mention, which unfortunately all use different baselines with little to no overlap

We are glad that you found the additional steps of derivation helpful, and we can certainly include them in the next update to the paper.

The diversity encouragement method uses a detrimental point process on the projected representations of a batch of data. It selects random projectors that produce maximally diverse representations as given by the determinant of a matrix constructed from those representations. For an in-depth discussion on DPPs and diversity encouragement, please see [A].

We have answered your question on why using multiple random projectors does not incur heightened computational costs compared to baselines - random projectors are not trained or backpropagated through, so we can pre-compute their representations once and reuse them throughout training. We used the same key hyperparameters across methods, including architectures and batch size. Based on prior work, we agree that performance could be improved on datasets like CIFAR-10 by greatly increasing model size, batch size, and training epochs, but we do not have the resources at this time to match such settings. While we agree that a full empirical review of all domain-agnostic SSRL methods across a wide range of baselines, each with a range of architectures and extensively tuned hyperparameters, would be a beneficial resource to the community, such a sweeping review of the topic is beyond the scope of our work and beyond the resources we have available. 

[A] Kulesza and Taskar, “Determinantal point processes for machine learning”, Foundations and Trends in Machine Learning 2012