Paper: LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architectures (ICLR-2024)

Reviewer comment (Evaluation):
No disucssions on computational overhead and runtime.

Author response:
We thank the reviewer for bringing up this point as its important for several reasons:

* First is to provide the community with information on run-time overhead of LiDAR to ensure that the overhead is within bounds for the use case considered in the paper. We conduct an experiment where-in we measure the wall-clock time for running RankMe vs LiDAR for I-JEPA evaluation and report our findings in **Appendix 12** and **Table 34**. We include the table at the end of our comment for convenience. We find that the wall-clock time for evaluation on a 1GPU machine has a median value of 0.71 seconds for LiDAR which is very reasonable albeit slower than RankMe which has a median value of 0.38 seconds. The difference is is understandable given the non-optimized nature of our implementation. We find that the times reported above are much smaller than the time taken for feature extraction also reported in **Table 34**; the feature extraction times are of the order of 10s of seconds up to several minutes. Also note that standard linear probing evaluation on the ImageNet-1K dataset takes considerably longer and/or requires multiple GPUs, potential data labeling and curation [can we mention the times we observed from probing runs?].
* The second reason why the above point raised by the reviewer is important is that our current research findings may serve as a starting point to explore the use of LiDAR directly as a training loss function (regularizer). We think this is a great direction for future work as it motivates exploring techniques from randomized numerical linear algebra (RanNLA) and beyond to ensure an efficient implementation that can work across a range of numerical precisions relevant to practice today.

-----------------
**Table 34**: Wall-clock time to calculate effective rank for \emph{RankMe} and \emph{LiDAR}. The emebddings are gathered from an I-JEPA~\citep{ijepa} checkpoint for all methods.
| Method          | Sample count     | Median time (std. dev.) | Feature extraction |
|-----------------|------------------|-------------------------|--------------------|
|                 |                  | (seconds)               | (seconds)          |
| RankMe          | 10K              | 0.38 (2.38)             | 11.29              |
| LiDAR (Student) | 10K (n=1K, q=10) | 0.53 (0.07)             | 23.22              |
| LiDAR (Student) | 50K (n=1K, q=50) | 0.71 (0.06)             | 96.95              |

We once again thank the reviewer for raising a great question. We hope that our response addresses your concerns raised during review. We request the reviewer to provide suggestions that can help us improve the paper further so that it meets the reviewer’s bar for acceptance as a research publication.

Reviewer comment (Experiments):
The manuscript lacks evaluation on out-of-distribution (OOD) target datasets. Although the ImageNet dataset results are promising, the evidence for LiDAR's effectiveness in more realistic downstream tasks, often involving OOD datasets, is missing. Comparing with datasets like iNaturalist-2018 or Stanford Cars, following the protocol in RankMe, would strengthen LiDAR's position as a useful proxy metric.

Author response:
This is a very important and valid point raised by the reviewer. Fortunately, we were able to evaluate the performance of all of the SSL methods considered in our paper namely I-JEPA, data2vec, DINO, VICReg and SimCLR on 5 commonly used datasets that are considered out-of-distribution (OOD) in the SSL literature. We are excited to include them in the paper to strengthen the contribution. The datasets considered here are CIFAR10, CIFAR100, EuroSAT, Food101 and SUN397, which represent a reasonable subset of OOD datasets considered in RankMe [Garrido 2023]. 

We follow the protocol used in RankMe [Garrido 2023] and select the most performant checkpoint for each metric (ImageNet Oracle (based on validation dataset), RankMe, augmented RankMe (where applicable) and LiDAR). These checkpoints are trained and evaluated with the datasets mentioned above. A detailed description of the evaluation protocol is in Appendix Section 10.1 of the paper. The results for the various methods are available in the following tables:

- **Table 22** (average accuracies) and **23** (breakdown by dataset) for I-JEPA where the hyperparameters are set via grid search as done in I-JEPA[Assran2023]
- **Table 24** (average accuracies) and **25** (breakdown by dataset) for I-JEPA where the hyperparameters are set via random search
- **Table 26** (average accuracies) and **27** (breakdown by dataset) for data2vec where the hyperparameters are set via grid search
- **Table 28** (average accuracies) and **29** (breakdown by dataset) for DINO where the hyperparameters are set via random search
- **Table 30** (average accuracies) and **31** (breakdown by dataset) for VICReg where the hyperparameter values are copied over from RankMe[Garrido2023]
- **Table 32** (average accuracies) and **33** (breakdown by dataset) for SimCLR where the hyperparameter values are set via random search

We observe from the average accuracies that LiDAR performs better than RankMe in most cases and matches RankMe performance for a handful of cases considered in our experiments. Notably LiDAR is able to perform better than the ImageNet Oracle in OOD evaluation in certain cases ( see Table 24, Table 28, Table 30) . This observation is consistent with and  similar to the observations made in RankMe[Garrido2023] that suggests new metrics are needed to evaluate downstream performance.

We have included the average linear probe accuracies calculated on the OOD datasets in the following comments for convenience. These tables and the per-dataset breakdown are also available in the updated draft

**Table 22:** I_JEPA: Average linear probe accuracy recovered by \emph{RankMe} and \emph{LiDAR} on OOD datasets. Hyperparameters set via grid search.
| Metric                            | LR                      | WD                      | Target mask scale            | Context mask scale    | Overall                 |
|-----------------------------------|-------------------------|-------------------------|-------------------------|-------------------------|-------------------------|
| \textcolor{gray}{ImageNet Oracle} | \textcolor{gray}{78.49} | \textcolor{gray}{78.49} | \textcolor{gray}{78.49} | \textcolor{gray}{78.49} | \textcolor{gray}{78.49} |
| RankMe                            | 72.09                   | 75.45                   | 75.45                   | 75.15                   | 72.09                   |
| RankMe (aug.) (Student)           | \textbf{77.78}          | \textbf{77.78}          | \textbf{77.78}          | 77.78                   |\textbf{ 77.78}                   |
| RankMe (aug.) (Teacher)           | 72.09                   | 72.14                   | 70.20                   | 75.15                   | 72.09                   |
| LiDAR (Student)                   | \textbf{77.78}          | \textbf{77.78}          | \textbf{77.78}          | 77.78                   | \textbf{77.78}                   |
| LiDAR (Teacher)                   | 74.33                   | 76.84                   | 73.91                   | \textbf{78.30}          | 74.33          |

Reviewer comment (Experiments):
The comparative analysis appears limited, with LiDAR mainly being evaluated against RankMe. Other potential baselines, like $\alpha$-ReQ mentioned in related work, are overlooked.

Author response:
We thank the reviewer for raising this excellent point. We agree that \alpha-ReQ is relevant to the topic of representation evaluation. However, we decided to focus on RankMe as it was shown to work better than \alpha-ReQ in-distribution and comparable OOD (and is currently the state-of-the-art approach for predicting downstream performance for joint-embedding architectures). But we will add comparisons in a future/camera-ready version of the paper.

Reviewer comment (Reproducibility):
In Section 4.1, the authors advocate for dimensionality reduction (DR) of embedding features, possibly adding another layer of dependency for LiDAR evaluation. The necessity of DR for LiDAR's effective application, and any specific DR algorithms used in experiments, remain ambiguous.

Author response:
We thank the reviewer for pointing out this ambiguity in our writing. First of all we apologize for the confusion caused by our statement on dimensionality reduction (DR). Our intention is not to argue for dimensionality reduction but to use the trivial bound in equation 6 in Section 4.1 as a guide to select the number of classes. We suggest that the users of LiDAR set n to be greater than the feature vector size to not impose any artificial constraint on the rank (LiDAR score). We follow this rule of thumb to select n for all of the SSL methods considered in our as explained in our answer above to the question on hyperparameter selection. We mention that dimensionality reduction could be potentially used to reduce computational complexity for cases where (p >> n) but we avoid this in our evaluations to ensure fairness. We once again apologize for any confusion.

Reviewer comment (Theory):
Although the authors claim LiDAR integrates the SSL objective, the manuscript does not delve into the relationship between LiDAR and different SSL losses.

Author response:
We agree with the reviewer that the paper does not attempt to establish any relationship between LiDAR and various SSL objectives used in practice. We proposed LiDAR that captures the discriminative aspect of the SSL methods (using image as a class) and conjecture that the discriminative directions LiDAR is sensitive to are the same discriminative directions learned by various SSL objectives considered in the paper. We focused our efforts in this paper to thoroughly evaluate the effectiveness of LiDAR as a metric that predicts downstream performance for several SSL objectives. We leave the topic of establishing potential relationship between LiDAR and different SSL losses as a topic for future research. We thank the reviewer for their excellent suggestion as we delve deeper into this topic.

Reviewer comment (Theory):
Scatter matrices, $\Sigma_w$ and $\Sigma_b$, having dimensions (p $\times$ p), are influenced by both n and the data dimensionality (p). In Section 4.1, the authors note they "maintain a total of 50 features" for VICreg. How does this align with rank constraints dictated by the data dimensionality?

Author response:
We thank the reviewer for the question. We apologize to the reviewer for the confusion caused above due to multiple typos in our initial draft. We note that the above (n=5K, q=10) LiDAR hyperparamters are used for SimCLR leading to a dataset with 50000 (and not 50) features. Secondly, we use (n=10K, q=10) for VICReg as noted in Section 9.1 in the Appendix of the original draft. We have updated our draft to remove any confusion. Once gain we apologize to the reviewer for causing confusion in our presentation. 

The value of n are chosen for various SSL objectives to ensure that the number of surrogate classes exceeds the length of feature vectors (p). Thus, our choice of n ensures that we do not impose any artificial constraints on the rank estimated by LiDAR.

Reviewer comment (Writing):
The abbreviation of the proposed method "LiDAR" is irrelavent of the problem it tries to solve. The irrelavent abbreviation misleads the readers what the paper want to express. Some readers may think the paper is on the lidar.

Author response:
We recognize the potential for confusion which can often occur when producing names for methods that are intended to be memorable and creative, which we feel strongly about. Therefore, to reduce potential for confusion, we expand the term LiDAR in the abstract to **Li**near **D**iscriminant **A**nalysis **R**ank. We hope that along with the title referring to the goal of the paper this makes it clear to future readers that LiDAR refers to a metric for evaluating representations of joint embedding methods where the name is meant to be a playful way to refer to ‘sensing’ representation quality, as mentioned in the title.

Reviewer comment (Writing):
In section 2.1: In practice, a set of downstream tasks {T_j} are used to asses -> In practice, a set of downstream tasks {T_j} are used to assess.

Author response:
We thank the reviewer for carefully reading our paper and bringing the above issues to our notice. We have fixed the issues in the current revision of our draft. We commit to fixing any issues that remain in a camera-ready version of our paper and will ensure the paper goes through a round of proofreading and spellcheck.