Paper: Explorative Latent Self-Supervised Active Search Algorithm (ELSA) (ICLR-2024)

Reviewer comment (Novelty):
The novelty of the work is very small as both active learning and self-supervised learning are well known.

Author response:
Thank you for your comments. Both active learning and self-supervised learning techniques are used in developing ELSA. ELSA is more of a search algorithm than an active learning technique. The primary motivation behind the development of an algorithm like ELSA is to ease the task of manual labelling.

Reviewer comment (Novelty):
The solution that this paper takes is quite common, including nearest neighbor search and random sampler.

Author response:
We have used an ordinary nearest neighbour search routine for this task, which forms one of our components, but our random sampler is unique. It differs from the usual random search because it tries to find the most probable cluster centres from the dataset. We have shown in detail in Appendix B how RandS differs from a vanilla random sampler. Moreover, we also provide the intuition behind why RandS works so well in Appendix B. RandS is designed to search orthogonally to the nearest neighbour sampler, making the two work independently and symbiotically.  Appendix F shows how adding the RandS improves the search compared to the nearest neighbour search.

Reviewer comment (Novelty):
It looks like the pape combines several existing methods into one piece without motivating the method well and explaining why they are combined.

Author response:
We have added Appendix A, where we provide a detailed description of the algorithm and the interplay of its components. We have also added Section 4.4 in the paper in order to motivate the readers to the problem statement and give a basic intuition on the working of ELSA. Both sections describe why we need all the components used to design the algorithm.

Reviewer comment (Writing):
The paper is poorly written without an official problem definition.

Author response:
We have made some changes following the comments of the reviewer. We have added section 4.4,  Appendix A, Appendix B and Appendix F to make the working of the algorithm and the problem statement more straightforward. If the reviewer has some additional comments, we are open to them.

Reviewer comment (Novelty):
The contribution is not clearly. For instance why nearest neighbor search would be listed as contribution since it is just a common method. Similarly, the random sampling method is also a very routine method.

Author response:
Our random sampler differs from the vanilla one, as described in Appendix B. We have not claimed the nearest neighbour search as our contribution rather, it has been described and formalised for the sake of completeness.

Reviewer comment (Theory):
The method lacks theoretical support. For instance, what is the error bound? How does the method perform under noisy setting?

Author response:
ELSA works on the embedding space of a self-supervised algorithm. The mathematical formation of these spaces is still an active field of research; hence, providing any theoretical bound of error for the algorithm will be very difficult due to the sheer complexity of such spaces. We believe this to be beyond the scope of our study and hence have not explored the theoretical error bound. Nevertheless, we have added experiments in the appendix (Appendix D) to show how our algorithm performs under noisy settings. ELSA works well even in moderately noisy settings, but the labelling efficiency sufferers as we add more noise. The discovery rate remains approximately constant to the overall noise added. If the reviewer wants any other theoretical justification, we will be happy to answer them.

Reviewer comment (Experiments):
Not enough empirical evaluations. It is necessary to evaluate other state-of-the-art tabular benchmarks in Table 5.

Author response:
Thank you for your comments. Table 5 already includes eight different state-of-the-art self-supervised methods on which we have benchmarked our algorithm.  It would be helpful if the reviewer could elaborate on this statement: *"necessary to evaluate other state-of-the-art tabular benchmarks"*; if the reviewer wants us to run more benchmarks, their suggestions are welcome.

Reviewer comment (Novelty):
Novelty and limitations. The theoretical justification is interesting but the novelty in the method itself is slightly incremental, and the proposed algorithm seems based on a simple modification.

Author response:
The novelty of the method lies in connecting the different search components. We have used two main components in our algorithm, one of which is the standard nearest neighbour search, the other one being RandS. We claim RandS as one of our main contributions to this paper. We have described in detail in Appendix A how RandS, along with the NN component, can maximise the search. In addition, we have also described in Appendix B why a sophisticated random sampler like RandS is required for this purpose and why the usual random sampler would fail in this scenario. There are multiple works on automatizing the task of labelling in the literature, some of which are described in the paper and how they are different from our work; moreover, we focus on a more general dataset like ImageNet and Food101 to benchmark and test our algorithm. We have also shown from our analysis in Appendix D how the two components of ELSA work orthogonal to each other and maximise the search without affecting each other.

Reviewer comment (Reproducibility):
The reviewer could not find the code to check reproducibility.

Author response:
All our source codes are in the supplementary material along with the analysis notebooks and log files. We plan to link the official Github repository in the camera-ready version.

Reviewer comment (Experiments):
Lack of analysis of the impact of hyperparameters such as “a” in the NEAREST NEIGHBOUR SEARCH: The authors do not provide a detailed analysis of the impact of hyperparameters on the performance of ELSA. This is an important aspect of the algorithm, as the choice of hyperparameters can have a significant impact on its effectiveness. A more thorough analysis of the impact of hyperparameters would help to identify the optimal settings for different datasets and applications.

Author response:
Thank you for your comments. Following the suggestions from the reviewers, we have done an in-depth analysis of several hyperparameters of our algorithm. We have added the hyperparameter analysis to the appendix of our paper (Appendix C).

Reviewer comment (Writing):
The meanings of some symbols such as $L_e$ and $d_r$ in Section 3.1 have not been provided with sufficient clarity.

Author response:
$L_e$ is the labelling efficiency, i.e. the ratio of the number of positive samples discovered to the total number of samples labelled during the search process.
$d_r$ is the discovery rate, i.e. the ratio of the number of positive samples discovered by the search algorithm to the total number of positive samples present in the dataset. To give a better understanding to the readers, we have updated section 3.1
to provide better clarity.

Reviewer comment (Experiments):
How sensitive is ELSA to the choice of hyperparameters? Can you provide a more detailed analysis of the impact of hyperparameters on the performance of the algorithm? This would help to identify the optimal settings for different datasets and applications.

Author response:
ELSA is robust when it comes to the choice of hyperparameters. Please see Appendix C for a detailed analysis.

Reviewer comment (Experiments):
How might ELSA be extended or modified to address other challenges in active learning-based labeling, such as the presence of noisy or mislabeled data?

Author response:
We have added a section for noise analysis for ELSA, which can be found in the appendix (Appendix D). From our study, we see that ELSA works even in moderately noisy settings, but the labelling efficiency of the algorithm suffers as more noise is added to the algorithm. The discovery rate remains approximately constant to the overall noise that is added. The issue can be tackled using a more sophisticated active learning sampler that works well in noisy settings. Due to the sheer simplicity of the algorithm, changing the active learning component to a more sophisticated one should not be very difficult.