Paper: Active Automated Machine Learning with Self-Training (ICLR-2024)

Reviewer comment (Presentation):
Should the line 2 and line 3 in Algorithm 1 be swapped?

Author response:
Q1: You are right, these lines should be switched.

Reviewer comment (Experiments):
**Hyperparameter Sensitivity:** The method exhibits a pronounced sensitivity to hyperparameters, which poses practical usability challenges. If users cannot ascertain and set hyperparameters effectively a priori, it could hinder the practical deployment and user-friendliness of this framework, making this aspect a significant limitation that needs addressing to bolster the model's utility in real-world applications.

Author response:
Q2: That is true, the choice of the hyperparameters is crucial and would be, in the best case, not hard coded, but chosen by the framework during inference (depending on the data distribution or whatsoever).

Reviewer comment (Experiments):
What guidance or strategy does the paper offer regarding the a priori selection of hyperparameters? Clear instructions or criteria for choosing hyperparameters beforehand are crucial for users who aim to apply this framework effectively in practical scenarios. Your elucidation on this aspect would significantly contribute to the model’s usability and overall applicability.

Author response:
Q2: That is true, the choice of the hyperparameters is crucial and would be, in the best case, not hard coded, but chosen by the framework during inference (depending on the data distribution or whatsoever).

Reviewer comment (Experiments):
Could the authors provide an ablation study for AutoASL? It would be insightful to discern which specific components contribute most significantly to the overall performance of the system.

Author response:
Q1+3: You are completely right. While doing ablation studies where we  
first incorporated only the self-labeling component and then only the active learning component, we actually found out, that the active learning component plays a major role. Therefore we definitely need more understanding mabe also about differences regarding binary or multiclass settings.

Reviewer comment (Evaluation):
The authors suggest extending the model to multi-class classification is relatively straightforward. Given this assertion, could you elucidate which components—between self-training and active learning—will likely be more pivotal or influential in a multi-class scenario? A clear delineation of the impact or contribution of each component in multi-class settings would be instrumental in understanding and applying the model effectively in broader contexts.

Author response:
Q1+3: You are completely right. While doing ablation studies where we  
first incorporated only the self-labeling component and then only the active learning component, we actually found out, that the active learning component plays a major role. Therefore we definitely need more understanding mabe also about differences regarding binary or multiclass settings.

Reviewer comment (Experiments):
Why disagreement set should be removed from the unlabeled set? I think labeling them by oracle should also be beneficial? The paper claims that removing these items "prevents them from getting wrongly labeled in future iterations", then why items in unconfident set are kept?

Author response:
Q1: Yes, of course labeling by oracle could be beneficial. But since we wanted to use the oracle as rare as possible, thats why we only labeled a subset of the instances inside the oracle set. (and discarding the disagreement set and the remaining instances in the oracle set)

Reviewer comment (Experiments):
Why in Algorithm 1, line 5, if no models have sufficiently high accuracy score, the algorithm should stop and get classifier from labeled dataset by TabPFN? I think collecting true labels with AL should still be beneficial to improve the classifier.

Author response:
Q2: During our experiments, we found out, that in those cases, where none of the algorithms has a sufficiently high accuracy score (i.e. 
could learn the data distribution), this means, that the data is just really hard to learn/noisy etc. And even having more labeled data would not help in improving the accuracy of the classifiers.