Paper: CLA-RA: COLLABORATIVE ACTIVE LEARNING AMIDST RELABELING AMBIGUITY (ICLR-2024)

Reviewer comment (Experiments):
The selection of the datasets is not very reasonable. The characteristics of all datasets are not clear. This is my main concern about the effectiveness of the experiment. Detailed traits of each dataset should be exhibited. The number of annotators is too small. This condition is really beneficial to the proposed method which drops the convincing of the experiment.

Author response:
**Query about the datasets.**

The datasets chosen here are a combination of datasets present in previous works in these domain (eg CEAL, ALIO, Proactive etc) which are simulated from standard UCI datasets. We have provided a summary of the datasets in table 1 due to lack of space. However we agree to more details about the individual datasets in the appendix.

Reviewer comment (Experiments):
How were the hyper-parameters, i.e. ratio of training data, annotation budget, exploration threshold, confidence threshold, and similarity threshold, selected in the experiment?

Author response:
**Q: How were the hyper-parameters, i.e. ratio of training data, annotation budget, exploration threshold, confidence threshold, and similarity threshold, selected in the experiment?**

Determining the correct set of hyper-parameter values is a challenging task here, since we assumed
no labelled validation set to be available. We used some heuristics to decide on the hyperpaameters. For
example, learning rate was decided based on looking at the convergence of the training loss,ratio of boot
to active learning samples were decided based on the how many examples are needed to have atleast
1 example for each class, confidence and similarity threshold were decided based on looking at the
distribution of the respective values. We experimented with various similarity thresholds ranging from 0.7 to 0.95 and we observed that for most of the datasets there was not much change in the performance of the classifier with varyingsimilarity threshold. This indicates that some datasets are well separated in the embedding space However we agree that these are very crude ways to determine the
hyperparameters, therefore we decided to fix the value of the hyperparameters for all the datasets to
show the generalizability of these values. However if the constraint of not having any validation dataset is
relaxed, one can tune the hyperparameters based on the validation dataset to possibly improve the
numbers even further.

Reviewer comment (Evaluation):
How about the time complexity of the proposed method compared with other methods? When the number of workers becomes greater, due to the training phase, the proposed method will

Author response:
**Q: How about the time complexity of the proposed method compared with other methods?**

Time complexity wrt the no. of annotators (N) is linear because operations in KB is linear in N and the
change in the annotator model is also in the output layer which is linear in N.

Reviewer comment (Experiments):
How to guarantee that the annotator model works well? If the training data is unbalanced or responses from annotators have great bias, the annotator model will fail. Also, too small a quantity of training datasets cannot overcome the random prediction brought by the random initialization of the neural network.

Author response:
**Q: How to guarantee that the annotator model works well? If the training data is unbalanced or responses from annotators have great bias, the annotator model will fail.**

We showcase the efficacy of our annotator model empirically in figure 2 where we show the accuracy of the labels obtained from the annotator model. Many of the datasets , especially the multi class datasets such as car, ecoli etc have unbalanced distribution of labels for which our model outperforms the baseline by a large margin. However as mentioned in the reply of review 1, in the situation where most of the time, majority is incorrect, our approach will not perform optimally, which remains a drawback of not only this approach but the other baselines as well.

 **too small a quantity of training datasets cannot overcome the random prediction brought by the random initialization of the neural network**

This is the reason of choosing to relabel an instance with some probability at each step. For instances on which the Annotator model is not very confident, those instances will be chosen during the relabelling step and the annotator model will be trained for those instance with more label information.

Reviewer comment (Novelty):
In the method proposed by the article, a weighted majority scheme is employed to compute the confidence among different annotators. Numerous extant methods exist to calculate confidence, such as utilizing the L1 norm of predictions. The question concerns the decision to employ the weighted majority algorithm—a relatively conventional approach—over more recent methodologies, and what the rationale was behind this choice.

Author response:
**Q: My question concerns the decision to employ the weighted majority algorithm—a relatively conventional  
approach—over more recent methodologies. What was the rationale behind this choice?**  

We used weighted majority as it was performing the best across all the datasets amongst few other  
techniques. However we do agree with the reviewer’s comment that there might be better techniques to  
infer annotator confidence.

Reviewer comment (Experiments):
There are thirteen datasets, but often the authors only show eight datasets when presenting their results.

Author response:
**There are thirteen datasets, but often the authors only show eight datasets when presenting their results.**

We decided to show eight datasets for the ablation study results to maintain the clarity of the charts and not make them too crowded.

Reviewer comment (Evaluation):
Why does the approach not work as well as CEAL for Mushroom and Vehicle?

Author response:
**Q: Why does the approach not work as well as CEAL for Mushroom and Vehicle?**

CEAL actually uses a small set of labeled data to initialise their algorithm. We went ahead with a much difficult setting of having no labeled data to bootstrap. Even without any labeled data, we outperform CEAL in most datasets.

Reviewer comment (Experiments):
Would this approach work in high-dimension domains, thus making the similarity function more of an issue?

Author response:
**Q: Would this approach work in high-dimension domains, thus making the similarity function more of an issue**

For higher dimensional cases such text and images, one can use pretrained models to obtain the
embeddings. For example for text, one might use models such as BERT to encode the text into an
embedding vector and for images, one can use CNNs to get the embeddings. These embeddings can
then be passed to the similarity function

Reviewer comment (Theory):
The Annotator Model makes a big assumption that the majority will be correct; can this work in cases where the majority isn't correct?

Author response:
**The Annotator Model makes a big assumption that the majority will be correct, can this work in cases where the majority isn't correct?**

The goal of having an annotator model which predicts annotator confidence taking in instance features as input was to help improve the labelling of an instance even when majority is not correct for that particular instance. Having said that, the way the annotator model is trained, if for most of instances, majority is incorrect, then the annotator model will be incorrectly trained and will not give optimal results. We agree that this is a drawback of the method, but all the existing methods which do not use any kind of prior information about the labellers (including having few labeled examples) also suffer from this drawback.