Paper: Active Domain Adaptation Of Medical Images Using Feature Disentanglement (ICLR-2024)

Reviewer comment (Writing):
The paper seems written in a rush and it’s difficult to read at times.

Author response:
Our Response: We regret that our writing was not clear to the Reviewer.We will proofread the manuscript again and make sure to improve the readability of the final version.

Reviewer comment (Theory):
Using features of a pre trained classifier for L_base does not need to identify the correct task-specific features. Nothing prohibits the model from a) extracting task-independent features and assigning them zero weights in the final classification layer and b) collapsing to the target variable already in the hidden layer (or the logit distribution).

Author response:
Our Response: We disagree with the reviewer.It is known in domain generalization that when finetuning models on a small amount of data from a different domain the output features tend to collapse in diversity.See section 2 of: Wuyang Chen et. al, "CONTRASTIVE SYN-TO-REAL GENERALIZATION".Using L_base helps us prevent such feature collapse.While using M_base we assume that the classifier is the optimal one. Thus the resulting set of features from M_base’s penultimate layer (not the logit distribution) are expected to be the optimal features.  

While reviewer's point (b) can possibly occur, the use of dropout after the feature layer mitigates such issues. L_base is not the only criterion for getting task specific and domain specific features. Combined with the other loss functions L_1,L_2,L_3, we ensure that the final z_task is not prone to inaccuracies.

Reviewer comment (Theory):
The density estimation approach is incorrect. You cannot reason about probability densities by comparing cosine similarities of **arbitrarily distributed** vectors (e.g., imagine the case where several dimensions are strongly correlated).

Author response:
Our Response: We believe there is a misunderstanding by the reviewer. We are not using density estimation/probability values in any of our proposed loss functions. The only instance is use of predictive entropy for Q_Unc in Eqn. 7, which is a standard technique. 
Can the reviewer be more specific to which equation they are referring to?

Reviewer comment (Theory):
In fact most of your objectives suffer from that same problem - the similarities can easily be inflated if multiple dimensions are not independent. Consider using a more probabilistically sound approach, e.g., by incorporating models such as normalizing flows.

Author response:
Our Response: We like to clarify that  we use an autoencoder, as described in the text and different loss functions- e.g. reconstruction loss is used for autoencoder training. An autoencoder is a better option than normalizing flows for feature disentanglement since we do not need density distributions.  (A note is that in Fig 1a of the paper I show VAE encoder. In reality it is simple autoencoder since I use only th reconsrucion loss. Perhaps the reviewer is confused because of that?)

Reviewer comment (Experiments):
How did you select the hyperparameters (e.g., the percentiles for similarity cutoffs)? Actually looks like you just fitted the hyperparameters to the final results? Which is incorrect? How did you select hyperparameters for the baselines?

Author response:
Our response: The percentile based values for eta_1 and eta_2 were chosen based on steps similar to what one would choose for other parameters. Insead of having a fixed value for these parameters we observe that a good threshold is highly dependent on the actual similarity values. Hence we experimented with different percentile values as threshold varying them from 5% to 60% for eta_1, and 20% to 90% for eta_2, across all datasets. We found that eta_1=30% and eta_2=75% gives the best results across all datasets.  

We have also selected the hyperparameters after careful consideration. We confirm that as per standard practices in machine learning the parameter values were set using a separate validation set which was not part of training or test datasets. We agree that this is a high number of parameters that need to be optimized due to the multiple loss terms in our method. Hence we have taken extra care to ensure that the optimization steps were done with utmost care and we have cross checked them multiple times. Reproducible results were ensured with fixed seed values. The high number of hyperparameters leads to much improved results.

Reviewer comment (Experiments):
The proposed method is not simple in terms of optimizations: 5 different losses are used, each with its own hyperparemeter. Also the informative score introduces many hyperparamers. Choosing many hyperparameters is not trivial, and authors report some arbitrary value for them. How were they chosen? It is not completely clear.

Author response:
We would like to clarify that we select the hyperparameters after careful consideration. We confirm that as per standard practices in machine learning the parameter values were set using a separate validation set which was not part of training or test datasets. We agree that this is a high number of parameters that need to be optimized due to the multiple loss terms in our method. Hence we have taken extra care to ensure that the optimization steps were done with utmost care and we have cross checked them multiple times. Reproducible results were ensured with fixed seed values. The high number of hyperparameters leads to much improved results.

Reviewer comment (Presentation):
The tables are hard to read, best results are not highlighted. I also suggest author report a standard devation, especially in the higher p-value cases.

Author response:
Due to space constraints we did not report standard deviations. However we will include them in the final manuscript. The standard deviations among close performing methods is such that there is negligible overlap in their values, especially methods with higher p-values

Reviewer comment (Experiments):
In your experimental protocol (Sec. 4.1) you select at each step 10% of the size of the training data from the target set. This means that the added samples will be in minority in the training set. Have you also tried reweighting them in the classification loss? Can this help in reducing the number of samples required for better AUC?

Author response:
We believe the reviewer is alluding to importance sampling. While updating the classifier we use the labels of the newly added samples to finetune the classifier. There is no explicit reweighting, although that can be explored in future work.

Reviewer comment (Experiments):
The two medical image datasets are quite large. In the experimental setting, at least 10% of the unlabeled data samples in the target domain were selected for labeling and added to the source domain's labeled data to update the classifier. While 10% still represents a significant number, considering the constraints in the research setting—such as limited and expensive expertise in the medical field—it would be more valuable to evaluate the effectiveness of the proposed method with much fewer labeled samples from the target domain, for example, 0.1% or 1%. Unfortunately, this aspect is missing in the paper.

Author response:
We thank the Reviewer for raising this point. Although we show results for every 10% increment to the training data, it is done over batches. We use a batch size of 96 in determining the informative samples. On an average in every batch 2-3 samples  of each class are chosen  to add to the training set. This represents 0.1% or less for each class in the NIH dataset. The increase of performance is very low over batches and becomes prominent after every 10% increment of dataset size. In the final manuscript we will provide results for 0.1% and 1% labeled target samples.

Reviewer comment (Evaluation):
The discussion section or the presentation of the paper could be improved. For instance, in the ablation study, each component of the loss function and the metrics for identifying informative data samples was thoroughly examined, and their contributions were reported in the tables. However, there is a lack of in-depth discussion and no clear claims have been made regarding which component contributes more. It is not readily apparent to readers which component has the most significant impact.

Author response:
We agree that our discussions could be improved and we should emphasize the most important components of our method. We have addressed this issue in the revised manuscript and  will elaborate further in the final manuscript.

Reviewer comment (Writing):
In the algorithm description, the authors initially state that feature disentangling was performed jointly using data samples from both the source and target domains. However, they later mention that the process was performed using data solely from the source domain. Which statement is correct?

Author response:
We apologize for the confusion. The feature disentanglement network was trained on source and target data. The confusion possibly arises from the text in Sec 4.1. We would like to clarify that in Sec 4.1 we refer to those target domain  samples that were not part of the training process.