Paper: DUDE: Deep Unsupervised Domain adaptation using variable nEighbors for physiological time series analysis (ICLR-2024)

Reviewer comment (Writing):
Some parts of the methods need more clarification or justification. For example, what are the encoders? Why is the Domain Shift Uncertainty (DSU) layer necessary?

Author response:
Our answer: The encoders correspond to the original algorithms developed for a given task and published in [1,2,3]. Each encoder is briefly described in the supplementary section “B.1 experiments and background” because of the page limit for the core paper. The following sentence in the manuscript has been edited accordingly:

“More details on the task, its clinical importance, the DL encoder used and the datasets for each experiment are provided in the supplement B1.”

[1] Levy, Jeremy, et al. Nature Communications 14.1 (2023): 4881.
[2] Kotzen, Kevin, et al. IEEE Journal of Biomedical and Health Informatics 27.2 (2022): 924-932.
[3] Ben-Moshe, et al. 2022 Computing in Cardiology (CinC). Vol. 498. IEEE, 2022.

The motivation for including a DSU layer is elaborated upon in question 4.

Reviewer comment (Novelty):
Technical contribution is limited. The model extends existing NNCLR algorithm by incorporating more nearest neighbors in the support set.

Author response:
Our answer: In this paper, we focus on the scarcely explored, yet critically important area of UDA applied to continuous physiological time series. These time series form the backbone of daily clinical practice, playing a pivotal role in the diagnosis and monitoring of patients. As researchers in this area we are confronted with a persistent, unresolved challenge in developing DL support tools for continuous physiological time series analysis: namely the effective deployment of our algorithms in real-world settings where the distribution of the data differs from the source domain. Our findings strongly support the importance of UDA in analyzing continuous physiological time series. Thus, our contribution not only enhances the algorithmic performance of NNCLR but also provides strong evidence on the potential of using UDA to bridge a crucial gap in medicine, potentially transforming patient care through more precise and adaptable diagnostic tools.

Reviewer comment (Presentation):
Citation format is wrong, which reduces readability of the paper.

Author response:
Our answer: We thank the reviewer for his/her remark and have now fixed the citations style.

Reviewer comment (Reproducibility):
What are the encoders for each task? Why is the Domain Shift Uncertainty (DSU) layer necessary? Please clarify in Methods.

Author response:
Our answer: We experimented with and without the DSU layer, and found that it was improving the performance based on the validation set. We thus decided to include it as one of the components of our framework. The ablation study presented in Figure 3 clearly shows that the DSU layer has a contribution to the overall DUDE performance. The DSU layer introduces some “uncertainty” in the encoder. The introduction of uncertainty makes the model less prone to overfitting on the source domain by reducing its excessive confidence, thereby facilitating domain adaptation.

Reviewer comment (Experiments):
It would be interesting to see how model performance varies across $\delta$ (i.e., number of neighbors), e.g., does model performance saturate or degrade with larger number of neighbors?

Author response:
Our answer: We have explored this question for Experiment 1 and have added Table S3 where the results are reported. We ran simulations for 1, 2, 5 and 10 neighbors. The results indeed show that performance tends to increase while increasing the number from 1 to 5. However, it then decreases for 10 neighbors, i.e. performance degrades with too large a number of neighbors.

Reviewer comment (Experiments):
In ablation study (Figure S1), what’s the setup for DSU and $NNCLR_{\Delta}$? Please clarify.

Author response:
Our answer: The ablation study is now reported in the core manuscript as Figure 3. The experimental setup used is the exact same as for the main experiments conducted in the manuscript. To clarify that, we have updated the caption of Figure 3 to:

“Ablation study on the main components of DUDE. The training of the Baseline and DSU models include the source domain training patients. The training of the NNCLR_Delta and DUDE model includes the exact same set of patients from the source domain as well as a subset of each target domain patients (as unlabeled samples). See "Experimental settings" in section 4.2.”

Reviewer comment (Experiments):
Hidden in the supplement, the authors discuss that including a supervised loss prevents latent space collapse. Please provide an ablation study to support this claim.

Author response:
Our answer: A new simulation for Experiment 1 was added to Table S2. The lack of supervised loss led to very poor model performance on all target domains. Indeed, without supervised loss, there is the issue of space collapse due to the numerous positive instances in contrastive learning. This underscores the importance of employing a supervised loss in our analysis. We added to section B.3 the reference to Table S3:

Reviewer comment (Presentation):
Writing style: the citations are neither hyperlinked nor separated from the text.

Author response:
Our answer: We thank the reviewer for this remark and have now fixed the citations style. Citations are hyperlinked and we use the command \citep when appropriate.

Reviewer comment (Novelty):
The technical novelty of the work is quite limited. The two main parts of DUDE are either using existing techniques (DSU) or making marginal improvements on existing work NNCLR. Using threshold to adaptively select neighbors is not new, and it also brings a question on how to determine the threshold for different or new tasks. The paper claims that the best hyperparameter Δ found on the validation set was 0.95 for both experiments and thus shows the consistency for different experimental settings, which is not convincing. The authors may need to conduct a more comprehensive sensitivity analyses on this hyper-parameter or propose a valid hyper-parameter selection guideline for new tasks or unseen datasets.

Author response:
In this paper, we focus on the scarcely explored, yet critically important area of UDA applied to continuous physiological time series. These time series form the backbone of daily clinical practice, playing a pivotal role in the diagnosis and monitoring of patients. As researchers in this area we are confronted with a persistent, unresolved challenge in developing translational DL support tools for continuous physiological time series analysis, namely the effective deployment of our algorithms in real-world settings where the distribution of the data differs from the source domain. Our findings strongly support the importance of UDA in analyzing continuous physiological time series. Thus, our contribution not only enhances the algorithmic performance of NNCLR but also provides strong evidence on the potential of using UDA to bridge a crucial gap in real-world applications of deep learning, potentially transforming patient care through more precise and adaptable diagnostic tools.

With respect to the choice of the delta hyperparameter, we have further elaborated the methods section describing the hyper-parameter approach we used:

“The evaluation was conducted on the validation set, which consists of labeled samples exclusively from the source domain. Hyperparameter tuning was carried out using a Bayesian search with 100 iterations, meaning training the model on the train set, and validating it against the validation set. The results on the source and target domains test sets are presented for the best hyperparameter configuration found.”

In that respect we confirm that there is no information leakage in the training process of DUDE as all hyperparameters, including Δ, are learned through a search performed over the validation set.

Reviewer comment (Novelty):
The paper highlights that DUDE is proposed for physiological time series analyses, but from the technical view, the framework can be used for general time series UDA problems. It is unclear why the context of physiological time series is necessary.

Author response:
The reviewer is correct in stating that DUDE could be applied to a broad range of time series UDA problems. Given the clinical significance of continuous physiological time series and the existing gap in research on utilizing UDA for their analysis, we chose to concentrate our efforts in this area. However, we have now included a sentence in the discussion to acknowledge that the DUDE framework could indeed be valuable for analyzing other types of time series.

Reviewer comment (Experiments):
Based on point 2, more general time series UDA baselines should be compared, e.g. [ref1] and [ref2], to name a few.

Author response:
The reviewer is correct in noting that a number of additional benchmarks could be included. Unlike other subfields of DL where SOTA approaches are well-defined, this is not the case in the field of UDA. Therefore, we have identified and included one important benchmark for each mainstream UDA approach. We then focused on demonstrating the added value of NNCLR_Delta versus NNCLR within the DUDE framework. However, we agree that expanding this research to include more UDA benchmarks is desirable. We have therefore included the following statement in the discussion section:

“Future work should include benchmarking DUDE against additional UDA approaches such as those of Cai et al.(2021); Tonekaboni et al. (2021); Ragab et al. (2022); Eldele et al. (2023).”

Reviewer comment (Novelty):
For DSU layer, what’s the difference between DSU and instance normalization used in [ref3].

Author response:
AdaIN (ref3) aligns the mean and variance of the feature from one domain to another, for the purpose of style transfer. In DSU, a non-Bayesian approach has been taken, and the goal is to insert uncertainty into the model.

Reviewer comment (Experiments):
Why different data augmentation used for different tasks? It is necessary to have an ablation study on different data augmentation techniques.

Author response:
Specific data augmentation techniques are particularly effective for certain physiological time series, owing to the unique characteristics of the data, such as cyclic patterns and types of noise. The specific augmentation techniques chosen for each task were informed by previous experiments demonstrating the effectiveness of certain techniques for specific time series and tasks, such as the random switch for SpO2 (Levy et al 2022). This choice was also guided by discrete experiments with common data augmentation techniques (Saeed et al. 2019) and their performance evaluation on the validation set thus preventing any information leakage. The careful selection of these augmentations is crucial, as an even more strategic choice could potentially enhance DUDE's performance further and a comprehensive study benchmarking the effects of a subset of classic augmentation techniques would be beneficial. A paragraph has been added to the discussion accordingly.

Reviewer comment (Experiments):
The paper lacks ablation studies on the two parts (DSU and NNCLR) of DUDE.

Author response:
We acknowledge the importance of an ablation study and agree that it should have been part of the main manuscript rather than the supplementary material. Accordingly, we have now included the ablation study in the main manuscript - Figure 3. This study highlights the added value of each critical component within the DUDE framework. Specifically, it shows the value of the NNCLR approach for the contrastive loss, the capability to vary the number of neighbors with NNCLR_Delta, and the incorporation of DSU layers into the architecture.