Paper: DEEP UNSUPERVISED DOMAIN ADAPTATION FOR TIME SERIES CLASSIFICATION: A BENCHMARK (ICLR-2024)

Reviewer comment (Presentation):
While Figure 1 compares model performance within hyperopt methods (a, b, c), it does not provide an overall comparison of all models with all hyperopt methods. As such, it is difficult to determine which complete approach (model + tuning approach) actually has the best performance. An additional critical difference diagram comparing the (5?) top methods for each tuning approach would make this much clearer.

Author response:
We acknowledge the reviewer's suggestion to incorporate an *additional critical difference diagram* for a comprehensive comparison of all experiments, presenting an overall perspective on performance. This point has been addressed in the Appendix A.2.1. The average accuracy rank across all algorithms and model selection methods delineates three clusters: \
i) the first cluster comprises most of the algorithms trained using the target-only method, \
ii) a second cluster includes OTDA and VRADA, identified as the less performing algorithms across the three hyperparameter selection methods, and \
iii) a third cluster encompasses the remaining algorithms, fairly/visually distributed between the source-only and IWCV methods. Due to space constraints, we kept this further analysis in the Appendix only.

Reviewer comment (Presentation):
While the figures provide some information, and the Appendix gives a full set of results for each dataset, it remains difficult to assess the margins between the approaches. A summary table of average accuracy across all datasets for each experimental configuration would be beneficial in conveying this information.

Author response:
We have included a comprehensive table, where each column corresponding to a specific method (Source Risk, IWCV, Target Risk), in the Appendix A.1.2. The table aims to provide *a summary by averaging accuracy scores across datasets for each classifier*, accompanied by reporting of standard deviations.

Reviewer comment (Experiments):
Further variations of Figure 4 for other models/datasets would be useful to see if the revealed trend is consistent.

Author response:
We have added *variations of Figure 4* with different datasets and UDA algorithms in Appendix A.2.5, as well as an analysis of these graphs. They confirm a certain correlation of the source risk and IWCV with the target risk, but show a plateau for the source risk, and some different optimum for IWCV, thus explaining the gap in performance compare to target risk.

Reviewer comment (Presentation):
I think the violin plots in Figure 7 are a strong way of communicating the results, and potentially should be moved to the main body if possible. As mentioned above, combining the some selection of the top methods for each tuning approach into a single plot would further aid comparison.

Author response:
We agree with the reviewer's comment regarding the effectiveness of the plots in Figure 8 for conveying results. 
    Due to space limitations, these additional plots have been placed in the appendix alongside all the extra graphs.

Reviewer comment (Reproducibility):
I believe this work has the most potential if the evaluation is released to allow further development of methods. I appreciate the source code is planned to be released upon acceptance, but potentially taking this a step further and allowing for easy reproducibility/extensibility would improve the impact of the work and help progress the research area.

Author response:
We completely agree with this point and we would like to answer with the following elements. First, the results are part of the git repository in a CSV format, and will be available when we open-source it for checking reproducibility and going further in the analysis. Furthermore,  the implementation of our benchmark relies on existing tools to strengthen its *reliability, scalability and modularity* and that we believe will help other research to easily add datasets and/or algorithms. These tools include Hydra for the configuration of the pipelines and parameters, HuggingFace for the datasets and evaluation of the models, PyTorch with HuggingFace, scikit-learn and POT for training the algorithms. As a proof of its modularity, we can mention that it only took us a few hours to add an additional algorithm (SASA) to the benchmark and run the experiment with it during the current rebuttal phase.

Reviewer comment (Evaluation):
To what extent is there dataset imbalance in the datasets? Additional results, for example using balanced accuracy, may be warranted if dataset imbalance is high. At the very least, a discussion on any dataset imbalances would be helpful. I appreciate F1 score results are given in the appendix.

Author response:
We appreciate your highlighting of this analytical aspect. To address this, we have included a new section in the Appendix A.1.3 specifically dedicated to emphasizing the *dataset imbalances*. In this section, we introduced a scoring metric, denoted as I, which quantifies the degree of imbalance, where a score of 0 indicates a highly balanced dataset, and a score of 1 signifies high imbalance. The results are presented in a table format, summarizing the number of classes, samples, and I scores for both the source (train, test) and target (train, test) datasets. It is important to note that only three datasets exhibit significant imbalance, posing additional challenges to the classification task. Note that two out of these three datasets are part of the original established set of datasets for UDA TSC.

Reviewer comment (Novelty):
The major concern of the work is on the technical novelty. All the datasets and baselines (including hyper-parameter tuning methods) are from existing literatures, and there is no novel technical contribution proposed.

Author response:
We agree with the reviewer that the technical novelty is limited, but we think this is expected from any benchmark study, as the goal is to assess the quality of existing algorithms on data which should be available openly to the community. Still, we have not limited ourselves to existing TSC UDA dataset and we adapt 7 diverse time series datasets to UDA settings which have never been used before for UDA. Moreover, we have not limited ourselves to simply comparing existing methods, as this could be expected from a benchmark, but we proposed the usage of the Inception backbone that, not only achieves the best performance, but also allows a fair comparison between SOTA methods having different backbones.

Reviewer comment (Novelty):
For UDA TSC, some important related works are missing, for instance (to name a few), unsupervised video domain adaptation [ref1], transfer gaussian process [ref2], and time-series domain adaptation [ref3]. The authors may need to present a more comprehensive related work section to discuss more related works.

Author response:
We thank the reviewer for pointing out these references. We believe the work on  UDA for video data [ref1] is a particular field and should not be included here, as it is a combination of image (spatial) and time series (temporal) problems and therefore goes beyond the scope of our study. Nevertheless we have added the latter paper as part of our references of UDA for computer vision tasks. 
    Furthermore regarding the Gaussian Process Regression [ref2], although the paper tackles time series and tabular data, it focuses on regression (data extrapolation) rather than classification tasks and does not tackle the unsupervised DA setting. Nevertheless we have now mentioned regression DA task in the introduction section.
    Finally, regarding the SASA model [ref3], we strongly agree with the reviewer that this reference is relevant to our benchmark. We therefore have added the Section A.4 in the appendix where we present the results of SASA (with both LSTM and Inception backbones) over all 54 datasets (accounting to almost 1296 hours of gpu time during the current rebuttal phase). However, we decided to keep it outside the main benchmark as it is more suited for the task of detecting an event within a time series, and not classifying the complete input time series, since it is making the decision based on the last few time stamps, which probably explains its bad performance in our study. Comments were also added in the main paper to mention SASA.

Reviewer comment (Experiments):
More analyses on the new datasets are expected, for instance, the domain discrepancy analyses (both marginal and conditional can be involved) on different domain pairs. From the experiments results of the source only baseline, the domain discrepancy differs considerably among different domain pairs, see, Table 5 domain 0 and domain 3, Table 9 domain 9 and domain 18.

Author response:
We strongly acknowledge the significance of quantifying the shift between various source and domain pairs, as it holds the potential to offer valuable insights into when domain adaptation proves beneficial and which algorithm is most effective. However, given that quantifying the shift in DA is already difficult, adding to that the complex dynamics of a time series data, we believe that this domain shift quantification is an ongoing area of research that we plan to address in our future work. The complexities stem from the unique nature of time series data, where shifts can manifest in time, features, or a combination of both. Measuring domain discrepancy, e.g. through metrics like Maximum Mean Discrepancy or Wasserstein distance, on different domain pairs may not provide sufficient information about time series shift. As an illustration of this difficulty, OTDA leverages this Wasserstein distance, which quantifies the shift, but does not work well for time series data in our benchmark.
    We found that using the performance of Inception (i.e. with no adaptation) was a quick and easy proxy to compute the shift, which can be seen in Figure 3, although we cannot assess how good a proxy it is.
    We also added a paragraph in Appendix A.1.1 to explain some of the discrepancies for specific cases (e.g. when we know that one domain is more noisy than others).

Reviewer comment (Evaluation):
More analyses on the comparison results are also expected. For instance, analyzing why some baselines achieve positive transfer on some tasks but negative transfer on others, e.g., see OTDA/VRADA in table 9. It would be more interesting to see constructive insights or conclusions that can benefit the community.

Author response:
We believe that there is already some interesting insight for the community even if they are indeed quite general. We cannot really enter too much into details of combination of algorithm/dataset.\
  i. The limited impact of IWCV compared to Source Risk, which is now additionally supported by the added Figure 6 that mix the methods.\
  ii. The limited impact of backbone, additionally supported by the comparison for different dataset theme and method.\
  iii. The fair comparison between algorithms with the same common Inception backbone, and the good performance of InceptionRain.\
    We have also added some more analysis in the Appendix following the other comments.

Reviewer comment (Experiments):
The discussion on the effect of model backbone in the paper is limited primarily to the Inception model. A broader examination involving diverse backbone models is crucial to substantiate the claim that "backbones do not have a significant impact".

Author response:
Note that we compare Inception with 3 different architectures that are proposed in the literature (CoDATS, Raincoat and CoTMix). Because it is hard to compare the different backbones of different methods directly, we chose the Inception backbone as reference. Following your remark, we added a section (Appendix 2.6) and three plots in Figure 11, that compare the 2 additional backbones individually with Inception. The same conclusion holds for each of them. Furthermore With this result, we believe that there is enough argument for the limited impact of backbone in time series UDA. Additionally, a proper comparison of another classical backbone would lead to around 5 months of GPU’s time.

Reviewer comment (Evaluation):
More discussions on different types of unsupervised domain adaptation methods would be beneficial. Specifically, it would be informative to explore under what specific conditions certain domain adaptation approaches may outperform others.

Author response:
We agree with the reviewer about the necessity of a deeper analysis. However, we believe part of this analysis relies on an estimation of the shift between the datasets to understand how much shift in the data each method can support. The estimation of the shift being still an open issue and difficult to tackle, the best we proposed here is an analysis about the difficulty of a dataset, which we can assess with accuracy scores as well as standard deviation from pairs of source-target.

Reviewer comment (Experiments):
Additional discussions regarding the choice of model backbones is helpful too. For example, I am curious if Inception is the best model backbone over all domains, or we need different model backbones for different time-series domains and data characteristics.

Author response:
Firstly, the InceptionTime algorithm has been found to be the best deep learning algorithm in a recent benchmark for Time Series Classification [ref1], which is the reason why we chose this backbone for a common comparison of the UDA approaches.
    Following your comment, we added in Section A.2.6 the Figure 12 to provide a better comparison about the differences between various types of dataset (Machinery, Medical, Motion and Remote sensing) and add comments, especially on a potential bias toward Motion dataset for existing methods due to existing and already established UDA TSC datasets. The latter observation justifies the need to expand the list of UDA TSC datasets and improve generalization to different TSC domains, which is one of the main contributions of this paper.

Reviewer comment (Presentation):
Figure 2 (b) and 5 should be merged since they lead to similar findings.

Author response:
Indeed, both figures have similar finding and can be merged but we still think that focusing specifically on Raincoat, which is the best method, is still important to support our claim: it is the Raincoat’s UDA method behind the better performance rather than its backbone.