Paper: LogoRA: Local-Global Representation Alignment for Robust Time Series Classification (ICLR-2024)

Reviewer comment (Theory):
The author claims that cross-attention in local-global fusion model is used to integrate local and global representations, which should be explained in detail. The cross-attention operation computes the similarity between query and key, and selects values based on the similarity score. It is confusing to compute the similarity of global and local feature, and use this score to select the local feature. What is the explanation behind it?

Author response:
**A2:** Firstly, the cross-attention operation is able to extract useful robust context information between global and local features. Thanks to the modeling capability of Transformers for long dependencies, we extract the global features of the data and force the global features to be time-step invariant through DTW. The local features extracted by the convolutional network remain in temporal order, which can be matched with the patch-wise global feature. Moreover, a global feature summarizes the content of a sequence, often leading to a compact representation. Local features, on the other hand, comprise pattern information about specific regions, which are especially useful for classification. Generally speaking, global features are better at recall, while local features are better at precision [1]. So, we assume that local features with more relevant information to global features are more important. As shown in the heatmap in Figure 6, the higher the similarity calculated between global and local features, the more useful the corresponding part is for classification. Therefore, based on similarity score, further selecting local features can yield features that contain more useful contextual information. Besides, with different kernel sizes, we obtain several local features of different scale. So, using similarity score to select the multi-scale local feature helps to extract the effective features on different positions more comprehensively.  Moreover, taking the HHAR dataset as an example, we have performed additional experiments by changing the fusion method from cross attention to addition and concatenation. The results are shown in the table below. From the experimental results, cross attention also proves to be the much more effective than others. We have updated the manuscript and provided more detailed explanations of the cross-attention operation in Appendix G.1. We hope it could answer your doubt.

| source → target | addition | concatenation | cross attention |
| --- | --- | --- | --- |
| 0 → 2 | 0.6849 | 0.7857 | 0.8235 |
| 1 → 6 | 0.8884 | 0.9163 | 0.9163 |
| 2 → 4 | 0.4462 | 0.3386 | 0.9363 |
| 4 → 0 | 0.262 | 0.2576 | 0.3886 |
| 4 → 1 | 0.6679 | 0.5672 | 0.9627 |
| 5 → 1 | 0.8582 | 0.8694 | 0.9851 |
| 7 → 1 | 0.8918 | 0.8545 | 0.9478 |
| 7 → 5 | 0.4402 | 0.5637 | 0.8147 |
| 8 → 3 | 0.8996 | 0.9738 | 0.9738 |
| 8 → 4 | 0.6175 | 0.7331 | 0.9681 |
| Avg | 0.6656 | 0.6860 | 0.8717 |

Reviewer comment (Experiments):
The paper only uses DTW for global features, how about the local features?

Author response:
**A3:**  Thanks to the patching operation, the obtained global feature is shorter than the local feature obtained by convolution. Because DTW involves dynamic programming algorithms, aligning the global feature with DTW can save more time. Moreover, we have also attempted to align the local feature with DTW, and the experimental results indicate that there is little difference between the two methods. Therefore, we chose the method with higher time efficiency.

Reviewer comment (Evaluation):
More limitations of the proposed method need to be discussed. For instance, in Table 2 of the ablation experiments, there is a noticeable performance degradation, particularly in the 4th and 5th rows compared to the 3rd row. This can be attributed to differences in performance across the first three scenarios. Can specific examples be used to illustrate this?

Author response:
**A2:** Thank you for your constructive suggestion. We have updated the manuscript and presented the detailed explanation with clear visualizations in Appendix F.7. We also shared some limitations in Appendix H. We hope it could address your concern.

Reviewer comment (Theory):
The training objective specified in Equation (5) involves five distinct loss functions, rendering it challenging to optimize the overall loss coherently during training. In essence, the presence of multiple training losses poses a challenge in comprehending the fundamental contributions of this paper, given the complexity associated with optimizing such a diverse set of loss functions.

Author response:
**A3:** **Firstly,** from the results of ablation experiments in Table 2, it can be observed that without these loss functions, the model's performance is mediocre. However, after optimizing with all loss functions, our method achieves a significant improvement. **Secondly,** the proposed loss functions complement each other. They can be directly optimized, and the entire optimization process is not complex, allowing for quick convergence. **In conclusion,** we believe this precisely demonstrates our contribution. The proposed loss functions can operate at different granularities, and ultimately contribute to the outstanding performance.

Reviewer comment (Evaluation):
The explanation for why the fusion of global and local features is effective is not clear. The interpretation of the heatmaps in Figure 6 is unclear as well. (1) How can different positions in the heatmaps be correlated with sequences? (2) Which weight activations in the heatmaps are meaningful? (3) How do different scales of heatmaps demonstrate their effectiveness?

Author response:
**A4:** Thank you for your feedback. We have updated the manuscript and provided detailed explanations of the cross-attention operation in Appendix G.1. We hope it could answer your doubt.

Reviewer comment (Theory):
Why is the method of fusing global and local features of time series using an attention mechanism effective?

Author response:
**A5**: **(1)** Because our local encoder uses a shallow convolutional network with a limited receptive field, the obtained features maintain the same temporal order as the original sequence, preserving a certain correspondence. **(2)** The weights obtained through cross-attention represent the weights of global features on the selection of local features. The higher the similarity calculated between global and local features, the more useful the corresponding part is for classification. Therefore, based on this score, further selecting local features can yield features that contain more useful contextual information. **(3)** With different kernel sizes, we obtain several local features of different scales. As shown in Figure 6, the weights assigned to different scales of local features are not identical. So, using similarity score to select the multi-scale local feature helps to extract the effective features on different positions more comprehensively.

Reviewer comment (Evaluation):
What are the merits and drawbacks of the approach introduced in this paper, in comparison to existing methods, concerning both runtime efficiency and the convergence behaviour of the model?

Author response:
**A6:** We present the comparison of computational complexity with other methods in Appendix F.1. Besides, we have updated the manuscript and analyzed limitations in Appendix H. Below is a portion of the training logs, hoping to alleviate your concerns about convergence.

```markdown
Epoch: [12][ 4/10]	Time  1.135	Data  0.007	Loss Cls 3.0972e+00	Macro F1   0.23	Loss Domain 7.9497e-01 	Loss dtw 2.7484e-01	Loss global 3.7793e-01	Loss center 3.4145e-01	Loss TOTAL 5.0477e+00
Epoch: [24][ 8/10]	Time  1.244	Data  0.008	Loss Cls 2.0759e+00	Macro F1   0.91	Loss Domain 7.2277e-01 	Loss dtw 2.9608e-01	Loss global 1.6139e-01	Loss center 1.0612e+00	Loss TOTAL 3.7608e+00
Epoch: [37][ 4/10]	Time  1.335	Data  0.019	Loss Cls 1.8064e+00	Macro F1   1.00	Loss Domain 7.4017e-01 	Loss dtw 2.8416e-01	Loss global 9.3070e-03	Loss center 1.1092e+00	Loss TOTAL 3.5444e+00
Epoch: [49][ 8/10]	Time  1.342	Data  0.005	Loss Cls 2.0430e+00	Macro F1   0.99	Loss Domain 7.2754e-01 	Loss dtw 2.1802e-01	Loss global 9.5955e-02	Loss center 8.5806e-01	Loss TOTAL 3.7131e+00 
Epoch: [62][ 4/10]	Time  1.017	Data  0.005	Loss Cls 1.7018e+00	Macro F1   1.00	Loss Domain 7.2700e-01 	Loss dtw 2.5508e-01	Loss global 1.3439e-03	Loss center 1.3648e+00	Loss TOTAL 3.4205e+00
Epoch: [74][ 8/10]	Time  1.188	Data  0.045	Loss Cls 1.5397e+00	Macro F1   1.00	Loss Domain 6.9720e-01 	Loss dtw 1.9302e-01	Loss global 5.3767e-03	Loss center 7.5330e-01	Loss TOTAL 3.1138e+00
Epoch: [87][ 4/10]	Time  1.120	Data  0.009	Loss Cls 1.6324e+00	Macro F1   0.99	Loss Domain 7.2536e-01 	Loss dtw 1.5999e-01	Loss global 3.7332e-02	Loss center 5.3994e-01	Loss TOTAL 3.2358e+00
Epoch: [99][ 8/10]	Time  1.257	Data  0.007	Loss Cls 1.5311e+00	Macro F1   1.00	Loss Domain 6.0868e-01 	Loss dtw 1.7000e-01	Loss global 5.6056e-03	Loss center 5.3012e-01	Loss TOTAL 2.8893e+00
Epoch: [112][ 4/10]	Time  1.388	Data  0.007	Loss Cls 1.5883e+00	Macro F1   0.99	Loss Domain 6.7380e-01 	Loss dtw 1.2501e-01	Loss global 2.1499e-02	Loss center 5.4366e-01	Loss TOTAL 3.0635e+00
Epoch: [124][ 8/10]	Time  1.276	Data  0.007	Loss Cls 1.7020e+00	Macro F1   0.99	Loss Domain 5.4085e-01 	Loss dtw 1.8139e-01	Loss global 6.4419e-02	Loss center 4.2474e-01	Loss TOTAL 2.9491e+00
```

Reviewer comment (Reproducibility):
I am interested in the implementation, as reproducibility is a key metric in ML publications nowadays. I wish authors can provide a code repo (hopefully using notebook so I can see the results without re-running everything). If authors can show the reproducibility during rebuttal, I would like to increase my rating.

Author response:
**A1:** The code repository is now publicly accessible via the following [anonymous link](https://anonymous.4open.science/r/LogoRA_rebuttal-DBE4/). Additionally, we have prepared a Jupyter notebook to facilitate the verification of the reproducibility of our experiments. The notebook is included in the code repository and can be accessed directly through the provided link. This is not the final official repo because we want to maintain anonymity. Once the anonymous review ends we will update it with a final official link.

Reviewer comment (Experiments):
The cross-attention is only validated by qualitative visualizations but not in ablation study. Although the heatmap looks good to me, it is interesting to know how much gain does cross attention bring vs. other fusion methods (e.g., addition or concatenation).

Author response:
**A2:** We have performed additional experiments by changing the fusion method from cross attention to addition and concatenation. The results are shown in the table below. Using cross attention is much more effective in fusing global features and local features than directly adding or concatenating them. As shown in the heatmap in Figure 6, by using the weights calculated between the global features, which have already learned time-step invariant information, and the local features, it is possible to focus on the parts of local features that are more useful for classification and learn more robust context information.

| source → target | addition | concatenation | cross attention |
| --- | --- | --- | --- |
| 0 → 2 | 0.6849 | 0.7857 | 0.8235 |
| 1 → 6 | 0.8884 | 0.9163 | 0.9163 |
| 2 → 4 | 0.4462 | 0.3386 | 0.9363 |
| 4 → 0 | 0.262 | 0.2576 | 0.3886 |
| 4 → 1 | 0.6679 | 0.5672 | 0.9627 |
| 5 → 1 | 0.8582 | 0.8694 | 0.9851 |
| 7 → 1 | 0.8918 | 0.8545 | 0.9478 |
| 7 → 5 | 0.4402 | 0.5637 | 0.8147 |
| 8 → 3 | 0.8996 | 0.9738 | 0.9738 |
| 8 → 4 | 0.6175 | 0.7331 | 0.9681 |
| Avg | 0.6656 | 0.6860 | 0.8717 |

Reviewer comment (Experiments):
Usually time series data are coupled with static data (tabular, images or texts), for example, MIMIC dataset. How to expand the current framework to deal with multi-modal data? You can share your thoughts as limitation/future work. If you have tried, you could share your exploration in the appendix too (even if it failed).

Author response:
**A3:** We thank the reviewer for useful suggestions. We presented several analysis on failure cases in Appendix F.4. And we have updated the manuscript and shared some new attempts in Appendix F.6, along with some ideas on how to extend our framework to multi-modal data in Appendix H. We hope these insights could be valuable to you. As for the MIMIC dataset you mentioned, unfortunately, our application for access is not replied by the official authorities. Consequently, we cannot to conduct experiments on this dataset. However, we express our willingness to explore its effects in the future if our application is approved. You are also welcome to try our publicly available code on your own. We hope it is helpful for you.

Reviewer comment (Novelty):
The technical details seem to be a bit lack of innovation. The author can improve the paper by further explain how global and local information work together to improve the quality of the classification.

Author response:
We thank the reviewer for useful suggestions. Firstly, the cross-attention operation is able to extract useful robust context information between global and local features. Thanks to the modeling capability of Transformers for long dependencies, we extract the global features of the data and force the global features to be time-step invariant through DTW. The local features extracted by the convolutional network remain in temporal order, which can be matched with the patch-wise global feature. As shown in the heatmap in Figure 6, the higher the similarity calculated between global and local features, the more useful the corresponding part is for classification. Therefore, based on similarity score, further selecting local features can yield features that contain more useful contextual information. Besides, with different kernel sizes, we obtain several local features of different scale. So, using similarity score to select the multi-scale local feature helps to extract the effective features on different positions more comprehensively. Besides, we have performed additional experiments by changing the fusion method from cross attention to addition and concatenation. The results are shown in the table below. It can be observed that using cross attention is much more effective than other fusion methods. We have also updated the manuscript and provided detailed explanations of the cross-attention operation in Appendix G.1. 

| source → target | addition | concatenation | cross attention |
| --- | --- | --- | --- |
| 0 → 2 | 0.6849 | 0.7857 | 0.8235 |
| 1 → 6 | 0.8884 | 0.9163 | 0.9163 |
| 2 → 4 | 0.4462 | 0.3386 | 0.9363 |
| 4 → 0 | 0.262 | 0.2576 | 0.3886 |
| 4 → 1 | 0.6679 | 0.5672 | 0.9627 |
| 5 → 1 | 0.8582 | 0.8694 | 0.9851 |
| 7 → 1 | 0.8918 | 0.8545 | 0.9478 |
| 7 → 5 | 0.4402 | 0.5637 | 0.8147 |
| 8 → 3 | 0.8996 | 0.9738 | 0.9738 |
| 8 → 4 | 0.6175 | 0.7331 | 0.9681 |
| Avg | 0.6656 | 0.6860 | 0.8717 |

Reviewer comment (Experiments):
Only four datasets are employed, it may be not sufficient enough to support the claim of the advantage of the proposed method.

Author response:
We acknowledge the importance of a diverse set of datasets for a comprehensive evaluation of our proposed method. Therefore, following recent work in time series domain adaptation (reference [1] and reference [2]), we employed a similar amount of datasets for a fair comparison. Certainly, we understand your concern, so we have expanded our experiments to include an additional dataset, CAP (detailed in reference [3]). Compared to the previous datasets, the time series in the CAP have longer lengths and more dimensions, making the dataset more challenging. Due to time constraints, we only test a portion of the baselines, which are reported in the Appendix F.6. The results presented in Table 9 further demonstrate that our method has significant improvements over previous works, achieving better results on more complex datasets. We hope this could address your concerns.

Reviewer comment (Evaluation):
The effectiveness of the algorithm proposed by the author compared to the baseline method varies significantly across four datasets, with an improvement of +12.52% on the HHAR dataset, but only +0.51% on the HAR dataset. Can the author explain the reason for this difference?

Author response:
**Firstly**, it should be noted that the performance on the HAR dataset has been saturated. For example, the current state-of-the-art RAINCOAT achieves 0.974 in terms of mean accuracy. So, there is a little room for improvement. Our work pushes that boundary without 2-stage training or any correction steps in the reference [2], achieving a 0.51% absolute improvement over RAINCOAT and 3.71% absolute improvement over CLUDA, which has been a relative significant improvement on the HAR dataset. **Secondly**, in most testing cases on the HAR dataset, our method achieves excellent results with mean accuracy of 1.000. Especially, we achieve a 3.62% improvement on the 23-13 domain pair. **Thirdly**, the domain gaps between different datasets often vary depending on various data collection environments and application backgrounds, and the time-step shifts are also different. Therefore, it is quite natural that the degree of improvements achieved by our method vary across different datasets.

Reviewer comment (Writing):
The author suggests that using both global and local information simultaneously can help improve algorithm quality. Can we use intuitive examples to explain this process?

Author response:
Simultaneously utilizing global and local information is a common and effective approach in various domains, such as computer vision (CV) as referenced in reference [4] and the time series domain as mentioned in reference [5]. Actually, we have presented an intuitive example as shown in Figure 1 in the section of Introduction. The example exhibits accelerometer data pieces of two actions, i.e., walking upstairs (upper) and walking downstairs (lower) from the HAR dataset. As shown in the lower figure in Figure1, compared to other regions, the local abrupt acceleration changes in the shaded area are more representative to characterize the class (walking downstairs), which underscores the significance of local features. Because the signal is relatively stable in other positions, it may be unable to capture this sudden signal change only using the global features obtained from the original data. This oversight could lead to a failure in recognizing this action. By contrast, if only using the local features in the shaded area in Figure 1, it is still not sufficient to distinguish the two sequences of walking downstairs and walking upstairs, as both sequences have similar local features. While it will be much easier to differentiate them through modeling the temporal dependencies between local features. Therefore, it is an optimal strategy to utilize global and local features simultaneously for improving the discriminative capability of time series models.