From: ICLR.cc/2023/Conference/Paper2590/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper2590/Authors
title: Response to Reviewer A747
comment: Thank you for your valuable comments. We thouroughly understand your comments in the first and the recent reviews.

We wanted to clarify that our FEAT encoder has high usability with its robust and effective performance. Even though each part of the model seems basic concept of deep and representation learning, we believe the interesting combination of these concepts is valuable. 

Thanks a lot again for your comments and hard work in reviewing our paper.

From: ICLR.cc/2023/Conference/Paper2590/Reviewer_A747
title: Overemphasized arguments
comment: I am afraid that the above argument is overemphasized. 

For multivariate time series, the feature-specific, temporal, interactive representations, etc., are well-known concepts in the literature, as the example papers shown in the first review. The appreciated aspect of this paper is the empirical study showing the effectiveness of materializing these concepts in the representation learning for different time series tasks. 

However, when it boils down to the technical implementation/design of the proposed method, Eq.(1) to (8) are mostly very basic in the context of deep and representation learning, despite the authors still arguing about the conceptual novelty of the work.

From: ICLR.cc/2023/Conference/Paper2590/Authors
title: Response to Reviewer oUN6's Question 3
comment: To recapitulate our previous answer to your question 3(Q3), we are aware of taking advantage of our reconstruction structure in anomaly detection tasks. 

Concerning your constructive comments, we trained our FEAT framework on a real-world public dataset MSL with 27 sub-datasets with 55 variables to explore the application of FEAT in the anomaly detection task. Then, **we trained the MLP reconstruction decoder on the fixed learned representations of normal data, taking advantage of our reconstruction architecture**. The anomaly scores are calculated with the reconstruction errors. The average performance of 27 sub-datasets is presented below.

We compare our performance with Anomaly Transformer’s paper result (Xu et al.,2021; ICLR 2022 spotlight), one of the SOTA anomaly detection specialized models. We followed the same evaluation protocols in their official repository(https://github.com/thuml/Anomaly-Transformer). 

Note that **Anomaly Transformer are taking task-specifically designed architecture** such as Anomaly-Attention blocks. **Although FEAT is not an anomaly-specific but a general representation learning model, it achieves competitive results with Anomaly Transformer**. The result certainly shows that FEAT is a general representation learning framework that robustly performs in a task-agnostic way.

Our valuable insight is that **FEAT encoder can represent the complex properties** of the various time-series datasets, including **normality** which is crucial to differentiate the anomalous points. FEAT encoder can represent the expressive and rich information in a data-driven manner that can enhance the downstream various task performances.


| Metric                    |   **P**   |   **R**  |   **F1**  |
|---------------------------|:---------:|:--------:|:---------:|
| **OCSVM**                 |   59.78   |   86.87  |   70.82   |
| **IsolationForest**       |   53.94   |   86.54  |   66.45   |
| **LOF**                   |   47.72   |   85.25  |   61.18   |
| **Deep-SVDD**             |   91.92   |   76.63  |   83.58   |
| **DAGMM**                 |    89.6   |   63.93  |   74.62   |
| **MMPCACD**               |   81.42   |   61.31  |   69.95   |
| **VAR**                   |   74.68   |   81.42  |    77.9   |
| **LSTM**                  |   85.45   |   82.5   |   83.95   |
| **CL-MPPCA**              |   73.71   |   88.54  |   80.44   |
| **ITAD**                  |   69.44   |   84.09  |   76.07   |
| **LSTM-VAE**              |   85.49   |   79.94  |   82.62   |
| **BeatGAN**               |   89.75   |   85.42  |   87.53   |
| **OmniAnomaly**           |   89.02   |   86.37  |   87.67   |
| **InterFusion**           |   81.28   |   92.7   |   86.62   |
| **THOC**                  |   88.45   |   90.97  |   89.69   |
| **Anomaly   Transformer** | **92.09** |   95.15  | **93.59** |
| **FEAT(Ours)**            |   89.12   | **1.00** |   92.67   |

From: ICLR.cc/2023/Conference/Paper2590/Authors
title: Response to AC
comment: We thank the AC for the valuable comments on our work. We did our best to address the reviewer oUN6’s concerns. FEAT, a general framework for feature-aware multivariate time-series representation learning, achieved the best classification and forecasting performance. However, we are convinced that FEAT can be applied to anomaly detection tasks by learning the normality through our sophisticatedly designed FEAT encoder architecture, as we mentioned in the conclusion. Therefore, we investigated FEAT’s performance on the anomaly detection task concerning the reviewer oUN6’s constructive comment. The detailed reply is presented below.

From: ICLR.cc/2023/Conference/Paper2590/Authors
title:  Response to Reviewer rRns's Question 2 
comment: 
| 　                 |       | **FEAT   (Ours)** |            | **Autoformer** |           | **Informer** |           |
|--------------------|-------|:-----------------:|:----------:|:--------------:|:---------:|:------------:|:---------:|
|       **Dataset**      | **H** |      **MSE**      |   **MAE**  |     **MSE**    |  **MAE**  |    **MSE**   |  **MAE**  |
|      **ETTm1**     |   24  |     **0.3117**    |  **0.379** |      0.383     |   0.403   |     0.323    |   0.369   |
|                    |   48  |     **0.4122**    | **0.4426** |      0.454     |   0.453   |     0.494    |   0.503   |
|                    |   96  |     **0.4617**    |   0.4819   |      0.481     | **0.463** |     0.678    |   0.614   |
|                    |  288  |     **0.5822**    |   0.5575   |      0.634     | **0.528** |     1.056    |   0.786   |
|                    |  672  |       0.7329      |   0.6414   |    **0.606**   | **0.542** |     1.192    |   0.926   |
|       **ECL**      |   96  |       0.2825      |   0.3795   |    **0.201**   | **0.317** |     0.274    |   0.368   |
|                    |  192  |       0.2891      |   0.3841   |    **0.222**   | **0.334** |     0.296    |   0.386   |
|                    |  336  |       0.2989      |   0.3928   |    **0.231**   | **0.338** |      0.3     |   0.394   |
|                    |  720  |       0.3108      |   0.4001   |    **0.254**   | **0.361** |     0.373    |   0.439   |
| **Exchange** |   96  |     **0.177**     |  **0.308** |      0.197     |   0.323   |     0.847    |   0.752   |
|                    |  192  |       0.386       |    0.463   |     **0.3**    | **0.369** |     1.204    |   0.895   |
|                    |  336  |       0.685       |    0.617   |    **0.509**   | **0.524** |     1.672    |   1.036   |
|                    |  720  |     **0.969**     |  **0.774** |      1.447     |   0.941   |     2.478    |    1.31   |
|  **Traffic** |   96  |       0.7177      |   0.4415   |    **0.613**   | **0.388** |     0.719    |   0.391   |
|                    |  192  |       0.7252      |   0.4423   |    **0.616**   |   0.382   |     0.696    | **0.379** |
|                    |  336  |       0.7376      |   0.4459   |    **0.622**   | **0.337** |     0.777    |    0.42   |
|                    |  720  |       0.7614      |   0.4571   |    **0.66**    | **0.408** |     0.864    |   0.472   |
|  **Weather** |   96  |     **0.2536**    | **0.3379** |      0.266     |   0.336   |      0.3     |   0.384   |
|                    |  192  |       0.3461      |   0.4006   |    **0.307**   | **0.367** |     0.598    |   0.544   |
|                    |  336  |       0.4682      |   0.4785   |    **0.359**   | **0.395** |     0.578    |   0.523   |
|                    |  720  |       0.6902      |   0.5981   |    **0.419**   | **0.428** |     1.059    |   0.741   |
|    **ILI**   |   24  |     **2.4683**    | **0.9816** |      3.483     |   1.287   |     5.764    |   1.677   |
|                    |   36  |     **2.4831**    | **0.9787** |      3.103     |   1.148   |     4.755    |   1.467   |
|                    |   48  |     **2.6378**    |  **1.01**  |      2.669     |   1.085   |     4.763    |   1.469   |
|                    |   60  |       2.7819      | **1.0419** |    **2.77**    |   1.125   |     5.264    |   1.564   |
|      **Avg.**      | 　    |     **0.839**     |    0.553   |      0.872     | **0.543** |     1.493    |   0.752   |

From: ICLR.cc/2023/Conference/Paper2590/Authors
title: Response to Reviewer rRns's Question 2 
comment: To recapitulate our previous answer to your question 2(Q2), the submitted paper's results (including both forecasting and classification) used the same fixed hyper-parameters regardless of specific datasets. In this paper, we aimed to show the overall robustness of FEAT with fixed hyper-parameters. 

**To improve our paper with your constructive comment, we fine-tune FEAT to achieve a competitive performance against Autoformer (Wu et al., 2021)**. We evaluated our representations on six datasets, including **four additional datasets** that Autoformer used – Exchange, Traffic, Weather, ILI. We fine-tuned FEAT with respect to epoch, batch size, input sequence length, and representation dimension by each dataset. We evaluated the performance on **the only last timestamp vector of learned representations by FEAT with a** ***simple, light ridge regression*** as we did in our paper. However, we compare our performance with Autoformer’s paper results, which were performed with its forecasting-specific encoder-decoder architecture. **We emphasize that FEAT is a representation learning framework, not a forecasting-specifically designed model.** 

The overall MSE and MAE scores are presented in the Table below. **Despite FEAT’s constraint as a task-agnostic model, it showed impressive results against Autoformer, and significantly outperformed Informer (Zhou et al., 2021)**. This experimental result confirms that **FEAT encoder can** ***represent*** the dynamics of the various time-series datasets, including the **complex properties** such as **feature-specific patterns, feature-temporal dependency, and trend or seasonality** that are all important for forecasting tasks. 

We present the forecasting result below due to the space limit.

From: ICLR.cc/2023/Conference/Paper2590/Authors
title: Response to Reviewer A747
comment: Thank you for your reply to our responses. We are glad that our responses addressed some of the concerns in the review. 

Although previous works have been studied for utilizing both feature and temporal patterns, no previous representation learning methods have been conducted the in-depth module-level analysis to take advantage of feature-level representations effectively. We want to clarify that FEAT is the **first work that learns the feature-specific representations** as well as feature-temporal dependencies **for time-series representation learning**. FEAT’s well-structured architecture is significant to robust performance in various downstream tasks. 

**We updated our general response discussed in phase 2. We described our paper’s valuable insights and analysis of additional experiments (forecasting and anomaly detection).** We look forward to your concerns being addressed with our new general response above sufficiently reflected.

From: ICLR.cc/2023/Conference/Paper2590/Authors
title: General Response for Discussion Phase 2 (Part 3)
comment: **[2] Forecasting**
- As a general presentation learning framework, FEAT showed competitive performance in Autoformer (Wu et al., 2021), one of the task-specific SOTA models, and significantly outperformed Informer (Zhou et al., 2021). **Our expressive representations learned by FEAT encoder** can perform forecasting tasks **effectively with a simple ridge regression model**. The detailed analysis and the result table are presented in our answer to the second question(Q2) of reviewer [rRns](https://openreview.net/forum?id=n9iRY8XFfXW&noteId=cIO1tjzUsl).

**[3] Anomaly Detection**
- Although FEAT is a representation learning framework that is not a task-specifically designed, **FEAT shows an impressive result compared to Anomaly Transformer** (Xu et al.,2021; ICLR 2022 spotlight), one of the SOTA anomaly detection-specifically designed methodologies. The detailed analysis and the result table are presented in our answer to the third question(Q3) of reviewer [oUN6](https://openreview.net/forum?id=n9iRY8XFfXW&noteId=yieF3hwHLEC).

From: ICLR.cc/2023/Conference/Paper2590/Authors
title: General Response for Discussion Phase 2 (Part 2) 
comment: **[1] Classification**
- Note that we used the same hyper-parameters fixed for both classification and forecasting tasks in our paper.  Although we've proved that FEAT can guarantee effective and robust performance on various datasets and tasks, we identified that FEAT could achieve significantly enhanced performance through hyper-parameter tuning for each dataset with respect to epoch, batch size, learning rate, and representation dimension. **Following the classification result below, FEAT resulted in the best for 24 out of 30 datasets and ranked in the Top 2 for 27 out of 30 datasets**. The average accuracy was increased to 75.4% on the 30 datasets, which is 3.2%p higher than our result with the default parameter settings. **Considering practical application on the certain dataset, FEAT can be considered as one of the most robust and highly potential representation learning frameworks**. 


* **Hyper-parameter Tuning results** (The rank with parentheses refers to the averaged rank that reflects the same rank. The bold numbers indicate improved accuracy or rank.)
 | 　                              |          **Before**          |          |   **After**  |            |
|---------------------------------|:------------:|:--------:|:------------:|:----------:|
| **Dataset**                     |**Accuracy** | **Rank** | **Accuracy** |  **Rank**  |
| ArticularyWordRecognition       |  0.991  |    1     |  **0.993**  |     1      |
| AtrialFibrillation              |             0.293            |    1     |  **0.400**  |     1      |
| BasicMotions                    |               1.000              |   1(2)   |    1.000     |   1(2)   |
| CharacterTrajectories           |             0.993            |  2(2.5)  |  **0.995**  | **1(1.5)** |
| Cricket                         |             0.969            |    5     |  **0.986**  |     **3**      |
| DuckDuckGeese                   |             0.564            |    5     |  **0.620**  |   **3(3.5)**   |
| EigenWorms                      |             0.811            |    4     |  **0.870**  |   **1**   |
| Epilepsy                        |             0.948            |    7     |  **0.971**  | **1(1.5)** |
| ERing                           |             0.896            |    2     |  **0.930**  |   **1**   |
| EthanolConcentration            |             0.322            |    2     |  **0.327**  |   **1**   |
| FaceDetection                   |             0.53             |    4     |  **0.574**  |   **1**   |
| FingerMovements                 |             0.488            |    4     |  **0.640**  |   **1**   |
| HandMovementDirection           |             0.378            |    1     |  **0.419**  |     1      |
| Handwriting                     |             0.542            |    1     |  **0.575**  |     1      |
| Heartbeat                       |             0.746            |   2(3)   |  **0.756**  |   **1**   |
| JapaneseVowels                  |             0.983            |    3     |  **0.987**  |     **2**      |
| Libras                          |             0.889            |    1     |  **0.894**  |     1      |
| LSST                            |             0.548            |    3     |  **0.574**  |     **2**      |
| MotorImagery                    |             0.562            |    3     |  **0.630**  |   **1**   |
| NATOPS                          |             0.921            |    2     |  **0.928**  | **1(1.5)** |
| PEMS-SF                         |             0.874            |    1     |  **0.896**  |     1      |
| PenDigits                       |             0.989            |  1(1.5)  |    0.989     |   1(1.5)   |
| PhonemeSpectra                  |             0.216            |    4     |  **0.222**  |   **3(3.5)**   |
| RacketSports                    |             0.888            |    1     |    0.888     |     1      |
| SelfRegulationSCP1              |             0.852            |    1     |  **0.881**  |     1      |
| SelfRegulationSCP2              |             0.562            |    2     |  **0.572**  |     2      |
| SpokenArabicDigits              |             0.986            |    2     |  **0.993**  |   **1**   |
| StandWalkJump                   |             0.533            |    1     |  **0.667**  |     1      |
| UWaveGestureLibrary             |             0.929            |    1     |  **0.938**  |     1      |
| InsectWingbeat                  |             0.462            |    3     |  **0.491**  |   **1**   |
| **On the first   30 datasets:** |              　              |    　    |      　      |     　     |
| **Best**                        |              12              |          |    **24**   |            |
| **Best or   Second-Best**       |              19              |          |    **27**   |            |
| **Avg. Acc.   Avg. Rank**       |             0.722            |   2.467  |   **0.754**  |  **1.443** |

From: ICLR.cc/2023/Conference/Paper2590/Authors
title: General Response for Discussion Phase 2 (Part 1) 
comment: We thank all reviewers and AC for the thoughtful comments. In discussion phase 2, we’ve discussed our paper’s technical and empirical significance with additional experimental results. We did our best to address the reviewers' concerns and improve our paper with the reviewers’ constructive comments. **We are convinced that FEAT provides a significant insight for future multivariate time-series data research in addition to its practical usability.** The official code is going to be available after the final decision.

We present the general response regarding discussion phase 2 as below. 

FEAT provides **valuable insights** as follows:
- FEAT is **designed with sophisticated techniques for general presentation learning**. Unlike previous representation learning methods, FEAT has **a feature encoder in parallel with temporal encoder** to increase the expressiveness of each variable **via feature-contrasting** and has **an element-wise gating layer** which can flexibly learns the importance of each variable as well as temporal dependency in data-driven manner. 
**No previous representation learning methods have been conducted the in-depth module-level analysis to effectively use feature-level representations**. However, we firstly analyze the effectiveness of proposed **feature-aware architecture** by quantitative ablation studies (Table 3, Table 8) and qualitative analyses (Figure 4 and Figure 6).
- Although FEAT is **not a task-specific or data-specific structure**, it shows competitive classification, forecasting, and anomaly detection performance, all of which are **comparable to the SOTA task-specific methods**. 
- FEAT is **a distinctive architecture for capturing both feature-specific and feature-agnostic information**. We provide **valuable insight that our FEAT encoder structure can well represent the time-series properties robustly and effectively** (feature-specific patterns, temporal dependency, trend, seasonality, and even normality) **through extensive evaluations on the 'learned' representations**.  Therefore, we assert that leveraging both feature-specific and feature-agnostic time-series patterns **by using our FEAT encoder architecture benefits a wide range of application areas**.

New analysis of additional experiments:
- **More experiments were conducted on three different tasks** (classification, forecasting, anomaly detection) based on the reviewer's constructive comments in Discussion 1. It was confirmed that FEAT works robustly on various datasets and tasks as a general representation learning model, and that FEAT encoder is significant for learning complex and implicit time-series pattern (feature-specific patterns, trend, seasonality, or normality).

From: ICLR.cc/2023/Conference/Paper2590/Reviewer_A747
title: Well empirical study, but less technical significance
comment: The authors' reply addressed some of the concerns in the review. 

The proposed time series representation learning method is practical and shows some competitive performance on some datasets. The empirical study of this paper is appreciated.

The remaining concern is mainly about the technical significance of the proposed method. Exploiting intra-, cross-variate, and temporal relations, and common factors (corresponding to the "variable-agnostic temporal pattern" in this paper) is a long-standing idea for multivariate time series. Given this context, the technical implementation for capturing the above relations in this paper is mostly basic, though having some interesting combinations, e.g., contrast loss, data augmentation, etc. Meanwhile, the lack of theoretical insights also weakens the technical significance. Thus, the score is still at 6.

From: ICLR.cc/2023/Conference/Paper2590/Area_Chair_rjLE
title: Please respond to author comments (Paper 2590)
comment: Reviewer oUN6,

Please respond to the comments by the authors of Paper 2590.
This is an important part of the review process and your job as a reviewer is not done until you read their response and ascertain whether it addressed your concerns.

Please indicate which of your concerns were addressed and update your score accordingly.

You indicated that you do not see the paper as sufficiently novel, can you please point out the prior work that you consider to be most similar to what the authors propose? The authors explain how their work is different from TS2Vec.

You indicated the paper does not handle anomaly detection, but handles classification and regression. In general, a method does not need to solve every category of problems to be valuable, so this should definitely not count against the paper.

If some of your concerns were not addressed, please explain what's missing from the author response.

All the best,

The AC

From: ICLR.cc/2023/Conference/Paper2590/Area_Chair_rjLE
title: Please respond to author comments (Paper 2590)
comment: Reviewer rRns,

The authors have responded to your review. They explained why their work is novel and explained why their work is better than Autoformer.
Does their response address your concerns? 

If so, please update your score.
If not, please explain why. 

As a general rule, when you make a statement such as 'X is not a new technique for time-series forecasting', you *should* provide references to papers that are the most similar to what the authors proposed. Ultimately, almost everything published these days in AI is a combination of existing prior work, but interesting combinations with emerging properties (i.e. high performance) are of still of interest in the field, as are papers that increase our understanding of the circumstances under which certain methods are better than others.

All the best,

The AC

From: ICLR.cc/2023/Conference/Paper2590/Authors
title: General Response and Summary of draft modification
comment: We thank all reviewers for the constructive and valuable comments. We’ve replied to all reviewers by responding to each question, and we especially clarified our novelty by answering Q1, which concerns each reviewer. We present the general response and the summary of the draft modification below. 

First, we are glad that you found our paper’s strengths as follows:

* "an intuitive and effective framework" and "practically interesting results and effectiveness" (Reviewer A747)
* "clearly written" and "interesting results with a good motivation" (Reviewer rRns)
* "novel idea that learns feature-specific patterns along with the feature-agnostic temporal patterns" and "good writing quality and clarity" (Reviewer oUN6)

Our novelty can be described as follows:
* FEAT learns time-series representation for the first time in terms of three diversified key perspectives through all components in its Encoder Layer: feature-specific patterns, feature-agnostic temporal patterns, and dependency between multiple feature-specific and temporal information.
* Unlike previous architectures, we first focus on learning the beneficial feature-level representation as well as how the features are dynamically aligned on the temporal axis. FEAT can capture the feature information which behaves uniquely in the given context or across the datasets before detecting feature-temporal dependency.
* Specifically, through the proposed gating layer, the model can automatically control the usage of feature-specific information at each timestamp.
* We design an optimal and robust multivariate time-series representation learning model that can simultaneously benefit the feature-specific information and the feature-agnostic. Moreover, we demonstrated that each component of FEAT contributes to performance improvement through experiments. 

In addition, the summary of our draft modification is as below:
* To describe the novelty of FEAT that models the feature-specific information, we revised the first paragraph in Section 2 and added a paragraph before explaining the Decoder Layer in Subsection 2.3.
* We added the sensitivity analysis of hyper-parameters in Appendix A.1 (Table 5).
* We added the result of the ablation study on the forecasting task in Appendix A.4 (Table 8).
* Minor grammar and expressions have been modified.

From: ICLR.cc/2023/Conference/Paper2590/Authors
title: Response to Reviewer oUN6 (Part 2)
comment: **Q3.** FEAT only conducts experiments on classification and forecasting tasks, but doesn't analyze the application or performance of other tasks such as anomaly detection, resulting in a lack of convincing to a certain extent. 

**A3.** As we mentioned as our future work in conclusion, we are aware of the application by taking advantage of our reconstruction structure in anomaly detection tasks. We are in the progress of evaluating anomaly detection datasets, and we can share the results as soon as possible in discussion phase 2.


\
**Q4.** In Section 2.2, what is the role of Random Shifting? What is the specific operation of padding or trimming? Different manipulations may greatly affect the results and there are no corresponding explanations.

**A4.** The main role of Random Shifting is to generate random augmented contexts. As shown in Figure 2a, we horizontally shift the input time series (left & right). We propose ‘shifting’ as an augmentation technique not only for satisfying contextual consistency but also for keeping the length $T$ of the input sequence ($T \times F$) as well as augmentation. We obtain feature-wise embedding through mapping dimension T of the transposed input sequence ($F \times T$) to $d_1$ by the fixed weight matrix ($T \times d_1$).
For example, we can get two augmented contexts as follows. (e.g., shift size = 2)

>	Input sequence: [1,2,3,4,5,6,7,8,9,10]

>	(I) Shifted left: [3,4,5,6,7,8, 9,10 0 0] (Padding the last two timestamps, and trimming the first two timestamps)

>	(II) Shifted right: [0,0,1,2,3,4,5,6, 7, 8] (Padding the first two timestamps and trimming the last two timestamps)

Then, the overlapped temporal representation ([3,4,5,6,7,8]) from (I), (II) are used for temporal contrasting and the shifted feature-wise representation (I), (II) are used for feature contrasting.


\
**Q5.** There are many dotted lines in the architecture diagram. What do they try to express and what are their specific roles?

**A5.** We described the role of the dashed line in the caption of Figure 2; The dashed line indicates the stream of the augmented context pair by random shifting. For the detailed explanation, the solid and dashed lines stand for the sources for calculating temporal contrasting loss, feature contrasting loss, and reconstruction loss. Specifically, the contrasting sources originate from the shifted augmented pairs, denoted as ‘context 1’ and ‘context 2’ in Figure 2.


\
**Q6.** In Section 2.4, the authors assume that the distribution of each feature sequence from the shifted pairs is similar to each other, which greatly limits the scope. How to apply the method when the assumption is not satisfied?

**A6.** First, we remind that the assumption is for the feature contrasting loss. As described in the last sentence in Section 2.2, we introduce the maximum shift size to make each distribution of shifted feature pair keep the similar distribution of the original feature sequence. We assume that marginal shifts (e.g., three timestamps) can avoid generating positive pairs with different sequences without significantly changing sequence patterns. This doesn’t harm learning feature-specific representations, given that each shift size is randomly selected at every iteration.

We can apply the method with a smaller shift size to avoid or resolve situations where the assumption is not satisfied. For your information, even the datasets with a short length, such as PenDigits($T=8$) and RacketSports($T=30$) showed the best performance on the classification task under the maximum shift size 3.

Additionally, as shifting is for data augmentation, it can be considered as a type of masking augmentation strategy. Therefore, to adapt our method, especially for feature contrasting, replacing the random shifting with other augmentation methods, such as masking or transformation techniques, can be considered.

From: ICLR.cc/2023/Conference/Paper2590/Authors
title: Response to  Reviewer oUN6 (Part 1)
comment: Thank you for your valuable comments and questions regarding our paper. We appreciate your recognition of our method, “novel idea that learns feature-specific patterns along with the feature-agnostic temporal patterns” and “good writing quality and clarity”. The detailed responses to your main concerns and questions are as below:

\
**Q1.** In the method part, almost every subsection refers to the TS2Vec method. Some processing and flow of data are similar to TS2Vec. However, the novelty for modeling the feature-specific information is less described, giving the impression that this method is quite incremental on the basis of TS2Vec.

**A1.** We would like to explain how FEAT methods differ from TS2Vec. In advance, taking advantage of temporal contrasting to satisfy contextual consistency is helpful to design a general framework as TS2Vec does. Meanwhile, **our most significant achievement is to propose an effective representation learning architecture for the first time** to leverage **both feature-specific and feature-agnostic information simultaneously.** We aim to address the limitation of feature-agnostic representation learning methodologies, including TS2Vec that their learned representation cannot reflect the explicitly modeled feature-specific patterns. 

The followings are **the modules we newly designed.**
**First,** we introduce random shifting (Section 2.2) that significantly contributes to pipelining feature-wise embedding (Section 2.1) and feature-contrasting (Section 2.4), as well as satisfying contextual consistency. **Second,** we design the effective layers, feature encoder (Section 2.3), and element-wise gating layer (Section 2.3) to adaptively model the feature-specific patterns. Besides, we adopt an auxiliary task with reconstruction decoder (Section 2.3) to help the final representation to be appropriately fused by gating between feature-agnostic and feature-specific information. **Through these modules, FEAT could effectively learn the time-series representation in terms of diversified perspectives** - feature-agnostic temporal representation($R_1$), feature-specific representation($R_2$), and feature-specific temporal representation($R^{\prime}_2$). 

In summary, we can rephrase the above paragraphs as follows. Although we refer to TS2Vec to explain its limitation or take advantage of its beneficial techniques (masking input embedding in Section 2.2 and temporal contrasting in Section 2.4), we **newly adapt the augmentation technique (random shifting in Section 2.2)** and **propose the well-structured architecture to properly utilize the feature-specific patterns (Sections mentioned in the above paragraph).** Indeed, the ablation study in Section 3.3.1 shows that **newly introduced architecture is organically taking an important role.**

\
**Q2.** The presentation of the experimental results is not rigorous. FEAT is not optimal on each classification dataset, but on average of 30 datasets. What is the criterial of chose 30 datasets from UEA? 

**A2.** First, we chose UEA datasets because most recent time-series representation learning models are evaluating multivariate time-series classification tasks on them, which are from various domains. UEA archive provides 128 univariate datasets and 30 multivariate datasets (https://arxiv.org/abs/1811.00075). Therefore, 30 datasets we used are not the selected subset, and we exactly used all UEA datasets.

Second, the results of all datasets in our paper (including classification and forecasting) were evaluated on the same fixed hyper-parameters regardless of datasets and tasks. We intended to show the overall competitive capacity by showing both results of each dataset and the average one. We expect that each dataset can obtain its own optimal solution by fine-tuning. For further explanation, we summarize the number of best-performing datasets and the number of second-best-performing datasets for each representation learning method in Table 1. FEAT resulted in the best in more than a third of the entire dataset, and it was ranked either first or second for nearly two-thirds of the datasets.

|                         | **FEAT** | **TS2Vec** | **T-Loss** | **TNC** | **TS-TCC** | **TST** | **DTW** |
|-------------------------|----------|------------|------------|---------|------------|---------|---------|
| **Best**                | **12**   | 7          | 4          | 2       | 6          | 1       | 2       |
| **Best or Second-Best** | **19**   | 16         | 12         | 5       | 7          | 4       | 5       |

From: ICLR.cc/2023/Conference/Paper2590/Authors
title: Response to Reviewer rRns
comment: Thank you for your constructive feedbacks regarding our paper. We appreciate that you found our paper "clearly written" and "interesting results with a good motivation”. We address your main concerns as below:

\
**Q1.** However, feature-wise and temporal-wise modeling to capture feature-specific and temporal patterns is not a new technique for time-series forecasting tasks and the suggested method seems to be a simple combination of pre-existing techniques.

**A1.** We want to clarify that our contribution is **to design an optimal and robust multivariate time-series representation learning model that can effectively benefit the feature-specific information and the feature-agnostic simultaneously** as well as can be adopted for both classification and forecasting.  Specifically,  **FEAT is the first work that disentangles   the time-series representation into three perspectives** – feature-agnostic temporal representation($R_1$), feature-specific representation($R_2$), feature-specific temporal representation($R^{\prime}_2$). Unlike previous works, we first focus on learning **the beneficial feature-level representation** as well as **how the features are dynamically aligned on the temporal axis.** This enables the model to **focus solely on extracting each unique feature information in the given input context** and **subsequently pass it to benefit learning dependency between all features and timestamps.**  Moreover, through **the proposed gating layer**, the model can **automatically control the usage of feature-specific information at each timestamp.** 

In addition, we compared FEAT and two forecasting models which take feature-wise and temporal-wise modeling in response to the reviewer A747’s comments. I would appreciate it if you could refer to our answer to the first question(Q1) of reviewer A747. 

\
**Q2.** The authors should compare its performance with Autoformer model as it is the one of the recent forecasting methods that shows a good performance for forecasting tasks.      
**A2.** Thank you for your suggestion in advance. For your information, the results (including both forecasting and classification) in the submitted paper used the same fixed hyper-parameters regardless of specific datasets. In this paper, we aimed to show the overall robustness of FEAT with fixed hyper-parameters. Therefore, if we fine-tune FEAT for the forecasting datasets, it can achieve a competitive performance against Autoformer. We are fine-tuning our model on the new datasets that Autoformer used. Because the experiments are still in progress and the results seem to be aggregated after discussion phase 1, we will report them by replying here in discussion phase 2.

From: ICLR.cc/2023/Conference/Paper2590/Authors
title: Response to Reviewer A747 (Part 2)
comment: **Q3.** In the experiment, the ablation study seems to be on the classification task. It would be good to also see the ablation study on the forecasting task. 

**A3.** We **conducted ablation study on the forecasting task and added it in Appendix A.4 (Table 8).** It was performed on three ETT datasets, and the results are the averaged MSE and MAE scores of the five repetitions at all forecasting horizons. We used the same hyper-parameters fixed for both classification and forecasting tasks in our paper. In terms of both MSE and MAE, all ablations exhibit degradation, and the full model achieved the best, which supports that each component of FEAT contributes to the performance improvement.

\
**Q4.** Meanwhile, as for the hyper-parameters, it seems they are set to fixed values. For the hyper-parameters of the training process, e.g., learning rate, batch size, regularization, etc, it would be good to have a hyper-parameter search process, since it might affect the performance significantly. 

**A4.** We **newly added the sensitivity analysis of hyper-parameters in Appendix A.1 (Table 5).** To explore the sensitivity of the hyper-parameters, we independently check the effectiveness of each hyper-parameter. We evaluate the model performance in terms of variation in batch size, learning rate, representation dimension, and reconstruction loss function. Hyper-parameters marked with * are used for the performance in the main paper. Considering the overall performance on 30 classification datasets, no harmful degradation occurs around the default settings. Based on this sensitivity analysis, we found the FEAT is not very sensitive to small change of hyper-parameters, which can be another advantage of practical implementation.

From: ICLR.cc/2023/Conference/Paper2590/Authors
title: Response to Reviewer A747 (Part 1)
comment: Thank you for the valuable comments. We are glad that you found our framework "an intuitive and effective" and "extensive experiments” showing “practically interesting results and effectiveness". We address your main concerns and respond to your questions as follows:

\
**Q1.** Modeling the temporal-wise and variable-wise relations (or temporal-spatial relations) in multivariate time analyses is an extensively studied subject, e.g., [1, 2, 3]. The techniques used in the proposed method are mostly standard and lack some new insights.

**A1.** We would like to clarify the new insights of FEAT compared to [1,2,3].  First, both [1] and [2] can be considered as a recurrent architecture that learns the covariate (or variable-wise inter-dependency) of a certain timestamp, or the cumulated variable information up to a certain timestamp. That means the variable-specific information is utilized recurrently in timestamp-wise dependent manner. In contrast, FEAT is a non-recurrent architecture **that models the variable-specific information under the given context.** Moreover, our main goal is to represent the variable-wise specific patterns independently prior to learn covariates only at certain timestamp. FEAT **first learns the variable-specific varieties independently** by feature contrasting, **then learns the dependency between each timestamp and multiple variables.** Although directly using covariates at each timestamp can provide helpful information, it is limited in that **it cannot capture the feature information which behaves uniquely in the given context or across the datasets** before the timestamp-dependent covariates which are locally detected. However, **FEAT is designed to capture variable-specific patterns, variable-agnostic temporal patterns, and dependency between multiple variable-specific information and temporal information. A key component of FEAT is to adjust the contribution of each part aforementioned information adaptively learned by gating layer.** It is more sophisticatedly designed and can be applied whether the variables are highly correlated.  
Second, [3] could seem similar to FEAT because it proposes variable-wise specific modeling. However, [3] uses a univariate-wise method that only captures the dependency between spectral and temporal information **present in ‘a’ variable (# var 1 : # time T; dependency).** In contrast, as we mentioned above, **FEAT takes a more multilateral perspective that captures variable-agnostic temporal pattern (# time T), variable-specific pattern (# var 1), and variable-specific temporal pattern (# var F : # time T).**

\
**Q2.** The final representation is aligned with the time dimension. The other case would be to align it to the feature dimension. Do the authors have some sense of whether the different design choices would make a difference?

**A2.** Our main goal is to represent the input sequence with length $T$ in a vector with the same length, in which each timestamp can be represented respectively. Specifically, the reason why we chose such an alignment design relates to the role of our temporal and feature encoder. FEAT’s temporal encoder learns feature-agnostic temporal patterns, the overall trend or seasonality that every feature inherently shares. Meanwhile, the feature encoder learns feature-specific information regardless of the exact timestamp order. We are concerned that if we align the feature-agnostic temporal patterns to feature-specific information, the feature dimension, there is a risk of information loss of the temporal axis, which is crucial in time-series. Therefore, we align the feature-specific information to the temporal dimension that can not only maintain the temporal axis but also reflect the feature-specific influences. 
Meanwhile, aligning the final representation to the feature dimension can be used to analyze the feature importance of a given sequence.

From: ICLR.cc/2023/Conference/Paper2590/Reviewer_A747
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper focuses on the representation learning framework for multivariate time-series data. The proposed method learns feature-specific and temporal patterns. Then, the feature-specific patterns are transformed into representations of the same time dimension as the temporal patterns, such that an aggregated representation can be derived. 
The learning objective consists of contrasting and reconstruction loss. Experimental evaluation shows the competitive performance of the proposed method on classification and regression tasks. 
strength_and_weaknesses: Strength:

The proposed method considers both feature-specific and -agnostic patterns. Through a gating layer, the final timestep-wise representation integrates both the temporal pattern and feature-specific pattern. The experimental results show some interesting performance improvement compared with baselines on several datasets. 

Weakness:

Modeling the temporal-wise and variable-wise relations (or tempral-spatial relations) in multivariate time analyses is an extensively studied subject, e.g., [1, 2, 3]. The techniques used in the proposed method are mostly standard and lack some new insights.

The final representation is aligned with the time dimension. The other case would be to align it to the feature dimension. Do the authors have some sense of whether the different design choices would make a difference?

In the experiment, the ablation study seems to be on the classification task. It would be good to also see the ablation study on the forecasting task.

Meanwhile, as for the hyper-parameters, it seems they are set to fixed values. For the hyper-parameters of the training process, e.g., learning rate, batch size, regularization, etc, it would be good to have a hyper-parameter search process, since it might affect the performance significantly. 

[1] Qin, Yao, et al. "A dual-stage attention-based recurrent neural network for time series prediction." Proceedings of the 26th International Joint Conference on Artificial Intelligence. 2017.

[2] Guo, Tian, Tao Lin, and Nino Antulov-Fantulin. "Exploring interpretable LSTM neural networks over multi-variable data." International conference on machine learning. PMLR, 2019.

[3] Pal, Soumyasundar, et al. "RNN with particle flow for probabilistic spatio-temporal forecasting." International Conference on Machine Learning. PMLR, 2021.

[4] Yang, Ling, and Shenda Hong. "Unsupervised time-series representation learning with iterative bilinear temporal-spectral fusion." International Conference on Machine Learning. PMLR, 2022.


clarity,_quality,_novelty_and_reproducibility: The paper is clearly presented. Though there is no significantly novel techniques developed in this paper, it shows practically interesting results and effectiveness. 
summary_of_the_review: This paper focuses on the representation learning framework for multivariate time-series data and proposed an intuitive and effective method. 
Extensive experiments on both classificaiton and forecasting tasks demonstrate the effectiveness of the method. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper2590/Reviewer_rRns
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper propose a multi-variate time-series forecasting model which captures feature-aware temporal representation and learns feature-specific patterns with an element-wise gating layer. They validate their model on time-series benchmark datasets (classification, regression) with its baseline and the experimental results shows that the propose model seems to outperform its baselines on some classification tasks and most forecasting tasks.

strength_and_weaknesses: The proposed method is technically simple, but it seems to show outstanding performance, compared to popular time-series forecasting models on the  benchmark datasets. The experimental setting is well-designed with many datasets for classification and regression tasks.

However, feature-wise and temporal-wise modeling to capture feature-specific and temporal patterns is not a new technique for time-series forecasting tasks and the suggested method seems to be a simple combination of pre-existing techniques.

Suggestion 1) The authors should compare its performance with  Autoformer model as it is the one of the recent forecasting methods that shows a good performance for forecasting tasks. 
clarity,_quality,_novelty_and_reproducibility: I found the paper is clearly written and easy to understand. 

summary_of_the_review: The authors provides a good description about their motivation and proposed method, if technically correct, seems to address the targeted problem. Although the shown results are quite interesting (regression tasks) with a good motivation, it is a bit hard to think the proposed method is novel. 

correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper2590/Reviewer_oUN6
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes a general framework for FEature-Aware multivariate Time-series representation learning, called FEAT, to resolve two limitations of most time series representation learning methods. This method explicitly models the feature-specific information for the first time along with feature-agnostic temporal dependency. And this method proposes an element-wise gating layer to flexibly learn the influence of feature-specific patterns per timestamp in a data-driven manner. The results show that FEAT outperforms existing state-of-the-art representation learning methods on most datasets of classification and forecasting tasks, and demonstrate its effectiveness.

strength_and_weaknesses: Strength:

The results show that FEAT outperforms existing state-of-the-art representation learning methods on most datasets of classification (improvement of 1.9%p on average in terms of accuracy ) and forecasting tasks (improvement of 9.54% on average in terms of MSE).
This paper proposes a novel idea that learns feature-specific patterns along with the feature-agnostic temporal patterns.
This paper gives a detailed mathematical description of their method and overall architecture, showing how to model the feature-specific information and the feature-agnostic simultaneously.

Concerns and Questions:

In the method part, almost every subsection refers to the TS2Vec method. Some processing and flow of data are similar to TS2Vec. However, the novelty for modeling the feature-specific information is less described, giving the impression that this method is quite incremental on the basis of TS2Vec. 

The presentation of the experimental results is not rigorous. FEAT is not optimal on each classification dataset, but on average of 30 datasets. What is the criterial of chose 30 datasets from UEA?

FEAT only conducts experiments on classification and forecasting tasks, but doesn't analyze the application or performance of other tasks such as anomaly detection, resulting in a lack of convincing to a certain extent.

In Section 2.2, what is the role of Random Shifting? What is the specific operation of padding or trimming? Different manipulations may greatly affect the results and there are no corresponding explanations.

There are many dotted lines in the architecture diagram. What do they try to express and what are their specific roles?

In Section 2.4, the authors assume that the distribution of each feature sequence from the shifted pairs is similar to each other, which greatly limits the scope. How to apply the method when the assumption is not satisfied?

clarity,_quality,_novelty_and_reproducibility: The writing quality and clarity are good. The novelty is limited. 
summary_of_the_review: The novelty is limited.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold