Paper: GAFormer: Enhancing Timeseries Transformers Through Group-Aware Embeddings (ICLR-2024)

Reviewer comment (Experiments):
The GAFormer goes through the SGE and then the TGE, what happens if the order of the two modules is reversed? Why is it modeled from the spatial point of view first?

Author response:
**1. Swapping the order of the SGE and TGE encoders**

Thank you for your insightful question! To better understand the effectiveness of SGE and TGE, we conducted experiments to switch the order of our group embedding layers in GAFormer. In this case, we compare our model with SGE followed by TGE to a model where we apply TGE and then SGE.

| | SGE -> TGE  | TGE -> SGE | 
| -------- | ------- | -------- | 
| MotorImagery | 61.0 | 58.0 | 
| SelfRegSCP2 | 56.11 | 48.33 | 

In this experiment, we find that there is a large drop in performance when we switch the order of both operations. This also seems to align with our previous finding inTable 4 where we ablate TGE and SGE and find that SGE provides more significant improvements. Thus, we think that adding SGE at the input layer can have more transformative effects on the model performance, as it learns a discriminative representation for each timepoint by attending to multi-channel relations, which is the basis of the the following TGE block which models inter-timepoint relations. Additionally, we adopt the SGE-TGE architecture to make the model adaptive to different tasks. Some tasks require time-wise representations and some tasks require global representation. We get time-wise representations from the last layer of our model and simply average all timepoints to get global representation.

Reviewer comment (Presentation):
Can the author explain Figure 4 again, I don't really understand the interpretability of the authors' proposal.

Author response:
**2. How group embedding assignments provides insights into the underlying spatial and temporal structure in timeseries datasets**

Thank you for your question, which is an important question in our rebuttal. Please refer to general response 2 for an example of neural spiking data. We show that the learned temporal groups can indicate temporal structures in timeseries data in Appendix Figure 6.

To explain Figure 4, we want to recall that there are 64 channels and 100 timepoints (after the patching layer proposed by [1]) for a single sample in the MotorImagery dataset. For SGE figures (upper row), we have 10 group embeddings and each channel will be assigned with 1 group embedding (the value of Matrix(i, j) indicates the probabilities that channel i is assigned with group j). The visualizations indicate how the 64 channels are grouped within each sample based on the token similarity. Similarly, TGE visulaizations (lower row) indicate how the timepoints are grouped within each sample. And we can also get insights of inter-sample correlations by comparing their visulizations because all samples share the same set of group embeddings.

In contrast to a standard position embedding (PE) which is **fixed** for each datapoint in the dataset, GE is **adaptive to data** and thus for each token and sequence, we build a data-adaptive embedding through a mixture of the learned group tokens. This in turn reveals different grouping assignments for each input token. We find that when we use a GE, we often obtain sparse assignments across the different group tokens, which provides insights into **how different channels or time points are grouped**. In the temporal embeddings, we find contiguous groups that divide the sequence into different phases: the results on neural population dynamics contains integer-valued bins and also results in very sparse and nearly binary group assignments and often correspond to visible transitions in the data (Appendix Figure 6). This point will be elaborated upon in the revision.

Reviewer comment (Evaluation):
Table 4 shows that SGE is more effective than TGE, can the authors explain why?

Author response:
**3. Why SGE is more effective than TGE?**

Thank you for your insightful question! We hypothesize that because the temporal structure is naturally preserved across nearby samples in the input sequence, spatial information is more important to harness and learn to inject into the input. Additionally, we hypothesize that the channel-wise relationship is even more critical for classification and regression tasks. This is in contrast to timeseries forecasting tasks [1], where performance remains consistent irrespective of whether the spatial relationship across channels is taken into account. A recent paper [2] also provides extensive discussion on the importance of modeling inter-channel relations.

**Reference**

[1] Nie, Yuqi, et al. "A time series is worth 64 words: Long-term forecasting with transformers." arXiv preprint arXiv:2211.14730 (2022).

[2]Yong Liu, et al. “iTransformer: Inverted Transformers Are Effective for Time Series Forecasting.” arXiv preprint arXiv:2310.06625.

Reviewer comment (Evaluation):
Though the approach claims to offer "a more interpretable decomposition," this is a subjective claim. The degree of interpretability might vary based on the user or the specific use case.

Author response:
**1. How to define interpretability**

Thanks for your insightful question on the definition of interpretability! Please refer to general response 2 for an example of neural spiking data which can provide some intuition of learned groups. We found the learned temporal groups can indicate the temporal structure of the timeseries data. 

While we do agree that interpretability is often subjective, the challenges in understanding and extracting the spatiotemporal structure of timeseries data is an outstanding challenge, especially given no pre-assumed underlying structure across channels. We hope our work could further advocate the research along this line, and thus is important to the community.

To address the reviewer’s concern, we attempted to quantify the interpretability through an additional zero-shot fine-grained classification experiment on a synthetic many-body dataset. Specifically, we only train the model to classify the total energy of the system (high energy v.s. low energy, a binary classification task), and later freeze the weight of the transformer encoder. At test time, we extract the token representation of each channel (object), and then train a KNN classifier (k=5, 8-class problem to classify the x, y axis of 4 objects) to cluster and classify the type of each object using the learned token representation. In this case, we can see how GE impacts the learned token representation without providing any information of the type of channel or additional training.

Classification accuracy with positional embedding and group embedding:

| | Positional Embedding  | Group Embedding |
| -------- | ------- | -------- | 
| Acc | 61.65 | 71.23 |


We can see that, with group embeddings, the learned token representation contains more information about the type of the channel in our synthetic setting, demonstrating its interpretability. We believe our architecture can be used to understand the spatiotemporal structure of timeseries datasets, giving the potential to perform zero-shot fine-grained classification.

Reviewer comment (Reproducibility):
How to tokenise the MTS data?

Author response:
**2. Details of the Time-Series Tokeniser**

Thank you for your insightful question! We believe that the choice of a “default tokenizer” for time-series is still an active research question, and thus explored two different versions of the tokenizer to make sure that our approach generalizes to both cases. Specifically, we attempted (i) a single-layer MLP to create token representation from a patch of continuous time points of a temporal window from each channel (adopted by PatchTST [1] and GAFormer in our paper); and (ii) a single-layer MLP that creates token representation from all channels of the timeseries at one time point (adopted by MVTS [2], one of the baselines in our paper). The MLP layer is trained end-to-end. Text tokenizer usually builds on a predefined vocabulary and there are discrete word embeddings for each word/token. Different from text tokenizer which process discrete words, time-series tokenizer processes overlapping patches of timeseries, which map continuous timepoints into a higher latent space.

Reviewer comment (Writing):
What's the different between a text tokeniser and a MTS tokeniser?

Author response:
**2. Details of the Time-Series Tokeniser**

Thank you for your insightful question! We believe that the choice of a “default tokenizer” for time-series is still an active research question, and thus explored two different versions of the tokenizer to make sure that our approach generalizes to both cases. Specifically, we attempted (i) a single-layer MLP to create token representation from a patch of continuous time points of a temporal window from each channel (adopted by PatchTST [1] and GAFormer in our paper); and (ii) a single-layer MLP that creates token representation from all channels of the timeseries at one time point (adopted by MVTS [2], one of the baselines in our paper). The MLP layer is trained end-to-end. Text tokenizer usually builds on a predefined vocabulary and there are discrete word embeddings for each word/token. Different from text tokenizer which process discrete words, time-series tokenizer processes overlapping patches of timeseries, which map continuous timepoints into a higher latent space.

Reviewer comment (Experiments):
What's the effect of different K?

Author response:
**3. Determine the appropriate size of K**

We conducted additional experiments to examine the robustness of our model as we vary the number of groups (K). The results are provided for MotorImagery and FaceDetect datasets below. 

| Values of K | 5  | 10 | 15 | 20 | 25 |
| -------- | ------- | -------- | ------- | ------- | ------- |
| MotorImagery | 55.0 | 61.0 | 62.0 | 63.0 | 58.0 |
| FaceDetect | 67.02 | 67.99 | 67.08 | 68.53 | 68.19 |

We found that our method is relatively stable across different values of K, with only small performance degradation when K is too small or too large. When visualizing the different group assignments it appears that larger values of K will produce a number of tokens with very sparse/few assignments, suggesting that additional unnecessary tokens may not be used and thus similar performance is obtained for the overcomplete case. 

We will include these results in an updated version of the paper. Thanks for your suggestion!

Reviewer comment (Evaluation):
The group structure exhibited in Figure 4 is claimed to provide interpretability of the model. However, these are some groups that have been learnt and the interpretability aspect is not clear as to what it means if some channels or time signals are grouped. This is similar to the groups formed in any attention mechanism and the method doesn’t seem to provide any added explanations or interpretability to the decisions. Thus it may not be right to claim interpretability based on group embedding structure.

Author response:
We thank you for your comments and agree that our argument for GE providing enhanced interpretability could be explained more clearly. Please refer to general response 2 for an example of neural spiking data. We show that the learned temporal groups can indicate temporal structures in timeseries data in Appendix Figure 6.

In particular, we hope to better highlight the ability of the model to capture both temporal structure and spatial structure in the data. Our approach can do this in two ways: (i) through the choice of assignments to group tokens, and (ii) by improving the representations learned by the model to enhance interpretability or explainability in other ways. 

In our extended supplemental figures, we now include more examples and visualizations of the time-series data along with the group assignment coefficients. In these examples, we can see how the switches in dynamics across the time-series are also captured in the group assignment matrix.

Reviewer comment (Evaluation):
The group structure exhibited in Figure 4 is claimed to provide interpretability of the model. However, these are some groups that have been learnt and the interpretability aspect is not clear as to what it means if some channels or time signals are grouped. Further analysis and explanation by the authors may help.

Author response:
We thank you for your comments and agree that our argument for GE providing enhanced interpretability could be explained more clearly. Please refer to general response 2 for an example of neural spiking data. We show that the learned temporal groups can indicate temporal structures in timeseries data in Appendix Figure 6.

In particular, we hope to better highlight the ability of the model to capture both temporal structure and spatial structure in the data. Our approach can do this in two ways: (i) through the choice of assignments to group tokens, and (ii) by improving the representations learned by the model to enhance interpretability or explainability in other ways. 

In our extended supplemental figures, we now include more examples and visualizations of the time-series data along with the group assignment coefficients. In these examples, we can see how the switches in dynamics across the time-series are also captured in the group assignment matrix.

Reviewer comment (Experiments):
As the authors have mentioned other mechanisms (spot attention etc.) to induce group structure, I would expect some empirical study to compare the proposed method with these PEs. Specifically, is the proposed method a unique way to induce PEs for multivariate time series forecasting or could other techniques provide similar or better performance?

Author response:
We would like to clarify that our baselines and ablations provide different variants of position embedding both in time and in space. However, comparisons with other methods mentioned like slot attention have not been applied to time series datasets and extending them to work across channels is not a trivial task.  To the best of our knowledge, we are the first to explore the concept of group embedding in time-series where there is no spatial information or known correlations across “nearby channels” and thus there are no other PE baselines to compare against.

Reviewer comment (Theory):
The proposed methodology is not well motivated and built on the existing Transformer. The paper tries to introduce the group embedding layer but does not provide clear motivation for the group embedding, i.e. how to define the "group" and why we need the group embedding.

Author response:
Thank you for raising this point about needing to improve the explanation behind group embeddings (GE). We plan to revise the introduction and methods to provide more intuition and explanation of why group embeddings are needed for multivariate timeseries.

Our approach is centered on developing data-adaptive spatiotemporal embeddings, which augment the conventional positional embeddings typically employed in sequential data processing. This is achieved by creating a compact set of universal embeddings that are dynamically combined, often through a sparse linear composition. This process results in customized positional embeddings, varying across different sequences, allowing the model to apply a "group embedding" either to specific channels or points in time. This can be perceived as a form of soft clustering, where input tokens are aligned with group tokens, leading to a shared or similar learned positional embedding. This methodology enables diverse group assignments across various channels and temporal segments when applied to multivariate timeseries, facilitating efficient grouping or segmentation through our approach.

We believe our results underscore the need for new methods to learn spatiotemporal structures that are distinct for each instance, and can be adapted to produce embeddings that are not tied to their absolute position in the sequence. This is particularly pertinent in multivariate timeseries data, where the significance of channels may fluctuate between different classes and across training and testing datasets. Consequently, we believe that both our group embedding approach and the GAFormer architecture present significant advancements, particularly for the timeseries analysis community, where standard Positional Embeddings (PEs) may prove inadequate.

Reviewer comment (Writing):
The explanation of the group embedding is vague. It is not easy to understand what the group embedding stands for.

Author response:
Thank you for raising this point about needing to improve the explanation behind group embeddings (GE). We plan to revise the introduction and methods to provide more intuition and explanation of why group embeddings are needed for multivariate timeseries.

Our approach is centered on developing data-adaptive spatiotemporal embeddings, which augment the conventional positional embeddings typically employed in sequential data processing. This is achieved by creating a compact set of universal embeddings that are dynamically combined, often through a sparse linear composition. This process results in customized positional embeddings, varying across different sequences, allowing the model to apply a "group embedding" either to specific channels or points in time. This can be perceived as a form of soft clustering, where input tokens are aligned with group tokens, leading to a shared or similar learned positional embedding. This methodology enables diverse group assignments across various channels and temporal segments when applied to multivariate timeseries, facilitating efficient grouping or segmentation through our approach.

We believe our results underscore the need for new methods to learn spatiotemporal structures that are distinct for each instance, and can be adapted to produce embeddings that are not tied to their absolute position in the sequence. This is particularly pertinent in multivariate timeseries data, where the significance of channels may fluctuate between different classes and across training and testing datasets. Consequently, we believe that both our group embedding approach and the GAFormer architecture present significant advancements, particularly for the timeseries analysis community, where standard Positional Embeddings (PEs) may prove inadequate.

Reviewer comment (Experiments):
The paper aims to learn generalizable representation within multivariate datasets. I did not find the related experiments to demonstrate the generalization ability.

Author response:
Thanks for your question. By generalization, we were referring to the model’s ability to perform well at test time on a new set of sequences which were not seen by the model in training time. In timeseries, the noise distribution and channel relevance can change abruptly and thus models that aren’t robust to these types of changes in the input may suffer at test time. We will make this clear in the revised paper. Thank you!

Reviewer comment (Reproducibility):
The details of transformer used seems to be missing. see questions

Author response:
**1. Details and specifications of our architecture**

Our GAFormer architecture is built based on a cascaded Spatial transformer and a Temporal transformer, with group embedding modules (SGE, TGE) added to both encoders. Specifically, our SGE/TGE module is typically a 3-layer, 8-head, 256-dim transformer with K=10 groups. Please refer to the updated Appendix for more details on the hyperparameters used in our experiments. We will also make sure to include these details in the revised paper. Thanks for your feedback!

Reviewer comment (Experiments):
Why does this model not compare to (Nie et al.), which seems to be an important related work.

Author response:
**2. Comparison to PatchTST**

While PatchTST has a similar channel separated encoder design, because it is designed for forecasting, we couldn’t directly report numbers from their work on the classification and regression tasks considered here. For our initial submission, we implemented a supervised variant of PatchTST [2] and reported those results in Table 2, where we demonstrate superior performance. Thanks to your comment, we ran additional experiments to also compare with supervised PatchTST for the univariate datasets in Table 1 and the neural datasets in Table 3 (see below). We find that across the board, GAFormer and group embeddings maintain their top performance and the patching-based model without group embeddings. We believe that these results help to further demonstrate the power of our approach.

Additional Baselines for Table 1:
| | InlineSkate  | Earthquakes | Adiac |
| -------- | ------- | -------- | ------- |
| PatchTST | 21.64 | 68.35 | 65.47 |
| PatchTST + TGE | 21.64 | 69.06 | 69.82 |
| $\Delta$ | $\uparrow$ 0.00 | $\uparrow$ 0.71 | $\uparrow$ 4.35 |

Additional Baselines for Table 3:

| | C-1  | C-2 | M-1 | M-2 | NLB-Maze | NLB-RTT |
| -------- | ------- | -------- | ------- | -------- | ------- | -------- |
| PatchTST (P=5) | 59.38 | 83.33 | 78.57 | 67.44 | 87.57 | 33.40 |
| PatchTST (P=1) | 78.13 | 86.11 | 88.10 | 72.09 | 88.19 | 43.16 |
| GAFormer (P=1) | 81.25 | 94.44 | 92.86 | 88.37 | 91.36 | 54.33 |

Reviewer comment (Reproducibility):
Does this model consisted of 2 layers of transformer? SGE and TGT?

Author response:
**1. Details and specifications of our architecture**

Our GAFormer architecture is built based on a cascaded Spatial transformer and a Temporal transformer, with group embedding modules (SGE, TGE) added to both encoders. Specifically, our SGE/TGE module is typically a 3-layer, 8-head, 256-dim transformer with K=10 groups. Please refer to the updated Appendix for more details on the hyperparameters used in our experiments. We will also make sure to include these details in the revised paper. Thanks for your feedback!

Reviewer comment (Reproducibility):
What's the specification of these transformers? It should be stated in the main paper.

Author response:
**1. Details and specifications of our architecture**

Our GAFormer architecture is built based on a cascaded Spatial transformer and a Temporal transformer, with group embedding modules (SGE, TGE) added to both encoders. Specifically, our SGE/TGE module is typically a 3-layer, 8-head, 256-dim transformer with K=10 groups. Please refer to the updated Appendix for more details on the hyperparameters used in our experiments. We will also make sure to include these details in the revised paper. Thanks for your feedback!