Paper: MoAT: Multi-Modal Augmented Time Series Forecasting (ICLR-2024)

Reviewer comment (Experiments):
As a single value at a specific timestamp provides little information, PatchTST and Crossformer patch time series to form informative tokens. Text data already contains a wealth of information, not to mention that there are multiple texts at each time step, so using patching is unreasonable here.

Author response:
**C1. Text already contains a wealth of information, so using patching is unreasonable here.**

As the reviewer commented, text provides sufficient information at each timestep while time-series data does not. However, we would like to emphasize the necessity of patching texts in our architecture. Specifically, we aggregate text embeddings within corresponding time intervals as the time series patches. This alignment is particularly crucial for feature-wise augmentation, where we concatenate temporally aligned time series and text patches as input for the shared multi-modal augmented encoder of MoAT.

Reviewer comment (Theory):
The decomposition of text data is unclear, particularly the definitions of "trend" and "season" for text. Equation (6) shows that the so-called trend-seasonal decomposition is just the same attention pooling with different sets of parameters, raising doubts about its ability to capture trend and seasonal dynamics as claimed.

Author response:
**C2. Decomposition of text data is unclear.**

We acknowledge the reviewer's comment and modified the attention pooling function for texts by using time series patches as query vectors during the attention computation for text aggregation. From the modification, texts are aggregated differently based on the seasonal and trend time sereis patches. For more details, please refer to the common response above.

Reviewer comment (Experiments):
Table 3 shows that the largest dataset, Bitcoin, contains only 741 * 4 = 2,964 scalars, while the smallest contains less than 1,000. This raises concerns about the suitability of training complex neural networks with such limited data.

Author response:
**C3. Table 3 shows that the largest dataset, Bitcoin, contains only 3K scalars. This raises concerns about the suitability of training complex neural networks with such limited data.**

As noted by the reviewer, the datasets we collected comprise a limited number of time series data points. To mitigate this scarcity in time series data, we employ multi-modal information, particularly leveraging the abundance of information within the text data. Specifically, text data contains a substantial amount of information that supplements the limited time series data. Moreover, the superiority of transformer-based models (e.g., PatchTST and Crossformer) over simpler linear models (e.g., DLinear and LightTS) suggests that complex neural networks like transformers indeed outperform simpler models within our datasets.

Reviewer comment (Experiments):
The main experiment focuses on an input8-output1 setting, with both input and output series being very short. While most selected baselines are for longer term forecasting, i.e. they perform at least input96-output96 task. So the comparison is not so fair.

Author response:
**C4. Both input and output series is very short. While most selected baselines are for longer term forecasting. So the comparison is not so fair. It is recommended to add traditional and straightforward models as baselines.**

While we incorporated basic linear models like LightTS and DLinear as baselines, we appreciate the reviewer’s feedback regarding the need for a fair comparison. We will take this into account and include baseline models to ensure a more comprehensive evaluation.

Reviewer comment (Experiments):
It is advisable to conduct experiments to validate the necessity of using patching for text data.

Author response:
**C1. Text already contains a wealth of information, so using patching is unreasonable here.**

As the reviewer commented, text provides sufficient information at each timestep while time-series data does not. However, we would like to emphasize the necessity of patching texts in our architecture. Specifically, we aggregate text embeddings within corresponding time intervals as the time series patches. This alignment is particularly crucial for feature-wise augmentation, where we concatenate temporally aligned time series and text patches as input for the shared multi-modal augmented encoder of MoAT.

Reviewer comment (Theory):
Could you clarify the meaning of "trend" and "season" in the context of text data? Additionally, please elaborate on how the pooling in Equation (6), without further constraints, extracts trend and season.

Author response:
**C2. Decomposition of text data is unclear.**

We acknowledge the reviewer's comment and modified the attention pooling function for texts by using time series patches as query vectors during the attention computation for text aggregation. From the modification, texts are aggregated differently based on the seasonal and trend time sereis patches. For more details, please refer to the common response above.

Reviewer comment (Experiments):
Given the small dataset size and short input/output lengths, it is recommended to add traditional and straightforward models as baselines, such as: 1) repeat last: just repeating the last timestamp's value $x_{L}$ as prediction; 2) Vector autoregressive moving average; 3) DeepAR.

Author response:
**C4. Both input and output series is very short. While most selected baselines are for longer term forecasting. So the comparison is not so fair. It is recommended to add traditional and straightforward models as baselines.**

While we incorporated basic linear models like LightTS and DLinear as baselines, we appreciate the reviewer’s feedback regarding the need for a fair comparison. We will take this into account and include baseline models to ensure a more comprehensive evaluation.

Reviewer comment (Theory):
It requires further clarification why the proposed textual decomposition components map to the trend and seasonality aspects of the time-series data.

Author response:
**C1. It requires further clarification why the proposed textual decomposition components map to the trend and seasonality aspects of the time-series data.**

We acknowledge the reviewer's comment and modified the attention pooling function for texts by using time series patches as query vectors during the attention computation for text aggregation. From the modification, texts are aggregated differently based on the seasonal and trend time sereis patches. For more details, please refer to the common response above.

Reviewer comment (Experiments):
The datasets primarily feature monthly or weekly sampling intervals. This raises concerns about the applicability of the approach to datasets with higher frequency sampling, such as hourly, where acquiring corresponding textual information could be challenging.

Author response:
**C3. The datasets primarily feature monthly or weekly sampling intervals. This raises concerns about the applicability of the approach to datasets with higher frequency sampling.**

As raised by the reviewer, testing on datasets with higher frequency is indeed an important consideration. However, we would like to emphasize that we collected six distinct datasets encompassing various sampling intervals, ranging from daily to monthly frequencies. In addition, elevating the sampling frequency can potentially introduce timesteps with missing text data, posing an additional challenge that requires careful attention. In our study, we assumed that at least one text is available at each timestep and leaved addressing this challenge as our future work.

Reviewer comment (Experiments):
Lack of experiments assessing whether the model performs well with scarce data, which is painted as the main motivation of MoAT. Furthermore, figure 5c does not seem to corroborate the story that MoAT performs significantly better than other methods with data scarcity (hard to say without variance). MoAT still seems to derive its main performance improvements from increasing the train ratio.

Author response:
**C1. Lack of experiments assessing whether the model performs well with scarce data.**

We would like to highlight that it is common in real-world scenarios to encounter time series datasets with a limited number of data samples or lengths, like the datasets collected for this study. Thus, our experimental results demonstrate the effectiveness of MoAT, in scenarios where datasets exhibit scarce time series information.

Reviewer comment (Presentation):
Lack of details in the caption of the T-SNE decomposition between time series and texts.

Author response:
**C2. Lack of details in the caption of the T-SNE decomposition between time series and texts.**

We apologize for the lack of clarity in the caption for Figure 4. The colors blue and green in the figure represent the trend and seasonal representations, respectively. In the figure, “O”s depict the four representations acquired through the encoder, while “X”s indicate the averaged value of these four representations. A visual observation reveals that utilizing the four representations as individual data samples, instead of merging them, expands the representation space, which potentially contributes to more accurate and robust time series forecasting.

Reviewer comment (Theory):
The remarks about information uniqueness of cross-modal vs unimodal representations are not backed up, no reason for their contained information to be unique.

Author response:
**C3. The remarks about the information uniqueness of cross-modal vs. uni-modal representations are not backed up, no reason for their contained information to be unique.**

We intended to convey that cross-modal and uni-modal representations are distinct from each other and may encompass different semantics, contributing to providing complementary information. An example is shown in Figure 1 (c) where time series and text present distinct information. We will refine this statement in the paper to avoid misunderstanding.

Reviewer comment (Experiments):
In fact, there is no comparison of model sizes and various scaling parameters for different methods. If you don't normalize, how do you know your performance increases aren't simply due to scaling up model size?

Author response:
**C5. There is no comparison of model sizes and various scaling parameters for different methods.**

We conducted experiments on MoAT using various dimensions and observed that it is not significantly affected by the number of parameters of the model. We will include more detailed results to support this finding.

Reviewer comment (Reproducibility):
How is text data decomposed into trend and seasonal components?

Author response:
**C4. Not obvious that the text data trend-seasonal decomposition actually decomposes into trend and seasonal data.**

We acknowledge the reviewer's comment and modified the attention pooling function for texts by using time series patches as query vectors during the attention computation for text aggregation. From the modification, texts are aggregated differently based on the seasonal and trend time sereis patches. For more details, please refer to the common response above.

Reviewer comment (Writing):
What does "(non-)overlapping patches" mean? Clearly patches are overlapping if they share S values.

Author response:
**C6. What does (non-)overlapping patches mean? Clearly patches are overlapping if they share S values.**

As the reviewer commented, patches are considered non-overlapping when S equals zero. We have phrased it as “(non-)overlapping” to highlight the flexibility in selecting S, which can be any non-negative integer smaller than the patch length.

Reviewer comment (Experiments):
Why are the texts unordered at each timestep? Is this a feature of the dataset used, or a design choice to ignore some of the granularity of the timestep?

Author response:
**C7. Why are the texts unordered at each timestep?**

At every timestep, there could exist multiple texts, which can indeed be temporally ordered, as noted by the reviewer. In our current approach, we focused on the temporal granularity of the time series and did not account for the specific ordering of texts within each timestep. Nevertheless, we find the reviewer’s suggestion to be an insightful perspective and acknowledge its potential direction for future work.

Reviewer comment (Experiments):
Is the forecasting in Figure 5 autoregressive? What are you providing as inputs at each time step? What are the document inputs when autoregressively predicting? The visualization in figure 5a is so zoomed out as to be uninformative (hard to tell the difference between the methods).

Author response:
**C9. Is the forecasting in Figure 5 autoregressive?**

The experiments are conducted in an autoregressive manner, as correctly mentioned by the reviewer.

Reviewer comment (Evaluation):
Any intuition as to why MoAT_{time} is the best performer on the Fuel dataset?

Author response:
**C11. Any intuition as to why MoAT_time is the best performer on the Fuel dataset?**

It is important to note that we fine-tuned each variant of MoAT, including MoAT_time, within the same hyperparameter search space. In the case of the Fuel dataset, text behaves somewhat like noise and does not contribute to performance improvement. MoAT is a more universally applicable configuration across all datasets, providing a more generalized approach.

Reviewer comment (Evaluation):
The ablations in table 2 suggest that the augmentations are not really helpful, considering the relative improvement compared to MoAT_{time} and MoAT_{text}. MoAT_{sample} seems pretty performant itself, being the best on Metal and Bitcoin, and second best on Fuel and Covid.

Author response:
**C12. The ablations in Table 2 suggest that the augmentation are not really helpful, considering the relative improvement compared to MoAT_time and MoAT_text.**

The comparison between MoAT, MoAT_time, and MoAT_text aims to highlight the effectiveness of leveraging multiple modalities for time series forecasting. Conversely, MoAT_sample and MoAT_feature are intended to demonstrate how employing both augmentation methods using these multi-modalities enhances performance. Therefore, these different variants are designed to confirm distinct aspects rather than competing with each other.

Reviewer comment (Experiments):
Ridge regression includes a loss penalty. What weight did you choose for this?

Author response:
**C13. Ridge regression includes a loss penalty. What weight did you choose for this?**

The weight for the regularization term in ridge regression is selected based on its performance evaluated on the validation set.

Reviewer comment (Novelty):
This paper is mostly applied and combines several existing techniques, e.g., patch-wise embedding, and pattern decomposition. The authors are expected to better position this work by clarifying the technical novelty, contribution, or new insights.

Author response:
**C1. This paper is mostly applied and combines several existing techniques, e.g., patch-wise embedding and pattern decomposition. The authors are expected to better position this work by clarifying the technical novelty, contribution, or new insights.**

We would like to emphasize the technical contributions and novel insights our work brings to both the multi-modal and time-series domains as follows:

- We have reformulated existing augmentation techniques within the multi-modal domain, categorizing them into sample-wise (modality-specific) and feature-wise (modality-fused) augmentations. This configuration of multi-modal augmentation provides novel and structured viewpoints on how multiple modalities can be effectively fused.
- Surprisingly, despite its potential, there was a lack of benchmark datasets incorporating both time series and text (or other modalities). Thus, we gathered six real-world datasets, and we plan to make them publicly accessible. We strongly believe these datasets will serve as valuable resources for future researchers in this field. 
- We proposed MoAT, a time-series forecasting method that leverages multi-modal information. Specifically, MoAT consists of a multi-modal augmented encoder, which benefits from two distinct augmentations applied to both time series and text inputs. Note that most existing multi-modal approaches have typically utilized only one of these augmentation methods. 
- As the reviewer commented, patching and decomposition are existing techniques in time series forecasting. However, we would like to emphasize that, to the best of our knowledge, no prior work has combined both patching and decomposition for time series forecasting. Our approach, integrating these techniques, has notably improved accuracy.

In summary, we would like to emphasize that our work makes substantial contributions to (1) redefining methodologies, (2) introducing new datasets, and (3) presenting technically novel methods. As the reviewer suggested, we will emphasize these aspects in the revised version.

Reviewer comment (Evaluation):
The evaluation is mostly on financial datasets. But for finance, the error metric MSE is not the main interest, since in the real world the prediction is to serve downstream tasks, e.g., portfolio construction, risk management, etc, and practically MSE does not directly translate to the improvement for downstream tasks. It would be better to show the prediction by the proposed method can facilitate an example downstream task. e.g., portfolio construction is commonly used.

Author response:
**C2. The evaluation is mostly on financial datasets. But for finance, the error metric MSE is not the main interest, since in the real-world, the prediction is to serve downstream tasks, e.g., portfolio construction, risk management, etc, and practically MSE does not directly translate to the improvement for downstream tasks.**

The reviewer’s perspective is interesting, and we recognize the relevance of various downstream tasks tailored specifically for financial datasets. Nevertheless, we would like to emphasize that MoAT is a versatile time series forecasting framework that leverages multi-modal information. Thus, our evaluation primarily focused on measuring performance through MSE, which is a conventional metric used to assess the accuracy of the model.

Reviewer comment (Writing):
On page 5, the part "multi-modal augmented encoder" is essentially a combination of time series and text modalities, and what does the "augmented" refer to?

Author response:
**C3. On page 5, the part “multi-modal augmented encoder” is essentially a combination of time series and text modalities, and what does the “augmented” refer to?**

In Section 2, we restructured two orthogonal approaches for augmenting multi-modal data. Figure 3 (a) illustrates the multi-modal encoder in MoAT, which utilizes both augmentation methods. This integration allows MoAT to leverage both modality-specific and modality-fused information extractable from time series and text data.

Reviewer comment (Theory):
On page 6, in the part "joint trend-seasonal decomposition", trend-seasonal decomposition is reasonable for time series because of the underlying generative process. But, for text data, especially the embedding of news text, applying trend-seasonal decomposition is not intuitively understandable. News content is highly dependent on real-world happenings where the trend-seasonal decomposition is not necessarily existent.

Author response:
**C4. For text data, especially the embedding of news text, applying trend-seasonal decomposition is not intuitively understandable.**

We acknowledge the reviewer's comment and modified the attention pooling function for texts by using time series patches as query vectors during the attention computation for text aggregation. For more details, please refer to the common response above.

Reviewer comment (Evaluation):
In Table 1, MSE are mostly low-magnitude values, while the real price of the finance assets of the data used in the experiment differs greatly in magnitudes. Is this due to some data standardization? If so, it is better to report the errors in the original scale of data, because the prediction on the standardized domain would be less useful for downstream tasks in many cases.

Author response:
**C5. In Table 1, MSE are mostly low-magnitude values, while the real price of the finance assets of the data used in the experiment differs greatly in magnitudes. It is better to report the errors in the original scale of the data, because the prediction on the standardized domain would be less useful for downstream tasks in many cases.**

In line with existing practices in time-series forecasting research, we standardized the entire time series dataset using the training set and conducted training and evaluation accordingly. Although the standardized values may vary in magnitudes, it is important to note that the relative rankings among different methods remain consistent. In addition, it is straightforward to revert to the original scales through de-standardization using the mean and standard deviation obtained for standardization.