Paper: Modeling Time Series as Text Sequence A Frequency-vectorization Transformer for Time Series Forecasting (ICLR-2024)

Reviewer comment (Theory):
Because the frequencies of the DFT basis depend on sequence length, my main concern is that the tokenization process is highly dependent on the length of the input sequence $L$: (1) The size of token dictionaries, $G, H \in \mathbb{R}^{L\times P}$ vary with the input sequence length $L$. (2) The correspondence between keys in the dictionary and their frequencies also changes with varying sequence lengths: when embedding a sequence of length 10, $g_5, k_5$ in the fixed dictionary correspond to frequency $\frac{1}{2}$, i.e. period of $2$. While for sequence of length 20, they correspond to frequency $\frac{1}{4}$, i.e. period of $4$. Therefore, the tokenization highly depends on the sequence length and we need a specific dictionary for each different length, even on the same data set. This limits the generalization of the proposed model.

Author response:
Q1: Dependency on sequence length. A very insightful observation. Due to the limitations of DFT, we acknowledge that our tokenization method is highly correlated with input length. However, this does not imply weak generalization of our method. On the one hand, upon careful examination of the expressions for G and H, we can find that regardless of changes in input length, as long as the embedding dimensions remain constant, the basis vectors at each position in G and H will not change. This ensures the generalization of token dictionaries. On the other hand, any time series can be partitioned into fixed-length segments. For instance, a time series of length 200 can be divided into 105 subsequences of length 96 (105 = 200 - 96 + 1), and these subsequences can be sampled at regular intervals for prediction. The average results from these predictions can be considered as the final prediction result. This approach allows for extension to time series tasks of arbitrary lengths.

Reviewer comment (Evaluation):
The tokenization should be orthogonal to the pretraining task. However, it seems that FreqTST highly relies on pretraining to achieve SOTA performance.

Author response:
Q2: Relation between tokenization and pretraining task. There is probably some misunderstanding here. Firstly, both the TSR and MPP pretraining tasks can be trained using the time series itself and the position of the maximum value, independent of specific tokenization. Secondly, in the main experiment, FreqTST achieves state-of-the-art results without pretraining tasks.

Reviewer comment (Experiments):
The paper uses a fixed input length of 96 in the main experiments, while the forecasting lengths are substantially larger (e.g., 336, 720). This design choice may not be realistic. Moreover, PatchTST and DLinear can get better forecasting accuracy with longer look-back window.

Author response:
Q3: Fixed input length. We fixed the input length to 96 because we aimed to model scenarios with missing data, such as in medical contexts, which are common occurrences.

Reviewer comment (Experiments):
The claim of zero-shot transferability might be overstated. In section 4.3.2, the model is trained on Electricity and tested on ETTh2 to show its transferability. However, both these two datasets are hourly-sampled and have a period of 24 hours. I infer that this pre-trained model is unlikely to achieve satisfactory results on weather as it is sampled every 10 minutes. Thus "These results indicate our model has capable transfer ability due to the unified frequency units and common dictionary among all kinds of time series" seems to be overclaimed.

Author response:
Q4: Completion of zero-shot experiments. It is very kind of you to point out this problem. In fact, we conduct more experiments but only represent one sample in our paper. Here are more results that show the powerful transferibility of FreqTST. 
|        Models       | Dataset | Weather |       |       |       | Ettm2 |       |       |       | Etth2 |       |       |       |
|:-------------------:|:-------:|:-------:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|
|                     |  Metric |    96   |  192  |  336  |  720  |   96  |  192  |  336  |  720  |   96  |  192  |  336  |  720  |
|  PatchTST_ sup_zero |   MSE   | 0.232   | 0.276 | 0.338 | 0.418 | 0.173 | 0.193 | 0.236 | 0.286 | 0.201 | 0.243 | 0.275 | 0.354 |
|                     |   MAE   | 0.273   | 0.306 | 0.350 | 0.402 | 0.294 | 0.309 | 0.342 | 0.373 | 0.302 | 0.333 | 0.359 | 0.416 |
| PatchTST_ self_zero |   MSE   | 0.234   | 0.280 | 0.329 | 0.397 | 0.170 | 0.190 | 0.220 | 0.275 | 0.248 | 0.276 | 0.304 | 0.360 |
|                     |   MAE   | 0.281   | 0.314 | 0.344 | 0.385 | 0.290 | 0.304 | 0.325 | 0.361 | 0.348 | 0.366 | 0.384 | 0.420 |
|  FreqTST_ sup_zero  |   MSE   | 0.204   | 0.249 | 0.304 | 0.392 | 0.177 | 0.206 | 0.276 | 0.405 | 0.203 | 0.237 | 0.271 | 0.361 |
|                     |   MAE   | 0.258   | 0.293 | 0.334 | 0.396 | 0.297 | 0.322 | 0.375 | 0.455 | 0.305 | 0.334 | 0.366 | 0.432 |
|  FreqTST_ self_zero |   MSE   | 0.247   | 0.264 | 0.328 | 0.373 | 0.142 | 0.183 | 0.205 | 0.258 | 0.209 | 0.235 | 0.264 | 0.333 |
|                     |   MAE   | 0.271   | 0.290 | 0.335 | 0.362 | 0.257 | 0.291 | 0.310 | 0.346 | 0.312 | 0.330 | 0.352 | 0.402 |

Reviewer comment (Experiments):
Section 4.1 mentions experiments conducted on the ETTm2 dataset, but the results are missing. Can you provide the outcomes for this dataset?

Author response:
Q5: Missing ETTm2 dataset. We use the ETTm2 dataset to evaluate zero-shot capacity of FreqTST, which is demonstrated by the table in Q4. We also evaluate main models on ETTm2, and the results are below.
|      Models     | Metric |   96  |  192  |  336  |  720  |
|:---------------:|:------:|:-----:|:-----:|:-----:|:-----:|
|  FreqTST_ self  |   MSE  | 0.119 | 0.150 | 0.190 | 0.245 |
|                 |   MAE  | 0.231 | 0.262 | 0.293 | 0.333 |
|   FreqTST_ sup  |   MSE  | 0.119 | 0.154 | 0.190 | 0.243 |
|                 |   MAE  | 0.233 | 0.269 | 0.301 | 0.340 |
| PatchTST_ self_ |   MSE  | 0.120 | 0.147 | 0.180 | 0.238 |
|                 |   MAE  | 0.231 | 0.260 | 0.286 | 0.331 |

Reviewer comment (Theory):
Why MSE was chosen instead of cross-entropy in Maximum Position Prediction task, considering this is a classification task and the target is a one-hot vector?

Author response:
Q6: Cross entropy for MPP task. Thanks for your valuable suggestion. We conducted validation experiments using the cross entropy loss and found that the results were not as good as those using MSE. We think this is because the pre-training loss function is the sum of the loss functions from the two pre-training tasks. If the loss functions of the two pre-training tasks are different, the optimization space of the entire loss function will be more rugged, making it difficult to converge to the global optimal solution.

Reviewer comment (Experiments):
Please compare FreqTST with DLinear and PatchTST with varying look-back window length $L$.

Author response:
Q7: Results with varying length L. As is mentioned in Q3, we focus on the scene that specializes in short input, so we do not include this experiment due to the limitation of time and compute resources. We will do our best to do this experiment in the near future.

Reviewer comment (Novelty):
The claim in the third paragraph of the introduction that "... transformer architectures cannot be directly applied to model time series data due to the lack of a clear tokenization scheme" seems unsubstantiated. Recent studies, such as [1], have showcased the feasibility of such an application.

Author response:
Q1: Overclaimed. Thank you for the reminder. After carefully reading the paper you recommended (sorry we did not notice this study previously), we found that the claims in our paper are not proper, and we will revise  by "... effectively applied to model time time series data". At the same time, we would like to emphasize as the counting ability of LLM is poor and the experimental results are quite mediocre, this work does not impact the value of our work.

Reviewer comment (Presentation):
In Fig 1, the demarcation between pre-training and downstream adaptation is not clearly delineated. This aspect remains vague in Sec 3, and one has to refer to the discourse in Sec 4.2 for clarity. For instance, the statement "the difference between fine-tuning and supervised learning is that we load ..." in Sec 3.5 appears ambiguous. Furthermore, the utilization of L_pre and L in Sec 3 isn't elaborated upon.

Author response:
Q2: Vagueness between pre-training and downstream adaption. There is probably some misunderstanding here. In Fig 1, we did not intend to introduce pretraining tasks and downstream tasks; rather, we aimed to showcase the motivation behind our model design and its core modeling ideas. However, there is a clear boundary between pretraining tasks and downstream tasks in Fig 2. Additionally, in the first paragraph of Sec 3, we provided a detailed definition of downstream tasks, complementing the discussion of pretraining tasks in Sec 3.4. The confusion may arise from the similarity between fine-tuning and supervised learning, as we conducted fine-tuning directly on downstream tasks. Therefore, the processes of fine-tuning and supervised learning are identical, with the distinction being that fine-tuning is the next stage after pretraining, while supervised learning is a complete training process. Besides, L is defined in the first paragraph of Sec 3 when L_pre was not used in Sec 3.

Reviewer comment (Theory):
The rationale for introducing G and H in Sec 3.2.2 is unclear. The link between positional encoding (specifically, Eq. 4 and 5) and the formulation of G and H requires more substantial justification.

Author response:
Q3: Motivation for frequency token formulation. The design principles for G and H align with the design principles of position encoding in Transformer. Firstly, each frequency position needs a unique and deterministic representation. Secondly, the representation distance between any two positions should be consistent across time series. Lastly, this representation should easily generalize to longer sequences with bounded values, as time series can be very long. Based on these principles, we constructed two frequency vector spaces, where G and H represent the basis vector of real and imaginary domains, respectively. After applying FFT to a time series, the coefficients obtained in the real and imaginary domains serve as coordinates for the basis of the two vector spaces. Finally, we use a linear mapping matrix to transform vectors from the imaginary domain to the real domain, and then sum the two, resulting in the final vector in the frequency space. This allows us to represent a time series as vectors in a frequency vector space.

Reviewer comment (Experiments):
Conducting evaluations on a broader set of datasets, for instance, the remaining three ETT datasets and M4, would enhance the robustness of this work.

Author response:
Q4: Limited Dataset. To address your concern, we have tried our best to add the required results. The table below presents the experimental results of our model on the Ettm2 dataset.  
|      Models     | Metric |   96  |  192  |  336  |  720  |  
|:---------------:|:------:|:-----:|:-----:|:-----:|:-----:|  
|  FreqTST_ self  |   MSE  | 0.119 | 0.150 | 0.190 | 0.245 |  
|                 |   MAE  | 0.231 | 0.262 | 0.293 | 0.333 |  
|   FreqTST_ sup  |   MSE  | 0.119 | 0.154 | 0.190 | 0.243 |  
|                 |   MAE  | 0.233 | 0.269 | 0.301 | 0.340 |  
| PatchTST_ self_ |   MSE  | 0.120 | 0.147 | 0.180 | 0.238 |  
|                 |   MAE  | 0.231 | 0.260 | 0.286 | 0.331 |

Reviewer comment (Writing):
The writing of the paper is overall quite poor with respect to the use of English and Latex, the clarity, and the way information is and is not provided. If the authors are not native English speakers I recommend asking a native English speaker to provide more in-depth feedback. I have also found that software like grammerly is very helpful.

Author response:
Thanks for your insightful feedback, which is valuable and very helpful for improving our paper. 

Q1: Writing Problems. Thank you very much for your meticulous reading and pointing out the various language errors in the paper. We will modify them and conduct thorough proofreading for our paper.

Reviewer comment (Writing):
The author uses words that do not exist in English (cosinuid should be cosine), and employs similar-sounding words that have completely different meanings (discretion is to speak in a way to not annoy or cause offense).

Author response:
Thanks for your insightful feedback, which is valuable and very helpful for improving our paper. 

Q1: Writing Problems. Thank you very much for your meticulous reading and pointing out the various language errors in the paper. We will modify them and conduct thorough proofreading for our paper.

Reviewer comment (Writing):
This paper's grammar needs improvements, as exemplified by: "we discrete the time series into frequency units" and "It represents we use L sinusoid."

Author response:
Thanks for your insightful feedback, which is valuable and very helpful for improving our paper. 

Q1: Writing Problems. Thank you very much for your meticulous reading and pointing out the various language errors in the paper. We will modify them and conduct thorough proofreading for our paper.

Reviewer comment (Writing):
The biggest issue with language is that this work is extremely vague which makes the arguments very weak and unconvincing. This vagueness at times leaves me to wonder how well the authors understand what they are saying. This is unfortunate because the authors may be experts with intimate knowledge of this subject, but their language does not always portray this. By detailing their ideas with more precise language the authors can write a significantly more compelling study that will also demonstrate the authors' expertise in the field. For example, in “Transformer-based models achieve better performances in long-term time series forecasting than previous models, so more and more transformer-based models are designed for better performance” — what about transformer models achieve better performance and why do people want to use them? Saying that "more and more" models are being made adds no information and sounds too colloquial. In “Another research trend is the utilization of the frequency domain in forecasting, as time series in the time domain can be transformed into the frequency domain. An intuitive way is to use the frequency domain to extract more features” — "time series in the time domain" is redundant, and “extract more features” is vague and misleading about the Fourier transform’s capabilities.

Author response:
Thanks for your insightful feedback, which is valuable and very helpful for improving our paper. 

Q1: Writing Problems. Thank you very much for your meticulous reading and pointing out the various language errors in the paper. We will modify them and conduct thorough proofreading for our paper.

Reviewer comment (Presentation):
Figure 3 is hard to understand because the values look the same. Only a reader with very good eyes and scale judgment can quantify the differences. This information should be put in a table.

Author response:
Q2: Overclaimed problems. (1) We conclude that "Firstly, our FreqTST outperforms all other supervised baseline models, including PatchTST and other frequency-based models, which verifies the effectiveness of our model" means compare FreqTST with other baseline excluding PatchTST with pretraining, which can be understood clearly with the last sentence in implementation details. (2) Figure 4 will be replaced by a more informative table with results on different datasets, which can validate the claim clearly in our paper. (3) The words "comprehensive" and "superior" does overclaim, and we will revise them by "amounts of" and "promising". We will also correct similar errors in the paper, such as instances of "outperform" in certain places. (4) There is probably some misunderstanding in the statement of "they are numerical values that cannot be tokenized", where we mean a single number in time series can not be tokenized as it contains no information alone. However, if a single number represents a frequency of a time series, it obtains semantic information so that it can be tokenized into a vector space. (5) In most scenarios, our method outperforms other baseline by a large margin, for example, in long-term prediction on Etth2 dataset, our method get an impressive results with a average decease of 6%.

Reviewer comment (Evaluation):
Many claims and statements about other works are overembellished and unsupported, sometimes contradicting the paper's own results. For example, “Firstly, our FreqTST outperforms all other supervised baseline models, including PatchTST and other frequency-based models, which verifies the effectiveness of our model” is false, since pretrained PatchTST sometimes outperforms the proposed model, as shown in the results.

Author response:
Q2: Overclaimed problems. (1) We conclude that "Firstly, our FreqTST outperforms all other supervised baseline models, including PatchTST and other frequency-based models, which verifies the effectiveness of our model" means compare FreqTST with other baseline excluding PatchTST with pretraining, which can be understood clearly with the last sentence in implementation details. (2) Figure 4 will be replaced by a more informative table with results on different datasets, which can validate the claim clearly in our paper. (3) The words "comprehensive" and "superior" does overclaim, and we will revise them by "amounts of" and "promising". We will also correct similar errors in the paper, such as instances of "outperform" in certain places. (4) There is probably some misunderstanding in the statement of "they are numerical values that cannot be tokenized", where we mean a single number in time series can not be tokenized as it contains no information alone. However, if a single number represents a frequency of a time series, it obtains semantic information so that it can be tokenized into a vector space. (5) In most scenarios, our method outperforms other baseline by a large margin, for example, in long-term prediction on Etth2 dataset, our method get an impressive results with a average decease of 6%.

Reviewer comment (Evaluation):
The claim that “Both FreqTST and FreqTST$_{pretraining}$ show more promising performances on the zero-shot setting than PatchTST, which even outperform some supervised models” is misleading because the proposed model only outperforms in some cases, and this should be stated clearly along with the magnitude of differences.

Author response:
Q2: Overclaimed problems. (1) We conclude that "Firstly, our FreqTST outperforms all other supervised baseline models, including PatchTST and other frequency-based models, which verifies the effectiveness of our model" means compare FreqTST with other baseline excluding PatchTST with pretraining, which can be understood clearly with the last sentence in implementation details. (2) Figure 4 will be replaced by a more informative table with results on different datasets, which can validate the claim clearly in our paper. (3) The words "comprehensive" and "superior" does overclaim, and we will revise them by "amounts of" and "promising". We will also correct similar errors in the paper, such as instances of "outperform" in certain places. (4) There is probably some misunderstanding in the statement of "they are numerical values that cannot be tokenized", where we mean a single number in time series can not be tokenized as it contains no information alone. However, if a single number represents a frequency of a time series, it obtains semantic information so that it can be tokenized into a vector space. (5) In most scenarios, our method outperforms other baseline by a large margin, for example, in long-term prediction on Etth2 dataset, our method get an impressive results with a average decease of 6%.

Reviewer comment (Experiments):
The statement “Through comprehensive experiments conducted on multiple datasets, we demonstrate superior performance ...” is inaccurate because the experiments are not comprehensive (e.g., missing datasets from Nie et al.), and the model is not regularly the best performer in Tables 2 and 3.

Author response:
Q2: Overclaimed problems. (1) We conclude that "Firstly, our FreqTST outperforms all other supervised baseline models, including PatchTST and other frequency-based models, which verifies the effectiveness of our model" means compare FreqTST with other baseline excluding PatchTST with pretraining, which can be understood clearly with the last sentence in implementation details. (2) Figure 4 will be replaced by a more informative table with results on different datasets, which can validate the claim clearly in our paper. (3) The words "comprehensive" and "superior" does overclaim, and we will revise them by "amounts of" and "promising". We will also correct similar errors in the paper, such as instances of "outperform" in certain places. (4) There is probably some misunderstanding in the statement of "they are numerical values that cannot be tokenized", where we mean a single number in time series can not be tokenized as it contains no information alone. However, if a single number represents a frequency of a time series, it obtains semantic information so that it can be tokenized into a vector space. (5) In most scenarios, our method outperforms other baseline by a large margin, for example, in long-term prediction on Etth2 dataset, our method get an impressive results with a average decease of 6%.

Reviewer comment (Evaluation):
The claim “The architecture makes full use of the modeling capacity of transformers for discrete and semantic token sequence, and outperforms other supervised models by a large margin” is unsupported and vague, with incorrect generalizations about performance.

Author response:
Q2: Overclaimed problems. (1) We conclude that "Firstly, our FreqTST outperforms all other supervised baseline models, including PatchTST and other frequency-based models, which verifies the effectiveness of our model" means compare FreqTST with other baseline excluding PatchTST with pretraining, which can be understood clearly with the last sentence in implementation details. (2) Figure 4 will be replaced by a more informative table with results on different datasets, which can validate the claim clearly in our paper. (3) The words "comprehensive" and "superior" does overclaim, and we will revise them by "amounts of" and "promising". We will also correct similar errors in the paper, such as instances of "outperform" in certain places. (4) There is probably some misunderstanding in the statement of "they are numerical values that cannot be tokenized", where we mean a single number in time series can not be tokenized as it contains no information alone. However, if a single number represents a frequency of a time series, it obtains semantic information so that it can be tokenized into a vector space. (5) In most scenarios, our method outperforms other baseline by a large margin, for example, in long-term prediction on Etth2 dataset, our method get an impressive results with a average decease of 6%.

Reviewer comment (Writing):
In the abstract, the statement “Transformer-based models … fail to understand time series sequences … numerical values that cannot be tokenized” contradicts the paper’s premise of tokenizing numerical values and conflicts with claims about the method’s performance.

Author response:
Q2: Overclaimed problems. (1) We conclude that "Firstly, our FreqTST outperforms all other supervised baseline models, including PatchTST and other frequency-based models, which verifies the effectiveness of our model" means compare FreqTST with other baseline excluding PatchTST with pretraining, which can be understood clearly with the last sentence in implementation details. (2) Figure 4 will be replaced by a more informative table with results on different datasets, which can validate the claim clearly in our paper. (3) The words "comprehensive" and "superior" does overclaim, and we will revise them by "amounts of" and "promising". We will also correct similar errors in the paper, such as instances of "outperform" in certain places. (4) There is probably some misunderstanding in the statement of "they are numerical values that cannot be tokenized", where we mean a single number in time series can not be tokenized as it contains no information alone. However, if a single number represents a frequency of a time series, it obtains semantic information so that it can be tokenized into a vector space. (5) In most scenarios, our method outperforms other baseline by a large margin, for example, in long-term prediction on Etth2 dataset, our method get an impressive results with a average decease of 6%.

Reviewer comment (Experiments):
The paper omits results on several benchmark datasets used in PatchTST (Nie et al., 2023), preventing a comprehensive comparison.

Author response:
Q3: Limited Dataset. To address your concern, we have tried our best to add the required results. The table below presents the experimental results of our model on the Ettm2 dataset.
|      Models     | Metric |   96  |  192  |  336  |  720  |
|:-:|:-:|:-:|:-:|:-:|:-:|
|  FreqTST_ self  |   MSE  | 0.119 | 0.150 | 0.190 | 0.245 |
|                 |   MAE  | 0.231 | 0.262 | 0.293 | 0.333 |
|   FreqTST_ sup  |   MSE  | 0.119 | 0.154 | 0.190 | 0.243 |
|                 |   MAE  | 0.233 | 0.269 | 0.301 | 0.340 |
| PatchTST_ self_ |   MSE  | 0.120 | 0.147 | 0.180 | 0.238 |
|                 |   MAE  | 0.231 | 0.260 | 0.286 | 0.331 |

Reviewer comment (Theory):
The claim that the Fourier dictionary tokenization method can be applied to any time series is not entirely true because the number of Fourier coefficients depends on the sequence length, which affects orthogonality and results when applied to sequences of different lengths than the training data.

Author response:
Q6: Dependency on sequence length. A very insightful observation. Due to the limitations of DFT, we acknowledge that our tokenization method is highly correlated with input length. However, this does not imply weak generalization of our method. On the one hand, upon careful examination of the expressions for G and H, we can find that regardless of changes in input length, as long as the embedding dimensions remain constant, the basis vectors at each position in G and H will not change. This ensures the generalization of token dictionaries. On the other hand, any time series can be partitioned into fixed-length segments. For instance, a time series of length 200 can be divided into 105 subsequences of length 96 (105 = 200 - 96 + 1), and these subsequences can be sampled at regular intervals for prediction. The average results from these predictions can be considered as the final prediction result. This approach allows for extension to time series tasks of arbitrary lengths.

Reviewer comment (Writing):
The reason for choosing to represent the Fourier embeddings with Eqs. 4 and 5 is unclear, including whether $F$ is meant to be $G$ and the second $G$ is meant to be $H$.

Author response:
Q4: Linear mapping matrix. We constructed two frequency vector spaces, where G and H represent the basis vector of real and imaginary domains, respectively. After applying FFT to a time series, the coefficients obtained in the real and imaginary domains serve as coordinates for the basis of the two vector spaces. Finally, we use a linear mapping matrix to transform vectors from the imaginary domain to the real domain, and then sum the two, resulting in the final vector in the frequency space. This allows us to represent a time series as vectors in a frequency vector space.

Reviewer comment (Experiments):
The model should be evaluated on all datasets in Nie et al., 2023, to allow a complete and fair comparison.

Author response:
Q3: Limited Dataset. To address your concern, we have tried our best to add the required results. The table below presents the experimental results of our model on the Ettm2 dataset.
|      Models     | Metric |   96  |  192  |  336  |  720  |
|:-:|:-:|:-:|:-:|:-:|:-:|
|  FreqTST_ self  |   MSE  | 0.119 | 0.150 | 0.190 | 0.245 |
|                 |   MAE  | 0.231 | 0.262 | 0.293 | 0.333 |
|   FreqTST_ sup  |   MSE  | 0.119 | 0.154 | 0.190 | 0.243 |
|                 |   MAE  | 0.233 | 0.269 | 0.301 | 0.340 |
| PatchTST_ self_ |   MSE  | 0.120 | 0.147 | 0.180 | 0.238 |
|                 |   MAE  | 0.231 | 0.260 | 0.286 | 0.331 |

Reviewer comment (Novelty):
The overall novelty of the proposed method is limited. Frequency domain has been widely used in deep learning methods for time series analysis. Although the authors claim they are the first to tokenize continuous time series with frequency domain, there is no fundamental difference between this tokenization method and previous work which utilize FFT to conduct feature extraction.

Author response:
Q1: Noverty of FreqTST. There are some fundamental differences between previous work and FreqTST. Firstly, we propose the concept of frequency token for the first time, which maps the frequency value into a high dimension vector. Secondly, we consider frequency spectrum as a common dictionary for all time series and it can model various time series in a unfied way, while previous work only extract features on single dataset. Thirdly, all previous frequency-based methods transform signals into temporal domain after extracting features in frequency domain. At the same time, FreqTST uses frequency-tokenised representation to predict future values directly, filling the gap between frequency domain and temporal domain.

Reviewer comment (Experiments):
No comparison was made with the latest methods that utilize frequency domain information or employ Transformer structures, e.g., TimesNet and CrossFormer.

Author response:
Q2: Comparison with latest model. PatchTST(ICLR 2023) and DLinear(AAAI 2023) are contemporaneous works with Timesnet(ICLR 2023) and Crossformer(ICLR 2023), so the first two models involved in our paper can serve as representatives of the latter two models.

Reviewer comment (Experiments):
The setting of zero-shot transfer experiment is quite limited. The author only gives results on a set of pre-training and transferred data sets. And the performance of FreqTST for zero-shot learning can be over-claimed.

Author response:
Q3: Completion of zero-shot experiments. It is very kind of you to point out this problem. In fact, we conduct extra experiments but only show one sample in our paper. Here are more results that show the powerful transferibility of FreqTST.  
|        Models       | Dataset | Weather |       |       |       | Ettm2 |       |       |       | Etth2 |       |       |       |  
|:--:|:-:|:--:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|  
|                     |  Metric |    96   |  192  |  336  |  720  |   96  |  192  |  336  |  720  |   96  |  192  |  336  |  720  |  
|  PatchTST_ sup_zero |   MSE   | 0.232   | 0.276 | 0.338 | 0.418 | 0.173 | 0.193 | 0.236 | 0.286 | 0.201 | 0.243 | 0.275 | 0.354 |  
|                     |   MAE   | 0.273   | 0.306 | 0.350 | 0.402 | 0.294 | 0.309 | 0.342 | 0.373 | 0.302 | 0.333 | 0.359 | 0.416 |  
| PatchTST_ self_zero |   MSE   | 0.234   | 0.280 | 0.329 | 0.397 | 0.170 | 0.190 | 0.220 | 0.275 | 0.248 | 0.276 | 0.304 | 0.360 |  
|                     |   MAE   | 0.281   | 0.314 | 0.344 | 0.385 | 0.290 | 0.304 | 0.325 | 0.361 | 0.348 | 0.366 | 0.384 | 0.420 |  
|  FreqTST_ sup_zero  |   MSE   | 0.204   | 0.249 | 0.304 | 0.392 | 0.177 | 0.206 | 0.276 | 0.405 | 0.203 | 0.237 | 0.271 | 0.361 |  
|                     |   MAE   | 0.258   | 0.293 | 0.334 | 0.396 | 0.297 | 0.322 | 0.375 | 0.455 | 0.305 | 0.334 | 0.366 | 0.432 |  
|  FreqTST_ self_zero |   MSE   | 0.247   | 0.264 | 0.328 | 0.373 | 0.142 | 0.183 | 0.205 | 0.258 | 0.209 | 0.235 | 0.264 | 0.333 |  
|                     |   MAE   | 0.271   | 0.290 | 0.335 | 0.362 | 0.257 | 0.291 | 0.310 | 0.346 | 0.312 | 0.330 | 0.352 | 0.402 |

Reviewer comment (Experiments):
The author doesn't seem to have conducted multiple repeat experiments. The standard deviation of the results is not given in the text, which reduces the persuasiveness of the results.

Author response:
Q4: Multiple repeat Experiments. In the field of deep learning, repeated experiments are indeed essential. However, many time-series-related works presented at ICLR 2023 (Timesnet, Crossformer) did also not report the results of repeated experiments and their standard deviation. This is because the experimental results in this field are relatively stable with strong reproducibility. Below is a set of repeated experiments from our experimental process, showing that there is almost no difference in the results at this level of precision.  
| weather |   96  |  192  |  336  |  720  |  
|:-:|:-:|:-:|:-:|:-:|  
| result1 | 0.175 | 0.221 | 0.277 | 0.356 |  
|         | 0.215 | 0.255 | 0.296 | 0.347 |  
| result2 | 0.174 | 0.221 | 0.277 | 0.355 |  
|         | 0.214 | 0.255 | 0.296 | 0.347 |

Reviewer comment (Experiments):
Why not use cross entropy loss for MPP task, which is more appropriate as it is a classification task.

Author response:
Q5: Cross entropy for MPP task. Thanks for your valuable suggestion. We conducted validation experiments using the cross entropy loss and found that the results were not as good as those using MSE. We think this is because the pre-training loss function is the sum of the loss functions from the two pre-training tasks. If the loss functions of the two pre-training tasks are different, the optimization space of the entire loss function will be more rugged, making it difficult to converge to the optimal solution.  
|        Models       | Dataset | Weather |       |       |       | Etth2 |       |       |       |  
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|  
|                     |  Metric |    96   |  192  |  336  |  720  |   96  |  192  |  336  |  720  |  
|     FreqTST_ sup    |   MSE   |  0.169  | 0.210 | 0.258 | 0.326 | 0.185 | 0.225 | 0.236 | 0.291 |  
|                     |   MAE   |  0.229  | 0.267 | 0.307 | 0.358 | 0.294 | 0.327 | 0.338 | 0.381 |  
|    FreqTST_ self    |   MSE   |  0.178  | 0.223 | 0.281 | 0.356 | 0.184 | 0.220 | 0.250 | 0.313 |  
|                     |   MAE   |  0.220  | 0.256 | 0.299 | 0.347 | 0.291 | 0.319 | 0.344 | 0.390 |  
| FreqTST_ self_cross |   MSE   |  0.183  | 0.223 | 0.279 | 0.356 | 0.181 | 0.220 | 0.253 | 0.321 |  
|                     |   MAE   |  0.223  | 0.256 | 0.297 | 0.348 | 0.287 | 0.319 | 0.346 | 0.394 |

Reviewer comment (Theory):
You mentioned FreqTST have lower computation costs than PatchTST, however theoretically, FreqTST applies self-attention on a longer sequence (L / 2) than PatchTST does (L / P). Why PatchTST can not be applied on some of the long-term forecasting tasks while FreqTST could?

Author response:
Q6: Long-term prediction of PatchTST. $PatchTST_{pretraining}$ can not afford to predict long-term time series as it has a huge linear layer. However, FreqTST in supervised costs less in memory.

Reviewer comment (Theory):
The motivation/need of a discrete tokenised representation is not clear for time series forecasting. Especially the formulation of pre-training not in the form of MLM indicates that it doesn’t necessarily builds on contextualised formulation.

Author response:
Q1: Motivation of discrete tokenised representation. We discussed our motivation in Sec 1. On the one hand, discrete tokenised representation has shown powerful capability to model text data, which is worthy of reference in the time series domain. On the other hand, it is an effective way to model different kinds of time series using a unified discrete tokenised representation, which is beneficial to train a large model of time series on various datasets.

Q2: MLM pretraining task. We have considered the issue once we designed this method. Here are the reasons that we do not use MLM task. Firstly, considering that the downstream task involves predicting time series, if the pre-training process focuses solely on correlation calculations in the frequency domain using MLM task, the model may struggle to grasp changes in the time series effectively. Secondly, the frequency basis vectors are independent from each other. Attempting to reconstruct missing vectors based on other representation vectors is difficult. Thirdly, the results below indicate that the MLM task is not suitable for this particular scenario. 
| Dataset | TSR | MPP | MLM | Metric | 96    | 192   | 336   | 720   |
|-|-|-|-|-|-|-|-|-|
| Weather |     |     | √   | MSE    | 0.179 | 0.230 | 0.282 | 0.358 |
|         | √   |     |     | MSE    | 0.174 | 0.221 | 0.276 | 0.354 |
|         |     | √   |     | MSE    | 0.176 | 0.220 | 0.276 | 0.354 |
|         | √   | √   |     | MSE    | 0.173 | 0.220 | 0.276 | 0.353 |

Reviewer comment (Writing):
There are some issues as over-useage of notation in the methodology section: In Equation 2 i is used for imaginary part, but in Equation 4 and 5 i is used for index of the position.

Author response:
Q3: Notation issue: Thank you very much for your meticulous reading and pointing out the notation issues in the paper. We will modify them and conduct thorough proofreading for our paper.

Reviewer comment (Reproducibility):
Some experimental details are missing, e.g. It is mentioned beta value is altered between different values.

Author response:
Q4: Beta value: We have conducted a considerable number of experiments to evaluate the effect of beta value. It concludes that it has little influence on the final results and the best results are achieved when beta is set to 1.
| Model                  | Metric | 96    | 192   | 336   | 720   |
|-|-|-|-|-|-|
|   FreqTST_ self_beta1  |   MSE  | 0.178 | 0.223 | 0.281 | 0.356 |
|                        |   MAE  | 0.220 | 0.256 | 0.299 | 0.347 |
|  FreqTST_ self_beta0.5 |   MSE  | 0.178 | 0.225 | 0.279 | 0.357 |
|                        |   MAE  | 0.219 | 0.258 | 0.298 | 0.348 |
| FreqTST_ self_beta0.75 |   MSE  | 0.178 | 0.225 | 0.280 | 0.356 |
|                        |   MAE  | 0.218 | 0.259 | 0.298 | 0.348 |
|  FreqTST_ self_beta1.5 |   MSE  | 0.182 | 0.223 | 0.280 | 0.356 |
|                        |   MAE  | 0.222 | 0.256 | 0.297 | 0.347 |
|   FreqTST_ self_beta5  |   MSE  | 0.177 | 0.224 | 0.281 | 0.356 |
|                        |   MAE  | 0.217 | 0.258 | 0.299 | 0.347 |

Reviewer comment (Novelty):
Some work on frequency domain forecasting is not mentioned in the related work  
- Chen et al. (2023) FrAug: Frequency Domain Augmentation for Time Series Forecasting  
- Sun et al. (2022) FreDo: Frequency Domain-based Long-Term Time Series Forecasting

Author response:
Q5: Relevant research. It is very kind of you for recommending papers highly related to our work. The Fredo model concludes that learning in frequency domain improves model performance with statistical significance, which is a great support material for our work. At the same time, we would like to emphasize that we cited FrAug in Sec 3.1 instead of in related work, as it mainly provides a method of data augmentation rather than model designing.

Reviewer comment (Experiments):
Did you try pre-training with MLM like loss, how did it behave?

Author response:
Q2: MLM pretraining task. We have considered the issue once we designed this method. Here are the reasons that we do not use MLM task. Firstly, considering that the downstream task involves predicting time series, if the pre-training process focuses solely on correlation calculations in the frequency domain using MLM task, the model may struggle to grasp changes in the time series effectively. Secondly, the frequency basis vectors are independent from each other. Attempting to reconstruct missing vectors based on other representation vectors is difficult. Thirdly, the results below indicate that the MLM task is not suitable for this particular scenario. 
| Dataset | TSR | MPP | MLM | Metric | 96    | 192   | 336   | 720   |
|-|-|-|-|-|-|-|-|-|
| Weather |     |     | √   | MSE    | 0.179 | 0.230 | 0.282 | 0.358 |
|         | √   |     |     | MSE    | 0.174 | 0.221 | 0.276 | 0.354 |
|         |     | √   |     | MSE    | 0.176 | 0.220 | 0.276 | 0.354 |
|         | √   | √   |     | MSE    | 0.173 | 0.220 | 0.276 | 0.353 |

Reviewer comment (Experiments):
How the beta value alters, is there a scheduling used for it?

Author response:
Q4: Beta value: We have conducted a considerable number of experiments to evaluate the effect of beta value. It concludes that it has little influence on the final results and the best results are achieved when beta is set to 1.
| Model                  | Metric | 96    | 192   | 336   | 720   |
|-|-|-|-|-|-|
|   FreqTST_ self_beta1  |   MSE  | 0.178 | 0.223 | 0.281 | 0.356 |
|                        |   MAE  | 0.220 | 0.256 | 0.299 | 0.347 |
|  FreqTST_ self_beta0.5 |   MSE  | 0.178 | 0.225 | 0.279 | 0.357 |
|                        |   MAE  | 0.219 | 0.258 | 0.298 | 0.348 |
| FreqTST_ self_beta0.75 |   MSE  | 0.178 | 0.225 | 0.280 | 0.356 |
|                        |   MAE  | 0.218 | 0.259 | 0.298 | 0.348 |
|  FreqTST_ self_beta1.5 |   MSE  | 0.182 | 0.223 | 0.280 | 0.356 |
|                        |   MAE  | 0.222 | 0.256 | 0.297 | 0.347 |
|   FreqTST_ self_beta5  |   MSE  | 0.177 | 0.224 | 0.281 | 0.356 |
|                        |   MAE  | 0.217 | 0.258 | 0.299 | 0.347 |

Reviewer comment (Reproducibility):
Is multi-head attention used? What is head size? What is the embedding size and hidden size for attention layers? Which activation function is used? Were there any drop out? Which optimization is used? What is the optimization hyperparameters? How the learning rate? How the baseline models trained?

Author response:
Q7: Experimental details. We apologize for not clarifying the detailed issues, and draw a table in appendix for all the parameters you mentioned. Note that we use a learning rate schedule the same as PatchTST, which is altered according to different datasets. The training settings of baseline models are consistent with the original paper.
| parameter | head size | embedding size | hidden size | dropout | optimizer | activation function |
|:-:|:-:|:---:|:--:|:---:|------|---------|
|   value   |     16    |       128      |     256     |    0    | Adam      | gelu                |