Paper: FITS: Modeling Time Series with $10k$ Parameters (ICLR-2024)

Reviewer comment (Experiments):
There are quite a few bits and pieces to the model pipeline, and at times the extent to which each component contributes towards overall performance is unclear. Perhaps a few ablation studies in the supplementary material can address this?

Author response:
*We understand that ablation study is very important to understand the impact of each part. However, **FITS is a very compact model. Removing any of its part will make it not functional. And FITS only have one hyperparameter which is the interpolation rate. We have done comprehensive ablation studies on the cutoff frequency, please see the Tab.5 of the manuscript and Section D of the appendix.***

Reviewer comment (Evaluation):
While I appreciated the experiment on anomaly detection, this is a field of study where (as referenced by the authors themselves), a few metrics may not be sufficient for fully capturing the success of different techniques. There is consequently a risk that the experiment featured here might not be sufficiently conclusive (although the results obtained on synthetic data are a step in the right direction).

Author response:
*We acknowledge that relying solely on simple metrics may not adequately capture the performance of FITS in real-world tasks. However, to ensure a fair comparison, we have followed the approach of previous works, specifically the Anomaly Transformer, to demonstrate the superior performance of FITS. Nevertheless, we recognize the importance of interpretability and plan to delve deeper into this aspect by investigating the interpretability of FITS using synthetic datasets. For a more comprehensive visualization and interpretation, we invite you to explore the jupyter notebook available in our anonymous repository.*

Reviewer comment (Experiments):
I would like to see more "basic" baselines (e.g., ARIMA, DeepAR for forecasting, and Random Cut Forest for anomaly detection).

Author response:
A: *In line with previous studies like PatchTST, the effectiveness of statistical methods like ARIMA and RNN-based methods such as DeepAR and LSTM has been found to be inferior compared to Transformer-based methods. Hence, we did not include them as baselines in our study. Additionally, DeepAR is specially designed as a probabilistic forecasting model which may not be appropriate to compare with it. It's worth mentioning that RCF, a proprietary anomaly detection algorithm by Amazon, is not included in our analysis. In our manuscript, we present results for Isolated-Forecast, an ensemble and decision tree-based algorithm.*

    *If we understand the ‘heuristic’ correctly,  FITS only have one hyper-parameter i.e., the cutoff frequency. It is easy to find the optimal value comparing to the large hyper-parameter space of other models.*

    *We use the ARIMA implementation provided in the LTSF-Linear. However, as described in the repo, ARIMA runs extremely slow. As a result, we only report the MSE performance on ETTh1 and ETTm1.*

    |Datasets|96|192|336|720|
    |:---:|:---:|:---:|:---:|:---:|
    |ETTh1|1.0642|1.1053|1.125|1.158|
    |ETTm1|1.4870|1.7228|2.0984|3.3069|

Reviewer comment (Evaluation):
To further substantiate the claims about performance (and to put them on a statistically sound foundation), a critical difference plot [1] may be a useful analysis tool.

Author response:
A: *We have added the critical difference plot in the **Appendix I**. FITS outperforms all the baselines with statistical significance. The critical difference plot also showcases the robustness of FITS' performance across various datasets, reinforcing its reliability.*  

We apologize for any confusion caused. As you correctly mentioned, we utilized a pre-existing repository for generating the critical difference (CD) plot. It is worth noting that the CD plot is commonly employed in classification tasks, where the evaluation metric is accuracy (higher is better). However, in our case, we evaluated a forecasting task based on mean squared error (MSE), where lower values indicate better performance. Consequently, being ranked sixth actually means that our model performs the best in terms of MSE.

To create the CD plot, we originally only considered the MSE performance over a 336-forecasting horizon, and we have now updated the CD plot in accordance with all the evaluation settings. Kindly refer to the appendix for the updated version. We also attempted to reverse the order of the models in the plot. However, we encountered some issues with the plotting repository when activating the reversal. Consequently, we have retained the original order for now. We are actively working to rectify this issue and will update the diagram in the final version.

The updated CD plot now demonstrates that our model, FITS, is no longer connected to the other methods, indicating statistical significance in its performance superiority.

We apologize once again for any confusion caused and hope that our responses have adequately addressed your concerns.

Reviewer comment (Experiments):
The method employs several heuristics that need to be tuned in the real world, a drawback it shares with other methods.

Author response:
*If we understand the ‘heuristic’ correctly,  FITS only have one hyper-parameter i.e., the cutoff frequency. It is easy to find the optimal value comparing to the large hyper-parameter space of other models.*

Reviewer comment (Theory):
Can you explain why $x(t-\tau)$ is a shift forward in time, rather than backward?

Author response:
A: *When we talk about shifting forward in time, it means we are essentially repositioning the starting point of our time axis in the direction of increasing time values. Alternatively, you can think of it as sliding the entire function in the opposite direction, which is the negative part of the time axis. Consider a time series represented as F(t). Shifting forward by a duration of τ means that we are relocating the original reference point τ units ahead in time. This action results in a new representation of the function, denoted as F(t-τ). In simpler terms, it's like moving our viewpoint along the time axis to where τ becomes the new starting point.*

Reviewer comment (Presentation):
Can you add a more thorough caption to Fig. 2? As it stands, it is not comprehensible enough to fully understand your method. Is it just a single linear layer/what is the output dimension? What are the typical dimensions of $\tilde{X}$? Are (i)RIN and (i)rFFT performed in parallel or sequentially? Please elaborate more on how RIN works, either in the Figure or its caption.

Author response:
A: *We have updated the caption for Figure 2 according to your advice.*
    
    *Yes, as described in Section 3.2, the complex linear layer is just a single linear layer with complex-valued weights.*
    
    *Suppose the input X dimension is (B, L, C). The $\tilde{X}$ should be the dimension of its rFFT which is (B, L//2+1, C). After a LPF, the $\tilde{X}$ is cut to the length of l according to the cutoff frequency. The input dimension of complex linear layer is (B, l, C) (Batch size, Length, Channel) and the output is (B, ηl, C) which extend the length according to the interpolation rate.* 
    
    *As mentioned in the paper, RIN stands for Reversable Instance Normalization. It is a simple normalization method that removes the bias and variance for each data instance before feeding into the model. After model processing, the variance and bias are added back to the result to reverse it back to the original data distribution. In most cases, RIN is used to eliminate the distribution shift so that models can reach a better result. However, FITS use RIN mainly to remove the bias of time series which can introduce a large 0 frequency component. Please check the paragraph under Figure 2 for detailed information.*  
    
    *RIN and FFT runs in sequential. As described, RIN is performed on the time domain. And we employ RIN to eliminate the bias in time series and use iRIN to add it back. Thus, RIN is performed ahead of the rFFT and iRIN in performed after the irFFT.*

Reviewer comment (Experiments):
Have you experimented with more sophisticated layers than just linear?

Author response:
A: *Thank you for raising this question. We appreciate your interest in exploring more sophisticated layers beyond linear ones in our study.*

    *The frequency domain poses unique challenges due to its complex nature, and it is widely acknowledged that complex-valued neural networks are better suited for processing such data. We share your curiosity regarding the performance of complex-valued layers in our model. However, it is important to note that complex-valued neural networks are still in the early stages of development. Many commonly used components, including the Attention block, present mathematical obstacles when adapting them to the complex domain. Moreover, critical elements such as activation functions and loss functions are currently unavailable, making it difficult to construct large-scale neural networks.*

    *Nonetheless, we recognize the potential of complex-valued neural networks in the frequency domain and plan to address this in our future work. We intend to investigate the use of complex-valued layers, such as a complex-valued frequency domain Transformer, to explore their impact on our model's performance.*

Reviewer comment (Writing):
What is the back-casting task, and how is it used in training?

Author response:
A: *As described in the fourth paragraph of Section 3.2. The back-casting is adding supervision on the reconstructed input. Since FITS outputs the extended time series segment which contains not only the forecasting horizon but also the reconstructed look-back window. By adding supervision on both of them, we were expecting a better performance.*

Reviewer comment (Writing):
Can you motivate in 1–2 sentences why the output length needs to be controlled (Section 3.3)?

Author response:
A: *According to the time series forecasting task definition, the model should make prediction on $x_{t, t+h}$ with given look-back window $x_{t-lbw, t}$. Generally, the forecasting model output length should be set as the length of target horizon. For FITS, we conduct the frequency interpolation on the input $x_{t-lbw, t}$ and generate the output as $x_{t-lbw, t+h}$. But anyway, the input and output length are decided according to the task setting (e.g.., input 720 output 96).*

Reviewer comment (Theory):
You say the LPF discards components beyond the model's learning capabilities. What do you mean by this? The model's capabilities are independent of the signal, and high frequencies are not always noise or irrelevant. It seems domain-specific. Furthermore, you say high-frequency components such as trends are filtered out. First, trend signals can be low-frequency, and second, they are particularly important in the forecasting realm.

Author response:
1. *The primary objective of using a Low-Pass Filter (LPF) is to simplify the model by **making a trade-off**—specifically, by discarding high-frequency components that have minimal impact on the final prediction. As shown in Figure 3 of the original paper, when a significant portion of high-frequency components is removed, the overall shape of the time series remains largely unchanged. This suggests that if the model can effectively fit the primary frequencies, it can achieve good results with fewer parameters.* 
    2. *FITS, being a compact linear model with very few parameters, struggles to handle data with complex patterns. High-frequency segments often contain information with extremely rapid temporal dynamics, such as noise. We believe that such information exceeds the learning capacity of FITS, which is why we employ the LPF to eliminate these high-frequency components and reduce the model's parameter count.*
    3. *As you rightly pointed out, the definition of "high-frequency" can vary across different datasets. Therefore, we conducted comprehensive ablation studies to demonstrate the impact of the cutoff frequency on the final prediction results across various datasets. This approach allows us to identify the primary frequency ranges that FITS can effectively learn, striking a balance between performance and memory usage.*
    4. *While trends are often considered low-frequency signals on a long time scale, time series analysis frequently operates on smaller segments of the data. In these cases, the long-term trend's influence on a particular segment can manifest as a bias. To preserve this bias and ensure it is not inadvertently removed by operations like LPF, we use a Reversable instance normalization (RIN). **Thanks for your suggestion, we have removed the description of ‘removing trend’ from the LPF part.***

Reviewer comment (Experiments):
The method is particularly interesting because of its parameter-effectiveness. Can you perform experiments on real-time inference time on simple hardware (not necessarily edge devices) to substantiate this (e.g., real-time anomaly detection)? What are the challenges regarding domain shifts and refitting, and how could they be addressed? This could substantially improve the focus and contribution of the paper.

Author response:
A: *Indeed, we have taken steps to evaluate FITS on relatively simple hardware setups. We have reported the time consumption of FITS when executed on a single CPU core, a common scenario in edge devices. Furthermore, FITS can be directly deployed on ubuntu based single board computers (e.g. Raspberry Pi) with our original code. Similarly, FITS achieves inference time at around 3ms on Raspberry Pi.*
    
    *Furthermore, we have also successfully deployed FITS on an ARM development board (specifically, the STM32F303K8T6, equipped with an ARM Cortex M4 core running at 72 MHz, 64 Kbytes of flash memory, and 16 Kbytes of SRAM). This board represents a computationally and memory-constrained device, making it a valuable test case.*
    
    *Domain shift poses a common challenge in time series forecasting. Typically, it manifests as changes in bias and variance. To address this, we utilize RIN (instance normalization) during inference, effectively handling this problem. In cases of other shifts, such as frequency and waveform changes, we can employ off-the-shelf continual learning strategies to update FITS in real-time. FITS' computational and memory-efficient design allows for easy fine-tuning with minimal data on resource-constrained devices.*

Reviewer comment (Evaluation):
The result on anomaly detection is not remarkable. Especially on the SMAP and MSL dataset. Some more in depth analysis is needed to find out the reason.

Author response:
*As analyzed in the conclusion. These two datasets are majorly composed with binary data where time-domain modeling is preferable as the raw data format is sufficiently compact. The binary-valued data also have messy frequency presentation which is difficult for FITS to learn. We provide a visualization in the interpretability notebook in the anonymous repository to show such finding.*

Reviewer comment (Evaluation):
Even though the FITS have far fewer parameters comparing to the DLinear, it still need more time to inference on the GPU (0.6 and 0.4 ms respectively). Does this mean the DLinear is still the best choice for the real-time application?

Author response:
*The inference time of FITS is sub-ms level which is comparable to the communication overhead, the 0.2ms overhead does not make a dealbreaker to fail FITS to work as a real-time time series analysis model. Furthermore,* t*he inference time on GPU does not match the common case on the edge-device real-time application. The edge-device often only have single core computation unit which should resemble the inference time result on the single-CPU setting which FITs achieves better result. The inference time is also influenced by the cuda optimization. As far as we know, the complex-valued neural network in cuda and pytorch is still an experimental feature, which indicate that the optimization is still not as perfect as the normal linear layer. This may cause some performance downgrade. Furthermore, the edge devices often have very constrained memory resources such as the arm development board we use (STM32F303K8T6, which is with ARM Cortex M4 core at 72 MHz, 64 Kbytes of flash memory, 16 Kbytes of SRAM). The parameter of DLinear can not fit into a board with 64 KBytes of flash memory. As a result, FITS is still more suitable for the edge-device deployment.*

Reviewer comment (Reproducibility):
How do author deploy the model on devices that do not support the complex computation?

Author response:
*We are aware of that many edge devices or even RTX4090 do not support complex computation. Thus, we provide a FITS-real model to simulate the complex-valued linear projection with normal real-valued linear layer as following:*
    
    $$ y_{r}= W_r * x_r - W_i * x_i $$
    
    and, 
    
    $$ y_{i}=W_r * x_i + W_i * x_r $$
    
    *The complex-valued weight matrix is divided as two real-valued linear projection $W_i, W_r$.* 
    
    *The FITS-real model achieves the same result as the original FITS, and they have the same memory footprints.*