Paper: Fourier Ordinary Differential Equations (ICLR-2024)

Reviewer comment (Writing):
I am not sure why the authors mention "Euclidean space" as a contrast to their frequency space modelling method. I don't think this is correct, specifically is FODE using any metric space different from "Euclidean", shouldn't it be just contrasted by mentioning in the data space or something like that? If one uses to choose specifics, these need to be pointed out, mentioning terms vaguely isn't helpful.

Author response:
Q1: Euclidean space in our paper is just used to contrast to the frequency space. We realized it may cause confusion, so we changed it to “Time domain” in the revision.

Reviewer comment (Theory):
It is mentioned that "time differentiation in the Fourier domain simplifies the modeling of dynamics as it becomes a multiplication operation". I believe authors refer to the convolution property in this case. Is this even being used implicitly and/or explicitly? If not, what is the point of mentioning some property? DFT has many properties; if they are not used, these shouldn't be presented as if they are helping the proposed method in any way.

Author response:
Q2: We appreciate the review’s question but we do not refer to convolution property here. We aim to say When one transforms a signal into the Fourier domain (using the Fourier Transform), differentiation with respect to time becomes a multiplication by the frequency variable. Mathematically, if f(t) is a time-domain function and F(ω) is its Fourier transform, then the Fourier transform of the derivative df(t)/dt​ is given by jωF(ω), where j is the imaginary unit and ω represents frequency. It is actually an important motivation for our work. Based on these properties, we plan to learn the derivative df(t)/dt​ in the Fourier domain and then use it for ODE based model.

Reviewer comment (Writing):
What is $T$ in equation (1)? Should it be $t_1$ instead?

Author response:
Q3 & Q4: 
Thanks for pointing them out. Yes, these are some typos, and we corrected them in the revision.

Reviewer comment (Theory):
Equation (3) is wrong; I will consider this as a typo.

Author response:
Q3 & Q4: 
Thanks for pointing them out. Yes, these are some typos, and we corrected them in the revision.

Reviewer comment (Theory):
So, DFT from equation (4) is being performed along the feature dimension. If this is the case, what do frequency even mean in this case? If authors can clarify this, it would be great.

Author response:
Q5: 
In our setting, the DFT(FFT) from equ (4) is being performed along the time series length dimension. Thus, I think the frequency of it will be easily understandable.

Reviewer comment (Experiments):
In experimental results, comparison to a frequency space based method is something that I would like to see. If nothing else, [2] can be a good baseline; any comment on this one?

Author response:
We realize that compared with frequency space-based method is important to support the superiority of the FODE model. 
we have conducted additional comparative experiments to support the superiority of the FODE model. We further compare FODE with FNO[1] (belongs to the frequency space-based method) and Neural Controlled ODE (Other reviewers mentioned) on time series classification tasks.  We read [2] and we think it is a good baseline, we added [2] in related work since the time limit.
For a fair comparison, FNO[1] is conducted with one Fourier layer where modes=2 and width=8. 
Despite this configuration resulting in the FNO having more parameters than our FODE model, as detailed in Table 2, our FODE model demonstrates superior performance. The results, presented in Table 3, show that our model achieves the lowest test loss on two out of the three ECG datasets used in our study.

Reviewer comment (Evaluation):
I would expect time-series classification results be evaluated on accuracy and not MSE (Table 3). Why is the choice for MSE justified for a classification task?

Author response:
Q7:
This is a typo. For time-series classification, we use Cross-Entropy Loss as the loss function, and the test loss is what we recorded during the experiment. Thus, table 3 describes the “Cross-Entropy Loss ”. Please see the revision.

Reviewer comment (Experiments):
In Table (2) what is the result for NODE with comparable parameters? Can the authors discuss this briefly if possible?

Author response:
Q8:
The NODE with comparable parameters will still have worse performance than FODEs. We can be convinced of this point from our rich empirical experiments. Also, even ANODEs and SONODEs are more advanced models and always have better performance than original NODEs with comparable parameters. We also compared more models as shown in Table 3.

Reviewer comment (Evaluation):
Figure 4 is good, this is expected, but is there any further analysis that authors plan to present?

Author response:
Q9:
From Figure 4, we can see how hidden states evolve in the frequency domain for time series. The short-time Fourier transform is utilized here to enhance the visualization of the signal’s frequency domain Information.

Reviewer comment (Presentation):
I am not sure about the utility of Figure 5; this needs to be explained and justified better.

Author response:
Q10:
Figure 5 is used to illustrate how the filter K be trained during training. The evolution of filter K with different initialization. From left to right, K is initialized by Zeros, Ones, and Xavier Uniform,  respectively. From top to bottom, each subfigure shows the timeline of the training process (the training loss is from high to low). The changing of colors represents the changing of weight values in K during training. We added more descriptions in the revision.

Reviewer comment (Experiments):
The baselines considered are fairly simple ones that are not truly SOTA. Authors should compare the results with FNO, DeepONet, or such frameworks which are considered to be SOTA.

Author response:
we have conducted additional comparative experiments to support the superiority of the FODE model. We compare FODE with NCDE[1] and FNO[2] on time series classification tasks.  
For a fair comparison, FNO is conducted with one Fourier layer where $modes=2$ and $width=8$.  
Despite this configuration resulting in the FNO having more parameters than our FODE model, as detailed in Table 2, our FODE model demonstrates superior performance. The results, presented in Table 3, show that our model achieves the lowest test loss on two out of the three ECG datasets used in our study.

Reviewer comment (Reproducibility):
What is the loss function? This should be clearly articulated.

Author response:
For the time series forecasting tasks, MSE is used as the loss function. For the time series classification tasks, the Cross-Entropy Loss is used as the loss function.  We recognize the importance of providing comprehensive details about our experimental procedures. We provide more experimental setup details in the revision, please see the environment setup section.

Reviewer comment (Experiments):
More baselines should be compared. How well does the model perform in comparison to FNO, DeepONet and other SOTA time series models. RNNs and LSTMs are some of the earliest models to be considered as baselines.

Author response:
we have conducted additional comparative experiments to support the superiority of the FODE model. We compare FODE with NCDE[1] and FNO[2] on time series classification tasks.  
For a fair comparison, FNO is conducted with one Fourier layer where $modes=2$ and $width=8$.  
Despite this configuration resulting in the FNO having more parameters than our FODE model, as detailed in Table 2, our FODE model demonstrates superior performance. The results, presented in Table 3, show that our model achieves the lowest test loss on two out of the three ECG datasets used in our study.

Reviewer comment (Experiments):
How well does the model scale to more complex data? It is not clear.

Author response:
We agree with the reviewer that how well the model scales to more complex data is an important question.  
In this work, we plan to demonstrate a new method that learns the derivative df(t)/dt​ in the frequency domain and we test two tasks on several datasets to show this idea can work. We will test more complex datasets in our future work.

Reviewer comment (Experiments):
What are the advantages of the FODE over NODE? Is it data-efficient? How robust is it to noisy data?

Author response:
There are several advantages of the FODE over NODE. First is the accuracy, as shown in Table 1 and Table 3, the performance of FODE is much better than NODE.  Second, the FODE provides an innovative way to calculate the derivative df(t)/dt​ in the frequency domain which may be an efficient way to capture the long-term dependence of time series data, and our experiments verify it.

Reviewer comment (Experiments):
Although FODE seems to have a reduced parameter footprint over NODE, what about the training and inference time?

Author response:
We did not record the training and inference time of FODE and other models. From our experience, basically, the training time of FODE will be a little more than NODE, depending on the specific datasets.

Reviewer comment (Experiments):
Why is a sliding window approach required? Can't the model predict the $x_{t+1}$ directly from $x_t$? Please clarify? If this is a hyperparameter, how does the model work when the only data point is used to predict the future? In the original NODE and FNO, only one data point is used for the future prediction.

Author response:
In our paper, there are two experiment tasks. One of them is time series forecasting, as the reviewer mentioned.  In the time series forecasting tasks, we try to predict future values based on these historical values. The number of future values and historical values can be more than 1, for example, we use 50 historical values to forecast values in the future. The reason we use a sliding window approach is to provide a more flexible way to divide the data for this task. When the only data point is used, the FFT transform can be applied to the feature dimension. However, if the feature dimension is 1, our model currently cannot handle this situation. But we left the forecasting task based on just one initial point as a future work.

Reviewer comment (Evaluation):
Problems of model generalization and applicability: Although the article proposes a new method for learning dynamics in the Fourier domain, the generalization and applicability of the method on different types and sizes of time series data are not fully discussed. In particular, the performance and applicability of the FODE model on time series data that are non-periodic or have relatively high signal noise is still a question mark. In addition, the Fourier transform is usually weak for non-smooth signals, which may limit the ability of FODE on complex or non-linear time series data.

Author response:
Thank you for raising these important concerns. We carefully considered these questions during the design of our model. 
We carefully considered these questions during the design of our model. We acknowledge that non-periodic and noisy signals can pose challenges to the FODE model's performance and applicability. To address these issues, we incorporated an element-wise filter into our model's architecture. The element-wise filter is a crucial component of our approach, designed to enhance the model's ability to handle various types of time series data, including those that are non-periodic or have high levels of signal noise. This filter allows the FODE model to adapt and capture the underlying dynamics even in challenging conditions.

Reviewer comment (Evaluation):
Model Complexity and Computational Efficiency Issues: Despite the paper's claim that FODE improves on computational efficiency, in reality, the computational complexity of the Fourier transform and inverse transform may increase dramatically as the amount of data increases, especially on real-time or large-scale datasets. The paper lacks an assessment of the computational efficiency of a wider range of computations, such as on multidimensional datasets and the specific advantages and disadvantages compared to other existing methods.

Author response:
We appreciate your concerns regarding the computational efficiency of our approach, and we'd like to provide further clarification and insight into our efforts to address these issues.
(1) we use the FFT and IFFT inside the “ODE Function” and it makes sure the dimensions of input and output dimensions remain the same with a Neural ODE. For example, if the input data has a shape of (d1, d2) = (length, #features) after applying the FFT, the tensor’s shape becomes (d1’, d2’) = (length, #features//2), where d2’ contains real part and Imaginary part. 
(2) we use the Fast Fourier Transform (FFT) to reduce the computational complexity of DFT. 
(3) the integration of FFT and IFFT inside the 'ODE Function', coupled with their interaction with the ODE solver, forms a synergistic relationship. This synergy not only improves computational efficiency but also aids in effectively learning the vector field. As a result, it can potentially reduce the number of function evaluations required, further contributing to the model's overall efficiency.

Reviewer comment (Experiments):
Insufficient experimental validation and comparative analysis: while the paper provides some experimental results to support the superiority of the FODE model, it lacks comparative experiments, especially with the latest time series analysis methods. The article does not provide sufficient details of the experimental setup, such as the choice of hyperparameters for the model, training details, and measures to prevent overfitting. In addition, the article does not discuss the confidence intervals or statistical significance of the model in terms of forecasting performance, which reduces the persuasiveness of the experimental results.

Author response:
Thank you for your feedback regarding the experimental validation of our study.
(1) we recognize the importance of providing comprehensive details about our experimental procedures. we provide more experimental setup details in the revision, please see the environment setup section, including the number of epochs, loss function, ODE solver method, details of layers, etc. Algorithm 1 also shows the detailed implementation of our model. 
(2) we have conducted additional comparative experiments to support the superiority of the FODE model. We compare FODE with NCDE[1] and FNO[2] on time series classification tasks. For a fair comparison, FNO is conducted with one Fourier layer where modes=2 and width=8. Despite this configuration resulting in the FNO having more parameters than our FODE model, as detailed in Table 2, our FODE model demonstrates superior performance. The results, presented in Table 3, show that our model achieves the lowest test loss on two out of the three ECG datasets used in our study.

[1] Kidger, P., Morrill, J., Foster, J. and Lyons, T., 2020. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 33, pp.6696-6707.
[2] Li, Zongyi, et al. "Fourier Neural Operator for Parametric Partial Differential Equations." International Conference on Learning Representations. 2020