From: ICLR.cc/2023/Conference/Paper1571/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper1571/Authors
title: Did our response address your concerns?
comment: Dear Reviewer dLYs,

We are sincerely grateful for your time and energy in the review process! 
Given the limited time left in this response phase, we wanted to make a final inquiry about whether our response has addressed your concerns. We would be eager to follow up on any additional comments you may have!

Thank you!

Paper1571 Authors

From: ICLR.cc/2023/Conference/Paper1571/Authors
title: Did our response address your concerns?
comment: Dear Reviewer cyPc,

We are sincerely grateful for your time and energy in the review process! 
Given the limited time left in this response phase, we wanted to make a final inquiry about whether our response has addressed your concerns. We would be eager to follow up on any additional comments you may have!

Thank you!

Paper1571 Authors

From: ICLR.cc/2023/Conference/Paper1571/Authors
title: Reponse to followup comments
comment: Dear Reviewer jXp9

We thank the reviewer for the response. We wish we could have delivered the main contribution of our work more clearly. Please allow us for the last-minute clarification:

1) Our work is a game-theoretic approach for aggregating temporal observations for time-series forecasting in a data-driven fashion, not an explainable AI method. As well-appreciated by Reviewer cyPc, our work is the first paper that connects multi-agent game theory with time series forecasting. Because our work is an application of well-established game theory to time-series analysis, we focused to provide i) intuitive interpretations of what the game-theoretic framework brings to time-series analysis in the manuscript, and ii) thorough mathematical backgrounds and theoretical proofs in the appendix.

2) Our novel game-theoretic framework allows our method to flexibly extract meaningful (predictive) information from past observations which leads to outperforming performance over the existing DE-based time-series forecasting methods. To support this claim, we conducted a set of additional experiments with different temporal dynamics to show how the contributions of each past observation for predicting the future can vary depending on the different scenarios. As such, we believe providing quantitative analysis on the explanation quality is beyond the scope of our work.

Best,

Paper1571 Authors

From: ICLR.cc/2023/Conference/Paper1571/Reviewer_jXp9
title: Followup comments
comment: Thank you for the detailed responses.

The explainability justification seems weak - there is no quantitative analysis on the explanation quality. 

I still think the paper writing needs to be improved to tie with the theoretical framework better.

From: ICLR.cc/2023/Conference/Paper1571/Authors
title: Dear Reviewer jXp9
comment: Once again, we would like to thank you for your invaluable feedback! We were wondering whether our response from November 17 has sufficiently addressed your concerns. If you have any leftover comments or concerns, please let us know - we would be happy to do our utmost to address them!

From: ICLR.cc/2023/Conference/Paper1571/Authors
title: Dear Reviewer cyPc
comment: Once again, we would like to thank you for your invaluable feedback! We were wondering whether our response from November 17 has sufficiently addressed your concerns. If you have any leftover comments or concerns, please let us know - we would be happy to do our utmost to address them!

From: ICLR.cc/2023/Conference/Paper1571/Authors
title: Dear Reviewer dLYs
comment: Once again, we would like to thank you for your invaluable feedback! We were wondering whether our response from November 17 has sufficiently addressed your concerns. If you have any leftover comments or concerns, please let us know - we would be happy to do our utmost to address them!

From: ICLR.cc/2023/Conference/Paper1571/Authors
title: Response to Reviewer dLYs (2/2)
comment: $\textbf{(Interpretation of the Temporal Weights)}$

Given the cooperative neural agents, the decision of $i$-th agent $X_t^{i, \boldsymbol{\alpha}}$ contains relatively important latent information if the system assigns high values $A(t, X_t^{i,\boldsymbol{\alpha}}) \uparrow$ to the temporal weights. That is, the past observation $y_{s_i}$ is highly informative to predict the future value $y_t \approx X_t^{i, \boldsymbol{\alpha}}$.

In Section 5.2, we illustrate that our proposed temporal aggregations capture four different scenarios which mimic the temporal dynamics in the real-world time series: decay, impulse, delay, and noisy observations. In addition, we have provided strong supporting evidence to further validate the learned temporal attentions for the BAQD dataset. As shown in Figure 2 (in the manuscript), our method demonstrates that the temporal attention function puts higher weights on the most recent observations for short-term forecasts to capture local variations while it evens out the attention weights to capture the general variation patterns. Such behavior is well-supported in a study of air quality forecasting [2].

[2] S. Jang et al., Long- and short-term time series forecasting of air quality by a multi-scale framework, 2021

$\textbf{(Efficient Network Architecture)}$

For computational efficiency, the proposed MaSDEs share the same network except for the $\textbf{agent identification layer}$ (AIL, Section A.6.1) that can separate representations made by each neural agent. Even though most networks are shared and reused, the AIL encourages each neural agent to propagate the individual stochastic trajectory to ensure the expressivity of the entire system. Thanks to the architectural efficacy, the computational cost stays less than $O(N)$ even when the number of observations $N$ becomes large.

From: ICLR.cc/2023/Conference/Paper1571/Authors
title: Response to Reviewer dLYs (1/2)
comment: Thank you for your time in reviewing our work. We also appreciate your positive feedback and questions. We have addressed all the comments and provided a point-by-point response below.

$\textbf{(Main Contribution)}$

Please allow us to clarify the novelty of our work compared to the prior work (i.e., CSDE-TP) [1]. We emphasize that the main contribution of our work is $\textbf{the novel game-theoretic framework to train multi-agent neural SDEs}$ for time series analysis. Our new framework enhances the existing neural SDEs by modeling an individual past observation as a neural agent and targets to find an optimal aggregation of those neural agents to extract meaningful information about the underlying temporal relevance directly from the data. 

To this goal, our work suggests a cooperative differential game to find the optimal solution for aggregating multiple neural agents in a sophisticated manner. More specifically, we formally define the Nash equilibrium in Definition 1, which assures the optimal solution for our objective functional in Eq. (3). Then, we propose a novel game-theoretic training scheme, which we refer to as deep neural fictitious play, that achieves the Nash equilibrium: each neural agent mutually finds the best response to the environment made by the other neural agents to comprehensively minimize the error between the (aggregated) prediction and the target, i.e., $|| \Lambda_t - y_t ||^2$. To summarize, the key driving factor of our performance improvement over CSDE-TP comes from our novel game-theoretic framework that achieves the optimal temporal aggregation of past observations that results in accurate predictions about the future. 

For a numerical justification, we show supporting evidence that our game-theoretic framework leads to significant performance improvements in the experiments. Please refer to $\textbf{(Numerical Comparison to CSDE-TP)}$ for more details.

[1] S.Park et al., Neural Markov Controlled SDE: Stochastic Optimization for Continuous-Time Data, 2021

$\textbf{(Numerical Comparison to CSDE-TP)}$

When the dataset contains relatively smooth transitions and small noises, CSDE-TP produces comparable results as the entire spatio-temporal variables carry equally important latent information. Thus, homogeneous attention can become a suitable model design to capture the relative importance in such a static environment. In contrast, when the dataset contains relatively severe noisy signals such as high irregularity, temporal missingness, and high-frequency spatial noises (e.g., BAQD and Mackey-Glass DDE), the assumption of CSDE-TP about the static environment becomes violated since the relative importance between past observations should have been considered. 

Under highly noisy observations, the cooperative agents of the proposed framework collectively balance the temporal weights to filter out the temporal dynamics of those noisy observations. This property of neural agents originated from the definition of Nash equilibrium (Definition 1) since the goal of the game-theoretic framework was to assure the best possible performance no matter how noisy the environment is given to MaSDEs. Overall, we can conclude that the CooPredict $\textbf{consistently outperforms}$ the CSDE-TP in these noisy environments.

From: ICLR.cc/2023/Conference/Paper1571/Authors
title: Response to Reviewer cyPc (2/2)
comment: $\textbf{(Summary of relevant concepts)}$

We thank the reviewer for the kind recommendation. Based on the reviewer's suggestions, we have added a paragraph to summarize relevant concepts of the proposed method in the revised version of the manuscript; please refer to Appendix A.4.

$\textbf{(Comparisons with Non-DE-based Methods)}$

We start by noting that we focused our experiments on comparing CooPredict to a variety of methods along the line of DE-based approaches ranging from Latent ODE to more recently proposed Neural Laplace and CSDE-TP. This is because non-DE-based counterparts are incapable of handling irregularly sampled or partially observed time-series data without careful consideration of pre-processing steps such as discretization and interpolation.  

To address the reviewer's suggestion, we have compared CooPredict with the state-of-the-art Transformer-based time-series forecasting methods on the Speech dataset. We chose the Speech dataset since it comprises complete time-series observations with regular time intervals which can be favorable for applying non-DE-based methods and can avoid potential bias due to design choices of pre-processing steps. 
As can be seen in following table, CooPredict outperforms its non-DE-based counterparts implying that our proposed method can be a promising tool for general time-series modeling under both regular and irregular settings.

| Method  | Test MSE $\downarrow$ $(\times 10^{-2})$  |
|--:|---|
| Transformer [2]  | $0.4476 \pm 0.069$  |
|  Informer [3] | $0.4258 \pm 0.002$  |
|  Autoformer [4] | $0.4854 \pm 0.022$  |
| CooPredict  |  $\mathbf{0.4136 \pm 0.017}$ |

[2] A. Vaswani et al., Attention is all you need, 2017

[3] H. Zhou et al., Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting, 2021

[4] H. Wu et al., Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting, 2021

$\textbf{(Notations)}$

We thank the reviewer and apologize for any confusion caused. $\textit{\textbf{Hess}}\boldsymbol{\mathcal{V}}$ in Eq. (14) denotes the Hessian matrix of function $\boldsymbol{\mathcal{V}}$. We have revised the notation accordingly throughout the manuscript.

From: ICLR.cc/2023/Conference/Paper1571/Authors
title: Response to Reviewer cyPc (1/2)
comment: Thank you for pointing out the strengths of our paper. Moreover, we thank the reviewer for the detailed review and insightful comments. Below, we address the comments raised by the reviewer.

$\textbf{(Theoretical Considerations)}$

We totally agree with the reviewer's direction to model time series as a stochastic process $y_{s \in \mathbb{O}}$
 with detailed assumptions on temporal information. For example, we can further utilize the information $var(Y_s) \propto a_s^2$ or $var(Y_s) \propto b^2$ for some $a_s, b > 0$ obtained from data statistics. Since our theoretical groundings and main interests are lying on stochastic differential games, it was unable to overcome every technical difficulties that come from the general mathematical assumptions considered in this literature. For example, in the conventional SDG via Markovian FBSDEs, it is assumed that the initial points $\{y_{s_i}\}$ of MaSDEs are $\textit{deterministic}$. If the system supports the stochastic representation of the target data $\{ y_{s_i}(w) \}$, $w \in \mathcal{G}$ for some filtration $\mathcal{G}$, then we immediately face the theoretical obstruction about the $\textit{weak solution}$ to FBSDEs, and the usual technical set-up in the proposed work is not expected to be satisfied due to the effect of $\mathcal{G}$ to FBSDEs. Similar to ours, prior work [1] also assumed fixed/deterministic initial points during the optimization to solve HJBEs.

[1] J. Han et al., Convergence of Deep Fictitious Play for Stochastic Differential Games, 2022

$\textbf{(On the Stochastic Optimality)}$

In our interpretation to solve HJBEs, the Hamiltonian deviation in Eq. (A.15) is defined for all instances $i \in [{1, \cdots, I}]$ and time $t \in \mathbb{T}$. Thus, the proposed framework deals with $I$ number of equations in Eq. (A.15) in the instance-wise level. In Algorithm 1, every individual agents are trained to solve their own decoupled adjoint problem so that Eq. (A.15) is satisfied for all $i$. To do so, each agent minimizes the temporal integral of the adjoint variable, i.e., $\int_{\mathbb{T}}dY_t^{i, \boldsymbol{\alpha}}$, to bound Eq. (A.41) and Eq. (A.42). Since the proposed admissible action set assures optimality in terms of the inequality in Eq. (A.52), the final result in Eq. (A.53) holds.

$\textbf{(On Initial Conditions)}$

In the Eq. (A.11), the specific form of the filtration at time $t$ can be specified as follows: $\mathcal{F}_t$ =  ${\bigvee}\_{i \in [1, \dots, I]} \mathcal{F}_t^{i, B^i}$, ${\mathcal{J}}^{i} (t, x, \sim)$, and $x = [\mathbb{E}X_t^{1, \boldsymbol{\alpha}, y\_{s_1}}, \dots, \mathbb{E}X_t^{I, \boldsymbol{\alpha}, y\_{s_I}}]$ given initial conditions $[y\_{s\_i}]\_{i \in \mathcal{O}}$. Here, we clarify that the time variable $t$ is considered as $t = \inf{(\mathbb{T})}$ since our task is predicting time series on interval $\mathbb{T}$. According to this setting, one can easily show that the marginal constant $\boldsymbol{\epsilon}$ is largely affected by initial conditions. Let us consider the case where initial conditions are poorly given. In this case, neural networks produces a large upper-bound of Eq. (A.38) since the gradient descent quickly converges to bad local optima. This leads to poor convergence results which eventually induces worsened Nash equilibrium than perfect equilibrium states due to the high margin value $\boldsymbol{\epsilon}$.

From: ICLR.cc/2023/Conference/Paper1571/Authors
title: Response to Reviewer jXp9 (2/2)
comment: $\textbf{(Clarification of Deep Neural Fictitious Play)}$

Please allow us to clarify our contribution in Section 3. Combining the deep neural networks and the Fictitious play to iteratively identify the Nash equilibrium in the multi-agent stochastic differential game (MaSDG) was studied in [2]. This existing method only considers linear quadratic forms of MaSDGs and the corresponding global convex Hamiltonian system to develop an efficient algorithm. However, such limited expressivity of the restricted form of linear SDEs in MaSDGs [2] can be incapable of learning the complex temporal dynamics from the data and, thus, unsuitable for complicated tasks such as multivariate time-series forecasting.  
To overcome such a limitation, we propose a novel framework based on Fictitious play that is generally applicable to non-linear MaSDGs and the corresponding non-convex Hamiltonian system. The strength of the proposed framework is the ability to deal with heterogeneous agents and complex temporal dynamics, which can be demonstrated in our experiments by the outperforming performance of CooPredict throughout a wide range of multivariate time-series forecasting tasks and by the heterogeneous attention profiles of the learned agents. 

[2] J. Han et al., Convergence of Deep Fictitious Play for Stochastic Differential Games, 2022.

From: ICLR.cc/2023/Conference/Paper1571/Authors
title: Response to Reviewer jXp9 (1/2)
comment: We thank the reviewer for the kind feedback on our work. We have addressed all the comments in the updated version of our manuscript and provide a point-by-point response below.

$\textbf{(Explainability)}$

We begin by highlighting that the main contribution of our work is the novel game-theoretic framework to train multi-agent neural SDEs for modeling irregularly sampled time series without any induction bias.  
More specifically, our method shares a similar motivation with that of Transformer-based methods which primarily focus on how we can assign optimal temporal aggregation to learn meaningful temporal dynamics. To the best of our knowledge, our work is the first DE-based framework that can directly reveal the underlying temporal relevance of the data. 

To address the reviewer's suggestion, we provide strong supporting evidence to further validate the learned temporal attentions for the BAQD dataset. As shown in Figure 2 (in the manuscript), our method demonstrates that the temporal attention function puts higher weights on the most recent observations for short-term forecasts to capture local variations while it evens out the attention weights to capture the general variation patterns. Such behavior is well-supported in a study of air quality forecasting [1].

[1] S. Jang et al., Long- and short-term time series forecasting of air quality by a multi-scale framework, 2021.

$\textbf{(Multivariate Forecasting)}$

We would like to emphasize that all the tested real-world datasets indeed consist of complex multivariate datasets -- i.e., the Speech, Physionet, and BAQD datasets contain 65, 35, and 6 time-varying features, respectively -- and the task is forecasting multivariate time series. 

$\textbf{(Hyper-Parameters)}$

We chose the hyper-parameters of the benchmarks by grid searching using the validation set (i.e., 20% of the training set), which then was used for training each method. For implementing CooPredict, we identically set the dimension of hidden layers of each neural agent and the drift network as 128 following the baseline model (i.e., CSDE-TP). We set the number of hidden layers as 2 for neural agents and 1 for the drift network. For the attention network, we used a relatively small dimension (i.e., 36) with 2 hidden layers across all the experiments. Please refer to Appendix A.6. for more implementation details of our network architecture.

$\textbf{(Statistical Significance)}$

We thank the reviewer for the recommendation. In our experiments, we followed the same evaluation settings suggested by the prior works (i.e., Latent ODE, CSDE-TP), where a single training/testing split is evaluated with a fixed random seed. To address the reviewer's suggestion, we report the mean and the standard deviation of the MSE on the BAQD dataset over 5 independent runs. Here, we only include the results of the best performing benchmarks on the BAQD dataset due to the limited rebuttal period. Following table hows that our proposed method outperforms the best-performing benchmarks and the results are statistically significant with $p$-value $<0.001$. We will include the mean and the standard deviation for all the experiments in the final version of our manuscript.
| Method  |  Latent ODE |  CRU  | CSDE-TP | CooPredict  |
|--:|---|---|---|---|
| Test MSE  | $0.3814 \pm 0.0072$  | $0.3641 \pm 0.0087 $ |  $0.3475 \pm 0.0045 $  |  $0.2891 \pm 0.0031$ |

The MSE (mean $\pm$ std) on the BAQD dataset.

$\textbf{(Proposition 1 and Corollary 1)}$

Please allow us to clarify the connection of the theoretical results with our proposed method. 
Proposition 1 demonstrates that we can obtain the Nash equilibrium by solving the proposed $\textbf{deep neural fictitious play}$ introduced in Section 3. Moreover, we can further state from Proposition 1 that neural agents with a large expressivity assure accurate evaluation of the perfect Nash equilibrium (i.e., $\boldsymbol{\epsilon} \approx 0$). Corollary 1 shows that the convergent error of the predictions is proportional to magnitude of errors induced by the proposed gradient descent-based algorithm (i.e., Algorithm 1). In other words, the near-perfect Nash equilibrium can be attained by the proposed deep neural fictitious play when the gradient descent algorithm ensures smooth and accurate convergent results of the loss landscape. (Figure A.1)

$\textbf{(Pseudo-Codes)}$

We thank the reviewer for the suggestion. We have added the pseudo-codes for the training and inference procedures of CooPredict in the revised manuscript; please see Appendix A.5 for more details.

From: ICLR.cc/2023/Conference/Paper1571/Reviewer_dLYs
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper studies the time series prediction problem with a new framework based on multi-agent cooperative
differential games in order to better model the temporal dynamics.  Each number of past observations is modeled as one agent which is further parameterized by a neural network. 

The entire multi-agent dynamics are formulated as as set of multi-agent stochastic differential equations. The predicted value at a future timestamp is  a weighted sum of the individuals' decisions. To train the neural agents, the authors used a gradient descent-based method similar to fictitious play, where each neural agent changes its action to find a value function that defines the optimal state of actions. 

Theoretically the authors showed the existence of an action set where an approximate Nash can be achived by Fictitious play. Experimentally, the authors compared the proposed framework with several existing differential-equation based continuous dynamical
models on three real datasets.
strength_and_weaknesses: Strength:
- Better modeling the temporal dynamics with partial observations is an important and difficult problem in time series prediction, this paper proposed a new framework for that which outperforms several existing differential-equation based continuous dynamical models. Overall the paper is well-written and the structure is fairly clear
- The multi-agent SDE framework learns the individual impact of partial observations by design, which is flexible and interpretable (contribution of each agent can be seem from the aggregation matrix)
- Experimentally the proposed framework showed improvements by a relatively large margin over a set of existing differential-equation based continuous dynamical models 

Weakness:
- It was a bit unclear to me the major novelty compared with the prior work (Park et al, Neural Markov controlled SDE: Stochastic optimization for continuous-time data), which proposed a very similar framework with neural agents. It seems that the main difference is that in this work, the aggregation matrix is time-dependent, and in (Park et al) the decision attention assigns uniform weights to all agents. However the new challenges and difficulties brought by such a change was not throughly discussed. 
- Experimentally, the performance of the proposed framework was also very close to the (Park et al) baseline, on two of the three datasets. Why on the BAQD dataset the improvement was more significant? What are the interpretations of the temporal weights in the aggregation?
- My another concern is about the scalability of the proposed method ---given that each observation is modeled as one neural agent, the number of agents in the SDE system scales linearly w.r.t. to the size of the observation size. Can the proposed method scale with large number of observations?



clarity,_quality,_novelty_and_reproducibility: I had some concern about the technical novelty of the proposed framework compared to prior work by (Part et al) (see details in the previous section). The theoretical results are accompanied by the complete proofs to my understanding; the implementation details for the experiments are nicely provided.
The code is not published in the supplementary.
summary_of_the_review: Overall I think this paper has some good idea in modeling the temporal dynamics in time series with multiple neural agents. However there are some concern about the novelty of this framework in comparison to the prior works. 
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1571/Reviewer_cyPc
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: In this paper, the authors propose a novel multi-agent based neural stochastic differential equations for time series forecasting. The proposed model is trained using deep neural fictitious play and agents will make a cooperative prediction to minimize the forecasting errors. Theoretical analysis shows that the Nash equilibrium could be reached by this way. Experimental results show efficacy of the proposed method compared with DE based continuous dynamical models.
strength_and_weaknesses: Strength:
By leveraging existing work on multi-agent stochastic differential equations and recent work on neural differential equations for time series forecasting, the paper looks pretty strong on theoretical results with algorithm convergence guarantees.

Weaknesses:
There are no characteristics of time series like trend and seasonality utilized in the modeling.  It is not convincing how well the proposed method works compared with state-of-the-art time series forecasting algorithms.

clarity,_quality,_novelty_and_reproducibility: Overall, the paper is well written and clear. Regarding novelty, It is the first paper connecting the multi-agent game theory with time series forecasting which sheds some light on this direction.
Reproducibility is unknown due to non-availability of the code.
summary_of_the_review: Major issues:
1. For proposition 1, it’s not clear for me how the Nash equilibrium is reached at each continuous time t. Let’s say if the time series is a non-stationary random process with heteroskedasticity, agents may need time to track changes in the time series and how to guarantee the existence of the optimal action set which makes eq. (14) hold is not clear to me. Furthermore, in the proof of the proposition, the authors starts from the deviation of Hamiltonian of each individual agent as shown in eq. (A.15), it’s not clear how optimal of the individual agent lead to the stochastic optimality of all agents as a system in eq. (A.53)
2. It is not clear what actions and states are for agents although the authors said X is decision and \alpha is action. It is recommended to define all relevant concepts in one paragraph or show a diagram of the system with all the components: agents, states, actions, rewards, loss function and how losses propagate backwards.
3. As defined in Eq. (1), each agent has a start state based on the past observations. Since differential equations are sensitive to initial conditions, the authors proposed temporal delay to resolve this issue. Will the temporal delay affect the equilibrium or convergence theoretically?
4. In the experiments, the authors only compared the proposed method with DE based continuous dynamical models. In practice, non-DE based models like transformer, deepar, N-beats and informer showed leading edge performance in time series forecasting. Comparisons with these state-of-the-art algorithms will make this work more convincing and applicable to real applications.
5. What is Hess in Eq. (14)? The authors are encouraged to go through all the equations and define all variables shown in this papers.

correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper1571/Reviewer_jXp9
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper proposes a novel multi-agent based neural stochastic differential equations, with the goal of analyzing the time series prediction through the lens of cooperative differential games for explainability, as well as the improved accuracy. The framework includes a  gradient descent based deep neural fictitious play to approximate the Nash equilibrium and the paper presents theoretical results assure the convergence.
strength_and_weaknesses: Strengths

- Improvements in task accuracy on benchmarks. 

- Detailed analyses on the important components.

- Cooperative game idea for time series modeling is somewhat novel.


Weaknesses

- Explainability is proposed as a value proposition but is not well demonstrated. It is unclear how the users can benefit from such explainability capability.

- Experimental section is weak. There are no results on complex multivariate datasets. Hyperparameter tuning on validation is unclear. No uncertainity around the results across experimental runs is presented, so it is hard to infer statistical significance. 

- The connection of the theoretical results with the proposed method is weak and poorly explained.
clarity,_quality,_novelty_and_reproducibility: Overall, the paper writing is okay, but there is definitely room for improvement. Theoretical sections are not well connected with the proposed framework. Including a pseudocode for training and inference would help. Figures are not high quality. The core method explanations can be much more clear. 

The quality of the work is pretty good, but the paper needs substantial improvements. 

The work is novel, contributing to the known methods in significant ways.

Reproducibility is unclear, not every detail is in the paper. 
summary_of_the_review: I do not think the paper would pass the acceptance bar yet. 

- Explainability is proposed as a value proposition but is not well demonstrated. It is unclear how the users can benefit from such explainability capability.

- Experimental section is weak. There are no results on complex multivariate datasets. Hyperparameter tuning on validation is unclear. No uncertainity around the results across experimental runs is presented, so it is hard to infer statistical significance. 

- The connection of the theoretical results with the proposed method is weak and poorly explained.

- Proposition 1 is not clearly tied with the method. 

- The background section is too long.

- I suggest including the pseudocode of training and inference.

- Section 3 is referred as deep but unclear where the depth comes from.



correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold