Paper: ClimODE: Climate and Weather Forecasting with Physics-informed Neural ODEs (ICLR-2024)

Reviewer comment (Experiments):
Limited Performance and Comparison: The proposed model, although surpasses vanilla Neural ODE and ClimaX, still cannot compete with IFS (ECMWF NWP), let alone bigger models such as FourCastNet and Pangu-Weather that have already reported better results than NWP. There lacks enough comparison to other models in general, such as Adaptive Fourier Neural Operator (practiced by FourcastNet), previous weather forecasting methods like [1, 2], even models for similar tasks such as spatio-temporal traffic forecasting or video forecasting, etc. Comparison to Neural ODE is more of an ablation study and ClimaX is a weak baseline marginally better than ResNet. Comparing only with ClimaX is definitely not convincing enough.

Author response:
**Comparison with FourCastNet.** Thank you for the suggestion. We could not compare to PanguWeather (no public code, model files do not support our data resolution) or GraphCast (public repository seemed incomplete). However, we include an additional experiment compared to FourCastNet. To have comparable results, we re-trained the FourCastNet on our dataset and used their recommended training regime and hyperparameters. 

Our results reinforce the efficacy of ClimODE. We note that our method is more accurate in terms of RMSE (in fact, quite significantly especially for the geopolitical variable z) and largely competitive in terms of ACC. We will include these results in the paper.

|          |                   | RMSE        |                        | ACC         |          |
|----------|-------------------|-------------|------------------------|-------------|----------|
| Variable | Lead Time (hours) | FourCastNet | ClimODE                | FourCastNet | ClimODE  |
| z        | 6                 | 149.4       | **102.9**   $\pm$ 9.3 | 0.73        | 0.99     |
|          | 12                | 217.8       | **134.8**   $\pm$ 12.3     | 0.58        | 0.99     |
|          | 18                | 275.0       | **162.7**   $\pm$ 14.4     | 0.58        | 0.98     |
|          | 24                | 333.0       | **193.4**   $\pm$ 16.3     | 0.58        | 0.98     |
| t        | 6                | 1.18        | **1.16**   $\pm$ 0.06      | 0.99        | 0.97     |
|          | 12               | 1.47        | **1.32**   $\pm$ 0.13      | 0.99        | 0.96     |
|          | 18               | 1.65        | **1.47**   $\pm$ 0.16      | 0.99        | 0.96     |
|          | 24               | 1.83        | **1.55**   $\pm$ 0.18      | 0.99        | 0.95     |
| t2m        | 6                | 1.28        | **1.21**   $\pm$ 0.09      | 0.99        | 0.97     |
|          | 12               | 1.48        | **1.45**   $\pm$ 0.10      | 0.99        | 0.96     |
|          | 18               | 1.61        | **1.43**   $\pm$ 0.09      | 0.99        | 0.96     |
|          | 24               | 1.68        | **1.40**   $\pm$ 0.09      | 0.99        | 0.96     |
| u10        | 6                | 1.47        | **1.41**   $\pm$ 0.07      | 0.93        | 0.91     |
|          | 12               | 1.89        | **1.81**   $\pm$ 0.09      | 0.91        | 0.89     |
|          | 18               | 2.05        | **1.97**   $\pm$ 0.11      | 0.89        | 0.88     |
|          | 24               | 2.33        | **2.01**   $\pm$ 0.10      | 0.89        | 0.87     |
| v10        | 6                | 1.54        | **1.53**   $\pm$ 0.08      | 0.95        | 0.92     |
|          | 12               | 1.81        | 1.81   $\pm$ 0.12      | 0.91        | 0.89     |
|          | 18               | 2.11        | **1.96**   $\pm$ 0.16      | 0.88        | 0.88     |
|          | 24               | 2.39        | **2.04**   $\pm$ 0.10      | 0.85        | 0.86     |

Reviewer comment (Theory):
Limited Physical Complexity: The partial differential continuity equation is indeed a fundamental concept in fluid dynamics as it describes how quantities such as mass, moisture, or energy are transported and conserved in the atmosphere over time and space. What makes it less predictive in the real world is that this equation comes with assumptions such as the homogeneity and isotropy of the fluid. In cases where these assumptions are violated, the equation might not hold perfectly. I am not certain whether relying too much on this physical equation would be optimal for modeling the complex dynamics of weather, especially in finer-granular resolution with more weather factors and potentially larger noises.

Author response:
These are all great points, and we agree. We believe that a performant climate model likely needs to strike a balance between data-driven deep learning (often black-box) practises, and theoretical modeling and understanding of the underlying physics. Earlier methods, such as PanguWeather or GraphCast, lean quite heavily on the deep learning side. In contrast, our contribution is to combine neural networks with advection equations, which makes our modeling significantly more principled over relying solely on deep learning. \\

To our knowledge, our study is the first method that introduces principled continuous-time dynamics to the problem, and further research is needed to identify the extent of physics to be injected in the models. In our model the advection primarily contributes mass conservation, which we show empirically to be a very useful inductive bias.

Reviewer comment (Theory):
Limited Physical Understanding: In all, the partial differential continuity equation is a general-form conservation equation rather than any specific equations explaining each weather factor. Other than that, because neural ODE is essentially still a black-box model due to using neural network, it remains hard to know, for example, the relationships and the interactions between different weather variables.

Author response:
Great remarks, and we again agree. The advection conservation equations have two roles in our study. On one hand, their foundation in physics allows us to interpret and analyse the proposed model from new perspectives (to the deep learning community), as well as  understand the model predictions in terms of divergence, transport and gradient fields (as shown in Figure 1 and 2). On the other hand, the induced `mass' conservation ensures that predictions do not lose or gain value, stabilising them and providing us with improved empirical accuracy. We are excited to study the physical modeling choices further in future, and experiment with diffusion, source and sink fields as well.

The interdependency of climate variables is another incisive question. To demonstrate the couplings of quantities (i.e., wind, temperature, potential), we plot below the emission model  $\mathbf{u}^{\mathrm{pred}}(\mathbf{x},t) \in \mathbb{R}^5$ pairwise densities averaged over space $\mathbf{x}$ and time $t$. These effectively capture the correlations between quantities in the simulated weather states. These show that temperatures (t,t2m) and potential (z) are highly correlated and bimodal; the horizontal and vertical wind direction are independent (u10,v10); and there is little dependency between the two groups. We will include these plots in the appendix.

Plot: https://postimg.cc/0McVK5r5

Reviewer comment (Experiments):
If the several ideas behind ClimODE is effective as suggested by the ablation study, I would really be interested to see its performance in a large model compared to Pangu or FourcastNet, for example. Those benchmarks are exceptional since they excel NWP results in many cases and have been tested/used even for recent disaster predictions. It is just like the LLM that, if it becomes powerful enough, everyone will be amazed and will use it. If the ultimate goal of this paper is to propose a strong SOTA model against existing models, it is strongly recommended to compare with those large models and consider adapting to this line in the future research.

Author response:
We agree and are keen to compare these models as well. However, there are some practical hurdles: the large DL models have been trained with higher resolution data, and we cannot directly obtain comparable prediction trajectories due to different test data.

These large weather models demonstrate significant scale, requiring substantial computational resources, extensive pre-training, and other prerequisites. These requirements impose practical limitations on the training process. We argue that there is also value in more compact models (in our case up to 100x less parameters) that allow rapid and more accessible development for the community.

We would also like to emphasize that the primary goal of our work is to demonstrate the benefits of incorporating physical inductive biases in neural networks, and present a qualitatively improved model that has (i) solid foundations in physics, (ii) outputs principled uncertainty distributions, and (iii) is fundamentally continuous-time. We have also demonstrated good empirical performance. However, our primary goal is not to best other large climate foundations models in benchmarks, and we have chosen instead to focus on principled method developments and see perhaps in future these ideas be integrated into the large foundation models as well (indeed, recently NowCastNet incorporated autoregressive advection equations to its Transformer pipeline).

Reviewer comment (Theory):
"Closed system assumption implies value-preserving manifold" is unclear and needs more justification. While the conclusion that: $$\mathbb{E}_x[u_k(\mathbf{x},t)]=\text{const.}$$ sounds intuitive, it's not justified rigourously. The expectation is taken with respect to which density? I would expect the quantity to be preserved would be $\int_{\Omega} u_k(\mathbf{x},t)dV$, since that's the total quantity over the whole "volume" you're considering which should be preserved (e.g. when $u_k=\rho$, the integral over the volume becomes the mass). In any case, since this is a Machine Learning submission, it would be better if this part is thoroughly justified.

Author response:
Thanks for pointing this out. Indeed, our expectation notation is imprecise, and we should have written a spatial integral $\int u_k(\mathbf{x},t) d\mathbf{x}$ as you suggest. We will fix this in the paper. We also analysed our model, and found out that the total spatial value is practically constant over time over the ODE forward solver (it shows a flat line over time). We will include this plot in the appendix.

Plot: https://postimg.cc/w709xD9n

Reviewer comment (Experiments):
Authors mention that other deep learning methods lacked open-source code, while that's true for PanguWeather (who only provide a pseudo-code "implementation"), it's not for the others:  
- FourCastNet: https://github.com/NVlabs/FourCastNet  
- GraphCast: https://github.com/google-deepmind/graphcast  
Given that GraphCast seem to outperform ClimaX, it would have been good to compare against it as well and also FourCastNet since it's one of the first papers to perform weather forecasting on such a scale.

Author response:
**Comparison with FourCastNet.** Thank you for the suggestion and for providing links to the repositories. PanguWeather has made their pre-trained model files available, but they use a different spatial resolution, so we cannot compare with them. FourCastNet similarly uses a different resolution, but we were able to retrain the FourCastNet model using our dataset for fair comparison and using their recommended hyperparameters and training.

We report below the results that show that in general ClimODE has better RMSE accuracy (especially significant for the geopotential variable z), while the performance with respect to ACC metric is largely comparable (same or slightly worse for ClimODE). We will include this experiment in the paper based on your feedback.

|          |                   | RMSE        |                        | ACC         |          |
|----------|-------------------|-------------|------------------------|-------------|----------|
| Variable | Lead Time (hours) | FourCastNet | ClimODE                | FourCastNet | ClimODE  |
| z        | 6                 | 149.4       | **102.9**   $\pm$ 9.3 | 0.73        | 0.99     |
|          | 12                | 217.8       | **134.8**   $\pm$ 12.3     | 0.58        | 0.99     |
|          | 18                | 275.0       | **162.7**   $\pm$ 14.4     | 0.58        | 0.98     |
|          | 24                | 333.0       | **193.4**   $\pm$ 16.3     | 0.58        | 0.98     |
| t        | 6                | 1.18        | **1.16**   $\pm$ 0.06      | 0.99        | 0.97     |
|          | 12               | 1.47        | **1.32**   $\pm$ 0.13      | 0.99        | 0.96     |
|          | 18               | 1.65        | **1.47**   $\pm$ 0.16      | 0.99        | 0.96     |
|          | 24               | 1.83        | **1.55**   $\pm$ 0.18      | 0.99        | 0.95     |
| t2m        | 6                | 1.28        | **1.21**   $\pm$ 0.09      | 0.99        | 0.97     |
|          | 12               | 1.48        | **1.45**   $\pm$ 0.10      | 0.99        | 0.96     |
|          | 18               | 1.61        | **1.43**   $\pm$ 0.09      | 0.99        | 0.96     |
|          | 24               | 1.68        | **1.40**   $\pm$ 0.09      | 0.99        | 0.96     |
| u10        | 6                | 1.47        | **1.41**   $\pm$ 0.07      | 0.93        | 0.91     |
|          | 12               | 1.89        | **1.81**   $\pm$ 0.09      | 0.91        | 0.89     |
|          | 18               | 2.05        | **1.97**   $\pm$ 0.11      | 0.89        | 0.88     |
|          | 24               | 2.33        | **2.01**   $\pm$ 0.10      | 0.89        | 0.87     |
| v10        | 6                | 1.54        | **1.53**   $\pm$ 0.08      | 0.95        | 0.92     |
|          | 12               | 1.81        | 1.81   $\pm$ 0.12      | 0.91        | 0.89     |
|          | 18               | 2.11        | **1.96**   $\pm$ 0.16      | 0.88        | 0.88     |
|          | 24               | 2.39        | **2.04**   $\pm$ 0.10      | 0.85        | 0.86     |

Reviewer comment (Theory):
Authors claim that the vaue-preserving manifold (that emanates from the closed system assumption), presents a strong inductive bias for long-term forecasts, yet, using Euler scheme to solve the ODE is known to be unstable and it'll especially not conserve the quantity we want preserved, so that inductive bias is no longer enforced. It would be better to include that limitation in the paper and acknowledge that while it is a strong inductive bias, it's hard to enforce in practice. This is further seen from Table 2 which shows that the error does increase dramatically with lead time and that suggests that ClimODE is better at inferring the true physics but not in mitigating the error propagation in long-term forecasts.

Author response:
Thanks for pointing this out. To empirically study this, we analysed how our current model retains the mass-conservation assumption and computed the integrals $I_{k,t} = \int u_k(\mathbf{x},t)d\mathbf{x}$ over time and quantities. We found out that the value is constant over time up to order of $10^{-12}$. Plot link: https://postimg.cc/w709xD9n

Thus it seems that our choice of second-order ODE with Euler with 1 hour increment is a good compromise in terms of accuracy and efficiency, but we agree that higher precision solvers should be preferred. We hypothesise that Euler is sufficient since our dynamics are relatively smooth (eg. Fig 6): the dynamics $\dot{u}_k$ largely models the spatiotemporal trends in climate, while we offload the day-night fluctuations to the instantaneous emission model.

Reviewer comment (Evaluation):
While Figure 6 shows qualitatively the soundness of the predicted bias and variance, there's no quantitative approach that evaluates the quality of the bias and variance output by the model. A metric like CRPS (Continuous Ranked Probability Score) can showcase that.

Author response:
**Quantifying quality of bias and variance outputs, and CRPS plots.** Thank you for another great suggestion. It essentially boils down to whether the predicted distributions $\mathcal{N}(u_k +  \mu_k, \sigma_k^2)$. These plots indicate that the emission model is highly aligned with data, and does not indicate any immediate biases or skews. These results are averaged over space and time, and spatially local variations are still possible. The mean $\mu$ plots show that means match data well. The standard deviation $\sigma$ plots show some bimodality of predictions with either no or moderate amount of uncertainty.

We also evaluated our model using CRPS (Continuous Ranked Probability Score). Our results show that ClimODE can indeed produce good estimates of variance and bias. We will include all these plots in the Appendix.

CRPS Plot: https://postimg.cc/RqGc04NG

Correlation Plots: https://postimg.cc/6TBmSWkh, https://postimg.cc/v47npyFW

Reviewer comment (Presentation):
Table 1 says that NowCastNet doesn't provide uncertainty estimation, but it's a generative model which can provide such estimates and in general approximate the true underlying distribution.

Author response:
**ClimODE vs. NowCastNet with respect to uncertainty estimation.**  Thank you for the opportunity to clairfy this important point. NowCastNet indeed operates as a generative model and to obtain uncertainty estimates one requires multiple samples and forward passes to obtain a scatter of outputs, whose summary statistics can be computed. However, for ClimODE, a single forward pass suffices for accurate uncertainty estimation. That is, our approach allows us to obtain reliable uncertainty estimates in a single pass without the need for repeated sampling that incurs extra costs and approximation errors.

Reviewer comment (Reproducibility):
In section 3.6, how is $\tilde{\dot{u}}(t_0)$ numerically apprixmated from past states?

Author response:
We compute $\tilde{\dot{u}}(t_0)$ by utilizing torchcubicspline (https://github.com/patrick-kidger/torchcubicspline) to fit data-points $\{u_k(t_0 -2),u_k(t_0 -1),u_k(t_0) \}$ in order to get a smooth derivative approximation.

Reviewer comment (Experiments):
Why not use different time-resolutions for solving the ODE and assessing their effect? Same goes for the ODE-solver. Given that you state that Runge-Kutta can be used with a low computational cost, it would add more quality to the paper overall if you include it as well.

Author response:
We utilize one-hour time resolutions for solving the ODE system, and finer time-stepping or higher precision solvers can (and perhaps should be) used. We settled on a compromise of 1 hour increment for a second-order ODE that already provided us with good efficiency, and also empirically maintained the mass conservation (see above).

Reviewer comment (Experiments):
How long does it take to train?

Author response:
For global forecast, We train our model on a single NVIDIA Tesla V100 64GB GPU, without any parallelization techniques for 300 epochs, and it takes around 20--24 hours.

Reviewer comment (Theory):
For the model of FLOW VELOCITY (section 3.2), as we already know $\dot{\mathbf{v}}_k(\mathrm{x}, t) = \ddot{u}_k(\mathrm{x}, t)$, why we still need to parameterize it? Is it because the computation of $\ddot{u}_k(\mathrm{x}, t)$ is too costly? There are also many methods that could approximate $\ddot{u}_k(\mathrm{x}, t)$ based on $\dot{u}_k(\mathrm{x}, t)$. More discussion or claims are encouraged.

Author response:
**Need to parameterize the flow velocity.** Thank you for raising an excellent point. In an advection system the climate quantities $u(\mathbf{x},t)$ are transported (moved and redistributed) around by a flow velocity $\mathbf{v}(x,t)$, which is a vector field that points where the climate `mass' is moving. Given a known starting state $u$, the system evolution is fully determined by the velocity $\mathbf{v}$ by solving eq (2) forward in time.

The velocity field function is generally unknown, and we thus learn it from data by parameterising the the velocity function as a neural network $\mathbf{v}(\mathbf{x},t | \theta)$. We also note that the remark that $\dot{\mathbf{v}}_k = \ddot{u}_k$ was a typo, and in general we cannot solve for $\mathbf{v}$ by, e.g., interpolating the value $u$.  

Apologies for any confusion: we have fixed this typo and our model equations remain unaffected.

Reviewer comment (Theory):
The model treats the advection PDE as independent for each task or quantity. However, in the intricate tapestry of climate dynamics, various quantities are interdependent. For instance, wind patterns can influence temperature fluctuations. The paper doesn't clearly illustrate how the proposed method addresses these inherent inter-correlations.

Author response:
**Explicating how ClimODE accounts for variable correlations.** This is another great point that we elucidate here. We present our equations for a single quantity $k$ at a time (for instance, equation (2)) because the convenient vector calculus notation $\dot{u}_k - \nabla \cdot (u_k \mathbf{v}_k)$ applies only to scalar $u$, and a `batch' version $\dot{\mathbf{u}}$ does not admit elegant notation. Nevertheless, our model is still able to consider the entire state vector $\mathbf{u} = (u_1, \ldots, u_K)$ at the same time, and all $K=5$ climate quantities are affected by each other. This can be seen in equation (4), where the velocity change $\dot{\mathbf{v}}_k = f( \mathbf{u}, \ldots)$ of quantity $k$ depends on all state variables, and thus subsequently $\dot{u}_k$ also depends on all state variables. Thus, for instance, wind can affect temperature and vice versa.

To demonstrate the emerging couplings of quantities (ie. wind, temperature, pressure potential), we plot below the emission model  $\mathbf{u}^{\mathrm{pred}}(\mathbf{x},t) \in \mathbb{R}^5$ pairwise densities averaged over space $\mathbf{x}$ and time $t$. These effectively capture the correlations between quantities in the simulated weather states. For example, these densities reveal that temperatures (t,t2m) and potential (z) are highly correlated and bimodal; the horizontal and vertical wind directions are independent (u10,v10); and there is little dependency between the two groups. We will include these plots in the appendix.

Plot: https://postimg.cc/0McVK5r5

Reviewer comment (Presentation):
A couple of Figures (e.g Figure 5) would benefit from a longer caption/description

Author response:
Thanks for the feedback. We will add more details to the captions of fig 3 (the pipeline), fig 5 (ODE ablations) and fig 6 (emissions over time) to make them more descriptive. For instance, fig 5 caption will be expanded to 

**The effect of including different ClimODE components on RMSE**. An ablation showing how enhancing the vanilla neural ODE (blue) incrementally with advection (orange), global attention (green) and emission (red) leads to progressively better RMSE scores. We note that that advection yields most improvement in accuracy, while attention turns out to be the least important.

Reviewer comment (Experiments):
There is Weatherbench 2 available now, maybe too recent to be included in this submission, but should be mentioned in future work: https://arxiv.org/abs/2308.15560, https://sites.research.google/weatherbench/

Author response:
Many thanks for the reference. We will include a citation and position appropriately in the paper.

Reviewer comment (Reproducibility):
Are any of the competing methods such as GraphCast, PanguWeather, FourCastNet now available?

Author response:
PanguWeather have not published their code, but released a pre-trained model file that can be used to forecast. However, we cannot evaluate our data with PanguWeather since, differently from us, their model file supports only a very high spatial resolution Earth grid of size (720, 1441). 

GraphCast have made a public implementation, but we found it to be incomplete and so were not able to make it work despite our efforts.

**Comparison with FourCastNet.** FourCastNet also trains with a higher spatial resolution, so we cannot use their pretrained models either. Fortunately, FourCastNet has released their implementation and recommended setting of hyperparameters. Based on your feedback, we produced a new experiment where we trained the FourCastNet model according to their training scheme on our dataset.

We report these new results below. We note that ClimODE compares favorably with FourCastNet in terms of RMSE: in particular, our model achieves significantly better RMSE for the $z$ (i.e., geopotential) variable. The ACC results are largely competitive across variables for the two methods.   We will include these results in the paper.


|              |                   | RMSE        |                        | ACC         |          |
|----------|-------------------|-------------|------------------------|-------------|----------|
| Variable | Lead Time (hours) | FourCastNet | ClimODE                | FourCastNet | ClimODE  |
| z        | 6                 | 149.4       | **102.9**   $\pm$ 9.3 | 0.73        | 0.99     |
|          | 12                | 217.8       | **134.8**   $\pm$ 12.3     | 0.58        | 0.99     |
|          | 18                | 275.0       | **162.7**   $\pm$ 14.4     | 0.58        | 0.98     |
|          | 24                | 333.0       | **193.4**   $\pm$ 16.3     | 0.58        | 0.98     |
| t        | 6                | 1.18        | **1.16**   $\pm$ 0.06      | 0.99        | 0.97     |
|          | 12               | 1.47        | **1.32**   $\pm$ 0.13      | 0.99        | 0.96     |
|          | 18               | 1.65        | **1.47**   $\pm$ 0.16      | 0.99        | 0.96     |
|          | 24               | 1.83        | **1.55**   $\pm$ 0.18      | 0.99        | 0.95     |
| t2m        | 6                | 1.28        | **1.21**   $\pm$ 0.09      | 0.99        | 0.97     |
|          | 12               | 1.48        | **1.45**   $\pm$ 0.10      | 0.99        | 0.96     |
|          | 18               | 1.61        | **1.43**   $\pm$ 0.09      | 0.99        | 0.96     |
|          | 24               | 1.68        | **1.40**   $\pm$ 0.09      | 0.99        | 0.96     |
| u10        | 6                | 1.47        | **1.41**   $\pm$ 0.07      | 0.93        | 0.91     |
|          | 12               | 1.89        | **1.81**   $\pm$ 0.09      | 0.91        | 0.89     |
|          | 18               | 2.05        | **1.97**   $\pm$ 0.11      | 0.89        | 0.88     |
|          | 24               | 2.33        | **2.01**   $\pm$ 0.10      | 0.89        | 0.87     |
| v10        | 6                | 1.54        | **1.53**   $\pm$ 0.08      | 0.95        | 0.92     |
|          | 12               | 1.81        | 1.81   $\pm$ 0.12      | 0.91        | 0.89     |
|          | 18               | 2.11        | **1.96**   $\pm$ 0.16      | 0.88        | 0.88     |
|          | 24               | 2.39        | **2.04**   $\pm$ 0.10      | 0.85        | 0.86     |

We are grateful for your constructive review. We hope that our response, including the new experimental results comparing ClimODE with FourCastNet, addresses your concerns and reinforces your support for this work.

Reviewer comment (Experiments):
Is the comparison against ClimaX fair? The main idea of ClimaX is to use pre-training, but you state you use all methods without pretraining

Author response:
Thank you for the opportunity to clarify this. We trained the ClimaX model from scratch using the same dataset as for ClimODE without pre-training to compare how much signal the two methods are able to extract from the same data. Comparing to pre-trained ClimaX would mean comparing to a model that has seen much more data than us, which would give unfair advantage to ClimaX. We acknowledge that pretraining can be effective for a practical weather predictor, and are very keen to leverage pre-training in future with larger scale implementations of our advection ODE model.

Reviewer comment (Novelty):
The statement "IFS is still far ahead of any deep learning method" doesn't really hold anymore, e.g.: https://arxiv.org/pdf/2307.10128.pdf

Author response:
Thank you very much for these  references. We will add these and update our statements about IFS.

Reviewer comment (Novelty):
Kreislers GNN work should be mentioned here as well: https://arxiv.org/pdf/2202.07575.pdf

Author response:
Thank you very much for these  references. We will add these and update our statements about IFS.