Paper: The KNN Score for Evaluating Probabilistic Multivariate Time Series Forecasting (ICLR-2024)

Reviewer comment (Evaluation):
While simple MAD, MSE, and CRPS may be oblivious to correlations or regularity of forecasts, they reflect the key desiderata of the forecasts stemming from the task: that they are close to the ground truth. As a model better captures correlations, simple metrics like MSE will also vanish. In other terms, one does not require metrics to be highly representative, but rather consistent and proper. For added complexity in metrics, there must be an argument that these metrics result in higher statistical power under realistic finite sample constraints and forecasting scenarios (realistic true distributions). Given the inconclusivity of the paper's empirical findings and the lack of a framework for tuning the KNNS parameter, this bar does not appear to have been cleared for KNNS.

Author response:
**“While simple MAD, MSE, CRPS may be oblivious to correlations or regularity of forecasts, they reflect the key desiderata of the forecasts stemming from the task: that they are close to the ground truth. As the model is better able to capture correlations, one expects that simple metrics like MSE will also vanish.”**

A probabilistic prediction ideally represents the uncertainty of the true data distribution correctly. Assume a distribution with two modes of equal size, then the MSE would be ideal if the model always predicts the mean of the distribution, where the density could be zero. A probabilistic univariate score would indeed vanish, e.g. an optimal probabilistic multivariate model would receive the optimal CRPS score, but the reverse is not true: a model that is optimal under CRPS does not have to produce the correct multivariate structure. Therefore, more complex scoring rules are needed to properly evaluate multivariate time series forecasts.

Reviewer comment (Evaluation):
Motivated by kNN density estimation, the authors introduce the L2 distance to the k-th nearest neighbor as the model selection metric, despite high dimensionality. This could result in notoriously high sampling variance depending on the true data distribution, and it makes the magnitude of the metric dependent on the sample size (in the paper, "ensemble"). In order to compare two models, one would need to compare them on the same number of sampled trajectories, otherwise the metric would be invalid.

Author response:
**“this makes the magnitude of the metric dependent on the size of the sample (in the paper, "ensemble"). i.e., in order to compare two models one would have to compare them on the same number of sampled trajectories or the metric would be invalid otherwise.”**

This is true. In practice, these would have to be defined for a particular benchmark, such as the parameters $p$ for the energy score or the variogram score, or $\alpha$ for the MSIS. We agree that deriving a guideline for choosing these parameters will be important for practical applicability. However, such a guideline would by necessity be scenario-dependent, since different applications will have vastly different prediction dimensionalities. We plan to investigate this in future work.

Reviewer comment (Evaluation):
The KNN score's intuition is that the best model places a forecast close in L2 distance to the ground truth. For a univariate forecast, this is equivalent to saying that the k-th best forecast in an ensemble, measured by squared error, has low squared error. Setting k = 1 is equivalent to choosing the model with the "best in hindsight" MSE. Thus, KNNS does not appear to measure the quality of distributional forecasts, a desired property set out in the paper, but only point forecast error.

Author response:
**“Finally, the KNN score's intuition is that the best model is able to place a forecast close in L2 distance to the ground truth. Note that for a univariate forecast this is equivalent to saying that the k-th best forecast in an ensemble, measured in squared error, has low squared error. Setting k=1, this is equivalent to choosing the model with the "best in hindsight" MSE. In other words, KNNS does not appear to measure the quality of distributional forecasts, a desired property set out in the paper, but only point forecast error. ”**

Evaluating based on a "best in hindsight" MSE is a better option for distributional predictions than the standard MSE, because a method that only produces a point prediction would perform worse than a method that produces multiple predictions based on a distribution. An optimal model under a "best in hindsight" MSE would spread its predictions over the area of possible futures, with more predictions in areas where it expects a higher probability.

Reviewer comment (Experiments):
The introduction of the random projections could be better motivated. Multiple random projections sampled and with sufficient latent dimension would be justified only for very high dimensionality tasks such as spatiotemporal forecasting in earth sciences, etc., which does not appear to be the case with the experimental setup.

Author response:
**“Multiple random projections sampled and with sufficient latent dimension would be justified for only very high dimensionality tasks such as spatiotemporal forecasting in earth sciences, etc. This doesn't appear to be the case with the experiment setup.”**

Many multivariate time series applications (e.g., stock market or energy forecasting) consist of tens or even hundreds of individual time series with a forecast horizon of about 10-50 time steps. Flattening such a forecast results in forecast vectors of several hundreds to thousands of dimensions, which is far too much for a valid density estimate with a manageable number of samples.

Reviewer comment (Theory):
The introduction of KNNS is somewhat counterintuitive. The paper first reports that out-of-sample likelihood (i.e., perplexity) is a desirable metric, but due to practicality immediately moves to KNN distance. This link should be substantiated.

Author response:
**“The introduction of KNNS is somewhat counterintuitive. The paper first reports that out of sample likelihood (i.e., perplexity) is a desirable metric; but due to its practicality immediately moves to KNN distance. I believe this link should be substantiated.”**

We start with the logarithmic score, which computes the log density of the prediction model at the observation. Then we replace the model density, which is not evaluable for many models, by a KNN density estimate. For $\mathbf x, \mathbf y \in \mathbb R^D$ and if $V_D$ is the volume of the unit sphere in $D$ dimensions, this results in $\log \frac{k}{|\mathcal X|V_D\|\text{NN}_{\mathcal X}^k(\mathbf y) - \mathbf y\|^D}$. Given a certain evaluation scenario, $D$, $k$, $|\mathcal X|$ and $V_D$ are constant, so to simplify the computation we omit them for our score. The result is the negative log distance to the kth-nearest neighbor.

Reviewer comment (Evaluation):
Why didn't you consider using the average of the first k distances to decrease variance?

Author response:
**“Why didn't you consider using the average of the first k distances to decrease variance?”**

We have actually thought about averaging over the closest $k$ neighbors, but we were worried that this would deviate even further from the idea of combining a density estimate with the log score. Instead of computing the score with $k=3$, one could compute the mean distance with $k=5$, or as an alternative the median distance with $k=5$. We would choose the latter, as this corresponds to the original score with $k=3$.

Reviewer comment (Theory):
The proposed method is introduced in Section 3, but it is unclear whether it qualifies as a proper scoring rule. If it does, it is essential to provide a formal proof.

Author response:
We agree, that a proof of propriety would be ideal, we only measure this property with our experiments. It should be sufficient to prove, that the inner part of the KNN score is a valid density estimate, do you agree? Because replacing $p(x)$ of the strictly proper logarithmic score with a valid density estimate should result in a proper scoring rule.

Reviewer comment (Writing):
The score's definition involves both a density model and a score, and the relationship between them needs to be clarified.

Author response:
We start with the logarithmic score, which computes the log density of the prediction model at the observation. Then we replace the model density, which is not evaluable for many models, with a KNN density estimate. For $\mathbf x, \mathbf y \in \mathbb R^D$ and if $V_D$ is the volume of the unit sphere in $D$ dimensions, this results in $\log \frac{k}{|\mathcal X|V_D\|\text{NN}_{\mathcal X}^k(\mathbf y) - \mathbf y\|^D}$. For a given evaluation scenario, $D$, $k$, $|\mathcal X|$ and $V_D$ are constant, so to simplify the computation we omit them for our score.

Reviewer comment (Theory):
A reference for the energy score proof in the Appendix should be included.

Author response:
What reference are you missing here? We refer to the proof in Appendix A in the second sentence of Section 2.5.1. The proof is our own work, so there is no reference to another paper.

Reviewer comment (Novelty):
The paper's contributions concerning scoring rules for multivariate distributions are unclear, and a recent and important reference, "Regions of Reliability in the Evaluation of Multivariate Probabilistic Forecasts" (ICML 2023), appears to be missing.

Author response:
Thanks for pointing out this work! It is indeed highly relevant.

Reviewer comment (Reproducibility):
There appear to be various approximations and challenges in implementing the method, such as not enforcing orthogonality of rows and the issue of sampling enough prediction vectors due to the curse of dimensionality. It is unclear how these challenges and design choices affect the proposed score.

Author response:
We have observed in preliminary experiments that omitting the orthogonality requirement has at most very small effects that are overshadowed by other factors. The number of prediction vectors is one such factor; we generally use 100 vectors, as this is often the default in related work. As these certainly depend on the internal data structure and also on the model prediction: do you think a broad experimental setup with various models and data sets would be sufficient to quantify deviations regarding the impact of omitting the orthogonality of rows and the number of samples?

Reviewer comment (Evaluation):
The statement, "Only the green one mimics dependencies between time steps correctly," is disputable. Having all realizations in the predictive region does not necessarily imply a correct capture of true uncertainty.

Author response:
Figure 1 should be an intuitive example of why a scoring rule should not score each point independently. The observed bivariate time series repeats the same values twice, and there is a clear correlation between the two individual series.The implicit assumption, of course, is that this is always the behavior of the true data. Now, if you look at a particular time step of a particular series, each model has exactly the same deviations around the observation, so the uncertainty around each point is the same. This is why scoring rules like CRPS score them all the same. However, assuming that the temporal and inter-series correlation structure shown by the observation is representative of the true data, only the green forecast properly reflects these dependencies. It is much more likely to observe the green forecast than the orange or red forecast.

Reviewer comment (Evaluation):
The assertion, "As a result, these metrics should be avoided when assessing probabilistic predictions," needs further elaboration and support. The authors should clarify under what circumstances these metrics should be avoided and why.

Author response:
An ideal probabilistic forecast should match the (unknown) distribution of the data. An optimal model for MSE or MAE does not need to predict a distribution at all, predicting the mean or median is sufficient for an optimal score. For example, if the data distribution has two equal-sized modes, the density at the mean could be zero, but would be optimal if evaluated by MSE.