Paper: Risk Assessment and Statistical Significance in the Age of Foundation Models (ICLR-2024)

Reviewer comment (Theory):
I think more discussion on the motivation/interpretability of the proposed measures is needed. For example, why is R-SSD a relevant and interpretable measure for comparing two language models and why would metrics portfolio be a reasonable way to combine different metrics?

Author response:
We have added in the revised paper in the introduction more motivation and interpretability of the portfolio and the method  based on the SSD used in the paper.  Please see page 1-3 in the revision. The motivation for R-SSD is now depicted in Figure 1 contrasting panels (a) for quantiles (b) Tail Value at Risk that is normalized second quantiles . Second quantiles give better order between LLMs, since they assess left tails and failure modes of the models. Regarding why the portfolio reasonable, please see the points below:
**Portfolio with R-SSD consistent with chatGPT score (proxy of human evaluation)**: We have incorporated in the revised manuscript comparison on the mix-instruct between a proxy of human evaluation given by chatGPT scores that evaluates LLMs and our portfolio approach. We have discussed this comparison in Figure 1 in the paper as well as in the introduction. We see our approach based on relative SSD is well aligned with chatGPT score, while the Min Win Rate (MWR) approach used widely in LLM  leaderboards such as HELM leads to very poor alignment with ChatGPT score.
**Portfolio with R-SSD is consistent with rank aggregation of per metric rankings**:  In the revised manuscript, on the Mix-instruct dataset we show in Appendix A (page 12,        Metrics Aggregation Versus Portfolio) that Portfolio with R-SSD is consistent with rank aggregation of per metric rankings. Indeed these two approaches lead to similar ranking, which justify that portfolio is not losing the comparison power of models, but portfolio based is 7x faster than per metric based ranking (this speedup is taking into account the portfolio computation, note that in this example the number of metrics computed is 8).
   
We hope this clears up the motivation and the interpretability of both SSD and portfolio.

Reviewer comment (Theory):
Although there are CLT results for the estimators, the validity of the statistical test is lacking — in particular, the validity of the Bootstrap approximation. The CLT result is not sufficient for the validity of the test since the limiting distribution contains the unknown variance term; does Bootstrap approximation solve this problem? A theory is needed. (Please correct me if I missed anything.)

Author response:
Thanks for this question. We have addressed this in page 7 in the manuscript and in Appendix K. One can compute the asymptotic variance of our tests using estimate of the density and the CDF but this is not a good option in finite sample regime, that is why we followed previous works such as  (Ulmer et al., 2022) that used a bootstrapping heuristic to perform testing of almost first order stochastic dominance. 
For the proof of consistency of bootstrapping, classical techniques of proofs require Hadamard or Frechet differentiability of the statistics. Note that our statistics are relaxations of FSD and SSD that can be either expressed in terms of CDFs and integrated CDF respectively or with quantiles; and integrated quantiles respectively. Appendix K considers statistics with CDFs and proves the consistency of the bootstrapping using classical results in (Shao & Tu, 2012). For the statistics with quantiles, a smoothing of the statistics is needed (Shao & Tu, 2012) for a similar proof to work, we leave this for future work since the length and detail of the derivation would be beyond the scope of the current work.

Reviewer comment (Writing):
The paper could be more clear on what it's main contribution is supposed to be. Is it "relative stochastic dominance" as a concept, the application of it to LLMs, the formulation of an asymptotic test statistic?

Author response:
We have improved in the revision the motivations for addressing the question of metric combination and statistical significance jointly, as well as the contributions. We also clarified the connection between the theory part that introduces the concepts and the application in evaluating LLMs. We have improved:

  1. The presentation of the motivation and the contribution of our work in the introduction. Please see the edits in red in the revised manuscript in the introduction pages 1-3. 
 2. We gave more intuitions and introduced all concepts graphically (portfolio, FSD and SSD) in the introduction in **Figure 1** in a running example showcasing our contributions. 
 3. Throughout all theory sections we  kept reminding the reader in the theory section as suggested by the reviewer the role of each random variable as a metric evaluation or as a portfolio that is the aggregation of multiple metrics.  (Please see edits in red in the beginning of Section 2, we also simplified Section 3 and Section 4, without substantial change in the content).

Reviewer comment (Theory):
Is relative stochastic dominance stable with respect to the sets of stochastic variables involved? I.e. if I have two variables X_1 > X_2, and I add a new variable X_3, could I have X_3 > X_2 > X_1?

Author response:
Let us assume that $X_1> X_2$ this means that $\varepsilon_1=\varepsilon_{12} < \varepsilon_2= \varepsilon_{21}$. When we add $X_3$, we have:

$\varepsilon_1^{'}=\frac{1}{2}( \varepsilon_{12}+ \varepsilon_{13})$
and
$\varepsilon_2^{'}=\frac{1}{2}( \varepsilon_{21}+ \varepsilon_{23})$
And
$\varepsilon_3^{'}=\frac{1}{2}(\varepsilon_{31}+ \varepsilon_{32})$

For $X_2$ to dominate $X_1$ in the new relative order considering 3 variables, we need to have $\varepsilon_2^{'} < \varepsilon_1^{'}  $, which means that we need to have: $\varepsilon_{23} - \varepsilon_{13}< \varepsilon_{12}- \varepsilon_{21} $ Note that the right side is negative by assumption. 

Hence the condition for the order to flip  between $X_1 $ and $X_2$ is:

 $$\varepsilon_{23} - \varepsilon_{13}< \varepsilon_{12}- \varepsilon_{21} <0 $$

Hence the order of variables of $X_1$ and $X_2$ can flip if  $\varepsilon_{23} < \varepsilon_{13}$, meaning that the violation of stochastic dominance of 2 on 3 is smaller than the one of 1 on 3 and if this delta  of violation is smaller than the delta of violation between 1 and 2. 

The order is only defined relative to the considered set. As we add variables, the order is sensible to the margins defined by the delta of violation ratios. That said, the flipping condition above should not occur in practice if the stochastic dominance between adjacent variables in the ordering is strong. In our own experiments, we find that the top-ranked models tend to be better-separated and more stable, the bottom-ranked models are often failing on many tasks and do not have a strong violation ratio between them, creating more unstable lower ranks.

Reviewer comment (Novelty):
Table 2 appears to suggest that mean-risk models give virtually equivalent rankings to relative SSD, yet are easier to understand. Why should one use relative SSD instead then?

Author response:
Mean Risk Models (MRM) are effectively useful, some of them are global statistics (integrated on all percentiles) like the Gini tail, but the tail value at risk, and the mean absolute  deviation   from a quantile are computed for a given percentile $p$ . We aggregate the ranks resulting from these MRM for a given $p$ using metric based rank aggregation (using pearson distance). Note that the resulting rank depends heavily on the percentile $p$ choice. The advantage of  Relative SSD testing, is that 1) statistically it has a confidence level which MRM lacks 2) Relative SSD is a global statistic that does not need any hyperparameter choice (except the confidence level). Hence Relative SSD is more reliable. 

To further compare Relative SSD and MRM we analyze the speed of their convergence versus sample size in Appendix A (Figure 2 page 12 in the revised paper), we see that R-SSD is more stable and has lower variance in low sample regime.

Reviewer comment (Novelty):
What aspect of your method is specific to LLMs as opposed to general model comparison along multiple dimensions?

Author response:
The work motivation was LLM comparison, but as the reviewer is pointing this applicable to general model comparison along multiple dimensions, where these dimension measure risks, and in the LLM space lot of metrics are assessing those risks.

Reviewer comment (Novelty):
**The main contribution of this paper is not very clear.** I have an impression that this paper discusses two independent topics in a paper. The first topic is the statistical robustness of risk assessment, and the second is how to balance tradeoffs among various metrics to compare LLMs. Though I admire that both of them have some motivation to work on, the issues of existing work in each topic are not clearly pointed out, and as discussed in the following Weakness 2, the paper fails to provide sufficient empirical analysis to compare the proposed method in both topics. Since the connection of two topics is also not very clear, I would suggest that splitting the paper into two might be useful.

Author response:
We have improved in the revision the motivations for addressing the question of metric combination and statistical significance jointly, as well as the contributions. We also clarified the connection between the theory part that introduces the concepts and the application in evaluating LLMs. We have improved:

  1. The presentation of the motivation and the contribution of our work  in the introduction . Please see the edits in red in the revised manuscript in the introduction pages 1-3. 
 2. We gave more intuitions and introduced all concepts graphically (portfolio, FSD and SSD) in the introduction in **Figure 1** in a running example showcasing our contributions. 
 3. Throughout all theory sections we  kept reminding the reader in the theory section as suggested by the reviewer the role of each random variable as a metric evaluation or as a portfolio that is the aggregation of multiple metrics. (Please see edits in red in the beginning of Section 2, we also simplified Section 3 and Section 4, without substantial change in the content).

Reviewer comment (Evaluation):
**No quantitative or qualitative assessment criteria are given to demonstrate the effectiveness of the proposed evaluation method.** In my understanding, this paper proposes a new, statistically robust risk-assessment metric to compare two (multi-variate) probability distributions. However, the paper lacks an empirical demonstration of how the proposed method overcomes the issues of existing metrics. Given the focus of this paper, just applying the proposed metric to evaluate various LLMs is not enough. In addition, for the portfolio selection part, the paper should investigate if the proposed metric aligns with human preference more than other existing metrics to demonstrate the benefit of the proposed method.

Author response:
We have incorporated in the revised manuscript comparison on the mix-instruct between a proxy of human evaluation given by chatGPT scores that evaluates LLMs and our portfolio approach. We have discussed this comparison in Figure 1 in the paper as well as in the introduction. We see our approach based on relative SSD is well aligned with chatGPT score, while the Min Win Rate (MWR) approach used widely in LLM  leaderboards such as HELM leads to very poor alignment with ChatGPT score.
Furthemore, we give in appendix A Figure 3, a quantitative assessment of this alignment on per sample basis  between relative SSD test and chatGPT score , and  between MWR chatGPT score.  We see clearly that our approaches lead to a robust ranking that is consistent with chatGPT scores, while the MWR approach does not show any consistency with ChatGPT.

Reviewer comment (Writing):
**The manuscript is hard to follow.** The paper sometimes lacks explanations of theorems or motivations to introduce theorems. This makes understanding the main contribution of this paper quite challenging. In particular, the four points I mentioned in Questions were hard to understand from the manuscript.

Author response:
We have taken into account all the mentioned points in our revision as clarified in our response to the first point in the rebuttal.

Reviewer comment (Theory):
Isn't Eq. (4) the definition of conditional value at risk (CVaR)? (because it seems that (4) takes expectation of the quartile value under p% quartile). Then, deriving (4) seems straightforward, and I couldn't understand why the complex theory is needed for SSD. The manuscript should provide more explicit motivation for the theoretical analysis.

Author response:
Thanks for the suggestion. Yes this is indeed almost CVaR,  $CVaR(p)=\frac{ F^{(-2)}(p)}{p}$.  In the literature of risk analysis,  CVaR is also referred as Tail Value at Risk TVAR, we use in the paper TVAR to stress the tail analysis that SSD performs. We have now motivated SSD in the introduced from conditional value at risk point of view, as it can be seen graphically in panel b in figure 1. Thanks for the suggestion.

Reviewer comment (Theory):
Are (1) and (2) always be equivalent? Similarly, are (3) and (5) always equivalent? In my understanding, (1) measures the cumulative probability of observing performance less than $\eta$, and (2) measures the $p$% quartile. It is not clear if (1) is always equivalent to (2) because they are measuring different quantities (though the statement may be true under some specific conditions about $\eta$ and $p$). Similarly, (3) measures the area under the CDF, while (5) measures CVaR. If the paper claims that (1) and (2) (or (3) and (5)) are equivalent, either citation or proofs should be included about this part.

Author response:
Yes (1) and (2) are equivalent;  (3) and (5)  are equivalent. We added in the paper reference to where proofs of these claims are given **without any assumptions needed** (Ogryczak & Ruszczynski, 2002). 
For (1) and (2), please note the CDF and quantiles are inverse one of another, hence if CDF of X is lower than CDF of Y , the quantiles will have the reverse relationship, Quantile of X are higher than Quantile of X.

For (3) and (5), the second performance function (area under CDF), and the second quantile (area under quantiles )  have very interesting properties (( See Ogryczak & Ruszczynski, 2002 ):  They are convex dual one of another. Hence we can write : 
$$ F^{-2}(p) = \sup_{\eta} p\eta - F^{(2)}(\eta),$$
Introducing the argument of the sup above in equation (3) and taking the sup on $\eta$ gives (5). A similar argument holds to go from (5) to (3).

Reviewer comment (Theory):
What does the paper mean by "Note that FSD implies SSD, hence SSD is a finer notion of dominance"? Does it mean that FSD is a special case of SSD?

Author response:
We meant that FSD implies SSD, i.e. if X dominates Y in FSD, we automatically have X dominates Y in SSD. But we can find random variables such that X > Y in SSD , but X < Y in FSD. In that sense SSD considers the tails of the distribution of the random variables and may be finer than FSD in a risk averse assessment of the order between the random variables. Given this interpretation, we would not use the terminology “special case.”

Reviewer comment (Writing):
What is the intuitive understanding of $\epsilon$-FSD? Does it mean the probability that $X$ is preferred to $Y$ under the FSD criteria?

Author response:
X> Y in $\varepsilon$ FSD means that all quantiles of X are higher than those of Y, except a smaller proportion of quantiles. This proportion where Y quantiles are higher, is the violation set.  $\varepsilon$ FSD  means that the area of this violation set is smaller than $\varepsilon$.

Reviewer comment (Evaluation):
I found the evaluation protocol and baselines section to be a bit too condensed. The way I am guessing the comparisons are done in the paper is to perform pairwise comparisons of all models, and then use the outcomes of the pairwise comparisons to produce a ranking?

Author response:
Thanks for the comment, that is correct.  We have clarified this,  please see page 7 Multi-Testing Algorithm paragraph, and now Algorithm 1 Appendix C page 14  has clearer stepwise explanation of the Algorithm for how to go from all pairs comparisons to producing a single rank.

Reviewer comment (Reproducibility):
How does one choose the lambda_i's in (17)?

Author response:
We used in portfolio uniform weights on the metrics. If there is a known order of importance between metrics, this order can be encoded in $\lambda$ and incorporated in the portfolio computation.

Reviewer comment (Evaluation):
Is there a way to account for the possible correlations between the different metrics? The way the portfolio is constructed now does not seem to account for this, as it just takes a geometric average of the individual CDF's?

Author response:
Thank you for bringing up this important question. Without making assumptions on the metrics, it is not easy to incorporate the correlations. One possible idea would be to estimate vine copulas from the metrics, in a form of pairwise CDF between metrics and a graph of correlations (w) and define the portfolio as the vine copula $\exp\left(\sum_{ij} w_{ij} \log F(X_i, X_j)\right)$. This is an interesting question but is beyond the scope of the current work.

Reviewer comment (Theory):
The connection between mathematical theory and the applied problem at hand is not emphasised enough.

Author response:
Thanks for  your comment and suggestions.  We have improved in the revision the connection between the theory part that introduces the concepts and the application in evaluating LLMs. We have improved :
  1. The presentation of the motivation and the contribution of our work  in the introduction . Please see the edits in red in the revised manuscript in the introduction pages 1-3. 
 2. We introduced all concepts graphically (portfolio, FSD and SSD) in the introduction in **Figure 1** in a running example showcasing our contributions 
 3. Throughout all theory sections we  kept reminding the reader in the theory section as suggested by the reviewer the role of each random variable as a metric evaluation or as portfolio that is the aggregation of multiple metrics.  (Please see edits in red in the beginning of Section 2, we also simplified Section 3 and Section 4, without substantial change in the content).

Reviewer comment (Presentation):
The transition between theoretical sections and experiments is somewhat abrupt.

Author response:
Thanks for  your comment and suggestions.  We have improved in the revision the connection between the theory part that introduces the concepts and the application in evaluating LLMs. We have improved :
  1. The presentation of the motivation and the contribution of our work  in the introduction . Please see the edits in red in the revised manuscript in the introduction pages 1-3. 
 2. We introduced all concepts graphically (portfolio, FSD and SSD) in the introduction in **Figure 1** in a running example showcasing our contributions 
 3. Throughout all theory sections we  kept reminding the reader in the theory section as suggested by the reviewer the role of each random variable as a metric evaluation or as portfolio that is the aggregation of multiple metrics.  (Please see edits in red in the beginning of Section 2, we also simplified Section 3 and Section 4, without substantial change in the content).

Reviewer comment (Experiments):
In section 5.1 the approach – in particular, the validation of statistical significance – is validated on an example with two Gaussian random variables. This example does not clearly connect to the “more realistic” experiments later on, and the fact that it works for two Gaussian random variables does not necessarily imply that it works similarly for more complex experiments.

Author response:
We would like to clarify a misunderstanding of our work  that the reviewer seems to have. Our work is not about Mean Risk Models but on stochastic dominance. The Second Stochastic order is consistent with  many Mean Risk models. However, the mean-variance model that is used in finance is indeed problematic as it is not consistent with the Second order and can only be used if the two random variables compared are Gaussians.  The reviewer’s question is valid if we were using mean-variance models since they don’t apply beyond Gaussians. We are not indeed using these models. 

For all our stochastic dominance we need minor assumptions on the random variables , we collected these assumptions in page 6 (Assumption 1). The random variables must be bounded (or with bounded second moments), and have some regularity on their density. 

To illustrate that our stochastic dominance  tests are valid beyond Gaussianity on synthetic examples, we added in Appendix L.1 , our Relative First and Second order tests between  log normals. We choose log normal to illustrate two distributions that are heavy tails, which occurs a lot in practice.

Reviewer comment (Theory):
The Gaussian example relates to the underlying assumptions of the method. While some mean-risk models assume an underlying Gaussian distribution, this assumption is considered unrealistic in finance and many attempts have been made to relax it. It is unclear how essential this assumption is for the problem at hand.

Author response:
We would like to clarify a misunderstanding of our work  that the reviewer seems to have. Our work is not about Mean Risk Models but on stochastic dominance. The Second Stochastic order is consistent with  many Mean Risk models. However, the mean-variance model that is used in finance is indeed problematic as it is not consistent with the Second order and can only be used if the two random variables compared are Gaussians.  The reviewer’s question is valid if we were using mean-variance models since they don’t apply beyond Gaussians. We are not indeed using these models. 

For all our stochastic dominance we need minor assumptions on the random variables , we collected these assumptions in page 6 (Assumption 1). The random variables must be bounded (or with bounded second moments), and have some regularity on their density. 

To illustrate that our stochastic dominance  tests are valid beyond Gaussianity on synthetic examples, we added in Appendix L.1 , our Relative First and Second order tests between  log normals. We choose log normal to illustrate two distributions that are heavy tails, which occurs a lot in practice.

Reviewer comment (Theory):
It is unclear what other assumptions need to hold for the method to have theoretical validity, and how one can check if these assumptions hold before applying the proposed method.

Author response:
We would like to clarify a misunderstanding of our work  that the reviewer seems to have. Our work is not about Mean Risk Models but on stochastic dominance. The Second Stochastic order is consistent with  many Mean Risk models. However, the mean-variance model that is used in finance is indeed problematic as it is not consistent with the Second order and can only be used if the two random variables compared are Gaussians.  The reviewer’s question is valid if we were using mean-variance models since they don’t apply beyond Gaussians. We are not indeed using these models. 

For all our stochastic dominance we need minor assumptions on the random variables , we collected these assumptions in page 6 (Assumption 1). The random variables must be bounded (or with bounded second moments), and have some regularity on their density. 

To illustrate that our stochastic dominance  tests are valid beyond Gaussianity on synthetic examples, we added in Appendix L.1 , our Relative First and Second order tests between  log normals. We choose log normal to illustrate two distributions that are heavy tails, which occurs a lot in practice.

Reviewer comment (Presentation):
Many results are mixed and presented in the appendix, even if described in the main text. The paper could use restructuring to highlight the most relevant results and theory in the main text.

Author response:
We hope the revision highlighted the relevant results and gave better intuitions.

Reviewer comment (Presentation):
There are long stretches where the text reads like it is from an econometrics paper, with the lack of emphasis on how the introduced concepts will be relevant for model comparison distracting from the main contributions of the paper. I recommend being more sensitive to the conference audiences' background, and frequently reminding the reader how a concept or result will help resolve an important sub/problem in the overall model comparison. See the first sentence of my summary of the paper for a simple example of an arguably more accessible framing.

Author response:
Thanks for the suggestions, we have improved in the revision the connection between the theory part that introduces the concepts and the application in evaluating LLMs. We have improved :
  1. The presentation of the motivation and the contribution of our work  in the introduction . Please see the edits in red in the revised manuscript in the introduction pages 1-3. 
 2. We introduced all concepts graphically (portfolio, FSD and SSD) in the introduction in **Figure 1** in a running example showcasing our contributions 
 3. Throughout all theory sections we  kept reminding the reader in the theory section as suggested by the reviewer the role of each random variable as a metric evaluation or as portfolio that is the aggregation of multiple metrics.  (Please see edits in red in beginning of Section 2, we also simplified Section 3 and Section 4, without substantial change in the content).

Reviewer comment (Evaluation):
In the experiments, the paper's results with SSD almost perfectly line up with those obtained by using previously known mean-risk models: I believe that how exactly the current methodology improves upon these simple and useful models is not sufficiently discussed.

Author response:
Mean Risk Models (MRM) are effectively useful, some of them are global statistics (integrated on all percentiles) like the Gini tail, but the tail value at risk, and the mean absolute  deviation   from a quantile are computed for a given percentile $p$ . We aggregate the ranks resulting from these MRM for a given $p$ using metric based rank aggregation (using pearson distance). Note that the resulting rank depends heavily on the percentile $p$ choice. The advantage of  Relative SSD testing is that 1) statistically it has a confidence level which MRM lacks 2) Relative SSD is a global statistic that does not need any hyperparameter choice (except the confidence level ). Hence Relative SSD is more reliable. 

To further compare Relative SSD and MRM we analyze the speed of their convergence versus sample size in Appendix A ( Figure 2 page 12 in the revised paper), we see that R-SSD is more stable and has lower variance in low sample regime.

Reviewer comment (Reproducibility):
How the existing tests are applied pairwise and then combined to achieve multi-testing should be described and discussed in more detail (this applies to main paper and Appendix A.) I recommend: expanding the end of Section 3 to include further details of how the pairwise statistical tests are combined to produce the full ranking with a confidence of $1-\alpha$, and expanding Appendix A to include a high-level description of Algorithm 1.

Author response:
Thanks for the suggestions, we have implemented them in main and in Appendix. (See page 7 Multi-Testing Algorithm paragraph, and now Algorithm 1 Appendix C page 14 has clearer explanation of the Algorithm )

Reviewer comment (Evaluation):
What are the justifications for portfolio-based vs. per metric ranking of models?

Author response:
Thanks for the question. Portfolio based is computationally more efficient than per metric ranking of models. For per metric ranking, one needs to compute Relalive FSD or SSD for each metric, then use rank aggregation to obtain a ranking.  While for Portfolio based , one computes a single R-FSD or R-SSD and obtains the ranking of the models. 

In the revised manuscript, on the Mix-instruct dataset we show in Appendix A (page 12, Metrics Aggregation Versus Portfolio ) that indeed these two approaches lead to similar ranking, which justify that portfolio is not loosing the comparison power of models, but portfolio based is 7x faster than per metric based ranking (this speedup is taking into account the portfolio computation, note that in this example the number of metrics computed is 8).

Reviewer comment (Evaluation):
Why is $\lambda$ is not used (instead of uniform weighting) for rank aggregation between metrics in the latter method?

Author response:
We used in portfolio and in rank aggregation uniform weights on the metrics. If there is a known order of importance between metrics , this order can be encoded in $\lambda$ and incorporated in both portfolio and rank aggregation of metric based ranking.

Reviewer comment (Reproducibility):
What multi-way aggregation method is used after per metric ranking?

Author response:
We discuss this in Appendix F.3 page 17 in revised manuscript,  we used metric based rank aggregation , with `pearson` as a distance.

Reviewer comment (Experiments):
How should $\epsilon$ be chosen? Why?

Author response:
It is not straightforward how to select $\varepsilon$, especially in multi-testing settings.  That is why we introduced the relative stochastic that is thresholdless. We recommend the use of our relative tests over the absolute test that require $\varepsilon$.

Reviewer comment (Evaluation):
Could the authors' methodology be extended to apply to metrics that are not continuous? (e.g. binary accuracy, human annotator ranking)

Author response:
Thanks, that is a very interesting question. Our test statistics are certainly defined for discrete valued metrics. Unfortunately the central limit theory in the paper relies heavily on the continuity of the random variables. Nevertheless, we can map human rankings  of multiple models to real-valued scores where our approach applies! We used these transformations on the chatGPT scores that we included in the revision; we followed Jiang et al. (2023) approach for this mapping. Please See appendix B page 13 that explains this approach. In future work, it would be interesting to explore the asymptotic distribution of the test statistics for discrete-valued metrics in order to design a valid hypothesis testing procedure in that setting - without needing to make the metrics continuous as we currently do.