Paper: Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarks (ICLR-2024)

Reviewer comment (Evaluation):
As noted in the paper, Borda aggregation yields very different results from mean aggregation, even in the absence of missing values. This should be investigated before accepting that the resulting rankings are truthful.

Author response:
**1. On the choice of Borda aggregation/lack of ground-truth/theoretical advantages of Borda.** To comment on the validity of using the ranking as ground truth. It's important to recognize the absence of a definitive ground truth ranking. Indeed, the complexity of social choice theory, exemplified by Arrow's impossibility theorem and the Gibbard–Satterthwaite theorem, underscores the inherent challenges in establishing a universally satisfying and consistent ranking system within the specified criteria.

In this paper, we choose to sum the ranking because it is exactly the Borda Count. The Borda Count is a 2 approximation of the Kemmeny consensus [1] and is an excellent choice in practical applications (see [2,3]).

Reviewer comment (Theory):
"enables us to leverage the Borda aggregation inheriting its theoretical and practical advantage" => what are they?

Author response:
**1. On the choice of Borda aggregation/lack of ground-truth/theoretical advantages of Borda.** To comment on the validity of using the ranking as ground truth. It's important to recognize the absence of a definitive ground truth ranking. Indeed, the complexity of social choice theory, exemplified by Arrow's impossibility theorem and the Gibbard–Satterthwaite theorem, underscores the inherent challenges in establishing a universally satisfying and consistent ranking system within the specified criteria.

In this paper, we choose to sum the ranking because it is exactly the Borda Count. The Borda Count is a 2 approximation of the Kemmeny consensus [1] and is an excellent choice in practical applications (see [2,3]).

Reviewer comment (Theory):
Why is sum of ranking matrices (and more generally Borda count) a good aggregation criterion?

Author response:
**1. On the choice of Borda aggregation/lack of ground-truth/theoretical advantages of Borda.** To comment on the validity of using the ranking as ground truth. It's important to recognize the absence of a definitive ground truth ranking. Indeed, the complexity of social choice theory, exemplified by Arrow's impossibility theorem and the Gibbard–Satterthwaite theorem, underscores the inherent challenges in establishing a universally satisfying and consistent ranking system within the specified criteria.

In this paper, we choose to sum the ranking because it is exactly the Borda Count. The Borda Count is a 2 approximation of the Kemmeny consensus [1] and is an excellent choice in practical applications (see [2,3]).

Reviewer comment (Evaluation):
How does the method handle misleading evaluation results, when the evaluation metric failed because of the dataset sample bias, or because it is itself an approximation of human evaluation?

Author response:
2. **We did not investigate the impact of misleading evaluation results in this paper**, but we do agree this is an interesting question. We did add it to future work (see Appendix).

Reviewer comment (Experiments):
Scaling corruption (Fig 3) is not detailed enough. How are the tasks selected for being scaled? How many tasks are being scaled? What is the evaluation metric? Why are there two regimes, starting at 1 or 0 when eta is 0? Font in Figure 3 is also too small.

Author response:
3. **On the robustness analysis in the experiments.** $\eta$ varies from 0 to 0.3 and the corrupted tasks are randomly chosen. The change in the final ranking is measured by the Kendall $\tau$. We did add a clarification in the main paper (see updated manuscript).

Reviewer comment (Evaluation):
Why is the correlation of sigma_l very different from sigma_2l for some datasets of Fig. 5 while it is very similar for others?

Author response:
4. **In Figure 5.** We agree that the label of the x-axis was misleading. We have updated it by setting it to 1- \eta to make it coherent with the discussion. We attribute the decline in performance on Flickr to the data distribution. However, our investigation into the score distribution did not reveal any discernible differences.

Reviewer comment (Presentation):
It is not clear from the figures whether # scores (%) is the percentage of removed or kept scores. The discussion is misleading in that regard.

Author response:
4. **In Figure 5.** We agree that the label of the x-axis was misleading. We have updated it by setting it to 1- \eta to make it coherent with the discussion. We attribute the decline in performance on Flickr to the data distribution. However, our investigation into the score distribution did not reveal any discernible differences.

Reviewer comment (Evaluation):
What is the proportion of missing scores in the comparison of rankings by sigma_l and sigma_mu in table 1 and 2?

Author response:
5. **In Tables 1 and 2**, we quantified the variation in rankings by calculating the average Kendall's tau (τ) between the generated rankings as we varied the proportion of missing rankings. We have included a clarification on this point in the revised version of the manuscript.

Reviewer comment (Experiments):
Confidence analysis (Fig 6) should be compared to other methods such as ANOVA. This section is not very useful to main point of the paper and can be removed in favor of more analysis of previous experiments.

Author response:
6. **We added the comparison with ANOVA** in the future work section. We agree with the reviewer this is outside the scope of the paper.

Reviewer comment (Novelty):
Originality is low and the contributions weak, as the main contributions are an efficient implementation for a combinatorial problem that allows to extend two pre-existing methods (Colombo et al. 2022) to missing scores, and enlarging an already existing dataset. Unsurprisingly the methods proposed in (Colombo et al. 2022) also are effective in this setting.

Author response:
**On the novelty.**

The first novelty of the paper is to identify and formalize the problem of benchmarking in the presence of missing data. Previous work in NLP often simply ignores the missing data when choosing the best system (see Pfeiffer et al., 2022; Lin et al., 2022; Martin et al., 2020; Guibon et al., 2021; Peng et al., 2019). This is mainly due to the difficulty of collecting data/private datasets or more recently the cost of running expensive models such as GPT. 

To the best of our knowledge, this is the first paper to tackle the problem of benchmarking with missing data in the NLP community.

**On the technical contribution.**

Our work builds on Colombo et al 2023. However, the addressed problem is different as such their method cannot be naively applied. Second, it seems that the reviewer missed the technical aspect of our paper. Specifically:
we introduce a new ranking algorithm for missing data. Our newly designed method can be adapted to both instance- and task-level aggregations. This method is not trivial and relies on counting the number of system orderings compatible with a partial rank. We respectfully refer the reviewer to Section 3. Our algorithm’s complexity is polynomial $O(n^3)$ while the naive version would be factorial. Additionally (see response to cmy7) our estimator is unbiased. 
We introduce confidence intervals for the ranking see Section 3.3 to better understand the uncertainty linked to the ranking.

Additionally, we conducted an extensive data collection and gathered a new dataset with a public release. Our effort is crucial for future work. Concretely, our dataset gathered over 131M score which is an order of magnitude larger than all existing datasets.  We believe that this will spur more research in benchmarking, a critical area in NLP, especially considering the surge in Generative Models.

Reviewer comment (Presentation):
The figures have very small fonts, unreadable without extensive zooming.

Author response:
**On the readability issue.**
We have diligently incorporated the reviewer's feedback into our manuscript. This includes rectifying typos, addressing citation issues, and increasing the size of figures as suggested.

Reviewer comment (Presentation):
Fig 3 is not readable, same with other figures.

Author response:
**On the readability issue.**
We have diligently incorporated the reviewer's feedback into our manuscript. This includes rectifying typos, addressing citation issues, and increasing the size of figures as suggested.

Reviewer comment (Presentation):
Citations use the wrong format "authors (year)" instead of "(authors, year)".

Author response:
**On the readability issue.**
We have diligently incorporated the reviewer's feedback into our manuscript. This includes rectifying typos, addressing citation issues, and increasing the size of figures as suggested.

Reviewer comment (Experiments):
In section 4, it seems that the toy experiment is only applied to robustness scaling and pairwise confidence analysis, but as I started to read I was expecting more development experiments. Could you mention why you only check those two factors and not other?

Author response:
1. **On the toy experiments.** The selection of these two factors, namely robustness to scaling and confidence, stems from the specific demands of NLP evaluation. In the realm of generative models, NLP practitioners frequently encounter metrics on diverse scales, some of which may even be unbounded, as exemplified by BartScore. Furthermore, practitioners often neglect confidence intervals, making this paper's focus on them a notable contribution. Given the already dense nature of the paper with numerous experimental results, our decision to explore additional factors, such as robustness to noise in real data experiments, serves to further substantiate the efficacy of our approach on authentic rankings.

Reviewer comment (Evaluation):
Lack of closer looks at the correlation between tasks, since similar tasks might be "easier" to predict

Author response:
Numerous studies in the literature explore the correlation between metrics (see [1] for example). We did consider this aspect in the early stages of the real data analysis, however, delving into it extensively proved challenging. Additionally, given the amount of experiments in the paper, we believe this would require work on its own. We do agree this would be an interesting follow-up work and added it to the paper's next research directions (see updated version of the manuscript). A promising avenue for studying this problem could lie in investigating confidence intervals. _If all metrics exhibit strong correlations, it could lead to a reduction in the size of the confidence interval—a potential starting point for a more in-depth examination._

Reviewer comment (Experiments):
Are there any other stronger baselines or previous works to compare with?

Author response:
To the best of our knowledge, this is the first work that addresses this issue of benchmarking in the presence of missing data for NLP systems. This is indeed a strong contribution to the paper.

Reviewer comment (Theory):
Although the methodology is well described overall and there is a lot of useful detail in the paper and the (extensive) appendix, the motivations are sometimes lacking. For example, is averaging still the right way to combine estimated ranks?

Author response:
The reviewer expresses concern regarding the appropriateness of assuming uniformity for unobserved rankings. In response, we demonstrate that in the realm of ranking data, unlike real-valued data, the estimation process remains consistent despite considerable noise. 
The conventional proofs affirming the consistency of the estimator for the Borda algorithm under various assumptions rely on demonstrating that for any pair $i,j$, if the data is drawn from a model satisfying the strong-unimodality property (such as Plackett-Luce or Mallows model just to name a few popular ones) with a median $\sigma_0$ for which  $\sigma_0(i)<\sigma_0(j)$, then $\mathbb{E}[\sigma(i)] < \mathbb{E}[\sigma(j)]$, and thus, the Borda algorithm accurately ranks such pairs [1,2,3] 
Here, we demonstrate that if Borda accurately ranks the pair $i,j$ without noise, i.e., if $\mathbb{E}[\sigma(i)] < \mathbb{E}[\sigma(j)]$, then it will also correctly rank $i$ and $j$ in expectation, given an equal noise rate $\eta$ for both $i$ and $j$. To see this, note that the expected imputed ranking value will be the average of all possible rankings, $0.5n$. Therefore,  the expected ranking of $i$ will be equal to $\mathbb{E}[0.5n\eta + (1-\eta)\sigma(i)] = 0.5n\eta + (1-\eta)\mathbb{E}[\sigma(i)] $, where the equality holds by linearity of the expectation. By noting that $\mathbb{E}[\sigma(i)]\mathbb<{E}[\sigma(j)]$ implies $0.5n\eta + (1-\eta)\mathbb{E}[\sigma(i)] <0.5n\eta + (1-\eta)\mathbb{E}[\sigma(j)] = \mathbb{E}[\sigma(i)] < \mathbb{E}[\sigma(j)]$ we conclude the proof.

Reviewer comment (Theory):
Imputation methods usually don't use naive distribution estimates, but try to leverage observed data to improve the missing data imputation. For example, if scores are missing for systems i and j on a given task, but i usually outperforms j whenever they are both observed, it seems sub-optimal to set M_ij to 0.5 (step 2, p. 5).

Author response:
The reviewer expresses concern regarding the appropriateness of assuming uniformity for unobserved rankings. In response, we demonstrate that in the realm of ranking data, unlike real-valued data, the estimation process remains consistent despite considerable noise. 
The conventional proofs affirming the consistency of the estimator for the Borda algorithm under various assumptions rely on demonstrating that for any pair $i,j$, if the data is drawn from a model satisfying the strong-unimodality property (such as Plackett-Luce or Mallows model just to name a few popular ones) with a median $\sigma_0$ for which  $\sigma_0(i)<\sigma_0(j)$, then $\mathbb{E}[\sigma(i)] < \mathbb{E}[\sigma(j)]$, and thus, the Borda algorithm accurately ranks such pairs [1,2,3] 
Here, we demonstrate that if Borda accurately ranks the pair $i,j$ without noise, i.e., if $\mathbb{E}[\sigma(i)] < \mathbb{E}[\sigma(j)]$, then it will also correctly rank $i$ and $j$ in expectation, given an equal noise rate $\eta$ for both $i$ and $j$. To see this, note that the expected imputed ranking value will be the average of all possible rankings, $0.5n$. Therefore,  the expected ranking of $i$ will be equal to $\mathbb{E}[0.5n\eta + (1-\eta)\sigma(i)] = 0.5n\eta + (1-\eta)\mathbb{E}[\sigma(i)] $, where the equality holds by linearity of the expectation. By noting that $\mathbb{E}[\sigma(i)]\mathbb<{E}[\sigma(j)]$ implies $0.5n\eta + (1-\eta)\mathbb{E}[\sigma(i)] <0.5n\eta + (1-\eta)\mathbb{E}[\sigma(j)] = \mathbb{E}[\sigma(i)] < \mathbb{E}[\sigma(j)]$ we conclude the proof.

Reviewer comment (Writing):
The paper feels rushed at times and there are lots of readability issues, including with the notation.

Author response:
We've invested considerable effort in meticulously collecting data, resulting in a robust experimental contribution that makes the paper dense. In response to the reviewer's valuable feedback, we've implemented changes, relocating figures to the appendix and meticulously addressing all their suggestions. This includes rectifying notations, refining legends, correcting typos, and ensuring the accuracy of citations. All the modifications are highlighted in yellow in the updated paper.

Reviewer comment (Presentation):
This is a substantial paper with a lot of material, but it is hard to pack that much material in 9 pages, and difficult to follow the paper without the appendices.

Author response:
We've invested considerable effort in meticulously collecting data, resulting in a robust experimental contribution that makes the paper dense. In response to the reviewer's valuable feedback, we've implemented changes, relocating figures to the appendix and meticulously addressing all their suggestions. This includes rectifying notations, refining legends, correcting typos, and ensuring the accuracy of citations. All the modifications are highlighted in yellow in the updated paper.

Reviewer comment (Theory):
It was not fully clear why the 'argsorts' are systematically doubled (p.3, p.6). For example, in Eq. 3, it seems that computing the average of estimated scores, one sort would be enough to recover the permutation with correct ranking.

Author response:
The argsorts are used to convert the combined scores of the systems into rankings. For instance, scores [1.0, 3.5, 2.0, 2.2] would be transformed into [0, 3, 1, 2], indicating that system 1 is ranked first, system 2 is ranked fourth, system 3 is ranked second, and system 4 is ranked third. The initial argsort provides an ordering of systems based on their scores, while the second one generates rankings of the systems relative to this ordering.

Reviewer comment (Writing):
In Clarity issues, in Sec. 3.2.1, "input" is likely "impute" (the missing data/information).

Author response:
We've invested considerable effort in meticulously collecting data, resulting in a robust experimental contribution that makes the paper dense. In response to the reviewer's valuable feedback, we've implemented changes, relocating figures to the appendix and meticulously addressing all their suggestions. This includes rectifying notations, refining legends, correcting typos, and ensuring the accuracy of citations. All the modifications are highlighted in yellow in the updated paper.

Reviewer comment (Writing):
In Clarity issues, still in Sec. 3.2.2, p_{i,j} pops up in the last paragraph — is that M_{i,j}?

Author response:
We've invested considerable effort in meticulously collecting data, resulting in a robust experimental contribution that makes the paper dense. In response to the reviewer's valuable feedback, we've implemented changes, relocating figures to the appendix and meticulously addressing all their suggestions. This includes rectifying notations, refining legends, correcting typos, and ensuring the accuracy of citations. All the modifications are highlighted in yellow in the updated paper.

Reviewer comment (Writing):
In Clarity issues, in Sec 3.2.3, step 3, there is a need for some reference to a publication or appendix for Borda count aggregation.

Author response:
We've invested considerable effort in meticulously collecting data, resulting in a robust experimental contribution that makes the paper dense. In response to the reviewer's valuable feedback, we've implemented changes, relocating figures to the appendix and meticulously addressing all their suggestions. This includes rectifying notations, refining legends, correcting typos, and ensuring the accuracy of citations. All the modifications are highlighted in yellow in the updated paper.

Reviewer comment (Presentation):
Figures are overall way too small and often unreadable.

Author response:
We've invested considerable effort in meticulously collecting data, resulting in a robust experimental contribution that makes the paper dense. In response to the reviewer's valuable feedback, we've implemented changes, relocating figures to the appendix and meticulously addressing all their suggestions. This includes rectifying notations, refining legends, correcting typos, and ensuring the accuracy of citations. All the modifications are highlighted in yellow in the updated paper.

Reviewer comment (Presentation):
Citation formatting is incorrect — citations lack brackets in most places, likely due to an issue with \cite[] usage with the style file.

Author response:
We've invested considerable effort in meticulously collecting data, resulting in a robust experimental contribution that makes the paper dense. In response to the reviewer's valuable feedback, we've implemented changes, relocating figures to the appendix and meticulously addressing all their suggestions. This includes rectifying notations, refining legends, correcting typos, and ensuring the accuracy of citations. All the modifications are highlighted in yellow in the updated paper.

Reviewer comment (Presentation):
On p.5, line -4: Superscripts of M seem messed up.

Author response:
We've invested considerable effort in meticulously collecting data, resulting in a robust experimental contribution that makes the paper dense. In response to the reviewer's valuable feedback, we've implemented changes, relocating figures to the appendix and meticulously addressing all their suggestions. This includes rectifying notations, refining legends, correcting typos, and ensuring the accuracy of citations. All the modifications are highlighted in yellow in the updated paper.

Reviewer comment (Writing):
On p.9: "in Ssec 5.3" is likely meant to be "5.2" (since we are in 5.3).

Author response:
We've invested considerable effort in meticulously collecting data, resulting in a robust experimental contribution that makes the paper dense. In response to the reviewer's valuable feedback, we've implemented changes, relocating figures to the appendix and meticulously addressing all their suggestions. This includes rectifying notations, refining legends, correcting typos, and ensuring the accuracy of citations. All the modifications are highlighted in yellow in the updated paper.