What are the best systems? New perspectives on NLP Benchmarking
Abstract
In Machine Learning, a benchmark refers to an ensemble of datasets associated with one or multiple metrics together with a way to aggregate different systems performances.
They are instrumental in (i) assessing the progress of new methods along different axes and (ii) selecting the best systems for practical use.
This is particularly the case for NLP with the development of large pre-trained models ( e.g. GPT, BERT) that are expected to generalize well on a variety of tasks.
While the community mainly focused on developing new datasets and metrics, there has been little interest in the aggregation procedure, which is often reduced to a simple average over various performance measures.
However, this procedure can be problematic when the metrics are on a different scale, which may lead to spurious conclusions.
This paper proposes a new procedure to rank systems based on their performance across different tasks.
Motivated by the social choice theory, the ﬁnal system ordering is obtained through aggregating the rankings induced by each task and is theoretically grounded.
We conduct extensive numerical experiments (on over 270k scores) to assess the soundness of our approach both on synthetic and real scores ( e.g. GLUE, EXTREM, SEVAL, TAC, FLICKR).
In particular, we show that our method yields different conclusions on state-of-the-art systems than the mean-aggregation procedure while being both more reliable and robust.
Introduction
This paper is about improving current practices regarding benchmarks of NLP systems.
As pointed out by [83], benchmarks are made of datasets, metrics and a way to aggregate performance.
Our point is that, if the bulk of the NLP community efforts on this domain is about collecting new datasets and introducing new metrics, little work is concerned with the third part, namely how to aggregate various performances .
Why are benchmarks vital?
Research advances in Machine Learning (ML) are crucially fueled by reliable evaluation procedures [34, 78].
The latter are indeed mandatory to fairly compare new methods and systems.
Usually, one relies on a well-chosen metric that reﬂects the ability to perform on a task – e.g. accuracy for classiﬁcation, mean-squared error for regression.
The multi-tasks evaluation setting.
If single-tasks problems are quite common, to best understand weakness and model in real-world scenario, the community is heading towards more complex evaluations involving ﬁne-grained evaluation [61] across several metrics (or criteria [46, 103]) and several tasks [36, 47, 63, 89, 91, 108].
This is due to the increasing performance of deep neural networks, which are nowadays designed to generalize in a great variety of situations and to solve complex tasks [86].
One is typically seeking for models with good transfer learning properties, meaning an ability to generalize well under distribution shift and/or task shift [51].
How to aggregate performances?
The multi-tasks setting has been investigated in recent works that provide benchmark of state-of-the-art models across a great variety of tasks [28, 62, 80, 90, 108], sometimes with more than ﬁfty [2, 84, 85, 94].
These papers provide tables of scores across the considered tasks, but the only non-qualitative way to compare systems consists in averaging the performances across tasks and then ranking systems according to their mean score values.
This is, for instance, done with the GLUE benchmark [91] and its derivatives [92].
However, taking the mean is seriously ﬂawed since the different metrics are usually not on the same scales and can even be unbounded [23, 102].
Even a pre-processing renormalization scheme would fail to capture the intrinsic difﬁculty of the tasks.
Contribution 1.
Our ﬁrst contribution is to provide a reliable tool to rank systems in a multi-tasks setting.
We rely on a ranking aggregation procedure which, from a set of rankings induced by each criterion, returns a single ranking that somehow aggregates the former.
This procedure, called the
Kemeny consensus [52], can be seen as a voting rule and stems from the social choice theory [66].
Aggregation when instance-level information is available.
As illustrated by Ruder [83], Zhong et al.
[109], a ﬁne-grained understanding of the model performance should include instance-level scores.
If taking the mean is quite natural in the classiﬁcation setting, this is not always the case, as recently pointed out by [73] in the NLG setting.
In this article, the authors investigate pairwise comparison of NLG systems for a single metric ( e.g. BLEU [71], ROUGE [59], METEOR [5, 35, 49],
CHRF [76, 77], BertScore [105]).
They prove that a comparison based on the mean or the median of the scores across test utterances can be highly ﬂawed.
They rather advise to rely on the BradleyTerry [10] pairwise comparison method, which consists, for two systems A and B, in computing the proportion of utterances on which A achieves a better score than B. Their work is a signiﬁcant advance but remains limited to pairwise comparisons.
Contribution 2.
Our second contribution consists in going one step further than [73] by applying our ranking procedure to an arbitrarily large set of NLG systems with respect to a group of ﬁxed criterion.
Our evaluation methodology can be seen as a natural extension of [73] since it coincides with the latter in the particular case of pairwise comparison.
In a more realistic multi-criteria scenario, we combine our two contributions and develop a two-stages ranking aggregation procedure which ﬁrst aggregates along utterances and then along criteria.
Experiments.
Our two contributions rely on our aggregation procedure which is proved to be effective through several experiments.
We explain on a simple synthetic example the superiority of our approach compared to the mean-aggregation procedure and the pairwise-aggregation procedure, both in terms of consistency and robustness.
We use our ranking procedure on 10 multi-tasks / multi-criteria benchmarks and observe it leads to different conclusions than mean- and pairwise-aggregation procedures.
We argue our procedure is more robust by investigating its stability with respect to the addition of criteria and with respect to the addition of systems.
Our code and the collected data will be released to accelerate the adoption of what we think is a reliable evaluation method for multi-tasks and multi-criteria benchmarks.
Problem Formulation and limitations of existing methods 2.1 General Considerations
When comparing systems performances, two settings can be distinguished depending on the information granularity at our disposal and on the way one wishes to use this information.
In general, each system is scored on each instance of a test set with respect to a given metric.
The ﬁnal (single) score of the system with respect to this metric is obtained through an aggregation procedure we scores instances task
Ks 1 , 1 ,s 1 , 1 ,Kscores instances taskT
K T s 1 ,T,s 1 ,T,K T (cid:110) system s 1 ,s 1 ,T instance-level aggregation stask-level aggregation s N,s N,T task taskT system systemN s N
Ks N, 1 ,s N, 1 ,K
K T s N,T,s N,T,K T (cid:110) systemN [Figure]
information
[Figure]
From the latter, one can derive the former relying on what we call an instance-level A
task level can then be per formed to synthesize per will call instance-level aggregation , and which has to be chosen by the practitioner (usually, the mean of the instances-scores). Then, the ﬁnal benchmark score of the system is obtained through an aggregation we call task-level aggregation of the scores of this system for each metric of the considered benchmark. See Fig.
1 for an illustration.
Notations.
Suppose we are given N ≥ 1 systems evaluated on T ≥ 1 tasks, each task t ∈ { 1 , . .
. , T } being associated with a metric and a test set made of K t ≥ 1 instances.
For every n ∈ { 1 , . . . , N } , t ∈ { 1 , . . .
, T } and k ∈ { 1 , . . .
, K t } , we denote by s n,t,k ∈ R the score of system n on the instance k of task t .
Instance-level aggregation.
The performance of system n on task t is an aggregation of its scores ( s n,t,k ) 1 ≤ k ≤ K t on each instances.
This aggregation is chosen by the practitioner and is usually the mean-aggregation deﬁned by
[Equation]
Task-level aggregation.
Sometimes, one only has access to the aggregated scores of each system on each task, that is, for every n ∈ { 1 , . . .
, N } and t ∈ { 1 , . .
, T } , to a score s n,t ∈ R which corresponds to an instance-level aggregation for system n on the instances of task t .
From these quantities, one can then compute, for each system, a synthetic score reﬂecting the overall performance of this system across every considered task.
Again, the usually-taken synthetic score of system n is the mean-aggregation: s mean n
[Equation]
2.2 Problem Formulation
Ranking objective.
When benchmarking systems, the goal is to output a ranking of each systems according to some objective criterion.
Formally, we need to introduce the symmetric group on N elements, denoted by S N , which elements are the N !
permutations of { 1 , . .
. , N } .
Equipped with this notation, our goal is to output a permutation
[Equation]
corresponding to the rankings of the systems.
For instance, one reads “system i is the σ i -th best system".
Depending on the granularity of the information at our disposal, we distinguish two problems.
Ranking Systems from Task Level Information.
Given a set of scores ( s n,t , 1 ≤ n ≤ N, 1 ≤ t ≤ T ) of N systems on T tasks, ﬁnd a proper aggregation procedure.
Ranking Systems from Instance Level Information.
Given a set of scores ( s n,t,k , 1 ≤ n ≤
N, 1 ≤ t ≤ T, 1 ≤ k ≤ K t ) of N systems on the different instances of T tasks, ﬁnd a proper aggregation procedure.
2.3 Limitation of Existing Methods
Mean-aggregation procedure.
The mean-aggregation procedure consists in taking the permutation σ mean ∗ that would rank the aggregated means s mean n , 1 ≤ n ≤ N .
This procedure suffers from several ﬂaws.
First, it is not well-suited when the metrics associated with the considered tasks are not on the same scale.
Consider, for instance, the situation where one of the tasks (say task t 0 ) is associated with a metric that is at a signiﬁcantly larger scale than the others.
In that case, the ranking obtained through mean-aggregation would probably correspond to the ranking induced by task t 0 .
One could argue that a remedy would ﬁrst normalize each metric so that everything is on the same scale.
However, the
[Table]
resulting aggregation would still fail to capture each task’s intrinsic difﬁculty.
Worse, this procedure is impractical in cases where some metrics are unbounded – for instance, this is the case of the
BARTScore [102].
Finally, another weakness of the mean-aggregation ranking procedure is that the score of a system is computed irrespective of its relative performance with respect to the others.
This simple observation has been pointed out by [73] who advises, in the special case of two systems and one metric, to compute the number of times a system is better than the other on the instances.
Pairwise ranking.
To be a bit more formal, the pairwise ranking aggregation proposed by [73] to rank two systems A and B , which scores on a given task are given by s A 1 , . . .
, s AK and s B 1 , . . .
, s BK , consists in computing
[Equation]
Then, A is better than B if and only if λ A > λ B .
As explained by the authors, this method is more relevant than the mean-aggregation in the context of NLG evaluation.
However, it is limited to the evaluation of two systems and does not apply for a general number N ≥ 3 of systems.
A solution would be to give a rank for each pair of systems and then aggregate these pairwise rankings.
However, this would lead to a prohibitive (cid:0) N 2 (cid:1) computational factor for the complexity.
Moreover, the conclusion of these pairwise rankings can be paradoxical.
Tab. 1 below provides a toy example where three systems A , B , and C are evaluated on 6 tasks, and where the pairwise comparisons give the paradoxical conclusion B > A , C > B and A = C .
Ranking via Kemeny consensus
We now turn to the description of our methodology to rank an arbitrary number of systems on multi-tasks / multi-criteria benchmarks.
3.1 Kemeny Consensus
Let us consider the problem of ranking N systems on T tasks based on the information of the scores s n,t of each system n on each task t .
We believe a robust approach to this problem consist in relying on the relative performance between systems on each task.
More precisely, for each task t ∈ { 1 , . .
, T } , we consider
[Equation]
where σ t n corresponds to the rank of system n on task t , in decreasing order.
Then, we would like to ﬁnd an appropriate procedure that aggregates the T rankings σ 1 , . . .
, σ T .
More formally, we would like to deﬁne a function
[Equation]
This function, from a set of T permutations corresponding to rankings, should return a ﬁnal permutation that summarizes them.
One difﬁculty is that the mean procedure makes no sense on the symmetric group, which is not a vector space.
It turns out that a very natural choice consists in taking f as the so-called Kemeny consensus [52] aggregation procedure, which somehow corresponds to compute a barycenter.
Kemeny consensus.
Let d be the Kendall distance on the symmetric group, deﬁned for every η, τ ∈ S N by d ( η, τ ) :
= (cid:80) 1 ≤ i,j ≤ N 1 ( η i − η j )( τ i − τ j ) < 0 .
A Kemeny consensus σ ∗ of σ 1 , . .
. , σ T is a solution of the following minimization problem min σ ∈ S N (cid:80) 1 ≤ t ≤ T d ( σ t , σ ) .
Why is Kemeny consensus natural?
As proved by Young and Levenglick [100], the Kemeny consensus aggregation procedure is the only rule that satisﬁes three natural properties: neutrality , meaning that it does not depend on the order of the tasks; consistency , meaning that if the tasks are split in two subsets and that the aggregation in the to subsets rank system i above system j then σ ∗ i > σ ∗ j ; and the Condorcet criterion [27], meaning that an item wining all its pairwise comparison is ranked ﬁst.
Moreover, the Kemeny consensus is also the maximum likelihood of the widely-used
Mallows statistical on the symmetric group [99].
3.2 Borda’s count approximation
If the Kemeny consensus is the ideal objective one would like to obtain, its computation is, in general, an NP-hard problem [6, 40] – although some regularity assumptions, rarely satisﬁed in practice, can speed up the computation, see for instance [8] and [11].
Fortunately, there exist many ways to get satisfying approximations of the latter: see for example [1] for a comprehensive empirical study.
For our experiments, we choose the so-called Borda’s count procedure, deﬁned hereafter for the instance-level and/or task-level aggregation.
Borda’s count.
The Borda’s count consists, from a set of permutations η 1 , . . .
, η L ∈ N corresponding to the ranking of N systems across L ≥ 1 tasks or instances, to sum the ranks of each system and then to rank the obtained sums.
Formally, it
Compute sum n :=
L (cid:80) l =1 η ln for every 1 ≤ n ≤ N , 2.
Output η := Borda( η 1 , . . .
, η L ) ∈ S N that ranks the sums, sum n ( argsort(argsort(sum 1 , . .
, sum T )) ).
There are at least four explanations for choosing Borda’s count procedure.
First, it coincides with the pairwise ranking procedure in the case of two systems, making it a natural generalization.
Second, there exists a theoretical result assessing it is a 5 -approximation of the Kemeny consensus [30] with respect to the Kendall distance [42].
Third, it is an unbiased estimator of the Kemeny consensus with low sample complexity for data distributed according to standard rankings models such as
Mallows [13, 44].
Fourth, from a practical perspective, [1] observe it is efﬁcient, accurate, and actually 10 times faster than the other approximation algorithms.
Fourth, We are now in a position to give our answers to the initial ranking problems from Task Level Information and from Instance
Level Information.
3.3 Our Ranking Procedures
How to rank Systems from Task Level Information.
Let σ t ∈ S N be permutation that ranks the scores s 1 ,t , . . .
, s N,t .
Our aggregation procedure ( σ ∗ ) output is σ ∗ :=
Borda (cid:0) σ 1 , . .
, σ t (cid:1) .
How to rank Systems from Instance Level
Information.
We actually give two different procedures.
For every task t ∈ { 1 , . . .
, T } and every instance k ∈ { 1 , . . .
, K t } of that task, let σ t,k be the permutation that ranks the scores s 1 ,t,k , . .
, s N,t,k .
See Figure 2 for an illustration.
Two-level aggregation ( σ 2 l ).
This procedure
Compute σ t :=
Borda( σ t,k , 1 ≤ k ≤ K t ) for each task t ∈ { 1 , . . .
, T } , 2.
Output σ 2 l := Borda( σ t , 1 ≤ t ≤ T ) .
One-level aggregation ( σ l ).
This procedure outputs σ l := Borda( σ t,k , 1 ≤ t ≤ T, 1 ≤ k ≤
K t ) .
[Figure] [Figure] [Figure]
[Figure]
1k
K T σ t, 1 σ t,σ t,k σ 1 ,Kσ T, 1 σ T,σ T,k σ T,K T
[Figure]
[Figure]
[Table] [Figure]
(cid:1)
[Figure]
σσ t σ T
3.4 How to compare rankings
The rest of the paper is dedicated to synthetic and empirical experiments, on which we demonstrate the soundness of our approach.
In order to obtain a quantitative result, one needs to be able to compare different rankings quantitatively.
Two measures can be used for that purpose: (1) the Kendall distance and (2) the Kendall correlation ( τ ) [53, 54].
The Kendall distance computes the number of inversions between two permutations and is therefore adapted for our purpose of ranking systems.
The values of τ range from [ − 1 , 1] where the value of 1 corresponds to a strong agreement, and close to -1 indicates strong disagreement.
Synthetic Experiments
In this section, we validate on simulated data the performance of our method on two criteria: robustness to manipulation and robustness to scaling.
4.1 Data Generation
The toy experiment analysis is carried out on synthetic scores over N = 20 systems, T =tasks and K = 20 instances.
For each n ∈ { 1 , . . . , N } , we model the performance of system n by a Gumbel r.v.
G n centered at φ ∗ n at scale β = 1 , where φ ∈ [0 , 1] is a dispersion parameter.
The scores of system n , ( s n,t,k ) t,k , are i.i.d.
samples of G n centered at φ ∗ n with scale β = 1 , where φ ∈ [0 , 1] is a dispersion parameter.
Moreover, the scores of different systems are sampled independently.
Since G n +1 − G n follows a logistic distribution with mean φ at scale 1 , this imply that P ( G n +1 − G n > 0) > 0 .
5 , the probability that system n + 1 performs better than system n is at least 0 .
Therefore, for all k, t , the rankings of systems is a realization of the ground-truth ranking [1 , ...N ] , with a noise term controlled by the ‘dispersion’ parameter φ .
Extreme scenarii correspond to the choices φ = 0 and φ = 1 .
More precisely, φ = 0 implies that all scores s n,t,k have the same distribution, whereas φ = 1 induces a strong consensus, i.e. , a clear system ranking emerges.
Remark 1.
Sampling s n,t,k according to the described procedure is equivalent to sampling the ranking of the systems from the well-know Plackett-Luce distribution [39, 75] with weights w i = φ ∗ n .
Interestingly, this distribution over ranking can be seen both from the utilitarian perspective in which the scores s n,t,k are real numbers and from a ranking-model perspective in which the ranking of systems have known distribution.
4.2 Robustness to manipulation
Setting.
To test the robustness of our ranking procedure, we analyze its stability with respect to perturbations of the scores.
More precisely, our way to corrupt scores of a given task t consists in sampling ( sn, t, k ) n,k as i.i.d.
samples following Gumbel distribution centered at − n .
This implies that, for that task t , the underlying ranking is [ N, ... 1] , namely the exact opposite of the ground truth [1 , . . .
, N ] .
The robustness to manipulation analysis shows how the error on the ﬁnal ranking of systems increases as the scores of some t tasks are ‘corrupted’.
Here, the error is computed relying on the normalized Kendall distance between the ground-truth ranking [1 , ...N ] and the ranking of systems obtained relying on the corrupted scores.
Results:
For each of the considered methods, σ mean , σ l and σ 2 l we report in Fig.
3 the results of the robustness analysis when φ and the number of corrupted task varies.
The results of the robustness analysis show that σ 2 l outperforms σ l which at the same time consistently outperforms σ mean .
Overall, for the same number of corrupted tasks and dispersion, the error of σ 2 l is always the smallest.
Moreover, the score-based method σ mean gets an error larger than .75 when just 2, 3, and 5 out of the total of T = 20 tasks have been corrupted, while for σ l the same error is achieved with 5, 7 andcorrupted tasks.
The most robust method is the two-level σ 2 l for which 10, 11 and 11 out of T =tasks have to be corrupted to get the same error of 0.75.
Takeaways: We conclude that the ranking-based methods are more robust than σ mean .
In particular, the 2-level aggregation σ 2 l is the most robust aggregation procedure.
4.3 Robustness to scaling
To further compare the ranking, we corrupt the scores of a given task by re-scaling them by a factor of x > 0 .
Whereas it does not affect our ranking procedure (every ranking induced by a task-instance pair remains the same), it increasingly perturbs the mean aggregation procedure as x increases.
Re-scaling the scores by a factor of 2 produces an error larger than 90%.
For larger φ , re-scaling the
[Figure] [Figure]
scores by a factor of 7 produces the same error (see Fig.
7 for detailed results).
Takeaways: Re-scaling one task’s score with an arbitrarily large number will always produce an arbitrarily large error for mean aggregation while not affecting ranking based aggregation.
Empirical Experiments
In this section, we present our results on real evaluation scores.
Our large scale experiments relies on real evaluation scores (over 270k scores) which are described in Ssec.
In Ssec.
5.2 we gather experimental results for Ranking Systems from Task Level Information , while Ssec.
5.3 is dedicated to the problem of Ranking Systems from Instance Level Information .
5.1 Data Collection
Datasets with Task Level Information We collect the results of GLUE [91], SGLUE [92] 1 and
XTREME [50].
For GLUE the dataset is composed of N = 105 systems that are evaluated ondifferent tasks: CoLA [93], SST-2 [87], MRPC [38], STS-B [14], QQP, MNLI [96], QNLI [80], RTE [7, 31, 48] and WNLI [57].
For SGLUE, the ﬁnal dataset gathers scores from N = 24 systems that are evaluated on 10 different tasks: BoolQ [17], CB [33], COPA [82], MultiRC [55], ReCoRD [104],
RTE, WiC [74], WSC and its derivatives AX-b AX-g [57].
XTREM benchmark is composed of
N = 15 systems and include tasks such as sentence classiﬁcation (using XNLI [29, 95] and PAXS-X [98, 106]), structured prediction (relying on Universal Dependencies v2.5 [68] and Wikiann [70, 79]), sentence retrieval (with BUCC [110, 111] and Tatoeba [3]) and question answering (via XQuAD [4, 80], MLQA [58], TyDiQA-GoldP [18]).
For all benchmarks, various types of metrics with various scales are reported ( i.e accuracy, f1, correlation).
Datasets with Instance-level information In this setting we focus on NLG evaluation as these scores are among the easiest to be collected.
We focus on ﬁve different tasks: summary evaluation, image description, dialogue and translation.
For summary evaluation , we use TAC08 [32], TAC10,
TAC11 [69], RSUM [9] and SEVAL [41].
For sentence-based image description we rely on FLICKR [101] and for dialogue we use PersonaChat (PC) and TopicalChat (TC) [64].
Finally for machine translation , we rely on the multilingual quality estimation (MLQE) introduced in Ranasinghe et al.
For all datasets except MLQE, we consider automatic metric based on S3 (both variant pyr/resp) [72], ROUGE [59] (including 5 of its variants [67]), JS [1-2] [60], Chrfpp [77], BLEU, BERTScore [105], MoverScore [107].
For MLQE we solely consider several version of BERTScore, MoverScore and ContrastScore.
We also add human evaluation which is speciﬁc to each dataset.
All details corresponding to these dataset can be found in Appendix A.
5.2 Task-level Aggregation Experiments
In this section, we address the aggregation problem when task-level information is available.
We ﬁrst study the ﬁnal ranking obtained by different methods on GLUE, SGLUE, and XTREM.
Then, we assess the robustness of σ ∗ when removing tasks.
Comparison with mean-aggregation To compare the rankings σ mean and σ ∗ , we compute (i) the
Dataset
[Table] [Table]
[Figure]
agreement rate (in %) which is the proportion of common top-ranked systems between σ mean and σ ∗ , and (ii) the Kendall Tau correlation ( τ ) between the rankings.
Results.
In Tab. 2, we compare the rankings of aforementioned methods for Top K systems (strongest systems) and Last K systems (weakest systems).
For the three benchmarks, we observe a high correlation between the ﬁnal rankings ( i.e. correlation values are in the range [0 . 80 , 1] ).
To a ﬁner degree, we also observe that methods tend to agree on which are the best/worst systems.
Although σ mean and σ ∗ agree on the best/worst systems, they do not rank them in the same order (see Tab. 3).
For instance, on XTREM, the third-best system according to σ mean (rank 2) is actually the sixth-best system according to σ ∗ .
Takeaways.
When changing the aggregation function, the response to our initial question "what are the best systems?" varies.
How does the addition/removal of new tasks/metrics affect the ranking?
When building a benchmark, practitioners can always add new tasks to reﬁne the model performance assessment (it boils down to adding a new column in Tab. 1).
In this experiment, we analyze how adding and removing tasks affect the rankings of the aggregation procedures.
Setting.
We compare the rankings obtained when considering a subset of the tasks and the one obtained using all the tasks.
Formally, for a given number of tasks t ≤ T , we randomly sample t tasks, compute the rankings obtain by our procedure and by the mean procedure, σ ∗ ,t and σ mean , on these t tasks, and ﬁnally compute the Kendall correlation between σ ∗ ,t (resp.
σ mean,t ) and the "ground truth" σ ∗ (resp.
σ mean ).
We repeat this random sampling 100 times to obtain a mean/variance plot of the correlations in the function of the number of sub-tasks.
Results.
We report in Fig.
5(a,f) the obtained results for varying size of subsets.
Interestingly we observe that correlation between the σ ∗ ,t and σ ∗ is consistently higher than the one between σ mean,t and σ mean .
This difference is particularly visible in the range [0 .
25% − 0 .
We observe a similar behavior when considering SGLUE (see Fig.
Takeaways.
The ranking from σ ∗ is more robust to task addition/drop than the one from σ mean .
5.3 Instance-level Aggregation Experiments
For instance level aggregation, we conduct experiments on the 9 aforementioned data-sets.
We study both the ﬁnal ranking obtained for each aggregation ( i.e. σ l , σ 2 l and σ mean ) as well as the effect of task addition/removal.
[Figure]
Global analysis When conducting our experiments, we observe that the three different aggregation procedures lead to three different stateof-the-art in 8 datasets out of 9.
Furthermore, they never agree on the top 3 systems.
In what follows, we compare σ l , σ 2 l and σ mean .
Setting.
We compare the obtained ranking by (1) comparing the Kendall correlation (see
Fig. 6), (2) comparing the number of agreements between the top N systems, (3) computing the
Kendall correlation between them.
Results.
When considering the agreement analysis of Fig. 4(a), we observe that σ 2 l and σ mean select a high number of common top systems.
However, the correlation of the rankings induced by σ 2 l and σ mean on these top-systems is low (see Fig. 4(b)).
This is also the case for the correlation of the entire rankings (see Fig. 4).
In short σ 2 l and σ mean select similar systems but rank them differently.
Similar analysis shows that σ l disagrees from σ 2 l and σ mean both on top systems and on their orders.
Takeaways.
σ 2 l exhibits a more similar behavior than σ l with respect to σ mean .
What is the impact of removing/adding tasks?
In NLG, different metrics ( i.e. task) assess the quality of a generated sentence along a different axis.
As adding a new metric may affect the ﬁnal ranking, we investigate the impact of task addition/removal on the ﬁnal system ordering.
Setting.
Similarly to ??
, we study the evolution between the correlation between the ranking computed on a subset of tasks and the ground truth ranking (computed on all tasks) for each of the three procedures.
Results.
We observe that for all datasets both σ 2 l , σ l obtain higher correlation and lower variance compared to σ mean when adding/removing tasks.
Results for RSUM reports a similar trends (see
Fig. 9).
Takeaways.
The ranking obtained with either σ l or σ 2 l are more robust to task addition/drop than the one from σ mean .
Conclusion and Future Works
In this paper, we introduced new aggregation procedures to rank systems when either task level scores or instance level scores are available.
Our methods, which are theoretically grounded and rely on
Kemeny ranking consensus, address fundamental ﬂaws of the widely used arithmetic mean.
We conducted extensive numerical experiments, which show that our methods are both more reliable and more robust than the mean aggregation while leading to different conclusions on which are the best systems.
Overall, when task-level (resp.
instance-level) information is available, we would recommend using the aggregation procedure σ ∗ (resp.
σ 2 l ) rather than the σ mean (resp.
σ mean and σ l ).
Although we focused on NLP benchmarks, our methodology could be applied in other modalities ( e.g. Computer Vision, Audio).
Another interesting avenue would be to consider a weighted version of the Kemeney consensus where weight could reﬂect task-speciﬁc criteria ( e.g fairness, robustness, complexity).  Acknowledgements and Fundings
We thank Maxime Peyrard, Saibo Geng, and Wei Zhao for sharing their collected dataset.
A warm thanks to Matthieu Labeau for his sharp eyes and excellent comments when reading our paper manuscript.
This work was granted access to the HPC resources of IDRIS under the allocation 2021-101838 and 2021-101913 made by GENCI.
Nathan is funded by the projet ANR LIMPID.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or [N/A] .You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] • Did you include the license to the code and datasets? [N/A] • Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] Our works aim at developing safe ai systems.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Both in the supplementary and main paper (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] In the supplementary.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes]
In the supplementary.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes]
In the supplementary. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] .
(b) Did you mention the license of the assets?
[N/A] .
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] .
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] .
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] .
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
We report in Tab. 5 and Tab. 5 global statistics of the data described in . Overall our conclusions are drawn from a total of over 270k scores.
[Table] [Table] B Additional Experiments In this section, we report additional experimental results including the details of the robustness to the scaling experiment (see Ssec.
B.1), the ranking on XTREM (see Ssec.
B.2), complete results on the experiments when adding adding/removing metrics/tasks when Task Level Information (see
Sssec.
B.3.1) and Instance Level Information is available (see Sssec.
B.3.2).
In the main paper we only report the aggregated score for the agreement analysis when instance level informatino is available, we report detailed results on Ssec.
B.4.
B.1 Toy Experiment on Scale
We display in Fig.
7 the results of the toy experiment on scaling robustness.
When corrupting one task by rescaling, we see that the error of the ranking induced by σ mean increases to 1 while the error of ranking-based aggregation remains constant.
[Figure]
B.2 Ranking of SGLUE
We display in Tab. 6 the resulting ranking on the three considered benchmark.
Although aggregation procedures tend to agree on good and bad systems, when changing the aggregation function, the
[Figure]
rankings vary.
Thus conclusion depending on the answer to the initial question ”what are the best systems?” might change
[Figure]
[Table]
B.3 Complete results on the task addition/removal experiments B.3.1 Task Level Aggregation
Results of task addition/removal experiments when Task level information is available are reported in Fig. 8.
Overall, we observe that ranking-based aggregation is more robust than mean-based aggregation.
B.3.2 Instance Level Aggregation
Results of task addition/removal experiments when instance-level information is available are reported in Fig.
Overall, we observe that ranking-based aggregation is more robust than mean-based aggregation.
B.4 Agreement analysis on Instance Level Aggregation
To further complete the experiment on agreement analysis of Fig.
5.3 report results on individual tasks.
We report in Fig.
10 the correlation of ranking when Top K systems for different values of K while Fig.
11 reports the agreement analysis.
B.5 Possible Extensions
For the futur we would like to extend the proposed ranking procedures to:
• Sequence Labelling Task [15, 16, 21] where instances are sequences, thus ordering matters.
• Emotion classiﬁcation [22, 37, 45, 97] where score are continuous.
[Figure]
• NLG [19, 20, 24–26, 88] where instance are sentences where both ordering and content matter.
C Dispersion Analysis
In this section, we introduce the notion of dispersion across a set of different rankings σ 1 , . . .
, σ T .
C.1 Dispersion as a measure performance
Suppose you have two ranking candidates, σ A and σ B , to summarize σ 1 , . . .
, σ T .
A natural question consists in measuring the performance of σ A and σ B .
Denoting by d the Kendall distance, a natural measure is
[Equation]
[Figure]
Of course, in our task-level framework, our candidate σ ∗ will achieve better performance than the mean since it is designed to minimize this quantity.
From a probabilistic point of view
[Figure]
one can see σ 1 , . . .
, σ T as an i.i.d.
realization of a r.v. Σ on the symmetric group S N .
Denoting by P is law and E the associated expectation, Equation (1) is an approximation of σ (cid:55)→ E
[ d ( σ,
[Equation]
C.2 Dispersion to measure ranking complexity
In order to take into account the intrinsic complexity of ranking σ 1 , . . .
, σ T , a natural way would be to compute some kind of Dispersion among these permutations.
Using the same notations as before, one can rely on the pairwise distance.
[Equation]
On a practical perspective, the computational complexity of Equation (3) is O
( T 2 N log N ) .
Nevertheless, it is possible to efﬁciently approximate this value via empirical stopping algorithms based on
Berstein or Hoeffding bounds [56, 65].
Notice that the pairwise distance has a solid theoretical foundation as to be the base for a measure of the spread.
It is an empirical approximation of E [ d (Σ , Σ (cid:48) )] where Σ and Σ (cid:48) are independent with law P .
Therefore, both are directly related to the noise level in different probabilistic models for permutations [43].
Moreover, the former is the basis of statistical uniformity tests for ranked data building upon it [12].
Remark 2.
A known relation between E [ d ( σ, Σ)] and E [ d (Σ , Σ (cid:48) )] has remarkable consequences on assessing the quality of our estimation.
It says that the former is bounded below (respectively, above)
[Figure]
by 0.5 (respectively, 1) times the latter [56].
This fact has broad practical application since it means that the measure of the expected quality of the estimators σ 2 l and σ l is lower an upper bounded by the intrinsic difﬁculty of the problem, which we can approximate via sample statistics in Equation (3) .
We conclude by noting that E [ d (Σ , Σ (cid:48) )] is the natural ranking counterpart of the variance for realvalued aggregation σ mean .
However, when the scores are not on the same scale, then the variance of the scores is no longer interpretable as a measure of spread in the population.
C.3 Experiments
We report in Tab.
7 the results of the dispersion analysis.
We compare the dispersion to measure performance obtained with the induced ranking by σ ∗ σ mean and the one obtained by 100 random permutations.
Takeaways As expected, σ ∗ obtains a lowest dispersion which further validate our approach.
[Table]