How not to Lie with a Benchmark: Rearranging NLP Leaderboards
Abstract
Comparison with a human is an essential requirement for a benchmark for it to be a reliable measurement of model capabilities.
Nevertheless, the methods for model comparison could have a fundamental ﬂaw - the arithmetic mean of separate metrics is used for all tasks of different complexity, different size of test and training sets.
In this paper, we examine popular NLP benchmarks’ overall scoring methods and rearrange the models by geometric and harmonic mean (appropriate for averaging rates) according to their reported results.
We analyze several popular benchmarks including GLUE, SuperGLUE, XGLUE, and XTREME.
The analysis shows that e.g. human level on SuperGLUE is still not reached, and there is still room for improvement for the current models.
Introduction
The benchmarking approach has a rich history throughout computer science and is now the leading method in machine learning progress validation. In the ﬁeld of Natural Language Processing (NLP), there exist at least 898 benchmarks 1 , the most prominent being GLUE, SuperGLUE, XGLUE, etc., created within a single paradigm. The SOTA-centricity of the language model benchmarking has been since criticized as misleading 2 : the battle for percentage fractions leads to extensive improvements – an increase in the data volume and number of parameters of models, – rather than an intensive improvement in their architectures.
The existing solutions suggest Pareto efﬁciency between the estimated accuracy and computational costs (Dodge et al., 2019) and draw attention to the question of model utility versus overall score (Ethayarajh and Jurafsky, 2021).
The recent work (Dehghani et al., 2021) has also stated that many factors, other than fundamental algorithmic superiority, can lead to a method being evaluated as superior, for example, various subsamples of the benchmark tasks’ result in different leaderboard arrangements. This forms the conception of the "benchmark lottery" , describing the fragility of the main model evaluation instruments.
We state that the delicate question of a general model assessment should not be solved by a simplistic yet incorrect method – the arithmetic mean over all the results.
Stated method does not take into account the scatter of results on various tasks, the differing size of the task test sets (e.g. in SuperGLUE they differ a hundred times, compare 146 test samples in Winograd Schema and 10’000 test samples in ReCoRd(Wang et al., 2019a)), different susceptibility to leaks (Elangovan et al., 2021), including the year of creation (Recognizing Textual Entailment data was collected in 2005 (Dagan et al., 2005), while BoolQ or CommitmentBank data was collected in 2019(Clark et al., 2019), (De Marneffe et al., 2019)).
In this article we present an analysis of the NLP benchmarks’ results, using not the arithmetic mean, but the other metrics: geometric mean and harmonic mean.
As F1 (harmonic mean) is frequently used to normalize Precision and Recall as they are fractions, optimizing the classiﬁer threshold to maximize
F1 leads to a better balance between metrics than the arithmetic mean because it penalizes systems more for the smaller values(Sasaki et al., 2007).
The geometric mean, as noted in the (Fleming and
Wallace, 1986), is the preferred metric to the arithmetic mean in computing performance benchmarks when it comes to normalized values and percentages.
The results change the usual idea of the models’ order on leaderboards: humans still occupy the ﬁrst place in the intellectual task solving, and the best results (1.5-2% worse than humans) belong to
DeBerta(He et al., 2020), T5+Meena (Raffel et al., 2020) and McAlbert+DKM models 3 .
Thus, the contribution of this paper is two-fold:
we present the reviewed model evaluation technique, core for benchmark design, 2.
we re-arrange the currently existing leaderboards of most popular benchmarks.
Previous Work
Evaluation and comparison of NLP models beget a rich history, rising with the Turing test (Turing, 2009).
The next step in the development and assessment of intelligent systems belongs to the MLbenchmark methodology, which aims to bring the solution of the Natural Language Understanding problem closer - General Language Understanding Evaluation(Wang et al., 2019b).
As stated in (Wang et al., 2019a), "Lacking a fair criterion with which to weight the contributions of each task to the overall score, we opt for the simple approach of weighing each task equally, and for tasks with multiple metrics, ﬁrst averaging those metrics to get a task score.
" All the GLUE-based benchmarks follow this methodology.
However, apart from the GLUE format, other benchmarks have provided several alternatives to evaluate the overall model contribution.
KILT, a Benchmark for Knowledge Intensive Language Tasks (Petroni et al., 2020), avoids calculating the overall result, and also do not compare the result with the human level, but only provides metrics for individual tasks.
DecaNLP (McCann et al., 2018) makes a rating using not the average, but the sum of points for all tasks.
This approach allows balancing the contributions of different tasks to the overall metric.
Method
We arrange the NLP benchmark results using the publicly available model scores for all the tasks to calculate new overall scores.
Other available options from Pythagorean means - harmonic mean and geometric mean - can also be considered: we have centered our research around 2 simple statistics that are widely used for averaging fractions (lun Chou, 1969) or normalized values (Fleming and Wallace, 1986) among the possible alternatives.
We did not consider other measures of central tendency, like median and mode, as the averaged samples more often consist of about 10 measurements, and on them such metrics can give the same results for competing systems. • The arithmetic mean (AM) desribed in eq. 1 is calculated as the sum of the task scores (Xs) divided by the total number of tasks, referred to as N. • The geometric mean (GM) desribed in eq. 2 is calculated as the N-th root of the product of all task scores (with the above conditions), where N is the number of values.
• The harmonic mean (HM) desribed in eq. 3 is calculated as the number of values N divided by the sum of the reciprocal of the values [Equation]
The sections below present the results of the leaderboard re-weighting as of May 2021.
3.1 Reevaluating the Benchmarks
The GLUE (11 tasks for English) , SuperGLUE (10 tasks for English) , XGLUE (11 tasks forlanguages) , and XTREME (4 tasks for 40 languages) provide different scoring metrics for each task, including Accuracy, F1, Matthew’s correlation coefﬁcient, Exact Match, while the overall score is calculated by their simple average.
In cases like these, the geometric mean is appropriate when the data contains values with different units of measure (lun Chou, 1969). The harmonic mean of the task results as a better overall metric has the same grounding as introduction of the F-1 measure over precision and recall (Sasaki et al., 2007): the harmonic mean is more intuitive than the arithmetic mean when computing a mean of ratios. Given the set of metrics with a large scatter, the harmonic mean will be less than the arithmetic mean, penalizing the system more for the errors made.
The harmonic mean is the appropriate mean if the data is comprised of rates, while the geometric mean is used as an unbiased estimation when working with normalized ratios, for example, in ﬁnance (Dittmann and Maug, 2008) or computing benchmarks (Fleming and Wallace, 1986). However, their applicability to a better summarization of the model performance to a single number has been widely discussed, see (Smith, 1988), discussing performance computing: • the harmonic mean is considered the appropriate metric to summarize benchmark results expressed as rates, • while geometric mean is applicable in case of the use of performance numbers that are normalized with respect to one of the results being compared (see 4), • and arithmetic mean should not be used as a summarizing metric with rates, making it the worst choice for results accumulation.
GLUE benchmark (Wang et al., 2019b) combines 11 tasks in various text classiﬁcation and question answering.
Overall score: average of all the task results.
If task has 2 main metrics, these metrics are averaged, then added to the overall average.
Human evaluation: collected on reported human performance numbers from original datasets, not exceeding 200 examples (heavily criticised in (Nangia and Bowman, 2019)).
The human baseline performance on the diagnostic set was provided by the project authors with the help of six NLP researchers annotating 50 randomly selected sentence pairs.
Rearranging the scores: the results of geometric and harmonic mean rearrangement are presented in Tab. 1.
GLUE benchmark seem to be the most reordered of all the ratings considered: the best result by geometric and harmonic means belongs to humans, DeBerta and McAlbert+DKM got apoint demotion, and the other models got severely rearranged their places.
N
Name
AM
HM
GM
CoLA
SSTMRPCMean
STS-BMean
QQPMean
MNLIm
MNLImm
QNLI
Human 87.
DeBERTa 90.
MacAlbert +DKM 90.
T 90.
PING-AN 90.
ERNIE 90, 84.
3.3 SuperGLUE
SuperGLUE (Wang et al., 2019a) is the sophisticated version of the GLUE benchmark, combiningtasks with a higher demand for higher intellectual abilities.
Task data must is available under various licenses that allow use and redistribution for research purposes.
Overall score: average of all the task results.
If task has 2 main metrics, these metrics are averaged, then added to the overall average.
Human evaluation: ready-made estimates for WiC, MultiRC, RTE, and ReCoRD datasets, the other tasks being evaluated by the project creators with the help of crowdworker annotators through
Amazon’s Mechanical Turk.
Rearranging the scores: Re-weighting the results using the geometric mean and harmonic mean again makes signiﬁcant changes to the original ranking: the top-3 result (human) is ranked top-1, the
DeBerta and T5 models are shifted down 1 position, PAI ALbert and Nezha Plus models swap their places, see Tab. 2.
[Table]
3.4 XTREME
The XTREME benchmark (Hu et al., 2020)covers 40 typologically diverse languages fromlanguage families and includes 9 tasks that require analysis of different levels of syntax or semantics.
Overall score: 2-step averaging: 1) calculating average for each task on all languages 2) calculating average on all tasks.
Human evaluation: 2-step averaging:
step 1: ready-made estimates from the original datasets taken and extrapolated to all unestimated languages; besides, for some datasets there were no original estimates provided (POS) and an empirical estimate of 97% was taken based on (Manning, 2011); no estimates for NER and sentence retrieval tasks; 2. step 2: all the task results averaged together.
Rearranging the scores: the results of applying the geometric mean and harmonic mean did not change the current ranking of the models - the quality spread between them is high enough for the metrics averaging them to retain the current order.
[Table]
3.5 XGLUE
The XGLUE benchmark(Liang et al., 2020)consists of 11 problems in 19 languages and evaluates the performance of multilingual pre-trained systems in terms of their ability to cross-language understanding and natural language generation.
Overall score: 2-step averaging: 1) calculating average for each task on all languages 2) calculating average on all tasks.
Human evaluation: not provided.
Rearranging the scores: Since the human level is not measured in the benchmark, we can only compare the 2 present models with each other.
Tab. 4 shows the results - the difference in the quality of the models is large enough to preserve their ranking on all averaging metrics.
[Table]
Results and Discussion
The results show that the ranking of results within a single leaderboard can ﬂuctuate signiﬁcantly.
So, in GLUE, the ﬁrst place in terms of the harmonic and geometric mean belongs to the result occupying the 16th line in the arithmetic mean.
In SuperGLUE, the permutation is not so striking - the third result is on the 1st place.
On the XTREME and XGLUE benchmarks system ranking is preserved.
Since all three averaging metrics considered are subject to different biases, we present the statistical measurements of the top-3 SuperGLUE results in Tab. 5.
Human results have the highest total points for all tasks (as in the DecaNLP methodology), while the standard deviation and variance are greater than top-2 and top-3 models.
[Table] The following topics remain debatable and need special attention of the community:
Different metrics for obtaining the average value (arithmetic, geometric, harmonic) have different restrictions on the accepted values (for example, not every one can take negative or zero values).
At the same time, metrics that take zero and negative values are actively used in measuring various skills - MCC metric on SuperGLUE diagnostics can be negative, other metrics can be equal to or greater than zero, and they are averaged altogether.
Correct averaging of the overall score for multilingual benchmarks creates additional problems while performing the averaging operation in 2 stages: for all languages and all tasks.
In addition to the problem of the main averaging metric, we left outside of the scope the problem that was also discovered within the framework of this study: human benchmark scores on various tasks were obtained in a very different way.
Conclusion
In this paper we present an alternative method to arrange the popular NLP benchmark results, elaborating on several task evaluation.
We analyze popular benchmark averaging methods and provide new insight into model comparison.
Namely, we obtain the following results:
• for popular benchmarks GLUE and SuperGLUE we can conclude that their overall score is subject to bias due to outliers; the alternative arrengement methods end with signiﬁcantly different ordering of the results; • rebuilding leaderboards using other metrics (geometric or harmonic mean) allows one to conclude that human result is the ﬁrst in the rankings; • in XGLUE leaderboard human result is obtained by extrapolation from one language to others, while in practice the level of problem-solving by native speakers of different languages varies; • the last ﬁnding could be extended to other multilingual benchmarks also.
We believe that an unbiased generalization of the benchmark scores is a necessity for the community to target language modelling.
The tracking of the complex process of model improvement over time can require the improvement of the benchmark design itself. The expansion towards multilingualism and multimodality of the new benchmark practices plus new models makes the stated problem more urgent and we hope our help to foster research in this direction. _ Appendix See tables below
5.1 Appendix5.2 Appendix5.3 Appendix T a b l e 6 : S up e r G L U E b e n c h m a r k r e s u lt s no r m a li ze d t
o
hu
m
a n l e v e l: g e
o m e t r i c m ea n i s t
h e m a i n m e t r i c
R a nk
Sup er G
L U
E
M o d e l
H M
G M
A M
B oo l Q
C B m e a n C O P A M u l t
i
RC M e a n R e C o RD m e a n
R TE W i C W S C
AX - b
H u m a n 87 ,( 1 ) 88 ,( 1 ) 89 ,( 1 ) 89 ( 1 ) 97 ,( 1 ) ( 1 ) 66 ,( 1 ) 91 ,( 1 ) 93 ,( 1 ) 80 ( 1 ) ( 1 ) 76 ,( 1 ) 99 , D e B E R T a 0 , 0 , 0 , 1 , 0 , 0 , 1 , 1 , 0 , 0 , 0 , 0 ,
T+ M ee n a 0 , 0 , 0 , 1 , 0 , 0 , 1 , 1 , 0 , 0 , 0 , 0 , T0 , 0 , 0 , 1 , 0 , 0 , 1 , 1 , 0 , 0 , 0 , 0 , 0 , R a nk
G L U E
M o d e l
H M G M A M C o L
A SS TM R
P C
M e a n S T S - B M e a n
QQ P M e a n M N
L I - m M N L Imm Q N
L I R TE W N
L I AX H u m a n 86 ,( 1 ) 86 ,( 1 ) 87 ,( 1 ) 66 ,( 1 ) 97 ,( 1 ) 83 ,( 1 ) 92 ,( 1 ) 69 ,( 1 ) 92 ( 1 ) 92 ,( 1 ) 91 ,( 1 ) 93 ,( 1 ) 95 ,( 1 ) 76 , 6 ( 1 ) D e B
E R T a 0 , 1 , 1 , 1 , 0 , 1 , 1 , 1 , 0 , 0 , 1 , 0 , 0 , 0 , M c A l b e r t 0 , 1 , 1 , 1 , 0 , 1 , 1 , 1 , 0 , 0 , 1 , 0 , 0 , 0 , T0 , 1 , 1 , 1 , 0 , 1 , 1 , 1 , 1 , 0 , 1 , 0 , T a b l e 7 :
G L U E r e s u lt s w it h a ll m e t
r i c s f o r t h e t a s k s , i n c l ud i ng t a s k s w it
h doub l e m e t r i c s . N M od e l G M
A M C o L A SS TM R P
C F M R P C
A cc M R P C M ea n
S T S - B P r s
S T S - B S p r S T S - B M ea n
QQ P F QQ P A cc QQ P M ea n M N
L I m M N L I mm QN
L I
R L I AX H u m a n 86 , 87 ,66 ,97 ,86 ,80 ,83 ,92 ,92 ,92 ,59 ,80 ,69 , 92 ,91 ,93 ,
D e B E R T a 86 , 90 ,71 ,97 ,   92 ,92 ,92 ,76 ,90 ,83 ,91 ,91 ,99 ,93 , H F L i F L Y
T E K 86 , 90 ,74 , 94 ,92 ,93 ,92 ,92 ,92 ,74 ,90 ,82 ,91 ,91 ,97 , T85 , 90 ,71 ,97 ,92 ,90 ,91 ,93 ,92 ,92 ,75 ,90 ,82 ,92 ,91 ,96 ,92 , P I NGAN 85 , 90 ,73 ,97 ,    92 ,92 ,76 , 83 ,91 ,91 ,97 ,91 ,
E R N I E 85 , 90 ,74 ,97 ,93 ,91 ,92 , 92 ,92 ,75 ,90 ,83 ,91 , 96 ,90 , T a b l e 8 : S u p e r G L U
E r e s u lt s w it h a ll m e t r
i c s f o r t h e t a s k s , i n c l ud
i ng t
a s k s w i t h doub l e m e t r i c s .
R a nk M o d e l G M A M
B oo l Q CB F CB A cc C B m e a n
C O P A M u lti RC
F M u lti RC A cc
M u l t i RC M e a n R e C o R D F R e C o R D
A cc R e C o RD m e a n R TE
W i H u m a n 88 , 89 , 95 ,98 ,97 , 81 ,51 ,66 ,91 ,91 ,91 ,93 ,
D e B
E R T a 87 , 90 ,90 ,95 ,97 ,96 ,98 ,88 ,63 ,75 ,94 ,94 ,94 ,93 ,77 ,95 ,
T+ M ee n a 87 , 90 ,91 ,95 ,97 ,96 ,97 ,88 , 75 ,94 ,93 ,93 ,92 ,77 ,95 , T86 , 89 ,91 ,93 ,96 ,95 ,94 ,88 ,63 ,75 ,94 ,93 ,93 ,92 ,76 , P A I A l b e r t 85 , 86 ,88 ,92 ,96 ,94 ,91 ,84 ,54 ,69 , 88 ,88 ,88 ,74 ,
N E Z HA
P l u s 82 , 86 ,87 ,94 , 95 ,93 ,84 ,55 ,69 ,90 ,89 ,89 ,89 ,74 ,