Ranger: A Toolkit for Effect-Size Based Multi-Task Evaluation (2023)

Abstract: In this paper, we introduce Ranger - a toolkit to facilitate the easy use of effect-size-based meta-analysis for multi-task evaluation in NLP and IR. We observed that our communities often face the challenge of aggregating results over incomparable metrics and scenarios, which makes conclusions and take-away messages less reliable. With Ranger, we aim to address this issue by providing a task-agnostic toolkit that combines the effect of a treatment on multiple tasks into one statistical evaluation, allowing for comparison of metrics and computation of an overall summary effect. Our toolkit produces publication-ready forest plots that enable clear communication of evaluation results over multiple tasks. Our goal with the ready-to-use Ranger toolkit is to promote robust, effect-size-based evaluation and improve evaluation standards in the community. We provide two case studies for common IR and NLP settings to highlight Ranger’s benefits.

## introduction we in the nlp (natural language processing) and ir (information retrieval) communities maneuvered ourselves into somewhat of a predicament: we want to evaluate our models on a range of different tasks to make sure they are robust and generalize well. however, this goal is often reached by aggregating results over incomparable metrics and scenarios @xcite . this in turn makes conclusions and take away messages much less reliable than we would like. other disciplines, such as social and medical sciences have much more robust tools and norms in place to address the challenge of meta-analysis. in this paper we present ranger -a toolkit to facilitate an easy use of effect-size based metaanalysis for multi-task evaluation. ranger produces beautiful, publication-ready forest plots to all authors contributed equally help everyone in the community to clearly communicate evaluation results over multiple tasks. ranger is written in python and makes use of matplotlib. thus it will be easy and timeefficient to customize if needed. with the effect-size based meta-analysis @xcite ranger lets you synthesize the effect of a treatment on multiple tasks into one statistical evaluation. since in meta-analysis the influence of each task on the overall effect is measured with the tasks' effect size, meta-analysis provides a robust evaluation for a suite of tasks with more insights about the influence of one task for the overall benchmark. with the effect-size based metaanalysis in ranger one can compare metrics across different tasks which are not comparable over different test sets, like ndcg, where the mean over different test sets holds no meaning. ranger is not limited to one metric and can be used for all evaluation tasks with metrics, which provide a samplewise metric for each sample in the test set. ranger can compare effects of treatments across different metrics. how the effect size is measured, depends on experiment characteristics like the computation of the metrics or the homogeneity of the metrics between the multiple tasks. in order to make ranger applicable to a wide range of multi-task evaluation, ranger offers effect size measurement using the mean differences, the standardized mean difference or the correlation of the metrics. in order to have an aggregated, robust comparison over the whole benchmark, ranger computes an overall combined summary effect for the multi-task evaluation. since these statistical analysis are rather hard to interpret by only looking at the numbers, ranger includes clear visualization of the meta-analysis comprised in a forest plot as in figure 1 . in order to promote robust, effect-size based evaluation of multi-task benchmarks we open source the ready-to-use toolkit at: https://github.com/metesertkan/ranger ## related work in the last years, increasingly more issues of benchmarking have been discussed in nlp @xcite and ir @xcite . bowman and dahl (2021) raise the issue that unreliable and biased systems score disproportionately high on benchmarks in natural language understanding and that constructing adversarial, out-ofdistribution test sets also only hides the abilities that the benchmarks should measure. @xcite notice that the now common practices for evaluation lead to unreliable, unrealistically positive claims about the systems. in ir one common multi-task benchmark is @xcite , which evaluates retrieval models on multiple tasks and domains, however in the evaluation the overall effect of a model is measured by averaging the ndcg scores over each task. as the ndcg score is task dependent and can only be compared within one task, it can not be averaged over different tasks and the mean of the ndcg scores does not hold any meaning. thus there is the urgent need in the nlp and ir community for a robust, synthesized statistical evaluation over multiple tasks, which is able to aggregate scores, which are not comparable, to an overall effect. to address these needs @xcite propose effect-size based meta-analysis with the use case of evaluation in ir for a robust, aggregated evaluation over multiple tasks. similarily, @xcite propose a new method for ranking systems based on their performance across different tasks based on theories of social choice. in nlp and ir research exist numerous single evaluation tools @xcite , however to the best of our knowledge there exists no evaluation tool addressing the needs for robust, synthesized multi-task evaluation based on meta-analysis. in order to make this robust, effect-size based evaluation easily accessible for a broad range of tasks, we present our ranger toolkit and demonstrate use cases in nlp and ir. ## case study nlp: glue benchmark in order to demonstrate the usage of the ranger toolkit for various multi-task benchmarks, we conduct an evaluation on the popular general language understanding (glue) benchmark @xcite . we train and compare two classifiers on the glue benchmark: one classifier based on bert @xcite , the latter based on a smaller, more efficient transformer model trained on bert scores, namely distilbert @xcite foot_0 . the official evaluation metric for two of the nine tasks (for cola and sts-b) is a correlation-based metric. since these correlation-based metrics can not be computed sample-wise for each sample in the test set, the effect-size based meta-analysis can not be applied to those metrics and we exclude these two tasks from our evaluation. we conduct the effect-size based meta-analysis based on the accuracy as metric and use standardized mean difference to measure the effect-size (type in ranger toolkit is 'smd'). we illustrate the meta-analysis of the bert and distilbert classifier in figure 2 . we also publish the walk-you-through jupyter notebook in the ranger toolkit to attain this forest plot for glue. the location of the black diamonds visualizes the effect of the treatment (distilbert) compared to the baseline (bert), whereas the size of the diamonds refers to the weight of this effect in the overall summary effect. we can see that using dis-tilbert as base model for the classifier compared to bert, has overall effect of a minor decrease in effectiveness. this behaviour is similar with the results on the mnli, qnli, and qqp where we also notice that the confidence intervals are very narrow or even non existent in the forest plot. for mrpc and sst-2 there is also a negative effect, however the effect is not significant, since the confidence intervals overlap with the baseline performance. for rte and wnli the effect of using distilbert compared to bert is rather big compared to the summary effect, where for rte the mean is 8% lower and for wnli the mean is 11% higher than for the bert classifier. however the large confidence intervals of these tasks indicate the large variability in the effect and thus the weight for taking these effects into account in the summary effect are rather low (0.7% and 1.7%). overall the summary effect shows that the dis-tilbert classifier decreases effectiveness consistently by 2%. since the confidence intervals are so narrow for the overall effect and do not overlap with the baseline (bert classifier), we see that the overall effect is also significant. we conduct a meta-analysis of the evaluation results based on ndcg@10 as metric and measure the effect-size with the mean difference (type is 'md'). the output of the ranger toolkit is illustrated in figure 3 . we publish a walk-through jupyter notebook in the ranger toolkit to attain this forest plot for beir benchmark evaluation. in figure 3 the effect size, the weight of the effect on the overall effect as well as the mean and confidence intervals of the effect are visualized. as an extension for ir we also visualize the ndcg@10 performance and j@10 judgement ratio from @xmath0. for nfcorpus and trec podcast we see a small positive effect of uni-colberter compared to tas-b, however the confidence intervals are overlapping with the baseline performance incdicating no clear positive effect on these tasks. for tripclick, dbpedia entity and trec robust 04 we see a consistent and significant small positive effect with narrow confidence intervals of uni-colberter and this effect is even greater for antique and trec covid. notice the great confidence intervals for trec covid, since the evaluation of trec covid is only based on 50 queries and thus 3 https://trec.nist.gov/ its influence for the overall effect should be and is the lowest (6.2%) among the test sets. the judgement ratio j@10 in the left most column shows the percentage of judged documents in the top 10 of retrieved results. analyzing the judgement ratio one can also get an understanding of how reliable the evaluation results are and how comparable the results of the two different retrieval models are, since a high difference in judgement ratio could indicate lower comparability of the two models with the respective test set. overall the summary effect of uni-colberter compared to tas-b is consistent and significantly positive, increasing effectiveness by 0.05. ## conclusion we presented ranger -a task-agnostic toolkit for easy-to-use meta-analysis to evaluate multiple tasks. we described the theoretical basis on which we built our toolkit; the implementation and usage; and furthermore we provide two cases studies for common ir and nlp settings to highlight capabilities of ranger. we do not claim to have all the answers, nor that using ranger will solve all your multi-task evaluation problems. nevertheless, we hope that ranger is useful for the community to improve multi-task experimentation and make its evaluation more robust.