Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurements (2022)

Abstract: Evaluation is a key part of machine learning (ML), yet there is a lack of support and tooling to enable its informed and systematic practice. We introduce Evaluate and Evaluation on the Hub—a set of tools to facilitate the evaluation of models and datasets in ML. Evaluate is a library to support best practices for measurements, metrics, and comparisons of data and models. Its goal is to support reproducibility of evaluation, centralize and document the evaluation process, and broaden evaluation to cover more facets of model performance. It includes over 50 efficient canonical implementations for a variety of domains and scenarios, interactive documentation, and the ability to easily share implementations and outcomes. The library is available at https://github.com/huggingface/evaluate . In addition, we introduce Evaluation on the Hub, a platform that enables the large-scale evaluation of over 75,000 models and 11,000 datasets on the Hugging Face Hub, for free, at the click of a button. Evaluation on the Hub is available at https://huggingface.co/autoevaluate .

## introduction evaluation is a crucial cornerstone of machine learning-not only can it help us gauge whether and how much progress we are making as a field, it can also help determine which model is most suitable for deployment in a given use case. however, while the progress made in terms of hardware and algorithms might look incredible to a ml practitioner from several decades ago, the way we evaluate models has changed very little. in fact, there is an emerging consensus that in order to meaningfully track progress in our field, we need to address serious issues in the way in which we evaluate ml systems @xcite @xcite . more recent papers use more datasets and metrics, while fewer of them report statistical significance test results. in order to have a clearer idea regarding the way model evaluation has evolved in our field, we have carried out our own analysis on a random sample of emnlp papers from the past two decades, and present our results in figure 1 . it can be observed that the number of evaluation datasets and metrics per paper has increased over time, suggesting that model evaluation is becoming increasingly complex and heterogeneous. however, auxiliary techniques such as testing for significance, measuring statistical power, and using appropriate sampling methods have become less common, making results harder to judge when comparing new results to previous work. we believe that while datasets are now more easily accessible thanks to shared repositories @xcite , model evaluation is still unnecessarily cumbersome, with a fragmented ecosystem and a lack of consensus around evaluation approaches and best practices. the goal of this work is to address three practical challenges in model evaluation for ml: reproducibility, centralization, and coverage. reproducibility: ml systems are extremely sensitive to small (and often undocumented) choices such as random seeds and hyperparameters @xcite . model performance is often not compared with proper statistical testing that takes this variance into account, making many self-reported comparisons unreliable. our goal is to standardize this process and thereby improve the reproduction of ml evaluations. centralization: historically, ml metrics have been poorly documented, exacerbating an already insufficient community-wide understanding of their usage and shortcomings @xcite . as metrics and datasets change, the onus is on the community to keep results up-to-date, causing unnecessary replication of work @xcite and the proliferation of outdated artifacts @xcite . coverage: ml as a field still focuses heavily on accuracy-based metrics. while important, this focus glosses over other critical facets such as efficiency @xcite , bias and fairness @xcite , robustness @xcite , and how these factor into choosing a model @xcite . we introduce the open source evaluate library and the evaluation on the hub platform to address many of these problems. we believe that better evaluation can happen, if we-as a communityestablish better best practices and remove hurdles. ## related work open-source tools for evaluation there is a long history of open source projects aiming to capture various measurements, metrics and statistical testing methods for ml. torchmetrics @xcite implements a large number of model evaluation metrics for pytorch @xcite , which is similar to evaluation metrics found in keras @xcite for tensor-flow. libraries like scikit-learn @xcite , scipy @xcite , statsmodels @xcite , nltk @xcite , trectools @xcite , rl reliability metrics @xcite , networkx @xcite , scikit-image (van der walt et al., 2014), gem @xcite , torchfidelity @xcite ) also support many evaluation measures across many domains. as integrating metrics into specific frameworks can be difficult, there are also many libraries dedicated to individual evaluations for example rouge_score, foot_0 bartscore @xcite , or sacrebleu @xcite . the fragmentation of the ecosystem leads to various problems, such as a wide range of incompatible conventions and apis, or misreporting due to differing implementations and results. in evaluate, we provide one single interface backed by a centralized hub. metrics can easily be shared, are version controlled, have a standardized interface, and allow for multimodal inputs. evaluation as a service the idea of evaluation as a service @xcite , whereby models are submitted for another party to be centrally evaluated, has recently gained traction as a more reproducible way to conduct model evaluation. central evaluation also facilitates holding challenges and competitions around datasets @xcite @xcite as opposed to simply evaluating selfreported model results or comparing model scores with benchmark suites @xcite @xcite @xcite @xcite @xcite . the advantages of conducting evaluation centrally are multiple, including better reproducibility, forward/backward compatibility, and the ability to measure models along multiple axes of evaluation (e.g. efficiency and fairness, in addition to accuracy), which can help contribute towards a more systematic approach to evaluation. issues with evaluation several studies of ml research and practice have been carried out in recent years on different aspects pertaining to ml evaluation, and together they paint a bleak picture of evaluation in our field. for instance, a 2019 largescale replication study of 255 ml papers found that only 63% of the results they reported could be systematically replicated @xcite . a complementary survey of 3,800 papers from papers with code has shown that a large majority of metrics used do not adequately reflect models' performance and that they largely do not correlate with human judgement @xcite . finally, a recent study of 770 papers in machine translation from the last decade found that while 108 new metrics have been proposed for the task, 99.8% of papers continue to use bleu score for reporting results @xcite , despite the fact that the original bleu score @xcite has been shown to vary based on user-chosen parameters such as tokenization, which vary across languages @xcite . these issues motivate the development of the tools presented in this work. ## library: evaluate the evaluate library provides canonical implementations of a large set of evaluation modules. modules are available to the community via a single, easy-to-use api. we provide extensive and detailed documentation cards for each, describing their correct usage, range of values and possible pitfalls, in a similar vein to model and dataset cards @xcite . to facilitate extensibility, each evaluation model lives in a separate git repository, and new modules can be easily contributed. the core library is released under the apache 2.0 license and is available on github, 2 making it easy to adopt and deploy. the library is designed to address the main challenges discussed in section 1. metrics are versioned and documented to support reproducibility within the framework. the core system is centralized to facilitate comparisons across models in a consistent manner supporting best practices, and data is stored in git to allow backups and cloning. finally, the tool is inherently designed for a multi-model, multi-evaluation paradigm supporting broad evaluation coverage by default. ## service: evaluation on the hub the evaluation on the hub platform extends the evaluate library to a free service model: anyone can evaluate any model on any dataset using any compatible metric, without requiring any code. this service utilizes models, datasets, and metrics standardized through the hugging face hub. all evaluation results using this method are produced by the same pipeline with versioned implementations, and so are inherently reproducible. when a new model, dataset, or metric is produced, anyone can rerun the evaluation. as such, evaluation on the hub facilitates large-scale evaluation of over 75,000 models and 11,000 datasets. the service model further supports the goals of reproducibility and centralization. while the evaluate library can ensure that the metrics used are consistent, it cannot ensure that the model was trained and evaluated using a reproducible set of hyperparameters and data. incorporating evaluate into a model hosting and training environment makes it possible to guarantee this consistency. centralization also provides a further benefit of joining these metrics with model and data card documentation. ## use cases evaluate and evaluation on the hub are already actively used by our community for a variety of tasks. there are many applications of these tools, and we highlight some of the most important use cases observed in practice. use case 1: choosing the best model. if the task is known and the aim is to find an appropriate model, the hub leaderboard (which aggregates all the evaluation results for a dataset representative of that task) can act as a trusted source. in case a particularly interesting model is not yet on the leaderboard, its evaluation can easily be triggered, directly from the hub, and its results will automatically appear on the leaderboard, allowing it to be compared to previous models. ## conclusion evaluate and evaluation on the hub aim to facilitate better evaluation of machine learning data and models by improving reproducibility, centralization, and coverage of evaluation tools. evaluate is an open-source, community-driven library that standardizes evaluation. evaluation on the hub is a reproducible no-code alternative for evaluation across models, datasets, and metrics. we hope that this set of tools can help facilitate better best practices for model and data evaluation.