E xplaina B oard: An Explainable Leaderboard for NLP (2021)

Abstract: With the rapid development of NLP research, leaderboards have emerged as one tool to track the performance of various systems on various NLP tasks. They are effective in this goal to some extent, but generally present a rather simplistic one-dimensional view of the submitted systems, communicated only through holistic accuracy numbers. In this paper, we present a new conceptualization and implementation of NLP evaluation: the ExplainaBoard, which in addition to inheriting the functionality of the standard leaderboard, also allows researchers to (i) diagnose strengths and weaknesses of a single system (e.g. what is the best-performing system bad at?) (ii) interpret relationships between multiple systems. (e.g. where does system A outperform system B? What if we combine systems A, B and C?) and (iii) examine prediction results closely (e.g. what are common errors made by multiple systems or in what contexts do particular errors occur?). So far, ExplainaBoard covers more than 400 systems, 50 datasets, 40 languages, and 12 tasks. We not only released an online platform at the website but also make our evaluation tool an API with MIT Licence at Github and PyPi that allows users to conveniently assess their models offline. We additionally release all output files from systems that we have run or collected to motivate “output-driven” research in the future.

## introduction natural language processing (nlp) research has been and is making astounding strides forward. compared to vanilla leaderboards, explainaboard allows users to perform interpretable (single-system , pairwise analysis, data bias), interactive (system combination, fine-grained/common error analysis), and reliable analysis (confidence interval, calibration) on systems in which they are interested. "comb." denotes "combination" and "errs" represents "errors". "per, loc, org" refer to different labels. this is true both for classical tasks such as machine translation @xcite , as well as for new tasks @xcite , domains @xcite , and languages @xcite . one way this progress is quantified is through leaderboards, which report and update performance numbers of state-of-the-art systems on one or more tasks. some prototypical leaderboards include glue and superglue for natural language understanding @xcite , xtreme and xglue @xcite for multilingual understanding, the wmt shared tasks @xcite for machine translation, and gem and genie for natural language generation @xcite , among many others. these leaderboards serve an important purpose: they provide a standardized evaluation setup, often on multiple tasks, that eases reproducible model comparison across organizations. however, at the same time, due to the prestige imbued by a top, or high, place on a leaderboard they also can result in a singular focus on raising evaluation numbers at the cost of deeper scientific understanding of model properties @xcite . in particular, we argue that, among others, the following are three major limitations of the existing leaderboard paradigm: in this paper, we describe explainaboard (see fig. 1 ), a software package and hosted leaderboard that satisfies all of the above desiderata. it also serves as a prototype implementation of some desirable features that may be included in future leaderboards, even independent of the provided software itself. we have deployed explain-aboard for 9 different tasks and 41 different datasets, and demonstrate how it can be easily adapted to new tasks of interest. we expect that explainaboard will benefit different steps of the research process: (i) system developement: explainaboard provides more detailed information regarding the submitted systems (e.g. fine-grained results, confidence intervals), allowing system developers to diagnose successes and failures of their own systems, or compare their systems with baselines and understand where improvements of their proposed methods come from. this better understanding can lead to more efficient and effective system improvements. additionally, explainaboard can help system developers uncover their systems' advantages over others, even when these systems have not achieved state-of-the-art performance holistically. (ii) leaderboard organization: the ex-plainaboard software both provides a readymade platform for easy leaderboard development over different nlp tasks, and helps upgrade traditional leaderboards to allow for more fine-grained analysis. for example, we have already established an explainaboard foot_4 for the existing xtreme benchmark. 6 (iii) broad analysis and understanding: because explainaboard encourages system developers to provide their system outputs in an easy-to-analyze format, these will also help researchers, particularly those just starting in a particular nlp sub-field, get a broad sense of what current state-of-the-art models can and cannot do. this not only helps them quickly track the progress of different areas, but also can allow them to understand the relative advantages of diverse systems, suggesting insightful ideas for what's left and what's next. 7 ## explainaboard as stated above, explainaboard extends existing leaderboards, improving their interpretability, interactivity, and reliability. it does so by providing a number of functionalities that are applicable to a wide variety of nlp tasks (as illustrated in tab. 1). many of these functionalities are grounded in existing research on evaluation and fine-grained diagnostics. ## tasks, datasets and systems we have already added to explainaboard 12 nlp tasks, 50 datasets, and 400 models, foot_10 which cover many or most of top-scoring systems on these tasks.we briefly describe them below, and show high-level statistics in tab. 2. text classification prediction of one or multiple pre-defined label(s) for a given input text. the current interface includes datasets for sentiment classification @xcite , topic identification @xcite , and intention detection @xcite . text-span classification prediction of a predefined class from the input of a text and a span, such as aspect-based sentiment classification task @xcite . we collect topperform system outputs from @xcite . text pair classification prediction of a class given two texts, such as the natural language inference task @xcite . sequence labeling prediction of a label for each token in a sequence. the explainaboard currently includes four concrete tasks: named entity recognition @xcite , part-of-speech tagging @xcite , text chunking @xcite , and chinese word segmentation @xcite . structure prediction prediction of a syntactic or semantic structure from text, where ex-plainaboard currently covers semantic parsing tasks @xcite . text generation explainaboard also considers text generation tasks, and currently mainly focuses on conditional text generation, for example, text summarization @xcite and machine translation . system outputs on text summarization are expanded based on the previous work's collection @xcite as well as recently state-of-the-art systems @xcite while outputs from machine translation are collected from the wmt20. foot_11 ## case study here, we briefly showcase the actual explain-aboard interface through a case study on analyzing state-of-the-art ner systems. ## implications and roadmap explainaboard presents a new paradigm in leaderboard development for nlp. this is just the beginning of its development, and there are many future directions. research revolving on system outputs foot_14 due to the ability to analyze, contrast, or combine results from many systems explainaboard incentivizes researchers to submit their results to explainaboard to better understand them and showcase their systems' strengths. at the same time, ex-plainaboard will serve as a central repository for system outputs across many tasks, allowing for future avenues of research into cross-system analysis or system combination. enriching explainaboard with glass-box analysis explainaboard currently performs blackbox analysis, solely analyzing system outputs without accessing model internals. on the other hand, there are many other glass-box interpretability tools that look at model internals, such as the allennlp interpret @xcite and language interpretability tool @xcite . expanding leaderboards to glass-box analysis methods (see @xcite ; @xcite for a survey) is an interesting future work. in the future, we aim to improve the applicability and usefulness by following action items: (1) collaborate with more leaderboard organizers of diverse tasks and set up corresponding explain-aboards for them. (2) cover more tasks, datasets, models, as well as functionalities.