DUMB : A Benchmark for Smart Evaluation of D utch Models (2023)

Abstract: We introduce the Dutch Model Benchmark: DUMB. The benchmark includes a diverse set of datasets for low-, medium- and high-resource tasks. The total set of nine tasks includes four tasks that were previously not available in Dutch. Instead of relying on a mean score across tasks, we propose Relative Error Reduction (RER), which compares the DUMB performance of language models to a strong baseline which can be referred to in the future even when assessing different sets of language models. Through a comparison of 14 pre-trained language models (mono- and multi-lingual, of varying sizes), we assess the internal consistency of the benchmark tasks, as well as the factors that likely enable high performance. Our results indicate that current Dutch monolingual models under-perform and suggest training larger Dutch models with other architectures and pre-training objectives. At present, the highest performance is achieved by DeBERTaV3 (large), XLM-R (large) and mDeBERTaV3 (base). In addition to highlighting best strategies for training larger Dutch models, DUMB will foster further research on Dutch. A public leaderboard is available at https://dumbench.nl.

## introduction to evaluate and compare new and existing language models, a reliable method of comparing model quality is essential. for this reason, several benchmark suites have been proposed, such as english glue @xcite and superglue @xcite . however, at present, there is no standard benchmark for dutch. the currently available dutch language models are @xcite , which is a dutch version of bert base @xcite , and three versions of robbert @xcite which are dutch versions of roberta base @xcite . direct comparisons between these models have focused on several tasks: sentiment analysis, natural language inference, coarse-grained part-of-speech tagging and three-class named entity recognition, where robbert often outperforms bertje @xcite . results are not completely consistent, however. some studies found that rob-bert performs better than bertje at specific tasks @xcite @xcite , whereas other studies find the opposite @xcite . other works assume higher performance for specific models and either exclusively experiment with bertje @xcite @xcite , or exclusively experiment with robbert @xcite . by developing a new benchmark, we aim to reduce the present unclarity of current evaluations and obtain insights into potential performance improvements for future development of new dutch models. we also hope that this work will foster further research on dutch, including the development of decoder-based models. indeed, we see the establishment of such a benchmark, together with the evaluation of existing encoder-based models, as a necessary step towards making it possible to also devise a solid evaluation framework for generative models, which is complicated by the high degree of variability related to prompts and outputs. ## dumb tasks a general language model should be able to perform well at different types of tasks. therefore, we balance our benchmark to contain wordlevel, word-pair-level, sentence-pair level and document-level tasks. moreover, we include tasks having different orders of magnitude of training items. we call these low-resource (10 2 ), midresource (10 3 ), and high-resource (>10 4 ) tasks. all included datasets are freely available and most source datasets use permissive licenses that allow (modified) redistribution. we do however not currently share the pre-processed benchmark directly because of licensing restrictions of some datasets. simple instructions to download these datasets and a single pre-processing script are available on our github so that the entire benchmark can be obtained. the included tasks are listed in ## evaluation we conduct an analysis of several pre-trained language models (plms) with dumb. the goal of this analysis is to identify strengths and limitations of current dutch language models, as well as to determine the overall effectiveness of the proposed benchmark. we evaluate the influence of model variants, model sizes and pre-training languages. ## analysis of tasks ## analysis of models to estimate the effects of pre-train languages, model types and model sizes, we fit a linear regression model with those variables as predictors (see appendix c). a model with all three predictors fits the data significantly better than using single predictors (model comparison: @xmath0.05) and explains 83.6% of variance (adjusted r 2 : 73.3%). all three predictors significantly contribute to this model (@xmath1.05); we discuss them in the following subsections. interactions between predictors did not significantly improve the model fit. the average rer scores are shown in ## conclusion we have introduced dumb, a benchmark with nine dutch language tasks, including four new tasks and relative error reduction (rer) as a comparative model evaluation metric. the tasks are internally consistent with positive correlations between tasks across models. some randomness in low-resource task performance might be due to model failure of non-dutch models. robbert models achieve up to 3.6 rer, but multilingual and even english models can achieve up to 14.4 (xlm-r large ) and 15.7 (debertav3 large ) rer, respectively. model comparisons across pre-training languages, model types and model sizes reveal that high multilingual and english performance on dutch tasks can be partially attributed to model size, but to an even larger extent this can be attributed to the de-bertav3 model type. a dutch debertav3 large model could achieve a substantially higher estimated rer of 38.0. this estimation shows that there is much room for improving benchmark scores with better future models. additionally, we encourage evaluating large generative models with dumb. our results are based on a robust standard approach and set a strong baseline that can also be useful to evaluate effective prompt engineering. a public leaderboard is available at dumbench.nl and the benchmark and reference model source code are available at github.com/wietsedv/dumb. the leaderboard will contain all models discussed in this paper, and will be kept up-to-date with newer models and updated versions of the benchmark.