N or B ench – A Benchmark for N orwegian Language Models (2023)

Abstract: We present NorBench: a streamlined suite of NLP tasks and probes for evaluating Norwegian language models (LMs) on standardized data splits and evaluation metrics. We also introduce a range of new Norwegian language models (both encoder and encoder-decoder based). Finally, we compare and analyze their performance, along with other existing LMs, across the different benchmark tests of NorBench.

## introduction this paper provides a suite of standardized tasks and probes for benchmarking of norwegian language models (lms). in addition to collecting a broad range of annotated datasets, we provide precise task definitions, pre-defined data splits and evaluation metrics, with corresponding scripts for streamlining the entire benchmarking pipeline. the resulting resource is dubbed norbench. we furthermore present a range of new transformer-based language models (lms) for norwegian, trained with optimized configurations and architectures, and on different corpora with different pre-processing. our contributions are as follows: ## norbench task descriptions we here briefly describe each task and associated dataset. the number of training examples for the different datasets and their train, development, and test splits are provided in bokmål and nynorsk norwegian has two official written standards: bokmål (bm), used by 85-90% of the norwegian population, and nynorsk (nn). while they are closely related, there can be relatively large lexical differences. the contextualised lms presented in this paper are therefore trained jointly on both varieties, but with the minority variant nynorsk represented by comparatively less data than bokmål (reflecting the natural usage). several previous studies have indicated that joint modeling of bokmål and nynorsk works well for many nlp tasks, like tagging and parsing @xcite and ner @xcite . in cases where the labeled data for our benchmark tasks are available as separate versions for bokmål and nynorsk, we fine-tune models jointly on the combined bm/nn data. one practical advantage of training joint and 'variety agnostic' models, is that only a single model needs to be maintained, and we bypass the need for a separate 'language identification' step. ## norbench baseline methodology below we describe various choices pertaining to fine-tuning the lms for the various tasks. note that, all of the approaches described here should be considered baselines, in the sense that the goal is not to produce state-of-the-art results, but rather to implement simple evaluation approaches allowing for a fair comparison of different lms across the various tasks. ## new norwegian language models a number of large norwegian language models have appeared in recent years: to name only the masked lms, @xcite trained norbert 1 , followed by norbert 2 , and kummervold et al. (2021) introduced nb-bert models, coming in different sizes. in this paper, we present a set of novel masked and text-to-text lms for norwegian trained according to the ltg-bert training recipe by @xcite . we dub these models norbert 3 and nort5 and evaluate their performance across different model sizes and training corpora. ## benchmarking results in addition to our norbert 3 models, we also benchmark these existing models: ## related work evaluating pre-trained language models for particular languages and cross-lingualy is a venerable research sub-field within nlp. well-known benchmark sets for english include glue @xcite , superglue @xcite , and glge @xcite , among others. however, up to now benchmarking lms for norwegian was limited to separate test sets with non-standardised evaluation workflows. scandeval (nielsen, 2023) aims to create a standard natural language understanding benchmark across scandinavian languages (danish, swedish, and norwegian). however, it does not focus on evaluating specifically norwegian tasks and half of its norwegian benchmarks (linguistic acceptability and question answering) are not human-annotated. we address this issue with nor-bench. ## future work we consider norbench to be a dynamic resource that we plan to continually extend in future work, to support additional tasks and additional architectures. while we anticipate including new annotated benchmark data as they may become available in the future, there are also existing datasets that we plan to include in the shorter term, like coreference resolution based on the narc dataset @xcite and negation resolution based on norec neg @xcite . finally, we also plan on adding tasks that more specifically target generative models, including sequence-generation tasks like summarization, but also prompt-based formulations of the existing norbench tasks for few-shot evaluation. ## summary in this paper we have presented norbench, a set of standardized benchmark tasks for systematically evaluating and comparing norwegian language models. the aim of this effort is to provide nlp practitioners with a comprehensive and streamlined service, including a leaderboard, human-annotated datasets, evaluation workflow, and open code implementing this workflow. this paper also describes and evaluates a set of novel norbert 3 masked lms trained on several different norwegian text collections in different model sizes. they are shown to outperform norwegian lms from prior work on the majority of norbench tasks.