NLEB ench+ N or GLM : A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in N orwegian (2024)

Abstract: Norwegian, spoken by only 5 million population, is under-representative within the most impressive breakthroughs in NLP tasks. To the best of our knowledge, there has not yet been a comprehensive evaluation of the existing language models (LMs) on Norwegian generation tasks during the article writing process. To fill this gap, we 1) compiled the existing Norwegian dataset and pre-trained 4 Norwegian Open Language Models varied from parameter scales and architectures, collectively called NorGLM; 2) introduced a comprehensive benchmark, NLEBench, for evaluating natural language generation capabilities in Norwegian, encompassing translation and human annotation. Based on the investigation, we find that: 1) the mainstream, English-dominated LM GPT-3.5 has limited capability in understanding the Norwegian context; 2) the increase in model parameter scales demonstrates limited impact on the performance of downstream tasks when the pre-training dataset is constrained in size; 3) smaller models also demonstrate the reasoning capability through Chain-of-Thought; 4) a multi-task dataset that includes synergy tasks can be used to verify the generalizability of LLMs on natural language understanding and, meanwhile, test the interconnectedness of these NLP tasks. We share our resources and code for reproducibility under a CC BY-NC 4.0 license.

## introduction recent advancements in generative language models (glms) have significantly improved natural language processing (nlp) tasks. however, most models remain partially closed-source due to business competition and data privacy concerns, which hinders transparency, flexibility, and progress in the nlp ecosystem. open-sourcing models can leverage community contributions, facilitate collaboration, and accelerate technological advancements while better controlling data use. this approach is especially beneficial for lowresource languages, aiding their preservation and development. currently, benchmarks focus mainly on languages like english and chinese, leaving low-resource languages (lrls) under-evaluated. most benchmarks for low-resourced languages either cater to discriminative models @xcite @xcite like bert @xcite or are adapted or translated from existing english datasets @xcite . @xcite proposes a closedsource platform, scaneval, for evaluating nordic languages. however, these benchmarks have two limitations: first, many nominal generation tasks are adapted from classification tasks, like multiplechoice questions, which restrict answer options and do not assess generative models' ability to produce longer texts. second, most benchmarks are singletask, with multi-task datasets being particularly scarce. we argue that by designing a multi-task dataset that includes several synergy tasks foot_0 in natural language understanding, it may be possible to evaluate the generalization ability of large language models (llms) in text comprehension. to address these gaps, we propose a comprehensive benchmark, nlebench, specifically tailored to evaluate the natural language generation capabilities in norwegian. nlebench comprises various real-world nlp tasks and provides relative comparisons for norwegian glms with different parameter scales and transformer-based architectures. specifically, our benchmark is purposefully designed to be capability probing, such as instructions specific to norwegian culture and special expressions, and a document-grounded multi-task dataset with human-annotated question-answer pairs and summaries. we hope that such a side-by-side performance benchmark will inspire future research on more advanced glms for norwegian and other lrls. in summary, this paper makes the following contributions: to the best of our knowledge, this is the first benchmarking dataset for norwegian causal/autoregressive language modelling foot_1 . ## norwegian benchmark dataset -nlebench this section introduces tasks in nlebench specifically designed for norwegian glms. the datasets are sourced from three categories: existing datasets, machine-translated datasets using the google translation api, and manually annotated datasets. our native norwegian colleagues evaluated random samples from both the google translation api foot_7 and another free translation api foot_8 supporting norwegian, finding that the former performs better, especially with confusing words and long texts. ## experimental results in this section, we only list key results for the benchmark datasets due to the page limit. more results can be seen in the appendix. ## discussion in this subsection, we present observations from the longitudinal comparison of different models in downstream tasks, as detailed in section 5: 1) while nb-gpt-j-6b did not achieve the highest scores across all tasks, it showed consistent performance and the best perplexity scores compared to our norglms on nearly all tasks. this consistency is likely due to its initial training on large english datasets before being continue-trained on norwegian data. ## conclusion in this paper, we introduced a suite of norwegian generative language models and a comprehensive 21 we have released more norwegian foundation models and datasets and will continue to update and integrate norwegianrelated resources. please follow our github repository for more information.