Large-Scale Contextualised Language Modelling for N orwegian (2021)

Abstract: We present the ongoing NorLM initiative to support the creation and use of very large contextualised language models for Norwegian (and in principle other Nordic languages), including a ready-to-use software environment, as well as an experience report for data preparation and training. This paper introduces the first large-scale monolingual language models for Norwegian, based on both the ELMo and BERT frameworks. In addition to detailing the training process, we present contrastive benchmark results on a suite of NLP tasks for Norwegian. For additional background and access to the data, models, and software, please see: http://norlm.nlpl.eu

## introduction in this work, we present norlm, an ongoing community initiative and emerging collection of largescale contextualised language models for norwegian. we here introduce the norelmo and nor-bert models, that have been trained on around two billion tokens of running norwegian text. we describe the training procedure and compare these models with the multilingual mbert model @xcite , as well as an additional norwegian bert model developed contemporaneously, with some interesting differences in training data and setup. we report results over a number of norwegian benchmark datasets, addressing a broad range of diverse nlp tasks: part-of-speech tagging, negation resolution, sentence-level and fine-grained sentiment analysis and named entity recognition (ner). all the models are publicly available for download from the nordic language processing lab-oratory (nlpl) vectors repository foot_0 with a cc by 4.0 license. they are also accessible locally, together with the training and supporting software, on the two national superclusters puhti and saga, in finland and norway, respectively, which are available to university nlp research groups in northern europe through the nordic language processing laboratory (nlpl). 2 the norbert model is in addition served via the huggingface transformers model hub. 3 norlm is a joint effort of the projects eosc-nordic (european open science cloud) and sant (sentiment analysis for norwegian), coordinated by the language technology group (ltg) at the university of oslo. the goal of this work is to provide these models and supporting tools for researchers and developers in natural language processing (nlp) for the norwegian language. we do so in the hope of facilitating scientific experimentation with and practical applications of state-of-the-art nlp architectures, as well as to enable others to develop their own large-scale models, for example for domain-or applicationspecific tasks, language variants, or even other languages than norwegian. under the auspices of the nlpl use case in eosc-nordic, we are also coordinating with colleagues in denmark, finland, and sweden on a collection of large contextualised language models for the nordic languages, including language variants or related groups of languages, as linguistically or technologically appropriate. ## background bokmål and nynorsk there are two official standards for written norwegian; bokmål, the main variety, and nynorsk, used by 10-15% of the norwegian population. norwegian language legislation specifies that minimally 25% of the written public service information should be in nynorsk. while the two varieties are closely related, there can also be relatively large differences lexically (though often with a large degree of overlap on the character-level still). several previous studies have indicated that joint modeling of bokmål and nynorsk works well for many nlp tasks, like tagging and parsing @xcite and ner @xcite . the contextualised language models presented in this paper are therefore trained jointly on both varieties, but with the minority variant nynorsk represented by comparatively less data than bokmål (reflecting the natural usage). datasets for all our models presented below, we used the following training corpora: the corpora contain ordered sentences (which is important for bert-like models, because one of their training tasks is next sentence prediction). in total, our training corpus comprises about two billion (1,907,072,909) word tokens in 203 million @xcite @xcite sentences. we conducted the following pre-processing steps: ## prerequisites: software and computing developing very large contextualised language models is no small challenge, both in terms of engineering sophistication and computing demands. training elmo-and in particular bert-like models presupposes access to specialised hardwaregraphical processing units (gpus) -over extended periods of time. compared to the original work at google or to our sister initiative at the national library of norway (see below), our two billion tokens in norwegian training data can be characterised as moderate in size. nevertheless, training a single norbert model requires close to one full year of gpu utilisation, which through parallelization over multiple compute nodes, each featuring four gpus, could be completed in about three weeks of wall clock time. at this scale, premium software efficiency and effective parallelization are prerequisites, not only to allow repeated incremental training and evaluation cycles to complete in practical intervals, but equally so for cost-efficient utilisation of scarce, shared computing resources and, ultimately, a shred of environmental sustainability. to prepare the norlm software environment, we have teamed up with support staff at the norwegian national e-infrastructure provider, uninett sigma2, and developed a fully automated and modularised installation procedure using the easy-build framework ( https://easybuild.io ). all necessary tools are compiled from source with the right set of hardware-specific optimizations and platform-specific optimised libraries for basic linear algebra ('math kernels') and communication across multiple compute nodes. this approach to software provisioning makes it possible to (largely) automatically create fully parallel training and experimentation environments on multiple computing infrastructures -in our work to date two national hpc superclusters, in norway and finland, but in principle just as much any suitable local gpu cluster. in our view, making available both a ready-to-run software environment on nordic national e-infrastructures, where university research groups typically can gain nocost access, coupled with the recipe for recreating the environment on other hpc systems, may contribute to 'democratising' large-scale nlp research; if nothing else, it eliminates dependency on commercial cloud computing services. ## related work large-scale deep learning language models (lm) are important components of current nlp systems. they are often based on bert (bidirectional encoder representations from transformers) @xcite and other contextualised architectures. a number of language-specific initiatives have in recent years released monolingual versions of these models for a number of languages @xcite @xcite @xcite @xcite @xcite . for our purposes, the most important such previous training effort is that of @xcite on creating a bert model for finnish -finbert foot_4 as our training setup for creating norbert builds heavily on this; see section 6 for more details. many low-resource languages do not have dedicated monolingual large-scale language models, and instead resort to using a multilingual model, such as google's multilingual bert model -mbert -which was trained on data that also included norwegian. up until the release of the models described in the current paper, mbert was the only bert-instance that could be used for norwegian. 6 another widely used architecture for contextualised lms is embeddings from language models or elmo @xcite . the elmofor-manylangs initiative @xcite trained and released monolingual elmo models for a wide range of different languages, including norwegian (with separate models for bokmål and nynorsk). however, these models were trained on very modestly sized corpora of 20 million words for each language (randomly sampled from wikipedia dumps and common crawl data). in a parallel effort to that of the current paper, the ai lab of the national library of norway, through their norwegian transformer model (no-tram) project, has released a norwegian bert (base, cased) model dubbed nb-bert @xcite . 7 the model is trained on the colossal norwegian corpus, reported to comprise close to 18,5 billion words (109.1 gb of text). in raw numbers, this is about ten times more than the corpus we use for training the norlm models. however, the vast majority of this is from ocr'ed historical sources, which is bound to introduce at least some noise. in section 7 below, we demonstrate that in some nlp tasks, a language model trained on less (but arguably cleaner) data can outperform a model trained on larger but noisy corpora. ## norelmo norelmo is a set of bidirectional recurrent elmo language models trained from scratch on the norwegian corpus described in section 1. they can be used as a source of contextualised token representations for various norwegian natural language processing tasks. as we show below, in many cases, they present a viable alternative to transformer-based models like bert. their performance is often only marginally lower, while the compute time required to adapt the model to the task at hand can be an order of magnitude less on identical hardware. currently we present two models, with more following in the future: each model was trained for 3 epochs with batch size 192. we employed a version of the original elmo training code from @xcite updated to work better with the recent tensorflow versions. all the hyperparameters were left at their default values, except the lstm dimensionality reduced to 2,048 from the default 4,096 (in our experience, this rarely influences performance). training of each model took about 100 hours on four nvidia p100 gpus. these are the first elmo models for norwegian trained on a large corpus. as has already been mentioned, the norwegian elmo models from the elmoformanylangs project @xcite were trained on very small corpora samples and seriously under-perform on semanticrelated nlp tasks, although they can yield impressive results on pos tagging and syntactic parsing @xcite . in addition, they were trained with custom code modifications and can be used only with the custom elmoformanylangs library. on the other hand, our norelmo models are fully compatible both with the original elmo implementation by @xcite and with the more modern simple elmo python library provided by us. 8 the vocabularies are published together with the models. for different tasks, different models can be better, as we show below. the published packages contain both tensorflow checkpoints (for possible fine-tuning, if need be) and model files in the standard hierarchical data format (hdf5) for easier inference usage. in addition, we have setup elmoviz, a demo web service to explore norwegian elmo models. 9 6 norbert our norbert model is trained from scratch for norwegian, and can be used in exactly the same way as any other bert-like model. the norbert training setup heavily builds on prior work on fin-bert conducted at the university of turku @xcite . norbert features a custom wordpiece vocabulary which is case-sensitive and includes accented characters. it has much better coverage of norwegian words than the mbert model or nb-bert (which uses the same vocabulary as mbert). this is clearly seen on the example of the tokenization performed by both for the norwe-gian sentence 'denne gjengen håper at de sammen skal bidra til å gi kvinnefotballen i kristiansand et lenge etterlengtet løft' norbert tokenization splits the sentence into pieces which much better reflect the real norwegian words and morphemes (cf. 'k vinne fo t ball en' versus 'kvinne fotball en'). we believe this to be extremely important for more linguistically-oriented studies, where it is critical to deal with words, not with arbitrarily fragmented pieces (even if they are well-performing in practical tasks). the vocabulary for the model is of size 30,000. it is much less than the 120,000 of mbert, but it is compensated by these entities being almost exclusively norwegian. the vocabulary was generated from raw text, without, e.g., separating punctuation from word tokens. this means one can feed raw text into norbert. for the vocabulary generation, we used the sen-tencepiece algorithm @xcite and tokenizers library. 10 the resulting tokenizers model was converted to the standard bert wordpiece format. the final vocabulary contains several thousand unused wordpiece slots which can be filled in with task-specific lexical entries for further finetuning by future norbert users. ## evaluation this section presents benchmark results across a range of different tasks. we compare norelmo and norbert to both mbert and to the recently released nb-bert model described in section 4. where applicable, we show separate evaluation results for bokmål and nynorsk. below we first provide an overview of the different tasks and the corresponding classifiers that we train, before turning to discuss the results. ## future plans in the future, separate bokmål and nynorsk bert models are planned, and we further expect to train and evaluate models with a higher number of epochs over the training corpus. while we plan to develop additional monolingual norwegian models based on other contextualised lm architectures beyond bert and elmo, we would also be interested to explore the usefulness of multilingual models restricted to scandinavian languages. further streamlining of the benchmarking process, in terms of both data access and computation of metrics, is something we also want to address in future work. in addition, the ready availability of a highly optimised software stack on multiple hpc systems (published as part of norlm) may contribute to other researchers developing very large contextualised language models for additional languages or language variants, e.g. domain-or application-specific sub-corpora. we hope that more pre-trained nlp models for norwegian from both academy and industry will be openly released, making it possible to study the interplay between training corpora sizes, hyperparameters, pre-preprocessing decisions and performance in different tasks. at the same time, given the resource demands and sustainability issues related to training such models, we believe it will be important to coordinate efforts and we hope to collaborate closely with other players moving forward. ## summary this paper has described the first outcomes of norlm, an initiative coordinated by the language technology group at the university of oslo seeking to provide norwegian (and nordic) largescale contextualised language models, while simultaneously focusing on maintaining a re-usable software environment for model development on national and nordic hpc infrastructure. we have here described the training and testing of norelmo and norbert -the first large-scale monolingual lms for norwegian. we have benchmarked the models across a wide array of norwegian nlp tasks, also comparing to the multilingual mbert model and another large-scale lm for norwegian developed in parallel work, nb-bert, trained on large amounts of text from historical sources. the results show that while the monolingual models tend to yield better results, which particular model ranks first varies across tasks. this underscores the importance of building an ecosystem of diversified models, accompanied by systematic benchmarking.