Operationalizing a National Digital Library: The Case for a N orwegian Transformer Model (2021)

Abstract: In this work, we show the process of building a large-scale training set from digital and digitized collections at a national library. The resulting Bidirectional Encoder Representations from Transformers (BERT)-based language model for Norwegian outperforms multilingual BERT (mBERT) models in several token and sequence classification tasks for both Norwegian Bokmål and Norwegian Nynorsk. Our model also improves the mBERT performance for other languages present in the corpus such as English, Swedish, and Danish. For languages not included in the corpus, the weights degrade moderately while keeping strong multilingual properties. Therefore, we show that building high-quality models within a memory institution using somewhat noisy optical character recognition (OCR) content is feasible, and we hope to pave the way for other memory institutions to follow.

## introduction modern natural language processing (nlp) models pose a challenge due to the massive size of the training data they require to perform well. for resource-rich languages such as chinese, english, french, and spanish, collections of texts from open sources such as wikipedia (2021a), variations of common crawl data @xcite , and other open-source corpora such as the bookscorpus @xcite are generally used. when researchers at google released their bidirectional encoder representations from transformers (bert) model, they trained it on a huge corpus of 16gb of uncompressed text (3,300m words) @xcite . later research has shown that the corpus size might have even been too small, and when facebook released its robustly optimized bert (roberta), it showed a considerable gain in performance by increasing the corpus to 160gb @xcite . norwegian is spoken by just 5 million people worldwide. the reference publication ethnologue lists the 200 most commonly spoken native languages, and it places norwegian as number 171. the norwegian language has two different varieties, both equally recognized as written languages: bokmål and nynorsk. the number of wikipedia pages written in a certain language is often used to measure its prevalence, and in this regard, norwegian bokmål ranges as number 23 and nynorsk as number 55. however, there exist more than 100 times as many english wikipedia pages as there are norwegian wikipedia pages @xcite . when it comes to building large text corpora, norwegian is considered a minor language, with scarce textual resources. so far, it has been hard to train well-performing transformer-based models for such languages. as a governmental entity, the national library of norway (nln) established in 2006 a mass digitization program for its collections. the language bank, an organizational unit within the nln, provides text collections and curated corpora to the scholarly community @xcite . due to copyright restrictions, the publicly available norwegian corpus consists mainly of wikipedia pages and online newspapers, and it is around 5gb (818m words) in size (see ## building a colossal norwegian corpus as the main norwegian memory institution, the nln has the obligation to preserve and give access to all published information in norway. a large amount of the traditional collection is now available in digital format. as part of the current legal deposit, many born-digital documents are also available as digital documents in the collection. the texts in the nln collection span hundreds of years and exhibit varied uses of texts in society. all kinds of historical written materials can be found in the collections, although we found that the most relevant resources for building an appropriate corpus for nlp were books, magazines, journals, and newspapers (see texts in the nln have been subject to a large digitization operation in which digital copies were created for long-term preservation. the nln em- non ocr library sources external sources digital safestore clean text corpus training set unpack create deduplicate sample mets/alto files text & meta files clean clean clean figure 1: the general corpus-building process. the basic inclusion criterion for our corpus was that as long as it was possible for a human to infer the meaning from the text, it should be included. however, the amount of text involved in building the model meant that this needed to be determined automatically. the mets/alto files contain information from the ocr process regarding the confidence of every word (from 0 for no confidence to 1 for certainty), so we used this assessment to calculate the average confidence for paragraphs and pages. setting the minimum paragraph confidence to 0.8 and the minimum page confidence to 0.9 allowed us to filter out a significant part of the text with the lowest quality. we also noticed that in the period of digitization from the beginning of 2006 until the end of 2008, the quality of the ocr was low and the estimated confidence values were too optimistic. we ended up excluding all text scanned in this period. to further filter out erroneous textual information, we calculated the number of words in the documents and averaged the number of words per paragraph. establishing a threshold of at least 20 words per document and an average of 6 words per paragraph, we could filter out text sources that had little value for training, such as cartoons and picture books. we estimated the language composition using various methods, including metadata tags in the collection and counting the frequency of words of certain types (e.g., personal pronouns). our estimate is that 83% of the text is in norwegian bokmål and 12% is in nynorsk. close to 4% of the texts are written in english, and the 1% left is a mixture of sami, danish, swedish, and a few traces from other languages. the aforementioned process was carefully orchestrated, with data moving from preservation storage, through error correction and quality assessment, and ending up as text in the corpus. as shown in figure 1 , after filtering, ocr-scanned documents were added to the other digital sources. after this step, the data went through the cleaning process, in which we ensured the consistency of the text encoding and special characters used. in the deduplication stage, all duplicated paragraphs in the entire collection were removed. finally, we drew out two pre-training-sets: one with a sequence length of 128 tokens, and one with a sequence length of 512 tokens. ## pre-training a norwegian bert model in order to build our own pre-trained language model for norwegian, we decided to use the original bert architecture pre-trained with a maskedlanguage model (mlm) objective, as published by @xcite . we evaluated the effect of changes in hyperparameters in terms of mlm performance and of the fine-tuning of the pre-trained models on various downstream tasks. all pretraining work was run on a v3-8 tpu (128gb) provided by the tpu research cloud, while the evaluation was done on in-house machines with a single nvidia quadro rtx6000 (24gb). our goal was to build a solid model that would perform well on all types of norwegian language tasks, ranging from old to modern text, and including texts that might be mixed with foreign languages like english. we therefore chose to initiate the model from the pre-trained mbert weights (tensorflow @xcite . model was trained on 104 languages, including both norwegian varieties (bokmål and nynorsk). the model uses a 119,547-token vocabulary, and its pre-trained weights might also benefit from cross-lingual transfer. our assumption is that using the mbert weights for norwegian should result in a better-performing model in comparison to starting with random weights. it might also keep some of its multilingual abilities, making it more robust when dealing with new words and texts containing fragments of other languages (wu and dredze, 2020). ## evaluation while pre-trained language models also can be used for direct mlm-predition and feature extractions, the most common use is to fine-tune it on a specific task. the base procedure for fine-tuning was described by @xcite , and it consists of training for a small number of epochs (typically 4), with a warmup of around 10% of the training steps; subsequently, a linear decay to zero is used. @xcite based their work on the same procedure and selected the best learning rate among 5e-5, 3e-5, and 2e-5, according to the performance of the model on the validation set. the optimal learning rate and number of epochs mainly depend on the size of and variance in the training corpus, but they can also be affected by the properties of the pre-trained model. to get optimal performance out of a pre-trained model, the hyperparameters in the fine-tuning should be adapted. however, in this work, we are not primarily interested in optimization but in a comparison of the performance of our models against the mbert model. ## results to evaluate the performance of our model, we searched for the optimal set of fine-tuning hyperparameters for each downstream task by running a small grid search (see for the token classification tasks, we selected the best-performing hyperparameters based on the seqeval (2018) f1 micro score on the validation set for bokmål after fine-tuning an mbert model. for sequence classification, we used the f1 macro score. ## discussion the majority of the training corpora used today for training transformer models are built using mainly open web sources. a major motivation for this project was to investigate whether the digital collections at the nln could be used to create a suitable corpus to train state-of-the-art transformer language models. the texts available through the library are heterogeneous in nature, including cartoons, novels, news articles, poetry, and government documents published over time and in dif for the target languages norwegian bokmål and norwegian nynorsk, the model performs significantly better than does the mbert model on both token classifications (pos and ner) as well as on the two sequence classification tasks. in the bokmål ner task, the level of improvement was +7.4 f1 points. because none of the datasets have been benchmarked against human performance, it is hard to measure how close this is to the theoretical maximum. the results show that our corpus is a valid training source, and this is by no means surprising. all research points to the possibility of improving transformer models' performance by training them on larger text corpora. however, the novelty of our results lies in that we were able to increase the performance on our domain-specific tasks while maintaining a lot of the multilingual properties of the mbert model. this was unexpected because english only comprised around 4% of the training set. still, we were able to improve the english capabilities of the model up to the level of the monolingual english model. part of the reason for this might be that we applied some training techniques that were not available when the englishonly model was trained and released, most notably the use of larger batch sizes and the lamb optimizer. we were also able to significantly improve the scores for swedish and danish, though it is hard to pinpoint how much of this was caused by the close linguistic similarities between the languages and how much by the fact that they were represented in the corpus to some degree. it should not be surprising that the capabilities of the model in relation to languages that were not included in the training corpus (i.e., spanish and finnish) did deteriorate. however, the drop in performance was not radical, and the results above indicate that we might have been able to prevent this by adding just a small portion of these languages to the large corpus. overall, our results suggest that collections such as the digital collection at the nln, even if they contain occational ocr-errors, may contribute significantly toward the creation of wellperforming language models by providing large training corpora. as discussed earlier, there are ocr errors in the included materials. an exhaustive removal of all ocr artifacts would either have required us to do a major reduction of the size of the corpus, or to invest an unmanageable amount of manual work. we have not seen any indication that the ocr errors negatively impacted the performance. we might speculate that the model has learned to distinguish ocr errors from ordinary text, indicating that quantity is more important than quality when building such corpora. all in all, size matters. ## conclusion and future work in this work, we have investigated the feasibility of building a large norwegian-only corpus for the training of well-performing transformer-based language models. we relied on the collections of the nln, and our model outperformed the existing multilingual alternatives. in the process, while the corpus produced might lack the cleanness of other textual resources, we proved that using somewhat noisy but available sources is an effective way to grow the ecosystem of resources for languages with fewer resources and for which enough open text in a digital format simply does not exist. as part of an effort to democratize the use of technology and digital resources at the nln, we are releasing our trained bert-based model (national library of norway ai lab, 2021a) and will be releasing other models based on the same corpus in the future. moreover, we are also releasing the set of tools and code we used so that others seeking similar results can easily reuse them (national library of norway ai @xcite . although our work may indicate that ocr errors in corpora have little to no impact on the quality of the resulting transformer model, this has not been explicitly proven in the current study. more systematic studies are needed to investigate the real effect of ocr noise and artifacts. another important aspect is that, to benefit from the pre-trained mbert weights, we used a 119,547-token multilingual vocabulary, of which only a small fraction pertained to norwegian. a natural follow up would be to investigate the performance gains of using only a tailored norwegian vocabulary. the decision to use a bert-based architecture as our target was guided by its simplicity to train and benchmark. however, newer and better-performing models have been released since the original bert work a few years ago. the current corpus could be used for training such models as well studying the differences between architectural styles and training objectives. while it is already large in size, there is still potential to grow our 109gb corpus to the limits of the extant norwegian holdings at the nln, which presents itself as an opportunity to release even larger models.