It’s Basically the Same Language Anyway: the Case for a N ordic Language Model (2021)

Abstract: When is it beneficial for a research community to organize a broader collaborative effort on a topic, and when should we instead promote individual efforts? In this opinion piece, we argue that we are at a stage in the development of large-scale language models where a collaborative effort is desirable, despite the fact that the preconditions for making individual contributions have never been better. We consider a number of arguments for collaboratively developing a large-scale Nordic language model, include environmental considerations, cost, data availability, language typology, cultural similarity, and transparency. Our primary goal is to raise awareness and foster a discussion about our potential impact and responsibility as NLP community.

## introduction deep transformer language models have become the weapon of choice in modern nlp (and in ai more generally). there is a rich, and evergrowing, flora of models available, including bert @xcite , xlnet @xcite , electra @xcite , t5 @xcite , and gpt (2 and 3) @xcite . these models present slight variations of architectural choices, training objectives, parameter settings, and size and composition of the training data. despite some internal variation in performance, transformer language models in general hold state of the art results in basically all nlp benchmarks and evaluation frameworks at the moment @xcite @xcite . the downside to this recent development is the computational cost of training deep transformer models. starting with bert-base with its (now viewed as modest, but at the time of publication seen as substantial) 110 million parameters, and bert-large with its 340 million parameters, there has been a virtual explosion in the number of parameters, culminating in the recent gpt-3 with its 175 billion parameters @xcite , gshard with 600 billion parameters @xcite , and the most recent switch transformer with a whopping 1,3 trillion parameters @xcite . this development translates into an acute need for access to powerful processing platforms, huge amounts of training data, and an ability to harbor extremely long training times. taken together, this is a perfect recipe for extreme energy consumption and cost, which risks leading to reduced inclusivity in research on largescale language models. there is a budding debate on the environmental and cultural impact of training and using largescale language models. two recent examples are @xcite and @xcite ; the former analyze the energy consumption and cost of training deep transformer language models, and the latter voice concerns regarding both the environmental and cultural impact of training and using large-scale language models. we hope to contribute to this discussion by providing a nordic perspective on the need for large-scale language models. we will assume the position that a collaborative effort towards training a largescale nordic language model is something worth striving for. we consider a number of arguments for this position, include environmental considerations, cost, data availability, language typology, cultural similarity, and transparency. it is of course impossible to get an accurate count on this, but one way to approximate an answer might be to consider how many models have been trained in the world so far. we obviously cannot know this either, but we might be able to get an idea by looking at the number of models published in open source libraries. luckily, much of the recent development is centered around one such library: the transformers library of the company hugging face. 2 the transformers library contains (at the time of submission) more than 6,800 models covering a total of 250 languages. a survey carried out by benaich and hogarth in the fall of 2020 claims that more than 1,000 companies are using the transformers library in production, and that it has been installed more than 5 million times. 3 6,800 models times a low estimate of 652 kg of co 2 sums to 4,434 tonnes of co 2 emissions. this is of course an extremely unreliable estimate. many of the models uploaded to the transformers library are merely finetuned and not trained from scratch (we have not been able to quantify this proportion). on the other hand, many of the uploaded models are significantly larger than bert-base, and one can assume that only a fraction of models that are built are actually uploaded to the transformers model repository. by comparison, the average swedish citizen emits around 8 tonnes of co 2 per year, foot_3 while rise (the research institutes of sweden) with approximately 2,800 employees emitted a total of 1,287 tonnes co 2 during 2019 according to the 2019 annual report. counting only the nordic models uploaded to hugging face, there are (at the time of submis- ## argument 2: cost it is anything but cheap to train large-scale language models. the cost for performing a single training pass for the largest t5 model is estimated to be @xmath04, 6 million. 5 to put these numbers into perspective, the average project funding in the eu horizon 2020 program is estimated to be around @xmath1150 thousand. 7 this means that, unless you happen to be in the possession of a sizeable computing infrastructure, training models on this scale will be out of the question for most researchers. however, even with access to suitable gpus, it is not obvious that it will be possible to train a model on the required scale. @xcite estimates that performing a single training run with the full gpt-3 using an nvidia tesla v100 gpu at its theoretical max speed would require 355 years. assuming access to an nvidia dgx-1, which features 8 v100 gpus, we would still need 44 years to build a replica of gpt-3. the cost of buying a dgx-1 machine is around $129 thousand -i.e. roughly the size of an average national research project. the sizeable cost (monetary as well as temporal) required to build a large-scale language model effectively excludes a large proportion of the nlp community from training models. this may not be entirely negative, considering the environmental concerns raised in the previous section, but it would be desirable if the production of large-scale language models was more inclusive and collaborative, with transparency and the possibility to influence the procedure even by smaller research groups. a communal effort would not only enable more researchers to have an influence on the model design, but it may also lead to broader usage of the resulting model, thereby reducing the need to constantly build new small (and probably not very useful) models. ## argument 3: data size and transfer it is a known fact that bigger training data leads to improved performance when using statistical learning methods in nlp @xcite . this has been eminently well demonstrated in the context of language models by the recent improvements using models that have been trained on very large data samples @xcite . it is a fascinating question whether there at all exists sufficiently large text data to build native models for all nordic languages. considering the biggest nordic language swedish as an example, sweden has legal deposit laws installed in 1661 for everything printed. during the the twentieth century it was gradually extended to include sound, moving images and computer games and electronic material. the law for legal deposit of electronic material was added in 2012. as a result, the national library of sweden (kb), has vast and ever growing collections, closing in on 26 petabyte of data. though only a fraction of the collections are digitized, the digital collections are nonetheless substantial. kb, through its data lab (kblab), works continuously to assembly corpora of swedish texts and to make them available for modeling. the latest corpus of cleaned, edited, raw swedish text is just over 104 gb of size (corresponding to approximately 1,4 billion sentences and 18,2 billion words). the sources for this corpus are: swedish wikipedia 2 gb; governmental texts 5 gb; electronic publications 0,4 gb; social media 5gb; monographs 2gb, and; newspapers 90 gb. the corpus currently under construction increases primarily the share of born digital text from legal electronic deposits and is expected to be around 1 tb of cleaned, edited, raw swedish text (thus approximately 14 billion sentences and 182 billion words). the upper limit (in terms of size) for subsequent corpora is expected to be between 2-5 tb, depending on the possibilities to transcribe spoken swedish present in the kb collections. the situations in the other nordic countries are similar, relative to the size of the population in the respective countries. there are consequently extensive danish and norwegian collections available, whereas the text/data resources in iceland and faroe islands are expected to be substantially smaller. combining all nordic text resources would likely lead to a fairly substantial data source, likely on the order of terabytes. the data conditions for the larger nordic languages look promising even when considered individually, but it is not obvious that there even exists enough data to train native large-scale models in the smaller nordic languages. fortunately, it has been demonstrated that multilingual models improve the performance for languages with less available training data, due to transfer effects @xcite . in particular, the transfer effects seems to be specifically beneficial for typologically similar languages @xcite . it is thus likely that in particular icelandic and faroese would benefit from a joint nordic language model. ## argument 4: typology the nordic languages belong to one of three germanic language groups, also referred to as north germanic languages (in addition to west and now extinct east germanic). the north germanic language group is further divided into two branches: east scandinavian languages, which includes swedish and danish, and west scandina-vian languages, which contains norwegian, icelandic and faroese. this genealogical categorization is sometimes contrasted with a distinction based on mutual intelligibility, which separates continental scandinavian (swedish, norwegian and danish) from insular scandinavian (icelandic and faroese). the nordic languages are so similar from a typological perspective that the language boundaries have been, if not in dispute, at least subject to some discussion @xcite . the difference between dialects within the nordic languages is in some cases probably larger than the difference between the languages. a telling example is the difference between norwegian bokmål, which is very similar to danish and as such is categorized as an east scandinavian language, and nynorsk, which is categorized as a west scandinavian language. another example is the difference between jamtlandish (or jamska, a dialect spoken in the swedish region jämtland, which is categorized as a west scandinavian language) and standard swedish (which is east scandinavian). from a typological perspective, it thus makes sense to entertain the idea of a joint north germanic language model, in particular when considering the potential for transfer effects to the smaller nordic languages. of course, one can always ask whether we should not aim for a combined germanic model instead? there will probably be something like an order of magnitude more data available if we consider all germanic languages rather than just the nordic ones. however, one can expect diminishing returns by adding more data at some point, and it is an interesting (and, as far as we are aware, open) question what is the trade-off between language similarity and data size? anglocentrism is not necessarily a disqualifying factor for the nordic countries, some of which (such as sweden) is sometimes considered to be among the most americanized countries in the world @xcite . we generally listen to the same type of music, watch the same type of movies, and watch the same type of tvshows. we don't, however, have similar political systems (as demonstrated by recent events). by contrast, there is arguably no (significant) difference in culture, politics, or economics between the nordic countries. in fact, there are probably more cultural differences within the countries than between. a relevant question is how to also include minority languages from other language families, such as sámi. a natural suggestion for this specific case is to consider a uralic language model, which would include languages such as finnish, hungarian, estonian, as well as the smaller languages erzya, moksha, mari, udmurt, sámi, and komi. ## argument 6: transparency the largest concurrent language models are not publicly available. few have probably missed the controversy surrounding the initial decision of open ai to not release gpt-2 due to concerns of adversarial usages. 9 as we know, gpt-2 was eventually released in full, and there are now gpt-2 models available in many other languages. the original gpt-3 model is however not yet openly available (open ai is beginning to look like a misnomer), but there are several open-source efforts to provide competing, or at least alternative, models. 10,11 this lack of transparency obviously limits the ability for other researchers not only to investigate this type of model, but also to contribute to its future development. a collaborative nordic effort would ensure inclusivity in the development, as well as accessibility to the final model. based on the considerations raised in this paper, we argue that we -the nordic nlp community should work together to build a truly largescale nordic language model, for the nordic languages, by nordic researchers. we believe that such a resource will be extremely beneficial for nordic nlp, and that it will have the potential to reduce the environmental impact of continuously training new models.