G iella LT — a stable infrastructure for N ordic minority languages and beyond (2023)

Abstract: Long term language technology infrastructures are critical for continued maintenance of language technology based software that is used to support the use of languages in digital world. In Nordic area we have languages ranging from well-resourced national majority languages like Norwegian, Swedish and Finnish as well as minoritised, unresourced and indigenous languages like Sámi languages. We present an infrastructure that has been build in over 20 years time that supports building language technology and tools for most of the Nordic languages as well as many of the languages all over the world, with focus on Sámi and other indigenous, minoritised and unresourced languages. We show that one common infrastructure can be used to build tools from keyboards and spell-checkers to machine translators, grammar checkers and text-to-speech as well as automatic speech recognition.

## introduction language technology infrastructures are needed for longterm maintenance of linguistic data and nlp applications derived from it. specifically in a nordic context, we have a selection of lan guages with very different requirements, and all differ from those that are commonly assumed in other nlp contexts, e.g. english and handful of most resourced languages in the world. the lan guages in the nordic area range from decently resourced indoeuropean languages (norwegian bokmål, swedish, danish and icelandic) to mod erately resourced uralic languages (finnish, es tonian) to all low and unresourced, minoritised languages (sámi languages, all other uralic lan guages, faroese, greenlandic). we have an infras tructure that supports all of these languages, with a focus on the smaller and less resourced languages and specifically on the sámi languages. the in frastructure we provide has been in use for over a decade and in this article we describe strategies and workflows that we have found successful. it cur rently supports over 100 languages, many outside of the nordic region. the technical infrastructure builds on the con cept that we aim to separate the technological work: programming and engineering, from the linguistic work: lexicography, grammar building, corpus annotation etc. in this way, we enable lin guists and native informants to work on the lan guage data and the engineers build and maintain the technological solutions in a meaningful way where both the technological solutions and the lin guistic data are kept up to date and functional. this workflow is important since both linguis tic and technological sides present ongoing chal lenges to be kept up to date. regarding the lin guistic content, the language norms change and grow, new words and expressions enter the lexicon regularly and other words and expressions become outdated. in technology, operating systems and environments, programming languages and apis change all the time, making the nlp tools built a few years ago not usable a few years later. the research question we solve with our infrastructure is, how both parts can be kept up to date while not burdening the people working with the parts with details irrelevant for their work. in other words, the infrastructure contains lin guistic data, and technological implementations to build end user nlpbased tools and software from it. the tools that we build nowadays include writ ing tools, such as spelling and grammar check ers and correctors, speech synthesis and recogni tion, machine translation, intelligent dictionaries and various linguistic analysis tools. the techno logical infrastructure is composed of tools like ver sion control systems, build systems and automa tion of building and distribution of the nlp tools. the underlying technologies here have changed a lot in the past 20 years, and will undoubtedly keep evolving. in this article we take a look on some concepts that have both stayed stable or evolved to be part of the core tools for us. in the nlp scene, the world has changed a lot in past years as well, with the traditional knowledgebased methodol ogy being gradually replaced by datadriven ap proaches; in the giellalt infrastructure we are still following the expertdriven knowledgebased ap proach as it continues to be the most appropriate for unresourced languages, but we do not cover this dichotomy in detail; for more details of this we refer to @xcite that discusses the issue extensively. in the past 20 years we have built language re sources for several sámi languages starting from virtually nothing; even though we had a num ber of nondigital resources available, these were far from exhaustive. this means that our work also included normative discussions, requests and suggestions to the language normative organs, er ror classifications, and grammatical descriptions of phenomena not included in grammar books. in several cases, these phenomena needed traditional linguistic research. based on this experience we suggest workflows and usage patterns along the technical solutions of the infrastructure that are effective for long term maintenance of linguistic software in support of continued digital existence of human languages. the contributions of this article are: we present a stable nordic language technology infrastructure that has supported nordic language technology de velopment for 20 years, we describe the best cur rent practices we have learned in the years and based on the current state of things we sketch the potential future developments. ## background the infrastructure presented in this article has been developed and maintained for at least 20 years now. the infrastrucutre has been discussed pre viously in nodalida some 10 years ago mosha gen et al. @xcite . in this work we aim to show updates and prove that the system has well stood the test of time in supporting nordic languages. on one hand everything has changed between the years; computers and mobile platforms, operating systems, programming environments, on the other hand, many solutions have stayed usable: rule based finite state morphologies, dictionaries and linguistic data. the foundation for the work presented in this article is the multilingual infrastructure giellalt, which includes over 100 languages, including most nordic ones: the sámi languages, faroese, finnish, norwegian, swedish, other uralic lan guages and many more. everything produced in the giellalt infrastructure is under free and open licences and freely available. the cor pora are available with free licensing where pos sible. the infrastructure is split codewise in three github organisations: giellalt containing the language data for each language, divvun contain ing language independent code for the infrastruc ture and various applications, and giellatekno for corpus infrastructure. end user tools served by the divvun group are at divvun.no & divvun.org, and tools served by the giellatekno group at giellate kno.uit.no, both at uit norway's arctic university. we build systems that include lexical data as well as rules governing morphophonology, syn tax and semantics as well as a number of applica tion specific information, e.g. grammatical rules for grammar checking, phonetic rules for textto speech (tts) and so forth. the languageindependent work is currently done within the infrastructure, the language independent features and updates that are relevant to all languages are semiautomatically merged as they are developed. to ensure that language inde pendent and common features and updates do not destroy existing language data or use case, we en force a rigorous continuous integration based test ing regime. the current system for testing is a combination of our longterm investment in testing within the infrastructure locally for developerscombined with modern automatic testing currently supplied by github actions. the automated testing and integration is one of the key features for upkeep and maintenance of the linguistic data: the linguists work with the dictio naries and rules on a daily basis and receive im mediate feedback from the system of the effects of the new word entries or rules. the testing system verifies that if the new words and rules did not af fect negatively the user experience of e.g. spelling checker, it can be immediately deployed to the end users of the mobile keyboards and spellcheckers on office platforms. another part of the giellalt philosophy is that of reusable and multipurposeful resources, cf. @xcite . this is true for all of our work, from corpus collection to crosslingual co operation. ## linguistic data there are two types of linguistic data we gather and develop in the infrastructure, one is the dictionar ies, grammars and descriptions for each language and the other is corpus data. even if our system is not corpusdriven in the way most other contempo rary systems are, once we develop the knowledge based systems we are working for, the realworld data from language users becomes a very impor tant resource for testing and evaluating the sys tems we have built. the corpus data we collect is also enriched by language experts by annotat ing spelling and grammar errors with corrections included, or by doing other linguistic annotations and corrections to automated annotations. for this reason and also because we work with many lan guages that have very little data available the cor pora we collect are carefully selected and curated. the linguistic data can be roughly evaluated without annotated large manually annotated gold corpora by calculating the number of words in the dictionaries and a naïve coverage. words counted are lemma entries, thus words covered by pro ductive morphology will not be included in the figure. 1 the naíve coverage will give an intu ition for the extents of the derivational morphology has with regards to real world wordform usage. here naïve coverage is calculated as a proportion of tokens that get any analyses of the whole cor pus, in this case we use the tokenisation provided by the corpus analysis tools, which is based on lefttoright longest match tokenisation that falls back on spaceseparated tokens with special cases for punctuation, i.e. mostly natural tokenisation for the western languages with latin and cyrillic scripts. 2 the figures are given in it is noteworthy that the naïve coverages we count are based on the corpora we have collected and this corpora has been seen by people working on the dictionaries, in other words it is technically not a clean test setup. for many of the languages we work with this is necessitated by the facts that the corpus we have is all texts that are available for the language at all. not making full use of it would hinder the development of the language model in a way that would be more valuable for the language communities than to hide parts of the corpus from the lexicographers for testing purposes. for this reason the figures should be considered as a rough guideline, as naïve coverage would be anyways. for our intents and purposes, we can see from the naïve coverage if the dictionaries need attention e.g., for spellcheckers to be usable enough as to not show too many red underlines in regular ev eryday texts. we collect texts for the nordic languages as well as several other languages that we use and develop. the largest corpora we have harvested are for the sámi languages: north, lule, south, inari and skolt sámi. the sámi corpus is owned by the norwegian sámi parliament, and all corpora are administered and made accessible to the public by the divvun and giellatekno groups. the cor pora for some of the uralic languages in russia are large, and for meadow mari even larger than for north sámi. some of the corpora for larger, nonminority languages (e.g. finnish, norwegian) are moderately sized, since they are already cov ered by other projects such as opus @xcite , and we only need to create specific corpora for our applications, such as grammar error cor pora by l2 speakers in order to develop a grammar checker. the corpora are split in two based on restrictions set by the copyright owners. researchers and any one else can freely download the free part. the whole corpus, also the restricted part, is accessi ble via a public search interface foot_2 . we have writ ten a tool named corpustools to administer, con vert and analyse the corpus texts. original texts and their metadata are saved in github reposito ries, then converted to a common xml format, to ease further use of the texts. the sizes of corpora are summarised in recently, we have also began collecting speech corpora for speech technology related projects. for example, for an ongoing lule sámi tts project we reused a part of a lule sámi gold corpus from 2013, and collected additional texts we knew to be well written and already proofread, before proofreading these texts once more to avoid con fusion when reading the text aloud during the tts recordings. the lule sámi tts text corpus con sists of various text styles (news, educational, par liament etc.) with altogether over 74,000 words. currently, we have recorded two lule sámi voice talents using this text corpus, and after processing the recordings, a speech corpus with altogether 20 hours will be ready to use for speech technology purposes. as spoken language technology is based on data and machine learning, the procedures and pipelines described above could be applied to any (minority) language with a lowresource setting, in the task of developing speech technology applica tions. most of the applications discussed here can be piloted with or further developed with relatively small data sets (even @xmath0, compared to the amounts of data used for respec tive tools for majority languages (see, e .g., ito and johnson (2017) 6 ). this is largely possible thanks to the available open source materials and tech nologies, especially those relying on, e.g., transfer learning, i. e. finetuning of models @xcite . ## conclusion in this article we have presented recent develop ments and status of the giellalt nordic multilin gual infrastructure built during the last 20 years. in the last years, we have added more support to speech technologies, and keyboards for various platforms such as mobile devices and modern op erating systems. the giellalt infrastructure contains building blocks and support for most of the language tech nology needs of indigenous and minority lan guages, from the very basic input technologies like keyboards to highlevel advanced tools like world class grammar checking and machine translation. it does this by using rulebased technologies that makes it possible for any language community to get the language technology tools they want and need. all that is needed is a linguist. we discussed the ways for longterm mainte nance of linguistic data and software tools for nlp of nordic and minority languages. we showed some best current practices and workflows on how to maintain the lexicons and keep end user tools unbroken and still uptodate. in conclusion, building corpora is based on big efforts, requires expertise and is timecostly. we have illuminated the work behind three important steps within building corpora firstly, collecting and digitalising, secondly upgrading, i.e. adding annotation for special purposes, and proofreading, and thirdly converting from one medium/language to another as in recording speech, translating, or other. with our multilingual infrastructure and our lan guage resources we show that while there is a need for corpus data for certain tasks, high quality tools needed by a language community can be built timeefficiently without big data in a rulebased manner.