Working Towards Digital Documentation of U ralic Languages With Open-Source Tools and M odern NLP Methods (2023)

Abstract: We present our work towards building an infrastructure for documenting endangered languages with the focus on Uralic languages in particular. Our infrastructure consists of tools to write dictionaries so that entries are structured in XML format. These dictionaries are the foundation for rule-based NLP tools such as FSTs. We also work actively towards enhancing these dictionaries and tools by using the latest state-of-the-art neural models by generating training data through rules and lexica

## introduction most of the languages spoken in the world are in danger of extinction. their documentation and revitalization are of a highest cultural value, for which they have received plenty of academic attention in various disciplines such as anthropology, typology, lexicography and computational linguistics. needless to say, the resources produced in each individual research project are not always published openly let alone made available to the community of native speakers. the goal of our paper is to describe our open infrastructure for documenting minority languages. we present our experiences with the following uralic languages: skolt sami (sms), erzya (myv), moksha (mdf), komi-zyrian (kpv) and komi-permyak (koi). as they belong to the uralic branch, they are languages that exhibit a complex morphology, which makes their computational processing a challenge for modern machine learning methods that would require a lot of data to cover this complexity. the quantity and quality of data is usually an issue when we deal with endangered languages @xcite . carrying out linguistic documentation in a structured machine readable format, however, makes it possible to create the resources needed for building nlp tools simultaneously with linguistic documentation. we are about to start working with the apurinã (apu) language, which allows us to reflect upon our uralic context from a broader perspective, and increases the relevance of our work in a latin american context. thus, we describe how our infrastructure can work in non-uralic contexts. linguistic documentation is a field of academic study that has developed considerably in recent decades. its purpose is to provide a complete record of the linguistic practices characteristic to a given speech community @xcite . the goal of language documentation is to describe the language of a community of speakers as fully as possible both for future generations and for language revitalization. the result of this work is typically manifested as a linguistic corpus or other type of material collection, which later on can be studied and analyzed in various ways. the question whether the collected materials actually describe the language use of a speech community is debatable, and this goal can never be fully achieved because a corpus can never describe a language in full. nonetheless, linguistically collected materials may be the only resources available for a small language. whether and how language documentation materials should be made accessible and distributed, has been a matter of debate. we believe it is important to understand that this is also a matter of granularity, and the question is not necessarily whether the materials are accessible, but rather which parties should be allowed what type of access. there are good reasons for keeping culturally sensitive materials available only to specific groups. at the same time, there are always materials in any language that are more neutral and such that the authors themselves may want to make accessible. especially for written publications, it may always be possible to negotiate a publication with open licenses, which would also allow the reuse of the same materials in different open research purposes. open materials are particularly important when we develop tools for nlp, because this work can greatly benefit from resources that are openly accessible with a permissive license. in the following sections we will discuss examples of such work, including our contribution to universal dependency treebanks. it must be emphasized that the open technology developed on an open infrastructure can also be used to process materials that are available only to a particular researcher or individual members of a community. therefore, open infrastructure benefits both open and closed environments, whereas a closed infrastructure only benefits a big commercial player. ## related work there are several individual projects in different parts of the world that work with online dictionaries for endangered languages. many projects, however, focus on one language only and work without knowing about other ongoing projects for other endangered languages. this has led to a situation where researchers solve the same type of problems individually for their language of interest reinventing the wheel over and over again. there are plenty of online dictionaries and language learning tools that have been developed from scratch for one particular language. work with endangered languages in north america has shown the importance of language learning tools for second language learners. lack of familiarity with lexicographical tradition can easily be a deciding factor in a beginner's learning experience. a learner of a new language cannot be expected to know exactly where an entry is located in a dictionary, nor can the learner be expected automatically to know the normative spelling. when the user of a language lacks a proper keyboard layout or knowledge of the correct orthography, the strategies of orthographic relaxation can be implemented in mobile and online dictionaries. morphological processing and spelling relaxation are used to cater to beginners in tsimshian and salishan languages in the use of dictionaries and nlp tools @xcite . on an entirely separate front, work has also been done to provide the yupik community of st. lawrence island unimpeded access to language materials online. this has been possible using a morphologically aware dictionary. in the system, a strategy of multiple input methods has been intro-duced that caters to different writing systems @xcite . the work here is tailor-made, and it maintains a strong link between the language and its community. the endangered language is seen as a low-resourced language in this context. the problem is that low-resourced language is a term that is used for almost any language with less internet presence than english. languages like hindi @xcite , arabic @xcite or persian @xcite are often considered low-resourced languages in the world of nlp, even though they have millions of speakers. in the work of nasution et al. (2018) , the ethnic indonesian languages are relatively small compared to the superstrate language that surrounds them. the approach consists of working simultaneously with a group of closely related languages in a multilingual, language-independent infrastructure. the authors analyze the use of bilingual dictionary entries and explain the difficulty of selecting the appropriate bilingual dictionaries to begin documentation. one of the largest infrastructures for minority language documentation from the point of view of computational linguistics is that of giella @xcite . their infrastructure is based on two main components: fst transducers (finite state transducers) and xml dictionaries. transducers are a way of documenting the morphology of a language computationally. that is to say, they are collections of rules about how the morphological system of a language works. these rules can be used directly for automatic text analysis and lemma conjugation in its morphological variants. transducers and xml dictionaries are used for spelling correction in word foot_0 , text prediction on android and ios keyboards foot_1 , interactive systems to learn languages @xcite and online dictionaries @xcite . our infrastructure is based on giella, which allows us to synchronize data between the two infrastructures. this means that advances in linguistic documentation in our infrastructure can be used directly in the tools produced in giella. ## our infrastructure using giella requires a relatively high proficiency in programming to be able to write dictionaries and morphological rules for fsts, and at the same time, it requires a good amount of knowledge in the language that is being documented. the infrastructure can be too complicated even for those who have studied computer science, and therefore it is not accessible to a community outside of those who collaborate directly with giella. for this reason, our infrastructure has several interfaces for different types of users; for users who do not have sufficient knowledge to write xml or program transducers and for developers who want to use the tools without knowing how to compile them right from the beginning with the make command. ## discussion and conclusions we hope that our work can be useful for others as well. we have put a lot of attention in opensourcing our tools and resources so that nobody needs to start building language documentation tools entirely from scratch. we have also paved a road towards using state-of-the-art neural models in the context of truly endangered languages with extremely limited resources. this is challenging and requires ingenuity. we are not interested in committing to the dichotomy of researches who defend rule-based tools as the only viable option for endangered languages nor to the researchers who frown upon rules and rely solely on the transformer architecture. the best solutions, we believe, are found by combining both worlds. our tools are compatible with the giella infrastructure. this has made it possible to use our dic-tionaries and translators directly on their online platform to learn languages @xcite , on android and iphone keyboards and spell checking for word and openoffice developed by divvun 7 at giella. flexible and interoperable design makes it also possible to integrate different lexical resources into our infrastructure once those are digitized or otherwise become available. digital documentation clearly has its benefits, since we can carry out machine learning with structured dictionaries and fst transducers. for this reason, a project conducted at the university of oulu 8 the goal of which was to author the new dictionary skolt finnish-sami has chosen to use ve'rdd to create the digital and printed dictionary. we have worked together with project employees to increase the functionality of our system. ve'rdd has made the simultaneous work of editors possible who, without ve'rdd, would have used excel and word. this would have meant a lost chance of producing a structured dictionary for the interest of nlp and a printed dictionary at the same time. we have started to explore non-uralic languages by building a ud treebak for apurinã @xcite . furthermore, we have built an initial fst for lushootseed (lut) @xcite and extended it with an lstm model. these are our initial steps towards non-uralic languages.