CLD ² Language Documentation Meets Natural Language Processing for Revitalising Endangered Languages (2022)

Abstract: Language revitalisation should not be understood as a direct outcome of language documentation, which is mainly focused on the creation of language repositories. Natural language processing (NLP) offers the potential to complement and exploit these repositories through the development of language technologies that may contribute to improving the vitality status of endangered languages. In this paper, we discuss the current state of the interaction between language documentation and computational linguistics, present a diagnosis of how the outputs of recent documentation projects for endangered languages are underutilised for the NLP community, and discuss how the situation could change from both the documentary linguistics and NLP perspectives. All this is introduced as a bridging paradigm dubbed as Computational Language Documentation and Development (CLD²). CLD² calls for (1) the inclusion of NLP-friendly annotated data as a deliverable of future language documentation projects; and (2) the exploitation of language documentation databases by the NLP community to promote the computerization of endangered languages, as one way to contribute to their revitalization.

## introduction there are around 6,500 mutually unintelligible languages in the world @xcite . however, several thousand minority languages are in danger of being lost forever without leaving systematic records. in response to this, in the last decades documentary linguistics has become a major and vibrant field in linguistics, which attempts to produce permanent records of the linguistic and cultural practices of the most threatened speech communities @xcite ; austin (2010); @xcite , among many others). the outcomes of documenting a language in the frame of contemporary documentary linguistics often comprise large amounts of audio and figure 1 : number of publications in the acl anthology where languages are explicitly named in the title or abstract, and they are classified by their vitality from the agglomerated endangerment status @xcite . vertical axis is in log-scale. international language archives comprises documentation databases for several hundred languages. for instance, the endangered language archive (elar) includes collections for 695 languages 1 , each of which may comprise several hours of transcribed and parsed speech, which represent several thousands of fully annotated sentences. these data has been produced in the frame of collaborative documentation projects with high ethical standards in terms of their methods, their outcomes and their dissemination. thus, in principle, the data available through international language archives have been published with the permission of the linguistic communities involved, and therefore it is expected that they will be incorporated into new research, education and revitalisation projects, ideally with the participation of members of the communities culturally and linguistically linked to the data @xcite . language databases, however, are often underexploited for further developments. although field linguists very often incorporate revitalisation components in their documentation projects, language documentation and language revitalisation are not equivalent in terms of their frames, methods and outcomes. language revitalisation will surely take advantage of the data produced in language documentation projects, by actively using such records in community-based revitalisation programs, which may take various shapes according to the needs of the community and/or the scope of the project. although it is true that creating a language repository alone cannot revert language endangerment or decay, there are several ways in which documentation data can be integrated into revitalisation projects. here, we focus on one, associated with the perspective of language technologies. language technologies offer a promising perspective for language revitalisation, not only because technological gadgets such smart phones are becoming more popular even in rural areas, but also because they are inexpensive. the concern about language endangerment is a fundamental issue in contemporary approaches to computational linguistics, and in the last years, the "computerisation" of minority languages has become a growing field in nlp research @xcite . nlp developments' potential contribution to revitalising endangered languages is high, but there is still moderate interaction between documentary linguistics and nlp research for language revitalisation. in this paper, we reflect on the necessity of increasing the interactions between documentary linguistics and nlp. this is not a novel point in collections/, consulted on february, 28th, 2022 the literature (see particularly @xcite ), but to our knowledge this is the first attempt to put some ideas on this topic together in a position paper. we hope that the proposals we dubbed here as computational language documentation and development (cld²) will stimulate debate and more vibrant interactions between documentary linguists and nlp developers. ## language documentation and language revitalisation language documentation (or documentary linguistics) emerged at the end of the last century as a research program whose primary motivation lies in the concern about the accelerating loss of language diversity in the world. as a response, language documentation aims to create permanent records of the linguistic and cultural practices of the most threatened speech communities @xcite @xcite . these records are framed as databases, ideally including several hours of audio and video recordings of monologue and dialogue texts belonging to various genres and topics (e.g. traditional tales and myths, verbal art, jokes, historical facts, life stories, cultural knowledge, among others). a good portion of these recordings is transcribed, translated and parsed. each transcribed sentence is expected to be timealigned and to include an orthographic or ipa representation, a morphemic parse, glossing, information about parts of speech and a free translation. producing such linguistic databases is a longterm and time-consuming task that may take several years and requires considerable funding. the expectation is that these linguistic databases, conceptualised as multipurpose repositories deposited and curated in international archives, will be preserved for posterity and thus will support communitybased revitalisation projects in the future. although it is true that language documentation projects very often incorporate revitalisation components, they are inevitably marginal since the documentation itself is the main focus of documentary linguistics. therefore, the contribution of language documentation to language revitalisation is potentially significant but mainly indirect: the linguistic repositories produced in the frame of language documentation projects can indeed contribute to future revitalisation projects, but crafting and archiving a repository is not expected to have an inherent positive impact on the vitality status of an endangered language. ## language documentation and computational linguistics most interactions between computational linguistics and documentary linguistics relate to the release of software tools for language documentation, processing and archiving @xcite . computational linguists and computer scientists have developed advanced software tools to assist field linguists in the various processes of contemporary language documentation, making them less time-consuming, more efficient and more systematic. these tools have been crucial for the exponential growth of language documentation on a global scale. contemporary language documentation implies a large amount of technical sophistication for managing, annotating, processing and archiving lasting and large repositories @xcite @xcite . this could not be achieved without the contribution of computer scientists (particularly software developers). in the last decades, we have witnessed the release of specialised software tools nowadays customary for language documentation, speech analysis and linguistic fieldwork. field linguist's toolbox (before "shoebox") (summer institute of linguistics, 2021a) and more recently fieldworks (flex) (summer institute of linguistics, 2021b) are data management and analysis tools for field linguists developed by the summer institute of linguistics, which are used in language documentation and taught in linguistics schools worldwide. toolbox and flex allow to create dictionaries, which can be used for morphosyntactic parsing and annotation of transcribed texts. transcription is often conducted in a different and nowadays very popular software called elan (max planck institute for psycholinguistics, 2021), developed by the max planck institute for psycholinguistics. elan allows to visualise and play audio and video files in order to create time-aligned transcriptions and translations. elan can also be used for morphological parsing, but most linguists prefer to conduct such tasks in toolbox or flex since elan transcriptions can be easily exported into these programs. in toolbox or flex, each sentence in an elan file (containing a transcription and a free translation) can receive morphemic parsing, morpheme-by-morpheme glossing and parts of speech tags, among any other relevant information in the frame of a specific project. the resulting toolbox/flex files are text files that can be opened back in elan, in praat (a phonetics analyser) @xcite , or to be processed in python or any other programming language as plain texts. this is shown in figure 2 . in sum, there have been several attempts from the computational side trying to create or incorporate intelligent components in language documentation tools and procedures @xcite @xcite @xcite . we find a one-direction application (computation into language documentation), but there are still few developments in the other direction (language documentation into computation). one of our takes in this paper is that language documentation can significantly contribute to computational linguistics by providing data and insights to develop nlp tools for endangered languages. ## nlp has not really met endangered language documentation as mentioned before, nlp has mainly focused on aiding the language documentation pipeline. however, has nlp taken advantage of the outputs of the documentation projects, especially for endangered languages? ## cld²: computational language documentation and development computational linguistics and language documentation share not only the assumption that technology plays an important role in the design and development of language-related projects, but also a crucial concern about language endangerment and loss. this concern is obvious from the perspective of language documentation, in the sense that it assumes itself as a response to language endangerment @xcite . a similar shift towards minority languages can be found in contemporary approaches to computational linguistics. @xcite regrets that less than 1% of the world's languages have been correctly "computerised". that is, for berment (2002), the fact that 99% of the world's languages lack computational tools (nlp tools as spell-checking or machine translation) requires immediate attention. since the seminal article by @xcite , language endangerment and language dormancy is a major concern for both current language documentation and computational linguistics. this paper takes the shared interest in linguistic diversity found in language documentation and computational linguistics further by proposing a paradigm that assumes an intense and multifaceted interaction between the two: computational language documentation and development (cld²). cld² assumes, following @xcite , that "computerisation" should be understood as one main task in language documentation and, at the same time, proposes a basic protocol to carry out this task. this basic protocol is based on a straightforward idea according to which any documentation project, in addition to its customary outcomes (audio and video recordings, transcriptions, morphological parsing and glossing, and free translations), should include nlp-friendly annotated data as its deliverables: we attempt then to draw documentary and computational linguists' attention towards the potentialities of a more integral and systematic collaboration between them. on the one hand, field linguists may get involved in creating relevant products from the nlp perspective (e.g. preparing representative treebanks taking as a starting point their own data). on the other hand, nlpers can get involved in the development of processes and protocols that may contribute to the transformation of linguistic data of the traditional sort into formats that may support nlp developments. according to @xcite , one feature for a language to be considered as a minor one is the few to zero availability of machine-readable resources. there are features such as the number of speakers or literacy speakers that may support the definition of a minor language in a general overview, but we want to emphasise the computational perspective in forcada's statement. dictionaries, translated text or annotated corpora, that are currently part of a standard language documentation process, are instances of machine-readable data. we consider that linguistic corpora are insufficient to disentangle the relationship between a language and its characterisation as a minor language. we claim the need to develop more multiple resources to support a consistent revitalisation of the language. however, we do not mean that all language documentation processes should include a massive technology development by itself. the magnitude of such a project would be cost-prohibitive. nevertheless, we have identified some elements that might be included in a documentation process that could drive a "computerisation" effect in the studied language. we want to emphasise the development of multipurpose linguistic databases, specifically aiming at language technologies, whose implementation will not radically increment the amount of expected work for the linguist. language technologies are purpose-specific programmes that try to address language-related tasks from spell-or grammarchecking to automatic machine translation. based on such databases, nlpers and field linguists may work together to develop nlp toolkits for minority languages. an nlp toolkit is a set of different tools made to computerise a language fully. we then take inspiration from the basic language resource kit @xcite and also consider established annotation frameworks, such as ud or unimorph, and current state-of-the-art methods in nlp, such as transfer learning. with transfer learning protocols, especially multilingual pretraining @xcite , cld² projects might automatise learning tasks by taking advantage of larger amounts of multilingual data and tools. a learning task in this context may refer to a specific nlp or functionality, such as a dependency parser, which has been trained to learn how to parse the syntax in a textual sentence. finally, we list the main tools that such basic toolkits could have: besides these tools, further developments that can be achieved for endangered languages, such as machine translation, are very appealing. however, we also need to point out that, despite the progress of the pretraining approaches and the use of few labelled examples, a translation system (or other kinds of nlp tools) should not be deployed with low-quality outputs, as it can mislead the user. limitations of their usage should be assessed according to the annotated data used and the purpose of the systems. ## conclusion cld² calls for an enrichment of language documentation projects by means of incorporating components, outcomes and methods from nlp research, as a strategy to promote the computarisation and revitalisation of minority languages. this paper shows that most of the interactions between computational linguistics and language documentation are framed as software developments that facilitate the various processes involved in documenting a language. the potential contributions of language documentation and language repositories to nlp research are under-exploited and deserve urgent attention from the nlp community. at the same time field linguists may also incorporate into the outcomes of their projects, data crafted into paradigms that can be automatically used for nlp developments (universal dependencies and/or universal morphology, for instance). this will benefit not only language documentation and computational linguistics scholars but also typologists and speech communities, as research in nlp has recently paid some attention to linguistic typology as a substantial source of linguistics knowledge to improve performance in different algorithms and technologies @xcite . indigenous communities, in turn, are highly enthusiastic about the computerisation of their languages as a political strategy that vindicates their languages and demonstrates that they are as valuable as major european languages. cld² can significantly contribute to this aim by promoting productive exchanges among field linguists, nlp researchers and members of indigenous communities as part of multi-component projects that put language revitalisation at their core.