Documenting Endangered Languages with L ang D oc: A Wordlist-Based System and A Case Study on M oklen (2024)

Abstract: Language documentation, especially languages lacking standardised writing systems, is a laborious and time-consuming process. This paper introduces LangDoc, a comprehensive system designed to address challenges and improve the efficiency and accuracy of language documentation projects. LangDoc offers several features, including tools for managing, recording, and reviewing the collected data. It operates both online and offline, crucial for fieldwork in remote locations. The paper also presents a comparative analysis demonstrating LangDoc’s efficiency compared to other methods. A case study of the Moklen language documentation project demonstrates how the features address the specific challenges of working with endangered languages and remote communities. Future development areas include integrating with NLP tools for advanced linguistic analysis and emphasising its potential to support the preservation of language diversity.

## introduction amongst the very first tasks in language documentation are to collect and record vocabulary of the language. traditionally, language data have been collected and stored in its most primitive form, often involving manual recording on paper or default word lists, sometimes with audio recording. this process is yet the most gruelling and labourintensive. despite the use of technology and/or computer-assisted systems in latter studies (e.g. @xcite , @xcite , @xcite , @xcite ), the heavy reliance on humans is inevitable, especially converting field notes into computer-stored data prior to any further analyses. human errors normally weaken the efficiency of a documentation project and contributes to various issues within the system (cf. @xcite , compton (2014)), including com-promising the overall quality of the information obtained, regardless of the limited resources and other constraints. langdoc foot_0 is then a system designed to streamline the recording and analysing process of the language data whilst mitigating errors associated with human involvement in collaborative projects, specifically for such languages including but not limited to which lacking conventionalised writing systems. its functionality extends to both online and offline environments, making it particularly well-suited for language documentation conducted in remote locations. in particular, this paper presents its idea, as well as system design, functionalities and features. it will also discuss the system's current limitations and outline the possible direction for future development. to illustrate the functionalities, this paper demonstrates langdoc with a real-world use case by its application in documenting the moklen in the southern thailand. this paper makes several key contributions to the field of language documentation. firstly, it aims to address common challenges faced in this domain, such as managing data from multiple sources, logistical difficulties in collaborative teamwork, and also extending to tackle such external limitations as the well-being of language informants. secondly, the paper proposes features to mitigate common errors and enhance the efficacy, whilst acknowledging the essential role of trained linguists. thirdly, the paper presents offline synchronisation feature is crucial for fieldwork in remote locations. the system allows users to collect data without an internet connection and syncs automatically when connectivity is restored. additionally, the system's architecture allows for future integration with tools for deeper linguistic analysis to further expand its capabilities. ## background issues and related work collecting vocabulary data for endangered languages presents significant challenges, particularly when the documentation effort is led by community outsiders. the conventional approach of conducting interviews and elicitation sessions with native speaker informants can be inefficient, costly, and potentially detrimental to the well-being of elderly informants who often serve as the primary sources of linguistic knowledge. one of the major issues is the limited productivity of data collection sessions, especially with elderly informants who may have physical limitations. as observed in moklen fieldwork, interviews with elders typically yield a maximum of 60 words per session, with several breaks required within a three-hour period. completing a modest vocabulary list of 250 words can take at least five days of work, and more extensive projects naturally require even greater resource investment. another challenge arises when multiple researchers are involved in the documentation effort. dividing informant interviews amongst project members can lead to wasted effort due to duplicate recordings of common vocabulary and the potential to miss more specific, culturally-related terms known to certain informants; not to mention the additional time to be spent merging data and identifying missing entries. furthermore, inconsistencies in the interpretations by different researchers can arise, especially when dealing with a semantically complex spoken languages like moklen. to resolve these discrepancies often requires revisiting informants in person, hindering the overall progress. even if larger team appear to be bring a faster data collection, the unique challenges of endangered language documentation suggest that a more focused approach tailored to the needs of the specific community is crucial. overwhelming elderly informants with lots of people can lead to shorter, less productive sessions due to factors such as fatigue and discomfort. in recent years, there have been efforts to integrate technologies for recording, transcribing, and analysing language records @xcite , as well as other nlp tasks @xcite to language documentation. nevertheless, most works focus on how can the data can be used to represent linguistic phenomena; little attention, however, has been given to tackle the fundamental problem of how linguists or researchers can actually and effectively collect and prepare the necessary linguistic data in the first place, especially for endangered languages with rapidly dwindling speaker populations. of course, good tools and applications have emerged to aid in field linguistics, such as aikuma @xcite , flex @xcite , and elan (max planck institute for psycholinguistics, 2023), yet often operate in silos and do not comprehensively address the multifaceted challenges faced by linguists in the field. there is a need for solutions that holistically address the data collection process whilst considering the unique logistical, ethical and communityrelated challenges faced when documenting such endangered languages. the issues highlighted above point to a primary use case that the proposed system aims to address, comprising a team of field linguists with varying experience working to document the vocabulary of an endangered language spoken by a remote community with few population of elderly native speakers. in the scenario when the opportunities to work with remaining fluent speakers are increasingly limited, efficiently and sensitively collecting high-quality data are paramount. ## the langdoc system langdoc is a comprehensive system designed to streamline the language documentation process, particularly for endangered languages lacking standardised writing systems. it incorporates several key features to address the challenges identified in the background section. ## evaluation and case study it is always difficult to find a good matrix to measure the performance of software development systems such as langdoc. however, evaluating its impact on language documentation projects is crucial for understanding its effectiveness and efficiency. ## discussion and future directions the langdoc system represents a step forward in optimising language documentation process, particularly for endangered languages in remote communities. however, it is essential to acknowledge the limitations of the current system. whilst excelling in data collection and organisation, lang-doc primarily focuses on the preliminary stages of language documentation, currently limited to managing wordlists, transcriptions, and basic metadata. additionally, the system's reliance on manual input and human involvement, even if mitigated through its collaborative features, may still introduce potential biases or inconsistencies, particularly in the transcription and annotation processes. integrating langdoc with state-of-the-art nlp techniques could significantly enhance its capabilities to help linguists doing their works. for example, automated transcription and annotation tools could reduce manual effort and potential biases from humans, allowing linguists to provide essential oversight, control quality, and go further with analysis of complex linguistic phenomena beyond lexicons. additionally, incorporating machine learning models trained on the collected data could assist in developing low-resource technologies, such as machine translation, parsing, and asr systems for the documented languages. exploring ways to involve language communities more actively in the documentation process could foster a sense of ownership and promote the preservation of linguistic heritage. this could involve developing user-friendly interfaces for community members to contribute to data collection, validation, and dissemination efforts, in addition to the tool used solely by linguists. ## conclusion this paper presented langdoc as a system to address challenges in documenting endangered languages without standardised writing not only in the form of software tools but also via presenting logical steps for human workflow. by incorporating project management, wordlist-driven recording, collaborative review, and offline access, it improves documentation efficiency and quality. the moklen case study demonstrated langdoc's capabilities in tackling data duplication, verification bottlenecks, and accommodating elder informants. whilst not a panacea, langdoc streamlines workflows and enhances collaborative project effectiveness, helping preserve linguistic diversity and sustain endangered languages in its most foundational process.