Dim Wihl Gat Tun: T he Case for Linguistic Expertise in NLP for Under-Documented Languages (2022)

Abstract: Recent progress in NLP is driven by pretrained models leveraging massive datasets and has predominantly benefited the world’s political and economic superpowers. Technologically underserved languages are left behind because they lack such resources. Hundreds of underserved languages, nevertheless, have available data sources in the form of interlinear glossed text (IGT) from language documentation efforts. IGT remains underutilized in NLP work, perhaps because its annotations are only semi-structured and often language-specific. With this paper, we make the case that IGT data can be leveraged successfully provided that target language expertise is available. We specifically advocate for collaboration with documentary linguists. Our paper provides a roadmap for successful projects utilizing IGT data: (1) It is essential to define which NLP tasks can be accomplished with the given IGT data and how these will benefit the speech community. (2) Great care and target language expertise is required when converting the data into structured formats commonly employed in NLP. (3) Task-specific and user-specific evaluation can help to ascertain that the tools which are created benefit the target language speech community. We illustrate each step through a case study on developing a morphological reinflection system for the Tsimchianic language Gitksan.

## introduction progress 12 in nlp research has primarily manifested in tools for the world's political and economic superpowers @xcite , and it is unclear how we can build more inclusive language technologies. even multilingual pretraining methods (e.g., @xcite , capable of producing effective models in the absence of large annotated training datasets require 1 dim wihl gat tun -"this is what the people should do" 2 first two authors contributed equally. unannotated corpora that are prohibitively large for 90% of the world's languages @xcite . nevertheless, many languages in this 90% have a body of resources. language documentation and linguistic fieldwork are an ongoing task worldwide, and many resources continue to be developed in these traditions @xcite . we have access to wordlists, bilingual dictionaries for over 1000 languages @xcite , aligned speech recordings for over 700 languages @xcite , multi-parallel texts for 1600+ languages @xcite , and knowledge of related languages @xcite . indeed, researchers have leveraged these resources to build impressive, useful computational systems for multilingual morphological analyzers @xcite , adapting pretrained language models for over 1000 languages @xcite , and building massively multilingual speech recognition systems @xcite ), among others. there are additional language documentation resources which have yet to be fully leveraged in the aim to produce more inclusive language technology. interlinear glossed texts (igts) depicted in figure 1 are semi-structured texts which comprise not only monolingual corpus data (e.g. al'algaltgathl) but also morphemelevel segmentations (e.g. cvc~algal-@xmath0, glosses for component-morphemes (e.g. pl~watch-3.@xmath1, word alignment information @xcite , and free translations. igts remain a major annotated datatype produced in the course of linguistic fieldwork: examples are continuously digitized in large databases for hundreds of languages @xcite , and entire corpora of igt are periodically published in volume series such as texts in indigenous languages of the americas. they have the potential to serve as training data for a wide variety of computational systems including bilingual lexicons, morphological analyzers, dependency parsers, part-of-speech taggers, and word-aligners @xcite . yet while they are accessible, they remain severely underutilized for these purposes. part of the general hesitancy in adoption of igt as training data may lie in the fact that the annotation format is only semi-structured and often language-specific. while the general igt format is governed by the leipzig glossing rules @xcite , there remains significant flexibility for the annotator to customize tags and conventions for any given language. this makes igt challenging as a format for training supervised nlp models. with this paper, we make the case that igt data can be leveraged in nlp research and language applications for speech communities, provided that target language expertise is available. specifically, we argue that it is essential to collaborate with documentary linguists who are familiar with the language-specific annotations in the igt data in order to leverage the data for nlp tasks. this may furthermore provide a foundation for co-designing language technologies with a given speech community @xcite . our paper provides a roadmap, portrayed in fig. 2 , for navigating three areas of significant uncertainty that arise when incorporating igt data for inclusive language technology. first, we need to define what nlp tasks can be accomplished with a given set of igt data, and whether they are of value to the speech community. second, after selecting useful tasks, we will need to preprocess the data, potentially by converting it to a structured format commonly employed in nlp tasks. finally, we need task-specific and user-specific evaluation procedures in order to be explicit about the failure modes of the technology, as it is ultimately being developed for end users like speakers and linguists rather than solely comparison with other researchers. we focus on the first two of these areas, forwarding our argument through a case study on developing a morphological reinflection system for the gitksan language (section 2.3) that has applications in language teaching. ## challenges in incorporating igt into nlp research because tokens in igt are already segmented and annotated, it forms an ostensibly convenient starting-point for further processing and tokenbased grouping. in many ways, igt is, however, a challenging data type for use in pedagogical and nlp applications. this section presents three specific challenges posed by igt data when nlp techniques are applied. first, while igt will contain a wealth of useful information for nlp models, it might also contain information which is far too fine-grained for automatic learning purposes, at least given the quantity of data which are available. second, igt often contain idiosyncratic or language-specific conventions which may not be easily converted to or represented in standardized frameworks. third, because igt is used as a device for language documentation, it will often contain dialectal variation, an important meta-characteristic which in aggregate cannot be easily distinguished from other types of variation or spelling errors. we argue that handling these issues for successful data preprocessing requires consultation with linguistic experts, and exemplify with instances from the gitksan igt and our use-case. ## steps toward building a language learning application the inflectional paradigms collected from the adapted igt corpus are overly sparse for automatically generating pedagogical exercises. to automatically fill in these paradigms, an example of which is shown in appendix b, we train and evaluate a morphological reinflection system. 10 data we train and test reinflection models on the gitksan morphological paradigms described in section 3. we generate three splits of the data from our complete set of paradigms: train (@xmath2, validation (@xmath3, and test (@xmath4. training we form training pairs by using the given forms in each table and learn to reinflect each given form in a table to another given form in the same table, following @xcite . model parameters are shown in appendix c. evaluation during test time, we predict forms for missing slots based on each of the given forms in the table and take a majority vote of the predictions. we evaluate accuracy on the test set by counting the number of the 124 forms that were correctly predicted. we find that the transformer model generates 87.09% of the test forms correctly. analysis. our model provides strong performance when measured by the standard metric of accuracy, in particular considering that it is trained on only 858 examples. accuracy, however, only provides one perspective on the efficacy of the model @xcite . the appropriate evaluation of the system is highly context dependent: for our goal of generating language learning exercises, we want to evaluate whether our system and automatically generated grammar exercises allow for more effective language learning; raw accuracy gleans little insight to the effectiveness of the system for this goal. if in contrast our goal was to facilitate language documentation, we would want to evaluate whether the model gives an overall significant reduction in documentation effort-this largely depends on whether the automatic annotations are of sufficient quality that correcting remaining errors takes less time than annotating all the data from scratch. further research, in collaboration with documentary linguists and the speech community, is required to determine whether our system can achieve the desired goals of building more practical, inclusive language technology. ## discussion incorporating igt data for nlp language documentation provides a valuable data source for many so called "left-behind" languages @xcite , which lack traditional annotated and unannotated nlp datasets. for example, igt data can be used to train systems for morphological inflection, segmentation and automatic glossing, among other applications. nevertheless, the annotations in igt are rarely ideally suited for typical nlp tasks, and may need to be significantly adapted. this will typically be hard without extensive knowledge of the target language and annotation conventions which were employed when the igt data were generated. linguists and community language experts are well-positioned to address questions related to igt usability, the structure of the target language, variation in the data, and other annotations in the source data. collaboration with language experts is not only vital for successful data preprocessing and conversion to the formats required for the typical nlp task, but can also naturally help define research goals and drive the project toward them. inclusive research goals nlp technologies for underdocumented languages have the capacity to speed up language documentation (e.g., anastasopoulos, 2019); assisting language revitalization (e.g., @xcite ; and creating digital infrastructure (e.g., anastasopoulos and neubig, 2019). these high-level goals are only a part of what it may mean to create inclusive language technology. equally valuable as a research goal may be inclusion: for speech communities to be acknowledged and engaged in the course of the the research project. 11 we encourage nlp projects on low-resource, minoritized, and/or endangered languages to begin by understanding the speech community context, proceed with community collaboration or endorsement, and ultimately produce concrete benefits that speech communities recognize. this might include outcomes for language teaching and pedagogy, or training opportunities in technology or research. evaluation methods can be compiled which address nlp researchers, linguists, and communities' overlapping and divergent goals. for example, pedagogical tools can be directly evaluated for dialect fairness and user/learner improvement. practical collaboration we suggest seeking out opportunities to collaborate directly with community members, in order to solicit their specific expertise when setting the research agenda (i.e. task definition) and conducting evaluation @xcite . when the nlp researcher has no existing contact or history with the speech community, this can be pursued via collaboration with a documentary linguist with established community relationships and a similar desire to engage in this research model. recognize that in any collaboration, different individuals contribute different skills and experience (e.g. pedagogy, annotation, knowledge of community attitudes) and may have different goals and preferred ways of participating, which should simply be discussed within the partnership to ensure things run smoothly. ## conclusion although a majority of the world's languages lack the kind of large annotated and massive unannotated datasets which are used to train modern nlp models for high-resourced languages like english @xcite , many languages have other potential data sources such as language documentation data, which so far have remained under-explored. however, care must be taken when applying this type of data, which originally is not intended for nlp use. this is important to ensure that the resulting technologies actually achieve their intended goals like accelerated language documentation or genuinely helpful computer-assisted language learning. collaboration with linguists can provide the expertise necessary to engage in modeling with igt data for underdocumented languages. linguists can help define an nlp task with good value propositions, given their familiarity and connections with the speech community. they can provide guidance on navigating the igt format so that we can extract the most useful information for the task at hand. finally, they can assist in evaluating whether the model achieves appropriate performance on the speech community use cases, and provide feedback on metrics for model success and fairness across dialects. throughout the development process, documentary linguists and speech community members should be consulted. this will further a greater understanding of the source data and lead to more equitable and effective technologies.