A Quest for Paradigm Coverage: The Story of N en (2023)

Abstract: Language documentation aims to collect a representative corpus of the language. Nevertheless, the question of how to quantify the comprehensive of the collection persists. We propose leveraging computational modelling to provide a supplementary metric to address this question in a low-resource language setting. We apply our proposed methods to the Papuan language Nen. Nen is actively in the process of being described and documented. Given the enormity of the task of language documentation, we focus on one subdomain, namely Nen verbal morphology. This study examines four verb types: copula, positional, middle, and transitive. We propose model-based paradigm generation for each verb type as a new way to measure completeness, where accuracy is analogous to the coverage of the paradigm. We contrast the paradigm attestation within the corpus (constructed from fieldwork data) and the accuracy of the paradigm generated by Transformer models trained for inflection. This analysis is extended by extrapolating from the learning curve established to provide predictions for the quantity of data required to generate a complete paradigm correctly. We also explore the correlation between high-frequency morphosyntactic features and model accuracy. We see a positive correlation between high-frequency feature combinations and model accuracy, but this is only sometimes the case. We also see high accuracy for low-frequency morphosyntactic features. Our results show that model coverage is significantly higher for the middle and transitive verbs but not the positional verb. This is an interesting finding, as the positional verb paradigm is the smallest of the four.

## introduction a key question in studying language is: when do we have enough data to fully understand the system? this is especially important in language documentation. as himmelmann (1998) states, 'the aim of language documentation is to provide a comprehensive record of the linguistic practices characteristic of a given speech community.'. @xcite extends this by asking, 'if a comprehensive record is unattainable in principle, is there a consensus on what an adequate record looks like. how would you quantify it?'. honouring their formulation, @xcite label this the 'himmelman-bird' problem. 1 in their paper, the authors strive to explore this himmelman-bird problem for the inventory of phonemes, which are the subdomain of language with the smallest and hence most frequentlyoccurring units. they set the bar even lower by simply requiring that at least one allophone of each phoneme occur. they then examine how much text it might take to capture a language's entire phoneme inventory, drawing on a sample of 137 distinct languages, some with additional dialectal or register variety taking the total to 158 speech varieties. full 'coverage' is achieved, for a given domain of language (say, its phoneme inventory) and a given corpus, it there is at least one incidence of each relevant unit (in this case, each phoneme) in that corpus. here we strive to follow a similar route for morphemes and their respective allomorphs, while still posing the problem in its simplest and hence most easily-satisfied form: we look just at verbs, and we restrict ourselves to one representative lexeme (the commonest) in each of the four main morphological classes -see below. the goal of collecting a representative sample has permeated many fields, from biology to sociology. researchers have explored the idea of having a gold standard process for collecting all required components to describe a system. for example, if we wanted to gather all the phonemes for english, the 'rainbow passage' by @xcite may be chosen. the first four lines of the passage cap-ture all phonemes for english. in morphology, we can discuss the idea of collecting all principal parts @xcite to construct the entire paradigm. this idea presents as a great solution to the difficulty faced by low-resource languages and, more specifically, language documentation. however, one caveat is the system knowledge required for designing such a task. for example, how might a linguist know all the phonemes before beginning their in-field analysis and recordings? accordingly, we make the distinction between heuristic and attestation coverage. the first refers to the discovery stage of a language, leading to a sketching of the dimensions of its design space -the logical space of all its possibilities in a particular domain, such as verbal inflections -through discovering the dimensions where it encodes contrasts (say 'dual number', 'future imperative', 'imperfect aspect'), and mapping out the ways these interact (say 'future imperfective dual imperative', as in nen nandowabe 'you two should be talking later on!' @xcite . the latter describes the scenario where a description exists, and the aim is to collect examples of language within the denoted design space. the concept of a 'whole language' is so vast and heterogenous that it is not operationally useful for many linguistic or practical purposes. to explore this question, we consider a particular component of language, inflectional morphology on the verb. we base our study on modelling morphological inflection in the nen language and examine the attestation coverage observed in the transcribed natural spoken corpus and inflection models built on the same data. in this paper, we address the following questions: (1) how can we test the degree to which a linguistic subsystem exhibits coverage in a given corpus (2) how does the model coverage compare with the corpus? (3) does corpus frequency relate to model accuracy? (4) can we use model-based learning curves to predict the data required for complete coverage? we propose a test case for the model that asks to predict a complete paradigm, i.e. the complete multidimensional array of inflected forms -english is too morphologically impoverished to furnish a good example (the best is with the copula to be: {am, (art), is, are; was, were; (to) be; being}. our results indicate that the generalisations afforded by the transformer model yield better coverage than the natural corpus. furthermore, we explore two separate correlations of the high dimensional axes of nen verbs; the undergoer and agent combinations and the agent and tense, aspect, and mood (tam) combinations. while frequent features tend to be captured correctly by the model, surprisingly, so are some low-frequency forms. finally, we use learning curves to predict the data needed for 100% coverage. ## related work to our knowledge, only two prior computational studies of nen exist. @xcite presents a finite-state description, while (muradoglu et al., 2020) explores the use of neural architecture, to model nen verbal morphology. the latter is based on two high performing submissions in the sigmorphon-conll 2017 shared task @xcite . between the two approaches, the finite-state description achieves a higher accuracy across the corpus. however, we note that the accuracies reported are not directly comparable given the ongoing development of the corpus. despite the performance difference, we opt to use a neural approach to enlist the aid of its generalising ability. moreover, the statistical nature of these models make the intersect with corpus linguistics an object of interest. specifically, we use a transformer @xcite based model. transformers have been successful in capturing complexities of phonological and morphological details @xcite , often achieving state-of-the-art performance. over the years, the inflection task has been extended to many languages, including other complex morphological systems such as murrinh-patha, kunwinjku and seneca. ## the nen language nen is a papuan language of the morehead-maro (or yam) family @xcite . it is spoken as a native language in the village of bimadbn in the western province of papua new guinea @xcite . most nen speakers are multilingual, typically speaking several of the neighbouring languages. verbs in nen are notoriously complicated and are described as the most complicated word-class in nen @xcite . they can be grouped in several ways, either as prefixing and ambifixing or by further breaking down the inflection patterns. prefixing verbs consist of the copula (and its derivatives 'go'/'come'/'have'), 'to walk' and positional verbs. another distinguishing feature of prefixing verbs, is the lack of infinitives. both ambifixing and middle verbs form infinitives through suffixing -s to the verb stem. in this study, we have listed the prefixing verb lemmas as the verb stem. ambifixing verbs can be separated into middle and transitive verbs. here, we separate the verb types beyond the prefixing and ambifixing categories as the corresponding paradigms are distinct. we provide details for the verbs we track below. ## data the nen corpus is made of 44 individual texts that were naturalistically recorded in the field. this amalgamates to approximately 8 hours of spoken text or over 30,000 words. this is filtered to over 6,000 verb instances representing 2,282 forms. some of these forms are the same, with different feature combinations due to syncretism or polysemy. for example, the sequence yn-can be parsed in two ways. it can either mean the prefix yn-coding first person nonsingular undergoer for the α series or y-n the third singular undergoer with the ventive (towards) directional. each of these instances are treated separately to expose the model to all possible meanings. a large portion of the texts in the corpus are coconut interviews 2 , these typically involve socalled biographical questions (parent names, place of birth etc), and questions about coconut trees that belong to the interviewee. this type of text was chosen as it can include a variety of tense -whether someone has planted or will plant a coconut treeand is a topic that easily inspires conversation from locals. although, these do not constitute a genre in the traditional sense, they do exhibit characteristic features, such as a high token count of the verb yis 'to plant' and third person non-past copula ym. the remaining texts range from anecdotal stories, folk tales, other narratives or procedural explanations. ## experiment we contrast the corpus-based account of the nen verbal paradigm to that modelled by a transformer model @xcite . our study is conducted in two parts: first, we follow the attestation coverage of the paradigm for one representative verb for each type in the corpus. second, we train transformer models to generate a complete paradigm 2 see evans (2020) for more details. for an unseen (barring the copula) verb for each type with incremental amounts of data. we establish a learning/coverage curve for each method @xcite . we use the term coverage here to mean the percentage of cells observed in the corpus or correctly predicted by the models out of the entire language design space. ## results and discussion a full paradigm for one verb is unlikely to be encountered in natural speech, or language learning contexts @xcite . although the focus of this paper is not language learning, the sparsity of paradigm coverage observed in these contexts is equally relevant here. based on various well-known corpora, @xcite shows that languages with larger verbal paradigms exhibit lower coverage. most notably, the only language with full coverage of its verbal paradigm is english, which only has six verbal forms. by contrast, finnish has 365 verb forms and only a 40.3% saturation even though the corpus size is almost double (2.1 million words compared to the brown corpus of 1.2 million words) that of the english counterpart. @xcite reports on the bleak data requirements to record each cell of the transitive verb in nen. here we have utilised the power of transformer models to leverage abstraction and statistical learning. figure 1 shows that the model based on the corpus does significantly better in terms of coverage. this suggests that while each combination might not be present in the corpus, the relevant information is. this typically parallels a mechanism utilised by field linguists to bootstrap the mapping of a linguistic paradigm since going through a complete paradigm for one particular verb is implausible. instead, the circumstantial context primes language informants to showcase verbs of different semantic domains. the field linguist typically obtains part of the paradigm (either through elicitation or by natural means) for each verb. these fragments likely allow for a reconstruction of the entire paradigm. dimensional independence allows the linguist to fill out parts of the paradigm. this task has been described as the paradigm cell filling problem (pcfc) @xcite ; @xcite ; @xcite . figure 1 shows the paradigm coverage across the four verb types in question. we contrast modelbased coverage with a corpus-based account. in both instances, we follow the trajectory of one rep-resentative verb. for the model, the four test verbs are detailed in the section 5.2.2. the corpus coverage curve follows akingr 'to be standing' for the positional, owabs 'to speak' for the middle, and räms 'to do/give' for the transitive verb. the model and the corpus follow m 'to be' since the copula verb is one entity. the most observable behaviour shown in figure 1 is the fluctuation across models trained across different training sizes. although, in general, the growth is positive, we see a significant difference across each step. one explanation might be the skew within the samples added. in other words, the added examples negatively influence the generalisations built by the model. another might be the model sensitivity to initial training data and data order. to account for the statistical variation, we report confidence bands for each verb type by measuring the variation in accuracy by dividing the test case for each verb into four random partitions. the partitions are randomly sampled as the test file is constructed in paradigmatic order. if the partitioning is performed sequentially, we might observe bias in one part of the paradigm, yielding large error margins. the model shows greater coverage for the transitive, middle and copula verb types than the corpus account. interestingly, the growth curve shows that the model-based account for positional verbs does worse than the corpus account. this is because the learning curve for the positional verb fluctuates substantially. the best-performing model for positional verbs is obtained with only 900 training examples (or 3,339 annotation units) at 16.5% coverage compared with the corpus account of akingr at 9% across the whole corpus. given that the paradigm of the positional verb is the smallest among the four, we would have expected coverage to be high. a possible explanation for this might be that there are few instances of positional verbs in the corpus (26 distinct forms across seven lexemes) and, thus, the training set. we also observe looping errors as described in @xcite , particularly for training sets below 1,000 instances. we describe the coverage growth relative to annotation units to capture the data requirements for paradigm representation fully. the texts are segmented into annotation units to retain some of the contextual information surrounding the verb in question. these units are typically one complete sentence and most commonly correspond to a segment in elan @xcite . on average, 4.7 words per intonation unit, one of which is usually a verb. with 6,446 annotation units across the corpus, on average, for every 2.88 units, there is a distinct form encountered. the model paradigm coverage is contrasted with that from the nen spoken corpus. we make a point to situate the required data size for training the model (i.e., train + dev) with units that relate to the corpus to help highlight the distillation process. typically, the model training size is measured in the number of instances. however, when collating a data set for a specific natural language processing (nlp) task -such as morphological inflection, the corpus is filtered from total words (assuming transcription exists) and later further distilled to types from tokens. to address our third question, we analyse the frequency of the verb features along the tam/actor and actor/undergoer dimensions. we expect a strong correlation between highly frequent features in the corpus and the model accuracy for that slot. figures 2 and 3 show the frequency of feature bundles. in both figures, the size of the bubbles corresponds to the frequency of the two sets of features in question (tam and actor or actor and undergoer). the saturation of the bubble shows how successful the model is in capturing the particular feature combination. the darker the bubble, the more likely the model will produce the correct corresponding form. these results are based on the model training with the entire training set available (2,000 instances). as expected, both figures show a correlation between the bubble size (corpus frequency) and saturation (model accuracy). nevertheless, there are cases where the corpus frequency is low, but the model proves to be proficient in producing the correct form. one such example is the imperfective imperative (ipfv.imp), the second person plural actor (which requires a prefix of the α series and the -tang suffix) makes up for 0.29% of the training data, but the model produces the correct form more than 66% of the time. one explanation might be that the rule's complexity and the chosen test verbs do not trigger allomorphic variants. we note the morphophonological element of inflecting. while we have tried to choose regular verbs, they still exhibit a phonological layer. it is hard to disentangle such effects. one possible future direction would be to choose a list of verbs across the categories presented here which exhibit the full range of phonological phenomena observed in nen. for example, verbs that might trigger vowel harmony and the consequent allomorphs. we further our analysis by providing a predictive quantity of data needed to reach 100% accuracy. we utilise scipy-based @xcite extrapolation by treating the resultant coverage curve as a learning curve. the predictions presented here are optimistic; to ensure that the predictions are based on monotonically increasing functions, we ensure that: it is tempting to draw parallels between language learning and the analysis presented here. however, we remind readers that we base our predictions on one representative verb and focus on attestation coverage rather than heuristic coverage. furthermore, we note that heuristic coverage would require a vastly more significant quantity of data. in addition, the numbers here are for one verb only, and it does not extend to include all parts of speech. ## conclusion we propose 'coverage' as a new way to measure the comprehensiveness of a corpus for morphological paradigms. here we present this application to nen verbal morphology. this methodology can be extended to include other parts of speech or languages. our results show that using deep learning approaches, more specifically the transformer architecture @xcite allows us to exploit the generalisable parts of a paradigm and thus grant us a higher coverage. the modelbased account yielded higher attestation for three of the four verbs considered. in an ideal setting, each inflection feature for each word would be observed and recorded naturally. however, this is an impossible feat in real-life. using statistics-based modelling like the transformer model allows us to synthesise forms based on examples encountered in the training data. as a result, the existing corpus can account for more of the system than a simple count within the corpus would suggest. we have explored the basis of the conventional wisdom of higher frequency yielding better model performance. while this holds, we observe a positive correlation between high-frequency feature combinations and model accuracy; we also see that the model can correctly generate less frequent feature combinations as well. we provide data quantity estimations based on the learning curves generated. these predictions are meant only as a guide rather than anything definitive, as they present an optimistic case defined by the enforcement of monotonicity. the extension of our proposed methodology to other languages with diverse morphological characteristics remains an open direction for future work.