Unsupervised part-of-speech induction for language description: Modeling documentation materials in Kolyma Y ukaghir (2023)

Abstract: This study investigates the clustering of words into Part-of-Speech (POS) classes in Kolyma Yukaghir. In grammatical descriptions, lexical items are assigned to POS classes based on their morphological paradigms. Discursively, however, these classes share a fair amount of morphology. In this study, we turn to POS induction to evaluate if classes based on quantification of the distributions in which roots and affixes are used can be useful for language description purposes, and, if so, what those classes might be. We qualitatively compare clusters of roots and affixes based on four different definitions of their distributions. The results show that clustering is more reliable for words that typically bear more morphology. Additionally, the results suggest that the number of POS classes in Kolyma Yukaghir might be smaller than stated in current descriptions. This study thus demonstrates how unsupervised learning methods can provide insights for language description, particularly for highly inflectional languages.

## introduction many nlp applications and linguistic investigations are facilitated by having part-of-speech (pos) tags for words in context. providing such tags flexibly and at scale for novel texts requires a pos tagger. when working with low-resource languages, it is often infeasibly labor-intensive to develop labeled data that would enable the training of a supervised tagger, or even to develop a lexicon that delimits the set of tags that may be appropriate for each word @xcite and thereby facilitates the training of an accurate unsupervised tagger (e.g. @xcite . working with such languages requires turning to pos induction, which clusters words according to the contexts in which they occur in an unannotated corpus. following the distributional hypothesis @xcite , words that occur in similar contexts are assumed to belong to the same pos class. pos induction is a potentially useful tool for the documentary linguist. however, its utility for endangered and underdocumented languages remains to be established, as it is not always clear what the pos class of a word should be in many such languages or even whether the notion of pos classes as established for high-resource european languages is appropriate @xcite , due to potentially high degrees of polyfunctionality @xcite @xcite . the goal of this paper is to evaluate the insights of pos induction for language documentation, through a case study on kolyma yukaghir (yukaghiric), a highly inflectional endangered language of northeastern siberia (republic of sakha, russia). ## pos induction in highly inflectional languages pos induction leverages distributional information by representing each word as a co-occurrence vector, which reflects how often it appears near each other word in a corpus, and clustering words with similar vectors. this approach is successful for languages that display fairly rigid word order and little inflection, like english, because the vectors are characterized by frequent function words that predominantly co-occur with words in certain pos classes, such as "the" and "to". however, it is not so successful for highly inflectional languages, in which the corresponding function elements are bound morphemes @xcite . successful unsupervised pos induction for highly inflectional languages requires building morphological information into the model. this approach leverages the fact that inflectional affixes are strongly associated with pos classes. if the pos classes of the affixes in a word are known, they can be used to delimit the set of possible pos classes for the root of that word @xcite . if the pos classes of affixes are not known, the distributional hypothesis can be applied at the morphological level: roots that have similar distributions of co-occurrence with affixes can be assumed to belong to the same pos class @xcite @xcite . however, building morphological information into pos induction may not be successful in all languages. there may be three major issues, which we illustrate with examples from kolyma yukaghir. the first issue is that the same affix may attach to roots that would traditionally be considered to have distinct pos classes, and as a result are analyzed as homonyms. in our examples, the suffix -n attaches to nouns when they modify a noun (1) or encode the arguments of a postposition (2), in which case it is glossed as "genitive". an identical morpheme attaches to numerals (3) and "adjectives" (4), but in these cases it is often glossed as "attributive" or "adverbializer", respectively. the second issue is that the same root may appear with affixes that are prototypically associated with traditionally distinct pos classes @xcite . numerals and "adjectives" appear in (3-4) with a suffix that is indistinguishable from prototypically nominal case-marking in (1-2), but they are also attested as the main predication of a clause bearing prototypical verbal morphology, such as aspect, evidentiality, person and number (5-6). the third issue, which is a consequence of the first two, is that two roots may have highly similar affix co-occurrence distributions but nevertheless be considered as having distinct pos classes. despite the similarities between numerals and "adjectives" in the examples (3-6), they are treated differently in grammars: "adjectives" are grouped with verbs @xcite @xcite , while numerals are either considered as a separate pos class @xcite or classified simultaneously with adnominals and verbs @xcite . these differences in conceptualization result in a different number of pos classes: 8 according to @xcite , and 6 to @xcite . the large degree of shared morphology across roots in such highly inflectional languages raises the question of whether applying the distributional hypothesis at the morphological level is appropriate, as well as the question of what the relevant pos classes might be in such languages in the first place. we explore these questions from a bottom-up, data-driven approach, with a case study on kolyma yukaghir. specifically, we seek to identify and evaluate the number of pos clusters through unsupervised induction, without specifying a predetermined value. ## kolyma yukaghir like other languages in the siberian linguistic area @xcite , kolyma yukaghir is strongly head-final, and it displays sv/aov constituent order with nominative-accusative alignment. morphologically, kolyma yukaghir is a predominantly agglutinating, suffix-dominant language, with partially fusional morphology. suffixes display some allomorphy due to residual vowel harmony and consonantal assimilation processes @xcite @xcite . in terms of morphological complexity, roots show differences in terms of the number and range of affixes they typically occur with. roots used "verbally" (i.e., for predication) have the largest number of affixal slots, some of which can be filled by a wide range of possible items (e.g., aspect). roots used "nominally" have fewer slots, which can typically be filled by fewer possible affixes, and sometimes occur without affixes at all (e.g., kind'ə in 6). in this study, we analyzed 19 of the 40 monologic texts collected in the late 20th century @xcite . these texts were narrated by five different speakers in the community and include a variety of genres: folktales, personal and fantastical stories, descriptions of games and competitions, an account of fortune telling, etc. to prepare the data, we stripped the texts of glosses, transliterated them into cyrillic orthography, and divided them into intonation units (iu; @xcite . ius are defined as "a stretch of speech uttered under a single coherent intonation contour" @xcite or the "spurts of language" in which speakers typically produce speech @xcite . affix boundary markers from the original transcriptions were maintained, so the choice of writing system did not impact the results. however, we removed root-internal boundary markers in compounds (13 words total). we also removed clitic boundary markers, replacing them with white space in the case of proclitics, and affix boundary markers in the case of enclitics. this treatment yielded texts that follow established written conventions as closely as possible. additionally, it meant that every word presented the same structure: if one or more morphological boundaries were present, the left-most morpheme was the root, and any subsequent elements were suffixes. after preprocessing, the data contained 3,513 word tokens (where a token was taken to be anything bounded by white space). these word tokens ## methods we obtained co-occurrence vectors for roots and affixes under four distinct definitions. the first definition, root(roots; iu), yielded a vector for each root, based on the roots it co-occurs with in an iu, as shown in the first row of we obtained vectors similarly for the remaining three definitions: root(affixes; word) yielded a vector for each root, based on the affixes that attach to it; affix(roots; word) yielded a vector for each affix, based on the roots it attaches to; and af-fix(affixes; word) yielded a vector for each affix, based on the affixes it co-occurs with in a word. examples of these definitions are shown in for each definition, we first removed elements that only ever occurred as isolates in the corpus (for root(roots; iu), roots that only ever occurred alone in an iu; for root(affixes; word), roots that only ever occurred without affixes; and for affix(affixes; word), affixes that never occurred alongside other affixes in a word). we then used k-means clus-tering on the vectors of remaining elements under each definition to induce classes of roots/affixes that have similar distributions within the corpus. we picked the number of clusters using the elbow method, where cluster quality was measured by inertia. for qualitative interpretation, we identified the 20 roots/affixes with the highest degree of centrality from each cluster. these four definitions represent different ways to approach pos induction. root(roots; iu) and root(affixes; word) assign each root to a class, as is typical for pos in european languages. we expect root(affixes; word) to be better than root(roots; iu) for kolyma yukaghir because it incorporates crucial morphological information; however, we do not expect it to be particularly useful, due to the large degree of shared morphology across roots. af-fix(roots; word) and affix(affixes; word) assign each affix to a class, which allows the pos of a root to be determined in context by the affixes that are attached to it. we expect these definitions to be more useful than the root-wise ones because they reflect the polyfunctional nature of the language. given the potential that affixes may mark different functional roles in kolyma yukaghir than is typically assumed for european languages, and may therefore co-occur with each other broadly, we might expect affix(affixes; word) to be less useful than affix(roots; word); however, the utility of af-fix(roots; word) ultimately depends on the extent to which roots have (gradient) prototypical associations with traditional pos roles. ## results as shown in figure 1 , the elbow method identified 2 clusters (of non-isolates) for 3 definitions, and 3 clusters for root(affixes; word). figure 2 visualizes the clusters under each definition using t-sne. the qualitative analysis of the 20 words with the highest degree of centrality under root(roots; iu) shows a lot of variability. words closest to the center in the small cluster (@xmath0.g., 'later'), pronouns (e.g., 'y'all', 'who'), nouns (e.g., 'hoof'), verbs (e.g., 'blow') and "adjectives" (e.g., 'fast'). similarly, the big cluster (@xmath1; we find the same categories as above. the clusters under root(affixes; word) show more consistency, as expected. the 20 words in the smallest cluster (@xmath2.g., 'river'), with the exception of (a) root(roots; iu) (b) root(affixes; word) (c) affix(roots; word) (d) affix(affixes; word) figure 2: k-means clusters under the 4 definitions the two copulas (of which one also functions as a placeholder), one verbal root (e.g., '(be) outside') and a homonym (e.g., айии-'shoot', 'only'). we find the opposite pattern in the slightly bigger cluster (@xmath3, where all the roots are more verbal in nature (e.g., 'hear') but one (e.g., 'bell'). the third and biggest cluster (@xmath4; we find nominal (e.g., 'old woman') and verbal roots (e.g., 'take'), along with "adjectives" (e.g., 'good'), pronouns (e.g., 'what'), and nouns that can function as postpositions (e.g., 'back'). as for the clusters of affixes, affix(affixes; word) yields similar behavior to root(roots; iu) with a very asymmetric split. all the affixes in the small cluster (@xmath5, and the 20 affixes returned for the big cluster (@xmath6, with the exception of a plural and a genitive/attributive allomorph. the results for affix(roots; word) are more insightful, as expected. all but 3 of the 20 affixes in the big cluster (@xmath7.g., inchoative); the exceptions are two case markers and the directional -ҕудэ. in the small cluster (@xmath8, two thirds of the affixes returned were nominal (e.g. 3 rd person possessive), whereas the remaining third were verbal and predominantly associated with non-finiteness. ## discussion & conclusion the number of clusters identified by the elbow method is rather small. this could be because there is not enough data to make finer distinctions in the clustering process, beyond a coarse split into prototypically "nominal" and "verbal" pos classes (and a third mixed class in root(affixes; word)). alternatively, it could be because kolyma yukaghir permits a given affix (or, to some extent, root) to be used in myriad ways, such that the treatment of each root/affix as monolithic (in terms of representing one feature in the vectors, and in terms of having only one pos class) obscures deeper complexity. as for the qualitative analysis of the clusters, the results suggest that root(affixes; word) indeed offers a more informative clustering than root(roots; iu). the latter definition fails to find structure in the data, whereas the former returns two cohesive clusters (with a nominal and a verbal tendency) and a third cluster with some variability. this variability, however, reflects in part the polyfunctional nature of the language. some roots that look prototypically nominal, like 'old woman', can bear verbal morphology to convey predicative possession (i.e., 'have a wife'), and thus their clustering with "adjectives", like 'good', that can also be marked with nominal and verbal morphology is coherent with their distributions. overall, the smaller number of function words makes the incorporation of morphological information particularly useful as anticipated. as for affixes, the clustering under affix(affixes; word) is less useful than that under affix(roots; word). these results suggest that, to a certain degree, some roots might be prototypically associated with noun and verb pos roles. in addition, the homogeneity of the bigger cluster in affix(roots; word) with verbal functions indicates that verbal affixes might be a more reliable source of information. this probably results from finite, assertion-making words being more morphologically complex: a "verbal" root can carry several affixes simultaneouslymarking it for tense, aspect, evidentiality, and person/number -whereas "nominal" roots tend not to carry many affixes at once. nominal stems can be marked for possession, case, and evaluatives, but rarely do all co-occur. thus, our removal of isolates -affixes that never occurred alongside other affixes in a word -is likely to have affected nominal affixes more than verbal affixes. taken together, the results suggest that applying the distributional hypothesis at the morphological level in a context with significant shared morphology can yield successful results, especially when clustering roots and affixes each on the basis of the other. clustering might be more reliable for words that typically bear more morphology. however, the results can be fairly coarse-grained; to obtain finergrained insights, more data and/or a more complex (mixture-based) approach may be necessary. additionally, the results also provide some insight into what the relevant pos classes in kolyma yukaghir might be. rather than the eight and six pos classes listed in grammatical descriptions @xcite , the clustering suggests a binary split at the morphological level centering around nominal and verbal functions, with the possibility of a third mixed class. further research is needed to investigate the degree to which this third distinction is categorical or represents a cline with nouns and verbs on opposite ends.