Leveraging E nglish Word Embeddings for Semi-Automatic Semantic Classification in Nêhiyawêwin ( P lains C ree) (2021)

Abstract: This paper details a semi-automatic method of word clustering for the Algonquian language, Nêhiyawêwin (Plains Cree). Although this method worked well, particularly for nouns, it required some amount of manual postprocessing. The main benefit of this approach over implementing an existing classification ontology is that this method approaches the language from an endogenous point of view, while performing classification quicker than in a fully manual context.

## introduction grouping words into semantic subclasses within a part of speech is a technique used widely throughout quantitative and predictive studies in the field of linguistics. @xcite use high level verb classes to predict the english dative alternation, @xcite uses verb class as one of the feature sets to help predict the alternation of finnish think verbs, and @xcite use polarity classifications (good vs bad) from pre-defined lexica such as wordnet @xcite . in many cases, classifications within word classes allow researchers to group words into smaller cohesive groups to allow for use as predictors in modelling. rather than using thousands individual lexemes as predictors, one can use a word's class to generalize over the semantic features of individual lexemes to allow for significantly more statistical power. while extensive ontologies of word classifications exist for majority languages like english @xcite , german @xcite , and chinese @xcite , minority languages, especially lesser resourced languages in north america generally do not boast such resources. 1 where such ontologies do exist, for ex1 there is one attempt at semantically classifying nêhiyawêwin through automatic means found in @xcite . this work makes use of similar techniques as desccribed in this paper, differing mainly in its mapping of nêhiyawêwin words onto wordnet classes. ## background we begun word classification in preparation for modelling a morpho-syntactic alternation in nêhiyawêwin verbs. one hypothesis we developed for this alternation, based on @xcite , is that the semantic classes of the verbs themselves as well as their nominal arguments would inform the verbal alternation. due to constraints of time, we investigated methods to automatically classify both verbs and nouns in nêhiyawêwin. although statistical modelling remains the immediate motivator for the authors, semantic/thematic classifications have a wide range of benefits for language learners and revitalization, particularly in online lexicographic resources, where one may want to view all words to do with a theme, rather than simply finding translations of single english words. in creating a framework for automatic semantic classification we make use of word2vec @xcite word embeddings. word embeddings are words represented by n-dimensional vectors. these vectors are ultimately derived from a word's 2 see http://rapidwords.net/ context in some corpus through the word2vec algorithm. unfortunately, the word2vec method is sensitive to corpus size. we initially attempted to create basic word and feature co-occurrence matrices based on a 140,000 token nêhiyawêwin corpus @xcite to create word vectors using principal components analysis, but in the end found the results to be not practically useful. similarly, an attempt at both tf-idf and word2vec using only the nêhiyawêwin dictionary produces mostly ill-formed groupings, though in these cases preprocessing by splitting verbs and nouns was not performed. regardless, the poor performance was most certainly due simply to the paucity of data. although the available corpora are small, nêhiyawêwin does have several english-to-nêhiyawêwin dictionaries, the largest being @xcite . although a bilingual nêhiyawêwin-english dictionary, it is one formed from an indigenous point of view, based on vocabulary from previous dictionaries, some of which have been compiled by nêhiyawêwin communities from their own perspectives, or gleaned from a number of texts collections rather than attempting to find nêhiyawêwin word matches for a pre-defined set of english words. this results in dictionary entries such as sakapwêw: it roasts over a fire (by hanging, with string on stick). definitions such as this take into account the nuanced cultural understanding reflected in the word's morphology. ## methodology to address the issue of corpus size, we attempted to bootstrap our classification scheme with pretrained english vectors in the form of the 3 million word google news corpus, which represents every word with a 300-dimensional vector. 3 we make use of the english definitions (sometimes also referred to as glosses) provided in @xcite and fit to each word its respective google news corpus vector. this dictionary makes use of lemmas as headwords, and contains 21,717 entries. the presumption is that the real-world referents (at least in terms of denotation) of english and nêhiyawêwin words are approximately comparable, in particular when taking the entire set of words in an english definition. stop words were removed, and where content words were present in definitions in @xcite but not available in the google news corpus, synonyms were used (one such example might be the word mitêwin, which is unavailable in the corpus and thus would replaced with something like medicine lodge or deleted if a synonym was given in the definition as well). because the google news corpus is based in american spelling, while @xcite is based in canadian spelling, american forms (e.g. color, gray) were converted into canadian forms (e.g. colour, grey). if such preprocessing is not performed, these words are simply unavailable for clustering, as they lack a matching vector. 4 where a nêhiyawêwin word had more than one word sense, each sense was given a separate entry and the second entry was marked with a unique identifier. finally, where needed, words in the nêhiyawêwin definitions were lemmatized. once every word in @xcite definitions matched an entry in the google news corpus, we associated each word in a nêhiyawêwin definition with its respective google news vector. that is, given a definition such as awâsisihkânis: small doll, the resulting structure would be: because all word-vectors in the google news corpus are of the same dimensionality, we then took the resulting definition and averaged, per dimension, the values of all its constituent word-vectors. this produced a single 300-dimensional vector that acts as a sort of naive sentence vector for each of the english glosses/definitions: @xcite mention this sort of naive representation and suggests the use of phrase vectors instead of word vectors to address the representation of non-compositional idioms; however, given the way wolvengrey (2001)'s definitions are written (e.g. with few idiomatic or metaphorical constructions), and for reasons of computational simplicity, we opted to use the above naive implementation in this paper. after creating the sentence (or english definition) vectors, we proceeded to cluster definitions with similar vectors together. to achieve this, we created a euclidean distance matrix from the sentence vectors and made use of the hclust package in r (r core team, 2017) to preform hierarchical agglomerative clustering using the ward method (based on the experience of @xcite in using the method to produce multiple levels of smaller, spherical clusters). this form of clustering is essentially a bottom-up approach where groupings are made by starting with individual labels with the shortest distance, then iteratively at a higher level making use of the clusters that result from the previous step or remaining individual levels; this second step is repeated until there is a single cluster containing all labels. this method of clustering creates a cluster tree that can be cut at any specified level after the analysis has been completed to select different numbers of clusters, allowing researchers some degree of flexibility without needing to rerun the clustering. this method is very similar to what has been done by both @xcite , @xcite , and @xcite . the choice of what number of clusters was made based on an evaluation of the effectiveness of the clusters, based on an impressionistic overview by the authors. for our purposes, we focused on the semantic classification of nêhiyawêwin nouns and verbs. nêhiyawêwin verbs are naturally morphosemantically divided into four separate classes: intransitive verbs with a single inanimate argument (vii), intransitive verbs with a single animate argument (vai), transitive verbs with an animate actor foot_2 and an inanimate goal (vti), and verbs with animate actors and goal (vta). for verbs, clustering took place within each of these proto-classes. among the viis, 10 classes proved optimal, vais had 25 classes, vtis with 15 classes, and vtas with 20 classes. the choice to preprocess verbs into these four classes was as not doing so resulted in a clus-tering pattern that focused mainly on the difference between transitivity and the animacy of arguments. any more or fewer classes and hac clusters were far less cohesive with obvious semantic units being dispersed among many classes or split into multiple classes with no obvious differentiation. similarly, verbs were split from nouns in this process because definitions in @xcite vary significantly between verbs and nouns. nouns are naturally divided into two main classes in nêhiyawêwin: animate and inanimate. 6 for our purposes we divide these further within each class between independent (i.e. alienable) and dependent (i.e. inalienable) nouns to create four main classes: independent animate nouns (na), dependent animate nouns (nda), independent inanimate nouns (ni), and dependent inanimate nouns (ndi). the reason for this further division is due to the morphosemantic differences between independent and dependent nouns in nêhiyawêwin. while independent nouns can stand on their own and represent a variety of entities, they are semantically and morphologically dependent on some possessor. we opted to pre-split ndis and ndas into their own classes, so as not to have the clustering focus on alienablity as the most major difference. 7 ## results in all cases, clusters produced by this procedure needed some amount of post-processing. for nouns, this post-processing was minimal and mostly took the form of adjustments to the produced clusters: moving some items from one class to another, splitting a class that had clear semantic divisions, etc. for the verbs, this processing was often more complex, especially for the vai and vta classes. items were determined to not belong in one class or another based on it's central meaning of the action or entity. if the majority of group members pertained to smoking (a cigarette), a word describing smokiing meat (as food preparation) would not be placed in this group, as the essence of the action and its intended purpose diverged significantly from the rest of the group. although most clusters produced somewhat cohesive semantic units, the largest clusters for the vai and vta classes acted as, essentially, catch-all clusters. although computationally they seemed to have similar vector semantics, the relationship between items was not obvious to the human eye. postprocessing for these clusters took substantial amounts of time and essentially comprised of using more cohesive clusters as a scaffold to fit words from these catch-all clusters into. in most cases, this resulted in slightly more clusters after postprocessing, though for vais this number was significantly higher, and for the ndis it was slightly lower. postprocessing grouped together words based on the most core semantic property of the word class: nouns were generally grouped based on the entity or state they represented, and verbs were generally grouped based on the most basic form action they represented. this is why, for example, ai-cover includes words for both covering and uncovering. in some cases a final class may seem like something that could be subsumed under another (e.g. ai-pray or ai-cooking might be understood as subsets of ai-action); however, in these cases, the subsumed class was judged to be sufficiently separate (e.g. cooking is an action of transforming resources into food for the purposes of nourishment, while verbs of ai-action are more manipulative, direct actions done for their own sake. further, the automatic classification already grouped words in these ways, further justifying their separation. finally, some grouping seem more morphosyntactic (e.g. ai-reflexive), though we argue that reflexivity, performing an action inwards, is in and of itself a salient semantic feature, and the inclusion of these terms into @xcite indicates their lexicalization and distinction from the non-reflexive forms. the actual quality of clustering varied form class to class. in general, nouns resulted in much more cohesive clusters out-of-the-box and required far less postprocessing. for example, nearly all of the hac class n i 14 items referred to parts of human bodies (and those that did not fit this description were terms clearly related to body parts like aspatâskwahpisowin, 'back rest'), n i 13 was made up of trapping/hunting words and words for nests/animals. the na classes produced through hac were similarly straightforward: n i 9 was made up of words for trees, poles, sticks, and plants; n i 8 was made up entirely of words form beasts of burden, carts, wheels, etc.; while much of n a 3 and n a 7 , and nearly all of n a 2 referred to other animals. once manually postprocessed, the na lexemes settled into 8 classes: na-persons, na-beast-of-burden, na-food, na-celestial, na-body-part, na-religion, na-money/count, and na-shield. the ndi and nda classes required almost no postprocessing: n da 1 and n da 3 were each made up of various family and non-family based relationships, while n da 2 was made up of words for body parts and clothing. the resulting classes for these were: nda-relations, nda-body, and nda-clothing. the ndi lexemes basically took two classes: the vast majority of ndi forms referred to bodies and body parts while two lexemes referred to the concept of a house, resulting in only two classes: ndi-body, and ndi-house. verbs, on the other hand, required quite a deal more postprocessing. viis showed the best clustering results without postprocessing. for example, v ii 6 was entirely made up of taste/smell lexemes, v ii 7 was almost entirely weather-related, v ii 8 contained verbs that only take plural subjects, v ii 9 had only lexemes referring to sound and sight, and v ii 1 0 had only nominal-like verbs (e.g. mîsiyâpiskâw '(it is) rust(y)'). despite these well formed clusters, v ii 1 through v ii 5 were less cohesive and required manual clustering. in the end, distinct classes were identified: ii-natural-land, ii-weather-time, ii-sensory-attitude, ii-plural, ii-move, ii-time, and ii-named. 8 although postprocessing was required, this was not too substantial in scope or time. the vais required significantly more work. some classes were well defined, such as v ai 23 whose members all described some sort of flight, but v ai 12 contains verbs of expectoration, singing, dancing, and even 8 the concepts of weather and time were combined here as many of the nêhiyawêwin words for specific times also contain some concept of weather (e.g. the term for 'day' is kîsikâw, clearly related to the word for 'sky/heavens', kîsik; similarly, the word for 'night' is tipiskâw, which is the same word used for the night sky. additionally, words like pipon, 'winter' and sîkwan 'spring' are obviously related to both time and weather. ## discussion in general, the best clustering was seen in classes with fewer items. the vai and ni lexemes required the most postprocessing, with each having roughly double the number of items as the next most numerous verb/noun class. verb classes in general seemed to produce less cohesive classes through hac. although the exact cause of this discrepancy in unknown, it could perhaps be due to the way words are defined in @xcite . in this dictionary, verb definitions almost always contain more words than noun definitions. almost every single verb definition will have at least two words, owing to the fact that nêhiyawêwin verbs are defined by an inflected lexeme. this means that if one looks up a word like walk, it would appear as: pimohtêw: s/he walks, s/he walks along; s/he goes along. meanwhile, nouns tend to have shorter definitions. the definition for the act of walking, a nominalized form of the verb for walk, is written as: pimohtêwin: walk, stroll; sidewalk. this difference is exacerbated by the fact that definitions are often translated fairly literally. something like pêyakwêyimisow might be translated simply as 's/he is selfish,' but contains morphemes meaning one, think, reflexive, and s/he. a gloss of this word is seen in (1). rather than simply defining the word as 's/he is selfish,' @xcite has opted to provide a more nuanced definition: pêyakwêyimisow: s/he thinks only of him/herself, s/he is selfish, s/he is self-centered. also worth noting is the relative distributions of each of the postprocessed classes mentioned above. perhaps unsurprisingly, the distribution of lexemes into different classes followed a sort of zipfian distribution. the na-person and na-other-animals accounted for the vast majority of noun lexemes for animate nouns. just under half of all ni lexemes were nominalized verbs, and roughly a quarter were smaller objectlike items (e.g. tools, dishes, etc.). the ndas were almost entirely dominated by words for family, while all but three ndis were body part lexemes. some categories such as ni-scent, ni-days, and na-shield have extremely low membership counts, but were substantially different from other categories that they were not grouped into another class. most interestingly, there appeared to be three ni lexemes that referred to persons, something usually reserved for nas only. these lexemes were okitahamâkêw 'one who forbids,' owiyasiwêwikimâw 'magistrate,' and mihkokwayawêw 'red neck.' in all three cases, the lexemes seem to be deverbal nouns (from kitahamâkêw 's/he forbids,' wiyasiwêw 's/he makes laws,' and mihkokwayawêw 's/he has a red neck.' verbs showed a similar distribution. overall, verb forms, especially the most numerous classes of vai and vta, required a large degree of manual postprocessing. because this approach assumes no underlying ontology, but rather attempts to work bottom-up (cf. @xcite ), the time taken to postprocess vai and vta classes is likely not too far from what it would take to manually classify these words based off a prebuilt ontology; however, the appeal of a bottom-up classification should not be overlooked, however. as an example, many ontologies place concepts like thinking, and being happy into separate classes; however, in our classification these words were combined into a single class of cognition. this is done because emotion words like môcikêyihtam, 's/he is happy (because of something)' (in addition to being verbs and not adjectives) contain a morpheme, {-êyi-}, meaning 'thought.' for these reasons, such emotion words are often translated as having to do specifically with thought and cognition: môcikêyihtam, 's/he thinks happily (because of something).' @xcite uses these sorts of definitions, and so unsurprisingly the majority of such emotion words were classified in the proposed scheme together with words of thought. where this was not the case, manual postprocessing from a bottom-up approach allows us to maintain the cultural understanding of emotions as directly related to cognition. furthermore, from the experiential standpoint of one of the authors, the use of semi-automatic clustering produces a kick-start that greatly aids to the starting of a semantic classification task, especially for non-native speakers. ## conclusion this paper describes an attempt at, for the first time, semi-automatically classifying nêhiyawêwin verbs and nouns. the process used in this paper is easily applied to any language that makes use of a bilingual dictionary with definitions written in a more resourced language. resulting clusters of nêhiyawêwin words are freely available on the online. although the technique worked quite well with nouns, which required very little manual adjustment, verbs required more directed attention. despite this, the technique presented in this paper offers a bottom-up, data-driven approach that takes the language on its own terms, without resorting to ontologies created primarily for other languages. if, however, one wishes to use a pre-defined ontology, the basis for this work (representing word definitions using pre-trained english word vectors) could be used in conjunction with existing ontologies to expedite the classification process. for example, @xcite compare the naive definition vectors for @xcite with the same for the english wordnet word senses; word senses