The More Detail, the Better? – Investigating the Effects of Semantic Ontology Specificity on Vector Semantic Classification with a P lains C ree / nêhiyawêwin Dictionary (2021)

Abstract: One problem in the task of automatic semantic classification is the problem of determining the level on which to group lexical items. This is often accomplished using pre-made, hierarchical semantic ontologies. The following investigation explores the computational assignment of semantic classifications on the contents of a dictionary of nêhiyawêwin / Plains Cree (ISO: crk, Algonquian, Western Canada and United States), using a semantic vector space model, and following two semantic ontologies, WordNet and SIL’s Rapid Words, and compares how these computational results compare to manual classifications with the same two ontologies.

## introduction despite the benefits and usages of semantically organised lexical resources such as dictionaries, ranging from uses as pedagogical tools @xcite to aids for machine translation @xcite , fully elaborated semantic dictionaries remain less common than those assembled with more routine alphabetical ordering systems. aside from the reason of convention, one prominent dissuasive factor towards creating semantic dictionaries is the sheer amount of effort necessary to create them if their lexical content is not already organised along some ontologically principled semantic lines; the manual semantic classification of even relatively small dictionaries of this nature frequently takes months. this may be a prohibitively costly procedure in situations where resources for linguistic analysis, be they temporal or economic, are limited. thus, a dilemma faced by the prospective compiler of a semantic dictionary is that of selecting an ontology, that is, a principled system of semantic categories, typically (but not universally) arranged hierarchically, into which lexical items may be grouped. the following investigation aims to address potential remedies to both of these limitations, with vector semantics as a first-pass alternative to manual semantic classification, and with princeton wordnet and sil's rapid words as two practical contenders for pre-existing semantic ontologies. in practice, these methods are to be demonstrated on an existing bilingual dictionary of plains cree (nêhiyawêwin), with results compared against human-made semantic classifications in both ontologies. ## vector semantics the first, and perhaps most daunting, obstacle in the process of creating a semantic dictionary (or indeed any semantically organised lexical resource) is the issue of time; even with a welldefined ontology and ample resources, manual semantic classification is a lengthy and expensive process, with teams of linguists and native speakers often requiring years to produce fully annotated semantic dictionaries @xcite . even with a more reduced ontology, semantically classifying an already existing full dictionary by hand takes months, and requires a thorough understanding of the chosen ontology @xcite . although the process of manually assigning semantic categories or correspondences to dictionary entries is generally not an exceptionally difficult task for a human annotator @xcite , the length of dictionaries, and the existence of highly polysemous lexical items, both complicate and lengthen the process of manual classification. as such, the mechanisation of the process of semantic classification assignment (or semantic annotation) appears to be one of the most direct routes to increasing overall efficiency with respect to time and resources, and to that end, the method of vector semantic classification is an alluring and well-attested alternative @xcite . in short, vector semantic classification is a method of computationally determining the semantic similarity between any two given lexical units based on commonalities in the usage contexts of those units in large corpora. this is accomplished by representing the meaning of a lexical unit (primarily a word) as a vector in multidimensional space, which is based on the co-occurrences of this lexical unit with other lexical units in its context, followed by a reduction of dimensionality using some heuristic to result in a compact, dense vector space (typically with several hundred dimensions). since this vector space is based on common contextual features, one may compare the multidimensional vector of one word with that of another, calculating their cosine distance to determine similarity; the closer this value is to 1, the more similar the average contexts of those two words are, and thus the more similar those words are semantically. in this way, the model functions largely on the assumptions of the distributional hypothesis as put forth by firth and harris in the 1950s @xcite @xcite , that semantic similarity begets distributional similarity, and vice versa. vector generation is not monolithic, and various tools using various methods exist in common use, including frequency-weighted techniques such as tf-idf and latent semantic analysis. in the context of this investigation, word2vec, a tool which makes use of predictionbased models rather than concurrence matrices to generate clusterable vector sets, has been used to generate all vectors; this decision was motivated chiefly by word2vec being readily available, easily applicable without lengthy training, and being able to leverage extensive, pre-existing pretraining on large english corpora, all advantages which largely offset the primary disadvantage of word2vec, being that it is a purely word-level vector generation tool, lacking the ability to model polysemy and contextual variances, a shortcoming which may possibly be addressed by using a sentence-level model such as bert (see section 5 and 7). the vector method is not a novelty, and its utility as a practical method of semantic classification assignment has been demonstrated on numerous occasions @xcite . however, useful as the method may be, in order to use vector semantics to classify entries in a dictionary, one requires a principled structure of semantic relationships into which to classify them. to this end, pre-existing semantic ontologies are a widespread and convenient solution. ## semantic ontologies although it is possible to computationally generate sets of semantic hierarchies, the results of such attempts generally indicate that humanmade, preset ontologies are preferable @xcite . many such premade ontologies exist, serving a wide variety of different classificational purposes; however, we will compare here only two, being a slightly modified version of the princeton wordnet and sil's rapid word collection method, both popular, general-purpose ontologies intended to cover the breadth of most semantic reference in a largely language-neutral fashion. a visual representation of the structures of both is detailed in figure 1 (see next page). ## plains cree / nêhiyawêwin plains cree (nêhiyawêwin) is an indigenous language of the algonquian family, spoken by ~30 000 throughout saskatchewan, alberta, and northern montana. although slightly less critically endangered in comparison with other canadian indigenous languages, the majority of speakers are elderly, and intergenerational transmission remains low. various revitalisation efforts have been undertaken in cree communities, including bilingual education and the creation of online lexical resources @xcite ; however, digital resources for cree remain limited overall. like most algonquian languages, plains cree is highly polysynthetic, with extensive morphology, particularly on verbs, which make up the bulk of the lexicon (e.g., @xcite . the lexical resource used for this investigation was a fully digitised copy of the database underlying nêhiyawêwin: itwêwina/cree: words (cw), a continually-updated bilingual cree-english dictionary compiled by arok wolvengrey across the late 20th and early 21st centuries @xcite . consisting currently of 21,347 words with morphological notes and pos-tagging, cw is the most extensive modern dictionary of plains cree, and its contents may be accessed through the university of alberta's online cree dictionary, itwêwina. ## method word vectors were obtained for every cree entry in cw using word2vec, a popular off-the-shelf vector generation tool @xcite . we used the pretrained google news corpus, which contains 3 million word embeddings trained on 3 billion tokens. cree word (or rather, dictionary entry) vectors were obtained as a simple, dimension-wise average of the individual english word vectors as extracted from the english definition phrases/sentences (glosses) of their respective entries, rather than the cree words themselves, as existing cree corpora (arppe 2020) are too small for meaningful dimensional vectors to be obtained @xcite . for example, the vector for the cree noun mahkahk (glossed in cw as 'tub, barrel; box') would be generated by averaging the vectors for the english words 'tub', 'barrel', and 'box', treated as a bag of words. similarly, for the cree verb nâtwânam (glossed as 's/he breaks s.t. apart; s/he breaks s.t. off by hand'), the vector would be derived from the average of the vectors for 's/he', 'breaks', 's.t.', 'apart', 'off', and 'hand'. cw noun glosses tend to be either single words or extremely curt noun phrases, and verb glosses are usually brief, utilitarian verb phrases, with no grammatical or derivational information included in the gloss itself; this fact is a further justification for using a word-level vector generation tool such as word2vec rather than a sentence-level tool like bert, as the pieces of linguistic information on which the cw vectors are based are typically either non-sentential or highly simplistic and formulaic, seemingly making the contextsensitivity of tools such as bert much less useful. the google news corpus and word2vec were similarly used to generate the vectors for the wordnet synsets, using the head words and synset description (definitions and example sentences) as context to create the vectors, and the head word(s) of the synset as labels @xcite . for example, the vector for the synset (n) barrel#2 (glossed as "barrel, cask (a cylindrical container that holds liquids)") would be the average of the vectors for the words 'barrel', 'cask', 'cylindrical', 'container', 'holds', and 'liquids'. the vectors for rapid words were created using the semantic domain levels as labels, with all example words and elicitation questions contained therein as context. for example, for the word 'barrel' in rapid words, which is contained in the semantic domain 6.7.7 container, the vector would be the average of the vectors for all of the english words in each elicitation question (i.e. "what words refer to a container", "what words refer to what is in a container", etc.), as well as all of the words listed as possible examples (such as 'container', 'vessel', 'bowl', 'pot', 'contents' etc.). these sets of vectors were then compared against the cw vectors using cosine distance, and for every cw entry, two lists were created. for each entry on the first list (the wordnet list), all wordnet synsets were listed, ordered by cosine similarity to that entry. on the second list (the four-level rapid words list), for each cw entry, all rapid words semantic domains at the fourth tier of the hierarchy were ordered by similarity. to provide an example for the second list, even if the manually-selected rw domain for the cree word acihkos ('caribou calf, fawn') was 1.6.1.1.3 hoofed animal, because, on this list, the vector method would only have access to the fourth hierarchy level, the ideal, most 'human-like' vector classification would instead be 1.6.1.1 mammal, as this domain is at the fourth level of the hierarchy and is identical to the manual classification up to the fourth level (1.6.1.1). the reasoning behind limiting the rw domains to the fourth level of the hierarchy in the vector method was threefold; firstly, tests in which the vector method was allowed to select domain classifications from any of the five levels returned notably poorer results than those which limited the choice to only one tier. (see in total, applying the vector semantic method to this end requires access to a fully digitised copy of the target dictionary (with entries and their glosses clearly delineated), access to wordnet, rapid words, and word2vec (all of which are freely available online), and a computer capable of both generating vectors for the dictionary entries and comparing those vectors with the pre-existing ontology vectors. to this end, a 2core laptop with 8gb ram is able to complete the cosine comparisons for the ~16k cw entries with the ~117k wordnet synsets in 4-5 days, and the same entries with the rapid words domains in no more than one and half days. on a highly parallelised computing cluster, such as computecanada's cedar (using 64 cores, each having 4-8gb ram), performing all of the cosine comparisons takes less than 90 minutes. the computational cost of the actual vector cosine comparisons is fairly negligible, and the lengthy runtime of this operation on more basic machines is likely due to the inefficiency of retrieving each vector from large matrices. to assess their quality, these vector classifications were compared against a gold standard of manual classifications for each entry in cw. these manual classifications were done following both wordnet and rapid words, with one or several synsets or rw elicitation questions assigned to each cw entry based on the meaning of the cree word. for the wordnet classifications, the part of speech of the english wn synset was ignored; for example, the manual classification of the cree verb mihkwâw ("it is red") was given in wordnet as the adjectival synset (adj) red#1. for rapid words classifications, given that rw elicitation questions do not have hard-coded parts of speech, whichever domain-internal elicitation question(s) were most semantically related to the target cree word were used, regardless of their domain level in the hierarchy. for example, for mihkwâw, the question 8. ## comparison of wordnet and rapid words results statistics: overall, although the results of both ontologies are comparable, semantic classifications using rapid words appear noticeably more human-like than those with wordnet, with 'human-like' here referring to how high the rank of the manual classification(s) is among the total vector classifications for each entry on average. for the vector classifications of cree nouns, the median position of the top manual classification was 2 for four-level rw domains (with 983 possible classes) and 36.5 when the vector method was allowed to choose from any level of domain (with 1789 possible classes). for cree verbs, the median position of the top manual classification was 15 for the fourlevel domains and 222 for any-level domains. in cases where there was more than one manual rw classification, the median position of the median of the multiple classes for cw nouns was 3 for four-levels, and for cw verbs, the median of the medians of multiple classes was 36 for four-levels. for the wordnet vector classifications, the median computationally selected position for the top manual classification was 15th for cree nouns and 333rd for verbs, and the median position of the manual classifications when there were several was 30.5 for the nouns and 1045 for the verbs. from this, it is clear that vector classifications with rapid words domains are, on average, much more human-like than their wordnet counterparts, being up to 22 times more accurate in the case of cree verbs, and that limiting the vector methods' potential selections to a single, moderately specific rw hierarchy level provides much more human-like results than allowing it to select from all domains at all levels. however, it is prudent to keep in mind that even with all of its domains, rapid words still has far fewer potential correspondences than wordnet (1789 total rw domains (with 983 four-levels) compared to 117,659 wn synsets), and in relative terms, relevant manual classifications occur on average in a higher position proportionate to the total number of possible choices in wn vector classifications than in those with rw; with four-level rw vector classifications, the median position of the top manual classification is in the top 0.203% for the nouns (2nd out of 983) and in the top 1.53% for the verbs (15th out of 983), compared with the top 0.0127% (15th out of 117659) and 0.283% (333rd out of 117659) respectively for wn. in general, the reduced specificity of rapid words, by virtue of both its inherently less detailed structure and its restriction here to a single hierarchical level of specificity, seemed to lend itself well to resolving a particular ill in the vector method, being its propensity to preferentially assign overly specific classifications to the high ranks of 'umbrellaterms', rather than the more appropriate general vocabulary. in this sense, rapid words semantic domains often represent concepts several steps higher in the hypernymic hierarchy than their wordnet equivalents. for example, with the wordnet classifications, the top classification for môhkomân (glossed as 'knife') was (n) knife blade#1, and the top 15 classifications consisted almost entirely of either specific types of knives or parts of knives, with the more appropriate generic term (n) knife#1 not appearing until 18th place. by contrast, in rapid words, in which such specific classifications are by nature impossible at the domain-level, the top ranking classifications are more appropriately general, with the any-level list, for example, having the appropriate 6.7.1 cutting tool as the top classification, and the similarly relevant 4.8.3.7 weapon, shoot in second place. the 'regift' problem: the in-built simplicity of rapid words also seems to have partially remedied, if not entirely solved, the so-called 'regift problem' which was prevalent in wordnet classifications; we discuss this problem in more detail in @xcite , but simply put, a small number of extremely low frequency wordnet synsets occurred disproportionately frequently in the high-ranking classifications of target cree words. the problem was so named due to such one low-frequency synset, (v) regift#1, being present in the top 1000 computational classifications of 65% of all cree verbs, despite almost always being entirely unrelated semantically to the target cree word. (v) regift#1 is not the only wordnet entry to exhibit this behaviour, and other words, such as (n) rumpelstiltskin#1 occurred in as many as 72% of the top 1000 vector classifications of cree verbs; other common regift words include (n) dido#1, (n) gumption#1, and (n) dingbat#1. as a rule, these 'regift' words were both low frequency in corpora and highly specific, often being proper nouns, however, there did not appear to be any pattern in the formatting or content of these entries' glosses. the rapid words vector classifications also exhibited this problem to an extent; for example, subdomains of the domain 4.1.9 kinship occurred in the top 1000 vector classifications of cw nouns and verbs an average of ~12 times, and appeared in the top 10 classifications 33.9% and 35.7% of the time for cw nouns and verbs respectively. however, as a whole, the regift problem was markedly less notable with rw classifications of both types than with wn classifications, with both fewer different regift words (or domains) and fewer occurrences of these words/domains overall. this broadly supported our initial theory that the 'regift' problem was at least partially caused by the excessive degree of specificity in wordnet synsets overwhelming the vector method and providing it with a large number of potential classification choices with poorly defined vectors (due the low frequency of 'regift' words in the google news corpus) which muddy the optimal, human-like choices. by contrast, since rapid words generally lacks highly specific vocabulary and is instead structured by more generic categories or 'domains', fewer of these low-frequency words are factored into the rapid words vectors, and these vectors are thus, in general, based on higher frequency, more contextually attested vocabulary, and are therefore (in theory) more accurate. in general, the lack of highly specific vocabulary in rapid words seems to contribute both to diminishing the number of semanticallyrelated, but overly specific correspondences in the computational classifications, as well as to reducing the prominence of semanticallyunrelated, overly specific 'regift' words (or in the case of rapid words, domains). one potential method to imitate this degree of simplicity in wordnet could involve using the hypernymic synsets of the current wordnet correspondences as labels, in essence, shifting all classifications one or more levels up in the wordnet hierarchy. this would appear to at least partially resolve the over-specificity issue (although it would do nothing to reduce the number of outright irrelevant classifications), despite incurring an obvious cost in terms of semantic richness. vector content: broadly speaking, the improved results with rapid words seem to be due not only to its simpler hierarchical structure and reduced level of specificity, but also due to its domain internal structure, in which domains generally include fewer irrelevant content words than wordnet synsets do. wordnet synsets frequently include example sentences in their glosses; although useful for human clarification, these inclusions inevitably lead to large amounts of semantically unrelated vocabulary influencing the respective synset vectors. as an example, the gloss for the synset (v) drive#2 (defined as "travel or be transported in a vehicle") includes the example sentences "we drove to the university every morning" and "they motored to london for the theater". as such, the semantically irrelevant words "university", "morning", "london", and "theater" are all factored equally into the vector for (v) drive#2 as the semantically relevant terms "drive", "motor", "vehicle". while the inclusion of these less relevant words may more accurately simulate natural linguistic contexts, the otherwise terse nature of wordnet synset glosses means that they introduce a potentially significant amount of distracting information, possibly skewing synset vectors towards the contexts of their irrelevant example sentence vocabulary rather than their relevant gloss vocabulary. by contrast, with the exception of infrequent descriptions of lexicalisation patterns, rapid words domains and questions contain only semantically related vocabulary, lessening potential 'distractions' for their vectors. ## conclusion the vector semantic method is a significantly faster and cheaper alternative to manual semantic annotation for tasks of semantic classification. however, the method is not yet capable of producing reliably human-like results across target-language parts of speech, and struggles to match natural levels of semantic specificity. to this end, using a consistent hierarchical level of a simpler, more generalistic semantic ontology, such as rapid words, seems to make vector semantic classifications appear more human-like, as restricting the breadth of choices available to the method as labels for correspondences seems to both reduce the number of potentially unrelated classifications and make the remaining classifications general enough that a less precise vector is necessary to generate a human-like correspondence. future avenues of research into dictionary vector semantics include the use of sentence-based vector generation tools such as bert which can more accurately model polysemy, although it should be kept in mind that even a model like bert cannot be expected to generate humanlike results for dictionary glosses if those glosses are non-sentential or otherwise overly brief. it may also prove productive to experiment with the further modification of existing semantic ontologies such as wordnet and rapid words (such as reducing the specificity of wn by using only synsets one or several levels higher in the hypernym hierarchy as correspondences), with one of the ultimate goals of this being the integration of the results of automatic vector classifications into online dictionaries in a form which is easily navigable and understandable to an untrained user.