Deciphering and Characterizing Out-of-Vocabulary Words for Morphologically Rich Languages (2022)

Abstract: This paper presents a detailed foundational empirical case study of the nature of out-of-vocabulary words encountered in modern text in a moderate-resource language such as Bulgarian, and a multi-faceted distributional analysis of the underlying word-formation processes that can aid in their compositional translation, tagging, parsing, language modeling, and other NLP tasks. Given that out-of-vocabulary (OOV) words generally present a key open challenge to NLP and machine translation systems, especially toward the lower limit of resource availability, there are useful practical insights, as well as corpus-linguistic insights, from both a detailed manual and automatic taxonomic analysis of the types, multidimensional properties, and processing potential for multiple representative OOV data samples.

## introduction even in a familiar language, unfamiliar words cause trouble for machine processing or comprehension of text. any dictionary is innately incomplete in its coverage, unable to provide novel coinages and exhaustive forms. without finding the word in a dictionary, the surface form and context afford only weak evidence for its meaning. the situation is even worse for languages other than english, especially morphologically rich languages, for two reasons: first, there is usually less annotated data available; and second, the coverage of such data is much lower due to the high number of different forms. moreover, many words not found in even a small training corpus are in fact related to quite common words by processes such as inflection, derivation, compounding, or misspelling. in the work described herein, we therefore concentrate on the problem of characterizing unknown words in terms of the processes by which they arise, and especially the relative frequencies at which such processes occur. this informs us of the distribution of out-of-vocabulary (oov) words with respect to different dictionary sources. to do so, we conduct a study on a sample of two bulgarian language corpora annotated by a native speaker. rather than treat oov tokens as a monolithic and undifferentiated problem, we progressively apply multi-faceted linguistic analyses to these corpora, characterizing both the words that these analyses explain and words yet to be explained, which we shall call the residual vocabulary. our methods are a mixture of the vintage and the vogue: specialized edit distances, composition of finite-state transducers, a noisy channel model for language identification fitted with empirical bayes, and neural network-based part of speech taggers. collectively, our processes accurately explain more than two in three (69%) unknown bulgarian words in a held-out set according to whether they are proper names, inflections, derivations, compounds, foreign words, or misspellings (as illustrated in both figures 1 and 3 , discussed in more depth in §5). we release our native speaker-annotated lexicon, intermediate analyses, and software at www.github.com/gbotev1/bg . ## motivation and related work previously unseen words often represent a significant portion of the vocabulary, due in part to the zipfian nature of language. figure 2 illustrates this for various vocabulary sizes. note that for the bulgarian training data, the oov rate remains high for both tokens (corpus instances of words) and types (vocabulary words) as found in a held-out set of 20,000 tokens. the rates are computed ignoring capitalization, punctuation, and numbers, so that these do not skew the count of unknown words. the frontier of natural language processing as an engineering discipline has adopted informationtheoretic subword tokenization @xcite to constrain the vocabulary size and provide a representation of all words, preventing any words from being out-of-vocabulary. because such models dominate so much of the field of nlp, one may ask what value there is in analyzing the residual vocabulary today. foremost, there is the corpus-linguistic and lexicographic value of characterizing this aspect of text: it is instructive about the patterns of lacunae in dictionaries or word formation processes in particular domains such as color @xcite . there are engineering applications as well. in languages with insufficient data for training large neural machine translation systems @xcite (or even for fine-tuning to new languages; see @xcite , statistical methods dominate @xcite . the methods described in this paper are of value for populating the phrase tables of statistical mt models beyond what can be done with existing bilingual dictionaries, as in @xcite who address spelling variants by online retokenization, or de @xcite who aims to reduce morphological variety. moreover, entity linking and the use of gazetteers in named entity recognition both benefit from exact word representations. we underscore the fact that resource-poor languages are the norm, not the exception. out of the world's roughly 7,000 languages, only 216 have more than 1,000 gloss definitions in wiktionary, a popular multilingual dictionary. 1 for the remaining ≈6,800 data-poor languages, unknown words are not only neologisms and proper names; items of the core vocabulary are regularly absent from bilingual dictionaries or small but extant corpora. lexicon stratification, the splitting of the lexicon based on words' origin and degree of assimilation into the language @xcite , is a powerful technique to hone the processing of oov words @xcite . the four identified levels are the core vocabulary, the partially assimilated words, the fully assimilated words, and peripheral lexemes. this paper proffers empirical relative frequencies of these degrees and showcases a series of models that roughly correspond to these degrees. ## the bulgarian language bulgarian is a member of the south slavic branch of the indo-european family, written in the cyrillic script. as a member of the balkan sprachbund, its lexis 2 and grammar have been influenced by areal effects. it thus displays several traits uncharacteristic of other slavic languages (except macedonian) which affect the apparent size of the lexicon: a postposed definite article marked for gender, the use of clitic pronouns, a lack of verbal infinitive, and limited case declension @xcite . as a case study, bulgarian is useful because it uses several widespread strategies for word formation. its rich verbal morphology yields over 50 forms per verb lexeme. derivational affixation and compounding are prevalent processes. in fact, derivation for nouns is both productive and regular @xcite . finally, a significant fraction of the bulgarian lexis is borrowed from russian, greek, or other languages, especially in technical contexts. these properties have made bulgarian a focus for linguistic examination and an area of interest in natural language processing. for example, @xcite devise a rich morphological tag set for bulgarian verbs. @xcite build a richly anno-1 https://en.wiktionary.org/wiki/wiktionary: 2 we distinguish between the lexis, i.e., the set of all words in a language, and the lexicon, i.e., the set of all lexemes. in contrast to these works, which offer an engineering approach to modeling bulgarian, our work relies on computational tools insofar as they help characterize properties of bulgarian text. namely, we explore the relative frequency of various processes by which words-especially unknown words-arise in naturally occurring bulgarian text. ## data for our study, we need a large and representative corpus of bulgarian text. we use the entirety of bulgarian wikipedia, which contains 1.3 million word types and 73.6 million word tokens (type-token ratio 0.018) after tokenization; a random sample of these is summarized in figure 1 . we also must define the set of known words. we merge three broad-coverage bilingual dictionaries: languagenet. 364,327 entries covering 155,703 unique english words. foot_0 panlex. 180,023 entries covering 70,986 unique english words @xcite . wiktionary. 51,537 entries covering 22,856 unique english words. we extract these with yawipa @xcite . in aggregate, these cover 165,644 unique engish words, with a median number of translations 1 and mean approximately 2.360. 4 to identify the residual vocabulary, we remove from bulgarian wikipedia all entries in our dictionaries as well as non-alphabetic entries, leaving 371,475 novel words-about one in every 200 tokens. 5 a random sample of 100 is summarized in figure 3 . the complete word lists and analyses are given in appendix a. all annotations were validated or adjudicated by a non-author professional bulgarian translator who is a native speaker. what becomes immediately apparent is that the residual vocabulary after dictionary entries are removed comes from five major groups: morphological variants of other words, foreign words, misspellings, compound words, and proper names like place names or people. we devise computational approaches to tackle these five major categories. because we discovered an abundance of russian words interspersed in the bulgarian text, we also extract russian-english bilingual entries from the same three dictionaries. we find 232,094 entries in wiktionary covering 75,284 unique english words; 2,379,638 entries in panlex covering 859,279 unique english words, and 1,633,709 unique entries in languagenet covering 879,438 unique english words. their union covers 932,738 unique english words, with potentially multiple russian candidate translations. the median number of translations was one, and the mean was 1.888. preprocessing to identify bulgarian tokens in context, we first preprocess the text using the rulebased spacy sentence segmenter and tokenizer @xcite . we found this to be faster than the stanza neural tokenizer @xcite . we use stanza for pos tagging, though its poor performance motivates the 'vintage' models we introduce below. in preliminary experiments, we also explored treetagger (schmid, 1994, 1999). 6 6 several avenues exist to improve part-of-speech tagging with minimal available resources. the most notable is projecting part-of-speech annotations across unsupervised word alignments into the language of interest, then using these silver annotations to train a new tagger @xcite @xcite @xcite . such methods could either complement a tagger such as stanza trained in the language of interest via classifier combination or we normalize all text to unicode nfkd form to increase coverage. 7 this also allowed us to remove accents, which were predominantly used to mark stress. we subsequently remove tokens with any letter not in the bulgarian alphabet. while this removes a few interesting cases like mp3-файлове 'mp3 files' and 2-то 'the second [thing]', on the whole the eliminations were useful: filtering urls, email addresses, and also less structured non-words. we found the need to preprocess the dictionaries by hyphen flattening. if a dictionary entry begins or ends with a hyphen, indicating that it is a prefix or suffix, we associate it with its non-hyphenated translational counterpart. for instance, the nonsensical english entry 'pra' is linked to the bulgarian transliteration 'пра', and the bulgarian prefix 'пра-' is correctly (and uniquely) associated with the english prefix 'great-'. after flattening, the bulgarian entry 'пра' would have both 'pra' and 'great' listed as candidate translations. this preprocessing both reduces the dictionary's size and is crucial to increasing the impact of the compound analysis ( §5.5). moreover, we define a heuristic to eliminate old bulgarian words, based on a 1945 orthographic reform that forbids word-final 'ь'. inspecting a sample of 50 words captured by this heuristic reveals that while none of the words filtered here were modern bulgarian, 44% were in fact old bulgarian. the remainder were transliterations (the "unassimilated foreign words" of tsvetkov and dyer, 2015) from disparate languages: italian (18%), turkish 8 (16%), kazakh (6%), chinese (6%), albanian (4%), and single exemplars of irish, portuguese, and moldovan. 9 ## modeling and analysis this work by its nature differs from a great deal of the empirical work in natural language processing. the object of its inquiry is language itself, not computational models, and so we do not evaluate in the standard positivist paradigm of comparing scores on standard benchmarks. instead, we build compuannotate the language in the absence of in-language annotations. the modularity of our approach lets us leverage prior tools and research in the language, and components can be upgraded as better models are devised (e.g., @xcite for morphological analysis, @xcite for inferring cognates). moreover, disparate models for a single word formation process can be combined in situ via classifier combination or meta learning. while many of the tools we use are tailored to the bulgarian language, such as hand-crafted derivational rules from a grammar, in principle our approach makes minimal assumptions about the nature of the language. it could easily be adapted to other slavic languages or, given sufficient prior typological information, other written languages writ large. the overall sequence of method application is given in figure 4 . in the following sections, we elaborate on the most telling among these: language identification, then modeling morphology, misspellings, and compounds. ## discussion and conclusion we have investigated the space of unknown lexical items in naturally occurring text. in a case study on bulgarian, a host of analytical models applied sequentially characterize the residual space of outof-vocabulary words. our models identify myriad processes responsible for these unknown words and map from such words to known words via heuristic and probabilistic processes. in this way, it complements @xcite who model unknown words based on affixal or contextual similarity, and it affords means to improve machine translation. the complete results of the residual space analyses are given in initially one might suspect the need for less aggressive inflection and compounding models, given that so many errors were typos. on balance, significant fractions of the analyses were reasonable: even if an inflected form is misspelled, it is useful to reduce it to a lemma that can then reduce the space of possible correct spellings to which it can be mapped. while our annotation convention allows for only a single category per word, several examples show the benefit of using annotations as heuristics with shades of nuance worthy of human validation. for instance, several misspelled proper names are identified as names rather than typos, and a case of two words inadvertently joined by a deleted space (i.e., a typo) is correctly decomposed into those words by the compounding model. in light of continued challenges in designing computational tools that effectively serve the world's thousands of languages, and that ignoring the linguistic traits of a language does not absolve the designer but rather induces greater harm @xcite , a detailed and taxonomized understanding of the behaviors of the language is vital. our analysis of the word formation processes in such a way that can be grounded in the known lexicon affords both broad-scale familiarity with the language and practical value: it can tailor the design of core nlp tools to the residual vocabulary of a new language. winston wu and david yarowsky. 2020a. computational etymology and word emergence. in proceedings of the 12th language resources and evaluation conference, pages 3252-3259, marseille, france. european language resources association. winston wu and david yarowsky. 2020b. wiktionary normalization of translations and morphological information. in proceedings of the 28th international conference on computational linguistics, pages 4683-4692, barcelona, spain (online). international committee on computational linguistics. david yarowsky and grace ngai. 2001. inducing multilingual pos taggers and np bracketers via robust projection across aligned corpora. in second meeting of the north american chapter of the association for computational linguistics. index word translation type sub-type features pos овесопроизводител oat producer compound n/a masc noun непорнографски non-pornographic compound n/a masc adj бързоразрастващи fast growing compound n/a pl adj окомерни eye sketching compound n/a pl adj преформироването the reformatting compound n/a neu+def noun метапознавателните the meta cognitive compound n/a pl+def adj трискатна of three gables compound n/a fem adj бундестреньор coach of german national team compound n/a masc noun многоцветниците nymphalidae compound n/a pl+def noun авиокосмическо (pertaining to) aerospace compound n/a neut adj геолокацията geolocation compound n/a fem+def noun следамерикански post american compound n/a masc adj китоловния whaling compound n/a masc+def adj видеоизкуство videoart compound n/a neut noun новоизграденият the newly built compound n/a masc+def adj първооснови primary basis compound n/a pl noun лейбгвардейците the life guards compound n/a pl+def noun сухолубиви drought-tolerant compound n/a pl adj наскоропоявилия appeared recently compound n/a masc+def adj леководолазът the scuba diver compound n/a masc+def noun косиха mowed conjugation bulgarian n/a verb разбъркаха mixed conjugation n/a n/a verb обследвайки investigating, inquiring conjugation n/a n/a part заобиколил go around, circumvent conjugation n/a n/a part преместилите moved around conjugation n/a pl+def part нарекъл called, named conjugation n/a masc part недостигнатия unattainable conjugation n/a masc+def part тероризиращ terrorizing conjugation n/a masc part досмила digesting, grinding conjugation n/a n/a part руководителей leaders foreign russian pl noun паутина spider web foreign russian fem noun питянтятяра pitjantjatjara geography australian n/a propn широколъшки (of) shiroka laka geography bulgarian pl adj сетница setnica geography bulgarian n/a propn замфировска zamphyrovska geography bulgarian fem adj (proper) гулянци guliantsi geography bulgarian n/a propn бенковската benkovska geography bulgarian fem+def adj (proper) знеполското the znepolsko geography bulgarian neut+def adj (proper) алитуска the alitusk geography bulgarian fem+def adj (proper) отроковице otrokovice geography czech n/a propn блекпулски blackpool geography english masc adj (proper) пфалцски (pertaining to) pfalz geography german masc adj (proper) сицилианска sicilian geography italian fem adj няманоро nyamanoro geography japan n/a propn твърдишкото (pertaining to) tvarditsa geography place neut+def adj можайският mojayska geography russian masc+def adj (proper) саянската of the sayan (mountains) geography russian fem+def adj лядунския (of) liaodong geography russian masc+def adj верхневилюйское verkhnevilyuysk geography russian n/a adj (proper) болградското bolgradski geography ukranian neut adj (proper) азраки azraqi georgraphy persian n/a propn купчето the small pile/bunch declension n/a neut+dim+def noun естуарното the estuarine declension n/a neut+def adj сметачната the calculating declension n/a fem+def adj просешката the beggary declension n/a fem+def adj трахити trachytes declension n/a pl noun миллети millets declension n/a pl noun ротердамци inhabitants of rotherdam declension n/a pl noun ресинтезът the resynthesis declension n/a masc+def noun флуксетинът the fluoxetine declension n/a masc+def noun неосъзнаваното the unconsciously declension n/a neut+def adj изсечена set in stone, cut down declension n/a fem adj сайгите the saiga anthlopes declension n/a pl+def noun смърчови (of) spruce declension n/a pl adj конецовидните thread-like declension n/a pl+def adj предлози prepositions declension n/a pl noun селяка peasant declension n/a masc+def noun ленни land granted by ottomans declension n/a pl adj екстрахепатичните extrahepatic declension n/a pl+def adj пуническото pertaining to punics declension n/a neut+def adj владишкото of the bishop declension n/a neut+def adj кратовския (pertaining to) kratovo declension n/a masc+def adj (proper) дудукът duduk declension n/a masc+def noun пастафории pastophoria declension transliteration pl noun създателю creator declension vocative masc noun костурчанка kosturchanka name inhabitants fem propn дъмбълдоров (pertaining to) dumbledore name person masc adj (proper) северо severo name person n/a propn квинтерна gittern target n/a fem noun назалност nasality target n/a fem noun нелютив mild (not hot) target n/a masc adj биткойн bitcoin target transliteration masc noun скакач springbok target zoology masc noun гюйсът the jack transliteration dutch masc+def noun дайнамикс dynamics transliteration english n/a propn обложака cover typo addition fem noun меджународната the international typo character swap fem+def adj палеографикатегория paleography category typo concatenation fem noun юдеизма the judaism typo declension masc+def noun наташната the next typo omission fem+def adj широкоразпространеия the widespread typo omission masc+def adj низхождение descent typo omission neut noun цитросуви (of) citrus typo substitution pl adj оковчателното (of) the final result typo substitution neut+def adj жинотните the animals typo substitution pl+def noun татраедър tetrahedron typo substitution masc noun имплементзция implementation typo substitution fem noun принадлежът belong typo substitution pl verb оръсия weapons typo substitution pl noun предозонова prednisone (therapy) typo transliteration fem adj table a.2: manual classification of 100 randomly sampled words from the tokenized bulgarian wikipedia corpus after eliminating entries from the union of dictionaries. we use the ud part-of-speech tags from: https:// universaldependencies.org/u/pos/ . table is summarized in figure 3. index word valid 1 витал yes 2 втрху yes 3 ихрам yes 4 синут yes 5 съкър yes 6 витрал yes 7 маквис yes 8 пераун yes 9 почест yes 10 ревабш yes 11 рененг yes 12 живееяг yes 13 модулин yes 14 гутболни yes 15 джутсуту yes 16 камбоурн yes 17 убеждавт yes 18 читирима yes 19 антатната yes 20 вододпади yes 21 наблядава yes 22 присъствт yes 23 художникт yes 24 обстрикция yes 25 преостъпва yes 26 ассортимент yes 27 монодрамата yes 28 присътствал yes 29 революциятс yes 30 числиността yes 31 продолжавало yes 32 нараставащата yes 33 пристрелването yes index word human trans. alg. trans. human type human sub-type alg. type alg. sub-type features pos н-к manage (abbreviated) n/a abbreviation n/a foreign russian masc noun полупансион half board semi board compound n/a compound n/a masc noun свръхелегантен overly well dressed svra|elegant compound n/a compound n/a masc noun по-нагъл more impudent n/a compound n/a foreign russian masc adj търговско-промишлена industrial-and-retail n/a compound n/a foreign russian fem adj русокоси blond russian compound n/a declension fuzzy pl adj новооткритото the newly found openings compound n/a declension fuzzy neut+def adj мироопазващите peace-keeping peace|keeping compound n/a