How to encode arbitrarily complex morphology in word embeddings, no corpus needed (2022)

Abstract: In this paper, we present a straightforward technique for constructing interpretable word embeddings from morphologically analyzed examples (such as interlinear glosses) for all of the world’s languages. Currently, fewer than 300-400 languages out of approximately 7000 have have more than a trivial amount of digitized texts; of those, between 100-200 languages (most in the Indo-European language family) have enough text data for BERT embeddings of reasonable quality to be trained. The word embeddings in this paper are explicitly designed to be both linguistically interpretable and fully capable of handling the broad variety found in the world’s diverse set of 7000 languages, regardless of corpus size or morphological characteristics. We demonstrate the applicability of our representation through examples drawn from a typologically diverse set of languages whose morphology includes prefixes, suffixes, infixes, circumfixes, templatic morphemes, derivational morphemes, inflectional morphemes, and reduplication.

## better representations are needed the past several years have seen the development of neural techniques capable of creating extremely high quality word embeddings, most notably bert @xcite and its many variants. in total, however, fewer than 300-400 languages have have more than a trivial amount of digitized text data, thus rendering data-driven nlp approaches including bert futile for more than 6000 remaining languages (representing over 1.2 billion people; @xcite , even with aggressive multilingual models, transfer learning, bilingual anchoring, and typologically-aware modelling @xcite @xcite . somewhere between 100-200 languages (most in the indo-european language family) have enough digitized text data @xcite for bert embeddings of reasonable quality to be trained using a combination of techniques including unsupervised sub-word segmentation methods, multilingual bootstrapping, and transfer learning. quality of word embeddings is substantially lower when corpus sizes are insufficiently large; @xcite , for example, constructed word embeddings using approximately 10 million tokens for yorùbá foot_0 and twi, 2 and found that the resulting embeddings are substantially poorer in quality those for high-resource languages. ## existing word representations are insufficient for most languages computational processing of natural language requires practical digital representations of the words of a language. we survey existing methods for representing words, arguing that while existing word representations work well for high resource ana-lytic languages like english, existing representations are insufficient for effectively representing morphologically complex words in thousands of languages for which large corpora do not exist. ## embedding and retrieving rich linguistic information as established in §1, there are thousands of languages which lack the large corpora needed for reliably training neural language models such as bert. for many of these cases, the size of corpora may be very small or even nonexistent. while multilingual and bootstrapping approaches certainly have a role to play, we ought not ignore the rich linguistic information embedded in morphological analyses. essentially every language that is even partly documented has numerous such analyses in the form of interlinear glossed text (ilgs) created by expert linguists. instead of relying on neural networks to induce linguistic patterns by processing massive corpora, we argue that for more than 6000 so-called "low-resource" languages, a more fruitful method for initializing meaningful word and subword embeddings is by directly embedding the rich linguistic information included in the morphological analyses found in ilgs and (when they exist) other morphologically analyzed corpora. ## supporting full linguistic diversity we now demonstrate the broad applicability of our technique for encoding rich linguistic information from morphologically analyses such as ilgs using examples drawn from a typologically diverse set of polysynthetic, agglutinative, fusional, and analytic languages. the following examples include prefixes, suffixes, infixes, circumfixes, templatic morphemes, derivational morphemes, inflectional morphemes, and reduplication. the notation in the following examples follows the conventions established in §3.3. ## conclusion while corpora of anything greater than trivial size exist only for a few hundred languages ( §1), morphologically analyzed examples in the form of interlinear glosses exist for essentially every human language. the vast array of human languages include a rich variety of morphological phenomenon that are not easily handled by existing word embedding methods ( §2). this work presents a straightforward mechanism whereby meaningful, linguistically interpretable word and morpheme embeddings can be created for any word in any language ( §3- §4). we have demonstrated the applicability of our method using linguistic examples of concatenation and zero inflection ( §4.1), circumfixation ( §4.2), fusion ( §4.3), polysynthesis ( §4.4), agglutination ( §4.5), infixation ( §4.6), reduplication ( §4.7), and templatic morphology ( §4.8). in addition to their direct use in future research involving language documentation and revitalization, we anticipate that embeddings created using the methods described in this work may provide an important initial step in bootstrapping vastly multilingual models capable of embedding words from thousands of languages.