Unsupervised Learning of T urkish Morphology with Multiple Codebook VQ - VAE (2024)

Abstract: This paper presents an interpretable unsupervised morphological learning model, showing comparable performance to supervised models in learning complex morphological rules of Turkish as evidenced by its application to the problem of morphological inflection within the SIGMORPHON Shared Tasks. The significance of our unsupervised approach lies in its alignment with how humans naturally acquire rules from raw data without supervision. To achieve this, we construct a model with multiple codebooks of VQ-VAE employing continuous and discrete latent variables during word generation. We evaluate the model’s performance under high and low-resource scenarios, and use probing techniques to examine encoded information in latent representations. We also evaluate its generalization capabilities by testing unseen suffixation scenarios within the SIGMORPHON-UniMorph 2022 Shared Task 0. Our results demonstrate our model’s ability to distinguish word structures into lemmas and suffixes, with each codebook specialized for different morphological features, contributing to the interpretability of our model and effectively performing morphological inflection on both seen and unseen morphological features.

## introduction in this paper, we introduce an interpretable unsupervised morphological learning model that achieves performances comparable to supervised models in the acquisition of complex morphological rules of turkish. we demonstrate its abilities in addressing one of the most studied problems in the literature, morphological inflection in the sigmorphon shared tasks @xcite @xcite @xcite @xcite . the unsupervised acquisition of morphological rules in humans is a natural process during language learning. this involves the analysis of word structures, recognition of stems and affixes, association of consistent meanings, and the integration of these elements into novel combinations, as explained by @xcite . these rules govern the appropriate structure of words to convey their intended meanings. for instance, when forming the present participle of the verb "to bike" it becomes "biking," not "bikeing" necessitating the exclusion of the last vowel. similarly, when evaluating the feasibility of a goal, we consider its "attainability" (attain+able+ity), not "attainityable" (at-tain+ity+able); maintaining the correct sequence of suffixes is crucial in this context. given the inherent ability of humans to learn morphology unsupervisedly, it is essential to develop unsupervised neural models that can replicate this process. this analogy suggests that it should be feasible for a model to acquire morphological rules without explicit supervision. in the intersection of computation and morphology, researchers have developed computational approaches to explore human morphology learning theories and address practical applications like spell checking, correction, automatic speech recognition, and statistical machine translation. the two-level morphology model @xcite , prevalent in the early stages, highlights the complexity of morphology, incorporating phonological alterations beyond a simple arrangement of morphemes. for example, in turkish words like bahçemden and garajımdan, both indicating movement from a possessed place, the morpheme sequences differ (+m+den vs. +ım+dan) based on turkish phonological rules. two-level morphology dissects this into lexical and phonological levels, resulting in the correct surface forms. finite-state transducers, exemplified by a turkish morphological analyzer @xcite , have been utilized to study morphological processes across languages within the two-level formalism. to evaluate unsupervised models from a morphological standpoint, it is essential to establish expectations for a model proficient in "learning" morphology. as outlined in @xcite , the questions an unsupervised morphology learner should address include identifying component morphemes in words, recognizing alternative forms (allomorphs) like -ler and -lar in turkish, understanding conditions for their usage, explaining alternative forms through phonological generalizations, determining permissible combinations of feature specifications, and unraveling morphological realization of each combination. in this work, we propose the multiple codebook vector quantized variational autoencoder (vq-vae) @xcite as an unsupervised morphology learner for text. our approach entails establishing a continuous space and utilizing multiple codebooks. the model integrates codebook entries with the continuous space to generate words. we expect the model to discretize various morphological features in codebooks, thereby representing a word's lemma in continuous space. for example, one codebook may encode person features (e.g., 1 st person singular, 3 rd person plural), another may represent the tense of the word (e.g., present, future, past), and a separate codebook may handle the polarity of the word (positive or negative). we evaluate our model's performance in morphological inflection, addressing the challenges it faces in learning crucial abilities such as allomorph recognition, phonological generalizations, and the realization of diverse morphological feature combinations. additionally, we examine its generalization capabilities under both high and low resource data scenarios by testing it on unseen suffixation scenarios within the sigmorphon-unimorph 2022 shared task 0 @xcite . to gain insights into the model's learning, we further employ probing techniques for interpreting encoded information within latent representations. our primary contributions are: ## model we extend the idea of vector quantised-variational autoencoders (vq-vae) @xcite for text. the original vq-vae is an encoderdecoder model that aims to model image and speech data using discrete latent variables picked from a codebook having embeddings. the encoder outputs are replaced with the nearest vectors in l 2 distance from the codebook. then, the codebook embeddings are fed to the decoder, and the data reconstruction is aimed. in our model, we employ multiple codebooks, in contrast to the original vq-vae that uses only one. additionally, we incorporate continuous variables, following the approach of the original vae. our expectation is that the model will specialize each codebook to capture distinct morphological features of a word, and as a result, the continuous space will be utilized to encode the lemma of a word. specifically, we construct a vq-vae model with continuous and discrete variables with the following blocks: bidirectional gru encoder, low-dimensional continuous space, varying number of codebooks for discrete space, and a unidirectional gru decoder, as seen in fig. 1 . while the continuous part is regulated by kl divergence to standard gaussian prior as in regular vae, the discrete latent variables are obtained with quantization through multiple codebooks. the encoder q with parameters ϕ has the last forward hidden state -→ h t , and the last backward hidden state ←h t with d dimensional vectors. the mean µ and variance σ are learned by applying a linear transformation to the last backward hidden state ←h t . then, using µ and σ, we estimate the continuous latent variable z c . to make the learning step differentiable, we use the reparameterization trick @xcite and calculate z @xmath0, 1). for quantization, we define the latent embedding space of codebooks as @xmath1, k is the number of entries in each codebook, d is the dimension of each embedding vector e (n) in the codebook. the -→ h t vector from the encoder is then linearly transformed into n vectors with dimension d. for each linearly transformed vector from encoder z and obtain z q (x) vector. we finally concatenate the quantized vector with the continous vector and feed it to the decoder as an initial hidden state. at each time step of decoding, we concatenate the continous vector z c , quantized vector z q , and the target token embedding. the total objective for our model becomes: the initial component of the loss involves the reconstruction loss, where the model conditions on the continuous latent variable z c and discrete latent variable z q to reconstruct the observed data x. the subsequent element pertains to the overall vector quantization loss for each vector z (n) e (x). similar to the original vq-vae, the stop gradient operation (denoted as sg) is employed to facilitate the learning of codebook embeddings e (n) . this operation ensures that the gradient of the applied term becomes zero during forward computation, converting it into a non-updated constant. in the second term, to minimize the l 2 distance between encoder outputs and codebook embeddings, only the codebook embeddings are updated. the third term involves updating only the encoder outputs, weighted by the parameter β to prevent the encoder outputs from growing faster than the codebook embeddings. lastly, in the fourth term, we regulate the continuous vector using a standard gaussian distribution. ## evaluation in this section, we evaluate the performance of our unsupervised model in morphological inflection (see section 3.1) and probe the latent variables of the model for morphological features (see section 3.2). we conduct further evaluation of our model in the context of sigmorphon-unimorph 2022 shared task 0 @xcite in section 3.3. ## importance of directionality in this section, we investigate the impact of our directional choice, where we assign the last backward hidden state ←h t to the continuous vector, aimed at encoding the lemma, and the last forward hidden state -→ h t to the codebooks, intended to encode mor-phological features in the suffix. given the structure of turkish, where the lemma typically starts on the left and suffixation occurs on the right, we anticipate this approach to be effective, introducing a form of inductive bias. to understand its effect, we concatenate the last forward and backward hidden states [ -→ h t ; ←h t ], and input the resulting vector into both the continuous vector and the codebooks. we conduct experiments with 4, 6, and 8 codebooks, each having 6, 8, and 12 entries, while maintaining other model dimensions. the experiments with three different random initializations reveal three types of observed problems: (1) suffix information is not entirely encoded in the discrete part, but partially encoded in the continuous part with lemma. in every setup, the lack of separation between lemma and suffix into continuous and discrete parts interferes with mapping morphological tags to codebook entries. thus, morphological inflection cannot be performed well. while the problems are partially observed in several runs of the model with an inductive bias, we could still achieve good convergence in most setups, which is a challenge to replicate without incorporating directionality. consequently, we argue that the directionality helps the model in distinguishing between lemma in the continuous and suffix in the discrete parts. nevertheless, further experiments without directionality may provide better insights. ## related work the unsupervised study of morpheme boundaries dates back years. harris (1955)'s pioneering work introduces a heuristic based on letter successor/predecessor tokens, counting the different letters after a morpheme candidate x. subsequent works enhance this approach by analyzing the frequency distribution of successor tokens and calculating entropy to measure predictability.the morfessor family, including morfessor baseline @xcite , morfessor flatcat @xcite , and morfessor em+prune @xcite , utilizes generative models for language morpheme learning. morfessor baseline optimizes parameters through map estimation, adhering to the minimum description length principle. morfessor em+prune starts with a seed lexicon of the most frequent subwords and prunes during training. additionally, adaptor grammar @xcite and mor-phagram @xcite contribute to unsupervised morphological segmentation, incorporating adaptors like the pitman-yor process @xcite . further work involves leveraging semantic features of words through neural networks for unsupervised morphological segmentation @xcite . previous work in morphological inflection includes supervised learning techniques. @xcite employs alignment and learns edit operations, while kann and schütze (2016) proposes a neural approach using an encoder-decoder architecture with soft attention @xcite and stacked grus @xcite . @xcite proposes data augmentation by generating hallucinated data in lemma-feature tag-target pairs. they replace shared substrings longer than three characters with random characters, resulting in hallucinated lemma-tag triples. some probing studies on rnns include @xcite ; @xcite . criticisms regarding probe reliability and classification limitations have prompted the consideration of simpler probes, emphasizing information-theoretic measures over accuracy @xcite @xcite . the studies also explore causal relations and latent ontologies, providing insights into feature usage and representations @xcite @xcite . ## conclusion & future work this work presents a novel and interpretable unsupervised model for learning turkish morphological rules, performing comparably to supervised models, particularly in low-resource settings. the model separates the lemma of words into continuous variables and their suffix into discrete variables within codebooks. across multiple runs, it customizes each codebook with distinct morphological features, contributing to enhanced interpretability. future work may involve exploring different morphological tasks, such as unsupervised paradigm completion and unsupervised paradigm clustering.