In search of isoglosses: continuous and discrete language embeddings in S lavic historical phonology (2020)

Abstract: This paper investigates the ability of neural network architectures to effectively learn diachronic phonological generalizations in amultilingual setting. We employ models using three different types of language embedding (dense, sigmoid, and straight-through). We find that the Straight-Through model out-performs the other two in terms of accuracy, but the Sigmoid model’s language embeddings show the strongest agreement with the traditional subgrouping of the Slavic languages. We find that the Straight-Through model has learned coherent, semi-interpretable information about sound change, and outline directions for future research.

## introduction historical phonology is an important area of diachronic linguistics, allowing scholars to explore the space of possible sound change trajectories and resulting synchronic patterns, as well as posit degrees of relatedness between languages on the basis of sound changes shared across them. the latter practice traditionally involves the identification of innovations that are probative with respect to historical subgrouping. the internal genetic structure of many linguistic groups is uncontroversial. for others, scholars disagree in terms of which isoglosses are relevant to subgrouping, and whether the relevant features are indeed shared across groups of languages. the use of computational methods has aided in resolving a number of outstanding questions in diachronic linguistics, though little work has been done assessing the ability of computational models to learn meaningful patterns of sound change as well as capture language-level information that may bear on degrees of genetic relatedness. this paper employs a neural encoder-decoder architecture to analyze patterns of sound change among slavic languages, training a series of models on data from an etymological dictionary. following the standard practice in multilingual nlp tasks, we make use of language embeddings concatenated to the model input. we make use of three different types of language embedding, comprising continuous real-valued dense, sig-moid (defined on the [0, 1] interval), and binary straight-through embeddings. we assess the accuracy with which these encoder-decoder models predict held-out forms in contemporary slavic languages from their corresponding proto-slavic input. we provide a detailed error analysis, observing differences across models in terms of the types of error introduced. we measure the extent to which the language embeddings learned by each model recapitulate the the most commonly accepted subgrouping of the slavic languages. finally, we assess the interpretability of the straightthrough embedding, investigating the degree to which embeddings in binary latent space represent meaningful information regarding sound change. we find that the model with straight-through language embeddings outperforms the dense and sigmoid models in terms of accuracy. at the same time, the language embeddings learned by the sigmoid model display a signal that shows the highest agreement out of the three models with received wisdom regarding the dialect grouping of slavic languages. we find that the latent binary representations learned capture meaningful and coherent information regarding sound patterns. we outline future directions for research using latent binary embeddings in neural historical phonology. ## background the slavic branch of indo-european is traditionally divided into east, west, and south slavic groups. many of the oldest and most de-cisive isoglosses differentiating the slavic languages are phonological in nature (cf. @xcite . for instance, tautosyllabic proto-slavic vowel+liquid sequences were subject to metathesis or re-ordering in west and south slavic languages, whereas east slavic languages underwent pleophony, inserting a vowel between the liquid and the following consonant. variation between liquid metathesis and pleophony, accompanied by language-specific vowel changes, can be seen in the cognates russian górod, ukrainian hórod, croatian grâd, czech hrad (< *gôrdȗ 'city'); the hfound in ukrainian (east slavic) and czech (west slavic) shows also that certain shared features do not cleanly follow the taxonomy defined above. it is traditionally assumed that the tripartite classification of slavic either reflects the dialectal diversity of the so-called slavic homeland, most probably situated on the outskirts of the carpathian mountains, or emerged as a result of the great slavic expansion in the 6th century ad @xcite . the extensive study of loanwords, however, suggests that postexpansional slavic was, despite the vast territory it occupied, still uniform. there seem to have been no significant differences between slavic spoken in areas located as far away from each other as the baltic sea and the peloponnese, at least with regard to phonology. it has therefore been argued that it is this post-expansional slavic that constitutes the ancestor of all slavic languages and not the slavic language spoken in the homeland @xcite . one of the arguments put forward in support of this claim is the still largely reconstructible post-proto-slavic dialect continuum @xcite . one objective of this paper to assess the degree to which neural models recapitulate the uncontroversial subgrouping of slavic as an indicator of whether they are capable of resolving outstanding issues in the field. ## related work a growing body of research assesses the information captured by language embeddings trained on large data sets using neural models. there is some debate as to whether embeddings learned in these tasks can pick up on genetic signal @xcite , or whether the information learned represents structural similarity @xcite . the majority of work of language embeddings involves models trained on large parallel corpora. @xcite approach the issue of sound change using a grubased neural machine translation model with soft attention to reconstruct latin forms from contemporary romance reflexes; the authors employ language embeddings, but do not provide an analysis of the information captured by these embeddings. phylogenetic approaches to sound change and the reconstruction of word forms incorporate a highly articulated genetic representation of language relatedness @xcite , but employ simplified representations of sound change in comparison to what can be captured by recurrent neural networks; at the same time, phylogenetic work explicitly models intermediate stages of change, a potential challenge for rnns, which are better suited to learning patterns resulting from the telescoping of multiple changes. related work seeks to disentangle genetic and areal pressures in shaping crosslinguistic patterns @xcite @xcite . in general, while the signal learned by embeddings can be analyzed via visualization techniques @xcite , it is a challenge to link the behavior of embeddings to individual features in the data analyzed. this difficulty undoubtedly stems in part from the fact that embeddings are generally continuous, lacking the sparsity or discreteness needed to identify the behavior of the neural model when features are active or inactive. this issue has been addressed by the development of de-noising approaches designed to induce sparsity @xcite . binary latent variables are of key interest to linguistic questions, but pose many challenges for inference. binary latent variable models such as the indian buffet process (ibp, @xcite have been used in some applications in computational phonology and typology @xcite using a combination of gibbs sampling and updates from the metropolis-hastings algorithm or hamiltonian monte carlo, but it is not clear that these inference procedures are scalable to neural models. discrete variables pose problems for differentiability in gradient-based optimization algorithms; marginalizing out all possible combinations of binary variables is generally unfeasible for binary latent variables. variational approaches have attempted to circumvent this issue via the concrete (alternatively, gumbel-softmax) distribution @xcite , which extends the reparameterization trick to categorical distributions and which produces gradient estimates that have lower variance than standard estimation techniques @xcite but are still biased; subsequent approaches reduce bias but are less straightforward to implement @xcite . while concrete-distributed versions of the ibp have been used in neural models @xcite , this work is limited to variational autoencoders, which use amortized variational inference to learn latent representations from the data via a global inference network; encoder-decoder mechanisms with attention like the one used in this paper cannot exploit this property of the data; training the latent variable with stochastic variational inference, while theoretically possible, is considerably more difficult @xcite . as an alternative, we use straight-through (st) embeddings @xcite in a maximum likelihood framework. straight-through layers are discrete but have underlying continuous weights; model output is predicted on the basis of the discrete representation, while model loss is differentiated with respect to the continuous underlying weights. while this approach has the same problems with biased estimates as the concrete distribution, it is straightforward to implement. we compare the quality of straight-through embeddings to embeddings with no activation and embeddings with sigmoid activation. ## data our data set consists of proto-slavic etyma and corresponding reflexes in medieval and modern slavic languages taken from a digitized version of a slavic etymological dictionary @xcite ; for alternative reconstructions see @xcite . in order to minimize the chance of introducing morphologically noncongruent forms into our data set, we extracted the first form provided for each slavic language in each entry, since these are most likely to agree morphologically with the proto-slavic headword. we converted forms in modern slavic languages to a narrow phonetic representation using ipa transcriptions from wiktionary (https: // www.wiktionary.org ), which were used to train a neural encoder-decoder; these models were used to obtain ipa transcriptions for forms not in wiktionary, and a portion was checked manually. in several cases we reconciled sources used in the etymological dictionary (e.g., @xcite with contemporary standardized orthographies, and made use of phonetic descriptions for languages where the training data were problematic @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . for the medieval languages old church slavic and church slavic, orthographic forms were converted to a broad phonemic transcription based on @xcite . suprasegmental features were marked for all modern languages (pitch accent for slovene and bcs and primary stress for the remainder; for consistency, we chose to mark primary stress on monosyllables in stress-timed languages). we excluded languages with fewer than 100 forms in the etymological dictionary (this resulted in the omission of macedonian, polabian and slovincian). we took additional steps to remove morphological mismatches in the data set. for bulgarian verbs, which reflect the proto-slavic 1sg present in their citation form, we replaced the proto-slavic headword (the infinitive form by default) with a morphologically congruent form, and excluded a small number of forms based on athematic verbs. additionally, proto-slavic adjectives are always given in the nominal or short form, although contemporary slavic languages often reflect the socalled long form, which arose from the addition of an inflected element *-jȋ to the ending; we converted short proto-slavic adjectives to their long form in the appropriate contexts. we tried to ensure that proto-slavic verbs matched their reflexes according to the presence/absence of reflexive morphology and preverbs. additionally, the original data source contains multiple gender inflections for certain proto-slavic etyma (e.g., *àblȗko n., *àblȗka f., and *àblȗkȗ m. for 'apple'), which are linked to the same reflexes irrespective of the reflexes' gender; for such forms, we discarded etymon-reflex pairs with mismatched gender. ultimately, this process yielded 11400 forms in 13 languages (see ## method to learn mappings between proto-slavic etyma and the slavic reflexes that descend from them, we use an lstm encoder-decoder with 0thorder hard monotonic attention @xcite , trained on all languages in our data set. the basic model architecture used for the experiments in this study has the following structure (schematized in figure 1 ): a trainable languagelevel embedding is concatenated to a one-hot representation of each input segment at each input time step; each concatenation is fed to a dense layer (with no activation) to generate a embedding for each time step that encodes information about the input phoneme and language id of the reflex; these embeddings subsequently are fed to the encoder-decoder in order to generate the output. the parameters of the encoder-decoder architecture are shared across languages in the data set; the sole language-specific variable employed is the language-level embedding fed to the model. in all experiments, we set the dimension of the language-level embedding and the language/character embedding to 128, and the hidden layer dimension to 256. in our experiments, we employ different representations of the languagelevel embedding, including a dense layer with no activation (dense model), a dense layer with sigmoid activation, (sigmoid model) and a dense layer with straight-through activation (st model), which uses the heaviside step function (negative values map to 0, non-negative values to 1). we train our model for 200 epochs with a batch size of 256 using the adam optimizer with a learning rate of .001 with the objective of minimizing the mean categorical cross-entropy between the predicted and observed distributions of the output. to evaluate model performance, we carry out k-fold cross-validation (@xmath0, randomly holding out 10% of the forms in each language, and greedily decoding the held-out forms using the trained model. for additional analyses regarding the interpretability of the embeddings learned, we train the model on all forms in the data set. models are implemented in keras @xcite and larq @xcite ). foot_0 6 results ## discussion and outlook this paper investigated the performance of multiple neural models in capturing patterns of sound change across slavic languages. we found that a model with binarized straight-through languagelevel embeddings outperformed other models in terms of accuracy, and shows great potential for learning coherent and interpretable information regarding sound change. we found that the discrete features learned by our model appear for the most part to correspond to meaningful, realistic variation in sound patterns, though representations are not particularly sparse. additionally, randomly sampling from discrete latent space tended to consistently generate coherent output; the preliminary attempts that we made to assess the likelihood of observing these samples in naturalistic contexts can be expanded considerably. we used straight-through embeddings as a lowcost alternative to more involved means of training discrete latent variables. in the immediate future, we plan to extend our approach to make use of variational approaches, the flexibility of which may help in inducing sparsity in order to learn more meaningful, realistic representations (there is additionally room for exploration of simpler approaches that we did not make use of in this paper, such as dropout regularization); however, since our encoder-decoder model is different from the autoencoding models used in previous work, directly extending these methods presents a challenge that require considerable experimentation to overcome (an early attempt to adopt the ibp prior of @xcite was unsuccessful, as the monotonically decreasing prior probabilities rarely yielded non-zero values; thus far, attempts to weight the kl divergence term have not yielded success). nevertheless, as low-variance, low-bias techniques for inferring discrete variables in neural models progress, we believe that they will be an increasingly valuable means of capturing meaningful, interpretable features in multilingual neural tasks like this paper's.