Disentangling dialects: a neural approach to I ndo- A ryan historical phonology and subgrouping (2020)

Abstract: This paper seeks to uncover patterns of sound change across Indo-Aryan languages using an LSTM encoder-decoder architecture. We augment our models with embeddings represent-ing language ID, part of speech, and other features such as word embeddings. We find that a highly augmented model shows highest accuracy in predicting held-out forms, and investigate other properties of interest learned by our models’ representations. We outline extensions to this architecture that can better capture variation in Indo-Aryan sound change.

## introduction the indo-aryan languages, comprising sanskrit (otherwise known as old indo-aryan, or oia) and its descendant languages, including medieval languages like pāl . i and modern languages such as hindi/urdu, panjabi, and bangla, form a wellstudied subgroup of the indo-european language family. at the same time, many aspects of the indo-aryan languages' history remain poorly understood. one reason is that there are large historical gaps in the attestation of indo-aryan languages, making it challenging to document when certain shared innovations took place. additionally, while the operation of sound changes are a diagnostic for subgrouping that historical linguistic often employ, indo-aryan languages have remained in close contact for millennia, borrowing words from each other and making it difficult to establish subgroupdefining sound laws using the traditional comparative method of historical linguistics. while a number of large digitized multilingual resources pertaining to the indo-aryan languages exist, these data sets have not been widely used in studies, and our understanding of indo-aryan dialectology stands to benefit greatly from the application of deep learning techniques. this paper seeks to move further towards closing this gap. we use an lstm-based encoder-decoder architecture to analyze a large data set of oia etyma (ancestral forms) and medieval/modern indo-aryan reflexes (descendant forms) extracted from a digitized etymological dictionary, with the goal of inferring patterns of sound change from input/output string pairs. we use language embeddings with the goal of capturing individual languages' historical phonological behavior. we augment this basic model with additional embeddings that may help in capturing irregular patterns of sound change not captured by language embeddings; additionally, we compare the performance of these models against a baseline model that is embedding-free. we evaluate the performance of models with different embeddings by assessing the accuracy with which held-out forms in medieval/modern indo-aryan languages are predicted on the basis of the oia etyma from which they descend, and carry out a linguistically informed error analysis. we provide a quantitative evaluation of the degree of agreement between the genetic signal of each model's embeddings and a reference taxonomy of the indo-aryan languages. we find that a model with embeddings representing data points' language id, part of speech, semantic profile and etymon id predicts held-out forms that are closest to the ground truth forms, but that a tree constructed from language embeddings learned by this model shows lower agreement with a reference taxonomy of indo-aryan than a tree constructed on the basis of a model with only language embeddings, and that in general, the ability of our models to recapitulate uncontroversial genetic signal is mixed. finally, we carry out experiments designed to investigate the information captured by specific embeddings used in our models; we find that our models learn meaningful information from augmented representations, and outline directions for future research. ## background: indo-aryan dialectology despite a long history of scholarship, there is no general consensus regarding the subgrouping of indo-aryan languages comparable to that regarding other branches of indo-european, such as slavic or germanic. scholars argue for a core-periphery @xcite @xcite ; @xcite or east-west split between the languages @xcite , or are agnostic to the higher-order subgrouping of indo-aryan, given the many challenges involved in establishing such groups (for discussion, see @xcite @xcite . disagreement between these groups stems largely from the fact that the different hypotheses are based on different linguistic features, and there is no agreed upon way in which to establish that individual features shared across languages are inherited from a common ancestor rather than due to parallel innovation. the traditional comparative method of historical linguistics @xcite tends to establish linguistic subgroups on the basis of innovations in morphology as well as shared sound changes, some of which are thought to be unlikely to operate independently. indeed, many scholars have agreed that indo-aryan subgrouping should be established according to sound change; however, the establishment of regular sound changes has proved challenging given the high degree of irregularity in the data @xcite . our method has the potential to detect regularities and bear on the questions described above. ## related work traditional computational dialectology @xcite identifies dialect clusters using edit distance; more recent work uses neural architectures for dialect classification based on social media data for languages such as english @xcite and german @xcite . computational methods have been applied to the related field of historical linguistics to identify cognates (words that go back to a common ancestor) and infer relationships between languages @xcite as well as the reconstruction of ancestral words through bayesian methods @xcite , gated neural networks @xcite and non-neural sequence labeling methods @xcite . other recent work infers language embeddings from large parallel corpora using different neural architectures @xcite @xcite . these embeddings tend to produce hierarchical clustering configurations that are close to the language classification trees inferred from historical linguistic research. these claims have been tested by @xcite who find that the distances between learned language representations may not be reflective of genetic relationship but of structural similarity. it is not always straightforward to interpret the sources of differentiation among these embeddings; typically, embeddings based on synchronic patterns of language use in corpora may be due to word order patterns, phonotactic patterns, or a number of other interrelated language-specific distributions. cathcart and wandl (2020) investigate the patterns of sound change captured by a neural encoder-decoder architecture trained on proto-slavic and contemporary slavic word forms, and find that embeddings dispay at least partial genetic signal, but also note a negative relationship between overall model accuracy and the degree to which embeddings reflect the communis opinio subgrouping of slavic. ## data and rationale for model design we use data from an etymological dictionary of the indo-aryan languages @xcite . 1 we extract oia etyma and their corresponding reflexes in medieval and modern indo-aryan languages (e.g., oia vākya 'speech, words' develops to pāl . i vākya, kashmiri wākh, etc.). as the traditional indological orthography used to transcribe forms in the dictionary is phonemic, we retain this representation and convert characters with diacritics to a normalization form canonical decomposition (nfd) unicode representation in order to reduce the number of input and output character types. additionally, we extract glosses provided for oia etyma (at the time of writing, the extraction of reflex glosses cannot be straightforwardly automated due to the unstructured nature of the markup language, plus the absence of glosses for certain reflexes). we match languages in the dictionary with the closest matching glottocode from the glottolog database @xcite , and omit languages with fewer than 100 entries. this results in a data set of 82431 forms in 61 languages; the number of forms in each language can be seen in a more interesting and poorly understood point (that is not merely an artifact of the data) is that etyma with certain semantic profiles may be more prone to certain analogical changes. for instance, nouns of certain semantic fields may be more likely to receive a diminutive suffix, which may then be reanalyzed as part of the noun stem; additionally, semantically related nouns are known to undergo analogical contamination @xcite or develop specific patterns of sound symbolism @xcite . finally, particular etyma may favor a particular "prototype" showing specific patterns of sound change. an example of this phenomenon can be seen in reflexes of oia vismarati 'forgets'. oia sm usually changes to m(h) or s(s) in descendant languages; however, only one reflex of vismarati shows m(h) (prakrit vimharaï), while the rest show s(s). it is possible that an early variant *visarati was diffused among neighboring dialects and an early date. all in all, while we do not explicitly model contact, accounting for the factors described above can improve model accuracy and allow us to tease apart legitimate patterns of sound change from orthogonal factors. in order to achieve this goal, we augment a basic model using language embeddings with different embedding types designed to account for idiosyncrasies of data collection as well as potential realworld sources of irregularity like those described above. we make use of embeddings that represent the part of speech (pos) of the oia etymon. additionally, we represent the semantic profile of each oia etymon by generating embeddings of each etymon's english language gloss using a pretrained bert model @xcite , though this approach does not fully encapsulate the oia word's semantics. finally, we wish to take into account idiosyncratic patterns displayed by individual etyma (such as vismarati, as discussed above). a one-hot encoding of etymon ids is costly, as there are 13580 unique etyma in our dataset; instead, we combine information from bert embeddings and the input string in order to produce a unique embedding for each etymon in the data set. in sum, these augmentations provide a way for our model to disentangle the orthogonal forces of sound change and other factors. ## model our experiments use an lstm encoder-decoder with 0th-order nonmonotonic hard attention @xcite . the authors' architecture works as follows: for each input x (for our purposes an oia etymon), a latent representation h enc @xmath0, ..., |x|} via a bidirectional lstm on the basis of the input symbol at time step j. for each output y (for our purposes a medieval/modern indo-aryan reflex), a latent representation h dec @xmath1, ..., |y|} on the basis of the output symbol at time step i -1. the probability that the output is aligned with the jth input symbol at time i is equal to softmax(h dec i t h enc j ), where @xmath2. the emission probability of the output symbol at time i given such an alignment is proportional to exp(w tanh(s[h dec i ; h enc j ])), and is hence also dependent on the previous output symbols (@xmath3. structural zeros are used in order to ensure that the alignment between the input and in our experiments, we concatenate embeddings encoding the features described in the previous section to our input at each time step, namely (l)anguage id, (p)art of speech, (s)emantic profile, and (e)tymon. we use a one-hot encoding of language id and pos id, and employ bert embeddings (reduced from 768 to 128 dimensions using principal component analysis) to represent an etymon's semantic profile. embeddings for etyma are represented by contatenating the first and last states of a bidirectional lstm encoding of the etymon string to the bert-based semantic embedding (denoted by e(gloss i ))). formally, these embeddings consist of the following, for a given data point index @xmath4, ..., |data|}: after one or more of these embeddings are concatenated to an input token, the resulting concatenation is passed to another mlp layer, which is then fed to the encoder-decoder architecture. we increment our models by concatenating embeddings to the input in a stepwise fashion, yielding four models (l, lp, lps, lpse). additionally, we compare our models against a baseline that does not use any embeddings. we set the embedding and hidden layer dimension size to 128. we carry out k-fold (@xmath5, training our models on mini-batches of 64 data points for a maximum of 200 epochs, validating on 10% of the training data and stopping early if the validation loss does not decrease over twenty consecutive epochs. we use the adam optimizer (kingma and ba, 2015) with a learning rate of .001. we greedily decode test data using the fitted models, generating each output token on the basis of the previous generated token. 3 6 results we assess the accuracy with which our models predict held-out medieval/modern indo-aryan forms on the basis of the corresponding oia etymon input by measuring the phoneme error rate (per), which we define as the levenshtein distance between the predicted and true form divided by the length of the longer string (normalized levenshtein distance), and the word error rate (wer), or the proportion of held-out forms where one or more errors occurs in the predicted form. mean per and wer values are found for each model in as expected, the baseline model performs the worst according to these metrics. of the nonbaseline models, the model with language and pos embeddings shows the worst overall performance; highly augmented models such as the model which uses language, pos, semantic and etymon embeddings shows the best performance in terms of per. however, the model with only language embeddings shows the best performance in terms of wer, indicating that the lpse model introduces errors in a higher number of individual predicted words even if it produces fewer errors overall. we carry out pairwise wilcoxon signed-rank tests to assess the significance of differences in per between models, using the bonferroni correction for multiple comparisons; all between-model differences are highly significant or significant, with the exception of the difference between the l and lps models. figure 1 displays the relationship between language-level per and the number of training examples for a given language. the correlation between these two variables is negative and significant (spearman's ρ is between -.39 and -.54) for all models. however, as shown by the lines of best fit plotted in the figure, this correlation is considerably weaker for the baseline model than for the other models. interestingly, the omission of language embeddings seems to have resulted in higher error rates for languages with larger numbers of training examples; if no information regarding language id is fed to the encoder-decoder, there seems to be no way to keep highly influential languages from interfering in the patterns learned for other highly influential languages. ## error analysis per based on unweighted levenshtein distance is agnostic to error type. in a task such as ours, some error types will indicate strongly that our models model per wer l .257 .808 lp .262 .818 lps .256 .813 lpse .255 .809 baseline .346 .940 with this in mind we turn to a quantitative but linguistically informed error analysis designed to investigate the types of errors made by each model and assess the degree to which each model makes certain errors. one error type consists of errors in vowel quality or quantity: the lps model erroneously generates kashmiri muhun (< oia mōháyati 'bewilders') as mȏhun. errors of this sort affect consonants as well: the pāl . i form addana (< oia ardana 'tormenting') is erroneously generated as ad . d . ana by the lp model. errors of this sort, particularly vowel type errors, often occur at the right word edge, perhaps due to confusion resulting from the restructuring of the oia case and gender systems in many indo-aryan languages; e.g., sindhi vali (< oia vallī 'creeper') is generated as vala '). along with source errors or morphological mismatches that simply cannot be detected by our model architecture, the errors mentioned above make up the bulk of errors produced by models. we align tokens in held-out forms with tokens in predicted forms using the needleman-wunsch algorithm @xcite , allowing us to automatically extract errors made by each model (e.g., instances where l . was generated instead of l). we classify mismatches between heldout and predicted tokens according to whether they involve an insertion, a deletion, a change affecting a vowel (other than deletion), or a change affecting a consonant (other than deletion). proportions of these changes across models are given in selected held-out reflexes of oia mátsya 'fish' are provided in ## genetic signal we investigate the degree to which the languagelevel embeddings learned by our models represent the genetic relatedness of indo-aryan languages in our sample. for the language embeddings produced by each model, we compute the cosine distances between each pair of embeddings, and use these distances to construct language trees using neighbor joining (nj, @xcite . we compare each tree to a reference taxonomy of the indo-aryan languages taken from glottolog @xcite , which contains relatively uncontroversial language groupings but does not resolve all subgroupings, and hence contains numerous polytomies (i.e., non-binary branchings). we measure the distance between two language trees is measured using generalized quartet distance (gqd, @xcite . a quartet in a phylogenetic language tree consists of four languages and can either be resolved ("butterfly") or unresolved ("star"). the generalized quartet distance is the ratio of the number of butterfly quartets which differ across trees to the total number of butterfly quartets in the reference tree. while our reference tree is non-binary, the trees inferred from our models' embeddings are binary. accordingly, the gqd measure does not penalize the inferred tree, ignoring the star quartets found in the reference tree. the gqd scores for all our models are given in notably, despite its good performance according to the per metric, the embeddings produced by the lpse model show the lowest agreement with a reference taxonomy of indo-aryan out of all of the models used in our experiments, particularly when compared with the model that uses only language embeddings. a possible explanation is that by including additional embeddings in our models designed to capture different patterns of sound change in different morphological, semantic and etymological scenarios, we have filtered out critical information relevant to subgrouping, removing valuable genetic signal displayed by morphological traits, which may explain why the model with language embeddings outperforms the other models. a similar negative relationship between model accuracy and genetic signal displayed by embeddings was found by @xcite . at the same time, all models are significantly outperformed by the ldn tree, indicating that string distances between contemporary forms capture inter-language relationships at more levels of granularity than the distances computed from the embeddings learned by our models. all models, including the lpse model, are successful at learning patterns of change within individual lineages and recapitulating shallow subgroups. for visualization, we map lpse embeddings to threedimensional space using multidimensional scaling and color the taxa of the reference tree using rgb color vectors produced by these normalized values, as shown in figure 2 . similarly colored taxa are close to each other in embedding space. in general, very close relatives appear to occupy the same region of embedding space, but small groups within large taxa may be distant from each other in the same space, indicating premature but not altogether unpromising results for our attempt to capture genetic signal in the data we analyze. ## representations learned by embeddings here, we investigate whether our architecture has learned properties of the different patterns displayed by verbal and non-verbal forms. oia verbs end in -ti and -tē; virtually no oia nouns end in -tē, and while some oia nouns end in -ti, they tend to have a different phonotactic profile from oia verb forms. hence, it is entirely possible that encoderdecoder models with embeddings that encode part of speech ignore this information and can simply learn mappings such as oia -@xmath8.g., from oia vaśati : assamese bahāiba), since -iba is a suffix that freqently co-occurs with the oia sequence -ati. the fact that a model with only language embeddings outperforms a model with language embeddings and part-of-speech embeddings provides a reason to suspect that our architecture does not learn anything about the morphological mismatches found between oia verb citation forms and medieval/modern indo-aryan forms, simply because it does not need to -surface patterns in phonology may be sufficient to learn the correct mapping between input and output. to investigate whether our models have learned anything from part-of-speech embeddings, we carry out an experiment where we feed held-out data to the lp, lps, and lpse models, holding the language id and semantic and etymological information constant but perturbing the pos id (i.e., changing it to verb for non-verbal data and changing it to non-verb for verbal data). we measure the normalized levenshtein distance between forms decoded with true pos ids and forms decoded with perturbed pos ids, and average these per values for verbal and non-verbal forms. as shown in ## discussion and outlook in this paper, we investigated the ability of lstmbased encoder-decoder architectures to capture recurrent patterns of sound change between oia and medieval/modern indo-aryan languages, as well as encode information regarding the genetic relationships between languages. we found that a model augmented with information regarding forms' language id, pos id, semantic profile, and etymon id showed the lowest phoneme error rate out of all models, but that language embeddings learned by this model showed low agreement with a reference taxonomy of indo-aryan, and that in general, our models struggled to capture uncontroversial genetic signal. this issue may be in part due to architectural choices we made, along with challenging aspects of the data set we used, drawn from a diverse group of languages. our experiments show that the use of different embeddings appears to allow our models to learn deviations from regular sound change that are found in words with certain parts of speech, semantic profiles, or that reflect particular etyma. at the same time, there are many avenues for improving the performance of models on this highly challenging data set. in future work, we plan to obtain glosses for reflex forms in the dictionary in order to determine whether the semantic distance between an etymon and a reflex can capture vagaries of analogical change that we were unable to model in this paper. additionally, our models learned embeddings for pos ids that did not vary across languages, rather than language-specific ones, and we built our models incrementally in a stepwise function rather than considering all possible subsets of predictors of interest, decisions that may have influenced our results. greater flexibility will play an important role in future deep learning approaches that hope to capture the multifaceted diachronic processes that yield synchronic linguistic similarity and dissimilarity.