Neural Unsupervised Reconstruction of Protolanguage Word Forms (2023)

Abstract: We present a state-of-the-art neural approach to the unsupervised reconstruction of ancient word forms. Previous work in this domain used expectation-maximization to predict simple phonological changes between ancient word forms and their cognates in modern languages. We extend this work with neural models that can capture more complicated phonological and morphological changes. At the same time, we preserve the inductive biases from classical methods by building monotonic alignment constraints into the model and deliberately underfitting during the maximization step. We evaluate our performance on the task of reconstructing Latin from a dataset of cognates across five Romance languages, achieving a notable reduction in edit distance from the target word forms compared to previous methods.

## introduction research has shown that groups of languages can often be traced back to a common ancestor, or a protolanguage, which has evolved and branched out over time to produce its modern descendants. words in protolanguages undergo sound changes to produce their corresponding forms in modern languages. we call words in different languages with a common proto-word ancestor cognates. the study of cognate sets can reveal patterns of phonological change, but their proto-words are often undocumented @xcite . to reconstruct ancient word forms, linguists use the comparative method, which compares individual features of words in modern languages to their corresponding forms in hypothesized reconstructions of the protolanguage. past work has demonstrated the possibility of automating this manual procedure @xcite @xcite @xcite . for example, @xcite developed probabilistic models of phonological change and used them to learn reconstructions of latin based on a dataset of romance languages, and @xcite extended their method to a large scale dataset of austronesian languages @xcite . nevertheless, previous approaches to computational protolanguage reconstruction have mainly considered simple rules of phonological change. in previous works, phonological change is modeled applying a sequence of phoneme-level edits to the ancestral form. although this can capture many regular sound changes such as lenitions, epentheses, and elisions @xcite , these edits are typically conditioned only on adjacent phonemes and lack more general contextsensitivity. phonological effects such as dissimilation @xcite , vowel harmony @xcite , syllabic stress @xcite , pre-cluster shortening @xcite , trysyllabic laxing @xcite , and homorganic lengthening @xcite , as well as many non-phonological aspects of language change @xcite , are all frequently dependent on non-local contexts. however, it is difficult to extend existing multinomial @xcite and log-linear @xcite @xcite models to handle more complex conditioning environments. motivated by these challenges, our work is the first to use neural models for unsupervised reconstruction. ancestral word forms and model parameters in previous unsupervised approaches are typically learned using expectation-maximization (e.g., @xcite . in applying neural methods to protolanguage reconstruction, we identify a problem in which the em objective becomes degenerate under highly expressive models. in particular, we find that neural models are able to express not just complex phonological changes, but also inconsistent ones (i.e., predicting vastly different edits in similar contexts), undermining their ability to distinguish between good and bad hypotheses. from a linguistic perspective, phonological change should exhibit regularities due to the constraints of the human articulatory and cognitive faculties @xcite , so we build a bias towards regular changes into our method by using a specialized model architecture and learning algorithm. we outline our approach in figure 1 . our work enables neural models to effectively learn reconstructions under expectationmaximization. in section 5, we describe a specialized neural architecture with monotonic alignment constraints. in section 6.4, we motivate training deliberately underfitted models. then, in section 7, we conduct experiments and show a significant improvement over the previously best performing method. finally, we conduct ablation experiments and attribute the improvement to ( 1 ) the ability to model longer contexts and (2) a training process that is well-regularized for learning under em. we release our code at https://github. com/andrehe02/historical_release . ## related work our work directly extends a series of previous approaches to unsupervised protolanguage recon-struction that model the probabilities of phonemelevel edits from ancestral forms to their descendants @xcite @xcite . these edits include substitutions, insertions, and deletions, with probabilities conditioned on the local context. the edit model parameters and unknown ancestral forms are jointly learned with expectation-maximization. the main difference between models in previous work is in parameterization and conditioning: bouchard-côté et al. (2007a) used a multinomial model conditioned on immediate neighbors of the edited phoneme; bouchard-côté et al. (2007b) used a featurized log-linear model with similar conditioning; and bouchard-côté et al. (2009) introduced markedness features that condition on the previous output phoneme. bouchard-côté et al. (2009) also shared parameters across branches so that the models could learn global patterns. bouchard-côté et al. (2013) used essentially the same model but ran more comprehensive experiments on a larger dataset. since the expectation step of em is intractable over a space of strings, past work resort to a monte-carlo em algorithm where the likelihood is optimized with respect to sample ancestral forms. however, this sampling step is still the bottleneck of the method as it requires computing data likelihoods for a large set of proposed reconstructions. bouchard-côté et al. (2007a) proposed a singlesequence resampling method, but this approach propagated information too slowly in deep phylogenetic trees, so bouchard-côté et al. ( 2009 ) replaced it with a method known as ancestry resampling @xcite . this method samples an entire ancestry at a time, defined as a thin slice of aligned substrings across the tree that are believed to have descended from a common substring of the proto-word. changes since the bouchard-côté et al. (2009) work, including shared parameters and ancestry resampling, are primarily concerned with reconstruction in large phylogenetic trees. while they improve reconstruction quality drastically on the austronesian dataset, these modifications did not bring a statistically significant improvement on the task of reconstructing latin from a family of romance languages @xcite . this is likely due to the romance family consisting of a shallow tree of a few languages, where the main concern is learning more complex changes on each branch. therefore, in this work we compare our model to that of @xcite but keep the single sequence resampling method from @xcite . previous work also exists on the related task of supervised protolanguage reconstruction. this is an easier task because models can be directly trained on gold reconstructions. @xcite trained a gru-based encoder-decoder architecture on cognates from a family of five romance languages to predict their latin ancestors and achieved low error from the ground truth. another similar supervised character-level sequenceto-sequence task is the prediction of morphological inflection. recent work on this task by @xcite improved output quality from out-of-the-box encoder-decoders by modifying the architecture to use hard monotonic attention, constraining the decoder's attention to obey left-toright alignments between source and target strings. in our work, we find that character-level alignments is also an important inductive bias for unsupervised reconstruction. ## task description in the task of protolanguage reconstruction, our goal is to predict the ipa representation of a list of words in an ancestral language. we have access to their cognates in several modern languages, which we believe to have evolved from their ancestral forms via regular sound changes. following prior work (e.g., @xcite , we do not observe any ancestral forms directly but assume access to a simple (phoneme-level) bigram language model of the protolanguage. we evaluate the method by computing the average edit distance between the model's outputs and gold reconstructions by human experts. concretely, let σ be the set of ipa phonemes. we consider word forms that are strings of phonemes in the set σ * . we assume there to be a collection of cognate sets c across a set of modern languages l. a cognate set @xmath0: @xmath1, consisting of one word form for each language l. we assume that cognates descend from a common proto-word x c through languagespecific edit probabilities p l (y l | x). initially, neither the ancestral forms {x c : @xmath2, @xmath3, and we wish to infer them from just the observed cognate sets c and a bigram model prior p(x). ## dataset in our setup, l consists of four romance languages, and latin is the protolanguage. we use the dataset from @xcite , which is a revision of the dataset of @xcite with the addition of cognates scraped from wiktionary. the original dataset contains 8799 cognates in latin, italian, spanish, portuguese, french, and romanian. we follow @xcite and use the espeak library foot_0 to convert the word forms from orthography into their ipa transcriptions. to keep the dataset consistent with the closest prior work on the unsupervised reconstruction of latin @xcite , we remove vowel length indicators and suprasegmental features, keep only full cognate sets, and drop the romanian word forms. the resulting dataset has an order of magnitude more data (|c| = 3214 vs. 586) but is otherwise very similar. we show example cognate sets in the appendix. ## model in this section, we describe our overall model of the evolution of word forms. we organize the languages into a flat tree, with latin at the root and the other romance languages @xmath4. following @xcite , our overall model is generative and describes the production of all word forms in the tree. proto-words are first generated at the root according to a prior p(x), which is specified as a bigram language model of latin. these forms are then rewritten into their modern counterparts at the leaves through branch-specific edit models denoted p l (y l | x). in using neural networks to parameterize the edit models, our preliminary experiments suggested that standard encoder-decoder architectures are unlikely to learn reasonable hypotheses when trained with expectation maximization. we identified this as a degeneracy problem: the space of possible changes expressible by these models is too large for unsupervised reconstruction to be feasible. hence, we enforce the inductive bias that the output word form is produced from a sequence of local edits; these edits are conditioned on the global context so that the overall model is still highly flexible. in particular, to construct the word-level edit models, we first use a neural network to model context-sensitive, character-level edits. we then construct the word-level distribution via an iterative procedure that samples many character-level edits. we describe these components in the reverse order as the character-level distributions are clearer in the context of the edit process: in section 5.1, we describe the edit process, while section 5.2 details how we model the underlying character-level edits. ## learning algorithm the problem of unsupervised reconstruction is to infer the ancestral word forms {x c : @xmath5: @xmath6: @xmath7, @xmath8. we use a monte-carlo em algorithm to learn the reconstructions and model parameters. during the e-step, we seek to sample ancestral forms from the current model's posterior distribution, conditioned on observed modern forms; during the m-step, we train the edit models to maximize the likelihood of these samples. we alternate between the e and m steps for several iterations; then in the final round, instead of sampling, we take the maximum likelihood strings as predicted reconstructions. ## discussion in this paper, we present a neural architecture and em-based learning algorithm for the unsupervised reconstruction of protolanguage word forms. given that previous work only modeled locallyconditioned sound changes, our approach is motivated by the fact that sound changes can be influenced by rich and sometimes non-local phono-logical contexts. compared to modern sequence to sequence models, we also seek to regularize the hypothesis space and thus preserve the structure of character-level edits from classical models. on a dataset of romance languages, our method achieves a significant improvement from previous methods, indicating that both richness and regularity are required in modeling phonological change. we expect that more work will be required to scale our method to larger and qualitatively different language families. for example, the austronesian language dataset of greenhill et al. ( 2008 ) contains order of magnitudes more modern languages (637 vs. 5) but significantly less words per language (224 vs. 3214) -efficiently propagating information across the large tree may be more important than training highly parameterized edit models in these settings. indeed, bouchard-côté et al. @xcite produce high quality reconstructions on the austronesian dataset by using ancestral resampling and sharing model parameters across branches. these improvements are not immediately compatible with our neural model; therefore, we leave it as future work to scale our method to settings like the austronesian languages. mous reviewers for their feedback on this project. nicholas tomlin is supported by a national science foundation graduate research fellowship, as well as the darpa lwll and semafor programs. 2008 )), and reconstruction in these datasets may benefit more from having efficient sampling algorithms and sharing parameters across branches @xcite . given the large amount of noise in the romance language dataset, we also do not overcome the restriction in @xcite of relying on a bigram language model of latin. moreover, inspecting learned sound changes is more difficult when using a neural model, so we leave a qualitative evaluation of unsupervised reconstructions from neural methods to future work.