Semisupervised Neural Proto-Language Reconstruction (2024)

Abstract: Existing work implementing comparative reconstruction of ancestral languages (proto-languages) has usually required full supervision. However, historical reconstruction models are only of practical value if they can be trained with a limited amount of labeled data. We propose a semisupervised historical reconstruction task in which the model is trained on only a small amount of labeled data (cognate sets with proto-forms) and a large amount of unlabeled data (cognate sets without proto-forms). We propose a neural architecture for comparative reconstruction (DPD-BiReconstructor) incorporating an essential insight from linguists’ comparative method: that reconstructed words should not only be reconstructable from their daughter words, but also deterministically transformable back into their daughter words. We show that this architecture is able to leverage unlabeled cognate sets to outperform strong semisupervised baselines on this novel task.

## introduction in the 19th century, european philologists made a discovery that would change the direction of the human sciences: they discovered that languages change in systematic ways and that, by leveraging these systematic patterns, it was possible to reproducibly reconstruct ancestors of families of languages (protolanguages) even when no record of those languages survived. this technique, called the comparative method, provided an unprecedented window into the human past-its cultures, its migrations, and its encounters between populations. the assumption that historical changes in pronunciation ("sound changes") are regular, known as 'the regularity principle' or 'the neogrammarian hypothesis', is fundamental to the comparative with some exceptions @xcite , recent attempts at automatic reconstruction models mostly take the form of neural sequence-to-sequence transduction models similar to those used for machine translation, trained with full supervision on a dataset where every cognate set is paired with a gold reconstruction @xcite . the development of supervised reconstruction systems has given the field insights into how reconstruction of protolanguages (unattested ancestor languages) can be modelled computationally. however, in a realistic scenario, these models only become usable once the hardest part of reconstruction has been done (since they rely on the linguist having already identified enough of the sound changes in the data to reconstruct a substantial part of the lexicon). computational models of comparative reconstruction are most useful if they can be deployed without training data, or if only a small volume of labeled data is needed to prime the comparative pump. we introduce a semisupervised protoform (reconstructed parent word) reconstruction task wherein the reconstruction model has access at training time to both a small number of cognate sets (sets of daughter words-reflexes-of a single parent) labeled with a protoform and a large number of unlabeled cognate sets, mirroring the situation of historical linguists early in their reconstruction of a protolanguage. though similar to semisupervised machine translation, the semisupervised reconstruction formulation entails the absence of target-side monolingual data. most semisupervised machine translation techniques rely on monolingual data in the target language, such as back-translation @xcite and pre-trained target-side language models @xcite . in contrast, in this task, models have access only to cognate sets (no monolingual text), meaning the structure of the problem is quite different. in this paper, we propose to incorporate the comparative method into a semisupervised reconstruction model via end-to-end multi-task learning. our proposed model, named dpd-bireconstructor, learns to improve its reconstructions by performing reflex predictions on an intermediate representation of its predicted reconstructions. reflex prediction losses are propagated into the reconstruction network, allowing the model to train on cognate sets without protoform labels. a hypothetical example from three tangkhulic languages is shown in ## results cross-strategy comparisons: interestingly, while consistent with our hypothesis that access to different subsets of labels affects learning outcomes, we observe high variations in performance between dataset groups. figures 6 and 7 visualize the performance of strategy-architecture combination by group, revealing that most strategies tend to do better on some dataset seeds. performance for varied labeling settings: despite only being tuned with 10% of labels, dpdbased strategies generalized to other labeling settings, often outperforming strong baseline strategies on all metrics on at least one of the architectures (see appendix f for detail). we see a nonlinear scaling between performance and the percentage of labels and notice higher performance variations between strategies for lower percentages of labels. at a 5% labeling setting, for example, gru-dpd-πm-bst almost doubles the accuracy of gru-supv on wikihan. at a 30% labeling setting, some strategies attain accuracy close to existing fully supervised reconstruction models, with trans-dpd-πm 5.14 percentage points be- ## related work computational historical linguistics: protoform reconstruction and reflex prediction are two central tasks in computational historical linguistics. protoform reconstruction predicts the protoform given reflexes in a cognate set, while reflex prediction models the changes from the protoform to its reflexes. computational reconstruction and reflex prediction methods vary and include rule-based systems @xcite , probabilistic models operating on phylogenetic trees @xcite @xcite , automated alignment systems @xcite , and more recently, neural networks (fourrier, 2022). supervised reconstruction and reflex prediction: ciobanu and dinu (2018) and ciobanu et al. (2020) formulate protoform reconstruction as a sequence labelling task and use conditional random field to reconstruct protoform phonemes at each position in the daughter sequences. meloni et al. ( 2021 ) reformulates the reconstruction task as a sequence-to-sequence task by concatenating reflexes into a single input sequence separated by language tags and uses gru to reconstruct latin on a new romance dataset. a group of subsequent researchers refined this task with additional datasets @xcite and improved neural methods @xcite . reflex prediction can be viewed as sequenceto-sequence transduction in the reverse direction. cathcart and rama (2020) propose an lstm encoder-decoder model to infer indo-aryan reflexes from old indo-aryan, aided by semantic embedding. @xcite replicates cathcart and rama (2020)'s experiments on south asia languages with both gru and transformer models. as reflex prediction maps from protolanguage to multiple daughter languages, a prompting token is often attached to the input to specify the target daughter language. non-supervised computational historical linguistics: past work in which comparative reconstruction was performed without full supervision included @xcite and @xcite . to the best of our knowledge, no research focuses specifically on semisupervised neural reconstruction. semisupervised learning: effective semisupervised learning should utilize unlabeled training data. one approach is proxy-labelling, by which synthesized labels are added to unlabeled training examples via heuristics @xcite . another approach is consistency regularisation, largely based on the smoothness assumption that is applicable regardless of whether labels are present: similar input data in high-density regions should have similar labels, whereas input data separated by low-density regions should not @xcite . our task is related to semisupervised machine translation @xcite . however, it differs crucially in that the model has no access to monolingual text, only labeled and unlabeled cognate sets. ## conclusion we introduce the task of semisupervised reconstruction, marking a step forward toward a practical computational reconstruction system that can assist early-stage protolanguage reconstruction projects. we design the dpd-bireconstructor architecture to implement historical linguists' comparative method, yielding performance that surpasses existing sequence-to-sequence reconstruction models and established semisupervised learning techniques, especially when protoform labels are scarce.