Ab Antiquo: Neural Proto-language Reconstruction (2021)

Abstract: Historical linguists have identified regularities in the process of historic sound change. The comparative method utilizes those regularities to reconstruct proto-words based on observed forms in daughter languages. Can this process be efficiently automated? We address the task of proto-word reconstruction, in which the model is exposed to cognates in contemporary daughter languages, and has to predict the proto word in the ancestor language. We provide a novel dataset for this task, encompassing over 8,000 comparative entries, and show that neural sequence models outperform conventional methods applied to this task so far. Error analysis reveals a variability in the ability of neural model to capture different phonological changes, correlating with the complexity of the changes. Analysis of learned embeddings reveals the models learn phonologically meaningful generalizations, corresponding to well-attested phonological shifts documented by historical linguistics.

## introduction historical linguists seek to identify and explain the various ways in which languages change through time. research in historical linguistics has revealed that groups of languages (language families) can often be traced into a common, ancestral language, a "proto-language". large-scale lexical comparison of words across different languages enables linguists to identify cognates: words sharing a common proto-word. comparing cognates makes it possible to identify rules of phonetic historic change, and by back-tracing those rules one can identify the form of the proto-word, which is often not documented. that methodology is called the comparative method @xcite , and is the main tool used to reconstruct the lexicon and phonology of extinct languages. inferring the form of proto-words from existing cog- * equal contribution nates in daughter languages is possible since historical sound changes within a language family are not random. rather, the phonological change is characterized by regularities that are the result of constraints imposed by the human articulatory and cognitive faculties @xcite . for example, we can find such regular change-commonly called "systematic correspondence"-by looking at the evolution of the first phoneme of latin's word for "sky": foot_0 @xcite . this pattern is systematic, and will be found throughout the languages. working this way, historical linguists reconstruct words in the protolanguage from existing cognates in the daughter languages, and determine how words in the protolanguage may have sounded. to what extent can a machine-learning model learn to reconstruct proto-words from examples in this way? and what generalizations of phonetic change will it learn? we focus on the task of proto-word reconstruction: the model is trained on sets of cognates and their known proto-word, and is then tasked with predicting the proto-word for an unseen set of cognates. our study concentrate on the romance language family 2 and the model is trained to reconstruct the latin origin. we show that a recurrent neural-network model can learn to perform this task well (outperforming previous attempts). 3 more interesting than the raw performance numbers are the learned generalizations and error patterns. the romance languages are widely studied @xcite ; @xcite ; @xcite among others), and their phonological evolution from latin is well mapped. the existence of this comprehensive knowledge allows exploring to what extent neural models internalize and capture the documented rules of language change, and where do they deviate from it. we provide an extensive error analysis, relating errors patterns to knowledge in historical linguistics. this is often not possible in common nlp tasks, such as parsing or semantic inference, in which the rules governing linguistic phenomenaor even the suitable framework to describe themare still in dispute among linguists. contributions inspection of existing datasets of cognates in romance languages has revealed inherent problems. we thus have collected a new comprehensive dataset for performing the reconstruction task ( §4). besides the dataset, our main contribution is the extensive analysis of what is being captured by the models, both on orthographic and phonetic versions of the dataset ( §6). we find that the error patterns are not random, and they correlate with the relative opacity of the historic change. these patterns were divided in different categories, each one motivated by a sound phonological explanation. moreover, in order to further evaluate the learning of rules of phonetic change, we evaluated models on a synthetic dataset ( §6.3), showing that the model is able to correctly capture several phonological change rules. finally, we analyze the learned inner representations of the model, and show it learns phonologically meaningful properties of phonemes ( §6.4) and attributes different importance to different daughter languages ( §6.5). ## related work the related task of cognates detection has been extensively studied. in this task, a set of cognates should be extracted from word lists in different languages. most effort in machine learn-ing approaches to this task has been focused on distance-based methods, which quantify the distance (according to some metric), or the similarity, between a given candidate of cognates. the similarity can be either static (e.g. levenshtein distance) or learned. once the metric is established, a classification can be performed either based on hard-decision (words below a certain threshold are considered cognates) or by learning a classifier over the distance measures and other features @xcite @xcite @xcite ; @xcite have evaluated an alternative approach, in which explicit rules of transformation are derived based on edit operations. see @xcite for a recent evaluation of the performance of several cognates detection algorithms. several studies have gone beyond the stage of cognates extraction, and used resulted list of cognates to reconstruct the lexicon of protolanguages. most studies in this direction borrowed techniques from computational phylogeny, drawing a parallel between the hypothesized branching of (latent) proto words into their (observed) current forms and the gradual change of genes during evolution. @xcite has applied such a model to the development of the romance languages, based on a dataset composed of aligned-translations. @xcite used an extensive dataset of austronesian languages and their reconstructed proto-languages, and built a parameterized graphical model which models the probability of a phonetic change between a word and its ancestral form; the probability is branch-dependent, allowing for the learning of different trends of change across lineages. while achieving impressive performance, even without necessitating a cognates lists as an input, their model is based on a given phylogeny tree that accurately represents the development of the languages in question. @xcite have automatically constructed cognate datasets for several languages, including romance languages, and used a character-level nmt system to complete missing entries (not necessarily the proto-form). several works studied the induction of multilingual dictionaries from partial data in related languages. @xcite reconstruct cognates in austronesian languages (where the proto-language is not attested). @xcite employ a mixture-of-experts approach for lexical translation induction, combining neural and probabilistic methods, and @xcite translate from a multi-source input that contains partial translations to different languages, concatenated. finally, ciobanu and dinu (2018) have applied a crf model with alignment to a dataset of romance cognates, created from automatic alignment of translations @xcite . the researchers also applied rnns on the same dataset, but reported negative results. ## proto-word reconstruction our proto-word reconstruction is as follows: the training set is composed of pairs (x i ,y i ), where each x @xmath0, ..., c ℓn i is a set of cognate words, each tagged with a language ℓ j , and y i is the protoword (latin word) of that set. we consider an orthographic task, where the cognates and protowords are spelled out as written. as the orthography is often arbitrary and more conservative than spoken language, we consider also a phonetic task, in which the cognates and proto-words are represented as their phonetic transcriptions into ipa. an example of a training instance (x, y) for the orthographic task is: ## comprehensive romance dataset the different experiments described in the paper were performed on a large dataset of our creation, which contained cognates and their proto-words in both orthographic and phonetic (ipa) forms. the dataset's departure point is @xcite , which consists of 3,218 complete cognate sets in six different languages: french, italian, spanish, portuguese, romanian and latin foot_3 . we augmented the dataset's items with a freely available resource, wiktionary, whose data were manually checked against diez and donkin (1864) to ensure their etymological relatedness with the latin source. the entries were transcribed into ipa using the transcription module of the espeak library foot_4 , which offers transcriptions for all languages in our dataset, including latin. the final dataset contains 8,799 cognate sets (not all of them complete), which were randomly splitted into train, evaluation and test sets: 7,038 cognate sets (80%) were used for training, 703 (8%) for evaluation and 1,055 (12%) for testing. overall, the dataset contains 41,563 distinct words for a total of 83,126 words counting both the orthographic and the phonetic datasets. vowel lengths were found to be difficult to recover (see a detailed description of the dataset collection process is available at the appendix §a.1. we make our additions to the dataset of ciobanu and dinu (2014b) publicly available foot_5 . ## results and analysis this disparity can be explained at least partially by a peculiarity of the phonetic dataset: it implicitly encodes vowel length, which was neutralized in the orthographic dataset. the reason for this difference is that length contrast in latin co-occurred with quality differences: short vowels tended to be more open than their long counterparts, a contrast also called "tense-lax" @xcite . this contrast is not present in latin orthography, but it appears in its phonetic transcription. this results in a noticeable gap between the results of the orthographic dataset with vowel lengths and without vowel lenghts (0.064 average normalized edit distance vs. 0.119), while the differences between the phonetic ipa dataset with vowel lenghts and without vowels lengths are much smaller. when the contrast "tense-lax" is manually neutralized foot_7 , the performances achieved are similar to the ones on the orthographic dataset (as it is possible to see from the performances on "ipa, no contrast", whose latin entries do not contain a "tense-lax" ## conclusions in this work, we introduce a new dataset for the task of proto-word reconstruction in the romance language family, and used it to evaluate the ability of neural networks to capture the regularities of historic language change. we have shown that neural methods outperform previously suggested models for this task. analysis of the linguistic generalizations the model acquires during training demonstrated that the mistakes are related to the complexity of the phonetic change. a controlled experiment on a set of rules for phonetic alternations between latin and its daughter languages demonstrated the model internalizes some of the systematic processes that latin had undergone during the evolution of the romance languages. visualizing the learned phonemeembedding vectors has revealed a hierarchical division of phonemes that reflects phonological realities, and inspection of attention patterns demonstrated the model attributes different importance to different languages, in a position-dependent manner. while the task examined in this paper is commonly called "proto-word reconstruction", in practice the task the model faces is considerably less challenging than the work of historical linguists, as the model is trained in a supervised setting. a future line of work we suggest is applying neural models for the end task of proto word reconstruction, without relying on cognates lists, in a way that would more naturally model the historical linguistic methodology.