Transformed Protoform Reconstruction (2023)

Abstract: Protoform reconstruction is the task of inferring what morphemes or words appeared like in the ancestral languages of a set of daughter languages. Meloni et al (2021) achieved the state-of-the-art on Latin protoform reconstruction with an RNN-based encoder-decoder with attention model. We update their model with the state-of-the-art seq2seq model: the Transformer. Our model outperforms their model on a suite of different metrics on two different datasets: their Romance data of 8,000 cognates spanning 5 languages and a Chinese dataset (Hou 2004) of 800+ cognates spanning 39 varieties. We also probe our model for potential phylogenetic signal contained in the model. Our code is publicly available at https://github.com/cmu-llab/acl-2023 .

## introduction languages change over time and sometimes diverge into multiple daughter languages. the common ancestor of a set of genetically related languages is their proto-language. while there are proto-languages such as latin that are attested, they are the exception 2 . reconstructed words and morphemes in proto-languages are called protoforms. the task of reconstructing unattested protolanguages is called protoform reconstruction. historical linguists reconstruct proto-languages by identifying systematic sound changes that can be inferred from correspondences between attested daughter languages (see the contributions of this paper are as follows: ## related work applying machine learning to protoform reconstruction is not new. @xcite learn an unsupervised protoform reconstruction model for the large oceanic language family using monte carlo expectation maximization @xcite , supervising the model with a gold phylogeny and using a probabilistic, generative model of sound change. @xcite apply an svm classifier to supervised reconstruction by treating sound correspondences as training examples. note that there were no word boundaries in the input matrix; that is, all sound correspondences across the training set are flattened into one matrix. furthermore, each language has an independent phonemic inventory. to learn contextual information, the authors experiment with adding features encoding the position of phonemes, among others. ciobanu and dinu (2018) learn a conditional random field @xcite using n-gram features for supervised reconstruction and ensemble 5 daughter-to-protoform models. they use a dataset of 3,218 complete cognate sets spanning latin (the proto-language) and 5 romance languages: romanian, french, italian, spanish, portuguese. @xcite employ a gru-based seq2seq approach @xcite to latin protoform reconstruction and achieve state-of-theart character edit distances. they extend @xcite 's romance data using data from wiktionary-for a total of 8,799 cognate sets across 5 romance languages plus latin-in both orthographic and phonetic (ipa) representations. in their model, all entries comprising the cognate set are concatenated together in a fixed order to form a training example. @xcite applied @xcite 's architecture to the reconstruction of middle chinese on a dataset of 5000+ cognate sets spanning 8 languages they compiled from wiktionary. 3 fourrier (2022) compares statistical machine translation, rnn, and transformer architectures for protoform reconstruction, but they evaluate their results using bleu scores @xcite instead of edit distance. they find that their transformer model did not outperform the rnn models on protoform reconstruction. in addition, their multilingual nmt (neural machine translation) model predicts many languages instead of one target language and is trained on bilingual pairs for protoform reconstruction (e.g. italian-latin and spanish-latin), unlike comparative reconstruction. in contrast, we encode the entire cognate set consisting of multiple daughter languages (5 for the romance dataset; 39 for chinese) and predict the corresponding protoform. ## datasets we train and test our model on romance and sinitic (chinese) language datasets. for romance languages, we use @xcite 's dataset which consists of 8,799 cognate sets of romanian, french, italian, spanish, portuguese words and the corresponding latin form (approximately, a protoform). there are two versions of this dataset: phonetic and orthographic. the phonetic dataset (rom-phon) represents words with ipa symbols whereas the orthographic dataset (rom-orth) represents words in the orthographic form of each language. we preserved all diacritics, except for vowel length. this dataset is an extension of @xcite 's original dataset of 3,218 cognate sets, which is not publicly available. refer to ## model we propose a transformer-based encoder-decoder architecture @xcite because such models have produced state-of-the-art results on many sequence processing tasks. transformers are by reputation data hungry, though, which poses a challenge to our problem setting, where the number of available training examples is often very small. we modify the standard encoder-decoder architecture to accommodate the structure of our datasets, where multiple daughter sequences correspond to a single protoform sequence. like @xcite , the daughter sequences are concatenated into a single sequence before being fed into the encoder. because we only care about the relative position between tokens within each daughter sequence but not across daughter sequences, positional encoding is applied to each individual daughter sequence before concatenation. along with positional encoding, an additive language embedding is applied to the token embeddings to differentiate between input tokens of different daughter languages. ## conclusion by showing that transformers can outperform previous architectures in protoform reconstruction despite the inherent data scarcity of the task, our work motivates future research in this area to take full advantage of the recent advancements in the transformer space. accurate supervised reconstruction can help pre-dict protoforms for cognate sets where linguists have not reconstructed one yet. future work could reconstruct proto-languages whose linguist reconstructions are not available, by transferring knowledge learned from languages with already reconstructed protoforms. furthermore, future work can leverage the abundance of work in unsupervised nmt to adapt our transformer model for the unsupervised setting, a more realistic scenario for the historical linguist.