Cognate Transformer for Automated Phonological Reconstruction and Cognate Reflex Prediction (2023)

Abstract: Phonological reconstruction is one of the central problems in historical linguistics where a proto-word of an ancestral language is determined from the observed cognate words of daughter languages. Computational approaches to historical linguistics attempt to automate the task by learning models on available linguistic data. Several ideas and techniques drawn from computational biology have been successfully applied in this area of computational historical linguistics. Following these lines, we adapt MSA Transformer, a protein language model, to the problem of automated phonological reconstruction. MSA Transformer trains on multiple sequence alignments as input and is, thus, apt for application on aligned cognate words. We, hence, name our model as Cognate Transformer. We also apply the model on another associated task, namely, cognate reflex prediction where a reflex word in a daughter language is predicted based on cognate words from other daughter languages. We show that our model outperforms the existing models on both the tasks, especially when it is pre-trained on masked word prediction task.

## introduction phonological reconstruction of a word in an ancestral proto-language from the observed cognate words, i.e., words of supposed common origin, in the descendant languages is one of the central problems in historical linguistics, a discipline that studies diachronic evolution of languages @xcite . for example, the cognate words french enfant, spanish infantes and italian infanti all trace to the proto-form infantes in latin meaning 'children', which is an attested language in this case. in most cases, the proto-language is not attested and has to be rather reconstructed. the process of arriving at such phonological reconstruction usually involves multiple steps including gathering potential cognate words, identifying systematic sound cor-respondences, and finally reconstructing the protophonemes. this procedure is known as the 'comparative method' (ringe and eska, 2013), which is traditionally carried out manually. several automated phonological reconstruction algorithms emerged in the last decade. some of these are drawn or inspired from computational biology, for example, @xcite . in general, computational historical linguistics draws techniques such as sequence alignment and phylogenetic inference from computational biology, in addition to the techniques known from historical linguistics and computational linguistics or natural language processing @xcite . on similar lines, we adapt the msa transformer, introduced in @xcite for modeling multiple sequence alignment (msa) protein sequences, for the problem of phonological reconstruction which takes as input a cognate word set in the form of msas. henceforth, we name the model introduced here as cognate transformer. we also apply our model on cognate reflex predicion task, where an unknown, i.e., a masked reflex in a daughter language is to be predicted based on the attested reflexes from other daughter languages @xcite . for instance, in the previous example, if we mask french enfant, the task would involve arriving at the word form correctly based on spanish infantes and italian infanti. one can notice that this task can serve as a pre-training objective for the proto-language reconstruction task described previously. hence, we also pre-train the cognate transformer on the cognate reflex prediction task. further, most of the existing models are fitted on a per language family basis, i.e., on one dataset at a time consisting of a single language family. thus, the utility of either transfer learning or simultaneous fitting across several language families has not yet been demonstrated. this is desirable even from the linguistic perspective since it is well known that the sound changes are phonologically systematic and, thus, often similar sound changes operate across different language families @xcite . for instance, the sound change involving palatalization of a velar consonant say /k/ > /ts/ can be observed in the case of latin caelum /kailum/ to italian cielo /tse:lo/ as well as in the supposed cognate pairs cold versus chill, which is a reminiscence of historical palatalization in old english. 1 hence, owing to the presence of commonalities across language families in terms of sound change phenomena, training models simultaneously across multiple language families should be expected to yield better results than when training on a single language family data at a time. this is well reflected in our present work. ## methodology in this section, the overall workflow is described. the input phoneme sequences are first aligned ( §3.1), the resulting alignments are trimmed ( §3.2), and then finally passed into the msa transformer with token classification head ( §3.3). in the training phase, the output sequence is also aligned while in the testing phase, trimming is not performed. the first two steps are the same as described in @xcite and are briefly described next. ## results in this section, we present and discuss in detail the results of our cognate transformer and other state-of-the-art models on the two tasks. ## conclusions in this paper, we adapted msa transformer for two phonological reconstruction tasks, namely, cognate reflex prediction and proto-language reconstruction. our novel architecture, called cognate transformer, performs either comparable to or better than the previous methods across various testtrain proportions consistently. specifically, the pretrained model outperforms the previous methods by a significant margin even at high test-train proportions, i.e., with very less trainable data reflecting a more realistic scenario. to the best of our knowledge, this work demonstrates the utility of transfer learning when applied to historical linguistics for the first time. in this paper, the data is in ipa representation, but this is not necessary as long as words can be aligned with properly defined sound classes in the respective orthographic representations. thus, relaxing the ipa input constraint can increase the amount of trainable data and pre-training with more data would most likely improve the performance of not only the problem of automated phonological reconstruction but can be demonstrated in the future for an important related task, namely automated cognate word detection. further, more standard ways of pre-training such as masking only a couple of tokens across all languages instead of a complete word of a single language can be adapted in future.