The SIGTYP 2022 Shared Task on the Prediction of Cognate Reflexes (2022)

Abstract: This study describes the structure and the results of the SIGTYP 2022 shared task on the prediction of cognate reflexes from multilingual wordlists. We asked participants to submit systems that would predict words in individual languages with the help of cognate words from related languages. Training and surprise data were based on standardized multilingual wordlists from several language families. Four teams submitted a total of eight systems, including both neural and non-neural systems, as well as systems adjusted to the task and systems using more general settings. While all systems showed a rather promising performance, reflecting the overwhelming regularity of sound change, the best performance throughout was achieved by a system based on convolutional networks originally designed for image restoration.

## introduction in historical-comparative linguistics, scholars typically assemble words from related languages into cognate sets. in contrast to the notion of cognacy in language teaching and synchronic nlp applications, cognate sets are understood as sets of words that share a common origin regardless of their meaning in historical-comparative linguistics and that should not contain borrowed words. the individual members of a cognate set are typically called cognate reflexes or simply reflexes @xcite . cognate reflexes typically show regular sound correspondences. this means that one can define a mapping across the individual phoneme systems of the individual languages. thus, english t typically corresponds to a german ts (compare ten vs. zehn), and english d corresponds to german t (compare dove vs. taube). the mappings often depend on certain contextual conditions and may differ, depending on the position in which they occur in a word. with the help of regular sound correspondences, linguists can often predict fairly well how the cognate counterpart of a word in one language might sound in another language. however, prediction by linguists rarely takes only one language pair into account. the more reflexes a cognate set has in different languages, the easier it is to predict reflexes in individual languages. ## systems four teams submitted their systems for our shared task. since these systems are described in individual papers @xcite @xcite , we will only briefly present their main features here. team crosslingference, represented by gerhard jäger (university tübingen), provided a workflow jäger 2022, implemented in the julia programming language, that makes specifically use of bayesian phylogenetic inference. in contrast to the remaining systems submitted to our shared task, jäger's approach takes phylogenetic information into account, extending an earlier workflow for phonological reconstruction @xcite . team mockingbird, represented by christo kirov, richard sproat, and alexander gutkin (google research), provided two models for the prediction of cognate reflexes. the first model, the neighbor transformer model, was originally designed to find problems in the readings of japanese place names spelled in kanji @xcite , and is based on the popular transformer architecture @xcite , which was specifically adjusted for the task. since the training data would be too small for the transformer model, the authors augmented it with new instances generated by randomly sampling subsets of a corresponding cognate set. in addition to that, they also enriched each set with synthetic instances using n-gram language modelling. the second model, the image inpainting model, compares the cognate reflex prediction task to the task of restoring corrupted parts of a 2d image, in which dimensions correspond to languages and cognate phonemic representations. the restoration is achieved with the help of convolutional neural networks @xcite . for this model, no data augmentation steps were undertaken. the authors provide four model configurations of the neighbor model, with the first three (n1-a, n1-b, and n1-c) differing in the number of training steps and not being publicly released, while the last one (n2), which was only applied to the 0.1 proportion of the data, being pub-licly released. for the image inpainting model, one configuration was provided (i1). team leipzig, represented by giuseppe g. a. celano (university leipzig), provided a trans-former-based architecture with character and position embeddings for the prediction of cognate reflexes @xcite , in which language information was one-hot encoded and the model was trained on individual reflex pairs on each language independently. in order to predict a word from several reflexes in different languages, the system first predicts individual target tensors of probabilities for each attested reflex and then averages them to produce the prediction. team ceot, represented by tiago tresoldi (uppsala university), provided a workflow that predicts cognate reflexes based on phonetic alignments @xcite , which is quite similar to the extended baselines of our shared task @xcite . in contrast to our baseline approaches, their system extalign-rf skips the trimming procedure (stage 2), varies the techniques for alignment enrichment by taking preceding and following context into account (stage 3), and uses a random forests classifier rather than a support vector machine. while all teams tried hard to provide results for all of their systems, some results could not be computed in time, to be included in the shared task. all teams were asked to share their data in such a way that users can easily replicate the results and also apply their methods to new data. unfortunately, there was no time for the team organizing the shared task to individually check all systems with respect to replicability and transparency. the team checked, however, that all systems were properly archived with repositories offering long-term storage of data, such as zenodo, and we communicated the importance of replication with all authors. ## results given that we measure system performance with four evaluation measures (edit distance, normalized edit distance, b-cubed f-scores, and bleu scores adjusted for word prediction), one might expect that systems perform differently with respect to different evaluation measures. as can be seen from the results in overall, all systems do quite a good job at recovering unknown words from their cognate sets, specifically in those cases, where only a small part of the test data was retained for the evaluation process. judging from our practical experience and independently published results on word prediction experiments @xcite , b-cubed f-scores higher than 0.7 and average edit distances of about 1 provide a good starting point for computer-assisted approaches and can already provide active help in various practical annotation tasks in historical linguistics. thus, scholars working on the reconstruction of certain language families could use predicted proto-forms and later manually correct them, or field workers could use automatically predicted words when trying to elicit specific lexical items to search for cognate words that might have shifted their meanings. ## discussion it was one of the crucial insights made by historical linguists in the early 19th century @xcite , that sound change proceeds in a surprisingly regular, systematic manner, affecting all sounds in the lexicon of a language that recur in similar phonotactic positions. without the systematicity and regularity of sound change, it would not be possible to predict the pronunciation of words in one language based on the pronunciation of cognate words in related languages. while it has been known for a long time to linguists that these kinds of predictions can be made on the basis of historical language comparison, the task of cognate reflex prediction has only recently attracted the attention of scholars working in the field of natural language processing and computational linguistics. with our shared task on cognate reflex prediction, we hoped to achieve two major goals. on the one hand, we wanted to highlight the importance of classical scholarship for computational applications in historical linguistics and linguistic