Mockingbird at the SIGTYP 2022 Shared Task: Two Types of Models for the Prediction of Cognate Reflexes (2022)

Abstract: The SIGTYP 2022 shared task concerns the problem of word reflex generation in a target language, given cognate words from a subset of related languages. We present two systems to tackle this problem, covering two very different modeling approaches. The first model extends transformer-based encoder-decoder sequence-to-sequence modeling, by encoding all available input cognates in parallel, and having the decoder attend to the resulting joint representation during inference. The second approach takes inspiration from the field of image restoration, where models are tasked with recovering pixels in an image that have been masked out. For reflex generation, the missing reflexes are treated as “masked pixels” in an “image” which is a representation of an entire cognate set across a language family. As in the image restoration case, cognate restoration is performed with a convolutional network.

## introduction the cognate reflex prediction task can be understood by considering a simple example. english dream is droom in dutch and traum in german. english stream is stroom in dutch and strom in german (so we can see that there is a bit of variation in how the cognate forms are realized). but now consider english tree, which is boom in dutch and baum in german. if there were a cognate in english, what would it look like? on analogy in particular with the dream example, one would expect the form to be beam-which is in fact cognate with the dutch and german forms, though the meaning has shifted. this study describes the two particular approaches to cognate reflex prediction code-named mockingbird. 1 both approaches use popular ma-chine learning techniques adapted to the cognate reflex prediction task. the transformer-based approach popularized by @xcite is a particular instance of sequence-to-sequence (seq2seq) recurrent neural bipartite encoder-decoder architecture @xcite equipped with multi-head attention mechanism. it has the advantages inherent to seq2seq models. in particular it can represent arbitrarily long-distance contextual dependencies both within and across words. this rich representational capacity comes at the cost of high model complexity and need for computational resources @xcite @xcite . our particular transformer-based model was originally developed for place name pronunciation @xcite , and models cognate sets as a neighborhood where we are trying to predict the pronunciation of a target feature given its neighbors. the image inpainting model @xcite relies on a simple convolutional neural network, or @xcite , which trains fairly quickly even on cpu. it treats cognate sets as holistic objects ("images"), with individual convolution filters representing partial joint alignments between all languages at once. unlike transformers or rnn-based models, however, convolutional models with finite kernel sizes cannot capture arbitrary amount of context. as an extreme example, it would be difficult for a convolutional model to learn that the first character in one language cognate should always be aligned with the last character in another language's cognate, especially if long words are possible. ## related work the establishment of cognate correspondences and the prediction of cognate forms has a long and venerable history that dates at least to the original work of william jones that established the indo-european language family @xcite , and later the work of the neo-grammarians @xcite . prior computational work related to this problem includes early work by @xcite and @xcite on methods for aligning cognate pairs at the segment level, and more recently work specifically on the prediction of cognate forms @xcite . the work of @xcite in particular is most similar to the current proposed methods in that they use a neural character-level encoderdecoder sequence-to-sequence model, and demonstrate that this shows good performance on the task of proto-form reconstruction for a fairly large dataset of romance languages. ## cognate reconstruction as image inpainting the neighborhood model presented above casts the reflex generation task as a straightforward extension of seq2seq modeling. here, we take a different tack, noting that if we treat the entire set of cognates as a unit (an "image"), then the task of generating unknown cognates is analogous to recovering missing/masked/corrupted parts of that image. in the field of image reconstruction, this is sometimes referred to as image inpainting @xcite @xcite . one of the popular state-of-the art methods employs convolutional neural networks (cnns) to recover missing pixel values. in this work we apply one such architecture by @xcite from nvidia to cognate generation. as we see below, cognate set "images" contain relatively few "pixels", so we can get away with small networks with a single pair or convolution and deconvolution layers that are fast to train and evaluate, even on cpu. ## results and discussion the sigtyp 2022 shared task was evaluated over 20 language-family specific datasets taken from the lexibank repository @xcite . each dataset consisted of a series of cognate groups presented in cldf @xcite format, with each cognate form stored as an ipa phonetic transcription. 10 datasets were provided for system development, and 10 "surprise" language families were held aside for final evaluation. furthermore, each dataset was provided in five sparsity conditions (dropping 10%, 20%, 30%, 40%, and 50% of the available cognate forms). for brevity, we only discuss the 10% ("dense") and 50% ("sparse") conditions in this paper. 9 we submitted five system configurations, shown in there are four neighbors transformer configurations. the first three configurations (n1-a, n1-b and n1-c) correspond to the models built using our internal pipeline. the only difference between these configurations is the number of training steps: 25,000 (n1-a), 35,000 (n1-b), and 100,000 (n1-c). for these configurations we mapped the language names to unique emojis during training and inference. the final transformer configuration n2 corresponds to the model trained using the released open-source pipeline. the training regime has some notable differences with the other transformer configurations. first, as noted above, no language name-to-emoji mapping was performed. second, when generating each random example {(p t , l t ), (p n , l n )} consisting of target t and neighbor n pronunciation/language pairs, the neighbor pronunciation p n is randomly generated using a first-order markov chain trained from all the bigrams constructed from the pronunciations available for language l n , as opposed to unigrams used by the n1 systems. also, the target pronunciation p t is generated from p n by sampling from the distribution obtained using sound class-based pairwise alignment algorithm implementation from @xcite described in @xcite . finally, the stopping criterion for the training process was rather ad-hoc: training for each language group was stopped after eyeballing the training set loss. unlike the rest of the 9 please see https://github.com/sigtyp/st2022/tree/ main/results for all the available results. 10 https://github.com/sigtyp/st2022/tree/main/ systems submitted systems, the n2 configuration was only trained in the 10% ("dense") sparsity condition. for the inpainting model, results were produced via a majority ensemble. for each language family, ten models were trained using the procedure described above corresponding to ten random 80%/20% train/dev splits of the available training data. predictions for each test sample were obtained from each of the models, and the most common prediction across the set was retained. model results were evaluated according to four metrics selected by the shared task organizers. the first two are raw and normalized (divided by the number of characters in the target form) levenshtein edit distances @xcite between the predicted and expected test cognate forms. for these metrics, lower is better as it indicates a closer match. the third metric, b-cubed f-scores, is designed to avoid overly penalizing systematic errors a system might make, discounting errors that occur across multiple trials @xcite . the metric derives from the b-cubed measure @xcite frequently used in historical linguistics to evaluate automatic cognate detection techniques @xcite . for this metric, higher is better. finally, standard bleu scores @xcite as used when evaluating machine translation (again, higher is better) were included. tables 2 and 3 summarize the evaluation in the dense and sparse data conditions. overall, both the inpainting and neighbor n1 models match or improve upon the baseline method provided by the shared task organizers. the inpainting model shows an overall advantage -its simplicity might mitigate against overfitting in such small datasets, but this isn't universal across all language families, and wanes as the task becomes more difficult moving from the dense to the sparse condition. in particular, the neighbor models are very effective in the felekesemitic language family in the sparse data condition, while the inpainting model behaves at baseline level. this could be due to the unique morphology of semitic languages, and the ability of the n1 models to use contextual cues that the simpler model architectures aren't able to represent, but which compensate for data sparsity to an extent. the condition may also benefit more than most from the data augmentation strategy used in training the neighbor models. in terms of overall results on the "surprise" ## conclusions we presented two approaches to the problem of cognate reflex prediction, one based on the transformer seq2seq architecture, and one based on convolutional networks. both approaches stem from natural, intuitive interpretations of the problem. the "neighbors" transformer approach treats the problem as one of reconstructing the phonetic form of a cognate by considering all the other cognates in the set, on analogy to the problem of reconstructing the reading of a geographical feature on the basis of the pronunciation of names of geographic neighbors. the inpainting approach treats the problem as being similar to filling in missing pixels in an image on the basis of the surrounding context pixels. we submitted 5 system variants (1 convolutional model and 4 transformer models) to the sigtyp 2022 shared task, where they performed well relative to the provided baseline and other submissions.