Probing Multilingual Cognate Prediction Models (2022)

Abstract: Character-based neural machine translation models have become the reference models for cognate prediction, a historical linguistics task. So far, all linguistic interpretations about latent information captured by such models have been based on external analysis (accuracy, raw results, errors). In this paper, we investigate what probing can tell us about both models and previous interpretations, and learn that though our models store linguistic and diachronic information, they do not achieve it in previously assumed ways.

## introduction in historical linguistics, cognates are words that share a common etymological origin in a common parent language. galician, portuguese and spanish gato, catalan and occitan gat, italian gatto, french chat and aromanian cãtushi, all meaning 'cat', as well as romanian cȃtus , ȃ 'manacle', foot_0 are cognates, as they all descend from the same word cattus 'cat' in their mutual parent language, latin. the parent word form cattus is called the proto-form. comparing the phonetic form of sets of cognates allows to identify patterns: in our example, initial [g] in galician to italian corresponds to [s] in french and [k] in romanian and aromanian. if said pattern is attested in more cognate sets, it is then considered to be a sound correspondence pattern, which emerge in related languages from the application of minimal, regular and exceptionless sound changes rules to the ancestral proto-forms. @xcite such sound correspondence patterns then help finding new cognates. the cognate prediction task aims at predicting, from a phonetised word, the plausible phonetic form of its cognate in a related language, according to known sound correspondence patterns; this has many applications, from identifying new words with field linguists (bodt et al., 2018; bodt and list, 2019) to inducing translation lexicons for lowresourced languages (mann and yarowsky, 2001). @xcite this task has been modelled as a sequence to sequence character level machine translation task in the most recent papers studying it (see the survey on cognate prediction in dekker and zuidema (2021)), which drew linguistic conclusions on the latent information learnt by such models by studying their outputs in a 'black-box' fashion. however, no paper that we know of tried to confirm or inform these conclusions by using modern interpretability tools, such as probing tasks, hidden representation analysis, or inner components analysis. in this paper, we therefore investigate whether the linguistic conclusions previously reached 1) can be reproduced, 2) hold under the scrutiny of modern interpretability techniques, and 3) can be extended. we first train several neural cognate prediction models, @xcite and analyse their outputs as such. then, we focus on applying modern interpretability techniques, and compare the insights they provide with prior hypotheses. ## synchronic probing using previously defined probes, we study whether our models learn synchronic linguistic information. ## conclusion after training and selecting the best multilingual machine translation models for the task of cognate prediction, we confirmed the black-box analysis previously made of similar models (they capture language relatedness information and phonetic similarity). we then probed our models and discovered that latent linguistic information learned by the model seemed to encode a phonetic 'contextual language model' rather than explicit phonology or phonotactics. we also discovered that our mod-els learn diachronic information: they are able to produce sound correspondences, and, even more interestingly, they contain enough historical linguistic information to allow the reconstruction of the proto-form with no fine-tuning, performing at least as well as models trained specifically for this task. we can therefore conclude that synchronic multilingual cognate prediction models learn latent diachronic information, though further work is needed to understand more precisely under which form this information is stored.