Automated Cognate Detection as a Supervised Link Prediction Task with Cognate Transformer (2024)

Abstract: Identification of cognates across related languages is one of the primary problems in historical linguistics. Automated cognate identification is helpful for several downstream tasks including identifying sound correspondences, proto-language reconstruction, phylogenetic classification, etc. Previous state-of-the-art methods are mostly based on distributions of phonemes computed across multilingual wordlists and make little use of the cognacy labels that define links among cognate clusters. In this paper, we present a transformer-based architecture inspired by computational biology for the task of automated cognate detection. Beyond a certain amount of supervision, this method performs better than the existing methods, and shows steady improvement with further increase in supervision proving the efficacy of utilizing the labeled information. We also demonstrate that accepting multiple sequence alignments as input and having an end-to-end architecture with link prediction head saves much computation time while simultaneously yielding superior performance.

## introduction words in genetically related languages with same descendance from a common ancestral language are termed as cognates. for example, sanskrit bhava and english be are cognates reconstructed as *b h ewh-in ancestral proto-indo-european. within historical linguistics, assembling potential cognates forms an essential step in the comparative method to proceed to further stages such as formulation of sound laws, reconstruction of proto-language, phylogenetic reconstruction, etc. @xcite . cognate identification has been traditionally carried out by tedious manual cross-comparisons of lexica across several concepts or meanings; this often requires sufficient linguistic expertise in the languages that are being compared. automated cognate detection attempts to alleviate manual labor and, thus, assists a historical linguist to quickly produce high-quality etymologies required for downstream tasks. over the past decade, several methods for automated cognate detection, mostly using sequence alignment and other techniques inspired by bioinformatics and evolutionary biology @xcite , have appeared. the best-performing methods primarily depend on similarity scores computed from distributions of phonemes in multilingual wordlists @xcite and make little or no use of the cognacy labels except for a clustering task at the end. in this paper, we advocate for a supervised learning scenario that utilizes the labeled information to the fullest. we demonstrate that such a scenario combined with the representational power of an appropriate deep neural network architecture can outperform previous methods above a certain amount of supervision. we also demonstrate that such a model is also capable of transfer learning. in other words, once trained on some data, it can perform well on any dataset unseen so far with little additional supervision. the typical procedure followed by the stateof-the-art methods for this problem is as follows. in each language family, attested words from all languages that have the same meaning, i.e., concept, are clustered based on the pairwise similarity measures computed by the respective procedure. we propose a different approach where instead of clustering based on pairwise similarity we directly take input a multiple sequence alignment (msa) of words of the same concept and predict linkage via an end-to-end architecture. this approach proves to be much better in performance and much faster than clustering from independent pairwise similarity measures. many of the algorithms in computational historical linguistics are heavily drawn or inspired by computational biology. continuing the trend, we adopt cognate transformer (akavarapu and @xcite , which yielded state-of-theart performance in automated phonological reconstruction task, as the base architecture. cognate transformer was adapted from msa transformer @xcite , a protein language model that excels in contact predictions. we additionally append to this architecture layers consisting of triangular multiplication and triangular attention modules inspired by alphafold2 @xcite , the state-of-the-art protein structure predictor, where the modules roughly capture triangle inequalities among the distances between amino acid residues. for our task, we applied these modules for capturing transitivity property among linkages in cognate clusters. we find that the addition of this particular module has a significant share in the performance of the overall architecture. our key contributions are as follows: the rest of the paper is organized as follows. related work is mentioned in §2. the problem statement is elaborated in §3. the methodology is described in §4. the details of the experimental setup including the datasets used, previous baselines, and evaluation measures are described in §5. the results of experiments and ablation studies along with error analyses and discussions are given in §6. finally, the article is concluded in §7. computational historical linguistics is a young field that emerged over the past two decades. notable works that lead to significant progress in automatic cognate detection are as follows. consonant class method of turchin et al. (2010) deems two words as cognate if the first two consonants fall under the same consonant class. in sound-class-based phonetic alignment (sca) of list @xcite , pairwise phoneme sequences are aligned and scored for similarity using sound classes that extend consonant classes. lexstat @xcite aligned and scored pairwise sequences using language phonemic-specific distributions combined with sca-based scores. the pairwise similarities thus obtained are clustered using upgma @xcite . the previous stateof-the-art results are attributed to lexstat combined with infomap clustering @xcite . equivalent performance was also reported in @xcite using chinese restaurant clustering. an expectation-maximization method over pairwise phonemic distributions is also found to yield similar performance (macsween and caines, 2020). information-weighted similarity measure was proposed by @xcite which reported a slight increase in evaluation scores over lexstat, albeit tested only on one dataset. supervised algorithms include the siamese-cnn-based model by @xcite which performs binary classification on a given pair of words. jäger et al. (2017 ## automated cognate detection the automated cognate detection problem statement is described here as follows. the gold data for a language family f , comprising of related languages l 1 , l 2 , . . . ∈ f , consists of words over several concepts, i.e., meanings, say m 1 , m 2 , . . . , etc. each word is a sequence of phonemes. for each concept m m , there are words w m i for several languages l i in that family, where w m i is a word of a language l i in concept m m . words in each concept are associated with labels say c m @xmath0. a single such cluster of words is called a cognate set. we also define links l k @xmath1, 1} between languages l i and l j for a concept m m which indicate if the corresponding words are cognates i.e., have the same cluster label. in other words, the goal of automated cognate detection is to correctly cluster a given set of words that mean a single concept in a language family. in a supervised setting, the aim is to predict the linkages correctly. for an illustration of the overall problem, consider the indo-european language family and the concept of 'all'. the attested lexica in the member languages are sanskrit sárve (vedic víśve), greek (ancient) hólos, latin omnes, german alle, english all, russian vse, czech vše, etc. among these vedic víśve, russian vse, czech vše form a cluster, i.e., a cognate set while sanskrit sárve and greek hóla form another cognate set. similarly, english skt. -s @ r v e -gr. ## methodology the overall workflow is described as follows. given some words from different languages for a concept in a language family, the words are first aligned ( §4.1), then converted into tokens and passed into the cognate transformer ( §4.2), whose outputs are converted into pairwise (along language axis) representations by outer product mean module ( §4.3), which are then passed into the layers of pairwise module ( §4.4) whose outputs are classified into two labels 0 or 1 indicating the pairwise linkage among the languages ( §4.5). since the linkage information is known in the form of cognacy labels, the architecture described can be thus trained end-to-end. the overall architecture is illustrated in figure 1 . ## experimental setup in this section, the details of the experiments including datasets, implementation, evaluation metrics, baseline models, etc. are described. ## results the results are summarized in ## conclusions in this paper, we have proposed a transformerbased model inspired by evolutionary biology for the task of automatic cognate detection. the model can harness efficiently the labeled data and consequently, with sufficient data, outperforms existing approaches that do not make efficient use of supervision data. in particular, better results are obtained with only 20 concepts per family on some of the datasets. to the best of our knowledge, we proposed for the first time in this particular problem a method that directly outputs link probabilities, i.e., pairwise similarities from an input msa in an end-to-end fashion, unlike all the previous methods which act on aligned pairs of words. we demonstrated through the primary results and ablation studies that this approach of inputting msa rather than paired alignments results not just in a significant increase in performance but also in drastically reducing the computation time. we have also demonstrated by observing few outputs that the model is capable of learning regular sound changes from just two example instances in the data for a particular sound change. evaluation of cognate transformer on phylogenetic reconstruction task @xcite is an unexplored problem and, thus, can be a potential topic of future work.