A Transformer Architecture for the Prediction of Cognate Reflexes (2022)

Abstract: This paper presents the transformer model built to participate in the SIGTYP 2022 Shared Task on the Prediction of Cognate Reflexes. It consists of an encoder-decoder architecture with multi-head attention mechanism. Its output is concatenated with the one hot encoding of the language label of an input character sequence to predict a target character sequence. The results show that the transformer outperforms the baseline rule-based system only partially.

## introduction the sigtyp 2022 shared task on the prediction of cognate reflexes investigates foot_0 the research question of to what extend machine learning can be employed to predict cognate reflexes, i.e., word forms assumed to derive from a common attested or reconstructed word form. such words prototypically present phonological features that are in part common to different languages and in part specific to each given language: consistent comparison between different words allows detection of structural similarities and recurrent change patterns, which can be explained by positing the existence of sets of cognate reflexes, i.e., sets of words belonging to different languages but deriving from a common form that has over time undergone consistent-and therefore to a large extent predictable-phonological/structural change according to the rules of each given language. the study of cognate reflexes, which is at the heart of modern historical-comparative linguistics, was first applied successfully to the indo-european languages, which have been argued, with ample and compelling evidence, to derive from a common ancestor, the proto-indo-european (e.g. see @xcite . in this respect, the comparative method has even allowed reconstruction of parallel grammars, with description of detailed phonological and morphosyntactic correspondences between languages (e.g., see @xcite for ancient greek and latin). the cognate-related computational research has been characterized by various tasks and approaches. some are rule-based: for example, dinu and ciobanu (2014) automatically identify cognates by linking word etymologies; @xcite proposes an algorithm to align cognate sound segments. others employ machine learning methods: for example, @xcite build an encoder-decoderwith-attention model to predict the latin root word common to romance language cognates. similarly, dekker and zuidema (2020) investigate the use of neural networks for prediction of cognates from slavic and germanic subfamilies. in the sigtyp 2022 shared task, participants are requested to predict a cognate reflex form from other cognate reflex forms. the present article reports on the model i have build for the challenge. in section 2, the dataset is described, while, in section 3, the method employed to tackle the task is detailed. section 4 contains the results, and section 5 their discussion. section 6 provides some concluding remarks. ## dataset the initial dataset provided for training consisted of cognate reflexes from 10 language families (see @xcite for a description of the language database). the data for each language family in turn consists of 5 tsv training files containing cognate reflexes for a variable number of languages: the 5 files contain training data with different percentages of missing data, ranging from 10% to 50%. this data structure has been designed to test model reliability on progressively sparser data, with the file containing 50% missing data being the most challenging. the 5 training files are matched by 5 test files presenting the exact same structure as the training files plus question marks in those table cells whose values are requested to be predicted. solutions and baseline results for each of the 5 test files are also provided. it is important to note that the initial training data are only provided in order for the participant to familiarize with data structure: indeed, the test data, which only are required to be submitted for the challenge, contain different languages (also called 'surprise languages'), whose model parameters need to be calculated singularly and independently. the test data also consists of 10 language families. in what follows, i present my work concerning the surprise languages foot_1 only. ## results results are shown in ## discussion the presence of 5 different training sets for each language family and the need of building a model for each language within a language family (see section 3) resulted in the training of as high a number of models as 495. since training of different algorithms and, especially, hyperparameter optimization for each model would have required a high computation load, i focused on a transformer architecture with fixed hyperparameters for all languages. 5 the levenshtein distance shows that the transformer always performs better than the rulebased system relative to the language family kesslersignificance. the transformer's performances for beidazihui and bremerberta are better relative to the datasets with proportions 0.1, 0.2, and 0.3 and the datasets with proportions 0.1 and 0.5, respectively. the major challenge posed by the sigtyp 2022 shared task seems to be data scarsity and sparsity, which affects data representativeness. the transformer's model tended to overfit the training data. a dropout layer and early stopping were employed, but more regularization and hyperparameter tuning 5 details can be found at https://github.com/ sigtyp/st2022/tree/main/systems/precor_ transformer . ## conclusions the paper presented the transformer model built for the sigtyp 2022 shared task. despite the transformer's complex architecture, which can model input characters in context and even rely on past target sequence steps, its performance was not overall superior to that of the baseline rule-based system. this may be due to data scarsity and sparsity. for this reason and in light of the considerable computational overhead that may be required at inference time, in that target sequence decoding may involve thousands of dictionary lookups-which can only be executed on cpu-one might prefer to test simpler model architectures.