ÚFAL Submission for SIGTYP Supervised Cognate Detection Task (2023)

Abstract: In this work, I present ÚFAL submission for the supervised task of detecting cognates and derivatives. Cognates are word pairs in different languages sharing the origin in earlier attested forms in ancestral language, while derivatives come directly from another language. For the task, I developed gradient boosted tree classifier trained on linguistic and statistical features. The solution came first from two delivered systems with an 87% F1 score on the test split. This write-up gives an insight into the system and shows the importance of using linguistic features and character-level statistics for the task.

## introduction the described system is a supervised model trained for three-way classifications aimed to distinguish cognate and cross-lingual derivatives or no relationship for pairs of words in different languages. cognates are pairs with similar meanings and come from the same root in an ancestral language. for instance, the german "vater" is cognate with the english "father" coming from the same proto-indo-european root. in contrast, multilingual derivatives are words borrowed from another language potentially with some modification, e.g., the word "restaurant" in english comes from a french word with the same spelling @xcite . the solution used only the data provided by the organizers, i.e., 232,482 bilingual pairs in 34 european languages. the data came with the relationship labels (cognate, derivative, or no relation) scraped from wiktionary. 1 in the examples containing derivative pairs, the order of words did not indicate the source and recipient language. the proposed system was evaluated on the test data with 876 bilingual word pairs with hidden target labels. the evaluation metric was a macroaveraged f1 score. for development purposes, i 1 https://www.wiktionary.org/ sampled 10% of the provided training data to create a validation set not used in the model fitting. my solution is based on gradient boosted tree classifier trained on the set of language features comprising multilingual language model embeddings, language and language group id, characterlevel levenshtein distance, and a binary variable marking capitalized words. the system obtained the f1 score of 87% on the test set and came first out of two submitted to the shared task. the source code for the submission is publicly available at github: https://github. com/tomlimi/cognate_detection . the system description is organized in the following way: in section 2, i describe the classification model and the hyperparameter search method; in section 3, i introduce the features selected as input for the classifier; lastly, in section 4, i present the results of the method together with the accumulation study and the analysis of feature importance. ## classification for classification tasks, i used a gradient-boosted tree implemented in the xgboost library @xcite . 2 the boosting tree is the method that enables the predictions of a large set of decision trees obtained with a gradient search. this section describes the hyperparameters used for the classifier and the method used to select them. i chose xgboost because it performs well for data containing real and discrete variables, and the set of input variables can be easily extended. moreover, xgboost can be interpreted through feature importance analysis. ## feature selection i used an ensemble of word embeddings, typological and orthographical information as input to the xgboost classifier. this section describes how those features were selected and pre-processed. the visualization of all the picked features is presented in figure 1 . ## results i trained the classifier on top of input vectors constructed from the features described in section 3 and using the hyperparameters picked by bayesian search described in section 2.3. i split the training set provided by the organizers into train and validation splits containing 90% and 10% randomly selected data examples. the submitted solution obtained 95.3% macro f1 score on the train set and 82.7% on the validation set. on the held-out test split, the system achieved 87% f1 score, as reported by the organizers. this section presents the results of the accumulation study and feature importance analysis. ## conclusions the developed supervised system achieves competitive results in cognate detection (87% on the test set). the model was trained on a diverse set of linguistic and statistical signals. the accumulation and importance analysis showed the importance of nuance aspects of the dataset, such as latin or capitalization, as an indication of derivative relation. the analysis also showed the high importance of using character-based representation for the task in the form of canine embedding and character-level levenshtein distance.