Findings of the SIGTYP 2023 Shared task on Cognate and Derivative Detection For Low-Resourced Languages (2023)

Abstract: This paper describes the structure and findings of the SIGTYP 2023 shared task on cognate and derivative detection for low-resourced languages, broken down into a supervised and unsupervised sub-task. The participants were asked to submit the test data’s final prediction. A total of nine teams registered for the shared task where seven teams registered for both sub-tasks. Only two participants ended up submitting system descriptions, with only one submitting systems for both sub-tasks. While all systems show a rather promising performance, all could be within the baseline score for the supervised sub-task. However, the system submitted for the unsupervised sub-task outperforms the baseline score.

## introduction cognates and derivatives have been studied in various fields of linguistics with different purposes @xcite . in historical linguistics, cognates are useful in the reconstruction of proto-languages and can aid in establishing the relationship between languages; in lexicography, cognates are helpful in the development of multilingual dictionaries. moreover, in recent years, nlp researchers have shown interest in using cognates to enhance the performance of multilingual tasks such as machine translation, lexical induction, word embeddings and many more @xcite . as there has been little work on automatic cognate identification, it is still a challenging task, especially for less-resourced languages @xcite . supervised identification of cognates and derivatives is requires a substantial amount of annotated linguistic data, which may need to be manually annotated @xcite . at the same time, finding linguists and annotators for less-resourced languages is impractical. thus we propose a shared task which aims to provide a new benchmark for differentiating between cognates and derivatives and introduce new unsupervised approaches for cognate and derivative detection in less-resourced languages. cognates are etymologically related word pairs across languages which may or may not have similar spelling, pronunciation and meaning @xcite . cognates can be traced back to a single ancestral word form in a common earlier language stage. on the other hand, derivatives are words which have been adopted into a language either from an earlier stage of the same language, or as a borrowing from a different language. to give an example, the spanish libro and french livre, are each derived from latin liber "book", and are cognates with each other because they share this common ancestor. by contrast, the irish word leabhar is derived from latin liber because it was borrowed into irish from latin, but leabhar is a cognate with spanish libro because libro has been derived from an earlier developmental language stage, i.e. leabhar was not borrowed from spanish, but from latin, a precursor to spanish. where multiple stages of direct derivation occur, each successive stage is considered a derivation from the last, but also from any earlier stages. for example, leabhar in modern irish is derived from old irish lebor, but also from latin liber. as will be discussed in section 3, data used in this shared task has been drawn from wiktionary. apart from cognates (cog), wiktionary distinguishes between derived (der), inherited (inh), and borrowed (bor). this distinction is not maintained in this shared task, and all three are treated broadly as derivation. languages are distinguished from one another in the shared task based on iso-639 codes. anything which has a discrete iso-639 code is considered a separate language. therefore, irish with the code ga is a completely separate language from old irish as this has a separate code, sga. this prevents any confusion as to the point at which something ceases to be a derivative and becomes a cognate. such confusion may occur in speech, for example, where one may say that a term was borrowed into english from french. such a statement could lead to the supposition that a modern english (en) word is derived from the modern french (fr) term, however, if the borrowing took place between earlier language stages, say into middle english (enm) from old french (fro), the modern english term is only derived from old french and precursors to it, like latin (la), not from modern french. this is the case with the english word liberal. it was borrowed into middle english from old french, and is ultimately derived from latin liber "free". hence, the modern english, liberal, would be considered a derivative from both old french and latin in our data, however, it would be a cognate with modern french libéral because liberal is not derived directly from modern french. the rest of this paper is organised as follows. section 2 describes the setup and schedule of the shared task. section 3 presents the dataset used for the competition. section 4 describes the evaluation methods and the baselines. section 5 describes the systems submitted by the teams in the competition, and section 6 presents and analyses the results obtained by the competitors. lastly, in section 7, we conclude the whole findings of the shared task. ## shared task setup and schedule the section describes how the shared task was organized. the shared tasks involve two sub-tasks to perform multiclass classification tasks, which require that the relationship between pairs of words be identified as either a cognate relationship, a derivative relationship, or no relationship. the subtasks are: ## cognate datasets in this section, we present the characteristics and the statistics of the dataset used for the task of cognate and derivative prediction. ## systems a total of 9 teams registered for the shared task: 7 teams registered to participate in both the supervised and unsupervised tasks while 2 teams registered for only the supervised task. out of these, only two teams submitted systems. both teams submitted for the supervised task and one team submitted for both the supervised and unsupervised task. the teams who submitted their systems were invited to submit system description papers describing their experiments in the proceedings of the workshop @xcite . since these systems are described in individual papers, we will only briefly present the main features here. úfal_supervised: the system submitted by team úfal, represented by tomasz limisiewicz from charles university, provided gradient boosted tree classifier trained on linguistic and statistical features. the features used by the team to train the classifiers were language model embeddings, typological information which included language identity and language group identity and orthographical information @xcite . cotohili_supervised: team cotohili, represented by liviu dinu from university of bucharest, experimented with a few different multi-class classification algorithms such as support vector machine, naive bayes, and sgd with the combination of three features graphic features, phonetic features and language features. at the end they selected the best performing classifiers to train a stackable ensemble classifier (liviu p. dinu, 2023). cotohili_unsupervised: the unsupervised system submitted by team cotohili employed a set of features including graphic, phonetic and language encoding to kmeans algorithms (liviu p. dinu, 2023). ## results the participants were asked to submit the final test results in the format of the training data files, with comma-separated fields for word pairs, language codes, and relationship labels. files had to be named team name_unsupervised/supervised to indicate both the team's name and the sub-task in question. ## conclusion we have reported the findings of the sigtyp 2023 shared task on cognate and derivative detection for less-resourced languages as part of the fifth edition of sigtyp workshop. with the two teams that participated, we have seen different and interesting non-neural and neural systems that deal with cognate and derivative prediction task. while the baseline for supervised sub-task were based on neural networks, team úfal used a gradient boosted tree classifier and team cotohili came up with an ensemble classifier. however, neither team could beat the baseline set for the supervised task: the difference in the f1-score was -0.04 for team úfal and -0.08 for team cotohili. although, team úfal's entry ranked first among the two supervised systems submitted, with an f1-score of 0.87, the unsupervised system submitted by team cotohili based on a kmeans algorithm beat the baseline for the unsupervised task with with an improvememt of 0.11 in accuracy. taraka rama. 2016. siamese convolutional networks for cognate identification. in coling 2016, 26th international conference on computational linguistics, proceedings of the conference: technical papers, december 11-16, 2016, osaka, japan, pages 1018-1027. acl.