Multi-lect automatic detection of S wadesh list items from raw corpus data in E ast S lavic languages (2023)

Abstract: The article introduces a novel task of multi-lect automatic detection of Swadesh list items from raw corpora. The task aids the early stageof historical linguistics study by helping the researcher compile word lists for further analysis.In this paper, I test multi-lect automatic detection on the East Slavic lects’ data. The training data consists of Ukrainian, Belarusian, and Russian material. I introduce a new dataset for the Ukrainian language. I implement data augmentation techniques to give automatic tools a better understanding of the searched value. The test data consists of the Old East Slavic texts.I train HMM, CRF, and mBERT models, then test and evaluate them by harmonic F1 score. The baseline is a Random Forest classifier. I introduce two different subtasks: the search for new Swadesh list items, and the search for the known Swadesh list items in new lects of the well-established group. The first subtask, given the simultaneously diverse and vague nature of the Swadesh list, currently presents an almost unbeatable challenge for machine learning methods. The second subtask, on the other hand, is easier, and the mBERT model achieves a 0.57 F1 score. This is an impressive result, given how hard it is to formalise the token belonging to a very specific and thematically diverse set of concepts.

## introduction the need for automatic tools that can aid human researchers has been pressing in computational linguistics for at least the last two decades @xcite . there are turnkey solutions for the word list data @xcite @xcite . however, when a researcher starts working with a new lect from scratch, they usually have nothing but raw data, from which they must extract this kind of a word list. this is where computational technologies may assist the researcher in the earlier stage of the study: they may execute preliminary detection of tokens that are of special interest -the swadesh list items @xcite . in this paper, i present a task of multi-lect automatic detection of swadesh list items from raw corpus data. swadesh list, named after its creator, morris swadesh, is a list of basic concepts that generally are universal among the human languages and may be used for historical linguistics purposes @xcite . i want to test, whether the computer can grasp the vague concept of swadeshness @xcite , if even human researchers often struggle with its formalisation. i define swadeshness by the following set of criteria: the multi-lect automatic detection of swadesh list items from raw corpora is challenging. the tool (a rule-based, statistical, or neural network-based model) should be able to perform it zero-shot and from the first attempt: otherwise, human researchers are not going to need it at all. ideally, the model should be able to grasp the concept of swadeshness and become proficient enough to perform the task on the languages, the relations of which to the others are completely unknown. such a model ideally should be at the forefront, laying the groundwork for a human researcher. however, currently, automatic tools are not able to efficiently zero-shot detect swadesh list items in the raw corpus of a randomly given language. it is only reasonable to start with an easier task, detecting swadesh list items in the language for which there is a strong hypothesis of its genetic relationships. to carry out this detection, a researcher needs raw corpus material from this language and a model trained on the material of the language's hypothetical relatives. thus, the task of multi-lect automatic detection of swadesh list items from raw corpus data transforms into the task of multi-lect automatic detection of swadesh list items from raw corpus data of a particular language group. i propose to start with the east slavic lects. in this paper, i use the term lect instead of dialect and/or language to denote any distinct variety without imposing any hierarchy, which generally distracts from the variation study. this is particularly relevant in the case of the slavic group due to the political circumstances of the last three decades. the east slavic group seems especially well-fit for the task because a group is quite a small unit of language classification, for which the concept of swadeshness may be easier to grasp. the east slavic group possesses some rather big corpora for both modern and historical data. i intend to train the models on the modern east slavic data from different lects (ukrainian, russian and belarusian) and to zero-shot test them on the historical data. i want to try different models, both simple probability-based tools and complex large linguistic models (llms). ## related work the desire to automatically extract swadesh list items from new data manifested itself in historical linguistics almost as soon as the computing powers became sufficient for this type of task @xcite . generally, it falls within the greater historical linguistics trend of implementing computational methods as researcher's assistants @xcite . hmm models are some of the most frequent solutions due to their simple yet effective architecture and overall dominance across the nlp horizon; with pairhmms, adapted for working with parallel data @xcite , being the most widely used. further steps are connected with different techniques in multiple sequence alignment @xcite and sequence comparison @xcite . after that, the automatic cognate detection and classification as a task emerges @xcite @xcite . the methods to extract large swadesh lists in the context of multi-lingual databases appear at this time @xcite and simultaneously the multi-lingual datasets for them to be tested on arise @xcite . the formalisation of swadeshness has become an important part of the discussion in recent years @xcite . the multi-lect automatic detection of swadesh list items requires other approaches, as it utilises raw corpus data rather than lexical databases. one such approach is part-of-speech tagging. part-ofspeech tagging is mostly dominated by universal methods, based on recurrent neural networks @xcite ). yet the tasks conducted on different language varieties demand agile models that can both be tuned for the needs of a specific tagset and work in the context of lowresourced and sparse data @xcite . hidden markov model (hmm)-based taggers present this opportunity @xcite @xcite . the other probabilistic tool used for part-of-speech tagging is conditional random fields (crf) @xcite . both these methods are regularly applied in the context of historical linguistics and language variation @xcite @xcite . crf is also used in named entity recognition, where it is rivalled by methods based on the use of transformer models @xcite . historical linguistics study often requires efficient resource utilisation. this fits the current nlp trend that gave rise to the distilled and tiny versions of transformers @xcite . historical data is usually quite low-resourced, which provides an additional challenge to the detection of sparsely distributed swadesh items. this requires using special metrics for imbalanced data @xcite . the harmonic f1 score, traditionally used for such cases @xcite , still finds its application in the analysis of nlp tasks @xcite . ## data the data consists of two subsets of different sizes and coming from different languages, one used to train the models and to test them on the first subtask, the search for new swadesh list items, and another -for the second subtask, to test their performance on completely new material. both datasets are stored in universal dependencies (ud) format @xcite . i use ud format as it contains information on the lemma, which makes it significantly easier to prepare the datasets for the experiments. the first subset is a large modern east slavic multi-lect dataset. it was vital to maintain the balance between these groups for the model to learn as many features of swadesh list items across the east slavic lects as possible. i call the main principle of balance a parent-node one, which means that the amount of data from the lects under the same node (i.e., sharing the last common ancestor) should be approximately equal. for instance, in the case of this research, it means that ukrainian and belarusian, the closest relatives out of the three present lects, should have the same amount of tokens on their part. russian, the sole representative of their sister group, should be presented with a corpus of the same size. the first corpus i use, the belarusian-hse corpus @xcite , consists of 305,000 tokens of different genres, such as fiction (including poetry), legal texts, non-fiction, news texts, wikipedia, social networks texts. ukrainian ud (iu) corpus consists of only 122,000 tokens foot_0 , so i need more data. for this purpose, i take the ua-gec corpus @xcite and tag it with the existing stanza model @xcite , acquiring silver data in ud format foot_1 . i get 183,961 samples of this corpus, and thus the ukrainian-belarusian branch of east slavic remains in balance. the russian corpus taiga (shavrina and shapovalova, 2017) consists of 197,000 tokens and is represented by a diverse set of genres, including poetry, fiction, non-fiction, wikipedia, blogs, social media, and news. taiga is designed to represent syntactic features of russian lexical units (obviously, taking in swadesh list items) in the best possible way. to balance the russian branch with the ukrainian-belarusian branch, i add data from syn-tagrus @xcite , a 1.5 million corpus of fiction, news, and non-fiction. i take 395,431 tokens, so the training corpus may achieve the balance. one may point out that this makes the dataset imbalanced in favour of the russian lect. however, it balances the russian branch of the east slavic tree with the ruthenian branch, while the ruthenian branch is still balanced within itself. this follows 2019 ). we illustrate this with figure 1 . the corpora of the training dataset and their key features are presented in the test dataset is the two corpora of historical east slavic lects, the old east slavic torot corpus @xcite , containing nearly 246,000 tokens. the torot corpus is predominantly later old east slavic (belarusian, ukrainian and russian ancestral lect continuum) and partly middle russian (when it split from ukrainian and belarusian) material. its texts are mostly legal documents and non-fiction (chronicles). old east slavic being the ancestral form for all three modern east slavic languages (thus containing within different texts proto-belarusian, proto-ukrainian, and proto-russian features) is the main reason i use every one of them, and not only russian. both datasets are additionally preprocessed to prepare them for the task. they are assigned a label c (non-swadesh list item) or i (swadesh list item). i use the 40-item swadesh list @xcite , enriching it with some concepts from the 110-item list @xcite , namely, woman, kill, eat, all, man, me, and you (indirect) (genitive stem). i chose these particular concepts as they are semantically close to the concepts of the 40-item list: woman to breast, kill to die, eat to drink, all to full, man to person, me to i, and you (indirect) to you. hopefully, this aids the models to better grasp the semantic component of swadeshness. i tag each possible morphological form of swadesh list items. genitive stems of you and i, you (indirect) and me respectively, get the treatment of separate concepts. yet this does not mean that i use only base forms for all the concepts in the dataset, as the east slavic languages are highly inflective. i would risk losing a lot of forms, tagging only base forms as swadesh list items. in this fashion, all the forms of i (я) and me (меня, мне, and мной) have an i (swadesh list item) label. while picking the exact lexical item for a concept, i generally follow guidelines by @xcite . the training dataset, while quite big, does not contain a lot of contexts for swadesh list items for the model to learn on. the fully automatic generation of new examples, contrarily to grammatical error detection, currently seems impossible. however, i apply artificial augmentation, using tokenlevel 3-grams that provide minimal left and right context. this is an approach that part-of-speech studies successfully implement @xcite . i wrote a script that generates 3-grams for each instance of the swadesh list item in the text. these may be represented as c i c, where i is a swadesh list item, and c is used for any other token, including [cls] (this denotes fragment-starting token) and [eos] (this denotes fragment-finishing token). an item of the dataset thus contains the original sentence and its labels and generated 3-grams with their labels. the script is also used for the test dataset. artificial augmentations of the test dataset are not going to be used in the evaluation, as they may seemingly boost results for a poorer-performing model, and compromise the intention of evaluating the model on the raw data. generated datasets are available on huggingface foot_2 . 4 method 4.1 task i treat multi-lect automatic swadesh list items detection as a sequence labelling and information extraction task, placing it among the part-of-speech tagging @xcite and named entity recognition @xcite tasks, as it shares common features (a clear split of all the items into categories with part-of-speechtagging and a heavy imbalance of two classes with named entity recognition) with both. one may see it as a reduced information extraction task with the extracted entity restricted to a single token, or as an unbalanced sequence labelling task with two labels, one of which is significantly less frequent than another. these different ways imply using particular methods for both creating the tool and its evaluation. whether one frames the task as a reduced information extraction task or an unbalanced sequence labelling task, one should use metrics that fit the case of unbalanced classes the most. i propose to use the traditional harmonic f1 score between precision (the number of correctly predicted items of a particular class, divided by the number of all items) and recall (the number of correctly predicted items of a particular class, divided by the number of items that belong to this class) @xcite . the formula for harmonic f1 score is given in (1). i am going to provide information on precision and recall to present a clearer picture. as an evaluation method, i use only the f1 score for the swadesh list items, as the average f1 score and f1 score for non-swadesh list items, the dominating class, are going to be very high, and, at the same time, not informative. ## experiments and results the experiments start with splitting the modern dataset into three parts, α, β, and ω. ω is a full dataset, α and β contain sentences that include only tokens from the a part of the swadesh list split, or the b one, respectively. i then augment each of the datasets with 3-gram addition. i train each architecture -hmm, crf, bert (but the baseline, random forest classifier) -separately on α, α-augmented, β, β-augmented, ω, and ω-augmented. the historical test dataset is not split, and later i refer to it as γ. i cross-validate α-and β-trained models. this is the first subtask, the search for new swadesh list items, and here the models are not going to show a high f1 score, as it is a hard task even for a human. for the second subtask, i test the ω-trained model with the γ dataset. here the results should be better, as there are obvious graphical similarities between modern and historical swadesh list concepts, and their semantic and syntactic stability possibly may allow for an easier capture of historical swadesh list concepts. the models' results comparison should lead to the discussion of possible reasons why the model with the best performance was the most successful and why others failed. ## conclusion automatic tools demonstrate modest yet inspiring results, achieving a maximum of 0.56 f1 score on the tokens they are familiar with in unfamiliar languages and a maximum of 0.15 f1 score on unfamiliar tokens in the familiar lects. this seems quite promising, as the swadesh list items is a very sparsely distributed class of lexical units. the average probability of encountering them in raw text (across 1000 random samples, 100 lexical units each) is 0.02 for ω-dataset and 0.04 for γ-dataset. bert outcompetes probabilistic tools, hmm and especially crf, as it grasps the deep core properties of swadesh list items, namely, syntactic independence, stylistic neutrality, and semantic preciseness. hmm, though, is the most stable one in terms of resisting the noise in the data. all the models perform better at memorising tokens than at generalising over the concept of swadeshness. this may still aid the search for concepts that are expressed by the forms most stable across the span of time, such as pronouns. they even sometimes find completely new candidates for swadesh list items, such as night. unfortunately, one still needs to deal with each case manually when a model labels something as a swadesh list item. effective evaluation systems are yet to appear. as for automatic evaluation, the last resort is still checking against an existing list. data augmentation, restricted to 3-gram generation from sentences, is harmful to both the probabilistic tools and the transformer models. it definitely leads to overfitting. for the automatic tools to aid human researchers better, further enhancements must be provided in the future. the extension of the datasets and the im-plementation of new, effective data augmentation techniques, such as providing quantified information on the described features of swadesh list items, are required. it seems crucial to add verification of the method on other language groups, not only east slavic. the task may also be approached with other methods based on other nlp tasks. i believe that at least a random forest classifier will become a much better baseline with information on syntactic independence, semantic precision, and stylistic neutrality. the other models are also going to benefit from this kind of feature engineering.