From web to dialects: how to enhance non-standard R ussian lects lemmatisation? (2023)

Abstract: The growing need for using small data distinguished by a set of distributional properties becomes all the more apparent in the era of large language models (LLM). In this paper, we show that for the lemmatisation of the web as corpora texts, heterogeneous social media texts, and dialect texts, the morphological tagging by a model trained on a small dataset with specific properties generally works better than the morphological tagging by a model trained on a large dataset. The material we use is Russian non-standard texts and interviews with dialect speakers. The sequence-to-sequence lemmatisation with the help of taggers trained on smaller linguistically aware datasets achieves the average results of 85 to 90 per cent. These results are consistently (but not always), by 1-2 per cent. higher than the results of lemmatisation with the help of the large-dataset-trained taggers. We analyse these results and outline the possible further research directions.

## introduction lemmatisation is a natural language processing (nlp) task that is a part of the basic language resource toolkit (blark) @xcite @xcite . lemmatisation may be defined as a transformation of a given token into the dictionary form, the latter being called a lemma. there may be different ways of lemmatisation, such as classifying a token by its particular supposed lemmatisation rule and the subsequent transformation by this rule (for instance, such model may classify shown into the group of tokens that are lemmatised with «delete last n and then add to before the token» rule, and then transformed by this rule into to show) @xcite . in this paper, we focus on the sequence-to-sequence approach, which takes input sequence and transforms it into output sequence directly @xcite . sequence-to-sequence approach generally requires additional information for the token, be-cause it is difficult for the model to lemmatise bare tokens @xcite . many smaller lects foot_0 do not possess gold morphological tagging. however, they are located nearby a closely-related high-resource lect, for which there are a lot of gold morphological datasets. we hypothesise that there is a reliable way to find a dataset with the specific distributional properties, train a tagger on it, use this tagger on a new, rather different dataset, and then lemmatise the tokens of this dataset with a preliminary fine-tuned large language model. we also presume that this approach is preferable to gathering the biggest data amount possible. we believe that the thoroughness in morphological training data selection becomes gradually more important with increasing variation within the lemmatisation evaluation data. so, if overall the better tactic is to get the largest and the most heterogeneous dataset possible, for some types of data one needs a more nuanced approach. we are going to demonstrate this on the material of the non-standard russian lects. this includes the web as corpora material, social media texts, and dialect texts, presenting the continuum of lects getting further away from the standard russian in terms of distributional properties. we hypothesise the following: h1: morphological tagging efficiency directly influences the lemmatisation accuracy. h2: if the model trains on the larger dataset, the morphological tagging it performs will present stable satisfactory results. h3: for the non-standard data, the distributional properties of the training dataset are generally more important than the sheer size. we impose a set of restrictions. both the model we use for morphological tagging and the lemmat-iser (at least in prediction mode) should be able to run on an individual device with no more than 6gb v-ram (the specs of nvidia geforce gtx1060, currently the most widespread gpu). the training data may vary in size, however, the datasets that we select on distributional properties basis should not exceed 500 000 tokens. section 2 contains previous research on the topics of lemmatisation in general and russian lemmatisation in particular. in section 3, we describe the data. section 4 includes a description of the method. section 5 describes the experiments and the analysis of their results. in section 6, we wrap up the research, stating either confirmation or refutation for each of the hypotheses, as well as the possible future directions of the research. ## related work currently, there are two predominant approaches to lemmatisation. the first is the classification approach: the model determines the rule of lemmatisation for a given token and then applies the rule @xcite @xcite @xcite . this approach tends to be monolingual @xcite . the second is the generally multilingual sequence-tosequence approach when the input (token and its features) is transformed directly to the output sequence, lemma @xcite @xcite . russian lemmatisation currently dominates the east slavic lemmatisation landscape @xcite , including historical varieties, with both rulebased and automatic methods @xcite . however, territorial lects have not yet gained the same kind of attention, while the lemmatisers designed for specific corpora are not open-source @xcite . russian web as corpora and social media texts are included in the evaluation pipelines but generally are not the centre of attention @xcite . there are different ways to enhance the performance of a lemmatisation model, morphological tagging being the most common @xcite . the ensemble models that enhance lemmatisation efficiency with external resources @xcite are gaining popularity, especially for historical low-resource territorial lects @xcite . and given that social media texts are similar to them @xcite , the contemporary vocabulary dictionaries are going to be of use in further research. ## data we employ two groups of datasets: the training datasets and the evaluation datasets. training datasets are generally well-established through russian nlp and mostly contain standard russian texts. the evaluation datasets group contains both the well-established ones and the ones that are not yet heavily adopted in the russian nlp. the largest training dataset is a collection of different russian national corpus foot_1 texts that vary diachronically (from the 1700s to 2010s), orthographically (containing texts in modern orthography, as well as premodern, used mostly before 1917), and genre-wise (including news, poetry, fiction, and social media). we later refer to this dataset as rnc-sampled. this dataset contains nearly 2 million tokens. it is also the dataset the lemmatisation model trained on. we also employ two subsets of rnc-sampled. the first one is taiga @xcite , which aims to represent texts from social media that demonstrate a higher level of variation and colloquiality. taiga contains 197 000 tokens. the second is syntagrus @xcite , the biggest universal dependencies russian corpus, containing fiction, non-fiction and news texts. the original syntagrus contains 1.5 million tokens, we downsampled it to 195 000 tokens for effective comparison with taiga. we use three sets of data for evaluation. the first is the tagged part of the russian general internet corpus @xcite , designed for morphrueval-2017 @xcite . it is 270264 tokens in size. later we refer to this dataset as gikrya. gikrya consists of different texts from the internet, which possess a high degree of variation and lack orthographical normalisation. the tagging of the gikrya part that we employ is human-checked. the second evaluation dataset is the scraped collection of tweets from 2022 to 2023, selected based on them containing words мокша 'moksha', эрзя 'erzya', and саратов 'saratov'. the tweets contain texts from the regional mass media, as well as everyday communication, concerning current politics, by speakers of different origins and backgrounds. we slightly manually normalised the texts, correcting the most obvious errors, such as *проектрование > проектирование 'design'. the variation degree in this dataset, despite the minor edits, remains high, mostly due to the nonstandard compounds, such as иворовал (<ива + воровал 'willow + steal.past.3.sg.m'), and nonstandard orthography, for instance, расейськая 'russian'. we provide human-checked lemmata (without pos/morphological tagging) for this dataset. we later refer to this dataset as mes-tweets. mes-tweets contains 6100 tokens. the third group of evaluation datasets is the transcribed recordings of interviews with speakers of russian continuum dialects (small territorial lects) belogornoje (saratov region, russia, southern type, the territory where russian speakers arrived after the russian dialect system had formed), and megra (vologda region, russia, northern type, the territory where russian speakers had arrived before the split of the old east slavic dialect continuum). we take the material for both belogornoje and megra (as we refer to them later) from saratov dialectological corpus @xcite . these datasets are in themselves homogeneous, yet they differ from the training datasets, representing small territorial lects, rather than variation within the standard. belogornoje and megra together contain 4372 tokens, with megra being slightly larger (2856 versus 1516 tokens). both datasets possess gold lemmatisation and morphological tagging, though annotation schema differences make the use of the latter hardly applicable to this study. we present the short summary for each dataset in ## method to determine the degree, to which the morphological properties of a training dataset may influence the lemmatisation efficiency of an evaluation dataset, we present the following experiment pipeline. beforehand, we fine-tune the lemmatisation model with the largest morphologically tagged dataset available, rnc-sampled. the lemmatisation model is a sequence-to-sequence one, employing the bart architecture with the largest number of parameters (430m) @xcite . this model, bart-large, is used for all the lemmatisation experiments. for part-of-speech tagging, we use the stanza tagger @xcite , modified for the lowresource lects @xcite . we train stanza on three different datasets, rnc-sampled, taiga, and syntagrus-downsampled. rnc-sampled has the largest variation degree and the largest size. taiga, being relatively smaller, consists of social media texts that inherently possess a high degree of variety. syntagrus-downsampled is comparable in size to taiga, but it is much more homogeneous genre-wise. training yields three taggers for each of the datasets (rnc-sampled, taiga, and syntagrus). we then test these models. for this, we use gikrya as a dataset both completely independent from the russian national corpus and possessing a significant variation degree. this provides us with the preliminary idea of whether the knowledge acquired through rnc-sampled, taiga, and syntagrusdownsampled data, may aid the model in tagging a completely different dataset. then we perform the three stages of the lemmatisation experiments. as a baseline for each stage, we use two different tactics. the first is a simple token-to-lemma method when each token is taken as its own lemma. the second is using bart-large on bare tokens (with input in the form of @xcite [morphological tagging information] and lemma as a desired output). for each stage, we tag the datasets of gikrya (stage 1), mes-tweets (stage 2) and belogornoje and megra (stage 3) with each of the morphological tagging models available, providing silver (non-human-checked, yet performed by a model that generally produces satisfactory results) morphological tagging. the stages represent the growing degree of distance between standard russian and the variations that form the datasets. after that, we lemmatise each of the acquired datasets with bart-large. we compare the results of the lemmatisation against the baseline. as gikrya provides the gold morphological tagging, for stage 1 we also lemmatise tokens with gold tagging to set the highest possible bar. for evaluation, we use accuracy score, combined with different string similarity measures: levenshtein distance (levenshtein, 1966), damerau-levenshtein distance @xcite , and jaro-winkler distance @xcite . levenshtein distance that scores additions, deletions, and substitutions of characters gives a more precise picture of sequence-to-sequence model performance in comparison to the accuracy score, reducing the cost of small mistakes and putting the models that generalise over the models that only memorise. damerau-levenshtein distance adds substitutions, providing an even more fine-grained picture. jaro-winkler distance shows exactly how well models capture the concept of lemmatisation in slavic languages, favouring the sequences that match from the beginning. we also use normalised versions of these metrics @xcite . normalisation generally highlights the ability of a model to generalise: if the normalised score is less than its raw counterpart, the model possibly learned to remember particular token-lemma pairs rather than to lemmatise. ## experiments and analysis we split the experiments into the morphological tagging section and the lemmatisation section. the lemmatisation section consists of three stages. for the first, we use gikrya, the web corpus that contains texts of different genres and variations, some further from the standard russian than others. the second includes the lemmatisation of the mes-tweets dataset, which possesses a higher variation degree. for the third, we take dialect data, pushing the ability of the models to generalise to the limit. ## conclusion the experiments prove that the silver morphological tagging allows a lemmatiser to perform much more efficiently than without any information on morphological tagging (over 40% improvement). we show that silver morphological tagging aids almost as efficiently as gold morphological tagging, lagging only by 4% for web as corpora datasets, such as gikrya. this is achieved with the bart-large model, fine-tuned for the standard language. both the prediction run of bart-large and any training run of modified stanza (scherrer, 2021) did not take more than 4 gb gpu on rtx 3060 (mobile). thus, even if fine-tuning large lemmatiser models themselves on personal computer hardware is still going to remain a focus of further study, morphological tagging and lemmatisation itself may be performed on the relatively small data. the lemmatiser enhanced with data provided by both the taiga-trained and the syntagrus-downsampled-trained taggers often performs better than the lemmatiser enhanced with data provided by the rnc-sampled-trained tagger. even when the situation is opposite, the distance between the results rarely exceeds five per cent. the hypotheses that we stated at the beginning of the research were the following: h1: morphological tagging efficiency directly influences the lemmatisation accuracy. h2: if the model trains on the larger dataset, the morphological tagging it performs will present stable satisfactory results. h3: for non-standard data lemmatisation, the distributional properties of the training dataset are generally more important than the sheer size. the first hypothesis, as gikrya experiments show, holds only partially. syntagrusdownsampled-trained tagger performs the best in terms of morphological tagging, but rnc-sampledtrained tagger performs the best as an aide for the lemmatiser. the second hypothesis holds: there are no sudden falls in lemmatisation accuracy when the rncsampled-trained tagger provides additional data, even if the results achieved are not the best. the third hypothesis generally holds. the less the dataset resembles the standard russian, the more efficient becomes the enhancement with data acquired from the taiga-trained tagger, and the less model a l l(n) d-l d-l(n) j-w j-w(n) efficient becomes the enhancement with data acquired from the syntagrus-downsampled-trained tagger. this is because taiga, social media texts, is much more heterogeneous than syntagrusdownsampled. additional morphological information from rnc-sampled-trained tagger run beats the one that taiga provides, but only for belogornoje. it is important to remember that parts of taiga are included in rnc-sampled, and interaction between these parts and other parts of the rnc-sampled enabled the lemmatiser to process belogornoje especially well. however, this case is an outlier. the future direction of the research becomes clear: further search for a dataset that provides the best silver morphological tagging for dialect data as well as attempts at efficiently using small transformers (such as tinybart @xcite ) that one can fine-tune with personal computer hardware.