Contextual Embeddings for U krainian: A Large Language Model Approach to Word Sense Disambiguation (2023)

Abstract: This research proposes a novel approach to the Word Sense Disambiguation (WSD) task in the Ukrainian language based on supervised fine-tuning of a pre-trained Large Language Model (LLM) on the dataset generated in an unsupervised way to obtain better contextual embeddings for words with multiple senses. The paper presents a method for generating a new dataset for WSD evaluation in the Ukrainian language based on the SUM dictionary. We developed a comprehensive framework that facilitates the generation of WSD evaluation datasets, enables the use of different prediction strategies, LLMs, and pooling strategies, and generates multiple performance reports. Our approach shows 77,9% accuracy for lexical meaning prediction for homonyms.

## introduction word sense disambiguation (wsd) task involves identifying a polysemic word's correct meaning in a given context. a task of wsd is applicable in various nlp fields @xcite , such as information retrieval, machine translation @xcite , and question answering. for well-resourced languages, this problem has many different approaches for solving that demonstrate competitive results @xcite . however, this task has received relatively little attention in the ukrainian language due to the absence of sense-annotated datasets. to address this issue, we propose a novel approach to the wsd task based on fine-tuning a pre-trained large language model (llm) to obtain better contextual embeddings for words with multiple senses. in this research, we present a method for generating a new dataset for wsd evaluation in the ukrainian language, which includes lemmas, example sentences, and lexical meanings based on the sum dictionary, of nas of ukraine (ulif-nasu). this dataset is used to evaluate the effectiveness of our proposed method. for supervised llm fine-tuning, we generate the dataset in an unsupervised way based on ubertext @xcite . additionally, we have developed a comprehensive framework foot_0 that facilitates the generation of wsd evaluation datasets, enables the use of different prediction strategies, llms, or pooling strategies, and generates multiple performance reports. ## related works early approaches in wsd utilized the concept of word embeddings, which were generated using pretrained algorithms such as @xcite or glove @xcite . however, these static word embeddings have a notable problem that all senses of a homonym word must share a single vector. to address this issue, several researchers have proposed techniques for capturing polysemy and generating more informative embeddings @xcite or @xcite . recently, there has been a trend toward utilizing contextual embeddings generated by llms instead of pre-trained word embeddings. these contextual embeddings provide a more nuanced representation of words, capturing context-specific information. as a result, a simple approach such as knn can be used in combination with these embeddings to predict word senses in word sense disambiguation tasks accurately @xcite . wsd can be approached as a binary classifica-tion problem. one such approach was proposed by @xcite , which involved adding a classification head to the bert model @xcite . the model takes a pair of sentences as input, with one sentence containing the target word and the other providing one of the possible definitions of the target word. the model then predicts whether the target word in the sentence has the same meaning as the definition. another noteworthy approach to word sense disambiguation is the one presented by @xcite , where the model not only takes into account the contextual information of the target word, but also the explicit senses assigned to neighboring words. despite the high performance of the previously mentioned supervised approaches for word sense disambiguation, their reliance on a large amount of annotated sense data can pose a challenge for their application to under-resourced languages. in contrast, unsupervised methods can also be applied to wsd tasks. one of the earliest and most wellknown solutions is using sense definitions and semantic relations from lexical graph databases such as babelfy @xcite . however, recent works such as @xcite have shown that llm-based solutions outperform those methods. given the limitations of prior research, particularly the shortage of annotated corpora in the ukrainian language, we present our proposed solution of supervised fine-tuning of an llm on a dataset generated in an unsupervised way. additionally, we have prepared a validation dataset for the ukrainian wsd task, derived from the sum (dictionary of ukrainian language) dictionary of nas of ukraine (ulif-nasu). our approach will enhance the model's understanding of semantic word meaning and improve the performance of the word sense disambiguation task in the ukrainian language. ## evaluation dataset to assess the efficacy of our methodology for addressing the ukrainian wsd task, we have established a validation dataset based on the sum dictionary. the sum dictionary is an appropriate resource as it employs componential analysis, a linguistic methodology used to differentiate common language phenomena such as polysemy and homonymy, by evaluating the presence or absence of shared semantic features among compared units. therefore, the dataset derived from the sum dictionary is well-suited for evaluating the performance of our approach. according to ukrainian lingua-information fund (2022), the examples in the sum dictionary were taken from a broad selection of resources, including fiction (from the end of the 18th century to the present day), ukrainian translations of the bible, folklore, publicistic, scientific, and popular scientific works, the language of the mass media, the language of the internet, etc. unfortunately, at the moment of publication, there is only part of the dictionary available (until word пiдкурювач (en: lighter, translit: pidkuryuvach)). the dataset was constructed by extracting each lemma, its lexical meaning, and examples of usage related to that meaning. while building the evaluation dataset, the lemmas with single possible lexical senses were filtered out, and the resulting dataset consisted of 78,000 samples. further data cleaning was performed to remove lemmas with a length of fewer than three characters, lemmas with missing senses or examples, lemmas that belong to functional parts of speech, and lemmas which lexical meaning reference for another lemma. after cleaning, the dataset consisted of 42,000 samples, with each sample consisting of a lemma, one of the possible lexical meanings of the lemma, and examples of this meaning. assembling the dataset involved part-of-speech (pos) detection for each lemma using the stanza library @xcite , and this information was utilized in the subsequent evaluation table. during our experiments, we observed that many lemmas in the ukrainian language have multiple similar lexical meanings, which significantly complicates the task, the examples presented in homonyms are unrelated words with the same written and spelling form but different lexical meanings. to construct a dataset of homonyms, we first filtered out lemmas with fewer than two entries in the sum dictionary. then, for each remaining lemma, we concatenated all the lexical meanings and examples of usage of each separate homonym. the resulting dataset consisted of 2,882 homonym samples, each sample including the lemma, its possible meanings, and examples for each meaning (see each lexical meaning group g l i , comprises all the possible lexical meanings of a particular lemma corresponding to the homonym. our objective is to predict the correct lexical meaning group g l i , from all the possible lexical meaning groups of the lemma g l , based on a list of examples of the lemma's usage. to accomplish this, we first calculate embeddings for the sentence example and obtain the target word embedding from it using various pooling strategies, which will be described later. subsequently, we measure the cosine similarity between the obtained embedding of the target word and the embeddings of each lexical meaning group. the lexical meaning group with the highest cosine similarity is considered to be the predicted context. figure 1 demonstrates an example of the single lemma prediction process utilizing our approach. ## results the we assume that a model trained on a larger dataset, which also has a larger average distance between positive and negative examples, generates better homonym-specific embeddings. we also observed that the model pmmbv2 tuned on 1,2m triplets with filtering out pairs with a small difference (less than 0.3) between the cosine similarity of the anchor and positive examples and that of the anchor and negative examples, resulting in the best accuracy. as the dataset used for training our model was constructed in an unsupervised manner, there existed a possibility of the model being biased towards the most frequently occurring senses of a given lemma. to assess this, we evaluated the model's accuracy based on the frequency of sense usage referring to the sum dictionary (see we have evaluated our approach on the polysemy dataset to investigate the correlation between the performance of the model on homonyms and polysemous lemmas. the ## conclusion our research proposes a novel approach for solving the wsd task in under-resourced languages such as ukrainian. we used a supervised approach to finetune llms on the unsupervised dataset generated by our method. furthermore, we built an evaluation dataset based on the sum dictionary, which other researchers can use for evaluating the wsd task in the ukrainian language. we implemented the u-wsd framework during the research, which preprocess and generate evaluation and fine-tuning datasets, perform inference, and measure performance. our approach achieved 77.9% accuracy on the homonym dataset, surpassing graph-based methods such as babelfy. future work aims to enhance the quality of the fine-tuning dataset by employing several measures. these measures include the removal of nearly identical anchor and positive examples, the exclusion of named entities detected as the target lemma, and the sampling of a more uniformly representative subset of examples for each lemma. we also want to improve the target lemma detection algorithm. additionally, we plan to explore more advanced embedding comparison mechanisms beyond cosine similarity.