Learning Word Embeddings for U krainian: A Comparative Study of F ast T ext Hyperparameters (2023)

Abstract: This study addresses the challenges of learning unsupervised word representations for the morphologically rich and low-resource Ukrainian language. Traditional models that perform decently on English do not generalize well for such languages due to a lack of sufficient data and the complexity of their grammatical structures. To overcome these challenges, we utilized a high-quality, large dataset of different genres for learning Ukrainian word vector representations. We found the best hyperparameters to train fastText language models on this dataset and performed intrinsic and extrinsic evaluations of the generated word embeddings using the established methods and metrics. The results of this study indicate that the trained vectors exhibit superior performance on intrinsic tests in comparison to existing embeddings for Ukrainian. Our best model gives 62% Accuracy on the word analogy task. Extrinsic evaluations were performed on two sequence labeling tasks: NER and POS tagging (83% spaCy NER F-score, 83% spaCy POS Accuracy, 92% Flair POS Accuracy).

## introduction word embeddings @xcite are fixed-length vector representations of words that have a variety of applications in natural language processing (nlp) tasks, including semantic text similarity @xcite , word sense disambiguation @xcite ), text classification @xcite ), question answering @xcite . word embedding techniques rely on the distributional hypothesis -the assumption that the meaning of a word is captured by the contexts in which it appears @xcite . even though unsupervised word embeddings can be learned directly from raw texts, gathering a significant amount of data for their training remains an immense challenge for low-resource languages * these authors contributed equally to this work. such as ukrainian. moreover, ukrainian is a morphologically rich language; its nouns decline for 7 cases, three genders, and two numbers. adjectives agree with nouns in case, gender, and number. verbs conjugate for four tenses, two voices, and two numbers. ukrainian verbs come in aspect pairs: perfective and imperfective. not surprisingly, traditional models that give excellent results for english may not be able to generalize well for highly inflected languages, such as ukrainian, without special tuning. to address these challenges, we worked with a high-quality, large dataset of different genres for learning vector representations of words in the ukrainian language. we identified the best hyperparameters to train fasttext language models on this dataset and performed the intrinsic and extrinsic evaluations of the generated word embeddings using firmly established methods and metrics. furthermore, the obtained vectors were compared with the ones previously published by the facebook team foot_0 @xcite and those trained using the default hyperparameters. our optimized models outperformed the baseline models by 7.1% in accuracy on the word analogy task, and showed a 6.4% improvement compared to the ukrainian word embeddings published by @xcite . the novel contributions of this work are: the obtained results allow nlp practitioners to reduce the computational resources and time required to develop algorithms for solving applied nlp tasks. the rest of the paper is organized as follows. section 2 contains an overview of related work. section 3 presents the data collection and preprocessing techniques. we describe the methodology for learning word vectors and conducted experiments in section 4. section 5 describes the evaluation methods and obtained results. we conclude and present future work in section 6. ## related work a standard approach @xcite for learning non-contextual word representations is to train a log-bilinear model based on the continuous bag-of-words (cbow) or the skipgram architectures. an example of such a language model is word2vec, described by @xcite . it learns continuous representations of words using a shallow neural net. @xcite showed that these representations capture syntactic and semantic regularities in language decently, and presented a novel method -word analogies -for evaluating word embeddings based on vector arithmetic. word2vec's main drawback is that it ignores the word's internal structure that contains rich information. this knowledge could be beneficial in computing representations of uncommon or misspelled words and for morphologically rich languages like ukrainian. to address this issue, the fasttext method @xcite proposed to take word morphology into account by representing each word as a bag of character n-grams. they evaluated the fasttext model on nine languages exhibiting different morphologies, but ukrainian was omitted. @xcite developed and released fast-text language models for 157 languages, including ukrainian, but they did not provide any evaluation of the embeddings for this language. they used wikipedia, which is of limited diversity, and quite noisy data from the common crawl (cc) project foot_3 for learning their word vectors. another shortcoming is that they did not optimize subword size and employed character n-grams of size 5-5 for all languages. also, the authors concluded that the quality of the obtained vectors for low-resource languages is significantly worse than for high-resource ones. @xcite discovered that subword sizes have a significant impact on the accuracy of fasttext language models, and their optimization enhanced the accuracy on word analogy tasks by up to 14%. the authors proposed a simple n-gram coverage model and discovered the optimal subword sizes on the english, german, czech, italian, spanish, french, hindi, turkish, and russian word analogy tasks. we utilized their findings for czech and russian as they also belong to the slavic language group. in the current study, we evaluate how the corpus size, specific models, and set of hyperparameters affect the quality of ukrainian word embeddings for different nlp tasks. also, we conduct a proper evaluation of the proposed by @xcite word embeddings for ukrainian and compare them with our results. ## training dataset this section presents our datasets and preprocessing techniques. ## embedding methods while recent advances in nlp have been dominated by transformer-based language models, there is still a place for simpler models like continuous bag-of-words (cbow) and skipgram @xcite in certain scenarios. these models offer several advantages over more complex ones, particularly in low-resource settings. for one, they are computationally efficient and can be trained on smaller datasets. additionally, they offer greater interpretability and transparency, making it easier to understand how the model makes its predictions. given these advantages, we choose to use cbow and skipgram methods for obtaining contextindependent word embeddings for our study. cbow model learns to predict a target word based on its context, using the sum of the background vectors. a predefined window size surrounding the target word represents the neighboring terms taken into account. skipgram is another architecture for creating word embeddings. the model uses a target word for predicting the context by summing the log probabilities of the surrounding words to the left and right of the target word. for the study, we have chosen the fasttext 16 implementation of these models, where morphology is taken into account. each word is represented as a bag of character n-grams (i.e., subwords), and the word vector is obtained by taking the sum of the vectors of the character n-grams appearing in that particular word @xcite . while being the golden implementation, it has two shortcomings that weren't described in the project documentation: cbow with positional weights is a variation of the cbow model that modifies the input vectors 16 https://fasttext.cc/ of context words to better depict the relationship between the target and context words based on their relative positions @xcite . the authors noted that the positional model more than doubles the training time since, for each gradient update of an input vector, we also need to update the weights of a positional vector. we utilized the implementation of the positional weighted model presented in the paper mentioned above. ## evaluation in this section, we describe various evaluation metrics on trained word vectors. prior work on evaluation of the embeddings can be divided into intrinsic and extrinsic evaluation methods @xcite . we start with the intrinsic evaluation on the word analogy task, then continue with named entity recognition (ner) and part-of-speech (pos) tagging for extrinsic estimation. ## conclusions and future work in this paper, we reviewed various aspects of learning word embeddings, including the quality and the quantity of the corpus texts, the choice of the word embeddings algorithm, and its hyperparameters. those variations were tested on real-world texts and nlp tasks, and the performance of the resulting word embeddings was carefully measured. during the research, more than forty variants of word vectors were trained and evaluated using the clean framework, which consists of one intrinsic and three extrinsic tests. the evaluation of the resulting word embeddings has indicated that: in the paper, we suggest a methodology to build and test word embeddings for low-resource languages, provide the code for training and evaluation, and describe the required data. such an approach allows conducting similar experiments for other languages and sets a good performance baseline for ukrainian, allowing us to revisit the results on an even bigger corpus. similar methods can be used to train other word vectors, such as classical word2vec @xcite , glove @xcite , or more recent alternatives like lexvec 32 or floret 33 .