Exploring Word Sense Distribution in U krainian with a Semantic Vector Space Model (2023)

Abstract: The paper discusses a Semantic Vector Space Model targeted at revealing how Ukrainian word senses vary and relate to each other. One of the benefits of the proposed semantic model is that it considers second-order context of the words and, thus, has more potential to compare and distinguish word senses observed in a unique concordance line. Combined with visualization techniques, this model makes it possible for a lexicographer to explore the Ukrainian word senses distribution on a large-scale. The paper describes the first results of the research performed and the following steps of the initiative.

## introduction word vector space model (vsm) is a distributional semantic technique initially developed in statistical natural language processing and is a principal tool in computational linguistics @xcite . such models treat text as multidimensional vector space, where a word, a combination of words, or a sentence are represented as a vector so that it is possible to apply various vector algebra operations: calculate distance between them, apply dimensionality reduction to the vector space, and cluster. among vsms, a group of models targets semantic items with more precise attention. according to @xcite , a semantic vector is a statistically processed frequency list of all collocates of a particular word in a given corpus, which expresses the idea that one can distinguish the meaning of a word by its context. for example, compare: "i have a great fan of rock among my friends" (fan as a person) to "i have a great fan with several heating options" (fan as a device). the representatives of the leuven variationist school have been advocates for semantic vector space models as statistical state-of-art for lexicog-raphers to identify semantic patterns in big unstructured corpora, helping linguists to avoid unfeasible manual data exploration @xcite . ## approach there are many techniques to build and examine a semantic vsm. the proposed approach is based on a specific kind of semantic vector space model that, in addition to considering immediate collocates of words, also accounts for the second-order collocates -typical collocates of words found in concordance lines built for a given word of interest. in such a way, a word is represented not only by friends but also by friends of a friend if to follow a social network metaphor. @xcite and @xcite used the second-order collocate vectors to examine semantic variations of the pluricentric dutch. @xcite applied this technique to english to investigate boundaries among various senses of a polysemic word and between different lexemes. the authors showed that semantic vsm and visual analytics could provide a solid support for lexicological analysis of polysemy in large corpora. in our research, we reproduce and expand this methodology to explore ukrainian word sense distribution to identify how words belonging to a particular synonymous set are related and contrasted. the experiment performed contains the following steps: 1) build semantic vector space model for the given ukrainian corpora, including a vocabulary of the most frequent words and their co-occurrence matrix; 2) from the corpora, extract concordance lines, and calculate second-order collocate vectors for them; 3) apply dimensionality reduction with the multidimensional scaling technique, as @xcite proposed, to the vectors and visualize them on a scatterplot for particular words from the validation set; 4) perform the pairwise calculation of cosine similarity to the vectors in question. ## data given that we are interested in identifying regional variation of ukrainian over time, we performed our initial set of experiments on the corpus with texts published in the kyiv region within 1940-1969. the texts in the corpus are fiction and periodic publications. the corpus size is relatively small for the word-sense exploration tasks (30 mln tokens total) but sufficient for the proof-of-concept. the source of the textual data investigated is the general annotated corpus of the ukrainian foot_0 @xcite . ## implementation for the experiment, a set of python data science and natural language processing libraries have been used (scikit-learn, pandas, numpy, matplotlib, beautiful soap, nltk). the morphological tagger and lemmatizer of the ukrainian foot_1 preprocessed the input texts. specific high-frequency words were removed from the texts. then a vocabulary model was created based on the word frequency across the corpus. the chosen size of the vocabulary for the experiment is 3,000 words so that the most frequent 3,000 lemmas (with the exclusion of highly-frequent grammatical words) comprised the vocabulary. as the next step, for each pair of words from the vocabulary, we calculated the amount of time they co-occur in the same 4-token-window context in each corpus. as a result, we obtained a sparse co-occurrence matrix of collocates for the vocabulary elements and then normalized it with the pmi (pointwise mutual information) index. to make the co-occurrence matrix less sparse, we kept only those columns and rows that contain at least one value with a @xmath0.0. in the next step, we extracted concordance lines for particular words of interest (with 5+5 and 10+10 words windows), shortened the lines to include only the words from the vocabulary, and calculated second-order collocate vectors for the lines. we consider only concordance lines with five or more vocabulary words for further processing. the second-order collocate vector calculation is the following: for each word in a shortened concordance line, get its vector representation in the co-occurrence matrix initially built (i. e. a corresponding column in the co-occurrence matrix). then average those word vectors for a particular concordance line. therefore, each word of interest obtained concordance lines with the corresponding second-order collocate vectors foot_2 . for the sake of assessing the quality of the model, the multidimensional scaling technique and calculation of cosine distance were applied to the averaged second-order collocate vectors. ## model assessment despite a relatively small input corpus, the proposed model turned out sensitive enough to distinguish different words and word senses and see commonalities among them. the pmi co-occurrence matrix with first-order collocates already gives some understanding of word senses. let us consider the top 10 collocates for dvygun ('an engine'; 'a driving force') with their pmi values (dvygun ('an engine') 4.53, potužnyj ('powerful') 3.64, atomnyj ('nuclear') 3.45, polit ('a process of flying') 3.38, vičnyj ('eternal') 3.27, švydkist' ('speed') 3.15, raketa ('a rocket') 3.10, zamovknuty ('to become silent') 3.05, korabel' ('a ship') 2.97. and for its synonym motor ('a motor', 'an engine'): gurkit ('roar') 3.88, motor ('a motor') 3.33, gudity ('to buzz') 3.16, kabina ('a cabin') 3.16, zavesty ('to start an engine') 2.996, litak ('an airplane') 2.991, potužnyj ('powerful') 2.78, traktor ('a tractor') 2.77, avtomašyna ('a car') 2.73, avtomobil' ('an automobile') 2.64. from these collocates, we can already see the difference between the two synonyms. in the kyiv corpus 1940-1969, motor often denotes common-life vehicles (autos, boats, tractors). in contrast, dvygun is associated with "serious" topics like space travel, nuclear power, and metaphorically a driving force and a cause of activities and events. trying to distinguish word senses only by immediate collocates may not capture sense similarity if words in the context do not overlap. that is why the second-order collocate approach gives better sensitivity to similar word senses even if direct collocates in the concordance lines differ. to evaluate the model's ability to capture word sense commonalities and differences, we have compared the cosine semantic similarity measure calculated for the second-order collocate vectors and an alternative vsm -the 200k ukrainian floret vectors available via the spacy framework foot_3 . for example, for dvygun in the two concordance lines: 1) "nočamy v tumani gorily svitliačky, i vytt'a zviriv inodi zaglušalo šum dvyguna v tabori." ("at night, fireflies burned in the fog, and the howling of the animals sometimes drowned out the noise of the engine in the camp.") 2) "kolia prysluhavsia. ni, dvyguny dyryzablia gudut' tak samo monotonno j nevpynno." ("kolia listened. no, the airship's engines were buzzing just as monotonously and incessantly.") the semantic similarity measure by the secondorder collocates is 0.9976, and by the ukrainian floret vectors -0.7508. in this example, the different context words in both lines express the same idea that an engine creates noise, represented with different context words, which the model was able to capture with the help of information about the second-order vectors. currently, the vocabulary of the model accounts for 3,000 lemmas. for the first concordance line, the following vocabulary words contributed to the calculation: zvir ('an animal'), inodi ('sometimes'), šum ('noise'), tabir ('a camp'), soldat ('a soldier'). and for the second line, those collocates are kolia (a person name), prysluhatysia ('to listen up'), gudity ('to buzz'). let us consider another example with the pair of synonyms dvygun and motor in the following contexts: 1) "zarevly dvyguny, dribnyi driž projšov po mašyni." ("the engines roared to life, and a small shudder went through the car.") 2) "do jogo čujnogo vucha doletilo poforkuvann'a motora, srkyp galm. prybuv komendant taboru -bil'ava bestija." ("a light engine whirring and brakes screeching reached his sensitive ears. the camp commandant, the blonde beast, had arrived.") the semantic similarity measure by the secondorder collocates is 0.9799, and by the ukrainian floret vectors -0.6416. the vocabulary elements for the former concordance line are smuga ('a lane'), myt' ('a momemnt'), zupynytys'a ('to stop'), dribnyj ('small'), projty ('to pass through'), mašyna ('a machine'). and for the latter: vuho ('an ear'), legkyi ('light'), prybuty ('to arrive'), komendant ('commander'), tabir ('camp'). the first results are promising. however, the initial version of the model has limitations due to a relatively small vocabulary size. it works well with the concordance lines with at least five collocates from the vocabulary. therefore, we will significantly increase the model's vocabulary to 20-50k lemmas on the project's next iteration to make it comprehend a wider range of words in more versatile contexts. it is also important to mention that we calculated the semantic similarity for vectors with already reduced dimensionality and only for the target words instead of the entire vocabulary. thus, the provided comparison to the alternative language model is made solely to show that the model can find commonalities in non-overlapping concordance lines. in addition to the cosine similarity metric, the multidimensional scaling technique made it possible to explore the model outputs visually. figures 1 2 3 4 5 demonstrate the scatterplots for several synonymous word pairs. multidimensional scaling was applied separately to the second-order collocate vectors of a particular group of words or a single word per each test case. the dots on the plot are the reduced vectors. for a word or a pair of words, one can investigate close and remote dots to validate whether they denote similar or distinct occurrences. such vectorspace-based visual representations of word concordances bring additional insights for lexicographers targeted at exploring polysemy, various semantic relations, and semantic variation in language. figure 1 : the second-order collocate vectors: dvygun concordance lines for dvygun. this word has two senses: 'engine as a mechanical device', and an abstract metaphoric sense, 'something that pushes, causes things to happen'. in the corpus, the first sense dominates. however, several vectors on the plot represent the second sense. let us consider vector d on the right of the plot with the following collocates: vplyv ('influence'), rozym ('intelligence'), istoria ('history'), rid ('lineage'), svitogl'ad ('a world view'). if to compare vector 2 to vector 10 having the vocabulary collocates: dokaz ('a proof'), kolia (a person name), prysluhatysia ('to listen up'), gudity ('to buzz'), the cosine similarity for these vectors is negative -0.7647, which indicates different senses. figure 2 : the second-order collocate vectors: motor cordance lines for motor ('engine'). the following two vectors demonstrate again that the model is able to capture sense similarity despite nonoverlapping collocates. consider vector 69 with the vocabulary collocates: prosto ('easily'), znyknuty ('to dissapear'), prolunaty ('to resound'), signal ('a signal'), zakryčaty ('to scream') and vector 74 with the vocabulary collocates: mašyna ('a machine'), movčaty ('to be silent'), rušyty ('to move'), pravoruč ('to the right'). for these two vectors, the semantic similarity by the second-order collocates is 0.99, and by the ukrainian floret vectors is 1.0. the model has captured the concordance lines with another sense of motor, usually occurring as an exclamation during a movie production: "motor!" as "action!". the vectors 86, 176-179 are located quite close to each other on the plot. the corresponding concordance lines contain a word denoting exclamation and attention (kruknuty, kryčaty ('to cry out'), vyguknyty ('to exclaim') and uvaga ('attention')), as well as nouns denoting movie pro-duction -režyser ('director') and the likes. vector 107 (with the vocabulary collocates vidro ('a bucket'), krutyty ('to spin'), kriz' ('through'), dirka ('a hole')) stays apart on the left for a reason. the context is atypical -a humorous story from a humoristic magazine perets, 1961 on how to construct an engine from a bucket: "vkladajete v take vidro vyprany bilyznu, motor krutyt' vidro, voda kriz' dirky vylitaje, bilyzna sohne na očah!.." ("put the laundry in the bucket, the motor spins the bucket, the water flies out through the holes, and the laundry dries before your eyes!") figure 3 shows that motor and dvygun have overlapping areas but also quite distinct ones, which proves the above observation that these two synonyms have separate areas of usage: motor for common everyday-life vehicles and devices, whereas dvygun is for large industrial machinery. figure 4: the second-order collocate vectors: volia (v) and svoboda (s) volia ('freedom', 'a will') and svoboda ('freedom') are distributed. this time, we calculated the secondorder collocate vectors for 10+10 concordance lines. the top pmi collocates for svoboda are svoboda ('freedom') 3.58, borec' ('a fighter') 3.43, demokratia ('democracy') 3.28, carstvo ('a kingdom') 2.52, myr ('peace') 2.46, demokratyčnyi ('democratic') 2.45, poniatt'a ('a concept') 2.4, ideal ('an ideal') 2.37, borot'ba ('a fight') 2.34. and for volia, the top collocates are pamjataty ('to remember') 4.16, odynyca 2.42 ('a unit'), volia ('freedom') 2.37, nevolia ('captivity') 2.31, nacia ('a nation') 2.26, zusyll'a ('an effort') 2.24, vlada ('power', 'authorities') 1.99, borec' ('a fighter') 1.94, rozum ('intelligence') 1.94, bažann'a ('a desire') 1.88. since both words are relatively frequent ((volia5k, svoboda 1.6k occurrences in the corpus), we plotted 100 random concordance lines per synonym. on the plot, the vectors for both words overlap significantly, which indicates that volia tends to denote the concept of freedom more often than the idea of will in the texts published in kyiv in 1940-1969. figure 5 provides an example with two semantically unrelated words (svoboda and dvygun)to demonstrate how the model distinguishes them. the plot contains two completely separate clusters. the provided observations for the test cases make us believe in the potential of the semantic vector space model with second-order collocate vectors for various semantic explorations of ukrainian, including but not limited to word-sense disambiguation problems, regional variation investigation, and diachronic semantics. combined with profound lexicological analysis, such formal semantic representation applied to large-scale corpora would make it possible to reveal hidden trends and model dynamics of language over time and across different regions. ## further work the initial set of experiments with the proposed semantic model has opened several directions for the subsequent research and development to enhance and extend this approach. the ambition is to reveal ukrainian word variation over regions, time, and registers following the prior work of von waldenfels (2014). therefore, we have to deal with the problem of a semantic model being generic enough to represent the ukrainian language as a whole and simultaneously being sensitive to regional and time-wise peculiarities. the open question that requires further exploration is building time-and-region-specific models vs. a single semantic model. in addition, certain steps of the current data processing pipeline and analytic modeling require enrichment. we aim to continue experimenting with pipeline configuration decisions, vectorizer algorithms, and dimensionality reduction algorithms, utilizing clustering techniques, and various visualization approaches, including building semantic graphs. moreover, to properly represent a wide range of word senses, the model must be trained on significantly larger corpora (ideally, billions of tokens) and consider a vocabulary of greater size (at least 20,000 words). another challenge is to make the semantic model able to deal with high-frequency words, like prepositions, since their semantic variation is of high research interest for many lexicographers. traditionally, such words are excluded from a vector space model as stop words, but we would like to treat them as another valuable target of semantic modeling. last but not least, there is a need to tackle several language standards of ukrainian in specific periods of its history, which requires both additional data normalization and model sensitivity to different standards. ## limitations during this initial phase of the research, we needed more digital textual data, especially for the period before wwii, and a poor representation of various