LU - BZU at S em E val-2021 Task 2: W ord2 V ec and L emma2 V ec performance in A rabic Word-in-Context disambiguation (2021)

Abstract: This paper presents a set of experiments to evaluate and compare between the performance of using CBOW Word2Vec and Lemma2Vec models for Arabic Word-in-Context (WiC) disambiguation without using sense inventories or sense embeddings. As part of the SemEval-2021 Shared Task 2 on WiC disambiguation, we used the dev.ar-ar dataset (2k sentence pairs) to decide whether two words in a given sentence pair carry the same meaning. We used two Word2Vec models: Wiki-CBOW, a pre-trained model on Arabic Wikipedia, and another model we trained on large Arabic corpora of about 3 billion tokens. Two Lemma2Vec models was also constructed based on the two Word2Vec models. Each of the four models was then used in the WiC disambiguation task, and then evaluated on the SemEval-2021 test.ar-ar dataset. At the end, we reported the performance of different models and compared between using lemma-based and word-based models.

## introduction as a word may denote multiple meanings (i.e., senses) in different contexts, disambiguating them is important for many nlp applications, such as information retrieval, machine translation, summarization, among others. for example, the word "table" in sentences like "i am cleaning the table", "i am serving the table", "i am emailing the table", refer to "furniture", "people", and "data" respectively. disambiguating the sense that a word denotes in a given sentence is important for understanding the semantics of this sentence. to automatically disambiguate word senses in a given context, many approaches have been proposed based on supervised, semi-supervised, or unsupervised learning models. supervised and semi-supervised methods rely on full, or partial, labeling of the word senses in the training corpus to construct a model @xcite . on the other hand, unsupervised approaches induce senses from unannotated raw corpora and do not use lexical resources. the problem in such approaches, is that unsupervised learning of word embeddings produces a single vector for each word in all contexts, and thus ignoring its polysemy. such approaches are called static word embeddings. to overcome the problem, two types of approaches are suggested @xcite : multi-prototype embeddings, and contextualized word embeddings. the latter suggests to model context embeddings as a dynamic contextualized word representation in order to represent complex characteristics of word use. proposed architectures such as elmo @xcite , ulmfit @xcite , gpt @xcite , t5 @xcite , and bert @xcite , achieved breakthrough performance on a wide range of natural language processing tasks. in multi-prototype embeddings, a set of embedding vectors are computed for each word, representing its senses. in @xcite , multi-prototype embeddings are produced based on the embeddings of a word. as such, a graph of similar words is constructed, then similar words are grouped into multiple clusters, each cluster representing a sense. as for @xcite , multi-prototype embeddings are produced by learning word and sense embeddings jointly from both, a corpus and a semantic network. in this paper we aim at using static word embeddings for wic disambiguation. works on arabic word sense disambiguation (wsd) are limited, and the proposed approaches are lacking a decent or common evaluation framework. additionally, there are some specificities of the arabic language that might not be known in other languages. although polysemy and disambiguating are challenging issues in all languages, they might be more challenging in the case of arabic @xcite and this for many reasons. for example, the word šāhd ( ) could be šāhid ( ) which means a witness, or šāhada ( ) which means watch. as such, disambiguating words senses in arabic, is similar to disambiguating senses of english words written without vowels. second, arabic is a highly inflected and derivational language. as such, thousands of different word forms could be inflected and derived from the same stem. therefore, words in word embeddings models will be considered as different, which may affect the accuracy and the utility of their representation vectors, as the same meaning could be incarnated in distributed word forms in corpora, which has led some researchers to think that using lemma-based models might be better than using word-based embeddings in arabic @xcite . this idea will be discussed later in sections 5 and 6. @xcite suggested an arabic wsd approach based on stem2vec and sense2vec. the stem2vec is produced by training word embeddings after stemming a corpus, whereas the sense2vec is produced based on the arabic word-net sense inventory, such that each synset is represented by a vector. to determine the sense of a given word in a sentence, the sentence vector is compared with every sense vector, then the sense with maximum similarity is selected. @xcite did not use either stemming or lemmatization. instead, they proposed to determine the sense of a word in context by comparing the context vector with a set of sense vectors, then the vector with the maximum similarity is selected. the context vector is computed as the sum of vectors of all words in a given context, which are learnt from a corpus of historical arabic. on the other hand, sense vectors are produced based on dictionary glosses. each sense vector is computed as the sum of vectors (learnt from the historical arabic corpus) of all words in the gloss of a word. other approaches to arabic wsd @xcite employ other techniques in machine learning and knowledge-based methods @xcite @xcite @xcite . in this paper, we present a set of experiments to evaluate the performance of using lemma2vec vs cbow word2vec in arabic wic disambigua-tion. the paper is structured as follows: section 2 presetns the background of this work. section 3 overviews the wic disambiguation system. section 4 and section 5, respectively, present the word2vec and lemma2vec models. in section 6 we present the experiments and the results; and in section 7 we summarize our conclusions and future work. the task aims at capturing the polysemous nature of words without relying on a fixed sense inventory. a common evaluation dataset is provided to participants in five languages, including arabic, our target language in this paper. the dataset was carefully designed to include all parts of speeches and to cover many domains and genres. the arabic dataset (called multilingual ar-ar) consists of two sets: a train set of 1000 sentence pairs for which tags (true or false) are revealed, and a test set of 1000 sentence pairs for which tags were kept hidden during the competition. figure 1 gives two examples of sentence pairs in the dev.ar-ar dataset. each sentence pair has a word in common for which start and end positions in sentences are provided. participants in the shared task were asked to infer whether the target word carries the same meaning (true) or not (false) in the two sentences. ## system overview this section describes our method to arabic wic disambiguation based on two types of embeddings: cbow word2vec and lemma2vec. given two sentences, s 1 and s 2 , and two words, v i from s 1 and w j from s 2 , the objective is to check whether v i and w j have the same meaning. to this end, our system extracts contexts of v i and w j from the sentence pair, represents them in two vectors and finally compares the two resulting vectors using the cosine similarity. the context of a word w of size n (denoted by context(w, n)) is composed of the words that surround the word w, with n words on the left and n words on the right (n varying between 1 and 10 in conducted experiments). to represent context(w, n) in a vector space, two methods are proposed: first one is based on cbow word2vec embeddings vectors @xcite of the words appearing in the context, whereas the second is based on the lemma2vec of lemmas of words appearing in the context. to select the best way to represent the context(w, n) by a vector, classification experiments were conducted using (i) different pooling operations, min, max, mean, and std to combine words/lemmas vectors of the context, (ii) different threshold values (between 0.55 and 0.85) and (iii) the removal of functional words (also called stop words). the later are used to express grammatical relationships among other words, they are characterized by they high frequency in the corpus which might affect the wic disambiguation accuracy. the cosine similarity is then used to compare vectors of context(v i , n) and context(w j , n). figure 2 illustrates how the cosine similarity is calculated from context(v i , 3) and context(w j , 3). classification experiments on semeval-2021 ar-ar datasets were conducted using the following two cbow word2vec models and two corresponding lemma2vec models: (i) wiki-cbow, a pretrained word2vec model from the set of aravec models @xcite , (ii) our cbow word2vec model that we trained ourselves, (iii) lemma2vec model that we constructed based on the wiki-cbow model, and (iv) lemma2vec that we constructed based on our cbow word2vec model. based on these four models, four experiments were conducted to tune the following parameters: context size (context size), threshold, pooling operation (pooling) and removing of functional words (stop words). ## corpus and trained word embeddings two cbow word2vec models were used in our experiments. the wiki-cbow @xcite , which consists of 234,173 vocabulary size, and another model we trained our self which consists of 334,161 vocabulary size. the wiki-cbow model was learnt from a corpus of arabic wikipedia articles of about 78 million words, the principal hyperparameters are: 5 for minimum word count and 5 for window size. our cbow word2vec model was trained on modern standard arabic corpora, such as @xcite @xcite of about 3 billion words; it was fit using 300-dimensional word vectors, 100 the minimum count of words, training epochs of 5 and window size of 5. before training the word2vec model, several normalization and preprocessing steps were performed. first, all diacritics, punctuations, madda character, digits (hindi and arabic), latin characters (including accented letters) were removed. second, some special arabic letters are unified. third, sequences of repeated characters with length larger than 2 were reduced to one character; repeated spaces were also replaced by one space. fourth, different forms of alifs ( ) are replaced with ( ). spaces followed by a period character and new lines were considered to be end of sentence marks. the split method in python is used in tokenization. the vocabulary size of the resulted model is 334,161. ## constructing the lemma2vec models two lemma2vec models were produced, based on both: the wiki-cbow word2vec model, and our cbow word2vec model. each vocabulary in each of the word2vec models was lemmatized first. then a vector for each lemma ( i.e., lemma2vec) is calculated as following: first all word forms belonging to this lemma are fetched, then their word2vec vectors are combined through a mean pooling operation. the lemmatization process was performed using in-house tools and lexicographic databases foot_0 belonging to birzeit university @xcite @xcite . in case of a word cannot be lemmatized due to misspelling, incorrect tokenization or in case of foreign word (not included in our database), then the corresponding lemma2vec is considered to be its word2vec vector. ## experiments results and discussion given our arabic wic disambigation method described in section 3, and given the semeval multilingual dev.ar-ar dataset provided by semeval-2021 @xcite , four classification experiments were conducted using the cosine similarity and based on the two word2vec models and the two lemma2vec models. the objective is to tune the following parameters for each model: context size (ranging from 1 to 10), threshold (we determined empirically the range from 0.55 to 0.85 with 0.1 step size), pooling (min, max, mean and std), and stop words (yes, no). the following to find the high f1-scores for t and f: for each context size (between 1 and 10) and for each value of the stop words (yes or no) we plotted 8 line plots (4 for t and 4 for f) for each of the four pooling operations (mean, max, min and std) and for threshold ranging from 0.55 to 0.85 (i.e., 20 plots for each model, resulting 80 plots). figures 3a, 3b, 3c and 3d show the best 4 f1scores line plots for each of the four models, and as shown in figure 3 , the lemma2vec models have the tendency to perform better with shorter context sizes compared with the word2vec models. a possible reason may be that, in case of lemma2vec, the narrow meaning of words is affected due to the increase number of words involved in lemma2vec vector calculation. the impact of lemma2vec on the narrow meaning of words is discussed in the next subsection. the results with yes for stop words are slightly better but not significant. additionally, the min pooling was generally the best operation to combine the context vectors, and the results of both min and max pooling were close to each other. ## conclusions and further work we presented a set of experiments to evaluate the performance of using word2vec and lemma2vec models in arabic wic disambiguation, without using external resources or any context/sense embedding model. different models were constructed based on two different corpora, and different types of parameters were tuned. the final results demonstrated that lemma2vec models are slightly better than word2vec models for arabic wic disambiguation. more specifically, we found that lemma2vec outperforms word2vec for true sentence pairs, but underperforms it for false sentence pairs. we plan to extend our work to use our lemma2vec model to build a multi-prototype embeddings using the large lexicographic database available at birzeit university. we plan also to fine tune the recently released arabic bert models, such as @xcite @xcite , using the same database.