A Method to Disambiguate a Word by Using Restricted Boltzmann Machine (2021)

Abstract: Finding the correct sense of a word is of great importance in many textual data related applications such as information retrieval, text mining and natural language processing. We have proposed one novel Word Sense Disambiguation (WSD) method according to its context. Based on collocation extraction score, the proposed method extracts three different features for each sense definition of a target word. These features create a feature vector and all the feature vectors create a sense matrix. Here, Restricted Boltzmann Machine (RBM) is used to enhance the sense matrix. Comparison of the proposed WSD method is made with current state-of-the-art systems using SENSEVAL and Sem Eval datasets. The proposed WSD method shows the practical implementation by applying on query-based text summary. For evaluation on query-based text summary, the proposed WSD method uses DUC datasets containing news-wire articles. Finally, the experimental analysis shows that our proposed WSD method performs better as compared to the current systems.

## introduction disambiguation of word is much important in the fields of natural language processing and ontology. in computational linguistics, most of the languages are polysemous @xcite . for example, 'i am going to bank.' the bank could be a financial institution or it could be a sloping land. in a sentence, sense of a word depends on context of the sentence @xcite . it is very troublesome to discover the sense of a word in computer programs because it does not possess endless information like a human being. according to jurafsky et al. @xcite , the task of selecting the correct sense of a word is known as wsd. many text applications depend on wsd technique in their process. the original lesk algorithm @xcite ) uses gloss or definition of the ambiguous word @xcite . lesk algorithm can be applied only in short phrases. @xcite developed two approaches based on the supervised and unsupervised method. the first method is an unsupervised method where log probability is computed from the sequences of word embedding vectors by considering the senses of the ambiguous word and from the context, it finds the correct sense @xcite . the second method is a supervised method where a multilayer neural network is used to find the sense of an ambiguous word. another unsupervised method was developed for word sense disambiguation by @xcite . here, the whole text document is considered as a context for the ambiguous word. this model is based on logistic normal topic model, which adds semantic information about the synsets as its priors. literature survey says that different supervised, unsupervised, and knowledge-based approaches are widely used in wsd ( @xcite ). a word contains different senses based on context and we need to disambiguate the target word for that given context. word sense disambiguation method is applied in many fields like sentence similarity measure. pawar et al. @xcite used 'max similarity' algorithm for wsd @xcite where they calculated sentence similarity score. they have implemented in p ywsd which is available in nltk library in python @xcite . however, the ac-curacy of this method is quite low as it does not always provide the exact sense of a word as maximum similarity does not give the surety that two senses will be exact for both sentences. here, an unsupervised learning algorithm named as deep belief network (dbn) is applied. dbn is a probabilistic learning model having multi-layers of hidden units @xcite . in dbn, a restricted boltzmann machine (rbm) is used to train the model. three different features are proposed to find the exact sense of a word. finally, rbm is used to enhance the extracted features to improve the result. ## contribution the main contribution is that proposed wsd method depends on three new features based on collocation extraction scores and further restricted boltzmann machine is applied to enhance these three features which give better result than other existing word sense disambiguation systems. to the best of my knowledge, there is no such kind of earlier work done in disambiguation of a word by using dbn. the proposed wsd method is the original one and can be applicable in many text mining applications. this proposed wsd method can be used in semantic relatedness score calculation between two sentences as it finds the word's correct sense on the basis of context of the sentence. further, semantic relatedness measure can be applied in many text mining applications like query-based text summarization, text clustering, plagiarism detection. the proposed wsd method is applied to query-based text summarization datasets to show its practical implementation. query-based text summarization is different from generic summarization as it extracts essential sentences from the input text based on the user's requirement @xcite . therefore, to find semantic relatedness between query and input text sentence, this wsd technique is applied to get accurate relatedness score. ## introduction to wordnet wordnet @xcite is a lexical dictionary. only content words are present in wordnet. these words are organized semantically. it is different from the traditional dictionary. nouns, verbs, adverbs, and adjectives are present in content words. 'wordnet' contains synonymous words set. it is known as synset or synonym set. synonym set contains words having same meaning. for example, shut and close are synonyms. polysemous words possess more than one synsets. for example, right; sometimes it means correct, morally good or justified and sometimes used as direction opposite to left. for each content word present in a synset, a gloss or a definition is present. most of the content words contain more than one sense definition. in wordnet, a word is represented as word#part of speech#sense number. wordnet dictionary is used here for calculating the semantic similarity or relatedness score between two words. therefore, before finding the score, it is important to disambiguate those words. ## proposed unsupervised deep leaning method for sense detection process of finding the appropriate sense of a word is shown in figure 1 . following steps are used to find the correct word's sense present in a sentence using unsupervised deep learning: here, pre-processing of text document uses stop word removal. stop words eliminates most common and unimportant words. example of stop word are: are, is, the etc. ## finding collocation extraction score between two words collocation refers as the use or occurrence of two words together. computational technique to find the collocation in a document or a corpus is known as collocation extraction score (rahman and borah, 2021a). to find collocation extraction score, wikipedia corpus (wc) (denoyer and gallinari, @xcite is used. bi-gram frequency is used to find the co-occurrence between two terms. associativity is found in while calculating collocation. if we take the example of cat and tiger, both are semantically similar. both are members of the feline family or superb hunters. in contrast, tiger and deer are associated as both occur frequently in language. this is known as functional relationship. association and similarity both are not even mutually exclusive or independent. two words tiger and deer are related two both relations to some degree @xcite . for each sense of ambiguous word, bi-gram collocation extraction score word 1 @xcite is found by calculating the frequency of words' available in sense definition for the first word (word 1 ) with the present words in the sentence and finally maximum value is taken by the proposed wsd method. we have taken one sentence: ram went to the state bank of india for depositing money. initially, for fining the accurate sense of word bank, we obtain all the senses present in wordnet. we find the collocation extraction score for each word present with the sense with the other content words present in the sentence. there are many senses present for the word bank. for example if i take the sense depository financial institution, then for each content word present in the sense deposit, financial, institution, we need to find the collocation extraction score with the content words present in the sentence ram, go, state, india, deposit, money. at the end, we need to take the maximum value. in this way we have to find the score for each sense. following equation 1 is used for finding the collocation extraction score between two words. one word is selected from the gloss of g and other word is selected from the sentence sen: for a target word (taw) exist in a sentence, the collocation extraction score (ces) of that sense is: we find the collocation extraction score for all the senses of taw. finally, we select that sense of taw for which the proposed wsd method gets the maximum collocation extraction score @xcite . ## feature extraction for finding exact sense of a word present in a sentence initially, proposed wsd method takes all the content words available in the same sentence to detect the sense of an ambiguous word. for an ambiguous word, the proposed method takes all senses present in the wordnet. for each sense, collocation extraction score is calculated for all content words. to find out the collocation extraction score, algorithm 1 shows the systematic steps: data: target word (t w ) and the sentence (s ws ) result: collocation score of t w for each sense s of t w do the stop word removal of s ws find out senses of t w for each sense (s) of t w do do the stop word removal (s swr ) of s for each word (w swr ) of s swr do find out the collocation extraction score between w swr and s ws by using the equation 2 end end algorithm 1: collocation extraction score for target word's each sense with words available in the sentence it is also seen that noun phrases always carry essential information which helps in finding the meaning of a sentence. therefore, noun phrases are considered and find collocation extraction score. we have calculated the collocation extraction score for each sense of target word with noun phrases. description of algorithm 2 is given below. data: target word (t w ) and sentence (s ws ) result: collocation score of t w for each sense s of t w do the removal of stop words s ws find the noun phrases (s np ) in s ws find out senses of t w for each sense (s) in t w do do the stop word removal (s swr ) of s for each word (w swr ) of s swr do find out the collocation extraction score between w swr and s np by using the equation 2 end end algorithm 2: collocation extraction score of each sense of the target word with the noun phrases present in the sentence it is also observed that sometimes, some words are not present in the wordnet, but they still can be considered as important words as they contribute to creating the context of the sentence. for example, we take three sentences: narendra modi visits china. shyam visited his uncle's house to attend the birthday party, and donald trump visited india. the word visit has different senses in wordnet. now the proposed method will find the most suitable sense present in wordnet. in first sentence, the word visit is much related to narendra modi. it can understand that this visit must be an official visit, as prime minister usually goes to other foreign countries for official purpose. the same meaning is also present for the third sentence. for the second sentence, the word visit is related to go to see a place which is certainly not official. to find exact sense, the collocation extraction score between each sense of visit with narendra modi, shyam, and donald trump will contribute to it. narendra modi, donald trump, and shyam are not present in wordnet. that refers that, sometimes, if a word is not present in wordnet, still it helps in contributing to finding exact sense. the word visit is identified as a verb here. from the above table ??, it is clear that the sense of visit word for the first and third sentence should be visit#v#4 as it matches with the context of the sentence and for the second sentence, it should be visit#v#1. the collocation extraction score between narendra modi and office is much higher in case of the first sentence; for the second sentence it is for party and entertainment and the third sentence it is for donald trump and office. this exact sense will help in finding an exact semantic relatedness score among three sentences. therefore, the following algorithm 3 is used to find the required collocation extraction score: .this sense matrix will be input for the rbm. the feature vector for each target word is passed through the hidden layer where each feature vector is multiplied with respective weight and a bias value. initially, train the rbm with the input vector v. now, the hidden units of rbm become deterministic. rbm has two layers: one is visible, or input layer, and the other is a hidden layer. no of units in the visible layer depends on how many senses are present for each word available in the text document. it calculates e(h j |v; w) with input v which serves again as visible units for rbm. this process can be repeated as many layers as it needs. after this unsupervised layer-wise training, back propagation is utilized to fine-tune of weights and biases @xcite . this unsupervised phase does not require labels. finally, we will get a refined and enhanced matrix. the method can be represented mathematically. input for the rbm is a sense matrix. each row in the matrix represents one particular sense of a word. whole content words present in the text document are represented as a sense matrix. the sense matrix @xmath0, s 2 , .....s n ) is a feature vector set contains all the three features extracted for each sense of a word s 1 (jain and lobiyal, 2022). the sense matrix is represented by the following figure 2 : the input to the rbm is the set of feature vector s. it acts as a visible layer. here, random values are selected for biases. rbm contains two hidden layers, the whole process can be represented in following equations: here s @xmath1, f 2 , f 3 ) and @xmath2, where n is the sense number for the ambiguous word present in the text document. as the rbm has two layers, two bias values h 0 and h 1 are selected randomly. to get a more refined matrix, rbm works in two steps. during the first phase, the new refined sense matrix is: the above expression is obtained in the follow-ing way: in step 2, the same procedure is followed by considering the bias h 1 and get more enhanced and refined sense matrix and which is given by: after obtaining refined sense matrix, a threshold value is taken for testing purpose with it. the threshold value is randomly generated for each vector. it is further tested with a for example: if the value of f 1 >= threshold f 1 , then only it will be considered. to generate optimal feature vector set, obtained feature vector sets are fine tuned by adjusting the units' weight of rbm. for optimal fine-tuning, back propagation algorithm is used. the enhanced feature vector values are added to obtain a score against each sense. finally, the highest scored sense will be considered as the best sense for that target word. note that the rbm will have to be freshly trained for each new content word that has to be disambiguated. ## comparison of results of use of three different features with and without the use of feature enhancement here, we have compared the word sense disambiguation results with or without the use of feature enhancement technique. through this comparison, it is quite clear that the feature enhancement through restricted boltzmann machine helps in getting better results. following ## conclusion and future work we have presented an unsupervised deep learning technique for detecting the word's sense. three different features are extracted based on the collocation score. restricted boltzmann machine is used to enhance the features. proposed wsd method is implemented on word sense disambiguation datasets to compare mainly with other existing and current word sense disambiguation methods. the evaluation shows that the proposed wsd method outperforms many current methods. as this method will be used in query-based text summarization, evaluations have done on duc datasets where the performance is much more better than many query-based text summarization methods. experimental analysis shows better performance of our proposed wsd method than many current methods. the result attains much better due to the use of collocation based features in deep belief network. in future, the proposed wsd method can be used in many fields like question-answering, information retrieval or query-based text summarization. the proposed method will also try to work on languages other than english.