Word Vectorization -Word2Vec
Tags: word2vec, gensim, textanalytics, wordvectorization
Published: 2017-10-31 04:42:44

---
title: Word Vectorization -Word2Vec
published: true
description: This article is about finding Word Vectorization and Word Embeddings useful in Text Analysis like finding word similarity, odd one out, analogies etc.
tags: Word2Vec,Gensim,Text Analytics,Word Vectorization
canonical_url: http://techscouter.blogspot.in/2017/10/word-vectorization.html
cover_image: https://thepracticaldev.s3.amazonaws.com/i/2un5cnxxpqixzfa5l6x4.png
---

<div dir="ltr" style="text-align: left;" trbidi="on">
<h2 style="text-align: left;">
<span id="goog_114634844"></span><span id="goog_114634845"></span></h2>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-EzGSSDTyxq4/Wff7sYIW3iI/AAAAAAAAB5c/NRsslGTJlb8qZBxZKVZGFI0lsOS0eu1sACLcBGAs/s1600/cropped.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="316" data-original-width="797" height="251" src="https://3.bp.blogspot.com/-EzGSSDTyxq4/Wff7sYIW3iI/AAAAAAAAB5c/NRsslGTJlb8qZBxZKVZGFI0lsOS0eu1sACLcBGAs/s640/cropped.png" width="640" /></a></div>
<h2 style="text-align: left;">

Introduction</h2>
Machine Learning has become the hottest topic in Data Industry with increasing demand for professionals who can work in this domain.
<div>
There is large amount of textual data present in internet and giant servers around the world.</div>
<div>
Just for some facts</div>
<div>
<span id="docs-internal-guid-152fa68b-635d-a06a-c668-fdf727374bb6"></span>
<ul style="margin-bottom: 0pt; margin-top: 0pt;"><span id="docs-internal-guid-152fa68b-635d-a06a-c668-fdf727374bb6">
<li dir="ltr" style="font-family: Arial; font-size: 11pt; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.6; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">1,209,600 new data producing social media users each day.</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.5999996; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">656 million tweets per day!</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.5999996; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">More than 4 million hours of content uploaded to Youtube every day, with users watching 5.97 billion hours of Youtube videos each day.</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.5999996; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">67,305,600 Instagram posts uploaded each day</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.5999996; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">There are over 2 billion monthly active Facebook users, compared to 1.44 billion at the start of 2015 and 1.65 at the start of 2016.</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.5999996; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">Facebook has 1.32 billion daily active users on average as of June 2017</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.5999996; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">4.3 BILLION Facebook messages posted daily!</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.5999996; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">5.75 BILLION Facebook likes every day.</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.5999996; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">22 billion texts sent every day.</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.6; margin-bottom: 8pt; margin-top: 0pt;">
<span style="font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">5.2 BILLION daily Google Searches in 2017.</span></div>
</li>
</span></ul>
<span id="docs-internal-guid-152fa68b-635d-a06a-c668-fdf727374bb6"> </span>
<h2 style="text-align: left;">
Need for Vectorization</h2>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-g-5dcPNryNE/Wff5k_l_JCI/AAAAAAAAB5U/D4W2kDJ5MjEgDoU0I2WBhy7fsFpnkRz_QCEwYBhgL/s1600/vector_space1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="225" data-original-width="300" src="https://2.bp.blogspot.com/-g-5dcPNryNE/Wff5k_l_JCI/AAAAAAAAB5U/D4W2kDJ5MjEgDoU0I2WBhy7fsFpnkRz_QCEwYBhgL/s1600/vector_space1.png" /></a></div>
<div>
<span style="font-family: &quot;arial&quot;;"><span style="font-size: 14.6667px; white-space: pre-wrap;">The amount of textual data is massive, and the problem with textual data is that it needs to be represented in a format that can be mathematically used in solving some problem.</span></span></div>
<div>
<span style="font-family: &quot;arial&quot;;"><span style="font-size: 14.6667px; white-space: pre-wrap;">In simple words, we need to get an integer representation of a word. </span></span><span style="font-family: &quot;arial&quot;; font-size: 14.6667px; white-space: pre-wrap;">There are simple to complex ways to solve this problem. </span></div>
<div>
<span style="font-family: &quot;arial&quot;; font-size: 14.6667px; white-space: pre-wrap;">
</span> 
<h3 style="text-align: left;">
<span style="font-family: &quot;arial&quot;; font-size: 14.6667px; white-space: pre-wrap;">One of the easiest ways to solve the problem is creating a simple <b>word to integer mapping</b>.</span></h3>
</div>
<div>
<pre style="background-color: rgba(0,0,0,0.1); border-bottom: 1px solid rgb(170, 204, 153); border-image: initial; border-left: none; border-right: none; border-top: 1px solid rgb(170, 204, 153); line-height: 15.6px; overflow-x: auto; overflow-y: hidden; padding: 5px;">#list of sentences to be vectorized
line="Hello this is a tutorial on how to convert the word in an integer format"

#dictionary to hold the words
word_list={}

#initialize the counter for assigning to different word
counter=0

#iterate over the words
for word in line:
 #check if the word is in dict
 if word not in word_list:
  word_list[word]=counter
  #update the counter
  counter+=1
  

</pre>
<div>

</div>
This will return us the dictionary of words with the corresponding integer representations.
<h3 style="text-align: left;">
Another way to get these numbers is by using TD-IDF</h3>
<div>
TF-IDF stands for term frequency-inverse document frequency which assigns some weight to the&nbsp;word based on the number of occurrences in the document also taking into consideration the frequency of the word in all the documents. This approach is better than the previous approach as it lowers the weight of the words that occur too often in all the sentences like 'a',&nbsp;'the', 'as' etc and increases the weight of the words that can be important in a sentence.</div>
<div>
This is useful in the scenarios where we want to get the important words from all the documents .</div>
<div>
This approach is also used in topic modelling.&nbsp;&nbsp;</div>
<div>

</div>
<h3 style="text-align: left;">
The third approach and the one on which this article will be focussing is Word2VEC</h3>
Word2vec is a group of related models that are used to produce so-called word embeddings. These models are shallow, two-layer neural networks, that are trained to reconstruct linguistic contexts of words.

After training, word2vec models can be used to map each word to a vector of typically several hundred elements, which represent that word's relation to other words. This vector is the neural network's hidden layer.

Word2vec relies on either skip-grams or continuous bag of words (CBOW) to create neural word embeddings. It was created by a team of researchers led by Tomas Mikolov at Google. The algorithm has been subsequently analyzed and explained by other researchers.

Luckily if you want to use this model in your work you don't have to write these algorithms.
<a href="https://radimrehurek.com/gensim/" target="_blank">Gensim</a>&nbsp;is one the library in Python that has some of the awesome features required for text processing and Natural Language Processing. In the rest of the article, we will learn to use this awesome library for word vectorization.
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-TCovoMhSwL4/Wff5lGQa45I/AAAAAAAAB5U/hK1zJ-FDlhElm5YqFxwrkEmO-KcfpSUIQCEwYBhgL/s1600/man_woman.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="775" data-original-width="1000" height="310" src="https://2.bp.blogspot.com/-TCovoMhSwL4/Wff5lGQa45I/AAAAAAAAB5U/hK1zJ-FDlhElm5YqFxwrkEmO-KcfpSUIQCEwYBhgL/s400/man_woman.jpg" width="400" /></a></div>
<h4 style="text-align: left;">
Installing Gensim</h4>
<div class="highlight-default" style="background-color: white; color: #333333; font-family: arial; font-size: 14px;">
<div class="highlight" style="background: rgb(238, 255, 204);">
<pre style="background-color: #f8f8f8; border: 1px solid rgb(204, 204, 204); font-family: Consolas, &quot;Deja Vu Sans Mono&quot;, &quot;Bitstream Vera Sans Mono&quot;, monospace; font-size: 0.95em; letter-spacing: 0.015em; line-height: 15.96px; overflow-x: auto; padding: 0.5em;"><span class="n">pip</span> <span class="n">install</span> <span class="o" style="color: #666666;">--</span><span class="n">upgrade</span> <span class="n">gensim</span></pre>
</div>
</div>
</div>
<div>
<span style="font-family: &quot;arial&quot;; font-size: 14.6667px; white-space: pre-wrap;">
</span> It has three major dependencies

<ul style="text-align: left;">
<li>Python</li>
<li>NumPy</li>
<li>SciPy</li>
</ul>
<div>
Make sure you install the dependencies before installing gensim.</div>
<div>
Lets' get our hands dirty on the code.</div>
<h4 style="text-align: left;">
Text Preprocessing:</h4>
<div>
In this step, we will pre-process the text like removing the stop words, lemmatize the words etc.</div>
<div>
You can perform different steps based on your requirements.</div>
<div>

</div>
<div>
I will use nltk stopword corpus for stop word removal and nltk word lemmatization for finding lemmas.</div>
<div>
I order to use nltk corpus you will need to download it using the following commands.</div>
<div>
<b>
</b></div>
<div>
<b>Downloading the corpus</b>
<b>
</b></div>
<div>
<pre style="background-color: rgba(0,0,0,0.1); border-bottom: 1px solid rgb(170, 204, 153); border-image: initial; border-left: none; border-right: none; border-top: 1px solid rgb(170, 204, 153); line-height: 15.6px; overflow-x: auto; overflow-y: hidden; padding: 5px;">import nltk
nltk.download()
#this will open a GUI from which you can download the corpus
</pre>
</div>
<div>

</div>
<div>
<b>Input initialization</b></div>

<pre style="background-color: rgba(0,0,0,0.1); border-bottom: 1px solid rgb(170, 204, 153); border-image: initial; border-left: none; border-right: none; border-top: 1px solid rgb(170, 204, 153); line-height: 15.6px; overflow-x: auto; overflow-y: hidden; padding: 5px;">#list of sentences to be vectorized
lines=["Hello this is a tutorial on how to convert the word in an integer format",
"this is a beautiful day","Jack is going to office"]</pre>
<div>
<b>
</b></div>
<div>
<b>Removing the Stop Words</b></div>
<div>
<b>
</b></div>
<div>
<pre style="background-color: rgba(0,0,0,0.1); border-bottom: 1px solid rgb(170, 204, 153); border-image: initial; border-left: none; border-right: none; border-top: 1px solid rgb(170, 204, 153); line-height: 15.6px; overflow-x: auto; overflow-y: hidden; padding: 5px;">from nltk.corpus import stopwords
stop_words=set(stopwords.words('english'))

lines_without_stopwords=[]
#stop words contain the set of stop words
for line in lines:
 temp_line=[]
 for word in lines:
  if word not in stop_words:
   temp_line.append (word)
 string=' '
 lines_without_stopwords.append(string.join(temp_line))

lines=lines_without_stopwords
</pre>
</div>
<div>
<b>
</b></div>
<div>
<b>Lemmatization</b>
<b>
</b></div>
<div>
<pre style="background-color: rgba(0,0,0,0.1); border-bottom: 1px solid rgb(170, 204, 153); border-image: initial; border-left: none; border-right: none; border-top: 1px solid rgb(170, 204, 153); line-height: 15.6px; overflow-x: auto; overflow-y: hidden; padding: 5px;">#import WordNet Lemmatizer from nltk
from nltk.stem import WordNetLemmatizer
wordnet_lemmatizer = WordNetLemmatizer()


lines_with_lemmas=[]
#stop words contain the set of stop words
for line in lines:
 temp_line=[]
 for word in lines:
  temp_line.append (wordnet_lemmatizer.lemmatize(word))
 string=' '
 lines_with_lemmas.append(string.join(temp_line))
lines=lines_with_lemmas


</pre>
</div>
<div>

</div>
<div>

Now we have done the basic preprocessing of the text. Any other preprocessing stuff can be achieved similarly.

<h3 style="text-align: left;">
Preparing Input</h3>
</div>
</div>
</div>
<div>
We have out input in the form of array of lines. In order for model to process the data we need covert our input to an array of array of words ( :\ ).</div>
<div>
<b>
</b></div>
<div>
<u><i>Our Input</i></u></div>
<div>
<div>
lines=["Hello this is a tutorial on how to convert the word in an integer format",</div>
<div>
"this is a beautiful day","Jack is going to office"]</div>
</div>
<div>
<b>
</b></div>
<div>
<u><i>New Input</i></u></div>
<div>
<div>
lines=[['Hello', 'this','tutorial', 'on', 'how','convert' ,'word',' integer','format'],</div>
<div>
['this' ,'beautiful', 'day'],['Jack','going' , 'office']</div>
</div>
<div>

</div>
<div>
<pre style="background-color: rgba(0,0,0,0.1); border-bottom: 1px solid rgb(170, 204, 153); border-image: initial; border-left: none; border-right: none; border-top: 1px solid rgb(170, 204, 153); line-height: 15.6px; overflow-x: auto; overflow-y: hidden; padding: 5px;">new_lines=[]
for line in lines:
 new_lines=line.split('')
#new lines has the new format
lines=new_lines
</pre>
</div>
<h3 style="text-align: left;">
Building the WORD2VEC Model</h3>
<div>
Building a model with gensim is just a piece of cake .</div>
<div>

</div>
<div>
<pre style="background-color: rgba(0,0,0,0.1); border-bottom: 1px solid rgb(170, 204, 153); border-image: initial; border-left: none; border-right: none; border-top: 1px solid rgb(170, 204, 153); line-height: 15.6px; overflow-x: auto; overflow-y: hidden; padding: 5px;">#import the gensim package
model = gensim.models.Word2Vec(lines, min_count=1,size=2)

</pre>
</div>
<div>

</div>
<div>
Here important is to understand the hyperparameters that can be used to train the model.</div>
<div>
Word2vec model constructor is defined as:</div>
<div>

</div>
<div>
<pre style="background-color: rgba(0,0,0,0.1); border-bottom: 1px solid rgb(170, 204, 153); border-image: initial; border-left: none; border-right: none; border-top: 1px solid rgb(170, 204, 153); line-height: 15.6px; overflow-x: auto; overflow-y: hidden; padding: 5px;">gensim.models.word2vec.Word2Vec(sentences=None, size=100, 
alpha=0.025, window=5, min_count=5, max_vocab_size=None, 
sample=0.001, seed=1, workers=3, min_alpha=0.0001, sg=0, 
hs=0, negative=5, cbow_mean=1, hashfxn=&lt;built-in function hash&gt;, 
iter=5, null_word=0, trim_rule=None, sorted_vocab=1,
batch_words=10000, compute_loss=False)</pre>
</div>
<div>

</div>
<div>

</div>
<div>
<i>sentence=</i>&nbsp;This is the input provided in the form of a list</div>
<div>

</div>
<div>
size= This defines the size of the vector we want to convert the word ('Hello'=[ ? ,&nbsp;? , ? ] if size=3)</div>
<div>

</div>
<div>
alpha= It is the initial learning rate (will linearly drop to min_alpha as training progresses).
<div>

</div>
<div>
window= It is the maximum distance between the current and predicted word within a sentence.</div>
<div>
<div>

</div>
<div>
min_count= ignore all words with total frequency lower than this.</div>
<div>

</div>
<div>
max_vocab_size = limit RAM during vocabulary building; if there are more unique words than this, then prune the infrequent ones. Every 10 million word types need about 1GB of RAM. Set to None for no limit (default).</div>
<div>

</div>
<div>
sample = threshold for configuring which higher-frequency words are randomly downsampled;</div>
<div>
default is 1e-3, useful range is (0, 1e-5).</div>
<div>

</div>
<div>
workers = use this many worker threads to train the model (=faster training with multicore machines).</div>
<div>

</div>
<div>
hs = if 1, hierarchical softmax will be used for model training. If set to 0 (default), and negative is non-zero, negative sampling will be used.</div>
<div>

</div>
<div>
negative = if &gt; 0, negative sampling will be used, the int for negative specifies how many “noise words should be drawn (usually between 5-20). Default is 5. If set to 0, no negative sampling is used.</div>
<div>

</div>
<div>
cbow_mean = if 0, use the sum of the context word vectors. If 1 (default), use the mean. Only applies when cbow is used.</div>
<div>

</div>
<div>
hashfxn = hash function to use to randomly initialize weights, for increased training reproducibility. The default is Python’s rudimentary built-in hash function.</div>
<div>

</div>
<div>
iter = number of iterations (epochs) over the corpus. Default is 5.</div>
<div>

</div>
<div>
trim_rule = vocabulary trimming rule specifies whether certain words should remain in the vocabulary, be trimmed away, or handled using the default (discard if word count &lt; min_count). Can be None (min_count will be used), or a callable that accepts parameters (word, count, min_count) and returns either utils.RULE_DISCARD, utils.RULE_KEEP or utils.RULE_DEFAULT. Note: The rule, if given, is only used to prune vocabulary during build_vocab() and is not stored as part of the model.</div>
<div>

</div>
<div>
sorted_vocab = if 1 (default), sort the vocabulary by descending frequency before assigning word indexes.</div>
<div>

</div>
<div>
batch_words = target size (in words) for batches of examples passed to worker threads (and thus cython routines). Default is 10000. (Larger batches will be passed if individual texts are longer than 10000 words, but the standard cython code truncates to that maximum.)</div>
</div>
<h3 style="text-align: left;">
Using the model</h3>
<div>
<pre style="background-color: rgba(0,0,0,0.1); border-bottom: 1px solid rgb(170, 204, 153); border-image: initial; border-left: none; border-right: none; border-top: 1px solid rgb(170, 204, 153); line-height: 15.6px; overflow-x: auto; overflow-y: hidden; padding: 5px;">#saving the model persistence
model.save('model.bin')

#loading the model
model = gensim.models.KeyedVectors.load_word2vec_format('model.bin', binary=True) 

#getting the most similar words
model.most_similar(positive=['beautiful', 'world'], negative=['convert'], topn=1)

#finding the odd one out
model.doesnt_match("bullish holding stock".split())

#getting the vector for any word
model[word]

#finding the similarity between words
model.similarity('woman', 'man')


</pre>
</div>
<div>

</div>
<div>

</div>
<div>
For more details, you can read the documentation of the word2vec&nbsp; gensim <span id="goog_193205381"></span><span id="goog_193205382"></span><a href="https://radimrehurek.com/gensim/models/word2vec.html" target="_blank">here</a>.</div>
</div>
</div>