GloVe for Word Vectorization
Tags: wordvectorization, python, glove
Published: 2018-05-22 04:00:23

---
title: GloVe for Word Vectorization
published: true
description: This article describe the usage of GloVe vectorization technique in python.
tags: Word Vectorization,Python,Glove
canonical_url: http://techscouter.blogspot.in/2018/04/getting-started-with-glove.html
cover_image: https://thepracticaldev.s3.amazonaws.com/i/sps3aj9mu0qaolbn5jbp.jpeg
---

<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-U8Cjp06jluc/Wt7rRjQ3KKI/AAAAAAAAbjE/DGD3_b3UOHYyUjsA4DbxJsxXgqG6jX_YwCLcBGAs/s1600/Text-to-vector-3_normal.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="508" data-original-width="1040" height="312" src="https://3.bp.blogspot.com/-U8Cjp06jluc/Wt7rRjQ3KKI/AAAAAAAAbjE/DGD3_b3UOHYyUjsA4DbxJsxXgqG6jX_YwCLcBGAs/s640/Text-to-vector-3_normal.png" width="640" /></a></div>

<h2 style="text-align: left;">
What is GloVe?</h2>
<div style="text-align: left;">
GloVe stands for global vectors for word representation. It is an unsupervised learning algorithm developed by Stanford for generating word embeddings by aggregating global word-word co-occurrence matrix from a&nbsp;corpus. The resulting embeddings show interesting linear substructures of the word in vector space.</div>
<div style="text-align: left;">

</div>
<div style="text-align: left;">
Examples for linear substructures&nbsp;are:</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-TcJsh6VB75E/Wt7xk0VBQxI/AAAAAAAAbjY/7_pS22MdugQBQ9vOBJUV2hsm2b9bcVbZQCEwYBhgL/s1600/vec2.jpg" imageanchor="1" style="display: inline !important; margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="779" data-original-width="1000" height="496" src="https://3.bp.blogspot.com/-TcJsh6VB75E/Wt7xk0VBQxI/AAAAAAAAbjY/7_pS22MdugQBQ9vOBJUV2hsm2b9bcVbZQCEwYBhgL/s640/vec2.jpg" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">

</div>
<a href="https://4.bp.blogspot.com/-8v4NzcL-vvs/Wt7xk6PbLZI/AAAAAAAAbjU/XBmQq_Jnb-UgW6FeoSZ6kQdrejwdjNAhQCLcBGAs/s1600/vec1.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="775" data-original-width="1000" height="496" src="https://4.bp.blogspot.com/-8v4NzcL-vvs/Wt7xk6PbLZI/AAAAAAAAbjU/XBmQq_Jnb-UgW6FeoSZ6kQdrejwdjNAhQCLcBGAs/s640/vec1.jpg" width="640" /></a>
<div style="text-align: left;">

</div>
<div style="text-align: left;">
These results are pretty impressive. This type of representation can be very useful in many machine learning algorithms.</div>
<div style="text-align: left;">

</div>
<div style="text-align: left;">
To read more about Word Vectorization you can read my other <a href="http://techscouter.blogspot.in/2017/10/word-vectorization.html" target="_blank">article</a>.</div>
<div style="text-align: left;">

</div>
<div style="text-align: left;">
In this blog post, we will be learning about GloVe implementation in python. So, let's get started.</div>
<h2 style="text-align: left;">
Let's create the Embeddings</h2>
<h3 style="text-align: left;">
Installing Glove-Python</h3>
<div style="text-align: left;">
The GloVe is implementation in python is available in&nbsp;library&nbsp;<b>glove-python.</b></div>
<div>
<div>
<div class="highlight-default" style="background-color: white; color: #333333; font-family: arial; font-size: 14px;">
<div class="highlight" style="background: rgb(238, 255, 204);">
<pre style="background-color: #f8f8f8; border: 1px solid rgb(204, 204, 204); font-family: consolas, &quot;deja vu sans mono&quot;, &quot;bitstream vera sans mono&quot;, monospace; font-size: 0.95em; letter-spacing: 0.015em; line-height: 15.96px; overflow-x: auto; padding: 0.5em;"><div style="color: black; font-family: &quot;Times New Roman&quot;; font-size: medium; letter-spacing: normal; white-space: normal;">
<span style="background-color: rgba(27 , 31 , 35 , 0.05); color: #24292e; font-family: , &quot;consolas&quot; , &quot;liberation mono&quot; , &quot;menlo&quot; , &quot;courier&quot; , monospace; font-size: 13.6px;">pip install glove_python</span></div>
<div style="color: black; font-family: &quot;Times New Roman&quot;; font-size: medium; letter-spacing: normal; white-space: normal;">
<div class="highlight-default" style="background-color: white; color: #333333; font-family: arial; font-size: 14px;">
<div class="highlight" style="background: rgb(238, 255, 204);">
</div>
</div>
</div>
</pre>
</div>
</div>
</div>
</div>
<h3 style="text-align: left;">
<div style="-webkit-text-stroke-width: 0px; font-family: &quot;Times New Roman&quot;; font-size: medium; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;">
<div>
<div>
<h3 style="margin: 0px; text-align: left;">
Text Preprocessing</h3>
<div style="margin: 0px;">
In this step, we will pre-process the text like removing the stop words, lemmatize the words etc.</div>
</div>
<div>
<div style="margin: 0px;">
You can perform different steps based on your requirements.</div>
</div>
<div>
<div style="margin: 0px;">

</div>
</div>
<div>
<div style="margin: 0px;">
I will use nltk stopword corpus for stop word removal and nltk word lemmatization for finding lemmas.</div>
</div>
<div>
<div style="margin: 0px;">
I order to use nltk corpus you will need to download it using the following commands.</div>
</div>
<div>
<div style="margin: 0px;">
<b>
</b></div>
</div>
<div>
<div style="margin: 0px;">
<b>Downloading the corpus</b></div>
<div style="margin: 0px;">
<b>
</b></div>
</div>
<div>
<pre style="background-color: rgba(0, 0, 0, 0.1); border-bottom: 1px solid rgb(170, 204, 153); border-left: none; border-right: none; border-top: 1px solid rgb(170, 204, 153); line-height: 15.6px; margin: 0px; overflow-x: auto; overflow-y: hidden; padding: 5px;">import nltk
nltk.download()
#this will open a GUI from which you can download the corpus
</pre>
</div>
<div>
<div style="margin: 0px;">

</div>
</div>
<div>
<div style="margin: 0px;">
<b>Input initialization</b></div>
</div>
<div style="margin: 0px;">

</div>
<pre style="background-color: rgba(0, 0, 0, 0.1); border-bottom: 1px solid rgb(170, 204, 153); border-left: none; border-right: none; border-top: 1px solid rgb(170, 204, 153); line-height: 15.6px; margin: 0px; overflow-x: auto; overflow-y: hidden; padding: 5px;">#list of sentences to be vectorized
lines=["Hello this is a tutorial on how to convert the word in an integer format",
"this is a beautiful day","Jack is going to office"]</pre>
<div>
<div style="margin: 0px;">
<b>
</b></div>
</div>
<div>
<div style="margin: 0px;">
<b>Removing the Stop Words</b></div>
</div>
<div>
<div style="margin: 0px;">
<b>
</b></div>
</div>
<div>
<pre style="background-color: rgba(0, 0, 0, 0.1); border-bottom: 1px solid rgb(170, 204, 153); border-left: none; border-right: none; border-top: 1px solid rgb(170, 204, 153); line-height: 15.6px; margin: 0px; overflow-x: auto; overflow-y: hidden; padding: 5px;">from nltk.corpus import stopwords
stop_words=set(stopwords.words('english'))

lines_without_stopwords=[]
#stop words contain the set of stop words
for line in lines:
 temp_line=[]
 for word in lines:
  if word not in stop_words:
   temp_line.append (word)
 string=' '
 lines_without_stopwords.append(string.join(temp_line))

lines=lines_without_stopwords
</pre>
</div>
<div>
<div style="margin: 0px;">
<b>
</b></div>
</div>
<div>
<div style="margin: 0px;">
<b>Lemmatization</b></div>
<div style="margin: 0px;">
<b>
</b></div>
</div>
<div>
<pre style="background-color: rgba(0, 0, 0, 0.1); border-bottom: 1px solid rgb(170, 204, 153); border-left: none; border-right: none; border-top: 1px solid rgb(170, 204, 153); line-height: 15.6px; margin: 0px; overflow-x: auto; overflow-y: hidden; padding: 5px;">#import WordNet Lemmatizer from nltk
from nltk.stem import WordNetLemmatizer
wordnet_lemmatizer = WordNetLemmatizer()


lines_with_lemmas=[]
#stop words contain the set of stop words
for line in lines:
 temp_line=[]
 for word in lines:
  temp_line.append (wordnet_lemmatizer.lemmatize(word))
 string=' '
 lines_with_lemmas.append(string.join(temp_line))
lines=lines_with_lemmas


</pre>
</div>
<div>
<div style="margin: 0px;">

</div>
</div>
<div>
<div style="margin: 0px;">

</div>
<div style="margin: 0px;">
Now we have done the basic preprocessing of the text. Any other preprocessing stuff can be achieved similarly.</div>
<div style="margin: 0px;">

</div>
</div>
</div>
</div>
</h3>
<h3 style="text-align: left;">
Preparing Input</h3>
<h3 style="text-align: left;">
<div style="font-size: medium; font-weight: 400;">
<div>
</div>
</div>
<div style="font-size: medium; font-weight: 400;">
We have out input in the form of an array of lines. In order for the model to process the data, we need covert our input to an array of array of words ( :\ ).</div>
<div style="font-size: medium; font-weight: 400;">
<b>
</b></div>
<div style="font-size: medium; font-weight: 400;">
<u><i>Our Input</i></u></div>
<div style="font-size: medium; font-weight: 400;">
<div>
lines=["Hello this is a tutorial on how to convert the word in an integer format",</div>
<div>
"this is a beautiful day","Jack is going to office"]</div>
</div>
<div style="font-size: medium; font-weight: 400;">
<b>
</b></div>
<div style="font-size: medium; font-weight: 400;">
<u><i>New Input</i></u></div>
<div style="font-size: medium; font-weight: 400;">
<div>
lines=[['Hello', 'this','tutorial', 'on', 'how','convert' ,'word',' integer','format'],</div>
<div>
['this' ,'beautiful', 'day'],['Jack','going' , 'office']</div>
</div>
<div style="font-size: medium; font-weight: 400;">

</div>
<div style="font-size: medium; font-weight: 400;">
<pre style="background-color: rgba(0, 0, 0, 0.1); border-bottom: 1px solid rgb(170, 204, 153); border-left: none; border-right: none; border-top: 1px solid rgb(170, 204, 153); line-height: 15.6px; overflow-x: auto; overflow-y: hidden; padding: 5px; text-align: left;">new_lines=[]
for line in lines:
 new_lines=line.split('')
#new lines has the new format
lines=new_lines
</pre>
<h3>
Building a Glove model</h3>
<div>
</div>
</div>
</h3>
<div>
<pre style="background-color: rgba(0,0,0,0.1); border-bottom: 1px solid rgb(170, 204, 153); border-image: initial; border-left: none; border-right: none; border-top: 1px solid rgb(170, 204, 153); line-height: 15.6px; overflow-x: auto; overflow-y: hidden; padding: 5px;">#importing the glove library
from glove import Corpus, Glove

# creating a corpus object
corpus = Corpus() 

#training the corpus to generate the co occurence matrix which is used in GloVe
corpus.fit(lines, window=10)

#creating a Glove object which will use the matrix created in the above lines to create embeddings
#We can set the learning rate as it uses Gradient Descent and number of components

glove = Glove(no_components=5, learning_rate=0.05)
 
glove.fit(corpus.matrix, epochs=30, no_threads=4, verbose=True)
glove.add_dictionary(corpus.dictionary)
glove.save('glove.model')</pre>
</div>
<div>

</div>
<div>
Creating a glove model uses the co-occurrence matrix&nbsp;generated by the Corpus object to create the embeddings.</div>
<div>
The <b>corpus.fit</b> takes two arguments:</div>
<div>
<ul style="text-align: left;">
<li>lines - this is the 2D array we created after the&nbsp; pre-processing</li>
<li>window - this is the distance between two words algo should consider to&nbsp;find some relationship between them</li>
</ul>
<div>
Parameters of Glove:</div>
</div>
<div>
<ul style="text-align: left;">
<li>no_of_components - This is the dimension of the output vector generated by the GloVe</li>
<li>learning_rate - Algo uses gradient descent so learning rate defines the rate at which the algo&nbsp;reaches towards the minima (lower the rate more time it takes to learn but reaches the minimum value)</li>
</ul>
<div>
Parameters of glove.fit :</div>
</div>
<div>
<ul style="text-align: left;">
<li>cooccurence_matrix: the matrix of word-word co-occurrences</li>
<li>epochs: this defines the number of passes algo makes through the data set</li>
<li>no_of_threads: number of threads used by the algo to run</li>
</ul>
</div>
<div>
After the training glove object has the word vectors for the lines we have provided. But the dictionary still resides in the corpus object. We need to add the dictionary to the glove object to make it complete.</div>
<div>

</div>
<div>
<pre style="background-color: rgba(0,0,0,0.1); border-bottom: 1px solid rgb(170, 204, 153); border-image: initial; border-left: none; border-right: none; border-top: 1px solid rgb(170, 204, 153); line-height: 15.6px; overflow-x: auto; overflow-y: hidden; padding: 5px;">glove.add_dictionary(corpus.dictionary)
</pre>
</div>
<div>

</div>
<div>
This line does the dictionary addition in the glove object. After this, the object is ready to provide you with the embeddings.</div>
<div>
</div>
<div>
<div>
<pre style="background-color: rgba(0,0,0,0.1); border-bottom: 1px solid rgb(170, 204, 153); border-image: initial; border-left: none; border-right: none; border-top: 1px solid rgb(170, 204, 153); line-height: 15.6px; overflow-x: auto; overflow-y: hidden; padding: 5px;">print glove.word_vectors[glove.dictionary['samsung']]
OUTPUT:
<pre style="background-color: white; border-radius: 0px; border: 0px; box-sizing: border-box; font-size: 14px; line-height: inherit; overflow: auto; padding: 0px; vertical-align: baseline; white-space: pre-wrap; word-break: break-all; word-wrap: break-word;">[ 0.04521741  0.02455266 -0.06374787 -0.07107575  0.04608054]</pre>
</pre>
</div>
</div>
<div>

</div>
<div>
This will print the embeddings for the word "samsung".</div>
<h2 style="text-align: left;">
End Notes&nbsp;</h2>
<div>
We have learned how to generate the vectors for the text data which is very useful for creating many Machine Learning models and techniques like SVM, KNN, K-Means, Logistic Classifiers, Sentiment Analysis, Document Classification etc.</div>
<div>
More can be learned about the GloVe on Stanford's <a href="https://nlp.stanford.edu/projects/glove/" target="_blank">website</a>.&nbsp;</div>
<div>

</div>
<div>

</div>
<div>

</div>
</div>