Topic Modeling
Tags: textanalysis, analysis, topicmining, textmining
Published: 2017-08-22 04:11:25

---
title: Topic Modeling
published: true
description: An introduction to topic modelling and a sample code for usage of LDA Model in python. 
tags: TextAnalysis, Analysis, TopicMining, TextMining
canonical_url: http://techscouter.blogspot.in/2017/08/text-analytics-part-2.html
cover_image: https://thepracticaldev.s3.amazonaws.com/i/j0agidog6ti3t9spf3bp.png
---

<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-2R8o8TuByy0/WZupbHOH4iI/AAAAAAAABtw/mXSUjcGlp7wDJ9jRxAlvxF-f67v8HqsoQCLcBGAs/s1600/tm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="185" data-original-width="410" src="https://2.bp.blogspot.com/-2R8o8TuByy0/WZupbHOH4iI/AAAAAAAABtw/mXSUjcGlp7wDJ9jRxAlvxF-f67v8HqsoQCLcBGAs/s1600/tm.png" /></a></div>
<div class="separator" style="clear: both; text-align: center;">

</div>
Hi readers,
<div>
In the previous post, I wrote about gaining the knowledge from the Text which is available from many sources. In this post, I will be writing about Topic Mining.</div>
<div>

</div>
<h2 style="text-align: left;">
Introduction</h2>
<div style="text-align: left;">
Topic Mining can be described as finding words from the group of words which can best describe the group.</div>
<div style="text-align: left;">
Textual Data in raw form is not associated with any context. A human can easily identify the context or topic for an article by reading the article and categorise&nbsp;it in one or other category like politics, sports, economics, crime etc.</div>
<div style="text-align: left;">
One of the factors any human will consider while classifying the text into one of the topics is the knowledge that how a word is associated with a topic e.g</div>
<div style="text-align: left;">
</div>
<div style="text-align: left;">
</div>
<ul style="text-align: left;">
<li>India won Over Sri Lanka in the <b>test match</b>.</li>
</ul>
<ul style="text-align: left;">
<li>World <b>Badminton </b>Championships: When and where to watch <b>Kidambi Srikanth</b>’s first round, live TV coverage, time in IST, live&nbsp;streaming&nbsp;&nbsp;</li>
</ul>
<div>
Here we may not find word sports explicitly in the sentences but the words marked in bold are associated with sports.</div>
<div>

</div>
<div>
Topic modelling can be broadly categorised into two type</div>
<div>
<ol style="text-align: left;">
<li>Rule-Based topic modelling</li>
<li>Unsupervised topic modelling</li>
</ol>
<h2 style="text-align: left;">
Rule-Based Topic Modelling</h2>
</div>
<div>
As the name suggests rule-based&nbsp;topic modelling depends on the rules which can be used to associate a given text with some topic.</div>
<div>

</div>
<div>
In the simplest rule-based approach, we can just search for some words in the text and associate it with a topic e.g finding the word sports for associating the topic with <i>sports</i>, finding travelling for associating with topic <i>travel</i></div>
<div>
<i>
</i></div>
<div>
This approach can be extended and a topic can be represented as a set of words with some given probabilities e.g For the category sports we can have a set of words with some weight assigned to each word.
Topic : Sports{"<i>sports"</i>:0.4,"<i>cricket</i>":0.1,"<i>badminton</i>":0.1 ,"<b>traveling</b>":0.05 .....}
Topic : Travel{"<i>travel</i>":0.4, "<i>hiking</i>":0.1,"<i>train</i>":0.05,"<b>traveling</b>":0.20 .......}
<i>
</i> <i>Notice the word &nbsp;"travelling", it occurs&nbsp;in both the Topics but has different weight.</i>

If we have a sentence "<i>Badminton</i>&nbsp;<i>players are travelling to UK&nbsp;&nbsp;for the tournament</i>", by the simple approach of finding the words for the topic then this sentence will go under the topic <i>Travel. </i>The second approach improves the prediction by checking the probabilities and weight for different words, in this case, "<i>Badminton" </i>&nbsp;and "travelling" and improves the result by predicting the more accurate result that is <i>Sports</i>.

The main disadvantage of the Rule-Based approach is that all the topics have to be known in the beginning and probabilities have to be determined and examined. This rules out the possibility of finding some new topic in the text corpus.</div>
<div>

</div>
<h2 style="text-align: left;">
Unsupervised Topic Modelling</h2>
<div>

</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-g3dI_6STH5s/WZuqBxnrhgI/AAAAAAAABt8/3AGn9JJPL7EHtkNl06Gpms2uMXLxKqj3QCLcBGAs/s1600/lda.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="481" data-original-width="914" height="336" src="https://1.bp.blogspot.com/-g3dI_6STH5s/WZuqBxnrhgI/AAAAAAAABt8/3AGn9JJPL7EHtkNl06Gpms2uMXLxKqj3QCLcBGAs/s640/lda.png" width="640" /></a></div>
<div>

</div>
<div>
The topic of a text sentence largely depends on the words used in the sentence and this property is exploited in unsupervised topic modelling technique to extract topics from the sentences.
It largely relies on the Bayesian Inference Model.
<h3 style="text-align: left;">
Bayesian Inference Model</h3>
</div>
<div>
It is a method by which we can calculate the probability of occurrence of some event-based on some common sense assumptions and the outcomes of previous related events.</div>
<div>
It also allows us to use new observations to improve the model, by going through many iterations where a prior probability is updated with the observational evidence in order to produce a new &nbsp;and improved posterior probability</div>
<div style="text-align: left;">
<h3 style="text-align: left;">
Some of the techniques for Unsupervised Topic Modelling are:</h3>
<div>
<ul style="text-align: left;">
<li>TF-IDF</li>
<li>Latent Semantic Indexing</li>
<li>Latent Dirichlet Allocation</li>
</ul>
I will not discuss the techniques in detail rather will focus on the implementation of these in python.
All the approaches use the vector space representation of the documents. In vector space model a document is represented by a document-term matrix.

<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-MF9AfD1OzdE/WZphHhbDM0I/AAAAAAAABtg/0YAaHqg6-H4EtrE4zA_2TBMHKxQiQzrHgCLcBGAs/s1600/Modeling2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="119" data-original-width="154" src="https://3.bp.blogspot.com/-MF9AfD1OzdE/WZphHhbDM0I/AAAAAAAABtg/0YAaHqg6-H4EtrE4zA_2TBMHKxQiQzrHgCLcBGAs/s1600/Modeling2.png" /></a></div>

</div>
Here D1,D2,D3...Dn are different documents and W1,W2,W3....Wn are the words that belong to our dictionary(all unique words in the corpus). DxWy gives the number of time the word has occurred in the document.

<pre style="background-color: lightgrey; border-radius: 4px; border: 1px solid rgba(0, 0, 0, 0.15); line-height: 20px; margin-bottom: 10px; overflow-x: auto; overflow-y: hidden; padding: 9.5px; word-break: break-all; word-wrap: break-word;"><div class="line number1 index0 alt2" style="background-image: none; border-radius: 0px; border: 0px; bottom: auto; box-shadow: none; box-sizing: content-box; color: #2b2b2b; direction: ltr; float: none; height: auto; left: auto; line-height: 1.1em; margin: 0px; outline: 0px; overflow: visible; padding: 0px 1em; position: static; right: auto; top: auto; vertical-align: baseline; width: auto;">
<div class="gmail_default" style="color: #222222; white-space: normal;">
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xxx-small;">#importing corpora from gensim for creating dictionary</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">from gensim import corpora</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;"></span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">#some raw&nbsp;documents .These documents can be stored in a file and read from the file</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">documents = ["Two analysts have provided estimates for Tower Semiconductor's earnings. The stock of Tower Semiconductor Ltd. (USA) (NASDAQ:TSEM) has \"Buy\" rating given on Friday, June 9 by Needham. The company's stock had a trading volume of 573,163 shares.",</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;"More news for Tower Semiconductor Ltd. (USA) (NASDAQ:TSEM)&nbsp;were&nbsp;recently published&nbsp;by:&nbsp;Benzinga.com, which released: “15 Biggest Mid-Day Gainers For Monday” on August 14, 2017.",</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;"Tower Semiconductor Ltd now has $2.33 billion&nbsp;valuation. The stock decreased 1.61% or $0.41 during the last trading session, reaching $25.06.",</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;"The investment company owned 74,300 stocks of the industrial products firms shares after&nbsp;disposing&nbsp;5,700 shares through out the quarter.", &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;"Tower Semiconductor Ltd now has&nbsp;$2.37B&nbsp;valuation. The stock rose 0.16% or $0.04 reaching $25.1 per share. About 115,031 shares traded.",</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;"Needle moving action has been spotted in Steven Madden Ltd (SHOO) as shares are moving today on volatility -2.00% or -0.85 from the open.",</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;"Shares of Steven Madden Ltd (SHOO) have seen the needle move -1.20% or -0.50 in the most recent session. The NASDAQ listed company saw a recent bid of 41.10 on 82437 volume.",</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;"Shares of Steven Madden Ltd (SHOO) is moving on volatility today -1.37% or -0.57 from the open. The NASDAQ listed company saw a recent bid of 41.03 on 279437 volume.",</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;"Shares of Steven Madden, Ltd. (NASDAQ:SHOO) are on a recovery track as they have regained 28.79% since bottoming out at $32.3 on Oct. 26, 2016."]</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;"></span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">'''</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">&nbsp; &nbsp; Text Preprocessiing before calculation vectors</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">'''</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;"># remove common words(stop words) and tokenize</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">stoplist = set('for a of the and to in its by his has have on were was which or as they since'.split())</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">texts = [[word for word in document.lower().split() if word not in stoplist]</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;for document in documents]</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
</div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;"></span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;"># remove words that appear only once in the corpus.</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">from collections import defaultdict</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">frequency = defaultdict(int)</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">for text in texts:</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">&nbsp; &nbsp; for token in text:</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">&nbsp; &nbsp; &nbsp; &nbsp; frequency[token] += 1</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;"></span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">texts = [[token for token in text if frequency[token] &gt; 1] for text in texts]</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;"></span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">from pprint import pprint &nbsp;# pretty-printer</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">pprint(texts)</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">'''</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">dictionary is collection of all the words that apprear in our corpus&nbsp;</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">'''</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">#creating dictionary from the corpus which represents each word in the text corpus as a unique integer</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">dictionary = corpora.Dictionary(texts)</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">dictionary.save('/tmp/<wbr></wbr>deerwester.dict') &nbsp;# store the dictionary, for future reference</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">#saved dictionary can be loaded from the memory for future references.Saves time for recreation of dictionary</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;"></span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;"></span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">#to print the token id of the words</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">print(dictionary.token2id)</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">'''</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">We can convert any document to vector format by just calling the doc2bow() method of dictionary object.</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">document="hello this is a good morning"</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">Input=['hello','this','is','<wbr></wbr>a','good','morning']</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">preprocessing can be done on the document before passing to doc2bow()</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">'''</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">corpus = [dictionary.doc2bow(text) for text in texts]</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">print corpus</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;"></span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;"></span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">'''</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">creating vector space representation for a single document</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">str="human interaction interface interface"</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">dictionary.doc2bow(text for text in str.split(' '))</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">'''</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;"></span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;"></span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;"></span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;"></span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">from gensim import corpora, models, similarities</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
</div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">#convertig the vector space representation to tf-idf transformation</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">tfidf = models.TfidfModel(corpus)</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">list(tfidf[corpus])</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;"></span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">#using LDA model for getting the topics</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">lda= models.LdaModel(corpus, id2word=dictionary, num_topics=2) # initialize an LDA transformation</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">corpus_lda= lda[corpus] # create a double wrapper over the original corpus: bow-&gt;tfidf-&gt;fold-in-lda</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">print lda.print_topics(2) #print the topics</span>
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">
</span>
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">#for classifying the text documents into topics&nbsp;</span>
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">#here I have used the old corpus on which it was trained ,one can also use new documents&nbsp;</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">for doc in corpus_lda:</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">&nbsp; &nbsp; print doc</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">&nbsp; &nbsp;&nbsp;</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;"></span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">Output:</span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<pre style="background-color: white; border-radius: 0px; border: 0px; box-sizing: border-box; color: black; font-size: 11px; line-height: inherit; overflow: auto; padding: 0px; vertical-align: baseline; white-space: pre-wrap; word-break: break-all; word-wrap: break-word;">[</pre>
<pre style="background-color: white; border-radius: 0px; border: 0px; box-sizing: border-box; color: black; font-size: 11px; line-height: inherit; overflow: auto; padding: 0px; vertical-align: baseline; white-space: pre-wrap; word-break: break-all; word-wrap: break-word;">(0, u'0.069*"shares" + 0.062*"ltd" + 0.059*"steven" + 0.044*"madden" + 0.044*"tower" + 0.043*"moving" + 0.043*"recent" + 0.041*"stock" + 0.040*"(shoo)" + 0.036*"semiconductor"'),</pre>
<pre style="background-color: white; border-radius: 0px; border: 0px; box-sizing: border-box; color: black; font-size: 14px; line-height: inherit; overflow: auto; padding: 0px; vertical-align: baseline; white-space: pre-wrap; word-break: break-all; word-wrap: break-word;"></pre>
<pre style="background-color: white; border-radius: 0px; border: 0px; box-sizing: border-box; color: black; font-size: 14px; line-height: inherit; overflow: auto; padding: 0px; vertical-align: baseline; white-space: pre-wrap; word-break: break-all; word-wrap: break-word;">(1, u'0.077*"shares" + 0.066*"tower" + 0.056*"semiconductor" + 0.050*"stock" + 0.048*"company" + 0.046*"ltd" + 0.045*"ltd." + 0.043*"(usa)" + 0.038*"(nasdaq:tsem)" + 0.035*"trading"')]</pre>
</div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;"></span></div>
<div class="gmail_default">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;">#the output gives the topics along with there probability distribution for different words</span></div>
<div class="gmail_default">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;"></span></div>
<div class="gmail_default" style="font-family: arial, sans-serif;">
<span style="font-family: &quot;verdana&quot; , sans-serif; font-size: xx-small;"></span></div>
</div>
</div>
</pre>

Parameters of LdaModel can be tweaked for result improvements.Also by just replacing LdaModel by LsiModel one can use LSI technique


</div>
</div>