Spidering the WEB 
Tags: scraping, mining, analysis
Published: 2017-09-17 11:37:28

---
title: Spidering the WEB 
published: true
description: This is an introduction of web scraping using python.
tags: Scraping, Mining, Analysis
canonical_url: http://techscouter.blogspot.in/2017/09/spidering-web-with-python.html
---

<div dir="ltr" style="text-align: left;" trbidi="on">
<h2 style="height: 0px;">
</h2>
<h2 style="height: 0px;">
</h2>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-wufi5ZCBw_E/Wbtjh0rRpFI/AAAAAAAAByU/O2ph9Y0cSJcRnIi23pNT4PkiFcifJ5RZQCLcBGAs/s1600/web-scraping-services.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="224" data-original-width="658" src="https://2.bp.blogspot.com/-wufi5ZCBw_E/Wbtjh0rRpFI/AAAAAAAAByU/O2ph9Y0cSJcRnIi23pNT4PkiFcifJ5RZQCLcBGAs/s1600/web-scraping-services.png" /></a></div>
<h2 style="height: 0px;">
Introduction</h2>
<div>

</div>
<div>
We will be talking about&nbsp;</div>
<div>
<ul style="text-align: left;">
<li>Spidering/Scraping</li>
<li>How to do it elegantly in python</li>
<li>Limitations and restriction&nbsp;</li>
</ul>
<div>

</div>
</div>
<div>
In the previous posts, I shared some of the methods of text mining and analytics but one of the major and most important&nbsp;tasks before analytics is getting the data which we want to analyze.</div>
<div>
Text data is present all over in forms of blogs, articles, news, social feeds, posts etc and most of it is distributed to users in the form of API's, RSS feeds, Bulk downloads and Subscriptions.</div>
<div>

</div>
<div>
Some sites do not provide any means of pulling the data programmatically, this is where scrapping comes into the picture.</div>
<div>

</div>
<div>
<i><b>Note: </b>Scraping information from the sites which are not free or is not publically available can have serious consequences.</i></div>
<div>

</div>
<div>
Web Scraping is a technique of getting a web page in the form of HTML and parsing it to get the desired information.

HTML is very complex in itself due to loose rules and a large number of attributes. Information can be scraped in two ways:

<ul style="text-align: left;">
<li>Manually filtering using regular expressions</li>
<li>Python's way -Beautiful&nbsp;Soup</li>
</ul>
<div>
In this post, we will be discussing beautiful soup's way of scraping.</div>
<div>

</div>
<div>

</div>
<h2 style="text-align: left;">
Beautiful Soup</h2>
<div>
As per the definition&nbsp;in its <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/" target="_blank">documentation</a>&nbsp;</div>
<div>

</div>
<div>
<i>"<a class="reference external" href="http://www.crummy.com/software/BeautifulSoup/" style="color: #355f7c; font-family: sans-serif; font-size: 16px; text-align: justify; text-decoration-line: none;">Beautiful Soup</a><span style="font-family: sans-serif; font-size: 16px; text-align: justify;">&nbsp;is a Python library for pulling data out of HTML and XML files. It works with your favorite parser to provide idiomatic ways of navigating, searching and modifying the parse tree. It commonly saves programmers hours or days of work.</span>"</i></div>
<div>
<i>
</i></div>
<div>
<i>
</i></div>
<div>
If you have ever tried something like parsing texts and HTML&nbsp;documents that you will understand how brilliantly this module is built and really save a lot of programmers work and time.</div>
<div>

</div>
<div>
Let's start with beautiful soup</div>
<h3 style="text-align: left;">
Installation</h3>
<div>
I hope python is installed in your system. To install Beautiful Soup you can use pip

</div>
<div style="background: rgba(0 , 0 , 0 , 0.1); display: inline-block; padding: 5px; width: auto;">
pip install beautifulsoup4</div>
<h3 style="text-align: left;">
Getting Started</h3>
</div>
<div>
<b>Problem 1: Getting all the links from a page.</b></div>
<div>
For this problem, we will use a sample HTML string which has some links and our goal is to get all the links</div>
<div>
<pre style="background-color: rgba(0,0,0,0.1); border-bottom: 1px solid rgb(170, 204, 153); border-image: initial; border-left: none; border-right: none; border-top: 1px solid rgb(170, 204, 153); line-height: 15.6px; overflow-x: auto; overflow-y: hidden; padding: 5px;"><span class="n" style="color: #333333;">html_doc</span><span style="color: #333333;"> </span><span class="o" style="color: #666666;">=</span><span style="color: #333333;"> </span><span class="s2" style="color: #4070a0;">"""</span><span style="color: #333333;">
</span><span style="color: #4070a0;">&lt;html&gt;
&lt;body&gt;
&lt;h1&gt;Sample Links&lt;/h1&gt;
&lt;br&gt;
&lt;a href="https://www.google.com"&gt;Google&lt;/a&gt;
&lt;br&gt;
&lt;a href="https://www.apple.com"&gt;Apple&lt;/a&gt;
&lt;br&gt;
&lt;a href="https://www.yahoo.com"&gt;Yahoo&lt;/a&gt;
&lt;br&gt;
&lt;a href="https://www.msdn.com"&gt;MSDN&lt;/a&gt;
&lt;/body&gt;
&lt;/html&gt;</span><span style="color: #333333;">
</span><span class="s2" style="color: #4070a0;">"""</span></pre>
</div>
<div>

</div>
<div>

</div>
<div>
<pre style="background-color: rgba(0,0,0,0.1); border-bottom: 1px solid rgb(170, 204, 153); border-image: initial; border-left: none; border-right: none; border-top: 1px solid rgb(170, 204, 153); line-height: 15.6px; overflow-x: auto; overflow-y: hidden; padding: 5px;">#to import the package 
from bs4 import BeautifulSoup


#creating an object of BeautifulSoup and pass 2 parameters
#1)the html t be scanned
#2)the parser to be used(html parser ,lxml parser etc)
soup=BeautifulSoup(html_doc,"html.parser")


#to find all the anchor tags in the html string
#findAll returns a list of tags in thi scase anchors(to get first one we can use find )
anchors=soup.findAll('a')

#getting links from anchor tags
for a in anchor:
    print a.get('href') #get is used to get the attributes of a tags element
    #print a['href'] can also be used to access the attribute of a tag

</pre>

</div>
<div>
This is it, just 5-6 lines to get any tag from the the html and iterating over it, finding some attriutes.
Can you this of doing this with the help of regular expressions. It will be one heck of a job doing it with RE. We can think of how well the module is coded to perform all this functions.

Talking about the parsers (one we have passed while creating a Beautiful Soup object), we have multiple choices if parsers.

<div style="background-color: white; font-family: sans-serif; font-size: 16px; line-height: 20.8px; text-align: justify;">
This table summarizes the advantages and disadvantages of each parser library:</div>
<div style="background-color: white; font-family: sans-serif; font-size: 16px; line-height: 20.8px; text-align: justify;">

</div>
<div style="overflow: auto;">
<table border="1" class="docutils" style="background-color: white; border-collapse: collapse; border: 0px; color: black; font-family: sans-serif; font-size: 16px;"><colgroup><col width="18%"></col><col width="35%"></col><col width="26%"></col><col width="21%"></col></colgroup><tbody valign="top">
<tr class="row-odd"><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: solid; border-left-color: initial; border-left-style: initial; border-right-color: initial; border-right-style: initial; border-top-color: initial; border-top-style: initial; border-width: 0px 0px 1px; padding: 1px 8px 1px 5px;"><span style="font-size: x-small;">Parser</span></td><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: solid; border-left-color: initial; border-left-style: initial; border-right-color: initial; border-right-style: initial; border-top-color: initial; border-top-style: initial; border-width: 0px 0px 1px; padding: 1px 8px 1px 5px;"><span style="font-size: x-small;">Typical usage</span></td><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: solid; border-left-color: initial; border-left-style: initial; border-right-color: initial; border-right-style: initial; border-top-color: initial; border-top-style: initial; border-width: 0px 0px 1px; padding: 1px 8px 1px 5px;"><span style="font-size: x-small;">Advantages</span></td><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: solid; border-left-color: initial; border-left-style: initial; border-right-color: initial; border-right-style: initial; border-top-color: initial; border-top-style: initial; border-width: 0px 0px 1px; padding: 1px 8px 1px 5px;"><span style="font-size: x-small;">Disadvantages</span></td></tr>
<tr class="row-even"><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: solid; border-left-color: initial; border-left-style: initial; border-right-color: initial; border-right-style: initial; border-top-color: initial; border-top-style: initial; border-width: 0px 0px 1px; padding: 1px 8px 1px 5px;"><span style="font-size: x-small;">Python’s html.parser</span></td><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: solid; border-left-color: initial; border-left-style: initial; border-right-color: initial; border-right-style: initial; border-top-color: initial; border-top-style: initial; border-width: 0px 0px 1px; padding: 1px 8px 1px 5px;"><code class="docutils literal" style="background-color: #ecf0f3; padding: 0px 1px;"><span style="font-size: x-small;"><span class="pre">BeautifulSoup(markup,</span>&nbsp;<span class="pre">"html.parser")</span></span></code></td><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: solid; border-left-color: initial; border-left-style: initial; border-right-color: initial; border-right-style: initial; border-top-color: initial; border-top-style: initial; border-width: 0px 0px 1px; padding: 1px 8px 1px 5px;"><ul class="first last simple" style="margin-top: 0px !important;">
<li style="line-height: 20.8px; text-align: justify;"><span style="font-size: x-small;">Batteries included</span></li>
<li style="line-height: 20.8px; text-align: justify;"><span style="font-size: x-small;">Decent speed</span></li>
<li style="line-height: 20.8px; text-align: justify;"><span style="font-size: x-small;">Lenient (as of Python 2.7.3 and 3.2.)</span></li>
</ul>
</td><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: solid; border-left-color: initial; border-left-style: initial; border-right-color: initial; border-right-style: initial; border-top-color: initial; border-top-style: initial; border-width: 0px 0px 1px; padding: 1px 8px 1px 5px;"><ul class="first last simple" style="margin-top: 0px !important;">
<li style="line-height: 20.8px; text-align: justify;"><span style="font-size: x-small;">Not very lenient (before Python 2.7.3 or 3.2.2)</span></li>
</ul>
</td></tr>
<tr class="row-odd"><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: solid; border-left-color: initial; border-left-style: initial; border-right-color: initial; border-right-style: initial; border-top-color: initial; border-top-style: initial; border-width: 0px 0px 1px; padding: 1px 8px 1px 5px;"><span style="font-size: x-small;">lxml’s HTML parser</span></td><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: solid; border-left-color: initial; border-left-style: initial; border-right-color: initial; border-right-style: initial; border-top-color: initial; border-top-style: initial; border-width: 0px 0px 1px; padding: 1px 8px 1px 5px;"><code class="docutils literal" style="background-color: #ecf0f3; padding: 0px 1px;"><span style="font-size: x-small;"><span class="pre">BeautifulSoup(markup,</span>&nbsp;<span class="pre">"lxml")</span></span></code></td><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: solid; border-left-color: initial; border-left-style: initial; border-right-color: initial; border-right-style: initial; border-top-color: initial; border-top-style: initial; border-width: 0px 0px 1px; padding: 1px 8px 1px 5px;"><ul class="first last simple" style="margin-top: 0px !important;">
<li style="line-height: 20.8px; text-align: justify;"><span style="font-size: x-small;">Very fast</span></li>
<li style="line-height: 20.8px; text-align: justify;"><span style="font-size: x-small;">Lenient</span></li>
</ul>
</td><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: solid; border-left-color: initial; border-left-style: initial; border-right-color: initial; border-right-style: initial; border-top-color: initial; border-top-style: initial; border-width: 0px 0px 1px; padding: 1px 8px 1px 5px;"><ul class="first last simple" style="margin-top: 0px !important;">
<li style="line-height: 20.8px; text-align: justify;"><span style="font-size: x-small;">External C dependency</span></li>
</ul>
</td></tr>
<tr class="row-even"><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: solid; border-left-color: initial; border-left-style: initial; border-right-color: initial; border-right-style: initial; border-top-color: initial; border-top-style: initial; border-width: 0px 0px 1px; padding: 1px 8px 1px 5px;"><span style="font-size: x-small;">lxml’s XML parser</span></td><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: solid; border-left-color: initial; border-left-style: initial; border-right-color: initial; border-right-style: initial; border-top-color: initial; border-top-style: initial; border-width: 0px 0px 1px; padding: 1px 8px 1px 5px;"><span style="font-size: x-small;"><code class="docutils literal" style="background-color: #ecf0f3; padding: 0px 1px;"><span class="pre">BeautifulSoup(markup,</span>&nbsp;<span class="pre">"lxml-xml")</span></code><code class="docutils literal" style="background-color: #ecf0f3; padding: 0px 1px;"><span class="pre">BeautifulSoup(markup,</span>&nbsp;<span class="pre">"xml")</span></code></span></td><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: solid; border-left-color: initial; border-left-style: initial; border-right-color: initial; border-right-style: initial; border-top-color: initial; border-top-style: initial; border-width: 0px 0px 1px; padding: 1px 8px 1px 5px;"><ul class="first last simple" style="margin-top: 0px !important;">
<li style="line-height: 20.8px; text-align: justify;"><span style="font-size: x-small;">Very fast</span></li>
<li style="line-height: 20.8px; text-align: justify;"><span style="font-size: x-small;">The only currently supported XML parser</span></li>
</ul>
</td><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: solid; border-left-color: initial; border-left-style: initial; border-right-color: initial; border-right-style: initial; border-top-color: initial; border-top-style: initial; border-width: 0px 0px 1px; padding: 1px 8px 1px 5px;"><ul class="first last simple" style="margin-top: 0px !important;">
<li style="line-height: 20.8px; text-align: justify;"><span style="font-size: x-small;">External C dependency</span></li>
</ul>
</td></tr>
<tr class="row-odd"><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: solid; border-left-color: initial; border-left-style: initial; border-right-color: initial; border-right-style: initial; border-top-color: initial; border-top-style: initial; border-width: 0px 0px 1px; padding: 1px 8px 1px 5px;"><span style="font-size: x-small;">html5lib</span></td><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: solid; border-left-color: initial; border-left-style: initial; border-right-color: initial; border-right-style: initial; border-top-color: initial; border-top-style: initial; border-width: 0px 0px 1px; padding: 1px 8px 1px 5px;"><code class="docutils literal" style="background-color: #ecf0f3; padding: 0px 1px;"><span style="font-size: x-small;"><span class="pre">BeautifulSoup(markup,</span>&nbsp;<span class="pre">"html5lib")</span></span></code></td><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: solid; border-left-color: initial; border-left-style: initial; border-right-color: initial; border-right-style: initial; border-top-color: initial; border-top-style: initial; border-width: 0px 0px 1px; padding: 1px 8px 1px 5px;"><ul class="first last simple" style="margin-top: 0px !important;">
<li style="line-height: 20.8px; text-align: justify;"><span style="font-size: x-small;">Extremely lenient</span></li>
<li style="line-height: 20.8px; text-align: justify;"><span style="font-size: x-small;">Parses pages the same way a web browser does</span></li>
<li style="line-height: 20.8px; text-align: justify;"><span style="font-size: x-small;">Creates valid HTML5</span></li>
</ul>
</td><td style="border-bottom-color: rgb(170, 170, 170); border-bottom-style: solid; border-left-color: initial; border-left-style: initial; border-right-color: initial; border-right-style: initial; border-top-color: initial; border-top-style: initial; border-width: 0px 0px 1px; padding: 1px 8px 1px 5px;"><ul class="first last simple" style="margin-top: 0px !important;">
<li style="line-height: 20.8px; text-align: justify;"><span style="font-size: x-small;">Very slow</span></li>
<li style="line-height: 20.8px; text-align: justify;"><span style="font-size: x-small;">External Python dependency</span></li>
</ul>
</td></tr>
</tbody></table>
</div>

<h3 style="text-align: left;">
Other methods and Usage</h3>
Beautiful soup is a vast library and can do things which are too difficult in just a single line.
Some of the methods for searching tags in HTML are:

<pre style="background-color: rgba(0,0,0,0.1); border-bottom: 1px solid rgb(170, 204, 153); border-image: initial; border-left: none; border-right: none; border-top: 1px solid rgb(170, 204, 153); line-height: 15.6px; overflow-x: auto; overflow-y: hidden; padding: 5px;"><div style="white-space: normal;">
<span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace; font-size: x-small;">#finding by ID</span></div>
<div style="white-space: normal;">
<span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace; font-size: x-small;">soup.find(id='abc')</span></div>
<div style="white-space: normal;">
<span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace; font-size: x-small;">
</span></div>
<div style="white-space: normal;">
<span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace; font-size: x-small;">#finding through a regex</span></div>
<div style="white-space: normal;">
<span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace; font-size: x-small;">#lmit the return to 2 tags</span></div>
<div style="white-space: normal;">
<span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace; font-size: x-small;">soup.find_all(re.compile("^a"),limit=2)</span></div>
<div style="white-space: normal;">
<span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace; font-size: x-small;">
</span></div>
<div style="white-space: normal;">
<span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace; font-size: x-small;">#finding multiple tags</span></div>
<div style="white-space: normal;">
<span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace; font-size: x-small;">soup.find_all(['a','h1'])</span></div>
<div style="white-space: normal;">
<span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace; font-size: x-small;">
</span></div>
<div style="white-space: normal;">
<span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace; font-size: x-small;">#fiind by custom or built in attributes</span></div>
<div style="white-space: normal;">
<span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace; font-size: x-small;">soup.find_all(attrs={'data':'abc'})</span></div>
</pre>

<h3 style="text-align: left;">
Problem 2:</h3>
<b>In the above example, we are using HTML string for parsing, now we will see how we can hit a URL and get the HTML for that page and then we can parse it in the same manner as we were doing for HTML string above</b>

For this will be using urllib3 package of python. It can be easily installed by the following command

<pre style="background-color: rgba(0,0,0,0.1); border-bottom: 1px solid rgb(170, 204, 153); border-image: initial; border-left: none; border-right: none; border-top: 1px solid rgb(170, 204, 153); line-height: 15.6px; overflow-x: auto; overflow-y: hidden; padding: 5px;">pip install urllib3</pre>

Documentation for urllib3 can be seen <a href="https://urllib3.readthedocs.io/en/latest/user-guide.html" target="_blank">here</a>.

<pre style="background-color: rgba(0,0,0,0.1); border-bottom: 1px solid rgb(170, 204, 153); border-image: initial; border-left: none; border-right: none; border-top: 1px solid rgb(170, 204, 153); line-height: 15.6px; overflow-x: auto; overflow-y: hidden; padding: 5px;">import urllib3
http = urllib3.PoolManager()
#hiitng the url 
r = http.request('GET', 'https://en.wikipedia.org/wiki/India')

#creating a soup object using html from the link
soup=BeautifulSoup(r.data,"html.parser")

#getting whole text from the wiki page
text=soup.text

#getting all the links from wiki page
links=soup.find_all('a')

#iterating over the new pages and getting text from them
#this can be done in a recursive fashion to parse large number of pages
for link in links:
    prihref=nt link.get('href')
    new_url='https://en.wikipedia.org'+href
    http = urllib3.PoolManager()
    r_new = http.request('GET', new_url)
    #do something with new page
    new_text=r_new.text


#getting source of all the images
src=soup.find('img').get('src')

</pre>

This was just a basic introduction to web scraping using Python. Much more can be achieved using the packages used in this tutorial. This article can serve as a starting point.

<h2 style="text-align: left;">
Points to Remember</h2>
<div>
Web Scraping is very useful in gathering data for different purposes like data mining, knowledge creation, data analysis etc but it should be done with care.</div>
<div>
As a basic rule of thumb, we should not scrape anything which is paid content. Being said that we should comply with the robots.txt file of the site to know the areas which can be crawled.</div>
<div>

</div>
<div>
<b>It is very important to look into the legal implications&nbsp;before scraping.</b>
<b>
</b> <b>
</b> Hope the article was informative.
--
TechScouter (JSC)
<b>
</b></div>
<div>

</div>
</div>
</div>