Introduction to Web Scraping with Embedded Python - Let’s Extract python job’s
Tags: tutorial, python, coding, beginners
Published: 2023-06-27 17:24:37

## **What is Web Scraping:**

In simple terms, **Web scraping**, **web harvesting**, or **web data extraction** is an automated process of collecting large data(unstructured) from websites. The user can extract all the data on particular sites or the specific data as per the requirement. The data collected can be stored in a structured format for further analysis.

##<img alt="What is Web Scraping? — James Le" data-noaft="1" jsaction="load:XAeZkd;" jsname="HiaYvf" src="https://images.squarespace-cdn.com/content/v1/59d9b2749f8dce3ebe4e676d/1566319595811-MFGFIVQGNNSY2W4TZ4AV/web-scraping.png?format=1500w" />  
  
**Steps involved in web scraping:**

  1. Find the URL of the webpage that you want to scrape
  2. Select the particular elements by inspecting
  3. Write the code to get the content of the selected elements
  4. Store the data in the required format

It’s that simple !!

## **The popular libraries/tools used for web scraping are:**

  * Selenium – a framework for testing web applications
  * BeautifulSoup – Python library for getting data out of HTML, XML, and other markup languages
  * Pandas – Python library for data manipulation and analysis

##  
What is Beautiful Soup? {#what-is-beautiful-soup?}

Beautiful Soup is a pure Python library for extracting structured data from a website. It allows you to parse data from HTML and XML files. It acts as a helper module and interacts with HTML in a similar and better way as to how you would interact with a web page using other available developer tools.

  * It usually saves programmers hours or days of work since it works with your favorite parsers like `lxml` and `html5lib` to provide organic Python ways of navigating, searching, and modifying the parse tree.
  * Another powerful and useful feature of beautiful soup is its intelligence to convert the documents being fetched to Unicode and outgoing documents to UTF-8. As a developer, you do not have to take care of that unless the document intrinsic doesn't specify an encoding or Beautiful Soup is unable to detect one.
  * It is also considered to be **faster** when compared to other general parsing or scraping techniques.

###  

## **In today's article we will be using Embedded Python with Object Script to scrape python vacancies and companies on  ae.indeed.com**

###  
**Step 1 - **Find the URL of the webpage that you want to scrape.  
 

Url = https://ae.indeed.com/jobs?q=python&l=Dubai&start=0

The webpage that we are gonna scrape data from looks like this  

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ktv5zsg5r0lfqr68h03q.png)

 

**_for simplicity and learning purposes we will be extracting "Job Title" and "Company", the output would be something similar to below screenshot._**


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/e61ckkake83vcdcv0i0l.gif) <span> </span>

 <span> </span>  
  
**We will be using two python libraries.**

  * **requests** Requests is a HTTP library for the Python programming language. The goal of the project is to make HTTP requests simpler and more human-friendly. <span> </span>
  * **bs4 for BeautifulSoup** Beautiful Soup is a Python package for parsing HTML and XML documents. It creates a parse tree for parsed pages that can be used to extract data from HTML, which is useful for web scraping.

**Lets install this python packages (windows)**

<pre class="codeblock-container" idlang="0" lang="ObjectScript" tabsize="4"><code class="language-cls hljs cos">irispip install --target C:\InterSystems\IRISHealth\mgr\python bs4

irispip install --target C:\InterSystems\IRISHealth\mgr\python requests</code></pre>


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rimejjkt5z862ka5jnoz.png) 

**Lets import the python libraries to ObjectScript**  
 

<pre class="codeblock-container" idlang="0" lang="ObjectScript" tabsize="4"><code class="language-cls hljs cos"><span class="hljs-keyword">Class</span> PythonTesting.WebScraper <span class="hljs-keyword">Extends</span> <span class="hljs-built_in">%Persistent</span>
{

<span class="hljs-comment">// pUrl = https://ae.indeed.com/jobs?q=python&l=Dubai&start=</span>
<span class="hljs-comment">// pPage = 0</span>
<span class="hljs-keyword">ClassMethod</span> ScrapeWebPage(pUrl, pPage)
{
    <span class="hljs-comment">// imports the requests python library</span>
	<span class="hljs-keyword">set</span> requests = <span class="hljs-keyword">##class</span>(<span class="hljs-built_in">%SYS.Python</span>).Import(<span class="hljs-string">"requests"</span>)
    <span class="hljs-comment">// import the bs4 python library</span>
	<span class="hljs-keyword">set</span> soup = <span class="hljs-keyword">##class</span>(<span class="hljs-built_in">%SYS.Python</span>).Import(<span class="hljs-string">"bs4"</span>)
	<span class="hljs-comment">// import builtins package which contains all of the built-in identifiers</span>
	<span class="hljs-keyword">set</span> builtins = <span class="hljs-keyword">##class</span>(<span class="hljs-built_in">%SYS.Python</span>).Import(<span class="hljs-string">"builtins"</span>)
}</code></pre>

**Lets collect the html data using requests;**  
  
_Note: The user agent us taken from googling "my user agent"_  
_The url is "https://ae.indeed.com/jobs?q=python&l=Dubai&start=", pPage is the page number_

We will do a http get request to the URL using requests and store the response on "req"

<pre class="codeblock-container" idlang="0" lang="ObjectScript" tabsize="4"><code class="language-cls hljs cos">    <span class="hljs-keyword">set</span> headers  = {<span class="hljs-string">"User-Agent"</span>: <span class="hljs-string">"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"</span>}
    <span class="hljs-keyword">set</span> url = <span class="hljs-string">"https://ae.indeed.com/jobs?q=python&l=Dubai&start="</span>_pPage
    
    <span class="hljs-keyword">set</span> req = requests.get(url,<span class="hljs-string">"headers="</span>_headers)
</code></pre>

The req object will have the html which was returned from the webpage.

**Let's run this through the BeautifulSoup html parser, so that we can extract the job data.**

<pre class="codeblock-container" idlang="0" lang="ObjectScript" tabsize="4"><code class="language-cls hljs cos"><span class="hljs-keyword">set</span> soupData = soup.BeautifulSoup(req.content, <span class="hljs-string">"html.parser"</span>)
<span class="hljs-keyword">set</span> title = soupData.title.text
<span class="hljs-keyword">W</span> !,title</code></pre>

**The title looks as follows**  

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3se0512had0zdfkoo5v5.png) 

###  

###  
**Step 2 :Select the required elements by inspecting.**

**In this scenario we are interested the list of jobs which usually sits in a 

<div>
  tag, in your browser you can inspect the element to find the div class.&lt;/strong><br /><img src="/sites/default/files/inline/images/images/image(4279).png" /><br /><br />In our case the required information is stored under &lt;div class="cardOutline tapItem <span> </span>...
</div>&lt;/p> 

###  

###  
Step 3: Write the code to get the content of the selected elements

**We will be using the find_all functionality on BeautifulSoup to look for all the 

<div>
  tags which contains the class name "cardOutline"&lt;/strong>&lt;/p> 
  
  <pre class="codeblock-container" idlang="0" lang="ObjectScript" tabsize="4"><code class="language-cls hljs cos"><span class="hljs-comment">//parameters to python would be sent as a python dictionary</span>
<span class="hljs-keyword">set</span> divClass = {<span class="hljs-string">"class"</span>:<span class="hljs-string">"cardOutline"</span>}
<span class="hljs-keyword">set</span> divsArr = soupData.<span class="hljs-string">"find_all"</span>(<span class="hljs-string">"div"</span>,divClass...)</code></pre>
  
  <p>
    This will return a list, which we can loop through and extract the Job Titles and Company
  </p>
  
  <h3>
    <br /><br />Step 4: Store/Display the data in the required format.
  </h3>
  
  <p>
    In the following example we will be writing the data to the terminal.
  </p>
  
  <pre class="codeblock-container" idlang="0" lang="ObjectScript" tabsize="4"><code class="language-cls hljs cos"><span class="hljs-keyword">set</span> len = builtins.len(divsArr)
	
<span class="hljs-keyword">W</span> !, <span class="hljs-string">"Job Title"</span>,<span class="hljs-built_in">$C</span>(<span class="hljs-number">9</span>)_<span class="hljs-string">" --- "</span>_<span class="hljs-built_in">$C</span>(<span class="hljs-number">9</span>),<span class="hljs-string">"Company"</span>
<span class="hljs-keyword">for</span> i = <span class="hljs-number">1</span>:<span class="hljs-number">1</span>:len {
    <span class="hljs-keyword">Set</span> item = divsArr.<span class="hljs-string">"__getitem__"</span>(i - <span class="hljs-number">1</span>)
    <span class="hljs-keyword">set</span> title = <span class="hljs-built_in">$ZSTRIP</span>(item.find(<span class="hljs-string">"a"</span>).text,<span class="hljs-string">"<>W"</span>)
    <span class="hljs-keyword">set</span> companyClass = {<span class="hljs-string">"class_"</span>:<span class="hljs-string">"companyName"</span>}
    <span class="hljs-keyword">set</span> company = <span class="hljs-built_in">$ZSTRIP</span>(item.find(<span class="hljs-string">"span"</span>, companyClass...).text,<span class="hljs-string">"<>W"</span>)
    <span class="hljs-keyword">W</span> !,title,<span class="hljs-built_in">$C</span>(<span class="hljs-number">9</span>),<span class="hljs-string">" --- "</span>,<span class="hljs-built_in">$C</span>(<span class="hljs-number">9</span>),company
}</code></pre>
  
  Note that we are using the builtins.len() to get the length of the divsArr list
 
     Identifier Names:
The rules for naming identifiers are different between ObjectScript and Python. For example, the underscore (_) is allowed in Python method names, and in fact is widely used for the so-called “dunder” methods and attributes (“dunder” is short for “double underscore”), such as __getitem__ or __class__. To use such identifiers from ObjectScript, enclose them in double quotes:

[Intersystems Documentation on Identifier Names](https://docs.intersystems.com/irisforhealthlatest/csp/docbook/DocBook.UI.Page.cls?KEY=AEPYTHON#AEPYTHON_sharedata_identifiers)

<h2>  
Example Class Method.
</h2>

ClassMethod ScrapeWebPage(pUrl, pPage)
    
```
    // pUrl = https://ae.indeed.com/jobs?q=python&l=Dubai&start=
// pPage = 0
ClassMethod ScrapeWebPage(pUrl, pPage)
{
    set requests = ##class(%SYS.Python).Import("requests")

    set soup = ##class(%SYS.Python).Import("bs4")
    
    set builtins = ##class(%SYS.Python).Builtins()

    set headers  = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"}
    set url = pUrl_pPage
    
    set req = requests.get(url,"headers="_headers)
    
    set soupData = soup.BeautifulSoup(req.content, "html.parser")
    
    set title = soupData.title.text
    
    W !,title

    set divClass = {"class_":"cardOutline"}
    set divsArr = soupData."find_all"("div",divClass...)
    
    set len = builtins.len(divsArr)
    
    W !, "Job Title",$C(9)_" --- "_$C(9),"Company"
    for i = 1:1:len {
            Set item = divsArr."__getitem__"(i - 1)
            set title = $ZSTRIP(item.find("a").text,"<>W")
            set companyClass = {"class_":"companyName"}
            set company = $ZSTRIP(item.find("span", companyClass...).text,"<>W")
            W !,title,$C(9)," --- ",$C(9),company
     }
}
```
  
  <p>
     
  </p>
  
  <h2>
    Next Steps..
  </h2>
  
  <p>
    Using Object Script and Embedded python and with few lines of code; we could easily scrape data form our favourite job web sites, collect the job name, company, salary, job description and emails/links.<br />for example if you have multiple pages you can traverse through them easily using the page<br />This data can be added to a pandas dataframe and remove duplicates, filters can be applied based on specific keywords that you are interested in.<br />Run this data through numpy, and get some lineplot's<br />Or perform One-Hot encoding on the data, and create/train your ML models and if there are specific vacancies that you are interested in, send a notification to yourself. 😉<br /><br />Happy Coding !!!
  </p>
  
  <p>
    and don’t forget to hit the like button 😃
  </p></body></html>