The Beginners Guide to Web Extraction and XPATH
Tags: technology, data, information
Published: 2018-10-23 21:06:13

---
title: The Beginners Guide to Web Extraction and XPATH
published: true
description: How to extract Data in easier methods
tags: technology, data, information
---
![computer data](https://thepracticaldev.s3.amazonaws.com/i/ik0uktoen6pckg8r4dyu.jpeg)
<p><strong>The Beginners Guide to Web Extraction and XPATH</strong></p>
<p><span style="font-weight: 400;">XPATH is a query language used for selecting nodes and computing values of an HTML or XML document. For some of you reading this, A LOT of information was just thrown at you. You're probably here because you've fallen down the rabbit hole of web extraction, or data scraping as it's widely called, and at this point you're just clicking around. Stop. You've arrived at the beginner's guide for all things web extraction and XPATH. Lets branch out and discuss what web extraction is and the two elements needed to extract the most data. </span></p>
<p>&nbsp;</p>
<p><strong>Data Scraping</strong></p>
<p><span style="font-weight: 400;">Web extraction, also known as </span><a href="https://www.webscraper.io/"><span style="font-weight: 400;">web scraping</span></a><span style="font-weight: 400;"> and web harvesting, is data scraping used for extracting data from websites. Scraping can be done both manually and through automated processes carried out by web crawlers and bots. The data is gathered from across the internet and copied into a central local database or spreadsheet. Google and Amazon AWS provide web scraping tools and public data free of charge. While that may sound somewhat invasive, there are many everyday uses such as:</span></p>
<ul>
<li style="font-weight: 400;"><span style="font-weight: 400;">online price change monitoring</span></li>
<li style="font-weight: 400;"><span style="font-weight: 400;">price comparison</span></li>
<li style="font-weight: 400;"><span style="font-weight: 400;">real estate listings</span></li>
<li style="font-weight: 400;"><span style="font-weight: 400;">weather data monitoring</span></li>
<li style="font-weight: 400;"><span style="font-weight: 400;">product review</span></li>
</ul>
<p>&nbsp;</p>
<p><span style="font-weight: 400;">Still, websites are finding new methods to prevent </span><a href="https://www.sciencedaily.com/terms/web_crawler.htm"><span style="font-weight: 400;">web crawlers</span></a><span style="font-weight: 400;"> and bots from extracting data. Data scraping systems have also evolved, but there are times when a human touch is needed and required. This requires the user to be able to both read and write in the language of the web. </span></p>
<p>&nbsp;</p>
<p><strong>The Language of the Web</strong></p>
<p><span style="font-weight: 400;">Web pages are built using text based markup languages such as </span><a href="https://www.w3schools.com/html/html_xhtml.asp"><span style="font-weight: 400;">HTML AND XHTML</span></a><span style="font-weight: 400;">. When web browsers receive HTML documents, they render them into multimedia web pages. HTML describes the structure of a web page semantically. In text form, the wealth of information to be parsed through offline is high. Understanding the language of the web will give you a foundation to be able to get the most out of web scraping tools, bots and web crawlers. It will also allow you yourself to actually read, write and troubleshoot what your looking for. Just to name a few, here are some other markup languages:</span></p>
<ul>
<li style="font-weight: 400;"><span style="font-weight: 400;">BNML (Business Narrative Markup Language)</span></li>
<li style="font-weight: 400;"><span style="font-weight: 400;">CFML (ColdFusion Markup Language)</span></li>
<li style="font-weight: 400;"><span style="font-weight: 400;">InkML</span></li>
<li style="font-weight: 400;"><span style="font-weight: 400;">VoiceXML</span></li>
<li style="font-weight: 400;"><span style="font-weight: 400;">XBEL (XML Bookmark Exchange Language)</span></li>
</ul>
<p>&nbsp;</p>
<p><strong>Roots, Trees and XPATH</strong></p>
<p><span style="font-weight: 400;">In the introduction of </span><a href="https://www.import.io/post/xpath-tutorial-intro-to-xpath/"><span style="font-weight: 400;">Using Xpath: An Xpath Tutorial</span></a><span style="font-weight: 400;">, it's established that XPATH is another language of the web. XPATH is based on a tree representation of the XML document. Based on the hierarchical structure of an XML document, they can be interpreted similar to a tree structure. An XML tree must have a root element. A root element is parent to all other elements. Each element can have sub elements. When dealing with XPATH, elements are referred to as nodes. XPATH allows the user to navigate around the tree and select specific nodes. XPATH's ability to both select and compute is essential as some sub nodes may contain values, conditions, text or even another data structure. </span></p>
<p><span style="font-weight: 400;">If you plan on working with the web, learning the language is an essential step. As website developers find new ways to block bots, knowing the language will allow YOU to find a work around. Knowing and understanding what you're looking for is crucial when data scraping. With XPATH, when you find nodes that contain values, you can compute them with no time being lost. Building your foundation in the language of the web will lead to a deeper understanding of the tree structure. A deeper understanding of the tree structure will have you scraping heaps of data in no time.</span></p>