Web Scraping With Java
Tags: java, webscraping, javascript, beginners
Published: 2022-11-29 13:46:50

<p class="c12"><span style="overflow: hidden; display: inline-block; margin: 0.00px 0.00px; border: 0.00px solid #000000; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px); width: 624.00px; height: 312.00px;"><img alt="" src="images/image3.jpg" style="width: 624.00px; height: 312.00px; margin-left: 0.00px; margin-top: 0.00px; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px);" title=""></span></p><p class="c12 c17"><span class="c20 c14 c26"></span></p><p class="c12"><span>Some of the popular languages used for web scraping are Python, JavaScript with Node.js, PHP, Java, C#, and many others. Every language has its strengths and weaknesses. For now, let&rsquo;s focus on web scraping with Java.</span></p><h2 class="c33" id="h.ftaxcsc0a8v6"><span>Web scraping frameworks</span></h2><p class="c12"><span>There are two most commonly used libraries for web scraping with Java&mdash; </span><span class="c10"><a class="c37" href="https://www.google.com/url?q=https://jsoup.org/&amp;sa=D&amp;source=editors&amp;ust=1669732597874486&amp;usg=AOvVaw1lgN0aLvpAMEYa5xr0QGQn">JSoup</a></span><span>&nbsp;and </span><span class="c10"><a class="c37" href="https://www.google.com/url?q=https://htmlunit.sourceforge.io/&amp;sa=D&amp;source=editors&amp;ust=1669732597874757&amp;usg=AOvVaw0Pv501ZbYpBEp7hoSTqQdd">HtmlUnit</a></span><span class="c23 c14 c26">.</span></p><p class="c12"><span class="c23 c14 c26">&nbsp;</span></p><p class="c12"><span class="c23 c14 c26">JSoup is a powerful library that can effectively handle malformed HTML. The name of this library comes from the phrase &ldquo;tag soup&rdquo;, which refers to the malformed HTML document.</span></p><p class="c12 c17"><span class="c23 c14 c26"></span></p><p class="c12"><span class="c23 c14 c26">HtmlUnit is a GUI-less, or headless, browser for Java programs. It can emulate the key aspects of a browser, such as getting specific elements from a page, clicking the elements, etc. As the name of this library suggests, it is commonly used for unit testing. It is a way to simulate a browser for testing purposes.</span></p><p class="c12 c17"><span class="c23 c14 c26"></span></p><p class="c12"><span>HtmlUnit can also be used for web scraping. The good thing is that with just one line, JavaScript and CSS can be turned off. It is helpful in web scraping as JavaScript and CSS are not required most of the time. In the later sections, we will examine both libraries and create web scrapers.</span></p><h2 class="c33" id="h.jvxy4rifu0r1"><span>Prerequisites for building a web scraper with Java</span></h2><p class="c12"><span class="c23 c14 c26">This tutorial assumes that you are familiar with the Java programming language. For managing packages, we will be using Maven.</span></p><p class="c12 c17"><span class="c23 c14 c26"></span></p><p class="c12"><span>Apart from Java basics, a primary understanding of how websites work is also expected. Good knowledge of HTML and selecting elements, either by using </span><span class="c10"><a class="c37" href="https://www.google.com/url?q=https://oxylabs.io/blog/xpath-vs-css&amp;sa=D&amp;source=editors&amp;ust=1669732597875852&amp;usg=AOvVaw3xrX_TPKDW6zIc3pFBPUuu">XPath or CSS selectors</a></span><span>, would also be required. Note that not all libraries support XPath. </span></p><h2 class="c33" id="h.e2b6j1yesyru"><span>Quick overview of CSS selectors</span></h2><p class="c12"><span class="c23 c14 c26">Before proceeding, let&rsquo;s review the CSS selectors:</span></p><p class="c12 c17"><span class="c23 c14 c26"></span></p><ul class="c32 lst-kix_7byeucpi9is3-0 start"><li class="c39 c7 li-bullet-0"><span class="c16">#firstname</span><span class="c9">&nbsp;&ndash; selects any element where </span><span class="c16">id</span><span class="c23 c14 c9">&nbsp;equals &nbsp;&ldquo;firstname&rdquo;.</span></li><li class="c39 c7 li-bullet-0"><span class="c16">.blue</span><span class="c9">&nbsp;&ndash; selects any element where the </span><span class="c16">class</span><span class="c23 c14 c9">&nbsp;contains &ldquo;blue&rdquo;.</span></li><li class="c7 c39 li-bullet-0"><span class="c16">p</span><span class="c9">&nbsp;&ndash; selects all </span><span class="c16">&lt;p&gt;</span><span class="c23 c14 c9">&nbsp;tags.</span></li><li class="c39 c7 li-bullet-0"><span class="c16">div#firstname</span><span class="c9">&nbsp;&ndash; selects </span><span class="c16">div</span><span class="c9">&nbsp;elements where </span><span class="c16">id</span><span class="c23 c14 c9">&nbsp;equals &nbsp;&ldquo;firstname&rdquo;.</span></li><li class="c39 c7 li-bullet-0"><span class="c16">p.link.new</span><span class="c9">&nbsp;&ndash; note that there is no space here. It selects </span><span class="c16">&lt;p class=&quot;link new&quot;&gt;</span><span class="c23 c14 c26">.</span></li><li class="c39 c7 li-bullet-0"><span class="c16">p.link .new</span><span class="c9">&nbsp;&ndash; note the space. Selects any element with class </span><span class="c16">&ldquo;new&rdquo;</span><span class="c9">, which is inside </span><span class="c16">&lt;p class=&quot;link&quot;&gt;</span><span class="c23 c14 c26">.</span></li></ul><p class="c12"><span>Now let&rsquo;s review the libraries that can be used for web scraping with Java.</span></p><h2 class="c33" id="h.gi5rmcuuc2w6"><span>Web scraping with Java using JSoup</span></h2><p class="c12"><span>JSoup is perhaps the most commonly used Java library for web scraping. Broadly speaking, there are three steps involved in web scraping using Java.</span></p><h3 class="c28" id="h.oxmc5l2k2ovq"><span class="c14 c38">Getting JSoup</span></h3><p class="c12"><span>The first step is to get the Java libraries. Maven can help here. Use any Java IDE and create a Maven project. If you do not want to use Maven, head over to </span><span class="c10"><a class="c37" href="https://www.google.com/url?q=https://jsoup.org/download&amp;sa=D&amp;source=editors&amp;ust=1669732597878737&amp;usg=AOvVaw1QfrcB_PMhQVAmSI6VSbhz">this page</a></span><span class="c23 c14 c26">&nbsp;to find alternate downloads.</span></p><p class="c12 c17"><span class="c23 c14 c26"></span></p><p class="c27 c7"><span class="c9">In the </span><span class="c16">pom.xml</span><span class="c9">&nbsp;(Project Object Model) file, add a new section for dependencies and add a dependency for JSoup. The </span><span class="c16">pom.xml</span><span class="c9">&nbsp;file would look something like this:</span></p><a id="t.5ab54c00405681635d5135594265d36a8d1e0a84"></a><a id="t.0"></a><table class="c29"><tr class="c22"><td class="c15" colspan="1" rowspan="1"><p class="c12"><span class="c4">&lt;</span><span class="c5">dependencies</span><span class="c4">&gt;</span></p><p class="c12"><span class="c5">&nbsp; &nbsp; </span><span class="c4">&lt;</span><span class="c5">dependency</span><span class="c4">&gt;</span></p><p class="c12"><span class="c5">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c4">&lt;</span><span class="c5">groupId</span><span class="c4">&gt;</span><span class="c5">org</span><span class="c5 c8">.</span><span class="c5">jsoup</span><span class="c4">&lt;/</span><span class="c5">groupId</span><span class="c4">&gt;</span></p><p class="c12"><span class="c5">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c4">&lt;</span><span class="c5">artifactId</span><span class="c4">&gt;</span><span class="c5">jsoup</span><span class="c4">&lt;/</span><span class="c5">artifactId</span><span class="c4">&gt;</span></p><p class="c12"><span class="c5">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c4">&lt;</span><span class="c5">version</span><span class="c4">&gt;</span><span class="c5 c24">1.14.1</span><span class="c4">&lt;/</span><span class="c5">version</span><span class="c4">&gt;</span></p><p class="c12"><span class="c5">&nbsp; &nbsp; </span><span class="c4">&lt;/</span><span class="c5">dependency</span><span class="c4">&gt;</span></p><p class="c13"><span class="c4">&lt;/</span><span class="c5">dependencies</span><span class="c4">&gt;</span></p></td></tr></table><p class="c13 c17"><span class="c21 c14 c30 c7"></span></p><p class="c12"><span>With this, we are ready to create a Java scraper.</span></p><h3 class="c28" id="h.mci5k3kvr1gh"><span>Getting and parsing the HTML</span></h3><p class="c12"><span>The second step is to get the HTML from the target URL and </span><span class="c10"><a class="c37" href="https://www.google.com/url?q=https://oxylabs.io/blog/what-is-data-parser&amp;sa=D&amp;source=editors&amp;ust=1669732597882682&amp;usg=AOvVaw2qjVSTwjwNbJPlKQz6VrAk">parse</a></span><span class="c23 c14 c26">&nbsp;it into a Java object. Let&rsquo;s begin with the imports:</span></p><p class="c13 c17"><span class="c5 c14 c8"></span></p><a id="t.ffac12ceece6118c00bec086c2677e982f9e53b7"></a><a id="t.1"></a><table class="c29"><tr class="c22"><td class="c15" colspan="1" rowspan="1"><p class="c12"><span class="c5 c31">import</span><span class="c5">&nbsp;org</span><span class="c5 c8">.</span><span class="c5">jsoup</span><span class="c5 c8">.</span><span class="c5">Connection</span><span class="c5 c8">;</span></p><p class="c12"><span class="c5 c31">import</span><span class="c5">&nbsp;org</span><span class="c5 c8">.</span><span class="c5">jsoup</span><span class="c5 c8">.</span><span class="c5">Jsoup</span><span class="c5 c8">;</span></p><p class="c12"><span class="c5 c31">import</span><span class="c5">&nbsp;org</span><span class="c5 c8">.</span><span class="c5">jsoup</span><span class="c5 c8">.</span><span class="c5">nodes</span><span class="c5 c8">.</span><span class="c5">Document</span><span class="c5 c8">;</span></p><p class="c12"><span class="c5 c31">import</span><span class="c5">&nbsp;org</span><span class="c5 c8">.</span><span class="c5">jsoup</span><span class="c5 c8">.</span><span class="c5">nodes</span><span class="c5 c8">.</span><span class="c5">Element</span><span class="c5 c8">;</span></p><p class="c13"><span class="c5 c31">import</span><span class="c5">&nbsp;org</span><span class="c5 c8">.</span><span class="c5">jsoup</span><span class="c5 c8">.</span><span class="c5">select</span><span class="c5 c8">.</span><span class="c5">Elements</span><span class="c5 c14 c8">;</span></p></td></tr></table><p class="c13 c17"><span class="c5 c14 c8"></span></p><p class="c27 c7"><span class="c9">Note that it is not a good practice to import everything with a wildcard &ndash; </span><span class="c16">import org.jsoup.*</span><span>.</span><span>&nbsp;</span><span class="c23 c14 c9">Always import exactly what you need. The above imports are what we are going to use in this tutorial.</span></p><p class="c27 c7"><span class="c9">JSoup provides the </span><span class="c16">connect</span><span class="c9">&nbsp;function. This function takes the URL and returns a </span><span class="c16">Document</span><span class="c9">. Here is how you can get the page&rsquo;s HTML:</span></p><a id="t.5e58bf788791474c93d7b1301be8f6eaac5805d4"></a><a id="t.2"></a><table class="c29"><tr class="c22"><td class="c15" colspan="1" rowspan="1"><p class="c13"><span class="c5">Document doc </span><span class="c4">=</span><span class="c5">&nbsp;Jsoup</span><span class="c5 c8">.</span><span class="c5 c16">connect</span><span class="c5 c8">(</span><span class="c5 c19">&quot;https://en.wikipedia.org/wiki/Jsoup&quot;</span><span class="c5 c8">).</span><span class="c5 c16">get</span><span class="c5 c8">();</span></p></td></tr></table><p class="c13 c17"><span class="c21 c14 c8 c7"></span></p><p class="c27 c7"><span class="c9">You will often see this line in places, but it has a disadvantage. This shortcut does not have any error handling. A better approach would be to create a function. This function takes a URL as a parameter. First, it creates a connection and then stores it in a variable. After that, the </span><span class="c16">get()</span><span class="c9">&nbsp;method of the connection object is called to retrieve the HTML document. This document is returned as an instance of the </span><span class="c16">Document</span><span class="c9">&nbsp;class. The </span><span class="c16">get()</span><span class="c9">&nbsp;method can throw an </span><span class="c16">IOException</span><span class="c9">, which needs to be handled.</span></p><a id="t.71d3d372c98845a9ccc8a708d28ce78cbeb476b9"></a><a id="t.3"></a><table class="c29"><tr class="c22"><td class="c15" colspan="1" rowspan="1"><p class="c12"><span class="c0">public</span><span class="c5 c7">&nbsp;</span><span class="c0">static</span><span class="c5 c7">&nbsp;Document </span><span class="c5 c6">getDocument</span><span class="c2">(</span><span class="c5 c7">String url</span><span class="c2">)</span><span class="c5 c7">&nbsp;</span><span class="c2">{</span></p><p class="c12"><span class="c5 c7">&nbsp; &nbsp; Connection conn </span><span class="c4 c7">=</span><span class="c5 c7">&nbsp;Jsoup</span><span class="c2">.</span><span class="c5 c6">connect</span><span class="c2">(</span><span class="c5 c7">url</span><span class="c2">);</span></p><p class="c12"><span class="c5 c7">&nbsp; &nbsp; Document document </span><span class="c4 c7">=</span><span class="c5 c7">&nbsp;</span><span class="c0">null</span><span class="c2">;</span></p><p class="c12"><span class="c5 c7">&nbsp; &nbsp; </span><span class="c0">try</span><span class="c5 c7">&nbsp;</span><span class="c2">{</span></p><p class="c12"><span class="c5 c7">&nbsp; &nbsp; &nbsp; &nbsp; document </span><span class="c4 c7">=</span><span class="c5 c7">&nbsp;conn</span><span class="c2">.</span><span class="c5 c6">get</span><span class="c2">();</span></p><p class="c12"><span class="c5 c7">&nbsp; &nbsp; </span><span class="c2">}</span><span class="c5 c7">&nbsp;</span><span class="c0">catch</span><span class="c5 c7">&nbsp;</span><span class="c2">(</span><span class="c5 c7">IOException e</span><span class="c2">)</span><span class="c5 c7">&nbsp;</span><span class="c2">{</span></p><p class="c12"><span class="c5 c7">&nbsp; &nbsp; &nbsp; &nbsp; e</span><span class="c2">.</span><span class="c5 c6">printStackTrace</span><span class="c2">();</span></p><p class="c12"><span class="c5 c7">&nbsp; &nbsp; </span><span class="c5 c41 c7">// handle error</span></p><p class="c12"><span class="c5 c7">&nbsp; &nbsp; </span><span class="c2">}</span></p><p class="c12"><span class="c5 c7">&nbsp; &nbsp; </span><span class="c0">return</span><span class="c5 c7">&nbsp;document</span><span class="c2">;</span></p><p class="c13"><span class="c2">}</span></p></td></tr></table><p class="c12 c17"><span class="c23 c14 c8"></span></p><p class="c27 c7"><span class="c9">In some instances, you would need to pass a custom user agent. This can be done by sending the user agent string to the </span><span class="c16">userAgent()</span><span class="c9">&nbsp;function before calling the </span><span class="c16">get()</span><span class="c9">&nbsp;function:</span></p><a id="t.f6a7c9c3dbea5d23f190593262d42978390a1db6"></a><a id="t.4"></a><table class="c29"><tr class="c22"><td class="c15" colspan="1" rowspan="1"><p class="c12"><span class="c5 c7">Connection conn </span><span class="c4 c7">=</span><span class="c5 c7">&nbsp;Jsoup</span><span class="c2">.</span><span class="c5 c6">connect</span><span class="c2">(</span><span class="c5 c7">url</span><span class="c2">);</span></p><p class="c12"><span class="c5 c7">conn</span><span class="c2">.</span><span class="c5 c6">userAgent</span><span class="c2">(</span><span class="c5 c19 c7">&quot;custom user agent&quot;</span><span class="c2">);</span></p><p class="c13"><span class="c5 c7">document </span><span class="c4 c7">=</span><span class="c5 c7">&nbsp;conn</span><span class="c2">.</span><span class="c5 c6">get</span><span class="c2 c14">();</span></p></td></tr></table><p class="c13 c17"><span class="c2 c14"></span></p><p class="c12"><span>This action should resolve all the common problems.</span></p><h3 class="c28" id="h.56apbl3588kf"><span>Querying HTML</span></h3><p class="c27 c7"><span class="c9">The most crucial step of any Java web scraper building process is to query the HTML </span><span class="c16">Document</span><span class="c23 c14 c9">&nbsp;object for the desired data. This is the point where you will be spending most of your time.</span></p><p class="c27 c7"><span class="c9">JSoup supports many ways to extract the desired elements. There are many methods, such as </span><span class="c16">getElementByID</span><span class="c9">, </span><span class="c16">getElementsByTag</span><span class="c23 c14 c9">, etc., that make it easier to query the DOM.</span></p><p class="c12 c7"><span class="c9">Here is an example of navigating to the JSoup page on Wikipedia. Right-click the heading and select </span><span class="c9 c25">Inspect</span><span class="c23 c14 c9">, thus opening the developer tools with the heading selected.</span></p>
![web scraping with java example](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bm0b3nh51dpcy3aerrqw.jpg)<p class="c12 c17 c7"><span class="c23 c14 c9"></span></p><p class="c12 c7"><span style="overflow: hidden; display: inline-block; margin: 0.00px 0.00px; border: 0.00px solid #000000; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px); width: 624.00px; height: 317.33px;"><img alt="" src="images/image1.jpg" style="width: 624.00px; height: 317.33px; margin-left: 0.00px; margin-top: 0.00px; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px);" title=""></span></p><p class="c35"><span>Image: HTML Element with a unique class</span></p><p class="c12 c17 c7"><span class="c23 c14 c9"></span></p><p class="c7 c27"><span class="c9">In this case, either </span><span class="c16">getElementByID</span><span class="c9">&nbsp;or </span><span class="c16">getElementsByClass</span><span class="c9">&nbsp;can be used. One important point to note here is that </span><span class="c16">getElementById</span><span class="c9">&nbsp;(note the singular </span><span class="c16">Element</span><span class="c9">) returns one </span><span class="c16">Element</span><span class="c9">&nbsp;object, whereas </span><span class="c16">getElementsByClass</span><span class="c9">&nbsp;(note plural </span><span class="c16">Elements</span><span class="c9">) returns an </span><span class="c16">Array</span><span class="c9">&nbsp;list of </span><span class="c16">Element</span><span class="c23 c14 c9">&nbsp;objects.</span></p><p class="c27 c7"><span class="c9">Conveniently, this library has a class </span><span class="c16">Elements</span><span class="c9">&nbsp;that extends </span><span class="c16">ArrayList&lt;Element&gt;</span><span class="c23 c14 c9">. This makes code cleaner and provides more functionality.</span></p><p class="c27 c7"><span class="c9">In the code example below, the </span><span class="c16">first()</span><span class="c9">&nbsp;method can be used to get the first element from the </span><span class="c16">ArrayList</span><span class="c9">. After getting the reference of the element, the </span><span class="c16">text()</span><span class="c9">&nbsp;method can be called to get the text.</span></p><a id="t.89fa8080c7ed6534d05e0351e93ada95c60f1367"></a><a id="t.5"></a><table class="c29"><tr class="c22"><td class="c15" colspan="1" rowspan="1"><p class="c13 c7"><span class="c3">Element firstHeading </span><span class="c4 c7">=</span><span class="c3">&nbsp;document</span><span class="c2">.</span><span class="c5 c6">getElementsByClass</span><span class="c2">(</span><span class="c5 c19 c7">&quot;firstHeading&quot;</span><span class="c2">).</span><span class="c5 c6">first</span><span class="c2">();</span></p><p class="c13 c7"><span class="c3">System</span><span class="c2">.</span><span class="c3">out</span><span class="c2">.</span><span class="c5 c6">println</span><span class="c2">(</span><span class="c3">firstHeading</span><span class="c2">.</span><span class="c5 c6">text</span><span class="c2">());</span></p></td></tr></table><p class="c13 c17 c7"><span class="c20 c14 c8 c7"></span></p><p class="c27 c7"><span class="c9">These functions are good; however, they are specific to JSoup. For most cases, the </span><span class="c16">select</span><span class="c9">&nbsp;function can be a better choice. The only case when select functions will not work is when you need to traverse up the document. In these cases, you may want to use </span><span class="c16">parent()</span><span class="c9">, </span><span class="c16">children()</span><span class="c9">, and </span><span class="c16">child()</span><span class="c9">. For a complete list of all the available methods, </span><span class="c34"><a class="c37" href="https://www.google.com/url?q=https://jsoup.org/cookbook/extracting-data/dom-navigation&amp;sa=D&amp;source=editors&amp;ust=1669732597896874&amp;usg=AOvVaw2mJf3xsq_gw7UhmrVNoMxz">visit this page</a></span><span class="c23 c14 c9">.</span></p><p class="c27 c7"><span class="c9">The following code demonstrates how to use the </span><span class="c16">selectFirst()</span><span class="c9">&nbsp;method, which returns the first match:</span></p><a id="t.ef12d2d86d8d2fef255558f93d3aed5a390c1598"></a><a id="t.6"></a><table class="c29"><tr class="c22"><td class="c15" colspan="1" rowspan="1"><p class="c13 c7"><span class="c3">Element firstHeading</span><span class="c4 c7">=</span><span class="c3">&nbsp;document</span><span class="c2">.</span><span class="c5 c6">selectFirst</span><span class="c2">(</span><span class="c5 c19 c7">&quot;.firstHeading&quot;</span><span class="c2">);</span></p></td></tr></table><p class="c13 c17 c7"><span class="c20 c14 c8 c7"></span></p><p class="c27 c7"><span class="c9">In this example, </span><span class="c16">selectFirst</span><span class="c16">()</span><span class="c9">&nbsp;method is used. If multiple elements need to be selected, you can use the </span><span class="c16">select()</span><span class="c9">&nbsp;method. This will take the CSS selector as a parameter and return an instance of </span><span class="c16">Elements</span><span class="c9">, which is an extension of the </span><span class="c16">ArrayList&lt;Element&gt; </span><span class="c9">type</span><span class="c23 c14 c9">.</span></p><h2 class="c33" id="h.th9hss4ok1kf"><span>Web scraping with Java using HtmlUnit</span></h2><p class="c12"><span class="c23 c14 c26">There are many methods to read and modify the loaded page. HtmlUnit makes it easy to interact with a web page like a browser, which involves reading text, filling forms, clicking buttons, etc. In this case, we will be using methods from this library to read information from URLs.</span></p><p class="c12 c17"><span class="c23 c14 c26"></span></p><p class="c12"><span>As discussed in the previous section, there are three steps involved in web scraping with Java.</span></p><h3 class="c28" id="h.jxmoyhhzt82w"><span>Getting and parsing the HTML</span></h3><p class="c27 c7"><span class="c9">The first step is to get the Java libraries. Maven can help here. Create a new maven project or use the one created in the previous section. If you do not want to use Maven, head over to </span><span class="c34"><a class="c37" href="https://www.google.com/url?q=https://sourceforge.net/projects/htmlunit/&amp;sa=D&amp;source=editors&amp;ust=1669732597900037&amp;usg=AOvVaw2rmo8BU09nUapjBNIKpYdZ">this page</a></span><span class="c23 c14 c9">&nbsp;to find alternate downloads.</span></p><p class="c27 c7"><span class="c9">In the </span><span class="c16">pom.xml </span><span class="c9">file, add a new section for </span><span class="c16">dependencies</span><span class="c9">&nbsp;and add a dependency for HtmlUnit. The </span><span class="c16">pom.xml</span><span class="c23 c14 c9">&nbsp;file would look something like this:</span></p><a id="t.21b485bdbb66cb162e64ddf8412f4926049a85d7"></a><a id="t.7"></a><table class="c29"><tr class="c22"><td class="c15" colspan="1" rowspan="1"><p class="c13 c7"><span class="c4 c7">&lt;</span><span class="c3">dependency</span><span class="c4 c7">&gt;</span></p><p class="c13 c7"><span class="c3">&nbsp; &nbsp; </span><span class="c4 c7">&lt;</span><span class="c3">groupId</span><span class="c4 c7">&gt;</span><span class="c3">net</span><span class="c2">.</span><span class="c3">sourceforge</span><span class="c2">.</span><span class="c3">htmlunit</span><span class="c4 c7">&lt;/</span><span class="c3">groupId</span><span class="c4 c7">&gt;</span></p><p class="c13 c7"><span class="c3">&nbsp; &nbsp; </span><span class="c4 c7">&lt;</span><span class="c3">artifactId</span><span class="c4 c7">&gt;</span><span class="c3">htmlunit</span><span class="c4 c7">&lt;/</span><span class="c3">artifactId</span><span class="c4 c7">&gt;</span></p><p class="c13 c7"><span class="c3">&nbsp; &nbsp; </span><span class="c4 c7">&lt;</span><span class="c3">version</span><span class="c4 c7">&gt;</span><span class="c5 c24 c7">2.51.0</span><span class="c4 c7">&lt;/</span><span class="c3">version</span><span class="c4 c7">&gt;</span></p><p class="c13 c7"><span class="c4 c7">&lt;/</span><span class="c3">dependency</span><span class="c4 c7">&gt;</span></p></td></tr></table><h3 class="c28" id="h.1i8a7bu8gpqv"><span>Getting the HTML</span></h3><p class="c12"><span class="c23 c14 c26">The second step of web scraping with Java is to retrieve the HTML from the target URL as a Java object. Let&rsquo;s begin with the imports:</span></p><p class="c13 c17 c7"><span class="c20 c14 c8 c7"></span></p><a id="t.baaae5b001506426e06a8c1d066af4f227806cd2"></a><a id="t.8"></a><table class="c29"><tr class="c22"><td class="c15" colspan="1" rowspan="1"><p class="c13 c7"><span class="c0">import</span><span class="c3">&nbsp;com</span><span class="c2">.</span><span class="c3">gargoylesoftware</span><span class="c2">.</span><span class="c3">htmlunit</span><span class="c2">.</span><span class="c3">WebClient</span><span class="c2">;</span></p><p class="c13 c7"><span class="c0">import</span><span class="c3">&nbsp;com</span><span class="c2">.</span><span class="c3">gargoylesoftware</span><span class="c2">.</span><span class="c3">htmlunit</span><span class="c2">.</span><span class="c3">html</span><span class="c2">.</span><span class="c3">DomNode</span><span class="c2">;</span></p><p class="c13 c7"><span class="c0">import</span><span class="c3">&nbsp;com</span><span class="c2">.</span><span class="c3">gargoylesoftware</span><span class="c2">.</span><span class="c3">htmlunit</span><span class="c2">.</span><span class="c3">html</span><span class="c2">.</span><span class="c3">DomNodeList</span><span class="c2">;</span></p><p class="c7 c13"><span class="c0">import</span><span class="c3">&nbsp;com</span><span class="c2">.</span><span class="c3">gargoylesoftware</span><span class="c2">.</span><span class="c3">htmlunit</span><span class="c2">.</span><span class="c3">html</span><span class="c2">.</span><span class="c3">HtmlElement</span><span class="c2">;</span></p><p class="c13 c7"><span class="c0">import</span><span class="c3">&nbsp;com</span><span class="c2">.</span><span class="c3">gargoylesoftware</span><span class="c2">.</span><span class="c3">htmlunit</span><span class="c2">.</span><span class="c3">html</span><span class="c2">.</span><span class="c3">HtmlPage</span><span class="c2">;</span></p></td></tr></table><p class="c13 c17 c7"><span class="c20 c14 c8 c7"></span></p><p class="c27 c7"><span class="c9">As discussed in the previous section, it is not a good practice to do a wildcard import such as </span><span class="c16">import com.gargoylesoftware.htmlunit.html.*</span><span>.</span><span class="c23 c14 c9">&nbsp;Import only what you need. The above imports are what we are going to use in this Java web scraping tutorial.</span></p><p class="c27 c7"><span class="c9">In this example, we will scrape this </span><span class="c34"><a class="c37" href="https://www.google.com/url?q=https://librivox.org/the-first-men-in-the-moon-by-hg-wells&amp;sa=D&amp;source=editors&amp;ust=1669732597907274&amp;usg=AOvVaw1OkoJvxrE62J3WnbS1dGB8">Librivox page</a></span><span class="c23 c14 c9">.</span></p><p class="c27 c7"><span class="c9">HtmlUnit uses the </span><span class="c36 c16">WebClient</span><span class="c9">&nbsp;class to get the page. The first step is to create an instance of this class. In this example, there is no need for CSS rendering, and there is no use of JavaScript as well. We can set the options to disable these two:</span></p><a id="t.91aa72fde83f4020a68fed08399ec63d1eefb2e7"></a><a id="t.9"></a><table class="c29"><tr class="c22"><td class="c15" colspan="1" rowspan="1"><p class="c13 c7"><span class="c3">WebClient webClient </span><span class="c4 c7">=</span><span class="c3">&nbsp;</span><span class="c0">new</span><span class="c3">&nbsp;</span><span class="c5 c6">WebClient</span><span class="c2">();</span></p><p class="c13 c7"><span class="c3">webClient</span><span class="c2">.</span><span class="c5 c6">getOptions</span><span class="c2">().</span><span class="c5 c6">setCssEnabled</span><span class="c2">(</span><span class="c5 c24 c7">false</span><span class="c2">);</span></p><p class="c13 c7"><span class="c3">webClient</span><span class="c2">.</span><span class="c5 c6">getOptions</span><span class="c2">().</span><span class="c5 c6">setJavaScriptEnabled</span><span class="c2">(</span><span class="c5 c24 c7">false</span><span class="c2">);</span></p><p class="c13 c7"><span class="c3">HtmlPage page </span><span class="c4 c7">=</span><span class="c3">&nbsp;webClient</span><span class="c2">.</span><span class="c5 c6">getPage</span><span class="c2">(</span><span class="c5 c19 c7">&quot;https://librivox.org/the-first-men-in-the-moon-by-hg-wells&quot;</span><span class="c2">);</span></p></td></tr></table><p class="c13 c17 c7"><span class="c20 c14 c8 c7"></span></p><p class="c27 c7"><span class="c9">Note that the </span><span class="c36 c16">getPage()</span><span class="c9">&nbsp;functions can throw </span><span class="c36 c16">IOException</span><span class="c23 c14 c9">. You would need to surround it in a try-catch.</span></p><p class="c27 c7"><span class="c9">Here is an implementation example of a function that returns an instance of </span><span class="c16 c36">HtmlPage</span><span class="c9">:</span></p><a id="t.37184845a3da69550cfcb46f5a7d0beb3294988e"></a><a id="t.10"></a><table class="c29"><tr class="c22"><td class="c15" colspan="1" rowspan="1"><p class="c13 c7"><span class="c0">public</span><span class="c3">&nbsp;</span><span class="c0">static</span><span class="c3">&nbsp;HtmlPage </span><span class="c5 c6">getDocument</span><span class="c2">(</span><span class="c3">String url</span><span class="c2">)</span><span class="c3">&nbsp;</span><span class="c2">{</span></p><p class="c13 c7"><span class="c3">&nbsp; &nbsp; HtmlPage page </span><span class="c4 c7">=</span><span class="c3">&nbsp;</span><span class="c0">null</span><span class="c2">;</span></p><p class="c13 c7"><span class="c3">&nbsp; &nbsp; </span><span class="c0">try</span><span class="c3">&nbsp;</span><span class="c2">(</span><span class="c3">final WebClient webClient </span><span class="c4 c7">=</span><span class="c3">&nbsp;</span><span class="c0">new</span><span class="c3">&nbsp;</span><span class="c5 c6">WebClient</span><span class="c2">())</span><span class="c3">&nbsp;</span><span class="c2">{</span></p><p class="c13 c7"><span class="c3">&nbsp; &nbsp; &nbsp; &nbsp; webClient</span><span class="c2">.</span><span class="c5 c6">getOptions</span><span class="c2">().</span><span class="c5 c6">setCssEnabled</span><span class="c2">(</span><span class="c5 c24 c7">false</span><span class="c2">);</span></p><p class="c13 c7"><span class="c3">&nbsp; &nbsp; &nbsp; &nbsp; webClient</span><span class="c2">.</span><span class="c5 c6">getOptions</span><span class="c2">().</span><span class="c5 c6">setJavaScriptEnabled</span><span class="c2">(</span><span class="c5 c24 c7">false</span><span class="c2">);</span></p><p class="c13 c7"><span class="c3">&nbsp; &nbsp; &nbsp; &nbsp; page </span><span class="c4 c7">=</span><span class="c3">&nbsp;webClient</span><span class="c2">.</span><span class="c5 c6">getPage</span><span class="c2">(</span><span class="c3">url</span><span class="c2">);</span></p><p class="c13 c7"><span class="c3">&nbsp; &nbsp; </span><span class="c2">}</span><span class="c3">&nbsp;</span><span class="c0">catch</span><span class="c3">&nbsp;</span><span class="c2">(</span><span class="c3">IOException e</span><span class="c2">)</span><span class="c3">&nbsp;</span><span class="c2">{</span></p><p class="c13 c7"><span class="c3">&nbsp; &nbsp; &nbsp; &nbsp; e</span><span class="c2">.</span><span class="c5 c6">printStackTrace</span><span class="c2">();</span></p><p class="c13 c7"><span class="c3">&nbsp; &nbsp; </span><span class="c2">}</span></p><p class="c13 c7"><span class="c3">&nbsp; &nbsp; </span><span class="c0">return</span><span class="c3">&nbsp;page</span><span class="c2">;</span></p><p class="c13 c7"><span class="c2 c14">}</span></p></td></tr></table><h3 class="c28" id="h.dsq3o3el7acq"><span>Querying HTML</span></h3><p class="c27 c7"><span class="c9">There are three categories of methods that can be used with </span><span class="c16">HTMLPage</span><span class="c9">. The first is DOM methods such as </span><span class="c16">getElementById()</span><span class="c9">, </span><span class="c16">getElementByName()</span><span class="c9">, etc., that return one element. These also have their counterparts like </span><span class="c16">getElementsById()</span><span class="c9">&nbsp;that return all the matches. These methods return a </span><span class="c16">DomElement</span><span class="c9">&nbsp;object or a List of </span><span class="c16">DomElement</span><span class="c9">&nbsp;objects.</span></p><a id="t.395ff5ed2ea8b95a8289f08872d78db011c5911b"></a><a id="t.11"></a><table class="c29"><tr class="c22"><td class="c15" colspan="1" rowspan="1"><p class="c13 c7"><span class="c3">HtmlPage page </span><span class="c4 c7">=</span><span class="c3">&nbsp;webClient</span><span class="c2">.</span><span class="c5 c6">getPage</span><span class="c2">(</span><span class="c5 c7 c19">&quot;https://en.wikipedia.org/wiki/Jsoup&quot;</span><span class="c2">);</span></p><p class="c13 c7"><span class="c3">DomElement firstHeading </span><span class="c4 c7">=</span><span class="c3">&nbsp;page</span><span class="c2">.</span><span class="c5 c6">getElementById</span><span class="c2">(</span><span class="c5 c19 c7">&quot;firstHeading&quot;</span><span class="c2">);</span></p><p class="c13 c7"><span class="c3">System</span><span class="c2">.</span><span class="c3">out</span><span class="c2">.</span><span class="c5 c6">print</span><span class="c2">(</span><span class="c3">firstHeading</span><span class="c2">.</span><span class="c5 c6">asNormalizedText</span><span class="c2">());</span><span class="c3">&nbsp;</span><span class="c5 c41 c7">// prints Jsoup</span></p></td></tr></table><p class="c13 c17 c7"><span class="c20 c14 c41 c7"></span></p><p class="c27 c7"><span class="c9">The second category of a selector uses XPath. Navigate to </span><span class="c34"><a class="c37" href="https://www.google.com/url?q=https://librivox.org/the-first-men-in-the-moon-by-hg-wells&amp;sa=D&amp;source=editors&amp;ust=1669732597920396&amp;usg=AOvVaw12_96qRo5H5Aak3zkQ5S8J">this page</a></span><span class="c9">, right-click the book title and click </span><span class="c25 c9">Inspect</span><span class="c9">. If you are already comfortable with XPath, you should be able to see that the XPath to select the book title would be </span><span class="c16">//div[@class=&quot;content-wrap clearfix&quot;]/h1</span><span class="c23 c14 c9">.</span></p>
![web scraping java](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ztkvkcifl7vip8ik3bwh.jpg)<p class="c12 c17 c7"><span class="c23 c14 c9"></span></p><p class="c12 c7"><span style="overflow: hidden; display: inline-block; margin: 0.00px 0.00px; border: 0.00px solid #000000; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px); width: 624.00px; height: 260.00px;"><img alt="" src="images/image2.jpg" style="width: 624.00px; height: 260.00px; margin-left: 0.00px; margin-top: 0.00px; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px);" title=""></span></p><p class="c7 c35"><span class="c9">I</span><span>mage: Selecting Elements by Xpath</span></p><p class="c12 c17 c7"><span class="c23 c14 c9"></span></p><p class="c27 c7"><span class="c9">There are two methods that can work with XPath &mdash; </span><span class="c16">getByXPath()</span><span class="c9">&nbsp;and </span><span class="c16">getFirstByXPath()</span><span class="c9">. They return </span><span class="c16">HtmlElement</span><span class="c23 c14 c9">&nbsp;instead of DomElement. Note that special characters like quotation marks will need to be escaped using a backslash:</span></p><a id="t.b6b59db9c3e0f3d56a523f2c36a61f007b59bf14"></a><a id="t.12"></a><table class="c29"><tr class="c22"><td class="c15" colspan="1" rowspan="1"><p class="c13 c7"><span class="c3">HtmlElement book </span><span class="c4 c7">=</span><span class="c3">&nbsp;page</span><span class="c2">.</span><span class="c5 c6">getFirstByXPath</span><span class="c2">(</span><span class="c5 c19 c7">&quot;//div[@class=\&quot;content-wrap clearfix\&quot;]/h1&quot;</span><span class="c2">);</span></p><p class="c13 c7"><span class="c3">System</span><span class="c2">.</span><span class="c3">out</span><span class="c2">.</span><span class="c5 c6">print</span><span class="c2">(</span><span class="c3">book</span><span class="c2">.</span><span class="c5 c6">asNormalizedText</span><span class="c2">());</span></p></td></tr></table><p class="c13 c17 c7"><span class="c20 c14 c8 c7"></span></p><p class="c27 c7"><span class="c9">Lastly, the third category of methods uses CSS selectors. These methods are </span><span class="c16">querySelector()</span><span class="c9">&nbsp;and </span><span class="c16">querySelectorAll()</span><span class="c9">. They return </span><span class="c16">DomNode</span><span class="c9">&nbsp;and </span><span class="c16">DomNodeList&lt;DomNode&gt;</span><span class="c23 c14 c9">&nbsp;respectively.</span></p><p class="c27 c7"><span class="c9">To make this Java web scraper tutorial more realistic, let&rsquo;s print all the chapter names, reader names, and duration from the page. The first step is to determine the selector that can select all rows. Next, we will use the </span><span class="c16">querySelectorAll()</span><span class="c9">&nbsp;method to select all the rows. Finally, we will run a loop on all the rows and call </span><span class="c16">querySelector()</span><span class="c9">&nbsp;to extract the content of each cell.</span></p><a id="t.c459d7aa9a5723774bcb3973ca2f525713e45225"></a><a id="t.13"></a><table class="c29"><tr class="c22"><td class="c15" colspan="1" rowspan="1"><p class="c13 c7"><span class="c3">String selector </span><span class="c4 c7">=</span><span class="c3">&nbsp;</span><span class="c5 c19 c7">&quot;.chapter-download tbody tr&quot;</span><span class="c2">;</span></p><p class="c13 c7"><span class="c3">DomNodeList</span><span class="c4 c7">&lt;</span><span class="c3">DomNode</span><span class="c4 c7">&gt;</span><span class="c3">&nbsp;rows </span><span class="c4 c7">=</span><span class="c3">&nbsp;page</span><span class="c2">.</span><span class="c5 c6">querySelectorAll</span><span class="c2">(</span><span class="c3">selector</span><span class="c2">);</span></p><p class="c13 c7"><span class="c0">for</span><span class="c3">&nbsp;</span><span class="c2">(</span><span class="c3">DomNode row </span><span class="c4 c7">:</span><span class="c3">&nbsp;rows</span><span class="c2">)</span><span class="c3">&nbsp;</span><span class="c2">{</span></p><p class="c13 c7"><span class="c3">&nbsp; &nbsp; String chapter </span><span class="c4 c7">=</span><span class="c3">&nbsp;row</span><span class="c2">.</span><span class="c5 c6">querySelector</span><span class="c2">(</span><span class="c5 c19 c7">&quot;td:nth-child(2) a&quot;</span><span class="c2">).</span><span class="c5 c6">asNormalizedText</span><span class="c2">();</span></p><p class="c13 c7"><span class="c3">&nbsp; &nbsp; String reader </span><span class="c4 c7">=</span><span class="c3">&nbsp;row</span><span class="c2">.</span><span class="c5 c6">querySelector</span><span class="c2">(</span><span class="c5 c19 c7">&quot;td:nth-child(3) a&quot;</span><span class="c2">).</span><span class="c5 c6">asNormalizedText</span><span class="c2">();</span></p><p class="c13 c7"><span class="c3">&nbsp; &nbsp; String duration </span><span class="c4 c7">=</span><span class="c3">&nbsp;row</span><span class="c2">.</span><span class="c5 c6">querySelector</span><span class="c2">(</span><span class="c5 c19 c7">&quot;td:nth-child(4)&quot;</span><span class="c2">).</span><span class="c5 c6">asNormalizedText</span><span class="c2">();</span></p><p class="c13 c7"><span class="c3">&nbsp; &nbsp; System</span><span class="c2">.</span><span class="c3">out</span><span class="c2">.</span><span class="c5 c6">println</span><span class="c2">(</span><span class="c3">chapter </span><span class="c4 c7">+</span><span class="c3">&nbsp;</span><span class="c5 c19 c7">&quot;\t &quot;</span><span class="c3">&nbsp;</span><span class="c4 c7">+</span><span class="c3">&nbsp;reader </span><span class="c4 c7">+</span><span class="c3">&nbsp;</span><span class="c5 c19 c7">&quot;\t &quot;</span><span class="c3">&nbsp;</span><span class="c4 c7">+</span><span class="c3">&nbsp;duration</span><span class="c2">);</span></p><p class="c13 c7"><span class="c2">}</span></p></td></tr></table><h2 class="c33" id="h.nryovtl7at4h"><span>Conclusion</span></h2><p class="c12"><span>Almost every business needs web scraping to analyze data and stay competitive in the market. Knowing the basics of web scraping and how to build a web scraper using Java can result in more informed and quick decisions, which are essential for a business to succeed.</span></p><p class="c12 c17"><span class="c20 c14 c26"></span></p></body></html>