Web Scraping With R Using rvest Tutorial
Tags: webscraping, webdev, beginners, tutorial
Published: 2022-11-24 08:12:35

<p class="c1"><span class="c16 c32">Let&rsquo;s be honest &ndash; if you need to learn a new programming language, getting started with web scraping can be daunting. Thankfully, more and more programming languages provide powerful libraries to help scrape data from web pages more conveniently.</span></p><p class="c1"><span class="c16 c32">In this tutorial, we&rsquo;ll cover the basics of web scraping with R, which is one of the most popular programming languages for data and statistical analysis. We&#39;ll begin with the scraping of static pages and shift the focus to the techniques that can be used for scraping data from dynamic websites that use JavaScript to render the content.</span></p><h2 class="c20" id="h.8hi03gqx4ozf"><span class="c22 c25 c24 c23">Installing requirements</span></h2><p class="c1"><span class="c32">We can break down the installation of required components</span><span class="c16 c32">&nbsp;into two sections &mdash; Installing R and RStudio and Installing the libraries.</span></p><h3 class="c27" id="h.sa147i7hgb9m"><span class="c3">Installing R and RStudio</span></h3><p class="c1"><span class="c16 c32">The first stage is to prepare the development environment for R. Two components will be needed &ndash; R and RStudio.</span></p><ul class="c26 lst-kix_5efqk1hp219s-0 start"><li class="c29 li-bullet-0"><span class="c32">To download and install R, visit </span><span class="c38"><a class="c9" href="https://www.google.com/url?q=https://cran.r-project.org/&amp;sa=D&amp;source=editors&amp;ust=1669279595950410&amp;usg=AOvVaw0qyubKeF1JOWSe31hg9S0w">this page</a></span><span class="c16 c32">. Installing the base distribution is enough.</span></li></ul><p class="c1"><span class="c16 c32">Alternatively, you can use package managers such as Homebrew for Mac or Chocolatey for Windows.</span></p><p class="c1"><span class="c16 c32">For macOS, run the following:</span></p><a id="t.8a9e4741548300f8c86907a038f0d663a2dbf847"></a><a id="t.0"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">brew install r</span></p></td></tr></table><p class="c1 c6"><span class="c16 c32"></span></p><p class="c1"><span class="c16 c12">For Windows, run the following:</span></p><a id="t.3c0d30233dab5e823a26fbc6ca90629c2caf87ff"></a><a id="t.1"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">choco install r</span><span class="c15">.</span><span class="c4">project</span></p></td></tr></table><p class="c6 c37"><span class="c22 c12 c34"></span></p><ul class="c26 lst-kix_ocqfqxbndd9n-0 start"><li class="c29 li-bullet-0"><span class="c12">Next, download and install RStudio by visiting </span><span class="c21"><a class="c9" href="https://www.google.com/url?q=https://www.rstudio.com/products/rstudio/download&amp;sa=D&amp;source=editors&amp;ust=1669279595951997&amp;usg=AOvVaw3SOHWlIbns2jfi20jXtFG4">this page</a></span><span class="c16 c12">. The free version, RStudio Desktop, is enough.</span></li></ul><p class="c1"><span class="c16 c12">If you prefer package managers, the following are the commands for macOS using Homebrew and for Windows using Chocolatey:</span></p><p class="c1"><span class="c16 c12">For macOS, run the following:</span></p><a id="t.3d6648c28cbe2cc2f2efb4d820bae202010cd210"></a><a id="t.2"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">brew install </span><span class="c8">--</span><span class="c4">cask r</span><span class="c8">-</span><span class="c4">studio</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c16 c12">For Windows, run the following:</span></p><a id="t.c5ed202291283e586f7e3dca68d79373c5a8782c"></a><a id="t.3"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">choco install r</span><span class="c15">.</span><span class="c4">studio</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c16 c12">Once installed, launch RStudio.</span></p>
![rstudio](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dt6h5tmjvhfhnyabnsjh.png)<p class="c1"><span style="overflow: hidden; display: inline-block; margin: 0.00px 0.00px; border: 0.00px solid #000000; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px); width: 624.00px; height: 385.33px;"><img alt="" src="images/image3.png" style="width: 624.00px; height: 385.33px; margin-left: 0.00px; margin-top: 0.00px; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px);" title=""></span></p><h3 class="c27" id="h.pr6mszhra0p"><span class="c3">Installing required libraries</span></h3><p class="c1"><span class="c12">There are two ways to install the required libraries. The first is using the user interface of RStudio. Locate the </span><span class="c12 c23">Packages</span><span class="c12">&nbsp;tab in the </span><span class="c12 c23">Help</span><span class="c12">&nbsp;section. Select the </span><span class="c12 c23">Packages</span><span class="c12">&nbsp;tab to activate the Packages section. In this section, click the </span><span class="c12 c23">Install </span><span class="c16 c12">button.</span></p><p class="c1"><span class="c12">The </span><span class="c12 c23">Install Package</span><span class="c12">&nbsp;dialog is now open. Enter the package names in the text box for Packages. Lastly, click </span><span class="c12 c23">Install</span><span class="c16 c12">.</span></p><p class="c1"><span class="c12">For the first section of the tutorial, the package that we&rsquo;ll use is </span><span class="c14">rvest</span><span class="c12">. We also need the </span><span class="c14">dplyr</span><span class="c16 c12">&nbsp;package to allow the use of the pipe operator. Doing so makes the code easier to read.</span></p><p class="c5"><span class="c12">Enter these two package names, separated with a comma, and click </span><span class="c12 c23">Install</span><span class="c16 c12">.</span></p>
![web scraping with r](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d9vkfbn5in9or61ar6lr.png)<p class="c5 c6"><span class="c16 c12"></span></p><p class="c5"><span style="overflow: hidden; display: inline-block; margin: 0.00px 0.00px; border: 0.00px solid #000000; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px); width: 624.00px; height: 194.67px;"><img alt="" src="images/image2.png" style="width: 624.00px; height: 194.67px; margin-left: 0.00px; margin-top: 0.00px; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px);" title=""></span></p><p class="c5 c6"><span class="c16 c12"></span></p><p class="c5"><span class="c16 c12">The second way is to install these packages using a console. To proceed, run the following commands in the console:</span></p><p class="c5 c6"><span class="c16 c12"></span></p><a id="t.69e776ca6db1a45c14b6adf1a97d6868c8b55963"></a><a id="t.4"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">install</span><span class="c15">.</span><span class="c11">packages</span><span class="c15">(</span><span class="c2">&quot;rvest&quot;</span><span class="c15">)</span></p><p class="c10"><span class="c4">install</span><span class="c15">.</span><span class="c11">packages</span><span class="c15">(</span><span class="c2">&quot;dplyr&quot;</span><span class="c15">)</span></p></td></tr></table><p class="c5 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c16 c12">The libraries are now installed. The next step is to start scraping data.</span></p><h2 class="c20" id="h.go76olnx2g0i"><span class="c22 c25 c24 c23">Web scraping with rvest</span></h2><p class="c1"><span class="c12">The most popular library for web scraping from any public web page in R is the </span><span class="c21"><a class="c9" href="https://www.google.com/url?q=https://rvest.tidyverse.org/&amp;sa=D&amp;source=editors&amp;ust=1669279595955810&amp;usg=AOvVaw2DMdwEr2vLi_Vf4uSwDvcH">rvest</a></span><span class="c12">. It provides functions to access a public web page and query-specific elements using CSS selectors and XPath. The library is a part of the </span><span class="c21"><a class="c9" href="https://www.google.com/url?q=https://www.tidyverse.org/&amp;sa=D&amp;source=editors&amp;ust=1669279595956087&amp;usg=AOvVaw1YdPRltkZxKwA4wr-Wepi_">Tidyverse</a></span><span class="c16 c12">&nbsp;collection of packages for data science, meaning that the coding conventions are the same across all of Tidyverse&#39;s libraries.</span></p><p class="c1"><span class="c16 c12">Let&#39;s initiate a web scraping operation using rvest. The first step is to send an HTTP GET request to a target web page. We&#39;ll be working with many rvest examples.</span></p><p class="c1"><span class="c16 c12">This section is written as a rvest cheat sheet. You can jump to any section that you need help with.</span></p><h3 class="c27" id="h.56zk01nejw36"><span class="c3">Sending the GET request</span></h3><p class="c1"><span class="c16 c12">Begin with loading the rvest library by entering the following in the Source area:</span></p><a id="t.e84e4735f8b12453b27d9ead3e300ddfd58df102"></a><a id="t.5"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c11">library</span><span class="c15">(</span><span class="c4">rvest</span><span class="c15">)</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c16 c12">All of the commands entered in the source areas can be executed by simply placing the cursor in the desired line, selecting it, and then clicking the Run button on the top right of the Source area. </span></p><p class="c1"><span class="c16 c12">Alternatively, depending on your operating system, you can press Ctrl + Enter or Command + Enter.</span></p><p class="c1"><span class="c12">In this example, we&#39;ll scrape publicly available data from a web page that lists </span><span class="c21"><a class="c9" href="https://www.google.com/url?q=https://en.wikipedia.org/wiki/List_of_ISO_3166_country_codes&amp;sa=D&amp;source=editors&amp;ust=1669279595957518&amp;usg=AOvVaw2GLiYoJwwUQ8s4s32kQnJ_">ISO CountryCodes</a></span><span class="c16 c12">. The hyperlink can be stored in a variable:</span></p><a id="t.fed4b9d95ac2be5ece1ca8a8a60d497c6b48d366"></a><a id="t.6"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">link </span><span class="c8">=</span><span class="c4">&nbsp;</span><span class="c2">&quot;https://en.wikipedia.org/wiki/List_of_ISO_3166_country_codes&quot;</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c12">To send an HTTP GET request to this page, a simple function </span><span class="c14">read_html()</span><span class="c16 c12">&nbsp;can be used.</span></p><p class="c1"><span class="c16 c12">This function needs one mandatory argument: a path or a URL. Note that this function can also read an HTML string:</span></p><a id="t.ab62e9f46cf0f78e6dcba84e6b3de8c6a745c87b"></a><a id="t.7"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">page </span><span class="c8">=</span><span class="c4">&nbsp;</span><span class="c11">read_html</span><span class="c15">(</span><span class="c4">link</span><span class="c15">)</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c12">The function above sends the HTTP GET request to the URL, retrieves the web page, and returns an object of </span><span class="c14">html_document</span><span class="c16 c12">&nbsp;type.</span></p><p class="c1"><span class="c12">The </span><span class="c14">html_document</span><span class="c16 c12">&nbsp;object contains the desired public data from the HTML document. Many rvest functions are available to query and extract specific HTML elements. </span></p><p class="c1"><span class="c16 c12">Note that if you need to use a rvest proxy, run the following to set the proxy in your script:</span></p><a id="t.615946ce00ee64a0454e03a85dfb3e06b415bc4f"></a><a id="t.8"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">Sys</span><span class="c15">.</span><span class="c11">setenv</span><span class="c15">(</span><span class="c4">http_proxy</span><span class="c8">=</span><span class="c2">&quot;http://proxyserver:port&quot;</span><span class="c15">)</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><h3 class="c27" id="h.7lwo75omvdiu"><span class="c3">rvest read_html timeout </span></h3><p class="c1"><span class="c12">The </span><span class="c14">read_html</span><span class="c12">&nbsp;doesn&#39;t provide any way to control the time out. To handle rvest </span><span class="c14">read_html</span><span class="c12">&nbsp;timeouts, you can use the </span><span class="c14">httr</span><span class="c12">&nbsp;library. The </span><span class="c14">GET</span><span class="c12">&nbsp;function from this library and </span><span class="c14">tryCatch</span><span class="c16 c12">&nbsp;can help you handle the time-out errors.</span></p><p class="c1"><span class="c16 c12">Alternatively, you can use the session object from rvest as follows:</span></p><a id="t.1c6a63681cefda0e1103ed0323c4b9d11e20ae6f"></a><a id="t.9"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">url </span><span class="c8">&lt;-</span><span class="c4">&nbsp;</span><span class="c2">&quot;https://quotes.toscrape.com/api/quotes?page=1&quot;</span></p><p class="c10"><span class="c4">page</span><span class="c8">&lt;-</span><span class="c11">read_html</span><span class="c15">(</span><span class="c7">GET</span><span class="c15">(</span><span class="c4">url</span><span class="c15">,</span><span class="c4">&nbsp;</span><span class="c11">timeout</span><span class="c15">(</span><span class="c7">10</span><span class="c15">)))</span><span class="c4">&nbsp;# Method </span><span class="c7">1</span></p><p class="c10"><span class="c4">page </span><span class="c8">&lt;-</span><span class="c4">&nbsp;</span><span class="c11">session</span><span class="c15">(</span><span class="c4">url</span><span class="c15">,</span><span class="c11">timeout</span><span class="c15">(</span><span class="c7">10</span><span class="c15">))</span><span class="c4">&nbsp;#Method </span><span class="c7">2</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><h3 class="c27" id="h.r4wf262c09uu"><span class="c3">Parsing HTML content</span></h3><p class="c1"><span class="c12">The </span><span class="c14">rvest</span><span class="c16 c12">&nbsp;package provides a convenient way to select the HTML elements using CSS Selectors, as well as XPath.</span></p><p class="c1"><span class="c12">Select the elements using </span><span class="c14">html_elements()</span><span class="c16 c12">&nbsp;function. The syntax of this function is as follows:</span></p><a id="t.ef412cbcfa9aa26d122e651f83d6557267099d34"></a><a id="t.10"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">page </span><span class="c8">%&gt;%</span><span class="c4">&nbsp;</span><span class="c11">html_elements</span><span class="c15">(</span><span class="c4">css</span><span class="c8">=</span><span class="c2">&quot;&quot;</span><span class="c15">)</span></p><p class="c10"><span class="c4">page </span><span class="c8">%&gt;%</span><span class="c4">&nbsp;</span><span class="c11">html_elements</span><span class="c15">(</span><span class="c4">xpath</span><span class="c8">=</span><span class="c2">&quot;&quot;</span><span class="c15">)</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c16 c12">An important aspect to note is the plural variation, which will return a list of matching elements. There&#39;s a singular variation of this function that returns only the first matching HTML element:</span></p><a id="t.4368db787e73ea3df38692a0da7c01a9128d1da1"></a><a id="t.11"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">page </span><span class="c8">%&gt;%</span><span class="c4">&nbsp;</span><span class="c11">html_element</span><span class="c15">()</span></p></td></tr></table><p class="c1 c6"><span class="c0"></span></p><p class="c1"><span class="c16 c12">If the selector type isn&#39;t specified, it&#39;s assumed to be a CSS Selector. For example, this Wiki web page contains the desired public data in a table.</span></p>
![public web data](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/s3kjhuidg6yv2hb3lgon.png)<p class="c1"><span style="overflow: hidden; display: inline-block; margin: 0.00px 0.00px; border: 0.00px solid #000000; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px); width: 624.00px; height: 366.67px;"><img alt="" src="images/image5.png" style="width: 624.00px; height: 366.67px; margin-left: 0.00px; margin-top: 0.00px; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px);" title=""></span></p><p class="c1"><span class="c12">The HTML markup of this table is as follows:</span></p><a id="t.819c9c70c422b2388b0f7ab63c24e5d1eaaeeda7"></a><a id="t.12"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c8">&lt;</span><span class="c4">table </span><span class="c19">class</span><span class="c8">=</span><span class="c2">&quot;wikitable sortable jquery-tablesorter&quot;</span><span class="c8">&gt;</span></p></td></tr></table><p class="c1 c6"><span class="c33 c22 c12"></span></p><p class="c1"><span class="c12">The only class needed to create a unique selector is the </span><span class="c14">sortable</span><span class="c12">&nbsp;class. It means that the CSS selector can be as simple as </span><span class="c14">table.sortable</span><span class="c16 c12">. Using this selector, the function call will be as follows:</span></p><a id="t.3abb27a90a861354ef0331bd2ffcdefcaa1f4ebd"></a><a id="t.13"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">htmlElement </span><span class="c8">&lt;-</span><span class="c4">&nbsp;page </span><span class="c8">%&gt;%</span><span class="c4">&nbsp;</span><span class="c11">html_element</span><span class="c15">(</span><span class="c2">&quot;table.sortable&quot;</span><span class="c15">)</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c12">It stores the resulting </span><span class="c14">html_element</span><span class="c12">&nbsp;in a variable </span><span class="c14">htmlElement</span><span class="c16 c12">.</span></p><p class="c1"><span class="c12">The next step of our web scraping project is to convert the public data contained in </span><span class="c14">html_element</span><span class="c16 c12">&nbsp;into a data frame.</span></p><h3 class="c27" id="h.kl3461y9a3tz"><span class="c3">Getting HTML element attributes with rvest</span></h3><p class="c1"><span class="c12">In the previous section, we discussed selecting an element using the </span><span class="c14">html_element</span><span class="c16 c12">&nbsp;function.</span></p><p class="c1"><span class="c12">This function makes it easy to use the rvest </span><span class="c14">select class.</span><span class="c16 c12">&nbsp;For example, if you want to select an element that has the class heading, all you need to write is the following line of code:</span></p><a id="t.f24850e7b7db9985672729a50a1d519a10066d1c"></a><a id="t.14"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">heading </span><span class="c8">&lt;-</span><span class="c4">&nbsp;page </span><span class="c8">%&gt;%</span><span class="c4">&nbsp;</span><span class="c11">html_element</span><span class="c15">(</span><span class="c2">&quot;.heading&quot;</span><span class="c15">)</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c12">Another use case is the rvest </span><span class="c14">div class</span><span class="c16 c12">. If you want to use rvest to select a div, you can use something like:</span></p><a id="t.9059920d9ddea610431ceeb140d3204ebd842c3e"></a><a id="t.15"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">page </span><span class="c8">%&gt;%</span><span class="c4">&nbsp;</span><span class="c11">html_element</span><span class="c15">(</span><span class="c4">&ldquo;div&quot;</span><span class="c15">)</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c12">If you also use rvest to select </span><span class="c14">div</span><span class="c16 c12">&nbsp;with a class:</span></p><a id="t.087a103114e85317226bf2268abb022fa07417d3"></a><a id="t.16"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">page </span><span class="c8">%&gt;%</span><span class="c4">&nbsp;</span><span class="c11">html_element</span><span class="c15">(</span><span class="c4">&ldquo;div</span><span class="c15">.</span><span class="c4">heading&rdquo;</span><span class="c15">)</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c12">You may come across to select HTML nodes is the </span><span class="c14">html_node()</span><span class="c12">&nbsp;function. Note that this way of selecting HTML nodes in rvest is now obsolete. Instead, you should be using </span><span class="c14">html_element()</span><span class="c12">&nbsp;and </span><span class="c14">html_elements()</span><span class="c16 c12">.</span></p><p class="c1"><span class="c12">From this element, you can extract text by calling the function </span><span class="c14">html_text()</span><span class="c16 c12">as follows:</span></p><a id="t.289be5516e1d3b3f93e0c1b9f4f643075f720465"></a><a id="t.17"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">heading </span><span class="c8">%&gt;%</span><span class="c4">&nbsp;</span><span class="c11">html_text</span><span class="c15">()</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c12">Alternatively, if you&#39;re looking for an attribute, you can use the rvest </span><span class="c14">html_attr</span><span class="c12">&nbsp;function. For example, the following code will extract the </span><span class="c14">src</span><span class="c16 c12">attribute of an element:</span></p><a id="t.e8a0bd27b9bbf048f31123b5027e144a4227139c"></a><a id="t.18"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">element </span><span class="c8">%&gt;%</span><span class="c4">&nbsp;</span><span class="c11">html_attr</span><span class="c15">(</span><span class="c4">&ldquo;src&quot;</span><span class="c15">)</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c12">You can use the rvest read table function if you&#39;re working with HTML tables. This function takes an HTML that contains </span><span class="c14">&lt;table&gt;</span><span class="c16 c12">&nbsp;elements and returns a data frame.</span></p><a id="t.05c89c8a294cfa72df8cfd0c806ca68d738c6029"></a><a id="t.19"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c11">html_table</span><span class="c15">(</span><span class="c4">htmlElement</span><span class="c15">)</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c16 c12">You can use this to build rvest extract table code:</span></p><a id="t.ec3a30e7612014b69c7a7974b8c0113cae57bff8"></a><a id="t.20"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">page </span><span class="c8">%&gt;%</span><span class="c4">&nbsp;</span><span class="c11">html_table</span><span class="c15">()</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c16 c12">As you can see, we can send the whole page and rvest reads tables, all of them.</span></p><h3 class="c27" id="h.hlr9rjbphrhl"><span class="c3">Using rvest to scrape a page with JavaScript</span></h3><p class="c1"><span class="c16 c12">If the page you are scraping uses JavaScript, there are two ways to scrape it. The first method is to use RSelenium. This approach is covered at length in the next section of this article.</span></p><p class="c1"><span class="c16 c12">In this section, let&#39;s talk about the second approach. This approach involves finding the hidden API that contains the data.</span></p><p class="c1"><span class="c21"><a class="c9" href="https://www.google.com/url?q=https://quotes.toscrape.com/scroll&amp;sa=D&amp;source=editors&amp;ust=1669279595972035&amp;usg=AOvVaw27INZjTLzgoasqGB9UW4hb">https://quotes.toscrape.com/scroll</a></span><span class="c16 c12">&nbsp;is an excellent example to learn how rvest JavaScript works. This site uses infinite scroll.</span></p><p class="c1"><span class="c16 c12">Open this site in Chrome, press F12, and go to the network tab. Once we have network information, we can implement rvest infinite scrolling easily.</span></p><p class="c1"><span class="c12">Scroll down to load more content and watch the network traffic. You&#39;ll notice that every time a new set of quotes are loaded, a call to the URL </span><span class="c21"><a class="c9" href="https://www.google.com/url?q=https://quotes.toscrape.com/api/quotes?page%3D2&amp;sa=D&amp;source=editors&amp;ust=1669279595972561&amp;usg=AOvVaw0lFP2RVP7BAIhvGVKvLhZi">https://quotes.toscrape.com/api/quotes?page=2</a></span><span class="c16 c12">&nbsp;is sent, where the page number keeps on increasing.</span></p><p class="c1"><span class="c16 c12">Another thing to note is that the response is returned in JSON. There&#39;s an easy way to build a rvest JSON parser.</span></p><p class="c1"><span class="c12">First, read the page.Then look for the </span><span class="c14">&lt;p&gt;</span><span class="c16 c12">&nbsp;tag. This will contain the JSON data in text format.</span></p><a id="t.243b9a381181b5b269c0568a55ba62eddf6ceb4f"></a><a id="t.21"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">page </span><span class="c8">&lt;-</span><span class="c4">&nbsp;</span><span class="c11">read_html</span><span class="c15">(</span><span class="c2">&quot;https://quotes.toscrape.com/api/quotes?page=1&quot;</span><span class="c15">)</span><span class="c4 c22">&nbsp;</span></p><p class="c10"><span class="c4">json_as_text </span><span class="c8">&lt;-</span><span class="c4">&nbsp;page </span><span class="c8">%&gt;%</span><span class="c4">&nbsp;</span><span class="c11">html_element</span><span class="c15">(</span><span class="c2">&quot;p&quot;</span><span class="c15">)</span><span class="c4">&nbsp; </span><span class="c8">%&gt;%</span><span class="c4">&nbsp;</span><span class="c11">html_text</span><span class="c15">()</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c16 c12">To parse this JSON text into an R object, we need to use another library &ndash; jsonlite:</span></p><a id="t.eccdf4a2fb3199a00e9f23cf9e3e0d3763c6de04"></a><a id="t.22"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c11">library</span><span class="c15">(</span><span class="c4">jsonlite</span><span class="c15">)</span></p></td></tr></table><p class="c5 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c12">Now, use the </span><span class="c14">fromJSON</span><span class="c16 c12">&nbsp;method to convert this rvest JSON text into a native R object.</span></p><a id="t.c96b269d3f6ff1d0279d1f826bd1ea815f5186f7"></a><a id="t.23"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">r_object </span><span class="c8">&lt;-</span><span class="c4">&nbsp;json_as_text </span><span class="c8">%&gt;%</span><span class="c4">&nbsp;</span><span class="c11">fromJSON</span><span class="c15">()</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c16 c12">You can use a loop to parse rvest javascript for a page with infinite scroll. In the following example, we&#39;re running this loop ten times:</span></p><p class="c5 c6"><span class="c16 c12"></span></p><a id="t.b380263d29545248415fe9f62212ac72c4d98b07"></a><a id="t.24"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c19">for</span><span class="c4">&nbsp;</span><span class="c15">(</span><span class="c4">x </span><span class="c19">in</span><span class="c4">&nbsp;</span><span class="c7">1</span><span class="c8">:</span><span class="c7">10</span><span class="c15">)</span><span class="c4">&nbsp;</span><span class="c15">{</span></p><p class="c10"><span class="c4">&nbsp; url </span><span class="c8">&lt;-</span><span class="c4">&nbsp;</span><span class="c11">paste</span><span class="c15">(</span><span class="c2">&quot;https://quotes.toscrape.com/api/quotes?page=&quot;</span><span class="c15">,</span><span class="c4">x</span><span class="c15">,</span><span class="c4">&nbsp;sep </span><span class="c8">=</span><span class="c4">&nbsp;</span><span class="c2">&#39;&#39;</span><span class="c15">)</span></p><p class="c10"><span class="c4">&nbsp; page </span><span class="c8">&lt;-</span><span class="c4">&nbsp;</span><span class="c11">read_html</span><span class="c15">(</span><span class="c4">url</span><span class="c15">)</span></p><p class="c10"><span class="c4">&nbsp; # parse page to </span><span class="c19">get</span><span class="c4">&nbsp;</span><span class="c7">JSON</span></p><p class="c10"><span class="c15">}</span></p></td></tr></table><p class="c5 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c16 c12">You can modify this code as per your specific requirements.</span></p><h3 class="c27" id="h.a9hcngu8zp1c"><span class="c3">Saving data to a data frame</span></h3><p class="c1"><span class="c16 c12">Data frames are fundamental data storage structures in R. They resemble matrices but feature some critical differences. Data frames are tightly coupled collections of variables, where each column can be of a different data type. It&#39;s a powerful and efficient way of storing a large amount of data.</span></p><p class="c1"><span class="c16 c12">Most data and statistical analysis methods require data stored in data frames.</span></p><p class="c1"><span class="c12">To convert the data stored in </span><span class="c14">html_element</span><span class="c12">, the function </span><span class="c14">html_table</span><span class="c16 c12">&nbsp;can be used:</span></p><p class="c5 c6"><span class="c16 c12"></span></p><a id="t.022a23ad73c27e2cf19602e8e34d7d4d84eddcdc"></a><a id="t.25"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">df </span><span class="c8">&lt;-</span><span class="c4">&nbsp;</span><span class="c11">html_table</span><span class="c15">(</span><span class="c4">htmlEl</span><span class="c15">,</span><span class="c4">&nbsp;header </span><span class="c8">=</span><span class="c4">&nbsp;</span><span class="c7">FALSE</span><span class="c15">)</span></p></td></tr></table><p class="c5 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c12">The variable </span><span class="c14">df</span><span class="c16 c12">&nbsp;is a data frame.</span></p><p class="c1"><span class="c12">Note the use of an optional parameter </span><span class="c14">header = FALSE</span><span class="c12">. This parameter is only required in certain scenarios. In most cases, the default value of </span><span class="c14">TRUE </span><span class="c16 c12">should work.</span></p><p class="c1"><span class="c16 c12">For the Wiki table, the header spawns two rows. Out of these two rows, the first row can be discarded, making it a three-step process.</span></p><ol class="c26 lst-kix_f5wfduewkjl8-0 start" start="1"><li class="c29 li-bullet-0"><span class="c12 c16">The first step is to disable the automatic assignment of headers, which we have already done.</span></li><li class="c29 li-bullet-0"><span class="c16 c12">The next step is to set the column names with the second row:</span></li></ol><a id="t.ce33cf20dbc75a47126602d1301980f7b4ba0cc1"></a><a id="t.26"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c11">names</span><span class="c15">(</span><span class="c4">df</span><span class="c15">)</span><span class="c4">&nbsp;</span><span class="c8">&lt;-</span><span class="c4">&nbsp;df</span><span class="c15">[</span><span class="c7">2</span><span class="c15">,]</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c16 c12">3. The third step is to delete the first two rows from the body of the data frame.</span></p><a id="t.d01ec82c4ba2059e00506a8c862394c211553925"></a><a id="t.27"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">df </span><span class="c8">=</span><span class="c4">&nbsp;df</span><span class="c15">[</span><span class="c8">-</span><span class="c7">1</span><span class="c8">:-</span><span class="c7">2</span><span class="c15">,]</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c16 c12">The data frame is now ready for further analysis.</span></p><h3 class="c27" id="h.a0kofcv11hxw"><span class="c3">Exporting data frame to a CSV file</span></h3><p class="c1"><span class="c16 c12">Finally, the last step of extracting data from the HTML document is to save the data frame to a CSV file.</span></p><p class="c1"><span class="c12">To export the data frame, use the </span><span class="c14">write.csv</span><span class="c16 c12">&nbsp;function. This function takes two parameters &ndash; the data frame instance and the name of the CSV file:</span></p><a id="t.e28efc49a7d80fe7a91111070cf2ae705f75742c"></a><a id="t.28"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">write</span><span class="c15">.</span><span class="c11">csv</span><span class="c15">(</span><span class="c4">df</span><span class="c15">,</span><span class="c4">&nbsp;</span><span class="c2">&quot;iso_codes.csv&quot;</span><span class="c15">)</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c16 c12">The function will export the data frame to a file iso_codes.csv in the current directory.</span></p><h3 class="c27" id="h.piiqhg1qwymm"><span class="c3">How to download image using rvest</span></h3><p class="c1"><span class="c16 c12">Images are easy to download with rvest. This involves a three-step process:</span></p><ol class="c26 lst-kix_aebpi9siysri-0 start" start="1"><li class="c29 li-bullet-0"><span class="c16 c12">Downloading the page;</span></li><li class="c29 li-bullet-0"><span class="c16 c12">Locating the element that contains the URL of the desired image and extracting the URL of the image;</span></li><li class="c29 li-bullet-0"><span class="c16 c12">Downloading the image. </span></li></ol><p class="c1"><span class="c16 c12">Let&#39;s begin by importing the packages.</span></p><a id="t.4081766a40bf5bf9265736154128f9c99b2fb06b"></a><a id="t.29"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c11">library</span><span class="c15">(</span><span class="c4">rvest</span><span class="c15">)</span></p><p class="c10"><span class="c11">library</span><span class="c15">(</span><span class="c4">dplyr</span><span class="c15">)</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c12">We&#39;ll download the first image from the Wikipedia page in this example. Download the page using the </span><span class="c14">read_htmlI()</span><span class="c12">&nbsp;function and locate the </span><span class="c14">&lt;img&gt;</span><span class="c16 c12">tag that contains the desired image. </span></p><a id="t.9c1190cbf0fa31bdbb81faea1fe8882aa0c702df"></a><a id="t.30"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">url </span><span class="c8">=</span><span class="c4">&nbsp;</span><span class="c2">&quot;https://en.wikipedia.org/wiki/Eiffel_Tower&quot;</span></p><p class="c10"><span class="c4">page </span><span class="c8">&lt;-</span><span class="c4">&nbsp;</span><span class="c11">read_html</span><span class="c15">(</span><span class="c4">url</span><span class="c15">)</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c16 c12">To locate the image, use the CSS selector&quot;.thumbborder&quot;.</span></p><a id="t.027dd06f49556ae2f70aebc2fff898e17ee741e7"></a><a id="t.31"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">image_element </span><span class="c8">&lt;-</span><span class="c4">&nbsp;page </span><span class="c8">%&gt;%</span><span class="c4">&nbsp;</span><span class="c11">html_element</span><span class="c15">(</span><span class="c4">&ldquo;</span><span class="c15">.</span><span class="c4">thumbborder&quot;</span><span class="c15">)</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c12">The next step is to get the actual URL of the image, which is embedded in the </span><span class="c14">src</span><span class="c12">&nbsp;attribute. The rvest function </span><span class="c14">html_attr()</span><span class="c16 c12">&nbsp;comes handy here.</span></p><p class="c5 c6"><span class="c16 c12"></span></p><a id="t.451c29eb822d39986115e07d292b9864af9e7215"></a><a id="t.32"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">image_url </span><span class="c8">&lt;-</span><span class="c4">&nbsp;image_element </span><span class="c8">%&gt;%</span><span class="c4">&nbsp;</span><span class="c11">html_attr</span><span class="c15">(</span><span class="c4">&ldquo;src&quot;</span><span class="c15">)</span></p></td></tr></table><p class="c5 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c12">This URL is a relative URL. Let&#39;s convert this to an absolute URL. This can be done easily using one of the rvest functions &mdash; </span><span class="c14">url_absolute()</span><span class="c16 c12">&nbsp;as follows:</span></p><a id="t.7091a264ebd6b1059681335f30e2fd7568a27006"></a><a id="t.33"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">image_url </span><span class="c8">&lt;-</span><span class="c4">&nbsp;</span><span class="c11">url_absolute</span><span class="c15">(</span><span class="c4">image_url</span><span class="c15">,</span><span class="c4">&nbsp;url</span><span class="c15">)</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c12">Finally, use another rvest function &mdash; </span><span class="c14">download()</span><span class="c16 c12">&nbsp;to download the file as follows:</span></p><p class="c5 c6"><span class="c16 c12"></span></p><a id="t.dc9560d09ae8238af5ec9467d40ab5ab7715aa86"></a><a id="t.34"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">download</span><span class="c15">.</span><span class="c11">file</span><span class="c15">(</span><span class="c4">image_url</span><span class="c15">,</span><span class="c4">&nbsp;destfile </span><span class="c8">=</span><span class="c4">&nbsp;</span><span class="c11">basename</span><span class="c15">(</span><span class="c2">&quot;paris.jpg&quot;</span><span class="c15">))</span></p></td></tr></table><p class="c5 c6"><span class="c16 c12"></span></p><h3 class="c27" id="h.ku4snignbek1"><span class="c3">Web scraping: rvest vs. BeautifulSoup</span></h3><p class="c1"><span class="c16 c12">The most popular languages for public data analysis are Python and R. To analyze data, first, we need to collect publicly available data. The most common technique for collecting public data is web scraping. Thus, Python and R are suitable languages for web scraping, especially when the data needs to undergo analysis.</span></p><p class="c1"><span class="c16 c12">In this section, let&rsquo;s quickly look at rvest vs beautifulsoup.</span></p><p class="c1"><span class="c16 c12">The BeautifulSoup library in Python is one of the most popular web scraping libraries because it provides an easy-to-use wrapper over the more complex libraries such as lxml. Rvest is inspired by BeautifulSoup. It&#39;s also a wrapper over more complex R libraries such as xml2 and httr.</span></p><p class="c1"><span class="c16 c12">Both Rvest and BeautifulSoup can query the document DOM using CSS selectors. </span></p><p class="c1"><span class="c16 c12">Rvest provides additional functionality to use Xpath, which BeautifulSoup lacks. BeautifulSoup instead uses its functions to compensate for the lack of XPath. Note that XPath allows traversing up to the parent node, while CSS cannot do that.</span></p><p class="c1"><span class="c16 c12">BeautifulSoup is only a parser. It&#39;s helpful for searching elements on the page but can&#39;t download web pages. You would need to use another library such as Requests for that.</span></p><p class="c1"><span class="c16 c12">Rvest, on the other hand, can fetch the web pages.</span></p><p class="c1"><span class="c16 c12">Eventually, the decision of rvest vs BeautifulSoup would depend on your familiarity with the programming language. If you know Python, use BeautifulSoup. If you know R, use Rvest.</span></p><p class="c1"><span class="c16 c12">Has the tutorial been valuable so far? Please leave your thoughts in the comments below and like the post if you find it useful. </span></p><h2 class="c20" id="h.6x7s2xi3hpm6"><span class="c22 c25 c24 c23">Web scraping with RSelenium</span></h2><p class="c1"><span class="c16 c12">While the rvest library works for most static websites, some dynamic websites use JavaScript to render the content. For such websites, a browser-based rendering solution comes into play.</span></p><p class="c1"><span class="c16 c12">Selenium is a popular browser-based rendering solution that can be used with R. Among the many great features of Selenium are taking screenshots, scrolling down pages, clicking on specific links or parts of the page, and inputting any keyboard stroke onto any part of a web page. It&#39;s the most versatile when combined with classic web scraping techniques.</span></p><p class="c1"><span class="c16 c12">The library that allows dynamic page scraping is RSelenium. It can be installed using the RStudio user interface as explained in the first section of this article, or by using the following command:</span></p><a id="t.0a1d899536be582da2ec0dc9c812acf55ea20e7b"></a><a id="t.35"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">install</span><span class="c15">.</span><span class="c11">package</span><span class="c15">(</span><span class="c2">&quot;RSelenium&quot;</span><span class="c15">)</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c16 c12">Once the package is installed, load the library using the following command:</span></p><p class="c5 c6"><span class="c16 c12"></span></p><a id="t.aec7b616eba8627efc1b2ec3c07fc703d553df4e"></a><a id="t.36"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c11">library</span><span class="c15">(</span><span class="c4">RSelenium</span><span class="c15">)</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c16 c12">The next step is to start the Selenium server and browser.</span></p><h3 class="c27" id="h.bzneo62rtuso"><span class="c3">Starting Selenium</span></h3><p class="c1"><span class="c16 c12">There are two ways of starting a Selenium server and getting a client driver instance.</span></p><p class="c1"><span class="c16 c12">The first is to use RSelenium only, while the second way is to start the Selenium server using Docker and then connect to it using RSelenium. Let&#39;s delve deeper into how the first method works. </span></p><p class="c1"><span class="c16 c12">RSelenium allows to setup the Selenium server and browser using the following function calls:</span></p><a id="t.5fab7289d0fcc19d399a05c08b2ca632590c7e06"></a><a id="t.37"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">rD </span><span class="c8">&lt;-</span><span class="c4">&nbsp;</span><span class="c11">rsDriver</span><span class="c15">(</span><span class="c4">browser</span><span class="c8">=</span><span class="c2">&quot;chrome&quot;</span><span class="c15">,</span><span class="c4">&nbsp;port</span><span class="c8">=</span><span class="c4">9515L</span><span class="c15">,</span><span class="c4">&nbsp;verbose</span><span class="c8">=</span><span class="c7">FALSE</span><span class="c15">)</span></p><p class="c10"><span class="c4">remDr </span><span class="c8">&lt;-</span><span class="c4">&nbsp;rD</span><span class="c15">[[</span><span class="c2">&quot;client&quot;</span><span class="c15">]]</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c16 c12">This will download the required binaries, start the server, and return an instance of the Selenium driver.</span></p><p class="c1"><span class="c16 c12">Alternatively, you can use Docker to run the Selenium server and connect to this instance.</span></p><p class="c1"><span class="c12">Install </span><span class="c21"><a class="c9" href="https://www.google.com/url?q=https://www.docker.com/products/docker-desktop&amp;sa=D&amp;source=editors&amp;ust=1669279595994490&amp;usg=AOvVaw0i9iOMs1yu9LYCJRu_DGhF">Docker</a></span><span class="c16 c12">&nbsp;and run the following command from the terminal.</span></p><a id="t.3a165fd836d975c143b815e366b026e4d3f6be74"></a><a id="t.38"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">docker run </span><span class="c8">-</span><span class="c4">d </span><span class="c8">-</span><span class="c4">p </span><span class="c7">4445</span><span class="c8">:</span><span class="c7">4444</span><span class="c4">&nbsp;selenium</span><span class="c8">/</span><span class="c4">standalone</span><span class="c8">-</span><span class="c4">firefox</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c16 c12">This will download the latest Firefox image and start a container. Apart from Firefox, Chrome and PhantomJS can also be used.</span></p><p class="c1"><span class="c16 c12">Once the server has started, enter the following in RStudio to connect to the server and get an instance of the driver:</span></p><a id="t.737cc02f2f3ccbe49f9de42e545f098a12f7957f"></a><a id="t.39"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">remDr </span><span class="c8">&lt;-</span><span class="c4">&nbsp;</span><span class="c11">remoteDriver</span><span class="c15">(</span></p><p class="c10"><span class="c4">&nbsp; remoteServerAddr </span><span class="c8">=</span><span class="c4">&nbsp;</span><span class="c2">&quot;localhost&quot;</span><span class="c15">,</span></p><p class="c10"><span class="c4">&nbsp; port </span><span class="c8">=</span><span class="c4">&nbsp;4445L</span><span class="c15">,</span></p><p class="c10"><span class="c4">&nbsp; browserName </span><span class="c8">=</span><span class="c4">&nbsp;</span><span class="c2">&quot;firefox&quot;</span></p><p class="c10"><span class="c15">)</span></p><p class="c10"><span class="c11">remDr$open</span><span class="c15">()</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c16 c12">These commands will connect to Firefox running in the Docker container and return an instance of the remote driver. If something isn&#39;t working, examine both the Docker logs and RSelenium error messages.</span></p><h3 class="c27" id="h.p49b6axllv64"><span class="c3">Working with elements in Selenium</span></h3><p class="c1"><span class="c16 c12">Note that after visiting a website and before moving on to the parsing functions, it might be essential to let a considerable amount of time pass. There&#39;s a possibility that data won&rsquo;t be loaded yet, and the entire parsing algorithm will crash. The specific functions could be employed that wait for the particular HTML elements to load fully.</span></p><p class="c1"><span class="c12">The first step is navigating the browser to the desired page. As an example, we&#39;ll scrape the name, prices, and stock availability for all books in the </span><span class="c21"><a class="c9" href="https://www.google.com/url?q=https://books.toscrape.com/catalogue/category/books/science-fiction_16/index.html&amp;sa=D&amp;source=editors&amp;ust=1669279595998255&amp;usg=AOvVaw2CJIE_Bk5rWo38aiulGQDv">science fiction genre</a></span><span class="c16 c12">. The target is a dummy book store for practicing web scraping.</span></p><p class="c1"><span class="c12">To navigate to this URL, use the </span><span class="c14">navigate</span><span class="c16 c12">&nbsp;function:</span></p><a id="t.c0b4fbfd32154298a2fb5e906908ec26b85df3ba"></a><a id="t.40"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c11">remDr$navigate</span><span class="c15">(</span><span class="c2">&quot;https://books.toscrape.com/catalogue/category/books/science-fiction_16&quot;</span><span class="c15">)</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c12">To locate the HTML elements, use </span><span class="c14">findElements()</span><span class="c12">&nbsp;function. This function is flexible and can work with CSS Selectors, XPath, or even with specific attributes, such as an id, name, name tag, etc. For a detailed list, see the </span><span class="c21"><a class="c9" href="https://www.google.com/url?q=https://www.rdocumentation.org/packages/seleniumPipes/versions/0.3.7/topics/findElements&amp;sa=D&amp;source=editors&amp;ust=1669279595999523&amp;usg=AOvVaw1OcrBT_5iZ4vYs6RTkgmtk">official documentation</a></span><span class="c16 c12">.</span></p><p class="c1"><span class="c16 c12">In this example, we&#39;ll work with XPath.</span></p><p class="c5"><span class="c12">The book titles are hidden in the </span><span class="c14">alt</span><span class="c16 c12">&nbsp;attribute of the image thumbnail.</span></p>
![web scraping using r](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/03imxfsdha4dzb0avep7.png)<p class="c5 c6"><span class="c16 c12"></span></p><p class="c5"><span style="overflow: hidden; display: inline-block; margin: 0.00px 0.00px; border: 0.00px solid #000000; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px); width: 624.00px; height: 366.67px;"><img alt="" src="images/image4.png" style="width: 624.00px; height: 366.67px; margin-left: 0.00px; margin-top: 0.00px; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px);" title=""></span></p><p class="c5 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c12">The XPath for these image tags will be </span><span class="c14">//article//img</span><span class="c16 c12">. The following line of code will extract all of these elements:</span></p><a id="t.0a03edf6999d427e58fe823d1211d88754eab2e2"></a><a id="t.41"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">titleElements </span><span class="c8">&lt;-</span><span class="c4">&nbsp;</span><span class="c11">remDr$findElements</span><span class="c15">(</span><span class="c4">using </span><span class="c8">=</span><span class="c4">&nbsp;</span><span class="c2">&quot;xpath&quot;</span><span class="c15">,</span><span class="c4">&nbsp;</span><span class="c2">&quot;//article//img&quot;</span><span class="c15">)</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c12">To extract the value of the </span><span class="c14">alt</span><span class="c12">&nbsp;attribute, we can use the </span><span class="c14">getElementAttribute()</span><span class="c16 c12">&nbsp;function. However, in this particular case we have a list of elements.</span></p><p class="c1"><span class="c12">To extract the attribute from all elements of the list, a custom function can be applied using the </span><span class="c14">sapply</span><span class="c16 c12">&nbsp;function of R:</span></p><a id="t.49f88f2bae140c74e957c32dbf8049ab70ec29ad"></a><a id="t.42"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">titles </span><span class="c8">&lt;-</span><span class="c4">&nbsp;</span><span class="c11">sapply</span><span class="c15">(</span><span class="c4">titleElements</span><span class="c15">,</span><span class="c4">&nbsp;</span><span class="c19">function</span><span class="c15">(</span><span class="c12 c30">x</span><span class="c15">){</span><span class="c11">x$getElementAttribute</span><span class="c15">(</span><span class="c2">&quot;alt&quot;</span><span class="c15">)[[</span><span class="c7">1</span><span class="c15">]]})</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c12">Note that this function will return the attribute value as a list. That&#39;s why we&#39;re using </span><span class="c14">[[1]]</span><span class="c16 c12">&nbsp;to extract only the first value.</span></p><p class="c1"><span class="c16 c12">Moving on to extracting price data, the following is an HTML markup of the HTML element containing price:</span></p><a id="t.7dca41f9a322c391324cb45be2c265e06aae882d"></a><a id="t.43"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c8">&lt;</span><span class="c4">p </span><span class="c19">class</span><span class="c8">=</span><span class="c2">&quot;price_color&quot;</span><span class="c8">&gt;</span><span class="c4">&pound;</span><span class="c7">37.59</span><span class="c8">&lt;/</span><span class="c4">p</span><span class="c8">&gt;</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c12">The XPath to select this will be </span><span class="c14">//*[@class=&#39;price_color&#39;]</span><span class="c12">. Also, this time we&#39;ll use the </span><span class="c14">getElementText()</span><span class="c16 c12">&nbsp;function to get the text from the HTML element. This can be done as follows:</span></p><a id="t.30946c58758df1bbbed8acf34d0bc076e601b2b0"></a><a id="t.44"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">pricesElements </span><span class="c8">&lt;-</span><span class="c4">&nbsp;</span><span class="c11">remDr$findElements</span><span class="c15">(</span><span class="c4">using </span><span class="c8">=</span><span class="c4">&nbsp;</span><span class="c2">&quot;xpath&quot;</span><span class="c15">,</span><span class="c4">&nbsp;</span><span class="c2">&quot;//*[@class=&#39;price_color&#39;]&quot;</span><span class="c15">)</span></p><p class="c10"><span class="c4">prices </span><span class="c8">&lt;-</span><span class="c4">&nbsp; </span><span class="c11">sapply</span><span class="c15">(</span><span class="c4">pricesElements</span><span class="c15">,</span><span class="c4">&nbsp;</span><span class="c19">function</span><span class="c15">(</span><span class="c30 c12">x</span><span class="c15">){</span><span class="c11">x$getElementText</span><span class="c15">()[[</span><span class="c7">1</span><span class="c15">]]})</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c16 c12">Lastly, the lines that extract stock availability will be as follows:</span></p><p class="c5 c6"><span class="c16 c12"></span></p><a id="t.355c2b4aa9dc37162ac2e976397083e0a0c6ee92"></a><a id="t.45"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">stockElements </span><span class="c8">&lt;-</span><span class="c4">&nbsp;</span><span class="c11">remDr$findElements</span><span class="c15">(</span><span class="c4">using </span><span class="c8">=</span><span class="c4">&nbsp;</span><span class="c2">&quot;xpath&quot;</span><span class="c15">,</span><span class="c4">&nbsp;</span><span class="c2">&quot;//*[@class=&#39;instock availability&#39;]&quot;</span><span class="c15">)</span></p><p class="c10"><span class="c4">stocks </span><span class="c8">&lt;-</span><span class="c4">&nbsp; </span><span class="c11">sapply</span><span class="c15">(</span><span class="c4">stockElements</span><span class="c15">,</span><span class="c4">&nbsp;</span><span class="c19">function</span><span class="c15">(</span><span class="c30 c12">x</span><span class="c15">){</span><span class="c11">x$getElementText</span><span class="c15">()[[</span><span class="c7">1</span><span class="c15">]]})</span></p></td></tr></table><p class="c5 c6"><span class="c16 c12"></span></p><h3 class="c27" id="h.syr8ypbpz8w1"><span class="c3">Creating a data frame</span></h3><p class="c1"><span class="c16 c12">At this point, there are three variables. Every variable is a list that contains a required data point.</span></p><p class="c1"><span class="c16 c12">Data points can be used to create a data frame:</span></p><a id="t.71e80c7d70047fd85775ff070fb946c17bfb308e"></a><a id="t.46"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">df </span><span class="c8">&lt;-</span><span class="c4">&nbsp;data</span><span class="c15">.</span><span class="c11">frame</span><span class="c15">(</span><span class="c4">titles</span><span class="c15">,</span><span class="c4">&nbsp;prices</span><span class="c15">,</span><span class="c4">&nbsp;stocks</span><span class="c15">)</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c16 c12">Once the data frame is created, it can be used for further analysis.</span></p><p class="c1"><span class="c16 c12">Moreover, the data frame can be easily exported to CSV with just one line:</span></p><a id="t.5e0801eb45cab80c551e2e071b7230e163af287f"></a><a id="t.47"></a><table class="c13"><tr class="c17"><td class="c18" colspan="1" rowspan="1"><p class="c10"><span class="c4">write</span><span class="c15">.</span><span class="c11">csv</span><span class="c15">(</span><span class="c4">df</span><span class="c15">,</span><span class="c4">&nbsp;</span><span class="c2">&quot;books.csv&quot;</span><span class="c15">)</span></p></td></tr></table><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1"><span class="c12">You can </span><span class="c21"><a class="c9" href="https://www.google.com/url?q=https://github.com/oxylabs/web-scraping-r&amp;sa=D&amp;source=editors&amp;ust=1669279596008320&amp;usg=AOvVaw0wCiUjmTvogft-6awInl4k">click here</a></span><span class="c16 c12">&nbsp;to find the complete code used in this tutorial for your convenience.</span></p><h2 class="c20" id="h.ondaoia3qdbh"><span class="c22 c24 c23 c25">Conclusion</span></h2><p class="c1"><span class="c16 c12">Web scraping with R is a relatively straightforward process if you are already familiar with R or programming in general. For most static web pages, the rvest library provides enough functionality; however, if any kind of dynamic elements come into play, a typical HTML extraction won&rsquo;t be up to the task. If so, RSelenium is the right solution to alleviate a more complicated load.</span></p><p class="c5"><span class="c12">If you want to find out more on how to scrape using other programming languages, check out the articles on our blog, such as </span><span class="c21"><a class="c9" href="https://www.google.com/url?q=https://oxylabs.io/blog/javascript-web-scraping&amp;sa=D&amp;source=editors&amp;ust=1669279596009006&amp;usg=AOvVaw0EesnnfIT7bbubrWFTVFmu">Web Scraping with JavaScript</a></span><span class="c12">, </span><span class="c21"><a class="c9" href="https://www.google.com/url?q=https://oxylabs.io/blog/web-scraping-with-java&amp;sa=D&amp;source=editors&amp;ust=1669279596009194&amp;usg=AOvVaw3MqC2qHoH8I4joeUE5Xhpn">Web Scraping with Java</a></span><span class="c12">, </span><span class="c21"><a class="c9" href="https://www.google.com/url?q=https://oxylabs.io/blog/csharp-web-scraping&amp;sa=D&amp;source=editors&amp;ust=1669279596009412&amp;usg=AOvVaw0HGQU-fUYp9AHJjFA-8oBV">Web Scraping with C#</a></span><span class="c12">, </span><span class="c21"><a class="c9" href="https://www.google.com/url?q=https://oxylabs.io/blog/python-web-scraping&amp;sa=D&amp;source=editors&amp;ust=1669279596009595&amp;usg=AOvVaw3nsQ-BSsGaY10rHY-g9cTt">Python Web Scraping Tutorial</a></span><span class="c12">, </span><span class="c21"><a class="c9" href="https://www.google.com/url?q=https://oxylabs.io/blog/what-is-jupyter-notebook&amp;sa=D&amp;source=editors&amp;ust=1669279596009804&amp;usg=AOvVaw02YP4Jx29lPugyMZ6zb-ev">What is Jupyter Notebook: Introduction</a></span><span class="c16 c12">, etc.</span></p><p class="c5 c6"><span class="c16 c12"></span></p><p class="c5"><span class="c16 c12">Enjoyed our content? Don&rsquo;t forget to like this post and leave a comment below &ndash; we&rsquo;ll be happy to hear your feedback and answer any relevant questions!</span></p><p class="c1 c6"><span class="c16 c12"></span></p><p class="c1 c6"><span class="c16 c12"></span></p><p class="c5 c6"><span class="c16 c24"></span></p></body></html>