How To Perform Web Scraping With JavaScript And Selenium
Tags: javascript, web, webdev, beginners
Published: 2023-01-30 09:23:35

<p>Web scraping with JavaScript is a very useful technique to extract data from the Internet for presentation or analysis. However, in this era of dynamic websites, it becomes difficult to accurately extract data from the web because of the ever-changing nature of data.</p><p>In the past, we used libraries such as <em>urllib</em> or <em>requests</em> to read data from webpages, but things started falling apart with dynamic websites. For websites that are not static, using a common ajax call or request will simply not work.</p><p>Selenium framework was introduced for such automation to help in the aspect of dynamic data and other important features.</p><p>When we were developing an automated job portal for backend engineers, we needed a reliable web scraping framework to help us scrape over 20 job portals and return over a thousand jobs each, and Selenium was the answer.</p><p>But before we delve into <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://www.lambdatest.com/selenium?fp_ref=solomon26">what is Selenium</a> and perform web scraping with JavaScript and Selenium, let’s explore what web scraping is all about.</p><h3 class="heading" levels="1,2,3,4,5,6">What is Web Scraping?</h3><p>Web scraping is the process of extracting data from the Internet for storage in different formats or for further processing and analysis. It can also be referred to as web harvesting or web data extraction.</p><p>By Web scraping with JavaScript and Selenium, we basically automate the manual process of extracting data from the Internet and storing it for further processing.</p><p>It can be useful to extract data for different purposes such as</p><ul><li><p>Academic or business Research</p></li><li><p>Historical Data Analysis</p></li><li><p>Price comparisons</p></li><li><p>SEO</p></li><li><p>Brand protection</p></li><li><p>Ad verification</p></li><li><p>and many other use cases</p></li></ul><p>You can use web scraping to generate product reviews from review websites to help you decide to buy a particular product.</p><p>You can also use web scraping to scrape thousands of job portals for specific jobs you’re interested in applying for.</p><h3 class="heading" levels="1,2,3,4,5,6">Benefits of Web Scraping with JavaScript</h3><p>Below are a few benefits of web scraping:</p><ul><li><p>Extracting product details from e-commerce websites such as prices, product names, images, etc.</p></li><li><p>Web scraping is very useful in research as it can help to gather structured data from multiple websites.</p></li><li><p>Gathering data from different sources for analysis can be automated with web scraping easily.</p></li><li><p>It can be used to collect data for testing and training machine learning models.</p></li></ul><p>The benefit of web scraping is endless and introduces different thoughtful insights and possibilities to businesses.</p><h3 class="heading" levels="1,2,3,4,5,6">Where to check for data protection?</h3><p>In general, web scraping is not illegal, but there are some caveats to it, and to some extent, it has become illegal.</p><p>It’s always very important to be careful and check out some of the different terms you should consider when web scraping to determine if your actions are illegal or not.</p><p>Now that we understand our stand on the legality of web scraping, what methods are there to web scrape data either for a small use case or web scraping at scale? Next, let’s explore the different methods and tools available for web scraping.</p><h3 class="heading" levels="1,2,3,4,5,6">Web Scraping&nbsp;Tools</h3><p>This is the most popular web scraping method where a business deploys an already made software for all their web scraping use cases.</p><p>If you want to access and gather data at scale, you need good web scraping tools that can surpass IP blocking, cloaking, and ReCaptcha. There are popular tools such as Scrapy, Beautiful Soup, Scrapebox, Scrapy Proxy Middleware, Octoparse, Parsehub, and Apify.</p><p>These tools help you with your web scraping task at scale and can surpass different obstacles to help you achieve your goals.</p><p>Selenium is a popular open-source web automation framework used for <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://www.lambdatest.com/automation-testing?fp_ref=solomon26">automated browser testing</a>. This framework helps you write <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://www.lambdatest.com/selenium?fp_ref=solomon26">Selenium</a> test scripts that can be used to <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://www.lambdatest.com/automation-testing?fp_ref=solomon26">automate the testing</a> of websites and web applications, then execute them in different browsers on multiple platforms using any programming language of your choice. However, it can be adapted to solve dynamic web scraping problems, as we will demonstrate in the blog on how you can do web scraping using JavaScript and Selenium.</p><p>Selenium has three major components:</p><ul><li><p><a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--li-anchor" href="https://www.lambdatest.com/blog/run-selenium-ide-test-on-online-selenium-grid/">Selenium IDE</a>: It is a browser plugin — a faster, easier way to create, execute, and debug your Selenium scripts.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--li-anchor" href="https://www.lambdatest.com/blog/selenium-webdriver-tutorial-with-examples/">Selenium WebDriver: </a>It is a set of portable APIs that help you write automated tests in any language that runs on top of your browser.</p></li><li><p><a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--li-anchor" href="https://www.lambdatest.com/blog/why-selenium-grid-is-ideal-for-automated-browser-testing/">Selenium Grid:</a> It automates the process of distributing and scaling tests across multiple browsers, operating systems, and platforms.</p></li></ul><p>When writing this blog on web scraping with JavaScript and Selenium, the latest version of Selenium is <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://www.lambdatest.com/learning-hub/selenium-4">Selenium 4</a>. If you are intrigued to know more about <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://www.lambdatest.com/blog/what-is-deprecated-in-selenium4/">what is new in Selenium 4</a>, you can go through this video:</p><h3 class="heading" levels="1,2,3,4,5,6">Benefits of using Selenium with JavaScript for Web&nbsp;Scraping</h3><p>The benefits of using Selenium are limitless and can be used in different aspects of web scraping with JavaScript — from scraping static to dynamic websites. Below are a few <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://www.lambdatest.com/blog/13-reasons-why-selenium-webdriver-should-be-your-first-choice-for-automation-testing/">benefits of using Selenium WebDriver</a> for your next web scraping project.</p><ul><li><p>The Selenium WebDriver can be configured to scrape a website on browsers whose browser drivers are available.</p></li><li><p>Selenium WebDriver to scrape complicated websites with dynamic contents since some needed data requires HTTP requests.</p></li><li><p>Perform <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--li-anchor" href="https://www.lambdatest.com/automated-screenshot?fp_ref=solomon26">automated screenshot</a> testing by taking numerous screenshots of the web pages as you scrape the web pages.</p></li><li><p>With the Selenium WebDriver, you can simulate a real user working with a browser by loading all resources and downloading browser cookies.</p></li></ul><p>In this <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://www.lambdatest.com/learning-hub/selenium-javascript">Selenium JavaScript tutorial</a>, we will learn how to perform web scraping with JavaScript and Selenium on our local machine and also on LambdaTest <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://www.lambdatest.com/selenium-automation?fp_ref=solomon26">cloud Selenium Grid</a>.</p><p>We will be web scraping the <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://www.youtube.com/channel/UCCymWVaTozpEng_ep0mdUyw?sub_confirmation=1">LambdaTest YouTube channel</a>, displaying the list of videos on a webpage using Selenium, and converting the information to JSON data.</p><p>If you are new to this channel, please subscribe to the LambdaTest YouTube Channel and stay updated with the latest tutorials on <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://www.lambdatest.com/selenium-automation?fp_ref=solomon26">Selenium testing</a>, <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://www.lambdatest.com/cypress-testing">Cypress testing</a>, CI/CD, and more.</p><p>Accelerate JavaScript Automation with Parallel testing. <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://accounts.lambdatest.com/register?fp_ref=solomon26">Try LambdaTest Now!</a></p><h3 class="heading" levels="1,2,3,4,5,6">Implementing Web Scraping with JavaScript and&nbsp;Selenium</h3><p>One of the benefits of Selenium is that it can web-scrape dynamic JavaScript pages where there are dynamic interactions such as hovering over menu items. In this section of this blog on web scraping with JavaScript and Selenium, we will demonstrate how to web scrape YouTube videos and display the data as JSON using Selenium and JavaScript.</p><p>Selenium encapsulates the complexity of building an automation tool for dynamic pages and makes it easy by simplifying installing and configuring the package.</p><h3 class="heading" levels="1,2,3,4,5,6">Prerequisites</h3><p>Before you continue below, you should be familiar with Node.js and Express.js — a Node.js framework. If you are new to <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://www.lambdatest.com/testing-cloud/express-testing">Express testing</a>, you can learn more about it through this guide on <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://www.lambdatest.com/blog/express-testing/">getting started quickly with Express testing</a>.</p><h3 class="heading" levels="1,2,3,4,5,6">Installation</h3><p>We will continue by installing the necessary packages and software to implement web scraping with JavaScript and Selenium.</p><ul><li><p><strong>Visual Studio Code:</strong> This is the code editor we will use in this demo, though you can go with any code editor of your choice. You could use any other IDE of your choice.</p></li><li><p><strong>Browser Driver:</strong> To run the Selenium testing drivers on your machine, you need to download the drivers from the following locations according to the browser of your choice.</p></li></ul><p>BrowserDownload Location</p><p>Opera</p><p><a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://github.com/operasoftware/operachromiumdriver/releases">https://github.com/operasoftware/operachromiumdriver/releases</a></p><p>Firefox</p><p><a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://github.com/mozilla/geckodriver/releases">https://github.com/mozilla/geckodriver/releases</a></p><p>Chrome</p><p><a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="http://chromedriver.chromium.org/downloads">http://chromedriver.chromium.org/downloads</a></p><p>Internet Explorer</p><p><a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://github.com/SeleniumHQ/selenium/wiki/InternetExplorerDriver">https://github.com/SeleniumHQ/selenium/wiki/InternetExplorerDriver</a></p><p>Microsoft Edge</p><p><a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://blogs.windows.com/msedgedev/2015/07/23/bringing-automated-testing-to-microsoft-edge-through-webdriver/">https://blogs.windows.com/msedgedev/2015/07/23/bringing-automated-testing-to-microsoft-edge-through-webdriver/</a></p><p>After downloading any of the drivers, make sure to install it and set it up properly according to your operating system.</p><p>After successfully installing and configuring the browser driver, we will explore how to create a new project and start scraping YouTube videos in the next section of this blog on web scraping with JavaScript and Selenium.</p><h3 class="heading" levels="1,2,3,4,5,6">Creating a&nbsp;Project</h3><p>After the installation, we will create a new VSCode Node.js project and add the following libraries to our package.json file.</p><ol><li><p>Ensure you have Node.js installed and configured properly in your operating system. You can download and install it from the <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--li-anchor" href="https://nodejs.org/en/">Node.js official website</a>.</p></li><li><p>Let’s install the following packages:</p></li></ol><pre><code>    npm install express selenium-webdriver</code></pre><img src="https://cdn-images-1.medium.com/max/1600/1*-61PJMPa1-pVtYIlm25bgg.png"><ol><li><p>After installing the libraries, let’s create an <strong><em>index.js </em></strong>file inside the root directory and set up our express server.</p></li></ol><pre><code>    touch index.js</code></pre><p>Create the express server to listen to incoming requests:</p><pre><code>const express = require('express')
    const { Builder, By } = require('selenium-webdriver');
    
    const app = express()
    const port = 3000
    app.get('/', (request, response) =&gt; {
     // Web Scraping Code here
    })
    app.listen(port, () =&gt; {
      console.log(`Example app listening at http://localhost:${port}`)
    })</code></pre><ol><li><p>The code snippet above creates a new Express server and listens to incoming requests from the user. In the next section of this blog on web scraping with JavaScript and Selenium, we will use the <strong><em>selenium-webdriver</em></strong> library we installed to web scrape our YouTube videos.</p></li></ol><h3 class="heading" levels="1,2,3,4,5,6">YouTube Web Scraping with JavaScript and&nbsp;Selenium</h3><p>In this demonstration, we scrape the following data from the LambdaTest YouTube channel and display it as JSON:</p><ul><li><p><strong>Video Title</strong></p></li><li><p><strong>Video Views</strong></p></li><li><p><strong>Video Published Date</strong></p></li></ul><img src="https://cdn-images-1.medium.com/max/1600/1*Ov4qe6TzMAthZqyn3IBukA.png"><p>You can implement it using the cloud-based <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://www.lambdatest.com/selenium-automation?fp_ref=solomon26">Selenium automation</a> grid provided by LambdaTest, which will take away all the manual configuration both in development and production.</p><p>Below is a simple walkthrough of how the scraping is going to happen before delving into the implementation.</p><ol><li><p>Open the YouTube channel videos page.</p></li><li><p>It scrolls until the end of the page to scrape all the videos available.</p></li><li><p>Retrieves the video title, views, and uploads details using a loop.</p></li><li><p>Convert the scraped information to JSON.</p></li></ol><p><strong>Implementation:</strong></p><pre><code>const express = require('express');
const { Builder, By } = require('selenium-webdriver');
 
const app = express();
const port = 3000;
app.get('/', async (request, response) =&gt; {
 // Web Scraping Code here
 try {
   const data = await WebScrapingLocalTest();
   response.status(200).json(data);
 } catch (error) {
   response.status(500).json({
     message: 'Server error occurred',
   });
 }
});
app.listen(port, () =&gt; {
 console.log(`Example app listening at http://localhost:${port}`);
});
 
async function WebScrapingLocalTest() {
 try {
   driver = await new Builder().forBrowser('chrome').build();
   await driver.get('https://www.youtube.com/c/LambdaTest/videos');
   const allVideos = await driver.findElements(
     By.css('ytd-grid-video-renderer.style-scope.ytd-grid-renderer')
   );
   return await getVideos(allVideos);
 } catch (error) {
   throw new Error(error);
 } finally {
   await driver.quit();
 }
}</code></pre><p><strong>Code Walkthrough:</strong></p><p>Let’s walk through the code together and understand the nitty-gritty of it.</p><p><strong>Step 1: Add the required packages</strong></p><p>First, we require the express and selenium-webdriver packages, and next, we created an Express server to handle the incoming request. The request is handled by calling the <code>WebScrapingTest</code> function.</p><pre><code>app.get('/', async (request, response) =&gt; {
 // Web Scraping Code here
 try {
   const data = await WebScrapingLocalTest();
   response.status(200).json(data);
 } catch (error) {
   response.status(500).json({
     message: 'Server error occurred',
   });
 }
});
 
async function getVideos(videos) {
 let videoDetails = [];
 try {
   for (const video of videos) {
     const title = await video.findElement(By.id('video-title')).getText();
     const views = await video
       .findElement(By.xpath(".//*[@id='metadata-line']/span[1]"))
       .getText();
     const date = await video
       .findElement(By.xpath(".//*[@id='metadata-line']/span[2]"))
       .getText();
     videoDetails.push({
       title: title ?? '',
       views: views ?? '',
       publishedDate: date ?? '',
     });
   }
 } catch (error) {
   console.log(error);
 }
 return videoDetails;
}</code></pre><p><strong>Step 2: Create an instance of Selenium WebDriver</strong></p><p>Inside the openChromeTest function, we created an instance of the Builder class from the Selenium Webdriver package we imported earlier. We also pipe the forBrowser method used to indicate the browser to use for the web scraping and lastly, we call the build method, which creates a new WebDriver client based on this builder’s current configuration.</p><p>With the instance of the builder, we passed in the official LambdaTest YouTube videos URL in the get method of the builder class.</p><pre><code>driver = await new Builder().forBrowser('chrome').build();
       await driver.get('https://www.youtube.com/c/LambdaTest/videos');</code></pre><p><strong>Step 3: Load all the YouTube Videos on the page</strong><br>&nbsp;Next, we will load all the videos within the page and collect specific information using a loop in JavaScript.</p><p>To do that, we need to target a specific element in the YouTube Videos page that contains all the videos using the Chrome inspector tool, we can select any element within the webpage. Below is a screenshot showing the element and the <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://www.lambdatest.com/blog/how-pro-testers-use-css-selectors-in-selenium-automation-scripts/">CSS selector</a> to use.</p><img src="https://cdn-images-1.medium.com/max/1600/1*WUiPAShYkFOEAxWnWrWpqg.png"><p>As of the time of writing, the class attribute to target is <code>ytd-grid-video-renderer style-scope ytd-grid-renderer</code>, which is a combination of the CSS class property of the parent container and the individual video container.</p><p>Next, we are going to convert the classes to CSS QuerySelector and target all the videos using the <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://www.lambdatest.com/blog/findelement-and-findelements-in-selenium/">FindElements method in the Selenium WebDriver</a> and <strong>By</strong> instance class.</p><p>Here is how it looks:</p><pre><code>
 const allVideos = await driver.findElements(
         By.css('ytd-grid-video-renderer.style-scope.ytd-grid-renderer')
       );</code></pre><p>Selenium uses the By instance class to represent different query selectors, so you can select elements using <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://www.lambdatest.com/blog/locators-in-selenium-webdriver-with-examples/">Selenium locators</a> like CSS name, ID, Class, etc.</p><p>Finally, the findElements method returns an object of all the selected HTML DOM Nodes called WebElement, which we will loop through to retrieve metadata of the videos while performing web scraping with JavaScript and Selenium.</p><p><strong>Step 4: Converting to JSON</strong></p><p>Lastly, we loop through the WebElement and retrieve metadata of the videos such as the video title, video views, and video published date.</p><pre><code>async function getVideos(videos) {
 let videoDetails = [];
 try {
   for (const video of videos) {
     const title = await video.findElement(By.id('video-title')).getText();
     const views = await video
       .findElement(By.xpath(".//*[@id='metadata-line']/span[1]"))
       .getText();
     const date = await video
       .findElement(By.xpath(".//*[@id='metadata-line']/span[2]"))
       .getText();
     videoDetails.push({
       title: title ?? '',
       views: views ?? '',
       publishedDate: date ?? '',
     });
   }
 } catch (error) {
   console.log(error);
 }
 return videoDetails;
}</code></pre><p>In the code above, we are using both the <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://www.lambdatest.com/blog/complete-guide-for-using-xpath-in-selenium-with-examples/">XPath selectors</a> to accomplish our task of retrieving basic video information. This screenshot below shows you how to retrieve the XPath of a particular element in a webpage.</p><img src="https://cdn-images-1.medium.com/max/1600/1*n6E8VbyIV3kkHz92w9Dglw.png"><p>Secondly, We are also using the <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://www.lambdatest.com/blog/making-the-move-with-id-locator-in-selenium-webdriver/">ID selector</a> to retrieve the video title. This screenshot shows where to get the ID for the video title element on YouTube using the Inspector tool:</p><img src="https://cdn-images-1.medium.com/max/1600/1*-wrAuDjzjgGtNbja_l7-gg.png"><h3 class="heading" levels="1,2,3,4,5,6">Display the JSON&nbsp;data</h3><p>After successfully performing YouTube web scraping with JavaScript and Selenium, we convert them into JSON and send them as a response to our request using the Express server. Finally, we are ready to test the new web scrapper.</p><ol><li><p>Start a new development server using the following command:</p></li></ol><pre><code>node index.js</code></pre><ol><li><p>Open the Chrome browser and visit localhost:3002, assuming you use the same port number from the demo.</p></li><li><p>You should be greeted with JSON data of the YouTube videos after a few seconds, like the screenshot below:</p></li></ol><img src="https://cdn-images-1.medium.com/max/1600/1*tn4mwhoyCk89MzuPndaUVA.png"><p>A screenshot showing the result of web scraping with JavaScript and Selenium on the LambdaTest YouTube videos page.</p><h3 class="heading" levels="1,2,3,4,5,6">Web Scraping with JavaScript and Selenium on cloud-based Selenium&nbsp;Grid</h3><p>In the previous section, we performed web scraping with JavaScript and Selenium on our local machine. One of the shortcomings of this approach is that you need to install and configure the Browser drivers locally.</p><p>Assuming you need to use multiple browsers to perform that automation, you must also manually install and configure each browser. This is where a cloud-based grid can be super impactful, as you can perform <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://www.lambdatest.com/javascript-automation-testing">JavaScript automation testing</a> in parallel on various browser and platform combinations.</p><p><a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://www.lambdatest.com/blog/benefits-of-cloud-testing/">Cloud testing</a> platforms like LambdaTest allow you to perform <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://www.lambdatest.com/live-testing">live testing</a> and run parallel tests on 3000+ real browsers and their different versions. It provides you access to thousands of browsers for mobile and <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://www.lambdatest.com/web-testing">web testing</a> to help you gain the maximum test coverage during the process of Selenium web automation.<br>&nbsp;Let’s look at how to run the Selenium automation test with JavaScript on LambdaTest:</p><p><strong>Step 1:</strong> Let’s start by creating a free account on the LambdaTest platform. Once you log in, fetch your unique username and access key from the LambdaTest Profile Section of the website.</p><p><strong>Step 2: </strong>Make the following configuration changes to your existing code:<br>&nbsp;Provide username and access key.</p><ul><li><p>Provide username and access key.</p></li></ul><pre><code>
const USERNAME = 'YOUR_USERNAME'; //replace with your username
const KEY = 'YOUR_KEY'; //replace with your accesskey</code></pre><ul><li><p>Provide the host corresponding to the LambdaTest platform.</p></li></ul><pre><code>const GRID_URL = 'hub.lambdatest.com/wd/hub';</code></pre><ul><li><p>Generate the desired browser capabilities using <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--li-anchor" href="https://www.lambdatest.com/capabilities-generator/">LambdaTest Capabilities Generator</a>.</p></li></ul><pre><code>const capabilities = {
     build: 'Web Scraping with JavaScript and Selenium',
     name: 'Youtube Videos',
     platform: 'Windows 10',
     browserName: 'Chrome',
     version: '90.0',
     selenium_version: '3.13.0',
     'chrome.driver': '90.0',
    };</code></pre><ul><li><p>Provide the gridUrl. You can also find this value from the <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--li-anchor" href="https://accounts.lambdatest.com/dashboard?fp_ref=solomon26">LambdaTest Dashboard</a>.</p></li></ul><pre><code>const gridUrl = 'https://' + USERNAME + ':' + KEY + '@' + GRID_URL;</code></pre><ul><li><p>Finally, edit the code to build and instantiate the browser on the designated platform.</p></li></ul><pre><code>
    async function WebScrapingWithLambdaTest() {
     try {
       driver = await new Builder()
         .usingServer(gridUrl)
         .withCapabilities(capabilities)
         .build();
       await driver.get('https://www.youtube.com/c/LambdaTest/videos');
       const allVideos = await driver.findElements(
         By.css('ytd-grid-video-renderer.style-scope.ytd-grid-renderer')
       );
       return await getVideos(allVideos);
     } catch (error) {
       throw new Error(error);
     } finally {
       await driver.quit();
     }
    }</code></pre><p>Here is the final implementation of Selenium automation with JavaScript that will be run on LambdaTest’s cloud-based Selenium Grid:</p>
<pre><code>app.get('/lambdatest', async (request, response) =&gt; {
     try {
       const data = await WebScrapingWithLambdaTest();
       response.status(200).json(data);
     } catch (error) {
       response.status(500).json({
         message: 'Server error occurred',
       });
     }
    });
    const USERNAME = 'YOUR_USERNAME'; //replace with your username
    const KEY = 'YOUR_KEY'; //replace with your accesskey
    const GRID_URL = 'hub.lambdatest.com/wd/hub';
    const capabilities = {
     build: 'JavaScript and Selenium Testing',
     name: 'Google search',
     platform: 'Windows 10',
     geoLocation: 'US',
     browserName: 'Chrome',
     version: '90.0',
     selenium_version: '3.13.0',
     'chrome.driver': '90.0',
    };
     
    const gridUrl = 'https://' + USERNAME + ':' + KEY + '@' + GRID_URL;
     
    async function WebScrapingWithLambdaTest() {
     try {
       driver = await new Builder()
         .usingServer(gridUrl)
         .withCapabilities(capabilities)
         .build();
       await driver.get('https://www.youtube.com/c/LambdaTest/videos');
     
       const allVideos = await driver.findElements(
         By.css('ytd-grid-video-renderer.style-scope.ytd-grid-renderer')
       );
       return await getVideos(allVideos);
     } catch (error) {
       throw new Error(error);
     } finally {
       await driver.quit();
     }
    }
</code></pre>

<p><strong>Step 3:</strong> Finally, it’s time to run the test on the LambdaTest grid. Trigger the following command on the terminal to run the <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://www.lambdatest.com/blog/automation-testing-with-selenium-javascript/">automation testing with Selenium JavaScript</a>.</p><pre><code>node index.js</code></pre><p>Next, visit <code>localhost:3000/lambdatest</code>. You should have the same result as before. If you check your LambdaTest Dashboard under Automation, you will see the execution snapshot, which indicates that the test execution was successful.</p><img src="https://cdn-images-1.medium.com/max/1600/1*809ePzrdcFbji8lhBZbFGQ.png"><p>You can also access the <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://analytics.lambdatest.com/test-overview">LambdaTest Analytics Dashboard</a>, which provides you with a complete overview of all your test results and a clear picture of their status over time.</p><img src="https://cdn-images-1.medium.com/max/1600/1*D-mwRfy-PJpjcBqAubAXGg.gif"><h3 class="heading" levels="1,2,3,4,5,6">Drawbacks of using Selenium for Web Scraping with JavaScript</h3><p>In this blog, we have seen how to perform web scraping with JavaScript and Selenium. However, there are some drawbacks to using any particular technology (or tool/framework), and Selenium is not exempted. Below are some of the drawbacks of using Selenium for web scraping.</p><ul><li><p>When using Selenium for web scraping, it can automatically download unnecessary files. For instance, if you’re only interested in the text data of a webpage, Selenium will also download the assets (i.e., CSS, HTML, JS, and other files) needed to render the web page.</p></li><li><p>It consumes time and resources when using Selenium WebDriver because it downloads unnecessary files to render the web page.</p></li><li><p>Web Scraping with Selenium is slow since it waits for the entire page to be loaded before you can start accessing data. To learn more about waits, you can read through this blog on <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--li-anchor" href="https://www.lambdatest.com/blog/types-of-waits-in-selenium/">types of waits in Selenium</a>.</p></li></ul><p>Watch this video to learn what are waits in Selenium and how to handle them using different methods like hard-coded pauses and by combining explicit waits with different design patterns.</p><p>So far, we have detailed a few drawbacks you need to look out for when web scraping with JavaScript and Selenium. Let’s also explore the limitless possibilities and benefits of using Selenium for web scraping. You can also identify obscure issues in the website with the aid of the <a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://www.lambdatest.com/visual-testing-tools-selenium">visual testing tool Selenium</a>.</p><h3 class="heading" levels="1,2,3,4,5,6">Summary</h3><p>Web scraping is an important technique to extract data. It eliminates the manual process of extracting public data. Web scraping is also very accurate and less time-consuming. It can also be configured to retrieve dynamic websites using Selenium and JavaScript.</p><p>In this blog on web scraping with JavaScript and Selenium, we have explored the benefits of web scraping, which can not be overemphasized, the different web scraping tools, and Selenium which stands as the best tool for automated testing and dynamic web scraping tools.</p><p>We scraped all the content of the official LambdaTest YouTube channel and converted them into JSON, automatically extracting the titles and authors to generate an author matrix, among other useful insights.</p><hr><p><em>Originally published at </em><a target="_blank" rel="noopener noreferrer nofollow" class="markup--anchor markup--p-anchor" href="https://www.lambdatest.com/blog/web-scraping-with-javascript-and-selenium/"><em>https://www.lambdatest.com</em></a><em> on July 18, 2022.</em></p>