Web Scraping with Python Tutorial - A Complete Guide with Examples
Tags: python, tutorial, beginners, programming
Published: 2023-11-22 07:41:14


We live in an era where we are surrounded by data that can be harnessed by extracting meaningful insights from it. As quoted by Tim Berners-Lee, inventor of the World Wide Web — “*Data is a precious thing and will last longer than the systems themselves*.”

If the data is the new oil, web scraping (or web harvesting) is the expeller that helps squeeze more oil.🙂Web scraping can be leveraged for analyzing and deriving actionable insights from tons of information available on the internet.

Irrespective of the business domain (e.g., eCommerce, EdTech, Fintech, etc.), scraping can be used for market research, pricing intelligence, lead generation, and sentiment analysis, to name a few! Though web scraping is immensely useful, it comes with a caveat — *Scraping should be done legally and ethically, respecting the website’s T&C and data privacy regulations*.

Popular programming languages like Python, Java, JavaScript, etc., are well-equipped with tools and libraries that can ease the job of web scraping. In this blog, I will limit the discussion of web scraping with Python — a popular programming language for [automated testing](https://www.lambdatest.com/learning-hub/automation-testing?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=learning_hub). You can check my earlier blog on [Python automation testing](https://www.lambdatest.com/blog/python-automation-testing/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog), highlighting why Python is the best-suited language for [automation testing](https://www.lambdatest.com/automation-testing?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=webpage).

By the end of this blog on web scraping with Python, you will learn to scrap static and dynamic websites using the best Python tools (or libraries) like PyUnit, pytest, and Beautiful Soup. The actionable examples will help you harness the capabilities of web scraping with Python to extract meaningful insights from websites.

Without further ado, let’s get started…

***Note: **Scrap & Scrape and document & page are used interchangeably throughout the blog.*
>  **[Text Uppercase](https://www.lambdatest.com/free-online-tools/text-uppercase?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=free_online_tools) is a free online tool for converting text into uppercase. Enter your text and click the convert button to make all lowercase letters upper case.**

## What is Web Scraping?

To set the ball rolling, let’s do a quick recap of the *What* & *Why* of web scraping. In simple terms, web scraping, or web harvesting (or data extraction), is the technique for deriving information (*that matters*) from websites.

Shown below is the simplistic representation of web scraping that shows that input is a website that needs to be scrapped by the scraping logic.

![](https://cdn-images-1.medium.com/max/2000/0*n6QnUk1zqWGYuVsP.png)

*Web Scraping Architecture*

Once the information from the HTML of the page is scraped, the raw data can be stored in a more presentable (or readable) format in Excel sheets, databases, etc.

With scraping options in the arsenal, it becomes imperative to opt for efficient and responsible scraping! In the examples presented later in this blog on web scraping with Python, the scraped information will be presented on the terminal.

## Prominent Use Cases of Web Scraping

Most new-age websites leverage JavaScript for loading and updating content on the page. AJAX (Asynchronous JavaScript and XML) is used to retrieve data from the server and update relevant parts of the page (without a full page reload).

For instance, videos on the [LambdaTest YouTube channel](https://www.youtube.com/@LambdaTest/videos?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=youtube) are dynamically loaded when the user performs a scroll operation. As seen in the below screenshots, the URL remains unchanged, but the videos are loaded on a dynamic basis.

![](https://cdn-images-1.medium.com/max/3200/0*QszbUKuavRA2fau8.png)

![](https://cdn-images-1.medium.com/max/3200/0*w_OzabPSmjk_6H4a.png)

*Example — Dynamic Website Content*

Popular Python frameworks like pytest and PyUnit, in conjunction with Selenium, can be used for dynamic web scraping websites (and SPAs) like YouTube, Netflix, [LambdaTest eCommerce Playground](https://ecommerce-playground.lambdatest.io/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=webpage), and more.

It goes without saying that scraping can be done on static websites where the entire HTML page content is downloaded and parsed to extract the desired information. You can check out my earlier blog highlighting the [difference between Static and Dynamic Web Scraping](https://www.lambdatest.com/blog/scraping-dynamic-web-pages/#staticdynamicdifference?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog).

Now that we know that it is possible to scrap static and dynamic websites, let’s look at some of the primary use cases of web scraping:

## Competitor Analysis

Irrespective of the size (or scale) of the business, businesses must keep a close watch on their competition. Insights into competitor’s products and services can be instrumental in having an upper edge over the competition.

Web scraping can effectively scrape relevant information (e.g., products, services, pricing, etc.) from competitors’ websites. The scraped data can be leveraged for tweaking product & pricing-specific information on the website.

## Lead Generation

Web scraping can gather details (e.g., names, phone numbers, email addresses, etc.) from websites to generate leads for your business.

The scraped data is then fed to CRM software, so the sales team can reach out to those leads.
>  **Rotate your text and add some creativity to your words with our [Text Rotater](https://www.lambdatest.com/free-online-tools/text-rotater?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=free_online_tools) tool. Try it out now and see the results in just a few clicks!**

## Data Analysis

Many websites have products that span across different categories. For instance, a horizontal eCommerce website might have products across electronics, apparel, baby care, etc.

Web scraper is instrumental in extracting product metadata, seller details, number of SKUs, etc., from such websites. Whether it is a SPA (Single Page Application) or paginated content, relevant web scraping tools & libraries let you scrape information from them.

Apart from the use cases mentioned above, you can also use web scraping for academic research and training data for ML projects!

## Python Libraries and Tools for Web Scraping

I have also tried [web scraping using C#](https://www.lambdatest.com/blog/scraping-dynamic-web-pages/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog), but the web scraping ecosystem is nothing like Python. One of the major benefits of using Python is the availability of many tools and libraries for web scraping.

It is recommended to perform the package installation in a virtual environment (venv) to isolate it from the packages in the base environment. Run the commands *virtualenv venv* and *source venv/bin/activate* on the terminal to create the virtual environment.

Though I will be covering a few libraries (or tools) using a detailed demonstration, let’s list some of the popular ones below:

## Selenium

For starters, [Selenium](https://www.lambdatest.com/selenium?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=webpage) is a popular open-source framework that helps with web browser automation. The tools and libraries offered by Selenium are instrumental in automating interactions with the elements in the document (or page). When writing this blog on web scraping with Python, the latest version was 4.14.0.

Selenium 3, the earlier version of Selenium, used the JSON-Wire protocol. On the other hand, Selenium 4 uses the W3C protocol because automated tests are less flaky (or more stable) when compared to tests implemented using Selenium 3. For a quick recap, you can check out our blog highlighting the [differences between Selenium 3 and Selenium 4](https://www.lambdatest.com/blog/selenium-3-vs-selenium-4/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog).

Most modern-day web pages use AJAX & JavaScript to load and display dynamic content. Consider [LambdaTest eCommerce playground](https://ecommerce-playground.lambdatest.io/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=webpage). The content is loaded when the user scrolls the page. As seen in the screenshot below, the images are lazy loaded and appear in the DOM once the user scrolls the page.

![](https://cdn-images-1.medium.com/max/3200/0*xJFKheutZqozUgLj.png)

![](https://cdn-images-1.medium.com/max/3200/0*GsqBE7OtgWP3b13S.png)

*Example of Lazy Loading of Images*

This is where Selenium proves to be an instrumental option for web scraping due to its ability to handle dynamically loaded content using JavaScript.

In the case of Python, [popular test automation frameworks](https://www.lambdatest.com/blog/best-test-automation-frameworks/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog) like pytest and PyUnit can be leveraged for scraping static and dynamic page content. Run the command *pip install selenium *(or *pip3 install selenium*) depending on the Python version installed in the machine.

PyUnit (or unittest ) is the default test automation framework, part of the Python standard library.

![](https://cdn-images-1.medium.com/max/3200/0*DR_pR_rHrrJaGuyU.png)

PyUnit is inspired by the [xUnit framework in C#](https://www.lambdatest.com/blog/xunit-testing-tutorial/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog) and is also compatible with older versions of Python. For pytest, check out our zero-to-hero [Selenium pytest tutorial ](https://www.lambdatest.com/learning-hub/selenium-pytest-tutorial?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=learning_hub)for a quick recap on running pytest with Selenium.

Run the command *pip install pytest* (or *pip3 install pytest*) to install the pytest framework on the machine.

![](https://cdn-images-1.medium.com/max/3200/0*QRLulTUo51tLwVLB.png)

[Explicit waits in Selenium](https://www.lambdatest.com/blog/selenium-webdriverwait/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog) will be leveraged for complex scenarios like waiting for the content to load, navigating through multiple pages, and more. In further sections of this blog on web scraping with Python, I will be using appropriate [locators in Selenium](https://www.lambdatest.com/blog/locators-in-selenium-webdriver-with-examples/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog) along with *find_element()* / *find_elements()* methods to scrape content on the document (or web page).
>  **Need to know how many characters are in your text? Get an accurate count of the characters in your text with our free online [character count](https://www.lambdatest.com/free-online-tools/character-count?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=free_online_tools) tool. Try it now and see how easy it is!**

## Beautiful Soup

Beautiful Soup is a popular Python library primarily built for web scraping. As specified in the official documentation, Beautiful Soup can navigate and parse through HTML & XML documents. Unlike Selenium, which can be used for static and dynamic web scraping, Beautiful Soup is apt for static web scraping with Python.

It supports unit test discovery using pytest. When writing this blog on web scraping with Python, the latest version of Beautiful Soup is 4.12.1. Hence, the library is also referred to as BS4. In case you are using BS3, simply changing the package name from *BeautifulSoup* to *bs4* will help in porting the code to BS4. For more details, you can refer to the official BS4 porting guide.

BS4 uses the *html.parser* module by default, which can be swapped out with *lxml* or *html5lib* libraries in Python. Web elements on the HTML page can be located using the CSS Selector and XPath locators. Akin to the [*find_element() *method](https://www.lambdatest.com/blog/findelement-and-findelements-in-selenium/) in Selenium Python, the *find()* method in BS4 returns the element located using the appropriate selector.

Similarly, the *find_all()* method in BS4 (like the *find_elements()* method in Selenium) scans through the entire page and returns a list of all the descendants that match the filters. You can refer to the official documentation of Beautiful Soup (or BS4) to get insights into all the methods provided by the library.

Before using BS4 for ***static web scraping***, the built-in *Requests* library makes HTTP requests. Text (or text) that contains the HTML content from the HTTP response object is then subjected to HTML parsing (or *html.parser*) using BS4. Once parsed, the Beautiful Soup object can be used for navigating & searching through the document’s structure.

BS4 can also be used with Selenium to give wings to ***dynamic web scraping***. In this case, BS4 is used to parse and extract data from the loaded (or rendered) HTML page, whereas Selenium is used to navigate to the relevant pages.

Run the command *pip3 install beautifulsoup4* or *pip3 install bs4* on the terminal to install Beautiful Soup 4 (or BS4). As seen below, the Beautiful Soup 4.12.2 installation was successful.

![](https://cdn-images-1.medium.com/max/3200/0*z1cBIC43dOggEDBw.png)

In further sections of the blog, I will demonstrate the usage of Beautiful Soup and Selenium for scraping content on various pages (1 through 5) on the [LambdaTest eCommerce playground](https://ecommerce-playground.lambdatest.io/index.php?route=product/category&path=57&page=1?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=webpage).

## Playwright

[Playwright](https://www.lambdatest.com/playwright?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=webpage) is another popular test automation framework suited for web scraping. It is primarily used for [end-to–end testing](https://www.lambdatest.com/learning-hub/end-to-end-testing?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=learning_hub) of modern web applications. Like Selenium, it also provides APIs that let you perform advanced interactions on the page using headless versions of Chrome, Chromium-based browsers, and WebKit.

Run the command *pip3 install playwright* (or *pip install playwright*) to install Playwright for Python on the machine.

![](https://cdn-images-1.medium.com/max/3200/0*OTCbK1WbmhPQKxwy.png)

When writing this blog on web scraping with Python, the latest version of Playwright is 1.39.0. Auto-waiting in Playwright ensures that waits on elements are performed before they become actionable. This eliminates the need to add artificial timeouts, making the scraping code much more maintainable & readable.

Over and above, Playwright plugins like Pyppeteer (Python port of Puppeteer) and scrapy-playwright can also be leveraged for static and dynamic web scraping with Python.

## Scrapy

Scrapy is a full-fledged web crawling and scraping framework that can be used to crawl websites to extract structured data from the page(s). Scrapy can be leveraged for data mining, monitoring, and automated testing.

When writing this blog, the latest version of Scrapy is 2.11.0. Run the command *pip3 install scrapy* (or *pip install scrapy*) on the terminal to install Scrapy.

![](https://cdn-images-1.medium.com/max/3200/0*5-BgV-YUTzaAGmXT.png)

![](https://cdn-images-1.medium.com/max/3200/0*_Uglz__5dzw7L0lU.png)

![](https://cdn-images-1.medium.com/max/3200/0*Qj1px4xGSuYQkePv.png)

As stated in the Scrapy documentation, Scrapy can extract data using APIs like Amazon Web Services (AWS). Though Scrapy is mainly used for static web scraping, it can also be used for scraping dynamic web pages (e.g. QuotestoScrape JS).

Similar to Beautiful Soup, Scrapy along with frameworks like Selenium or Playwright, can be instrumental in scraping dynamic web content.

![](https://cdn-images-1.medium.com/max/3200/0*CSjlBkTW95ZpJMiw.png)

In such cases, the *scrapy-playwright* plugin is handy for scraping dynamic web content using Playwright and Scrapy.

PyQuery, lxml, and Mechanical Soup are other popular libraries for scraping with Python.

In the further sections of the blog on web scraping with Python, the discussion & demonstration will be limited to the following libraries (or tools):

* Web scraping using Selenium PyUnit

* Web scraping using Selenium pytest

* Web scraping using Beautiful Soup

The learnings from the demonstration will be useful for creating sample code for scraping with other Python-based tools. So, let’s get our hands dirty with some code. 🧑‍💻
>  **[Sentence Count](https://www.lambdatest.com/free-online-tools/sentence-count?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=free_online_tools) is a free, easy-to-use tool for counting sentences in text. Simply upload your text, click Count and you’ll get the number of sentences in seconds!**

## Demonstration: Web Scraping With Python

For demonstrating web scraping using Selenium PyUnit (& pytest) and Beautiful Soup, I will be using the following websites:

![](https://cdn-images-1.medium.com/max/2000/1*89rHuqEnOuYqldEmPj5UNg.png)

## Directory Structure

As seen in the project structure, scraping using PyUnit, pytest, and Beautiful Soup is driven via a *Makefile*. Let’s look at the important directories and files in more detail below:

![](https://cdn-images-1.medium.com/max/2000/0*jBtAb4hR_-al5PIa.png)

Here is a closer look at the overall structure:

* **pageobject**

[Page Object Model](https://www.lambdatest.com/blog/page-object-model-in-selenium-python/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog) (POM) design pattern is used for separate web locators from the core test logic. *locators.py* contains details of locators used for interacting with the test websites mentioned earlier.

![](https://cdn-images-1.medium.com/max/3200/0*1FU3Ya8laU6TBzTS.png)

*helpers.py* contains the implementation of the wrapper functions for triggering actions, performing [explicit waits](https://www.lambdatest.com/blog/selenium-wait-for-page-to-load/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog), and more. Since the use cases for scraping with PyUnit and pytest are the same, two wrapper functions are created for reduced code duplication and improved maintenance:

![](https://cdn-images-1.medium.com/max/2000/1*bQVB6VuuuB9dk1JiRem0RA.png)

Both the functions return a list where the scraped information is stored for further operations.

![](https://cdn-images-1.medium.com/max/3200/0*cLb8YDzc1OciOiaU.png)

* **tests**

As the name suggests, the tests folder contains the implementation of the core test methods that are responsible for scraping the content using the respective scraping library (or tool).

To keep the code more modular and maintainable, we have created subfolders (or sub-directories) for the libraries used for scraping the test websites:

![](https://cdn-images-1.medium.com/max/2000/1*onXisAIC3_dYgnYmQCggYQ.png)

We will cover more about the implementation in the code walkthrough section of the blog.

* **Configuration (or setup) files**

The conftest.py file in pytest helps to fixture and share fixtures, hooks, and other configuration options used in the test code. For demonstration, scraping with Selenium and/or Beautiful Soup is performed using browsers on the local machine and cloud grid like LambdaTest.
LambdaTest is an AI-powered test orchestration and execution platform that lets you perform automated testing using Selenium Python on an [online browser farm](https://www.lambdatest.com/online-browser-farm?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=webpage) of 3000+ real browsers and operating systems.
{% youtube SqQ8SugRDos %}
Catch up on the latest tutorials around [Selenium automation testing](https://www.lambdatest.com/selenium?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=webpage), [Appium automation](https://www.lambdatest.com/appium-mobile-testing?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=webpage), and more. Subscribe to the [LambdaTest YouTube Channel](https://www.youtube.com/c/LambdaTest?sub_confirmation=1?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=youtube).
>  **Want to decode a messy URL? Unleash the full potential of your website with our online [URL Parse](https://www.lambdatest.com/free-online-tools/url-parse?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=free_online_tools) tool. Optimize your website’s URLs to improve your site’s visibility.**

![](https://cdn-images-1.medium.com/max/2000/0*TNvA-F4TKBnL5Qcu.png)

The scope of the [pytest fixture](https://www.lambdatest.com/blog/end-to-end-tutorial-for-pytest-fixtures-with-examples/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog) (i.e., *@pytest.fixture*) is set to *function*, which means that the fixture is set up and torn down before & after executing a test function (or method). As seen below, execution is controlled using the environment variable *EXEC_PLATFORM*, which can be set to *local *or *cloud*.

![](https://cdn-images-1.medium.com/max/3200/0*ZvXoEPhAhF3ovXMs.png)

Username (*LT_USERNAME*) and Access Key (*LT_ACCESS_KEY*) for using the [cloud Selenium Grid ](https://www.lambdatest.com/selenium-grid-online?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=webpage)on LambdaTest can be obtained by navigating to [LambdaTest Profile Page](https://accounts.lambdatest.com/security?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=webpage). Set them using the *export* (for macOS and Linux) or *set* (for Windows) command to export the environment variables.

![](https://cdn-images-1.medium.com/max/3200/0*tyoQjIBsacp5iGoS.png)

In the *teardown* function, the *quit()* method is invoked to terminate the browser session. The [JavaScriptExecutor in Selenium](https://www.lambdatest.com/blog/how-to-use-javascriptexecutor-in-selenium-webdriver/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog) [i.e., *execute_script()*] is used to update the *lambda-status* variable, which indicates the test execution status on LambdaTest Selenium Grid.

![](https://cdn-images-1.medium.com/max/2312/0*Arm17vzlxUrY5vIT.png)

Similarly, we have created puynitsetup.py that contains configurations related to the PyUnit (or unittest ) framework. Implementation under the *__init__()* method is used for initializing the test cases.



![](https://cdn-images-1.medium.com/max/2480/0*86F9M2nd1vSQLXoA.png)

Irrespective of whether the tests are executed on a local grid or cloud Selenium grid, the browser is initialized in headless mode. If the environment variable *EXEC_PLATFORM* is set to the *cloud*, Chrome browser on the LambdaTest grid is instantiated in headless mode.

![](https://cdn-images-1.medium.com/max/2000/0*Bz7VVtWa5tqEYMh_.png)

The said capability (i.e., *headless*), along with the other capabilities, can be set using the [LambdaTest Capabilities Generator](https://www.lambdatest.com/capabilities-generator/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=webpage).

![](https://cdn-images-1.medium.com/max/3200/0*AcK5SQq2Tjfx9BFv.png)



As stated in the official Selenium documentation, Chrome is instantiated in the headless mode by adding *–headless=new* to Chrome options.

![](https://cdn-images-1.medium.com/max/3200/0*pCV8VB4AHk4njrtF.png)

Lastly, the *setUp()* and *tearDown()* methods are implemented to set up and tear down the test environment. In fact, *__init__()* is optional, and implementation under it can be moved under the *setUp()* method.

![](https://cdn-images-1.medium.com/max/2000/0*N_aC094ee1lvI4cW.png)

* **Makefile**

As stated earlier, scraping using the respective libraries (or tools) is controlled via a Makefile. Typing *make help* provide all the options available for execution.

![](https://cdn-images-1.medium.com/max/2372/0*27s2UV7DQC4cF0SX.png)

The *make install* command will install the packages/libraries specified in requirements.txt.

    pytest-selenium
    selenium>=4.6.0
    urllib3==1.26.12
    requests
    py
    bs4

![](https://cdn-images-1.medium.com/max/2000/1*ttUO-UmFeUWbYTqlhvhk4g.png)

## Pre Requisites

First and foremost, trigger *virtualenv venv* and *source venv/bin/activate* on the terminal to create the virtual environment.

![](https://cdn-images-1.medium.com/max/3200/0*SfMCUVTsZGZx1ZFc.png)

Since there is a provision to perform scraping on the cloud Selenium grid, it is recommended to have an account on LambdaTest. To scrap websites using Selenium on LambdaTest Grid, you must update LT_USERNAME and LT_ACCESS_KEY (in Makefile) from the [LambdaTest Profile Page](https://accounts.lambdatest.com/security?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=webpage).

![](https://cdn-images-1.medium.com/max/3200/0*7nn0Yhv5g2yLuU1p.png)

Install the required frameworks and libraries (mentioned in *requirements.txt*) by triggering the *make install* command on the terminal.

![](https://cdn-images-1.medium.com/max/3200/0*AP8L_kZrpdi8eLHS.png)

With this, the stage is finally set to scrap the test websites. Let’s dive in!

Since the scraping logic with Selenium & Python remains the same for PyUnit and pytest, we have combined both frameworks in this section. To begin with, we look into the configuration aspects of the framework, after which we would look into the common scraping logic.
>  **Need to [extract text from HTML](https://www.lambdatest.com/free-online-tools/extract-text-from-html?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=free_online_tools) code? Our free online tool makes it easy and fast. Save time by extracting only the text you need from HTML code in no time.**

## Web Scraping using Selenium and PyUnit

For simplification, all setup-related code (i.e., instantiating the browser, setting explicit timeouts, etc.) is in *pyunitsetup.py*. Essential aspects of the setup are already discussed in the earlier section of this blog on web scraping with Python.

![](https://cdn-images-1.medium.com/max/2000/0*XWEjNljlMh-1m_4w.png)

Let’s deep dive into some of the integral parts of the configuration file pyunitsetup.py

    import sys
    import os
    from os import environ
    sys.path.append(sys.path[0] + "/../..")
    from pageobject.locators import locators
    from pageobject.locators import *
    
    exec_platform = os.getenv('EXEC_PLATFORM')
    
    class pyunit_setup:    
        def __init__(self):
            if exec_platform == 'cloud':
                username = environ.get('LT_USERNAME', None)
                access_key = environ.get('LT_ACCESS_KEY', None)
    
                ch_options = webdriver.ChromeOptions()
    
                lt_options = {}
                lt_options["build"] = "Build: Web Scraping with Selenium & Pyunit"
                lt_options["project"] = "Project: Web Scraping with Selenium & Pyunit"
                lt_options["name"] = "Test: Web Scraping with Selenium & Pyunit"
    
                lt_options["browserName"] = "Chrome"
                lt_options["browserVersion"] = "latest"
                lt_options["platformName"] = "Windows 11"
    
                lt_options["console"] = "error"
                lt_options["w3c"] = True
                lt_options["headless"] = True
    
                ch_options.set_capability('LT:Options', lt_options)
    
                gridURL = "@hub.lambdatest.com/wd/hub">https://{}:{}@hub.lambdatest.com/wd/hub".format(username, access_key)
                self.browser = webdriver.Remote(
                    command_executor = gridURL,
                    options = ch_options
                )
            elif exec_platform == 'local':
                ch_options = ChromeOptions()
                ch_options.add_argument("--headless=new")
                self.browser = webdriver.Chrome(options=ch_options)
      
        def setUp(self):
            self.browser.implicitly_wait(10)
            self.browser.maximize_window()
    
        def tearDown(self):
            if (self.browser != None):
                self.browser.quit()

The value of the *EXEC_PLATFORM* environment variable (i.e., *cloud* or *local*) decides whether the instantiation of the browser is on the local machine or cloud grid on LambdaTest.

![](https://cdn-images-1.medium.com/max/2480/0*tbOCeJryVb6TdqQD.png)

[Remote WebDriver](https://www.lambdatest.com/blog/selenium-remotewebdriver/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog) with grid URL & browser options as parameters are instantiated using the *webdriver.Remote() *method. Capabilities (or browser options) can be obtained from [LambdaTest Capabilities Generator](https://www.lambdatest.com/capabilities-generator/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=webpage).

![](https://cdn-images-1.medium.com/max/3200/0*wd2swgcaojaVN2JB.png)

Chrome in headless mode is instantiated as we do not require GUI for web scraping with Python. The *w3c* flag is set to true since we are using Selenium 4 (which is W3C compliant) for the tests.

![](https://cdn-images-1.medium.com/max/2000/0*SvUDJwzFZtS3tbqx.png)

When using a local grid (or machine), we set the *–headless=new* argument for Chrome Options.

![](https://cdn-images-1.medium.com/max/2000/0*sWFj3rTye8Wvuju1.png)

Finally, the *setUp() *and *tearDown() *methods contain the implementation of maximizing the browser and releasing the resources post text execution respectively.

![](https://cdn-images-1.medium.com/max/2000/0*5Wzw_x2dmGBe5Qph.png)
>  **Perform browser [test automation](https://www.lambdatest.com/automation-testing?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=free_online_tools) on the most powerful cloud infrastructure. Leverage LambdaTest automation testing for faster, reliable and scalable experience on cloud.**

## Web Scraping using Selenium and pytest

Since the demo websites scraped with PyUnit and pytest are the same, there are changes only in the setup-related code. Like web scraping with PyUnit, all setup-related code (i.e., instantiating the browser, setting explicit timeouts, etc.) is segregated in conftest.py.

For starters, *conftest.py* is a special file in pytest that contains the configuration of the test suite. Hence, hooks, fixtures, and other configurations are all a part of *conftest.py*

![](https://cdn-images-1.medium.com/max/2000/0*zF-BQqsxi8COe3UA.png)

Let’s deep dive into some of the integral parts of the configuration file conftest.py

    # Import the locators file
    import sys
    import os
    from os import environ
    sys.path.append(sys.path[0] + "/../..")
    from pageobject.locators import locators
    from pageobject.locators import *
    
    exec_platform = os.getenv('EXEC_PLATFORM')
    
    @pytest.fixture(scope='function')
    def driver(request):
        if exec_platform == 'cloud':
            username = environ.get('LT_USERNAME', None)
            access_key = environ.get('LT_ACCESS_KEY', None)
    
            gridURL = "@hub.lambdatest.com/wd/hub">https://{}:{}@hub.lambdatest.com/wd/hub".format(username, access_key)
            
            ch_options = webdriver.FirefoxOptions()
            ch_options.browser_version = "latest"
            ch_options.platform_name = "Windows 11"
    
            lt_options = {}
            lt_options["build"] = "Build: Web Scraping with Selenium & Pytest"
            lt_options["project"] = "Project: Web Scraping with Selenium & Pytest"
            lt_options["name"] = "Test: Web Scraping with Selenium & Pytest"
    
            lt_options["browserName"] = "Firefox"
            lt_options["browserVersion"] = "latest"
            lt_options["platformName"] = "Windows 11"
    
            lt_options["console"] = "error"
            lt_options["w3c"] = True
            lt_options["headless"] = True
    
            ch_options.set_capability('LT:Options', lt_options)
    
            browser = webdriver.Remote(
                command_executor = gridURL,
                options = ch_options
            )
    
            yield browser
    
            def fin():
                if request.node.rep_call.failed:
                    browser.execute_script("lambda-status=failed")
                else:
                    browser.execute_script("lambda-status=passed")
                    browser.quit()
    
            request.addfinalizer(fin)
        elif exec_platform == 'local':
            options = ChromeOptions()
            options.add_argument("--headless=new")
            browser = webdriver.Chrome(options=options)
    
            yield browser
    
            def fin():
                browser.quit()
    
    @pytest.hookimpl(tryfirst=True, hookwrapper=True)
    def pytest_runtest_makereport(item, call):
        outcome = yield
        rep = outcome.get_result()
    
        setattr(item, "rep_" + rep.when, rep)

The *@pytest.fixture(scope=’function’)* decorator is used for defining a fixture with a *function* scope.

![](https://cdn-images-1.medium.com/max/2344/0*QYnmi9aeqJioSi0N.png)

Depending on the value of the *EXEC_PLATFORM* environment variable (i.e. *cloud* or *local*), the browser is instantiated on the local machine or cloud grid on LambdaTest.

![](https://cdn-images-1.medium.com/max/3200/0*hD-0wga4DMLSktWp.png)

In case you are planning to run the [Selenium Python tests](https://www.lambdatest.com/selenium-python-testing?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=webpage) (i.e., scrap website under test) on browsers in LambdaTest cloud, instantiate Remote WebDriver with grid URL & browser options as parameters to the *webdriver.Remote()* method. Capabilities (or browser options) can be obtained from LambdaTest Capabilities Generator.

![](https://cdn-images-1.medium.com/max/3200/0*qEvFgJRn2qndYIS4.png)

Since we are performing web scraping with Python, the browser (i.e., Chrome) is instantiated in the headless mode. [Headless Chrome](https://www.lambdatest.com/blog/headless-chrome/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog) is faster than the real browser (with the GUI). Hence, it is best suited for web scraping with Python. Since Selenium 4 (W3C compliant) is used for testing, the *w3c *flag is set to *true*.

![](https://cdn-images-1.medium.com/max/2000/0*Xg5zKoTDorklbBlB.png)

For scraping using Selenium on local headless browsers (e.g., Chrome) on a local grid (or machine), simply set the *–headless=new* argument for Chrome Options.

![](https://cdn-images-1.medium.com/max/2000/0*Bq-4_xsXnlhKqgIE.png)

Lastly, the *yield* statement provides the resource (i.e., *browser*) in the *setUp* and *tearDown* sections of the code. Hence, all the resources the browser uses will be cleared once the execution is complete.

![](https://cdn-images-1.medium.com/max/3200/0*_EIZRX7WO9-757sd.png)

With this, we are all set to dive deep into the scraping implementation!

### Test Scenario 1 — Scraping LambdaTest YouTube Channel

In this example, meta-data (i.e., title, views, duration, etc.) associated with [LambdaTest YouTube Videos](https://www.youtube.com/@LambdaTest/videos?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=youtube) is scraped using the PyUnit (or unittest framework).

![](https://cdn-images-1.medium.com/max/3200/0*xW79DZZkDzoSSN66.png)

Since the LambdaTest YouTube channel has more than 600+ videos, a vertical scroll (till the end of the page) is performed so that required information can be scraped from those videos!
>  **Need a great solution for Safari browser testing on Windows? Forget about emulators or simulators — use real online browsers for testing on [safari for windows](https://www.lambdatest.com/safari-browser-for-windows?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=webpage). Try LambdaTest for free!**

**Implementation (pyUnit)**

The file test_yt_scraping.py contains the test logic where the *scrap_youtube()* method scraps the channel. It returns a list that contains the metadata of the videos.

    # Import the locators file
    import sys
    sys.path.append(sys.path[0] + "/../..")
    from pageobject.locators import locators
    from pageobject.locators import *
    
    from pageobject.helpers import helpers
    from pageobject.helpers import *
    
    from pyunitsetup import pyunit_setup
    
    test_setup = pyunit_setup()
    
    def scrap_youtube(test_url) -> list:
        meta_data_arr=[]
        test_setup.setUp()
        test_setup.browser.get(test_url)
    
        # Click on 'Accept All' in case the said window comes up
        # This occurs since cookies are cleared and machines are sanitized each run
    
        # Stack Overflow - https://stackoverflow.com/questions/66902404/selenium-python-click-agree-to-youtube-cookie
        # Locators were located using the below link
        # https://consent.youtube.com/m?continue=https%3A%2F%2Fwww.youtube.com%2F&gl=FR&m=0&pc=yt&uxe=23983172&hl=en&src=1
    
        # Below issue is not observed in local machines since there is no clean-up of cookies
    
        try:
            # Wait until the "Accept all" button is clickable
            accept_all = WebDriverWait(test_setup.browser, 5).until(EC.element_to_be_clickable(
                (By.CSS_SELECTOR, "form:nth-child(3) > div > div > button[aria-label= 'Accept all']")))
    
            # Click the "Accept all" button
            accept_all.click()
    
            print("Click on Accept all successful")
        except Exception as e:
            # Even if this exception is raised, we can still continue test execution
            # This means that the button was not found & the YouTube channel link has opened successfully
            print(f"'Accept All' button not present, proceed with the tests: {str(e)}")
    
        meta_data_arr = helpers.scrap_yt_content(test_setup.browser)
    
        test_setup.tearDown()
        return meta_data_arr
    
    def main():
        meta_data_arr = scrap_youtube(locators.test_yt_url)
        helpers.print_scrapped_content(meta_data_arr)
    
    if __name__ == '__main__':
        main()

To get started, we create an object of the *pyunit_setup* class which contains implementation of *setUp()* and *tearDown()* methods.

![](https://cdn-images-1.medium.com/max/2000/0*-TCTig2-xDNIrWCQ.png)

The browser instantiation is done in the *__init__()* method implemented in *pyunitsetup.py*. An empty list named *meta_data_arr *is created, and the browser is navigated to the URL under test.

![](https://cdn-images-1.medium.com/max/2000/0*br8tX7NgdTkzmaRY.png)

Scraping YouTube content is legal as long as we comply with regulations that deal with personal data and copyright protection. After repeated scraping, we observed that YouTube showed the Consent Page, where we had to programmatically click the Accept all button to navigate to the LambdaTest YouTube channel.

![](https://cdn-images-1.medium.com/max/3200/0*atth7nsRv4o7mmD0.png)

Since the above consent page was shown intermittently, we used the *try…catch *loop to counter the above situation. We first locate the Accept all button using the [CSS Selector](https://www.lambdatest.com/learning-hub/css-selectors?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=learning_hub) *form:nth-child(3) > div > div > button[aria-label= ‘Accept all’] *Alternatively, we could have also located the element using the [XPath locator](https://www.lambdatest.com/blog/python-xpath/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog) *//div[@class=’csJmFc’]/form[2]//div[@class=’VfPpkd-RLmnJb’]*

![](https://cdn-images-1.medium.com/max/3200/0*69apY2_MSoDAn28r.png)

To locate elements with ease, we used the *POM Builder *that provides CSS Selector & XPath of elements at the click of a button. The [*WebDriverWait class](https://www.lambdatest.com/blog/webdriverwait-in-selenium-c-sharp/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog)* is used along with the *element_to_be_clickable [*expected condition in Selenium](https://www.lambdatest.com/blog/expected-conditions-in-selenium-examples/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog) to realize an explicit wait for 5 seconds.

Once the element is located, the [*click *method in Selenium](https://www.lambdatest.com/blog/selenium-click-button-with-examples/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog) is invoked to perform a click operation on the Click all button.

![](https://cdn-images-1.medium.com/max/3200/0*JDQTXM5U1MQJCK_b.png)

![](https://cdn-images-1.medium.com/max/3200/0*FL68NvvD-k808jmW.png)

In case the *YouTube Consent Page* does not show up, we simply handle the exception in the *expect* block and print the exception.

![](https://cdn-images-1.medium.com/max/3200/0*tSarDDWME8l08S33.png)

Now that we are on the LambdaTest YouTube Channel, we invoke the [*scrap_yt_content()](https://github.com/hjsblogger/web-scraping-with-python/blob/main/pageobject/helpers.py#L79)* method that is a part of the [*helpers](https://github.com/hjsblogger/web-scraping-with-python/blob/main/pageobject/helpers.py#L79) *class. We will look into its implementation in the further sections of the blog.

![](https://cdn-images-1.medium.com/max/2512/0*WevKeaYPA-5IT53r.png)

Upon successful scraping, a list *meta_data_arr* containing the scraped content is returned, and the content is printed on the terminal using the [*print_scrapped_content()](https://github.com/hjsblogger/web-scraping-with-python/blob/main/pageobject/helpers.py#L171)* method.

**Implementation (Pytest)**

Like PyUnit, the core logic for scraping the YouTube channel is implemented in the *scrap_youtube()* method in [test_yt_scraping.py](https://github.com/hjsblogger/web-scraping-with-python/blob/main/tests/pytest/test_yt_scraping.py). It returns a list that contains the metadata of the videos.

Since the majority of the implementation remains the same, we would be focussing only on the changes that are specific to the pytest framework.

    # Import the locators file
    import sys
    sys.path.append(sys.path[0] + "/..")
    from pageobject.locators import locators
    from pageobject.locators import *
    
    from pageobject.helpers import helpers
    from pageobject.helpers import *
    
    @pytest.mark.usefixtures('driver')
    
    class TestYoutubeScraping:
        def test_scrap_youtube(self, driver) -> list:
            meta_data_arr=[]
            driver.get(locators.test_yt_url)
    
            try:
                # Wait until the "Accept all" button is clickable
                accept_all = WebDriverWait(driver, 5).until(EC.element_to_be_clickable(
                    (By.CSS_SELECTOR, "form:nth-child(3) > div > div > button[aria-label= 'Accept all']")))
    
                # Click the "Accept all" button
                accept_all.click()
    
                print("Click on Accept all successful")
            except Exception as e:
                print(f"'Accept All' button not present, proceed with the tests: {str(e)}")
    
            meta_data_arr = helpers.scrap_yt_content(driver)
            helpers.print_scrapped_content(meta_data_arr)

The *@pytest.mark.usefixtures(‘driver’)* decorator indicates that the *driver* (i.e., instantiated Chrome browser) fixture must be used in the execution of the test methods.

![](https://cdn-images-1.medium.com/max/2000/0*SpihqUCzYVnH84WW.png)

As mentioned in the earlier section, the click to Accept all button is initiated in case the YouTube consent form is displayed on the screen.

![](https://cdn-images-1.medium.com/max/3200/0*dm0knTOWzDQwVZgp.png)

Once you are on the LambdaTest YouTube channel, the [*scrap_yt_content()](https://github.com/hjsblogger/web-scraping-with-python/blob/main/pageobject/helpers.py#L79)* method is invoked to scrap the channel content. Lastly, the scraped content is printed on the screen.

![](https://cdn-images-1.medium.com/max/2108/0*7N7Lc4AaN04Nw6Ex.png)

Since the rest of the implementation is the same as in the PyUnit framework, hence we are not touching upon those aspects in this section.
>  **Run your [Selenium Automation](https://www.lambdatest.com/selenium-automation?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=webpage) Testing scripts on the LambdaTest cloud grid. Test on 3000+ desktop & mobile environments. Try it for free**

**Implementation (Core Scraping Logic)**

The [helpers.py](https://github.com/hjsblogger/web-scraping-with-python/blob/main/pageobject/helpers.py) file contains the core implementation for scraping the LambdaTest YouTube channel.

    # Import the locators file
    import sys
    from pageobject.locators import locators
    from pageobject.locators import *
    
    def create_actions(driver):
        actions = ActionChains(driver)
        return actions
    
    def create_waits(driver, duration):
        # Explicit wait of 10 seconds
        wait = WebDriverWait(driver, duration)
        return wait
    
    class helpers(object):    
        def scrap_yt_content(driver)->list:
            meta_data_arr=[]
            # Explicit wait of 10 seconds
            wait = create_waits(driver, 10)
    
            actions = create_actions(driver)
    
            # Explicit wait of 10 seconds
            wait = WebDriverWait(driver, 10)
    
            # Wait for 10 seconds till the Document State is not complete
            wait.until(lambda driver: driver.execute_script('return document.readyState') == 'complete')
    
            # Once the page has loaded, scroll to the end of the page to load all the videos
            # Scroll to the end of the page to load all the videos in the channel
            # Reference - https://stackoverflow.com/a/51702698/126105
            # Get scroll height
            start_height = driver.execute_script("return document.documentElement.scrollHeight")
    
            # Repeat scrolling until reaching the end of the page
            # Taking cues from my own blog https://www.lambdatest.com/blog/scraping-dynamic-web-pages/
            while True:
                # Scroll to the bottom of the page
                driver.execute_script("window.scrollTo(0, " + str(start_height) + ")")
                # Wait for the content to load
                time.sleep(2)
                scroll_height = driver.execute_script("return document.documentElement.scrollHeight")
                if (scroll_height == start_height):
                    # If heights are the same, we reached the end of page
                    break
                # print("scroll_height = " + str(scroll_height))
                time.sleep(2)
                start_height = scroll_height
    
            time.sleep(2)
    
            elem_1 = driver.find_elements(By.CSS_SELECTOR,
                "#dismissible > #details")
    
            for video_metadata_1 in elem_1:
                elem_2 = video_metadata_1.find_element(By.CSS_SELECTOR,
                    "#meta")
    
                elem_3 = elem_2.find_element(By.CSS_SELECTOR,
                    "#video-title")
    
                elem_4 = elem_2.find_element(By.CSS_SELECTOR,
                    "#metadata > #metadata-line > span:nth-child(3)")
    
                elem_5 = elem_2.find_element(By.CSS_SELECTOR,
                    "#metadata > #metadata-line > span:nth-child(4)")
    
                video_title = elem_3.get_attribute('innerText')
                video_views = elem_4.get_attribute('innerText')
                video_time = elem_5.get_attribute('innerText')
    
                # Create a dictionary of the video meta-data
                meta_data_dict = {
                    'video title': video_title,
                    'video views': video_views,
                    'video duration': video_time
                }
    
                meta_data_arr.append(meta_data_dict)
    
            return meta_data_arr
    
        def print_scrapped_content(meta_data):
            for elem_info in meta_data:
                print(elem_info)

**Code Walkthrough**

To get started, we wait for a maximum duration of 10 seconds till the document (or page) is fully loaded (i.e. *Document.readyState* is equal to *complete*). Feel free to check out the detailed [DOM tutorial](https://www.lambdatest.com/blog/document-object-model/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog) if you want a quick refresher about the nuances of DOM.

![](https://cdn-images-1.medium.com/max/3200/0*wh9m1PjTIH4XyM8Z.png)

Once the page is loaded, we first get the scrollable height of the document using the JavaScript function *document.documentElement.scrollHeight*. Before doing so, we maximize the browser window since that is considered one of the [best practices in Selenium automation](https://www.lambdatest.com/blog/selenium-best-practices-for-web-testing/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog).

![](https://cdn-images-1.medium.com/max/3200/0*I5vAhh9K0vl-cM9K.png)

Next, we run a *while* loop where a vertical scroll is performed using the *window.scrollTo()* method with *document.documentElement.scrollHeight *as the input argument.

![](https://cdn-images-1.medium.com/max/3200/0*pEqA6jOsXeU1Vpix.png)

In order to see the *window.scrollTo()* method in action, open the [LambdaTest YouTube channel](https://www.youtube.com/@LambdaTest/videos?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=youtube) on your browser. Then trigger the following commands in the browser console:

* *document.documentElement.scrollHeight*

* *window.scrollTo(0, document.documentElement.scrollHeight)*

![](https://cdn-images-1.medium.com/max/3200/0*KV990hXXhQ0AVMrW.png)

In the implementation shown above, we perform the above two actions till the end of the page is reached. This is when we break from the while loop!

The video meta-data (i.e., *name*, *views*, *duration*, etc.) is present in the div with ID *details, *which is under another div *dismissible*. The said element(s) are located using the CSS Selector (i.e., *dismissible > #details).*

![](https://cdn-images-1.medium.com/max/2040/0*YnA42HCf6kWqVJkD.png)

As seen in the Inspect Tools screenshot, there are 581 entries (or elements) when searched with CSS Selector (i.e., *dismissible > #details). *In summary, the LambdaTest YouTube channel has 581 videos when writing this blog on web scraping with Python.

![](https://cdn-images-1.medium.com/max/3200/0*27bJWC8oGb4i9b4N.png)

Next up, we locate the element(s) using CSS Selector *#meta,* which is nested under the div *#details.*

![](https://cdn-images-1.medium.com/max/3200/0*9v-BB8_MhVDX1Mta.png)

![](https://cdn-images-1.medium.com/max/2748/0*yEd6GAdRPhHwqNb8.png)

Now that we have located the *#meta* tag, we procure the video title, which is nested in the #meta tag. The title of the video is located using the *#video-title* CSS Selector.

![](https://cdn-images-1.medium.com/max/3200/0*CBWh0gAJZ4fsDrW0.png)

![](https://cdn-images-1.medium.com/max/2480/0*9IT7QlhRKkbzAKW9.png)

As seen below, the *innerText* attribute of the element (CSS Selector *#video-title*) provides the title of the video!

![](https://cdn-images-1.medium.com/max/3200/0*IUSQHIa3YsIuz7m-.png)

![](https://cdn-images-1.medium.com/max/2076/0*2aAOzjUOJ1c6l9tE.png)
>  **Automate [Cypress testing](https://www.lambdatest.com/cypress-testing?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=webpage) and perform browser automation testing with LambdaTest. Our cloud infrastructure has 3000+ desktop & mobile environments. Try for free!**

The next task is to find the video views, which are located using the nested CSS Selector *#metadata > #metadata-line > span:nth-child(3)*

As seen in the Inspect Tools screenshot, *#metadata-line* is nested inside *#metadata* (i.e., Element with ID *#metadata-line* is a child of an element with ID *#metadata*). To find the element’s locator that gives *video views*, simply hover over the views (of the first video) and copy the selector from the Inspect tools in the browser.

![](https://cdn-images-1.medium.com/max/3200/0*je5UZrAbnLbAg_WI.png)

The pseudo class *nth-child(3)* selects the third child span element among the direct children of the element with the ID *metadata-line*. Hence, the element located using CSS Selector *#metadata > #metadata-line > span:nth-child(3)* provides the views of the video in the earlier step.

![](https://cdn-images-1.medium.com/max/3200/0*sC3Et0ygVuQlmrL6.png)

*Locating the Video Views element using Inspect Tools*

![](https://cdn-images-1.medium.com/max/2748/0*hXIc4KgNB7chNVJj.png)

The *innerText* attribute of the located element provides the number of views of the said video.

![](https://cdn-images-1.medium.com/max/2276/0*5DLqJG0OqeHHGOAZ.png)

On similar lines, the publishing date of the respective video is also procured using the *innerText* attribute of the element located using the CSS Selector *#metadata > #metadata-line > span:nth-child(4).*

![](https://cdn-images-1.medium.com/max/3200/0*tE8PqgkjmMVIvt0W.png)

![](https://cdn-images-1.medium.com/max/2276/0*0xsOk95dcbsGKE4E.png)

All the above-mentioned steps are executed in a for loop that iterates through the elements that were located using *driver.find_elements(By.CSS_SELECTOR, “#dismissible > #details”*)

![](https://cdn-images-1.medium.com/max/3200/0*qDcTXdxH2oM2o7cM.png)
>  **Automate Cypress testing and perform browser automation testing with LambdaTest. Our [Cypress cloud](https://www.lambdatest.com/cypress-testing?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=webpage) infrastructure has 3000+ desktop & mobile environments. Try for free!**

### Test Scenario 2 — Scraping LambdaTest eCommerce Playground

In this example, we scrap the product meta-data (i.e., product name and price) from [LambdaTest eCommerce Playground](https://ecommerce-playground.lambdatest.io/index.php?route=product/category&path=57). As mentioned earlier, the scraping logic works seamlessly on the local grid and cloud Selenium grid like LambdaTest.

![](https://cdn-images-1.medium.com/max/3200/0*MPgGUGGxWB3HGb9h.png)

**Implementation (Core Scraping Logic)**

Since the overall setup & configuration-related implementation remain unchanged, we would directly deep dive into the scraping implementation.

![](https://cdn-images-1.medium.com/max/2000/1*x8cJ4rOro5wAFlJZvyXMPw.png)

Akin to the YouTube test scenario, the *helpers.py* file contains the core implementation where the [*scrap_ecomm_content()](https://github.com/hjsblogger/web-scraping-with-python/blob/main/pageobject/helpers.py#L16)* method is responsible for scraping the eCommerce Playground. It returns a list that contains the metadata of the SKUs on the page.

    # Import the locators file
    import sys
    from pageobject.locators import locators
    from pageobject.locators import *
    
    def create_actions(driver):
        actions = ActionChains(driver)
        return actions
    
    def create_waits(driver, duration):
        # Explicit wait of 10 seconds
        wait = WebDriverWait(driver, duration)
        return wait
    
    class helpers(object):
        def scrap_ecomm_content(driver)->list:
            meta_data_arr=[]
            # Explicit wait of 10 seconds
            wait = create_waits(driver, 10)
    
            actions = create_actions(driver)
    
            element_cat = wait.until(EC.visibility_of_element_located((By.XPATH,
                    locators.shopcategory)))
    
            # Move to the element and perform click operation
            actions.move_to_element(element_cat).click().perform()
    
            element_phcat = wait.until(EC.visibility_of_element_located((By.XPATH,
                    locators.phonecategory)))
    
            actions.move_to_element(element_phcat).click().perform()
    
            nested_elements = wait.until(EC.visibility_of_element_located((By.XPATH,
                "//div[@id='entry_212391']//div[@id='entry_212408']//div[@class='row']")))
    
            # Tough nut : 2 - nested locators!
            actual_items = nested_elements.find_elements(By.CLASS_NAME,
                    "product-layout.product-grid.no-desc.col-xl-4.col-lg-4.col-md-4.col-sm-6.col-6")
    
            count = len(actual_items)
    
            for ind_elem_props in actual_items:
                nested_product_name_elem = ind_elem_props.find_element(By.CSS_SELECTOR,
                    "div.product-thumb > div.caption")
                
                ################ Product Name ################
                nested_title_elem = nested_product_name_elem.find_element(By.CSS_SELECTOR,
                        ".title .text-ellipsis-2")
                
                ################ Price #######################
                nested_price_elem = nested_product_name_elem.find_element(By.CSS_SELECTOR,
                        ".price .price-new")
                
                # Create a dictionary of the meta-data of the items on e-commerce store
                meta_data_dict = {
                    'product image': nested_title_elem.get_attribute('href'),
                    'product name': nested_title_elem.text,
                    'product price': nested_price_elem.text
                }
                
                meta_data_arr.append(meta_data_dict)
            
            return meta_data_arr
    
        def print_scrapped_content(meta_data):
            for elem_info in meta_data:
                print(elem_info)

**Code Walkthrough**

Let’s directly look into the major aspects of the *scrap_ecomm_content()* method!

First, we create an instance of the *ActionChains* class to perform mouse movement operations on the page. We suggest having a look at our detailed blog on [*ActionChains *in Selenium Python](https://www.lambdatest.com/blog/how-to-automate-mouse-clicks-with-selenium-python/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog) in case you are new to *ActionChains* or want to have a quick refresher of this important concept!

![](https://cdn-images-1.medium.com/max/2000/0*RT47N7hd3je6xAk0.png)

[*WebDriverWait ](https://www.lambdatest.com/blog/selenium-webdriverwait/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog)*and [*ExpectedConditions *in Selenium](https://www.lambdatest.com/blog/expected-conditions-in-selenium-examples/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog) are used to initiate an explicit wait till the visibility of a specified element is true. The element in question is the menu item that is located using the XPath *“//a[contains(.,’Shop by Category’)]*”

![](https://cdn-images-1.medium.com/max/3200/0*ZDmjGlYFvBXZi9QP.png)

Once the menu is visible, the *move_to_element()* method of *ActionChains* class is used to move (or focus) on the located menu item. Since only operation (i.e., click) is involved in the chain, we invoke the *click()* and *perform()* methods to trigger the required action.

![](https://cdn-images-1.medium.com/max/2848/0*6kJbH7--_zTT-geX.png)

Now that we are on the Menu, the menu item containing *Phone, Tablets & Ipod* uses the XPath selector. Once the menu item is located, the combination of *move_to_element()* and *click* operations is performed to open the [Product Page](https://ecommerce-playground.lambdatest.io/index.php?route=product/category&path=57).

![](https://cdn-images-1.medium.com/max/3200/0*aLyRoeXxtuwXNJY8.png)

![](https://cdn-images-1.medium.com/max/2312/0*pS_TiWBy5efIBq7L.png)

Now that we are on the Product Page, we wait for the visibility of the grid containing the respective products. By default, 15 products are available in a single go.

![](https://cdn-images-1.medium.com/max/3200/0*kS-nNQOzhtBhQYIj.png)

Nested XPath locator *//div[@id=’entry_212391′]//div[@id=’entry_212408′]//div[@class=’row’]* is used for locating the grid containing the products. Alternatively, you can also use the CSS Selector to locate the grid.

Simply right-click on the row and choose *Copy → Copy CSS Selector* to locate the grid using CSS Selector. You can check out our [XPath vs CSS Selectors](https://www.lambdatest.com/blog/xpath-vs-css-selectors/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog) guide to get more insights into the advanced features of these selectors.

![](https://cdn-images-1.medium.com/max/3200/0*3Fn_GFvEzYWkVTRn.png)

Now that we have located the grid (containing the 15 products), the next step is to locate every row housing the corresponding product. As shown below, we have <*div*>’s for each nested product under the locator used in the previous step.

![](https://cdn-images-1.medium.com/max/3200/0*XS7NgUhQuHohuu9S.png)

The [*find_elements() *method in Selenium](https://www.lambdatest.com/blog/findelement-and-findelements-in-selenium/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog) is used along with the Class Name selector to find the elements containing product(s) information. Since there are 15 products on the page, the length of the list returned by the *find_elements()* method will also be 15!

![](https://cdn-images-1.medium.com/max/3200/0*vAUzEQ1BGhXCD9S2.png)

![](https://cdn-images-1.medium.com/max/3200/0*Ywf4lR8r6veypQ0J.png)
>  **This Cypress automation testing tutorial will help you learn the benefits of [Cypress test automation](https://www.lambdatest.com/blog/cypress-test-automation-framework/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog), and how to install Cypress and execute Cypress automation testing over scores of browsers and operating systems online.**

Next, we run a *for* loop and iterate through the list to scrap details of every product in the list. Since there are 15 products on the page, the loop will be executed 15 times.

As seen below, The element(s) located using CSS locator *div.product-thumb > div.caption* provides the meta-data (i.e., product name & price) of every product on the page.

![](https://cdn-images-1.medium.com/max/3200/0*5p7EtPzQN1jU1t5X.png)

![](https://cdn-images-1.medium.com/max/3020/0*CAKbvNE4Qz5q3vQ2.png)

With the meta-data handy, the next step is to find the *Product Name* from the element located in the previous step.

![](https://cdn-images-1.medium.com/max/3200/0*271R-ypg81CMe_vW.png)

As seen from the DOM layout, the element located using nested CSS Selector *.title .text-ellipsis-2* provides product details.

![](https://cdn-images-1.medium.com/max/2984/0*96Pcn3thhaBdlIMF.png)

The text attribute of the element provides the Product Name of the respective product.

![](https://cdn-images-1.medium.com/max/2884/0*tnqMqgjnO4VHGSTL.png)

On similar lines, the href attribute of the located element provides the link to the product (e.g., [Product Link](https://ecommerce-playground.lambdatest.io/index.php?route=product/product&path=57&product_id=28&limit=15)).

![](https://cdn-images-1.medium.com/max/3200/0*7qBGSrtnLQvZDiGb.png)

![](https://cdn-images-1.medium.com/max/2884/0*zMwxytwQcFtlE_ck.png)

Like the earlier step, the price of the current product is scraped by locating the element using the nested CSS Selector .*price .price-new*

![](https://cdn-images-1.medium.com/max/3200/0*0_tyGJT3Qfgw7Sro.png)

![](https://cdn-images-1.medium.com/max/2984/0*7EUPSzmWMVRaXEC5.png)

Now that the element is located, the *text* attribute of the element gives the price of the product.

![](https://cdn-images-1.medium.com/max/2884/0*XnwPYm5h5eRL6rZj.png)

Finally, the *scrap_ecomm_content()* method returns a dictionary of the scraped content of the page under test!
>  **Test your website or web app online for iOS browser compatibility. Perform seamless cross browser testing on the latest [iPhone tester](https://www.lambdatest.com/test-on-iphone-simulator?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=webpage) Simulator. Try for free!**

### Execution (PyUnit + Selenium)

As stated earlier, we can use the browser on a local machine (or grid) and cloud Selenium grid.

Set environment variable ***EXEC_PLATFORM*** to ***local ***using Chrome browser (headless mode) with Selenium for web scraping. Invoke the command *make scrap-using-pyunit* to start scraping content from the test website(s).

![](https://cdn-images-1.medium.com/max/3200/0*oToOMCHxE0ebZrSS.png)

As shown below, the Accept all button does not appear when the scraping is done on a local machine.

![](https://cdn-images-1.medium.com/max/3200/0*zQ9mTCoeHvQwMUAR.png)

On the other hand, the Accept all button is seen when the scraping is done using Chrome (headless mode) on cloud Selenium Grid.

![](https://cdn-images-1.medium.com/max/3200/0*Fx5DKaZFiEbfDEWP.png)

As seen below, the content from the LambdaTest YouTube channel and LambdaTest eCommerce playground was scrapped successfully.

![](https://cdn-images-1.medium.com/max/3200/0*ApRVjeztIIqoHzXP.png)

![](https://cdn-images-1.medium.com/max/3200/0*JPBOml8HIQZeS6f4.png)

Set environment variable ***EXEC_PLATFORM*** to ***cloud ***for using Chrome browser (headless mode) on cloud Selenium Grid.

![](https://cdn-images-1.medium.com/max/3200/0*xyQ0svtzi_an2fzl.png)

Log on to the [LambdaTest Automation Dashboard](https://automation.lambdatest.com/build?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=webpage) to view the status of the test execution.

![](https://cdn-images-1.medium.com/max/3200/0*7lOjvtsfGRxFRhz_.png)

As seen below, the test execution on LambdaTest was successful!

![](https://cdn-images-1.medium.com/max/3200/0*5V1enehABzalkn_7.png)

### Execution (pytest + Selenium)

Like earlier, set ***EXEC_PLATFORM*** to ***local ***for using Chrome browser (headless mode) with Selenium for web scraping with Python. Invoke the command *make scrap-using-pytest* to start scraping content from the test website(s).

![](https://cdn-images-1.medium.com/max/3200/0*YeKUTNTa786Bki7L.png)

As seen below, scraping data from the test websites was successful.

![](https://cdn-images-1.medium.com/max/3200/0*r2smPbSTV2J8sBk3.png)

![](https://cdn-images-1.medium.com/max/3200/0*BhCnf8E-769MsiZR.png)

Set environment variable ***EXEC_PLATFORM*** to the ***cloud ***and invoke *make scrap-using-pytest* for using Chrome browser (headless mode) on cloud Selenium Grid.

![](https://cdn-images-1.medium.com/max/3200/0*J0tpw7VvC5dh_7uP.png)

![](https://cdn-images-1.medium.com/max/3200/0*DPV6fwisgm04zaGM.png)

Shown below is the status on the dashboard, which indicates that web scraping using Selenium and pytest was successful.

![](https://cdn-images-1.medium.com/max/3200/0*4tdgNNmzguNlvF5z.png)

![](https://cdn-images-1.medium.com/max/3200/0*zFpfKwD9XiwUUin7.png)
>  **Explore Jest tutorial, your go-to guide for mastering [Jest](https://www.lambdatest.com/jest?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=webpage) testing. Run Jest automation tests in parallel across multiple browser and OS combinations.**

## Web Scraping using Beautiful Soup and Selenium

For a demonstration of web scraping with Python using Beautiful Soup and Selenium, we would be scraping content from the following websites:

![](https://cdn-images-1.medium.com/max/2000/1*1xf1FSyaiwV_vlVyz_SxRg.png)

We are using the combination of Beautiful Soup and Selenium for a very specific reason.🙂As you can see, both the test websites have dynamic content which is where Selenium has an upper-hand over Beautiful Soup.

Selenium is used for performing actions (e.g., scrolling, clicking, etc.) on the page (or document) so that the content to be scraped is available for our perusal. On the other hand, Beautiful Soup is used for parsing and navigating the structure of the HTML page.

The prerequisites and directory structure are already covered in the earlier sections of this blog on web scraping with Python. *BeautifulSoup4* (or *bs4*) & *Selenium (v 4.13.0)* are used for scraping the ScrapingClub website, whereas bs4, along with *Requests (v 2.31.0)* help in scraping the eCommerce playground product page(s).

### Demonstration: Scraping eCommerce Playground

In this test scenario, we would scrap the products from the eCommerce Playground product page. The website uses pagination, and 15 products are displayed on a single page.

 1. Navigate to the [eCommerce Playground product page](https://ecommerce-playground.lambdatest.io/index.php?route=product/category&path=57).

 2. Send an HTTP GET request to the URL using the Requests library.

 3. Using bs4, scrap product information from all the products of the said category.

**Implementation**

Similar to the configuration for the PyUnit & pytest frameworks, the locator & URL details are present in locators.py.

![](https://cdn-images-1.medium.com/max/3200/0*4XK6Uf_jzE6tPr-7.png)

The core scraping logic is implemented in [*test_ecommerce_scraping.py.](https://github.com/hjsblogger/web-scraping-with-python/blob/main/tests/beautiful-soup/test_ecommerce_scraping.py)* Let’s dive deep into the same!

    # Beautiful Soup Official Documentation - https://www.crummy.com/software/BeautifulSoup/bs4/doc/
    
    # Import the locators file
    import sys
    from pprint import pprint
    sys.path.append(sys.path[0] + "/../..")
    
    from pageobject.locators import locators
    from pageobject.locators import *
    
    from pageobject.helpers import helpers
    from pageobject.helpers import *
    
    def scrap_ecommerce(url) -> list:
        response = requests.get(url)
    
        if response.status_code != 200:
            print(f"Unable to fetch the page. Status code: {response.status_code}")
            return None
    
        soup = BeautifulSoup(response.text, 'html.parser')
    
        rows = soup.select('.product-layout.product-grid.no-desc.col-xl-4.col-lg-4.col-md-4.col-sm-6.col-6')
    
        meta_data_arr = []
    
        for row in rows:
            link = row.find("a", class_='carousel d-block slide')
            name = row.find("h4", class_='title')
            price = row.find("span", class_='price-new')
    
            # Create a dictionary of the meta-data of the items on e-commerce store
            meta_data_dict = {
                'product link': link.get_attribute_list('href'),
                'product name': name.get_text(),
                'product price': price.get_text()
            }
            
            meta_data_arr.append(meta_data_dict)
        return meta_data_arr
    
    # Pagination - 1:5
    # Page 1: https://ecommerce-playground.lambdatest.io/index.php?route=product/category&path=57&page=1
    # Page 5: https://ecommerce-playground.lambdatest.io/index.php?route=product/category&path=57&page=5
    if __name__ == '__main__':
        for iteration in range(1,6):
            test_url = locators.test_bs4_url + "&page=" + str(iteration)
            meta_data_arr = scrap_ecommerce(test_url)
            print('\n')
            print("Product Page = " + test_url)
            print("*********************************************************************************************************\n")
            helpers.print_scrapped_content(meta_data_arr)

**Code Walkthrough**

Inside the *__main__ *construct, we set up the test URL, which constitutes the [base URL](https://ecommerce-playground.lambdatest.io/index.php?route=product/category&path=57?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=webpage) appended by the page number (i.e. &*page=< page-number >*).

![](https://cdn-images-1.medium.com/max/3200/0*7k6HAK5J2kr7dKqL.png)

![](https://cdn-images-1.medium.com/max/3200/0*dyLm5y2QvTVL0ErM.png)

Since there are 5 pages in total for the said product category, the scraping implementation is executed in an interactive loop (from 1..6).

![](https://cdn-images-1.medium.com/max/3200/0*NqCQ--W1MUoVrFdX.png)

To get started with scraping, an HTTP GET request is initiated using the *get()* method of the *requests* library. The method takes the URL under test as the input parameter and returns a response object.

![](https://cdn-images-1.medium.com/max/2000/0*_igrQRypmQdLR0J_.png)

The status code (i.e., *status_code*) attribute of the response object is compared with 200 (i.e., *STATUS_OK*). If the status_code is not 200 (e.g., 404, 403, etc.), the test is terminated.

![](https://cdn-images-1.medium.com/max/3152/0*zRUsj8Lqh70-e6Fy.png)

The *response.text* attribute contains the HTML content of the page, which is retrieved as a string. As stated earlier, the HTML parser (i.e., *html.parser*), the default parser in bs4, is used to parse the HTML content.

*BeautifulSoup(response.text, ‘html.parser’)* returns a *BeautifulSoup* object that will be used throughout scraping. The *select()* method of Beautiful Soup finds elements using the CSS Selector property — .*product-layout.product-grid.no-desc.col-xl-4.col-lg-4.col-md-4.col-sm-6.col-6*

As expected, 15 elements match the CSS Selector since there are 15 products on the page. The *select()* method returns a list that will be further used for iteratively scraping information (i.e., name, cost) of each product.

![](https://cdn-images-1.medium.com/max/3200/0*FBM1pqNPwSB8tKJ5.png)

![](https://cdn-images-1.medium.com/max/3200/0*NIAkIetVFKYyxGSC.png)

Next up, we run a for loop for scraping information of all the 15 products (or elements) under the *div* located in the previous step.

![](https://cdn-images-1.medium.com/max/3200/0*4lf6O9FQiuK8EHvB.png)
>  **Test native, hybrid, and web apps on any mobile OS with our free [Android emulator](https://www.lambdatest.com/android-emulator-online?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=webpage) online. Sign up to optimize app performance.**

The **product link** is obtained by locating the element using the *find()* method of Beautiful Soup. As seen in the Inspect Tools screenshot above, the first argument is the tag that needs to be searched for (i.e., “a” — anchor tag), and the second is the CSS Class attribute.

![](https://cdn-images-1.medium.com/max/2412/0*qhjJDAb3DDaHFigb.png)

Link to the product is procured by using the *get_attribute_list()* method with the located element. In case you need more information on the *find()* method of Beautiful Soup, we recommend checking out the Beautiful Soup Official Documentation.

![](https://cdn-images-1.medium.com/max/3200/0*rKRak680c9H8z7CW.png)

![](https://cdn-images-1.medium.com/max/3020/0*ChfmbrWrlgwmPJvW.png)

On similar lines, the **product name** is also scraped by using the *find()* method in Beautiful Soup. In this case, the first argument is the “*h4*” tag, and the second argument is the Class Name locator “*title*”.

![](https://cdn-images-1.medium.com/max/3200/0*e9spfdSnjq_XZuLJ.png)

Finally, the [*get_text()](https://www.crummy.com/software/BeautifulSoup/bs4/doc/#get-text)* method of Beautiful Soup, when used with the recently located element, provides the product name of the current product.

![](https://cdn-images-1.medium.com/max/3020/0*pK3VKApzfvfGyuLN.png)

The last step is to scrap the **price** of the respective product. As shown below, the text under the element with Class Name “*price-new*” provides the product price.

![](https://cdn-images-1.medium.com/max/3200/0*jznWYjvG7we7q8vO.png)

Like before, the *find()* method of Beautiful Soup is used with “*span*” (i.e., tag to be located) as the first argument and Class Name “*price-new*” as the second argument. The *get_text()* method returns the text inside the element.

![](https://cdn-images-1.medium.com/max/3020/0*ZC_TOkMnVFFsjIW0.png)

Finally, a dictionary (i.e., *meta_data_dict*) of the scraped information (i.e., product link, name, and price) is created, and the same is appended to the list (i.e., *meta_data_arr*).

![](https://cdn-images-1.medium.com/max/3020/0*P5nfaPqpJ298X1WS.png)

The same steps are repeated for all the other product pages (i.e., [Product Page 1](https://ecommerce-playground.lambdatest.io/index.php?route=product/category&path=57&page=1) → [Product Page 5](https://ecommerce-playground.lambdatest.io/index.php?route=product/category&path=57&page=5)), and the scraped data is displayed on the screen.

![](https://cdn-images-1.medium.com/max/3200/0*397zM9JJdLy_E8_n.png)

### Demonstration: Scraping Infinite Scrolling website

In this test scenario, we would be scraping the ScrapingClub website that has content loaded on a dynamic basis. Since the website involves dynamic content, the combination of Selenium & Beautiful Soup is used for web page interactions & scraping.

 1. Navigate to Infinite Scroll Page.

 2. Scroll till the end of the page.

 3. Scrap product content using Beautiful Soup.

**Implementation**

Akin to the PyUnit & pytest frameworks, the locator & URL details are present in locators.py

![](https://cdn-images-1.medium.com/max/3200/0*lr9k17jdMPmZJ1Pa.png)

The core scraping logic is implemented in [*test_infinite_scraping.py.](https://github.com/hjsblogger/web-scraping-with-python/blob/main/tests/beautiful-soup/test_infinite_scraping.py)* Let’s deep dive into the same!

    # Selenium WebDriver used for infinite scrolling
    # Beautiful Soup used for scraping content
    
    # Beautiful Soup Official Documentation - https://www.crummy.com/software/BeautifulSoup/bs4/doc
    
    import sys
    # Import the locators file
    import os
    from pprint import pprint
    sys.path.append(sys.path[0] + "/../..")
    
    from pageobject.locators import locators
    from pageobject.locators import *
    
    from pageobject.helpers import helpers
    from pageobject.helpers import *
    
    exec_platform = os.getenv('EXEC_PLATFORM')
    
    def scrap_inifite_website(url) -> list:
        meta_data_arr = []
    
        if exec_platform == 'cloud':
            username = environ.get('LT_USERNAME', None)
            access_key = environ.get('LT_ACCESS_KEY', None)
    
            gridURL = "@hub.lambdatest.com/wd/hub">https://{}:{}@hub.lambdatest.com/wd/hub".format(username, access_key)
            
            ch_options = webdriver.ChromeOptions()
            ch_options.browser_version = "latest"
            ch_options.platform_name = "Windows 11"
    
            lt_options = {}
            lt_options["build"] = "Build: Web Scraping with Selenium & Beautiful Soup"
            lt_options["project"] = "Project: Web Scraping with Selenium & Beautiful Soup"
            lt_options["name"] = "Test: Web Scraping with Selenium & Beautiful Soup"
    
            lt_options["browserName"] = "Chrome"
            lt_options["browserVersion"] = "latest"
            lt_options["platformName"] = "Windows 11"
    
            lt_options["console"] = "error"
            lt_options["w3c"] = True
            lt_options["headless"] = True
    
            ch_options.set_capability('LT:Options', lt_options)
    
            driver = webdriver.Remote(
                command_executor = gridURL,
                options = ch_options
            )
        elif exec_platform == 'local':
            options = ChromeOptions()
            options.add_argument("--headless=new")
            driver = webdriver.Chrome(options=options)
    
        driver.get(url)
    
        # Took some support from https://stackoverflow.com/a/41181494/126105
        start_height = driver.execute_script("return document.documentElement.scrollHeight")
    
        while True:
            # Scroll to the bottom of the page
            driver.execute_script("window.scrollTo(0, " + str(start_height) + ")")
            # Wait for the content to load
            time.sleep(2)
            scroll_height = driver.execute_script("return document.documentElement.scrollHeight")
            if (scroll_height == start_height):
                # If heights are the same, we reached the end of page
                break
            time.sleep(2)
            start_height = scroll_height
    
        time.sleep(2)
    
        # Why features='html.parser' is required
        # The code that caused this warning is on line 44 of the file <file>.
        # To get rid of this warning, pass the additional argument 'features="html.parser"'
        # to the BeautifulSoup constructor.
        soup = BeautifulSoup(driver.page_source, features='html.parser')
        
        # Instantiated Chrome instance is no longer required
        # since we have already read the source
        driver.quit()
    
        # Code changes as per 28/07/2023
        # In case if elements are not located, please change the locators accordingly
        rows = soup.find_all('div', class_='w-full rounded border post')
        print("\nTotal items on " + url + " are " + str(len(rows)) + "\n")
    
        for row in rows:
            dress = row.find('h4')
            link = dress.find('a')
            price = row.find('h5')
    
            # Create a dictionary of the meta-data of the items on e-commerce store
            meta_data_dict = {
                'dress': dress.text.strip(),
                'link' : link.get_attribute_list('href'),
                'price': price.text.strip()
            }
            
            meta_data_arr.append(meta_data_dict)
        return meta_data_arr
    
    if __name__ == '__main__':
        meta_data_arr = scrap_inifite_website(locators.test_bs4_infinite_url)
        helpers.print_scrapped_content(meta_data_arr)

**Code Walkthrough**

Inside the *__main__ *construct, we invoke the [*scrap_inifinite_website()](https://github.com/hjsblogger/web-scraping-with-python/blob/main/tests/beautiful-soup/test_infinite_scraping.py#L20)* method that is primarily responsible for scraping the website under test.

![](https://cdn-images-1.medium.com/max/2952/0*LmoaWK0mvhvKNP5I.png)

Since we are using Selenium for performing actions on the page, the EXEC_PLATFORM environment variable helps us choose between local grid (or machine) and the cloud Selenium Grid.

We have already covered Remote WebDriver and [Headless browser testing](https://www.lambdatest.com/learning-hub/headless-browser-testing?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=learning_hub) in the earlier sections of the blog.

![](https://cdn-images-1.medium.com/max/3138/0*HraxRQiP7LdeoeOZ.png)

Since the content is loaded dynamically (i.e., on page scroll), scroll till the end of page is performed using the combination of the *execute_script()* method with *document.documentElement.scrollHeight* as the input argument.

![](https://cdn-images-1.medium.com/max/3200/0*hpnvdj1Ln5tzjKgY.png)

*Content & Images loaded on a dynamic basis*

![](https://cdn-images-1.medium.com/max/3200/0*MUNNbmZisyq0I0wL.png)

Now that the page load is complete, create a Beautiful Soup object that parses the page source obtained using Selenium. You can check out our detailed blog that deep dives into [getting page sources with Selenium WebDriver](https://www.lambdatest.com/blog/how-to-get-page-source-in-selenium-webdriver/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog) for more information on retrieving page sources using Selenium.

![](https://cdn-images-1.medium.com/max/3200/0*ajypaDcm4AyIR9HS.png)

Since we already have the page source, it’s safe to release resources used by the instantiated browser instance.

![](https://cdn-images-1.medium.com/max/2244/0*OfetJDlHrfuxLwXE.png)
>  **Discover 31 [best automation testing tools](https://www.lambdatest.com/blog/automation-testing-tools/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog) in 2023 for powerful and efficient testing solutions.**

Next, the *find_all()* method of Beautiful Soup is used with “*div*” (i.e., tag to be located) as the first argument and Class Name “*w-full rounded border post*” as the second argument. It returns a list of elements that meet the specified conditions.

![](https://cdn-images-1.medium.com/max/3200/0*uIotlV06vZ_OWCD0.png)

As seen from the Inspect Tools screenshot, the length of the list returned by *find_all()* method should be 60 (i.e. 60 products are present on the page).

![](https://cdn-images-1.medium.com/max/3084/0*fA1vPH0j4lLFVcnV.png)

Next up, we use the for loop for iteratively scraping information (i.e. dress name, link, price) of each product.

![](https://cdn-images-1.medium.com/max/2000/0*iot1PemZKrdsEaY0.png)

The *find()* method of Beautiful Soup is used to locate the element with input as “*h4*” tag.

![](https://cdn-images-1.medium.com/max/3200/0*vh1SRH-ID4B_H1sv.png)

The *text()* attribute of the located element provides the **product name** (e.g., Short Dress). The leading & trailing spaces of the output are done using the *strip()* method.

![](https://cdn-images-1.medium.com/max/3020/0*YZOO-Sp47s6919Pj.png)

The “*href*” attribute of the element located earlier using the “*h4*” tag [i.e., *dress = row.find(‘h4’)*] gives the **product link**.

![](https://cdn-images-1.medium.com/max/3200/0*4KDUW7UtUeyNjW3H.png)

![](https://cdn-images-1.medium.com/max/3020/0*FSib2REtlcgb_lAV.png)

On similar lines, the price of the respective product is obtained by locating the element using the *find()* method. The method takes the “*h5*” tag as the input argument since the *text()* attribute of the element contains the **product price**.

![](https://cdn-images-1.medium.com/max/3200/0*Kqza-Gx6KYGDLDVc.png)
>  **Deep dive to learn about test automation; [what is automation testing](https://www.lambdatest.com/learning-hub/automation-testing?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=learning_hub)**,**its uasage, types and also gain insights on how to get started with automated testing.**

Like the earlier step, leading & trailing spaces from the output of *text()* attribute to get the final price of the respective product.

![](https://cdn-images-1.medium.com/max/3020/0*qf2xqVKIifw_Kis2.png)

With this, we have successfully scraped the meta-data of products from the ScrapingClub (Infinite Scrolling) website. Let’s trigger the same on the local as well as LambdaTest Cloud Selenium grid.

### Execution

As stated earlier, we can instantiate the Chrome browser (in headless mode) on a local machine (or grid) as well as cloud Selenium grid.

Set environment variable ***EXEC_PLATFORM*** to ***local*** for using cloud Selenium grid. Invoke the command *make scrap-using-beautiful-soup* to start scraping content from the test website(s).

![](https://cdn-images-1.medium.com/max/3200/0*_c2_5KLZR87LiBSr.png)

As seen below, we are able to scrap content from the LambdaTest E-commerce Playground Product Page(s) and ScrapingClub website.

![](https://cdn-images-1.medium.com/max/3200/0*ze-kj2X-jqRuLXIu.png)

![](https://cdn-images-1.medium.com/max/3200/0*i0rPluJra9tXolSN.png)

Set environment variable ***EXEC_PLATFORM*** to ***cloud*** for using Cloud Selenium Grid on LambdaTest.

![](https://cdn-images-1.medium.com/max/3200/0*iPXF2Xc8-cl00GEF.png)

![](https://cdn-images-1.medium.com/max/3200/0*8uKJsUFUkBf49ntA.png)

![](https://cdn-images-1.medium.com/max/3200/0*zs2WJSk0e5d20Cow.png)

![](https://cdn-images-1.medium.com/max/3200/0*eW-4wsdpcicLrkMR.png)

Shown below is the status on the dashboard, which indicates that web scraping with Python using Beautiful Soup and Selenium was successful.

![](https://cdn-images-1.medium.com/max/3200/0*4_9ZCJvc3wTmBvIo.png)

![](https://cdn-images-1.medium.com/max/3200/0*jpwMvwz-KgI7fE25.png)

With this, we have covered web scraping with Python using popular libraries (or tools) like Selenium (with PyUnit & pytest) and Beautiful Soup. The popularly-quoted Spider-Man theme of “*With great power, comes great responsibility*” also applies to web scraping!

This is because web scraping with Python must be performed while taking legal and ethical considerations into account. It is unethical to scrape confidential (or sensitive) information from any website.

## Web Scraping Done Right!

As stated, web scraping is a super-powerful tool to have in your data collection armory. However, its benefits should be leveraged with utmost caution! Python provides numerous tools (or libraries) for web scraping, making it one of the preferred languages for scraping web content.

Though the choice of web scraping tool purely depends on the requirements, Selenium can be preferred for scraping dynamic web content. On the other hand, Beautiful Soup is one of the preferred choices for scraping static web content.

Playwright is another popular framework for scraping web content. You can refer to our detailed blog on [Web Scraping with Playwright and Python](https://www.lambdatest.com/blog/playwright-for-web-scraping/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog) in case you intend to leverage the framework for scraping.

As a part of the web scraping series, I dabbled into [Web Scraping with C# using Selenium](https://www.lambdatest.com/blog/scraping-dynamic-web-pages/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog). To summarize, web scraping is a great tool that helps make the most of public data, but it should be used with utmost responsibility.🙂

*Happy Scraping !*
>  **This blog explains different [types of automation testing](https://www.lambdatest.com/blog/types-of-automation-testing/?utm_source=devto&utm_medium=organic&utm_campaign=nov_22&utm_term=bh&utm_content=blog) along with their benefits and best practices. Also we have discussed different frameworks and tools at the end.**

## Frequently Asked Questions (FAQs)

### Is Python good for web scraping?

Yes, Python is widely recognized as an excellent programming language for web scraping. Python offers several libraries and frameworks, such as BeautifulSoup and Scrapy, that simplify the web scraping process. Its simplicity, readability, and rich ecosystem of packages make it a popular choice for web scraping tasks. Additionally, Python is platform-independent, which means you can use it on various operating systems to collect data from websites efficiently.

### What is web scraping with Python?

Web scraping with Python refers to automatically extracting data from websites. Python provides various libraries and frameworks, such as BeautifulSoup, Scrapy, and Requests, that enable developers to write scripts to access web pages, parse the HTML content, and extract specific information or data. This data can be used for various purposes, including data analysis, research, content aggregation, or any other application where web data is needed. Web scraping with Python is a versatile and powerful technique for collecting information from the internet.

### Is it legal to web scrape?

The legality of web scraping depends on various factors, including the website’s terms of service, the nature of the data being collected, and local or international laws. Here are some key points to consider:

* Website’s Terms of Service: Websites often have service or usage policies that specify whether web scraping is allowed or prohibited. It’s essential to review and comply with these terms. Violating a website’s terms of service could lead to legal issues.

* Publicly Accessible Data: Web scraping publicly accessible data, such as information without logging in, is typically considered more acceptable. However, this does not grant unlimited rights to use or redistribute the data.

* Copyright and Intellectual Property: If the data being scraped is protected by copyright or contains intellectual property, scraping and using that data may infringe on those rights.

* Personal Data and Privacy: Scraping and using personal data without consent can raise privacy and legal concerns, particularly under data protection laws like the GDPR in the European Union.

* Anti-Competitive Behavior: Web scraping for anti-competitive purposes, like price scraping to undercut competitors unfairly, may be subject to legal action under antitrust laws.

* Rate Limiting and Politeness: It’s advisable to implement rate limiting and be polite in your scraping activities. Excessive requests can strain a website’s servers and may be viewed as a denial-of-service attack.

* Laws and Jurisdictions: The legal aspects of web scraping can vary by jurisdiction. What’s legal in one country may not be in another.

To ensure compliance with laws and ethical standards, it’s essential to consult with legal experts if you have concerns about the legality of your web scraping activities, particularly when dealing with sensitive or proprietary data. Additionally, always respect a website’s terms of service and robots.txt file, which can specify rules for web crawlers.