Web scraping with Selenium and Python
Tags: selenium, python, automation, webscraping
Published: 2023-03-31 15:55:47

---
title: Web scraping with Selenium and Python
published: true
date: 2023-03-29 15:44:56 UTC
tags: selenium, python, automation, webscraping
canonical_url: https://blog.apify.com/web-scraping-with-selenium-and-python/
---

In this step-by-step tutorial, we show you how to scrape data from a website using Python with Selenium. Learn how to locate and wait for elements, take screenshots, and execute JavaScript code with examples.

## What is Selenium?

Until 2003, the word selenium was known only as a chemical element, but in 2004, it became the name of one of the most popular software testing frameworks. Initially designed for cross-browser end-to-end tests, [Selenium](https://www.selenium.dev/?ref=blog.apify.com) is a powerful open-source browser automation platform that supports Java, Python, C#, Ruby, JavaScript, and Kotlin.

Why is it called Selenium? The name came from a joke in an email by its creator, Jason Huggins. Wishing to mock his competitor, Mercury Interactive Corporation, Huggins quipped that you could cure mercury poisoning by taking selenium supplements. Thus the name Selenium caught on, and the rest, as they say, is history.

## Why use Selenium with Python for web scraping?

Python is by far the most popular choice of programming language for scraping web data. Combining it with Selenium WebDriver provides an easy API to write functional tests and web scraping scripts.

Selenium offers several ways to interact with websites, such as clicking buttons, filling in forms, scrolling pages, taking screenshots, and executing JavaScript code. That means Selenium can be used to scrape dynamically loaded content. Add to this its cross-language and cross-browser support, and it's little wonder that Selenium is one of the preferred frameworks for [web scraping in Python](https://blog.apify.com/web-scraping-python/).

![](https://blog.apify.com/content/images/2023/03/Monty-Python-And-now-for-something-completely-different.gif)

**_Fun fact:_** _The Python language was not named after the snake. When Guido van Rossum was implementing the language, he wanted a name for it that would be short, unique, and somewhat mysterious. It just so happened that he was reading the published scripts of_ **_Monty Python and the Flying Circus_** _at the time. That influenced him to go with the name Python._

![How to scrape a website with Selenium in Python](https://blog.apify.com/content/images/2023/03/web-scraping-with-selenium-and-python-1.jpg)

## How to scrape a website with Selenium in Python

With that brief introduction out of the way, its time to show you how to scrape a website using Python with Selenium.

### Setting up the environment for web scraping

To follow this tutorial, youll need to have the following installed:

1. Python 3.6 or later

2. Selenium package (`pip install selenium`)

3. [Chrome web browser](https://www.google.com/chrome/?ref=blog.apify.com)

4. The [ChromeDriver](https://sites.google.com/a/chromium.org/chromedriver/downloads?ref=blog.apify.com) that matches your Chrome browser version

You'll have to import the necessary packages for your Selenium script. For this tutorial, you'll need:

```python
import time 
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys

```

### 🚀 Launching the browser and navigating to the website

Once you have the required packages, you can launch the browser with the `webdriver` module and navigate to the website you want to scrape. In this case, we'll be using Chrome as the browser and navigating to the Monty Python online store: [https://www.montypythononlinestore.com/](https://www.montypythononlinestore.com/).

```python
DRIVER_PATH = "/usr/local/bin/chromedriver" # This path works for macos
driver = webdriver.Chrome(executable_path=DRIVER_PATH)
driver.get("https://www.montypythononlinestore.com/")

```

If you want to switch to [headless Chrome](https://blog.apify.com/headless-browsers-what-are-they-and-how-do-they-work/#what-is-the-difference-between-chrome-and-chrome-headless), you need to instantiate an `Options` object and set `add_argument` to `--headless=new`.

```python
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)
driver.get("http://www.montypythononlinestore.com/")

```

Selenium is deprecating the convenience method (`options.headless = True`) due to Chrome's recent [upgrade to headless mode](https://developer.chrome.com/articles/new-headless/?ref=blog.apify.com).

### 🔎 Locating and interacting with elements

Now that you've navigated to the website, you'll need to locate elements on the page and interact with them. For example, you might want to search for a product in the e-shop.

```python
search_box = driver.find_element(By.ID, "search-field")
search_box.send_keys("t-shirt")
search_box.send_keys(Keys.ENTER)

```

With Selenium WebDriver, you can use `find_element` for a single element or `find_elements` for a list of them. For example, if you want to select the `<h2>` element in an HTML document:

```python
h2 = driver.find_element(By.TAG_NAME, 'h2')

```

If you want to select all elements with the class name 'product' on a page:

```python
all_products = driver.find_elements(By.CLASS_NAME, 'product')

```

### Waiting for elements to load

Sometimes, the content on the web page is dynamically loaded after the initial page load. In such cases, you can wait for the required element to load using the `WebDriverWait` function.

In the example below, we wait 10 seconds for the 'h2' to load.

```python
wait = WebDriverWait(driver, 10)
element = wait.until(EC.presence_of_element_located((By.TAG_NAME, "h2")))

```

Once the element is loaded, you can scrape its content using the `element.text` method.

```python
element_text = element.text

```

### 📸 Taking a screenshot

If you need to take a screenshot of the website at any point, you can do that in your script using the `save_screenshot()` function.

```python
driver.save_screenshot("screenshot.png")

```

### 🪓 Executing JavaScript code

To execute JavaScript code, use the `execute_script()` method. For example, if you want to scroll to the bottom of the page to take a screenshot:

```python
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

```

You can use `time.sleep` to wait for the browser to scroll down before taking the screenshot. In the example below, we wait 5 seconds for the browser to scroll down.

```python
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(5)
driver.save_screenshot("screenshot.png")

```

### 🚪 Closing the browser

When youre done, you can close the browser window with the `driver.quit()`method. Note that the `quit` method is different from the `close` method. `close()` only closes the current window, but the WebDriver session will remain active, so use `quit()` to close all browser windows and end the WebDriver session.

### 📝 Final code

Now lets put it all together into a script for scraping the Monty Python store:

```python
import time 
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys

DRIVER_PATH = "/usr/local/bin/chromedriver"
driver = webdriver.Chrome(executable_path=DRIVER_PATH)
driver.get("<https://www.montypythononlinestore.com/>")

search_box = driver.find_element(By.ID, "search-field")
search_box.send_keys("t-shirt")
search_box.send_keys(Keys.ENTER)

wait = WebDriverWait(driver, 10)
element = wait.until(EC.presence_of_element_located((By.TAG_NAME, "h2")))

element_text = element.text
print(element_text)

all_products = driver.find_elements(By.CLASS_NAME, 'product')
print(f'There are {len(all_products)} t-shirts on the page.')

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(5)
driver.save_screenshot("screenshot.png")

driver.quit()

```

## Conclusion & further reading

Weve shown you how to use Selenium with Python to scrape the Monty Python online store, but you can use what you've learned here to scrape data from any site you like with the `driver.get` method.

If you want to learn more about Python, Selenium, and other powerful web scraping and browser automation tools, check out the online literature below.

**Selenium for web scraping, testing, and automation:**

🔖 [Selenium documentation](https://www.selenium.dev/documentation/?ref=blog.apify.com)  
🔖 [Playwright vs. Selenium: which one to use for web scraping?](https://blog.apify.com/playwright-vs-selenium-webscraping/)  
🔖 [Puppeteer vs. Selenium for automation](https://blog.apify.com/puppeteer-vs-selenium-for-automation/)  
🔖 [Cypress vs. Selenium: choosing the right web testing and automation framework for your project](https://blog.apify.com/cypress-vs-selenium/)

**Web scraping with Python:**

🔖 [The Apify SDK for Python](https://docs.apify.com/sdk/python/?ref=blog.apify.com)  
🔖 [Web scraping with Python](https://blog.apify.com/web-scraping-python/)  
🔖 [Web scraping with JavaScript vs. Python](https://blog.apify.com/web-scraping-javascript-vs-python-2022/)  
🔖 [Why is Python used for web scraping?](https://blog.apify.com/why-is-python-used-for-web-scraping/)

{% embed https://www.youtube.com/watch?v=r_n5_8NtHVc %}