Selenium Proxy Integration With Oxylabs
Tags: proxy, web, selenium, scraping
Published: 2022-06-17 09:30:00


![selenium proxy integration](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y4f8h4vh1le9202t7s1r.png)

If you’re into web scraping, we believe you’ve heard of [Selenium](https://www.selenium.dev/) – an open-source tool that helps automate web browser interactions for website testing and more.

This testing framework is especially useful when you need to interact with a browser to perform a number of tasks, such as clicking on buttons, scrolling, etc. But even if primarily Selenium is used for website testing, it can also be used for web scraping because it helps locate the required public data on a website. 

So, in this article, let’s go through the Selenium integration process with Oxylabs’ [Residential Proxies](https://oxy.yt/YtYC) for a smooth web scraping process.

**How to integrate Oxylabs’ proxies with Selenium?**

Here, we explain how to integrate Oxylabs’ Residential Proxies with Selenium in Python. Note that the required version of Python is Python 3.5 (or higher).

**Setting up Selenium**

Firstly, you’ll need to install [Selenium Wire](https://github.com/wkeeling/selenium-wire) to extend Selenium’s Python bindings because using the default Selenium module for implementing proxies that require authentication makes the whole process complicated. You can do it using the pip command:

```
pip install selenium-wire
```

Another recommended package for this integration is webdriver-manager. It’s a package that simplifies the management of binary drivers for different browsers. In this case, there’s no need to manually download a new version of a web driver after each update.
You can install the webdriver-manager using the pip command as well:

```
pip install webdriver-manager
```

**Proxy authentication**

Once everything is set up, you can move on to the next part – proxy authentication. For proxies to work, you’ll be prompted to specify your account credentials.

```
USERNAME = "your_username" PASSWORD = "your_password" ENDPOINT = "pr.oxylabs.io:7777"
```

You’ll need to adjust `your_username` and `your_password` fields with the username and password of your proxy user.

**Testing proxy server connection**
If you need to check if the proxy is working, you should visit ip.oxylabs.io. If everything is working correctly, it will return an IP address of a proxy that you’re using.

```
try: driver.get("https://ip.oxylabs.io/") return f'\nYour IP is: {re.search(r"[0-9].{2,}", driver.page_source).group()}' finally: driver.quit()
```

**Full code for Oxylabs’ Residential Proxies integration with Selenium**

```
import re
from typing import Optional

from seleniumwire import webdriver
# A package to have a chromedriver always up-to-date.
from webdriver_manager.chrome import ChromeDriverManager

USERNAME = "your_username"
PASSWORD = "your_password"
ENDPOINT = "pr.oxylabs.io:7777"


def chrome_proxy(user: str, password: str, endpoint: str) -> dict:
    wire_options = {
        "proxy": {
            "http": f"http://{user}:{password}@{endpoint}",
            "https": f"http://{user}:{password}@{endpoint}",
        }
    }

    return wire_options


def execute_driver():
    options = webdriver.ChromeOptions()
    options.headless = True
    proxies = chrome_proxy(USERNAME, PASSWORD, ENDPOINT)
    driver = webdriver.Chrome(
        ChromeDriverManager().install(), options=options, seleniumwire_options=proxies
    )
    try:
        driver.get("https://ip.oxylabs.io/")
        return f'\nYour IP is: {re.search(r"[0-9].{2,}", driver.page_source).group()}'
    finally:
        driver.quit()


if __name__ == "__main__":
    print(execute_driver())
```



**Wrapping it up**

All in all, Selenium is a great tool for public web scraping, especially when learning the basics. Plus, if you’re using high-quality and reliable proxies, public web scraping becomes even more efficient. 
Got any questions about this post or generally about integrations with our proxies? You can [reach out](https://oxy.yt/Dytz) to us any time!