Selenium in Google Colab: howTo Start with?
Tags: webdev, beginners, tutorial, python
Published: 2024-01-23 09:48:05

this is the beginning of a tiny - manual of what can go wrong ever  - if you want to run little scraper that does all that - just read on. 

btw: this is a very rough manual - lots of  you can add thoughts ideas and more... so come on - add your ideas ...

To use Selenium in Google Colab, you first need to install the necessary packages and set up the WebDriver. Here's a simple example of a Selenium web scraper in Google Colab:




# `Install necessary packages
!pip install selenium
!apt-get update
!apt install -y chromium-chromedriver

# Set up WebDriver
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options

chrome_options = Options()
chrome_options.add_argument('--headless')  # Run Chrome in headless mode (no GUI)
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')

# Specify the path to the chromedriver executable
chrome_driver_path = "/usr/lib/chromium-browser/chromedriver"

# Create a WebDriver instance
driver = webdriver.Chrome(chrome_driver_path, options=chrome_options)

# Example: Open Google and search for a term
search_term = "Hello, Google Colab!"
driver.get("https://www.google.com/")
search_box = driver.find_element("name", "q")
search_box.send_keys(search_term)
search_box.send_keys(Keys.RETURN)

# Wait for a few seconds to let the results load
driver.implicitly_wait(5)

# Print the title of the page
print("Page title:", driver.title)

# Close the browser window
driver.quit()

`
**important**:  Make sure to adjust the chrome_driver_path based on the version of the Chrome browser installed in the Colab environment.


**my guess**: well  i guess that i have to take care for the adjustment of the "chrome _driver_path"  based on the version of the chrome-brwoser that is installed in the colab environment 

see what i ve got back - how  to proceed?

`Requirement already satisfied: wsproto>=0.14 in /usr/local/lib/python3.10/dist-packages (from trio-websocket~=0.9->selenium) (1.2.0)
Requirement already satisfied: pysocks!=1.5.7,<2.0,>=1.5.6 in /usr/local/lib/python3.10/dist-packages (from urllib3[socks]<3,>=1.26->selenium) (1.7.1)
Requirement already satisfied: h11<1,>=0.9.0 in /usr/local/lib/python3.10/dist-packages (from wsproto>=0.14->trio-websocket~=0.9->selenium) (0.14.0)
Hit:1 https://cloud.r-project.org/bin/linux/ubuntu jammy-cran40/ InRelease
Hit:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  InRelease
Hit:3 http://archive.ubuntu.com/ubuntu jammy InRelease
Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease
Hit:5 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:6 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Hit:7 https://ppa.launchpadcontent.net/c2d4u.team/c2d4u4.0+/ubuntu jammy InRelease
Hit:8 https://ppa.launchpadcontent.net/deadsnakes/ppa/ubuntu jammy InRelease
Hit:9 https://ppa.launchpadcontent.net/graphics-drivers/ppa/ubuntu jammy InRelease
Hit:10 https://ppa.launchpadcontent.net/ubuntugis/ppa/ubuntu jammy InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
chromium-chromedriver is already the newest version (1:85.0.4183.83-0ubuntu2.22.04.1).
0 upgraded, 0 newly installed, 0 to remove and 33 not upgraded.
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-87c7a8d826d3> in <cell line: 19>()
     17 
     18 # Create a WebDriver instance
---> 19 driver = webdriver.Chrome(chrome_driver_path, options=chrome_options)
     20 
     21 # Example: Open Google and search for a term

TypeError: WebDriver.__init__() got multiple values for argument 'options'
`
- how  to proceed?



It seems like there's a version mismatch or an issue with the webdriver.Chrome initialization in your code. The error suggests that there's a problem with the arguments being passed to the WebDriver constructor.

Try modifying the webdriver.Chrome initialization line like this:


`# Create a WebDriver instance`


driver = webdriver.Chrome(executable_path=chrome_driver_path, options=chrome_options)


This ensures that the chrome_driver_path is explicitly provided as the executable_path argument. Please update your code accordingly and see if it resolves the issue.
If the issue persists, there might be a compatibility problem between the ChromeDriver version and the Chrome browser version. In such cases, you may need to download the appropriate version of ChromeDriver that matches the installed version of Chrome in your Colab environment.

we can download ChromeDriver from the official website: https://sites.google.com/chromium.org/driver/


After downloading, upload the ChromeDriver executable to your Colab environment and adjust the chrome_driver_path accordingly.