cythonselenium 0.13
Web Scraping with Selenium / SeleniumBase / Undetected Chromedriver and Cython+Pandas
License: MIT
Keywords: SeleniumBase, Selenium, Cython, Pandas


# Fast Web Scraping with Python and Selenium / SeleniumBase / Undetected Chromedriver

## `pip install cythonselenium`

### Cython and a C/C++ compiler must be installed! The module will be compiled the first time you import it!

- Provides a structured DataFrame format for web elements, simplifying data manipulation.
- Seamlessly integrates with Selenium / SeleniumBase WebDriver for robust browser interactions.
- Capable of handling dynamic content and content within iframes, crucial for modern web applications.
- Enables execution of JavaScript functions directly on web elements.
- Offers customizable CSS selectors, waiting conditions, and method attachments for flexible web scraping.
- Incorporates explicit waits to enhance the reliability of web interactions.
- Supports repeat queries for dealing with asynchronously loaded content.
- Automatically switches to the correct frame before performing actions like clicking, ensuring seamless interaction with elements across multiple frames.
- Retrieves all elements in a single request to optimize performance and reduce the load on the web server.


```py

# Attributes:
#     driver (WebDriver): The Selenium WebDriver instance used to control the browser.
#     By (By): Selenium By class used to locate elements on a web page.
#     WebDriverWait (WebDriverWait): Selenium WebDriverWait class used for implementing explicit waits.
#     expected_conditions (expected_conditions): Module in Selenium used to set expected conditions for explicit waits.
#     queryselector (str): CSS selector used to query and return elements from the DOM. Defaults to '*' which selects all elements.
#     repeat_until_element_in_columns (optional): Specific element to be checked for its presence in the dataframe columns before stopping the query. Useful for waiting on AJAX or dynamically loaded content.
#     max_repeats (int): Maximum number of iterations to perform when checking for the presence of 'repeat_until_element_in_columns'. Defaults to 1.
#     with_methods (bool): Flag to determine if JavaScript methods should be attached to the elements in the resulting dataframe. Defaults to True.

# Methods:
#     __call__(queryselector=None, with_methods=None, repeat_until_element_in_columns=None, max_repeats=None, driver=None, By=None, WebDriverWait=None, expected_conditions=None):
#         Generates a dataframe of web elements based on the specified query selector. The dataframe can include methods attached to these elements if 'with_methods' is True. This method allows for overriding class attributes during its call for flexibility in querying different elements without needing to create multiple instances of the class.

from cythonselenium import SeleniumFrame
from seleniumbase import Driver
from selenium.webdriver.support import expected_conditions
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.by import By
import subprocess
from time import sleep

# for SeleniumBase zombie processes
subprocess.run("taskkill /IM uc_driver.exe /F", shell=True)
sleep(5)

max_col_width = 30


if __name__ == "__main__":
    driver = Driver(uc=True)
    getframe = SeleniumFrame(
        driver=driver,
        By=By,
        WebDriverWait=WebDriverWait,
        expected_conditions=expected_conditions,
        queryselector="*",
        repeat_until_element_in_columns=None,
        max_repeats=1,
        with_methods=True,
    )
    driver.get(r"https://www.whitehouse.gov/")

    df = getframe("*")
    # Use https://github.com/directvt/vtm for a line break free experience
    print(df)

    # get only links - query selector "a"
    df2 = getframe("a")

    # Selector
    df3=df2.loc[df2.aa_text=='News'] 

    # Click on the item, without worrying about switching to the correct frame
    # JS click - always works
    df3.iloc[0].js_click()
    # Selenium Click
    df3.iloc[0].se_click()


```

## Example of a DataFrame (cropped to 50 chars)

| index                                              | element                                            | frame                                              | elements_in_frame                                  | aa_relList                                         | aa_text                                            | aa_origin                                          | aa_host                                            | aa_hostname                                        | aa_pathname                                        | aa_hash                                            | aa_href                                            | aa_offsetParent                                    | aa_offsetTop                                       | aa_offsetLeft                                      | aa_offsetWidth                                     | aa_offsetHeight                                    | aa_innerText                                       | aa_outerText                                       | aa_className                                       | aa_classList                                       | aa_innerHTML                                       | aa_outerHTML                                       | aa_scrollWidth                                     | aa_scrollHeight                                    | aa_clientWidth                                     | aa_clientHeight                                    | aa_nextElementSibling                              | aa_parentNode                                      | aa_parentElement                                   | aa_firstChild                                      | aa_lastChild                                       | aa_nextSibling                                     | aa_textContent                                     | aa_rel                                             | aa_title                                           | aa_firstElementChild                               | aa_lastElementChild                                | aa_childElementCount                               | aa_target                                          | js_toString                                        | js_attachInternals                                 | js_blur                                            | js_click                                           | js_focus                                           | js_hidePopover                                     | js_showPopover                                     | js_togglePopover                                   | js_after                                           | js_animate                                         | js_append                                          | js_attachShadow                                    | js_before                                          | js_checkVisibility                                 | js_closest                                         | js_computedStyleMap                                | js_getAnimations                                   | js_getAttribute                                    | js_getAttributeNS                                  | js_getAttributeNames                               | js_getAttributeNode                                | js_getAttributeNodeNS                              | js_getBoundingClientRect                           | js_getClientRects                                  | js_getElementsByClassName                          | js_getElementsByTagName                            | js_getElementsByTagNameNS                          | js_getHTML                                         | js_hasAttribute                                    | js_hasAttributeNS                                  | js_hasAttributes                                   | js_hasPointerCapture                               | js_insertAdjacentElement                           | js_insertAdjacentHTML                              | js_insertAdjacentText                              | js_matches                                         | js_prepend                                         | js_querySelector                                   | js_querySelectorAll                                | js_releasePointerCapture                           | js_remove                                          | js_removeAttribute                                 | js_removeAttributeNS                               | js_removeAttributeNode                             | js_replaceChildren                                 | js_replaceWith                                     | js_requestFullscreen                               | js_requestPointerLock                              | js_scroll                                          | js_scrollBy                                        | js_scrollIntoView                                  | js_scrollIntoViewIfNeeded                          | js_scrollTo                                        | js_setAttribute                                    | js_setAttributeNS                                  | js_setAttributeNode                                | js_setAttributeNodeNS                              | js_setHTMLUnsafe                                   | js_setPointerCapture                               | js_toggleAttribute                                 | js_webkitMatchesSelector                           | js_webkitRequestFullScreen                         | js_webkitRequestFullscreen                         | js_appendChild                                     | js_cloneNode                                       | js_compareDocumentPosition                         | js_contains                                        | js_getRootNode                                     | js_hasChildNodes                                   | js_insertBefore                                    | js_isDefaultNamespace                              | js_isEqualNode                                     | js_isSameNode                                      | js_lookupNamespaceURI                              | js_lookupPrefix                                    | js_normalize                                       | js_removeChild                                     | js_replaceChild                                    | js_addEventListener                                | js_dispatchEvent                                   | js_removeEventListener                             | js_wheel                                           | js_change_html_value                               | se_send_keys                                       | se_find_elements                                   | se_find_element                                    | se_is_displayed                                    | se_is_enabled                                      | se_is_selected                                     | se_clear                                           | se_click                                           | se_switch_to_frame                                 | se_location_once_scrolled_into_view                | se_get_screenshot_as_file                          | se_screenshot                                      | aa_window_handle                                   | aa_window_switch                                   |
| -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- |
| 0                                                  | <seleniumbase.undetected.webelement.WebElement (se | mainframe                                          | 35                                                 |                                                    | Skip to content                                    | https://www.whitehouse.gov                         | www.whitehouse.gov                                 | www.whitehouse.gov                                 | /                                                  | #wp--skip-link--target                             | https://www.whitehouse.gov/#wp--skip-link--target  | [object HTMLBodyElement]                           | -1                                                 | -1                                                 | 1                                                  | 1                                                  | Skip to content                                    | Skip to content                                    | skip-link screen-reader-text                       | skip-link screen-reader-text                       | Skip to content                                    | <a class="skip-link screen-reader-text" href="#wp- | 69                                                 | 93                                                 | 1                                                  | 1                                                  | [object HTMLDivElement]                            | [object HTMLBodyElement]                           | [object HTMLBodyElement]                           | [object Text]                                      | [object Text]                                      | [object HTMLDivElement]                            | Skip to content                                    | <NA>                                               | <NA>                                               | <NA>                                               | <NA>                                               | <NA>                                               | <NA>                                               | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | None                                               | ()                                                 | C:\Program Files\Microsoft VS Code\seleniumpicture | ()                                                 | A9CDA196DDDB8CEBE79EAB2ECF20D18C                   | ()                                                 |
| 1                                                  | <seleniumbase.undetected.webelement.WebElement (se | mainframe                                          | 35                                                 |                                                    | News                                               | https://www.whitehouse.gov                         | www.whitehouse.gov                                 | www.whitehouse.gov                                 | /news/                                             | <NA>                                               | https://www.whitehouse.gov/news/                   | [object HTMLDivElement]                            | <NA>                                               | 289                                                | 93                                                 | 50                                                 | NEWS                                               | NEWS                                               | <NA>                                               |                                                    | News                                               | <a href="https://www.whitehouse.gov/news/">News</a | 93                                                 | 50                                                 | 93                                                 | 50                                                 | <NA>                                               | [object HTMLLIElement]                             | [object HTMLLIElement]                             | [object Text]                                      | [object Text]                                      | <NA>                                               | News                                               | <NA>                                               | <NA>                                               | <NA>                                               | <NA>                                               | <NA>                                               | <NA>                                               | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | None                                               | ()                                                 | C:\Program Files\Microsoft VS Code\seleniumpicture | ()                                                 | A9CDA196DDDB8CEBE79EAB2ECF20D18C                   | ()                                                 |
| 2                                                  | <seleniumbase.undetected.webelement.WebElement (se | mainframe                                          | 35                                                 |                                                    | Administration                                     | https://www.whitehouse.gov                         | www.whitehouse.gov                                 | www.whitehouse.gov                                 | /administration/                                   | <NA>                                               | https://www.whitehouse.gov/administration/         | [object HTMLDivElement]                            | <NA>                                               | 382                                                | 181                                                | 50                                                 | ADMINISTRATION                                     | ADMINISTRATION                                     | <NA>                                               |                                                    | Administration                                     | <a href="https://www.whitehouse.gov/administration | 181                                                | 50                                                 | 181                                                | 50                                                 | <NA>                                               | [object HTMLLIElement]                             | [object HTMLLIElement]                             | [object Text]                                      | [object Text]                                      | <NA>                                               | Administration                                     | <NA>                                               | <NA>                                               | <NA>                                               | <NA>                                               | <NA>                                               | <NA>                                               | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()                                                 | ()