selenium-support 0.0.1
Usefull function for Selenium umbrella project
License: Apache 2.0
Requires Python: >=3.8
Keywords: Selenium support BMP browsermobproxy web_driver driver Chrome Firefox har download  wait wait_page_loaded click click_sync wait_chrome_file_finished_downloades wait_for_display

## selenium-support

I’ve started to use Selenium’s related products and how found that documentation is misleading, it doesn’t promote 
best practice. 

Moreover, many components has memory/resource leaks that are not fixed for years.
 
So I’ve created utility project that encapsulated my “fixes” to Selenium and promotes best practice.

I will list some of the capabilities of my library:

* create/destroy BmpDaemon(aka browsermobproxy.Server).
* create/destroy BmpProxy (aka browsermobproxy.Client).
* create/destroy SeleniumWebDriver (for example selenium.webdriver.Chrome.webdriver). (Can be any supported browser).
* Taking screenshots.
* Preparing browser’s data-dir for usage.
* Enabling browser to download files.
* Capturing network in har format.
* Waiting for page to load.
* Synchronous click (on the button).
* Wait for Google Chrome to finish to download file (Chrome specific).
* Wait for display.

### Getting Help


### QuickStart
```bash
python -m pip install -U selenium-support
```


### Installing from Github

```bash
python -m pip install -U https://github.com/alex-ber/selenium-support/archive/master.zip
```
Optionally installing tests requirements.

```bash
python -m pip install -U https://github.com/alex-ber/selenium-support/archive/master.zip#egg=alex-ber-utils[tests]
```

Or explicitly:

```bash
wget https://github.com/alex-ber/selenium-support/archive/master.zip -O master.zip; unzip master.zip; rm master.zip
```
And then installing from source (see below).


### Installing from source
```bash
python -m pip install -r req.txt # only installs "required" (relaxed)
```
```bash
python -m pip install . # only installs "required"
```
```bash
python -m pip install .[tests] # installs dependencies for tests
```

#### Alternatively you install install from requirements file:
```bash
python -m pip install -r requirements.txt # only installs "required"
```
```bash
python -m pip install -r requirements-tests.txt # installs dependencies for tests
```


### Using Docker
`alexberkovich/selenium_support:latest`  contains all `selenium_support` dependencies.
This Dockerfile is very simple, you can take relevant part for you and put them into your Dockerfile.

##
Alternatively, you can use it as base Docker image for your project and add/upgrade 
another dependencies as you need.

For example:

```Dockerfile
FROM alexberkovich/selenium_support:latest

COPY requirements.txt etc/requirements.txt

RUN set -ex && \
    #latest pip,setuptools,wheel
    pip install --upgrade pip setuptools wheel && \
    pip install selenium_support 
    pip install -r etc/requirements.txt 

CMD ["/bin/sh"]
#CMD tail -f /dev/null
```

where `requirements.txt` is requirements for your project.

##

From the directory with setup.py
```bash
python setup.py test #run all tests
```

or

```bash

pytest
```

## Installing new version
See https://docs.python.org/3.1/distutils/uploading.html 

## Installing new version to venv

```bash
python38 -m pip uninstall --yes selenium-support
python38 setup.py clean sdist bdist_wheel
python38 -m pip install --find-links=./dist selenium-support==0.1
```


```bas  h
python setup.py sdist upload
```

## Requirements


selenium-support requires the following modules.

* Python 3.8+
* selenium
* browsermob-proxy
* psutil


# Changelog

Useful functions for Selenium umbrella project.

All notable changes to this project will be documented in this file.

\#https://pypi.org/manage/project/selenium-support/releases/

## [Unrelased]


## [0.0.1] - 18/04/2021
### Added
All functions below are listed in `alexber.seleniumsupport` package.  
See https://alex-ber.medium.com/selenium-support-19330843c63a for more details.

* `BMPDaemon` is  context-manager that responsible for starting BMP Daemon. In the exit from the code block 
inside context-manager, BMP Daemon will be stopped, see `closeBmpDaemon()`.

See https://github.com/AutomatedTester/browsermob-proxy-py/blob/master/browsermobproxy/server.py#L59 
for undocumented values.

If you have multiple application that uses BMP Daemon, you have 2 basic choices:

*  Start BMP Daemon outside of scope of these application (maybe as 3 application or as OS Daemon or just manually) 
in some predefined port (the default is 8080) and write your application code that assumes that BMP Daemon is up 
and running.

* Each application will start BMP Daemon on different port.

I want to *emphasize*, technically it is sufficient to have only 1 running BMP Daemon for all application 
(You will create BMP Proxy per application).

Personally, I’ve found the second option easier to manage — namely to have multiple BMP Daemons, one per application. 

This is indeed waste of resources, but the application lifecycle is much easier to manage and you don’t 
have some external dependencies. Note, however, that in such case you should, at least in one application, 
explicitly provide the port number. It is better that they will far away one from another, because BMP Proxy is 
created as next port number.

This context-manager also worries to close BMP Daemon. See `closeBmpDaemon()` below. 

If you use it as regular function, this will not happen. 

* `closeBmpDaemon` If you’re using BMPDaemon as context-manager, it will worry to close the BMP Daemon. 
If you want to do it yourself, you can call `closeBmpDaemon()` function.
See https://github.com/AutomatedTester/browsermob-proxy-py/issues/8#issuecomment-679150656

* `BrowserDataDir` -  this context manager can be used for reuse of user data dir. It unzips file template 
(*path/to/chrome_data_dir.zip* in our case) to work_dir (*logs* in our case).
It returns directory  with extracted content from template.

If you want your browser to work with some predefined user data dir ("profile" in Mozilla Firefox).

For more on Google Chrome see https://stackoverflow.com/a/55636113/1137529, for 
Mozilla Firefox see https://firefox-source-docs.mozilla.org/testing/geckodriver/CrashReports.html.

For example, on Windows you can create shortcut: 
`"C:\Program Files (x86)\Google\Chrome\Application\chrome.exe" --user-data-dir=C:\tmp\chrome_profile`. 

Click on it to create all internal folders and than pass `C:\tmp\chrome_profile` as `--user-data-dir` to 
Google Chrome option's argument.

* `BMPPRoxy` -  this context manager is designed to create BMP Proxy on new port. It assumes that BMPDaemon is 
already up. 

It doesn't require BMPDaemon object, but only some of it's parameters.

By default it assumes, that BMPDaemon is running on localhost. By default it also assumes that it is 
running on port 8080, you may want to override this value.

This context manager returns BMP Proxy.

In the exit from the code block inside context-manager, it closes BMP Proxy.

* `SeleniumWebDriver` - this context manager is designed to create Web Driver. It assumes that BMPDaemon is already up. 
You may pass BMPPRoxy if you want to use it. It doesn't require BMPDaemon object. It returns Web Driver. 
In the exit from the code block inside context-manager, it close Web Driver, see 
`closeSeleniumWebDriver()` for the details.

*Note:* Web Driver consists of Python wrapper and some executable file (chromedriver.exe, for example). 
The term 'Web Driver' (confusingly) refers to both.

*Note*: There is some complexity on how to create general-purpose context-manager that will work for any Web Driver. 
For example, Google Chrome and Mozilla Firefox has different option's class. 

My implementation is inspired by 
https://github.com/clemfromspace/scrapy-selenium/blob/develop/scrapy_selenium/middlewares.py

* `closeSeleniumWebDriver()` If you're using SeleniumWebDriver as context-manager, it will worry to close Web Driver. 
If you want to do it yourself, you can call closeSeleniumWebDriver() function.

You may wonder why call to web_driver.quit() is not sufficient. 
*Note* (**be aware!**): you shouldn't confuse this call with `web_driver.close()`. 
The last call will close only the tab and not browser itself.

Why web_driver.quit() is not enough?

I have noticed, that sometimes, when exception is raised, but not always, Google Chrome browser and/or 
chromedriver.exe remain residents in memory.

When I've added the same logic to close all ancestor processes that was opened in Web Driver initialization, 
this never happens again.

See `closeBmpDaemon()`

See https://github.com/AutomatedTester/browsermob-proxy-py/issues/8#issuecomment-679150656

* `Screenshot` is designed to be used as context-manager.

If you want API for simple function call, please use `save_screenshot()`.

You may want to guard piece of you code with this context-manager. It is required that you instantiated `web_driver` 
first.

* `save_screenshot()` is regular function API. If you want a context-manager, please use `Screenshot`.

Maybe, you have some try-finally block and you want when you've caught an exception from `web_driver` to get 
screenshot in order to understand better what went wrong. 

Personally, I prefer to use Screenshot, but in some complex scenario you may want to have better control.

*Note*: you can define `dd['browser_download_folder']` as following
`dd['files']['browser_download_folder'] = str(Path(Path.home(), 'Downloads'))`

See see https://stackoverflow.com/questions/37480641/how-do-i-view-the-screenshot-available-via-screen

* `enable_chrome_download()` is Google Chrome specific function.

In Google Chrome in headless mode download is disabled by default. 
It's a "feature", for security. If you want to enable download you can use this function. 


See https://stackoverflow.com/questions/45631715/downloading-with-chrome-headless-and-selenium for more details.
See also https://bugs.chromium.org/p/chromium/issues/detail?id=696481.


* `set_new_har()` is convenient wrapper to `bmp_proxy.new_har()` with `capture*` parameters. 

It is used to get network transmission. For example, if you want to get response body. 

See https://medium.com/@jiurdqe/how-to-get-json-response-body-with-selenium-amd-browsermob-proxy-71f10335c66 
for more details.

Note:
 
1. You should call `bmp_proxy.wait_for_traffic_to_stop(5, 70)` first (you can change parameters).
Here we're waiting (5 ms) for network traffic to stop (with timeout of 70 ms). It is needed to ensure that BMP Proxy 
has recorded all network traffic (up to this point).

2. You should use `for ent in reversed(d['log']['entries']):`, note reversed builtin (it is absent in the link above).
If you have multiple calls to same URL, you should look on last result, so you should reverse the order of log entries.


This call is equivalent to click on *"Start recording"* on *Network* tab of *Page Inspector*.

* `wait_page_loaded()` is helper function to ensure that some basic elements of the page, such as title are loaded.

Usage example:

```python
from alexber.seleniumsupport import wait_page_loaded
wait = WebDriverWait(web_driver, timeout=70, poll_frequency=1)
wait_page_loaded(wait)
```

* `click_sync()`- sometimes calling click() on WebElement raise some weird exception. 
The best practice will be to use `wait.until(EC.element_to_be_clickable((By.XPATH, 'xpath')))`. 
This is "dirty" solution that make synchronous call (by using JavaScript) on `WebElement.click()` 

(WebElement is typically button). See https://stackoverflow.com/a/58378714/1137529 for more details.

Usage example:
```python

from alexber.seleniumsupport import click_sync
wait = WebDriverWait(web_driver, timeout=70, poll_frequency=1)
click_sync(web_driver,
                 wait.until(EC.element_to_be_clickable(
                    (By.XPATH, 'xpath'))))
```

See see https://stackoverflow.com/a/58378714/1137529 

* `wait_chrome_file_finished_downloades() `is Google Chrome specific function.

It works directly with file system. You should know the file_name beforehand. 
It relies on Google Chrome following internal mechanism:

* When Google Chrome downloads file, it has extension ".crdownload". 
* When downloads is finished it Google Chrome rename the file removing this extension.

This function doesn't rely on Google Chrome's downloads status.

On Windows, we can see wrong state of the filesystem. This is the reason for rerty-with-sleep mechanism.


See 
https://docs.python.org/3/library/os.html#os.scandir

https://msdn.microsoft.com/en-us/library/windows/desktop/aa364418(v=vs.85).aspx

https://msdn.microsoft.com/en-us/library/windows/desktop/aa364428(v=vs.85).aspx

*Note:* if the file is very bigger (more than 200MB) you may need to increase retries number.

*` wait_for_display()` - Sometimes, we want to make Selenium Web driver wait until elements style attribute has changed. 

This is usefull for dynamically loaded material. 

For example, we want to wait for the display style to change to none (or to "inline-block" or some other value). 

See https://stackoverflow.com/questions/34915421/make-selenium-driver-wait-until-elements-style-attribute-has-changed

Usage example:
```python
from alexber.seleniumsupport import wait_for_display
wait = WebDriverWait(web_driver, timeout=70, poll_frequency=1)
wait.until(wait_for_display((By.XPATH, 'xpath')))
```

See https://stackoverflow.com/questions/34915421/make-selenium-driver-wait-until-elements-style-attribute-has-changed

<!--
### Added 
### Changed
### Removed
-->