Meet Bose Framework: 🚀 Your Swiss Army Knife as a Ninja Scraper ✨
Tags: webscraping, python, selenium, webdev
Published: 2023-06-12 15:59:06


## 💡 What is Bose Framework?

Bose Framework is a framework built for selenium developers packed with best practices from experienced bot developers to help create **undetectable**, **low boilerplate** and **easy to debug** bots. 🤖

It helps you scrape or automate target websites with ease and gives you the mental powers of experienced bot developers at your fingertips, saving you **hours** of development time. 👨🏻‍💻

## 🏆 Top Feautres 

Bose is a battery packed framework that implements following feautres to help you create bots faster, saving your valuable Development Time. ✨

- **Go Low Boilerplate with Bose**
Launching a browser using Bare Selenium requires writing a significant amount of code:

```python
from selenium import webdriver

driver_path = 'path/to/chromedriver'

driver = webdriver.Chrome(executable_path=driver_path)

driver.get('https://www.example.com')

driver.quit()
```

With Bose, you can launch browser in few lines of code without having to worry about specifying paths:

```python
from bose import *
        
class Task(BaseTask):
    def run(self, driver):
        driver.get('https://www.example.com')
```

- **Configure Profile, Window Size, and User Agent with a Single Line of Code**

In bare Selenium, if you want to configure options such as the profile, user agent, or window size, it requires writing a lot of code, as shown below:

```python
from selenium.webdriver.chrome.options import Options
from selenium import webdriver

driver_path = 'path/to/chromedriver.exe'

options = Options()

profile_path = '1'

options.add_argument(f'--user-data-dir={profile_path}')

user_agent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.37")'
options.add_argument(f'--user-agent={user_agent}')

window_width = 1200
window_height = 720
options.add_argument(f'--window-size={window_width},{window_height}')

driver = webdriver.Chrome(executable_path=driver_path, options=options)
```

With Bose, you can specify them in a single line of code using predefined variables. Here's an example:

```python

class Task(BaseTask):
    browser_config = BrowserConfig(user_agent=UserAgent.user_agent_106, window_size=WindowSize.window_size_1280_720, profile=1)
```

- **Exception Handling that every Bot Developer wants**

Bose also addresses a common frustration with Selenium - when an exception occurs, the browser crashes and closes automatically. This behavior is not desirable for bot developers and makes debugging hard. 

Bose, on the other hand, keeps the browser open in the event of an exception, allowing for easier debugging of the problem.

![error prompt](https://www.omkar.cloud/bose/assets/images/error-prompt-83de79e560f129197afb9f831d388383.png)

- **Remembers the Past Runs**

Let's say you went to drink coffee while your bot was running, and when you came back, you noticed that the bot had closed.

As a developer, you might have an itch to see the last screenshot taken before the browser was closed or to know how much time the bot took to complete its task.

Well, Whenever you run Bose, it automatically stores important details such as 
- the screenshot taken before closing
- the time it took to run the bot
- any errors that occurred.

After, each run a new folder is created in `tasks` folder which contains three files, listed below:

### `task_info.json`
It contains information about the task run such as 

- duration for which the task run, 
- the ip details of task 
- the user agent
- window_size 
- profile 

![task info](https://www.omkar.cloud/bose/assets/images/task-info-1ad8d89552138e2edc900434144dfbe0.png)

### `final.png`
This is the screenshot captured before driver was closed. 

![final](https://www.omkar.cloud/bose/assets/images/final-d2ca24d2717d17576eb8233ad0cd2b10.png)


### `page.html`
This is the html source captured before driver was closed. Very useful to know in case your selectors failed to select elements.


![Page](https://www.omkar.cloud/bose/assets/images/page-cffce10976b4bf201b49a479c2340075.png)


### `error.log`
In case your task crashed due to exception we also store error.log which contains the error due to which the task crashed. This is very helful in debugging.

![error log](https://www.omkar.cloud/bose/assets/images/error-log-9ebb09dca133b2d7df1ae6cfc67df909.png)

- **Built to defeat Cloudflare, Perimeterx by default**

Multi Million Dollar Companies protects their valuable data with the help of Cloudflare and PerimeterX.

Now, you might be wondering how to bypass systems like Cloudflare and PerimeterX. Well, a brilliant Developer named [Leon](https://github.com/ultrafunkamsterdam) created a [ChromeDriver](https://github.com/ultrafunkamsterdam/undetected-chromedriver) that has excellent support for bypassing **all major bot detection systems** such as Distil, Datadome, Cloudflare, and others. 

To use it, simply pass the `use_undetected_driver` option to the `BrowserConfig` in your code, as shown below:

```python
from bose import BaseTask, BrowserConfig

class Task(BaseTask):
    browser_config = BrowserConfig(use_undetected_driver=True)
```

- **Output Data in CSV or JSON with a Single Line of Code**

Outputting data in CSV or JSON requires a significant amount of imperative code, as shown below:

```python
import csv
import json

def write_json(data, filename):
    with open(filename, 'w') as fp:
        json.dump(data, fp, indent=4)

def write_csv(data, filename):
    with open(filename, 'w', newline='', encoding='utf-8') as csvfile:
        fieldnames = data[0].keys()  # get the fieldnames from the first dictionary
        writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
        writer.writeheader()  # write the header row
        writer.writerows(data)  # write each row of data

data = [
    {
        "text": "\u201cThe world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.\u201d",
        "author": "Albert Einstein"
    },
    {
        "text": "\u201cIt is our choices, Harry, that show what we truly are, far more than our abilities.\u201d",
        "author": "J.K. Rowling"
    }
]

write_json(data, "data.json")
write_csv(data, "data.csv")
```

Bose simplifies these complexities by encapsulating them in the Output module for easy reading and writing of data:

```python
from bose import Output

data = [
    {
        "text": "\u201cThe world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.\u201d",
        "author": "Albert Einstein"
    },
    {
        "text": "\u201cIt is our choices, Harry, that show what we truly are, far more than our abilities.\u201d",
        "author": "J.K. Rowling"
    }
]

Output.write_json(data, "data.json")
Output.write_csv(data, "data.csv")
```

- **Run the Same Code Everywhere, whether it's on Mac, Linux, or Windows. Forget the need to change driver paths.**

Bose simplifies cross-platform development by abstracting away the differences between operating systems such as Windows, Mac, and Linux. 

You no longer need to specify driver paths specific to each OS when launching browser.

![](https://ded9.com/wp-content/uploads/2022/08/mac_linux_windows_banner.png)

- **Adds Powerful Methods to Supercharge Bot Development**

The driver you receive in the **`run`** method of the Bose Task is an extended version of Selenium that adds powerful methods to make working with Selenium much easier. Some of the popular methods added to the Selenium driver by Bose Framework are:

| METHOD | DESCRIPTION |
| --- | --- |
| get_by_current_page_referrer(link, wait=None) | simulate a visit that appears as if you arrived at the page by clicking a link. This approach creates a more natural and less detectable browsing behavior. |
| js_click(element) | enables you to click on an element using JavaScript, bypassing any interceptions(ElementClickInterceptedException) from pop-ups or alerts |
| get_cookies_and_local_storage_dict() | returns a dictionary containing "cookies" and "local_storage” |
| add_cookies_and_local_storage_dict(self, site_data) | adds both cookies and local storage data to the current web site |
| organic_get(link, wait=None) | visits google and then visits the “link” making it less detectable |
| local_storage | returns an instance of the LocalStorage module for interacting with the browser's local storage in an easy to use manner |
| save_screenshot(filename=None) | save a screenshot of the current web page to a file in tasks/ directory |
| short_random_sleep() and long_random_sleep(): | sleep for a random amount of time, either between 2 and 4 seconds (short) or between 6 and 9 seconds (long) |
| get_element_or_* [eg: get_element_or_none, get_element_or_none_by_selector, get_element_by_id, get_element_or_none_by_text_contains,] | find web elements on the page based on different criteria. They return the web element if it exists, or None if it doesn't. |
| is_in_page(target, wait=None, raise_exception=False) | checks if the browser is in the specified page |

In simple words, Bose is an excellent framework that simplifies the boring parts of Selenium and web scraping.

## 🚀 Get Started with Bose

Now, let's see how you can have the magic of Bose at your finger tips. 

Start by Cloning the Template

```bash
git clone https://github.com/omkarcloud/bose-starter my-bose-project
```

Then change into that directory, install dependencies, open vscode, and start the project:

```bash
cd my-bose-project
python -m pip install -r requirements.txt
code .
python main.py
```

The first run will take some time as it downloads the chrome driver executable, subsequent runs will be fast.

Once started it will scrape data from [quotes.toscrape.com](https://quotes.toscrape.com/) and store the results in /output/finished.json

<!-- TODO: USe JSON ONE  -->
![](https://www.omkar.cloud/bose/assets/images/final-d2ca24d2717d17576eb8233ad0cd2b10.png)

## ✨ Upcoming Features in V2 to supercharge 🔋 Bot Development 

- Kubernetes Integration to help you scrape data at Google’s Scale [Priority]
- Save Storage by storing the profile in a single JSON file by storing cookies and local storage for the website. [Priority]
- Provide a temporary email service [Priority]
- Purchase hundreds of pre-created Google and Microsoft accounts [Priority]
- Built-in IP rotation for requests [Priority]
- Captcha Solving implemented right into Bose [Priority]
- Generate names, emails, usernames, etc., for users in countries such as India, Russia, Europe, China, and America.

## 📚 Summary

Simply put, Bose empowers you to effortlessly automate or scrape your Target Website and its content with the ease of cutting butter with a knife. 

---

👋 Hi Reader, 

What do you think? Do you see the value of **Bose Framework**?

Share your thoughts in the comments and I will reply to every single comment.