Bose Web Scraping Tutorial
Tags: webscraping, python, tutorial, webscrapingtools
Published: 2023-05-13 14:01:28


![Featured](https://www.omkar.cloud/bose/assets/images/featured-0bacb14445a1b4cf367c10cd01454000.jpg)

In This tutorial you will learn about the Bose framework, a framework which provides an easier and structured way of using Selenium for web scraping. Think of it Swiss Army knife for Web Scraping

When using Selenium to scrape websites, there is usually a lot of boilerplate work involved such as:

- Downloading the appropriate Chrome driver for Selenium
- Creating the driver by specifying the driver's path, which can be challenging on Windows
- Specifying the correct ChromeOptions to make Selenium undetectable by bot protection sites
- passing profiles and user agents to Selenium
- Debugging in case of errors.

The Bose framework solves all these problems and puts an end to these nuisances for all developers. It is Django for web scrapers.

We will use the Bose framework to scrape quotes.toscrape.com, a website that lists quotes from famous authors.

Along the way, I will also share some great features of the Bose framework. So let's get started.

## Getting Started

First, let's download the Bose starter project by cloning the starter template:

```bash
git clone https://github.com/omkarcloud/bose-starter my-bose-project
```

Then, change into `my-bose-project` directory, install dependencies, and start the project:

```bash
cd my-bose-project
python -m pip install -r requirements.txt
python main.py
```

Whenever we use Selenium, we need to download the correct Selenium version for Chrome.

In past you must have visited chrome website to download the correct driver but thankfully with bose when you run the project for the first time, Bose automatically downloads the correct driver in the `build/` directory.

## Scraping quotes.toscrape.com

We are going to scrape quotes.toscrape.com for quotes and their authors.

![Dashboard](https://www.omkar.cloud/bose/assets/images/dashboard-6536dc47aa2adc244e41bcbebb81a568.png)

Write following code in src/scraper.py:

```python
from selenium.webdriver.common.by import By
from bose import BaseTask, Wait, Output

class Task(BaseTask):

    def run(self, driver):

        driver.get("https://quotes.toscrape.com/")
        els = driver.get_elements_or_none_by_selector('div.quote', Wait.SHORT)

        items = []
        for el in els:
            text = driver.get_element_text(el.find_element(By.CSS_SELECTOR, "span.text"))
            author = driver.get_element_text(el.find_element(By.CSS_SELECTOR, "small.author"))
            item = {
                "text" : text,
                "author" : author,
            }
            items.append(item)

        Output.write_finished(items)
```

This code defines a Python class Task that inherits from a BaseTask class. All Scraping Tasks must inherit from BaseTask to perform scraping.

Now in the run method of Task, we recieve driver object as a parameter, which is an instance of the BossDriver.

BossDriver extends Selenium WebDriver to add much powerful utlility methods for scraping. For example here `get_elements_or_none_by_selector` finds all elements with selector div.quote and wait upto 4 seconds (Wait.SHORT) to find them. The same thing in selenium would have been quite verbose.

Then, the method initializes an empty list items and iterates over the els list, which contains the div elements with the class name quote. For each element, it extracts the text of the quote and the author name using the find_element and get_element_text methods.

Finally, it appends a dictionary item to the items list containing the quote text and author name.

Then we use the Output Object from bose which aims to simplify reading, writing of JSON, csv files easy in selenium. Now using Output Object we write in it output/finished.json using the Output.write_finished method.

---

Now, To run the project, execute the command:

```bash
python main.py
```

You see it start scraping quotes.toscrape.com and put them in output/finished.json

![Result](https://www.omkar.cloud/bose/assets/images/result-908f901c7b8961b3d84853c103a277f9.png)

Furthermore, the /tasks/1/ directory will also be generated.

Web scraping can often be fraught with errors, such as incorrect selectors or pages that fail to load. When debugging with raw Selenium, you may have to sift through logs to identify the issue. Fortunately, Bose makes it simple for you to debug by storing information about each run.

If you observe the tasks/1/ directory, you will notice that it contains three files, which are listed below:

### `task_info.json`

It contains information about the task run such as duration for which the task run, the ip details of task, the user agent, window_size and profile which used to execute the task.

![task info](https://www.omkar.cloud/bose/assets/images/task-info-1ad8d89552138e2edc900434144dfbe0.png)

### `final.png`

This is the screenshot captured before driver was closed.

![final](https://www.omkar.cloud/bose/assets/images/final-d2ca24d2717d17576eb8233ad0cd2b10.png)

### `page.html`

This is the html source captured before driver was closed. Very useful to know in case your selectors failed to select elements.

![Page](https://www.omkar.cloud/bose/assets/images/page-cffce10976b4bf201b49a479c2340075.png)

### `error.log`

In case your task crashed due to exception we also store error.log which contains the error due to which the task crashed. This is very helful in debugging.

![error log](https://www.omkar.cloud/bose/assets/images/error-log-9ebb09dca133b2d7df1ae6cfc67df909.png)

## Exception handling

In bose, when an exception occurs in a scraping task, the browser will remain open instead of immediately closing. This is useful for debugging purposes, as it allows you to see the live browser state when the exception occurred.

For example, if we replace the code in scraper.py with the following code which selects a non exisiting selector causing an exception to be raised and run it:

```python
from selenium.webdriver.common.by import By

from bose import BaseTask, Wait, Output

class Task(BaseTask):

    def run(self, driver):
        driver.get("https://quotes.toscrape.com/")
        els = driver.get_elements_or_none_by_selector('div.some-not-exisiting-selector', Wait.SHORT)

        items = []

        for el in els:
            text = driver.get_element_text(el.find_element(By.CSS_SELECTOR, "span.text"))
            author = driver.get_element_text(el.find_element(By.CSS_SELECTOR, "small.author"))
            item = {
                "text" : text,
                "author" : author,
            }
            items.append(item)

        Output.write_finished(items)
```

You will notice that the browser does not close, and Bose prompts you to press enter to close the browser. This feature is very handy when you are trying to debug your code in the browser when the exception occurred.

![error prompt](https://www.omkar.cloud/bose/assets/images/error-prompt-cdcbd3f0419dd68b5d8264c2c96661f6.png)

## Browser Configuration

Bose makes it easy to configure the Selenium driver with different options, such as

- which profile to use
- which user agent to use
- which window size to use

You can easily configure these options using the BrowserConfig class. For example, here's how you can configure the driver to use Chrome 106 user agent, window size 1280x720, and profile 1:

```python
from selenium.webdriver.common.by import By
from bose import BaseTask, BrowserConfig, UserAgent,WindowSize

class Task(BaseTask):
    browser_config = BrowserConfig(user_agent=UserAgent.user_agent_106, window_size=WindowSize.window_size_1280_720, profile=1)

    def run(self, driver):
        driver.get("https://quotes.toscrape.com/")
```

In this example, we set the `BrowserConfig` using the `browser_config` property of the `Task` class.

## Using Undetected Driver

Bose also supports the `undetected_driver` library, which provides a robust driver to help evade detection by anti-bot services like Cloudflare. Although it is slower to start, it is much less detectable. To use it, pass the `use_undetected_driver` option to `BrowserConfig`, like so:

```python
from selenium.webdriver.common.by import By
from bose import BaseTask, BrowserConfig, UserAgent,WindowSize

class Task(BaseTask):
    browser_config = BrowserConfig(use_undetected_driver=True, user_agent=UserAgent.user_agent_106, window_size=WindowSize.window_size_1280_720, profile=1,)
```

## Outputting Data in Bose

Bose provides great support to easily output data as CSV, Excel, or JSON using the `Output` class. To use it, call the `write` method for the type of file you want to save.

All data will be saved in the `output/` folder:

```python
from bose import Output

data = [
    {
        "text": "\u201cThe world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.\u201d",
        "author": "Albert Einstein"
    },
    {
        "text": "\u201cIt is our choices, Harry, that show what we truly are, far more than our abilities.\u201d",
        "author": "J.K. Rowling"
    }
]

Output.write_json(data, "data.json")
Output.write_csv(data, "data.csv")
Output.write_xlsx(data, "data.xlsx")
```

## Using LocalStorage

Just like how modern browsers have a local storage module, Bose has also incorporated the same concept in its framework.

You can import the LocalStorage object from Bose to persist data across browser runs, which is extremely useful when scraping large amounts of data.

The data is stored in a file named `local_storage.json` in the root directory of your project. Here's how you can use it:

```python
from bose import LocalStorage

LocalStorage.set_item("pages", 5)
print(LocalStorage.get_item("pages"))
```

## Conclusion

In summary, Bose is an excellent framework that simplifies the boring parts of Selenium and web scraping. We encourage you to read the reference of BossDriver at [Boss Driver](https://www.omkar.cloud/bose/docs/reference/boss-driver/), which is an extended version of Selenium that adds some great methods to help you in scraping.