Scrape Brave Images with Python
Tags: webscraping, tutorial, python, programming
Published: 2022-11-15 15:42:02

- <a href="#intro">Intro</a>
- <a href="#what_will_be_scraped">What will be scraped</a>
- <a href="#brave">What is Brave Search</a>
- <a href="#fullcode">Full Code</a>
- <a href="#preparation">Preparation</a>
- <a href="#code_explanation">Code Explanation</a>
- <a href="#output">Output</a>
- <a href="#links">Links</a>


<h2 id="intro">Intro</h2>

Currently, we don't have an API that supports extracting data from Brave Search.

This blog post is to show you way how you can do it yourself with provided DIY solution below while we're working on releasing our proper API.

The solution can be used for personal use as it doesn't include the [Legal US Shield](https://serpapi.com/#features) that we offer for our paid [production and above plans](https://serpapi.com/pricing) and has its limitations such as the need to bypass blocks, for example, CAPTCHA.

You can check our public roadmap to track the progress for this API: [[New API] Brave Search](https://github.com/serpapi/public-roadmap/issues/322)


<h2 id="what_will_be_scraped">What will be scraped</h2>

![wwbs-brave-images](https://user-images.githubusercontent.com/81998012/197317672-392a9605-5e5f-4764-91e2-863f209100df.png)


<h2 id="brave">What is Brave Search</h2>

The previous Brave blog post previously described [what is Brave search](https://serpapi.com/blog/scrape-brave-search-organic-results-with-python/). For the sake of non-duplicating content, this information is not mentioned in this blog post.


<h2 id="fullcode">Full Code</h2>

If you don't need explanation, have a look at [full code example in the online IDE](https://replit.com/@chukhraiartur/blog-brave-search-images#main.py).

```python
import requests, json


def scrape_brave_images():
	# https://docs.python-requests.org/en/master/user/quickstart/#passing-parameters-in-urls
	params = {
		'q': 'dune 2021', 		# query 
		'source': 'web',		# source 
		'size': 'All',			# size (Small, Medium, Large, Wallpaper) 
		'_type': 'All',			# type (Photo, Clipart, AnimatedGifHttps, Transparent) 
		'layout': 'All',		# layout (Square, Tall, Wide) 
		'color': 'All',			# colors (Monochrome, ColorOnly, Red etc) 
		'license': 'All',		# license (Public, Share, Modify etc)
		'offset': 0
	}

	# https://docs.python-requests.org/en/master/user/quickstart/#custom-headers
	headers = {
		'content-type': 'application/json',
		'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'
	}

	data = []
	old_page_result = []

	while True:
		html = requests.get('https://search.brave.com/api/images', headers=headers, params=params).json()
		
		new_page_result = html['results']

		if new_page_result == old_page_result:
			break
		
		for result in new_page_result:
			data.append({
				'title': result.get('title'),
				'link': result.get('url'),
				'source': result.get('source'),
				'width': result.get('properties').get('width'),
				'height': result.get('properties').get('height'),
				'image': result.get('properties').get('url')
			})

		params['offset'] += 151
		old_page_result = new_page_result

	return data


if __name__ == "__main__":
	brave_images = scrape_brave_images()
	print(json.dumps(brave_images, indent=2))
```


<h2 id="preparation">Preparation</h2>

**Install libraries**:

```lang-none
pip install requests
```

**Reduce the chance of being blocked**

Make sure you're using [request headers](https://docs.python-requests.org/en/master/user/quickstart/#custom-headers) [`user-agent`](https://developer.mozilla.org/en-US/docs/Glossary/User_agent) to act as a "real" user visit. Because default `requests` `user-agent` is [`python-requests`](https://github.com/psf/requests/blob/589c4547338b592b1fb77c65663d8aa6fbb7e38b/requests/utils.py#L808-L814) and websites understand that it's most likely a script that sends a request. [Check what's your `user-agent`](https://www.whatismybrowser.com/detect/what-is-my-user-agent/).

There's a [how to reduce the chance of being blocked while web scraping blog post](https://serpapi.com/blog/how-to-reduce-chance-of-being-blocked-while-web/) that can get you familiar with basic and more advanced approaches.


<h2 id="code_explanation">Code Explanation</h2>

Import libraries:

```python
import requests, json
```

| Library       | Purpose                                                            |
|---------------|--------------------------------------------------------------------|
| [`requests`](https://requests.readthedocs.io/en/latest/user/quickstart/)    | to make a request to the website.                                  |
| [`json`](https://docs.python.org/3/library/json.html)        | to convert extracted data to a JSON object.                        |

Create URL parameters and request headers:

```python
# https://docs.python-requests.org/en/master/user/quickstart/#passing-parameters-in-urls
params = {
	'q': 'dune 2021', 		# query 
	'source': 'web',		# source 
	'size': 'All',			# size (Small, Medium, Large, Wallpaper) 
	'_type': 'All',			# type (Photo, Clipart, AnimatedGifHttps, Transparent) 
	'layout': 'All',		# layout (Square, Tall, Wide) 
	'color': 'All',			# colors (Monochrome, ColorOnly, Red etc) 
	'license': 'All',		# license (Public, Share, Modify etc)
	'offset': 0
}

# https://docs.python-requests.org/en/master/user/quickstart/#custom-headers
headers = {
	'content-type': 'application/json',
	'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'
}
```

| Code            | Explanation     |
|-----------------|-----------------|
| [`params` ](https://docs.python-requests.org/en/master/user/quickstart/#passing-parameters-in-urls)  | a prettier way of passing URL parameters to a request. |
| [`content-type` ](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type)  | to indicate the original media type of the resource (prior to any content encoding applied for sending). In responses, a `Content-Type` header provides the client with the actual content type of the returned content. |
| [`user-agent`](https://developer.mozilla.org/en-US/docs/Glossary/User_agent) | to act as a "real" user request from the browser by passing it to [request headers](https://docs.python-requests.org/en/master/user/quickstart/#custom-headers). [Default `requests` user-agent is a `python-reqeusts`](https://github.com/psf/requests/blob/589c4547338b592b1fb77c65663d8aa6fbb7e38b/requests/utils.py#L808-L814) so websites might understand that it's a bot or a script and block the request to the website. [Check what's your `user-agent`](https://www.whatismybrowser.com/detect/what-is-my-user-agent). |

Create the `data` list to hold all the data, and the `old_page_result` list that we'll need later:

```python
data = []
old_page_result = []
```

To scrape Brave images with pagination, you need to use the `offset` parameter of the URL, which defaults to `0` for the first page, `151` for the second, and so on. Since data is retrieved from all pages, it is necessary to implement a `while` loop:

```python
while True:
	# pagination will be here
```

In each iteration of the loop, you need to make a request to the Brave API, pass the created request parameters and headers. Using the [`json()` method](https://www.geeksforgeeks.org/response-json-python-requests/), the response is converted into a JSON object for further work:

```python
html = requests.get('https://search.brave.com/api/images', headers=headers, params=params).json()
```

The `new_page_result` list contains all the results on the current page. The `new_page_result` list is compared with the `old_page_result` list. If they are the same, then this means that we have reached the last page and there is no more new data. Therefore, you need to `break` the loop:

```python
new_page_result = html['results']

if new_page_result == old_page_result:
    break
```

📌Note: In the first iteration of the loop, there is no data in the `old_page_result` list. Therefore, the check will fail.

By looping through the `new_page_result` list in a `for` loop, you can get the data. For each result, data such as `title`, `link`, `source`, `width`, `height`, and `image` are retrieved:

```python
for result in new_page_result:
    data.append({
        'title': result.get('title'),
        'link': result.get('url'),
        'source': result.get('source'),
        'width': result.get('properties').get('width'),
        'height': result.get('properties').get('height'),
        'image': result.get('properties').get('url')
    })
```

📌Note: The `image` key contains a full resolution image.

After extracting the data, you need to increase the value of the `offset` parameter by `151`. This value also increases on the site when you click on the button responsible for showing more data, that is, we simulate this behavior:

```python
params['offset'] += 151
```

This is shown more clearly in the GIF below:

![brave-images-pagination](https://user-images.githubusercontent.com/81998012/197318628-48882e51-d638-40c1-8ec4-85164442ff5d.gif)

On each iteration, the data from the `new_page_result` list will be written to the `old_page_result` list until they are the same:

```python
old_page_result = new_page_result
```


<h2 id="output">Output</h2>

```json
[
  {
    "title": "Dune (2021) | The Poster Database (TPDb)",
    "link": "https://theposterdb.com/posters/42710?page=2",
    "source": "theposterdb.com",
    "width": 1365,
    "height": 2048,
    "image": "https://image.tmdb.org/t/p/original/2sxSn0jjjQoIIZfZjC6j5GZkMVR.jpg"
  },
  {
    "title": "Dune (2021) - Posters \u2014 The Movie Database (TMDB)",
    "link": "https://www.themoviedb.org/movie/438631-dune/images/posters",
    "source": "The Movie Database",
    "width": 2000,
    "height": 3000,
    "image": "https://www.themoviedb.org/t/p/original/7S56MF6XA1jIzD9I2ejMjd6aNvN.jpg"
  },
  {
    "title": "Dune (2021) - Posters \u2014 The Movie Database (TMDb)",
    "link": "https://www.themoviedb.org/movie/438631-dune/images/posters",
    "source": "The Movie Database",
    "width": 956,
    "height": 1333,
    "image": "https://www.themoviedb.org/t/p/original/AqjrlcNRSKx84CeNJyNueg6V1SR.jpg"
  },
  {
    "title": "Dune - Pel\u00edcula 2021 - SensaCine.com",
    "link": "http://www.sensacine.com/peliculas/pelicula-133392/",
    "source": "Sensacine",
    "width": 600,
    "height": 800,
    "image": "http://es.web.img2.acsta.net/pictures/20/04/15/09/53/3283826.jpg"
  },
  {
    "title": "DUNE 2021 Movie Poster : dune",
    "link": "https://www.reddit.com/r/dune/comments/kh9som/dune_2021_movie_poster/",
    "source": "reddit.com",
    "width": 1890,
    "height": 2800,
    "image": "https://preview.redd.it/3fl2s0q1ug661.jpg?auto=webp&s=ed5e4418f962103b0d47b5b466036d7b40aa761b"
  },
  ... other images
]
```


<h2 id="links">Links</h2>

- [Code in the online IDE](https://replit.com/@chukhraiartur/blog-brave-search-images#main.py)
- [Reduce the chance of being blocked while web scraping](https://dev.to/dimitryzub/how-to-reduce-chance-being-blocked-while-web-scraping-search-engines-1o46)


<p style="text-align: center;">Join us on <a href="https://twitter.com/serp_api">Twitter</a> | <a href="https://www.youtube.com/channel/UCUgIHlYBOD3yA3yDIRhg_mg">YouTube</a></p>

<p style="text-align: center;">Add a  <a href="https://github.com/serpapi/public-roadmap/issues">Feature Request</a>💫 or a <a href="https://github.com/serpapi/public-roadmap/issues">Bug</a>🐞</p>