Create your first web scraper with ScrapingBee API and Python
Tags: scrapingwithpython, scraping, webscrapingservice, automation
Published: 2022-10-10 18:20:14

---
title: Create your first web scraper with ScrapingBee API and Python
published: true
date: 2020-04-04 14:39:36 UTC
tags: scrapingwithpython,scraping,webscrapingservice,automation
canonical_url: https://itnext.io/create-your-first-web-scraper-with-scrapingbee-api-and-python-addf0dc5df58
---

In this post, I am going to discuss another cloud-based scraping tool that takes care of many of the issues you usually face while scraping websites. This platform has been introduced by [ScrapingBee](https://www.scrapingbee.com?fpr=adnan-35), a cloud-based Scraping tool.

### What is ScrapingBee

If you visit their [website](https://www.scrapingbee.com?fpr=adnan-35), you will find something like below:

> **ScrapingBee API handles headless browsers and rotates proxies for you.**

![](https://cdn-images-1.medium.com/max/1024/0*puVXgypdw5hI9QAj.png)

As it suggests, it is offering you all the things to deal with the issues you usually come across while writing your scrapers, especially the availability of proxies and headless scraping. No installation of web drivers for Selenium, yay!

### Development

ScrapingBee is based on REST API hence it can be consumed in any programming language. Since this post is related to Python so I’d be mainly focusing on requests library to use this tool.

Once you get registered and confirmed with them they will provide you an API KEY which you can see on the dashboard. They provide 1000 free API calls which are enough to test their platform. Otherwise, they offer different plans from starter to the enterprise which you can view [here](https://www.scrapingbee.com/#pricing).

Let’s try a simple example.

On executing this code, you will see output like below:

![](https://cdn-images-1.medium.com/max/1024/0*Fa6v4iuhq-sRtx-u.png)

Did you notice, every time it returns a new IP address, cool, isn’t it?

You can also pass your own custom headers where needed. If I change URL\_TO\_SCRAPE to https://httpbin.org/headers, it will dump the default headers.

So I changed the URL of _httpbin_ to display headers. First I set the parameter forward\_headers as True and then I override the existing header, in this case, User-Agent parameter as Scn-User-Agent. As per documents, you have to set Scn- as a prefix to override the existing one or add your own one. In our case, I added a new header as Scn-X-MyOwn-Header. When I run this it displays the following:

![](https://cdn-images-1.medium.com/max/1024/0*fvP8AHGVor8xRVSD.png)

Cool, No?

### Creating OLX Scraper

Like previous [scraping related posts](http://blog.adnansiddiqi.me/tag/scraping/), I am going to pick OLX again for this post. I will iterate the list first and then will scrape individual items. Below is the complete code.

I am using Beautifulsoup to parse HTML. I have only extracted Price here because the purpose is to tell about the API itself than _Beautifulsoup_. You should see my post [here](http://blog.adnansiddiqi.me/write-your-first-web-scraper-in-python-with-beautifulsoup/) in case you are new to scraping and Python.  
 Here I introduced another parameter, country\_code which I used in the USA. This parameter ensures to use a proxy from the US only instead of any random country. Anyways, when I scrape _Tablets section_ of OLX it prints following:

### Conclusion

In this post, you learned how to use ScrarpingBee API for scraping purposes. Whatever you can do with this API you can do it by other means as well; this API provides you everything under the umbrella, especially rendering of pages via Javascript for which you need headless browsers which, at times become cumbersome to set things up on remote machines for headless scraping. ScrarpingBee API is taking care of it and charging nominal charges for individuals and enterprises. The company I work with spend 100s of dollars on a monthly basis just for the proxy IPs.

Oh if you sign up [here](https://www.scrapingbee.com?fpr=adnan-35) with my referral link or enter promo code **ADNAN** , you will get a **10% discount** on it. In case you do not get the discount then just let me know via email on my site and I’d sure help you out.

In the coming days, I’d be writing more posts about ScrarpingBee API discussing further features.

_Originally published at_ [_http://blog.adnansiddiqi.me_](http://blog.adnansiddiqi.me/create-your-first-web-scraper-with-scrapingbee-api-and-python/?utm_source=r_python_scrapingbee&utm_medium=reddit&utm_campaign=c_r_python_scrapingbee) _on April 4, 2020._

* * *