AutoScraper and Flask: Create an API From Any Website in Less Than 5 Minutes
Tags: python, webdev, beginners, tutorial
Published: 2020-09-21 09:44:01

In this tutorial, we are going to create our own e-commerce search API with support for both eBay and Etsy without using any external APIs. With the power of [AutoScraper](https://github.com/alirezamika/autoscraper/) and [Flask](https://flask.palletsprojects.com/), we are able to achieve this goal in fewer than 20 lines of Python code for each site. I recommend reading my last [article](https://medium.com/better-programming/introducing-autoscraper-a-smart-fast-and-lightweight-web-scraper-for-python-20987f52c749) about AutoScraper if you haven’t done so yet.


-------

# Requirements
Install the required libraries using pip:
```
pip install -U autoscraper flask
```

-------

# Let’s Do It

First, we are going to create a smart scraper to fetch data from eBay’s search results page. Let’s say we want to get the `title`, `price`, and `product link` of each item. Using AutoScraper, it would be easily done by just providing some sample data:

{% gist https://gist.github.com/alirezamika/b148aa12e62dd7bfc1bee14cb6cfb4f7 %}

Note that if you want to copy and run this code, you may need to update the `wanted_list`.

Now let’s get the results grouped by scraping rules:

```
scraper.get_result_similar(url, grouped=True)
```

From the output, we’ll know which rule corresponds to which data, so we can use it accordingly. Let’s set some aliases based on the output, remove redundant rules, and save the model so we can use it later:

{% gist https://gist.github.com/alirezamika/98f53cb72a6cd1f9866c08555356a2ca %}

Note that the rule IDs will be different for you if you run the code.

OK, we’ve got eBay covered. Let’s add support for Etsy search results too. We’ll start by building its scraper. This time, we will use `wanted_dict` instead of `wanted_list`. It will automatically set aliases for us:

{% gist https://gist.github.com/alirezamika/743234bb98ba1e59e9727ef7788c7da9 %}

As the links are generated with a unique ID on Etsy each time, we added one sample product ID to the `wanted_dict` so we can create the link from it. Also, we provided two samples for title and price, as the structure of items on Etsy search result pages is different and we want the scraper to learn them all.

After analyzing the output, let’s keep our desired rules, remove the rest, and save our model:

{% gist https://gist.github.com/alirezamika/45a51ce21740060f9fc638e5e090f14f %}

Now that we have our scrapers ready, we can create our fully functioning API for both sites in fewer than 40 lines:

{% gist https://gist.github.com/alirezamika/ef540b87a63f12cf6a70ea9e03f23555 %}

Here, we are defining an API with the parameter `q` as our search query. We will get and join eBay and Etsy search results and return them as a response. Note that we are passing `group_by_alias=True` to the scraper to get the results grouped by our defined aliases.

By running this code, the API server will be up listening on port 8080. So let’s test our API by opening `http://localhost:8080/?q=headphone` in our browser:

<figure>
  <img src="https://miro.medium.com/max/5744/1*i13MQE455mvv7ouOZoOC6Q.png">
  <figcaption>Some results from eBay</figcaption>
</figure>


<figure>
  <img src="https://miro.medium.com/max/5752/1*UJt7y3MXB1xheBwDPWiN1Q.png">
  <figcaption>Some results from Etsy</figcaption>
</figure>


Voila! We have our e-commerce API ready. Just replace `headphone` in the URL with your desired query to get its search results.

-------

# Final Notes

The final code for this tutorial is available on [GitHub](https://github.com/alirezamika/tutorials/tree/master/api_server).

This is a development setup suitable for developing and testing. Flask’s built-in server is not suitable for production. For production usage, please check [Flask’s deployment options](https://flask.palletsprojects.com/en/1.1.x/deploying/).

This tutorial is intended for personal and educational use. If you want to scrape websites, you can check their policies regarding the scraping bots.

I hope this article is useful and helps to bring your ideas into code faster than ever. Happy coding!