Web Scraping with no coding
Published: 2020-02-18 22:37:45

Hello World 👋 🌍,

In this article, I will show how easy it can be to do Web Scraping.

I will show how to extract content ( text, HTML, links, images, etc..) form a webpage  **without writing code**.

The only thing you will need to do is to send an HTTP request and specify CSS selectors of elements you want to scrape.

----------

Below you can see an example of a basic request body.

{% gist https://gist.github.com/princePeterHansen/b53bee0757ffe2efffe1c371d0b20e3f %}


A response can be an array with extracted values or it could be in JSON like format.

{% gist https://gist.github.com/princePeterHansen/20226bd6ff5bfd32add24d1653163dc0 %}

----------

For the Demo I’m going to use:

1)  [Books to scrape](http://books.toscrape.com/) — a playground for web scraping.  
2)  [Postman](https://www.postman.com/)  — app for sending HTTP requests. 
3)  [Proxybot](https://proxybot.io/)  — API service helper tool for web scraping.

Let’s get started 👨‍💻

----------

For people who prefer watching videos, there is a quick video showing how to scrape basic webpages.

{% youtube 9AmY-K7iRC8 %}

----------

The idea is very simple, we just need to:

1.  **Find a page we want to scrape**
2.  **Get CSS selector of desired elements**
3.  **Send HTTP POST request with a Body containing CSS selectors** from step 

And now the same steps but just with more details 🔎.

----------

## 1) Find a page we want to scrape

We will use the ‘Books to Scrape’ ([http://books.toscrape.com/](http://books.toscrape.com/)) website as our web scraping playground.


![Web Scraping with no coding](https://dev-to-uploads.s3.amazonaws.com/i/hhuesb1zfwjo7p2opy1y.png)

The ‘Books to Scrape’ website contains dummy information about various books.

The website is ideal if you want to practice basic web scraping skills.

----------

## 2) Get CSS selector of desired elements

In order to get CSS for desired elements, we need to open Dev tools of your desired browser and inspect elements.

[![Web Scraping with no coding](https://dev-to-uploads.s3.amazonaws.com/i/86r1w0gjfxxx6y4qnk3h.png)](https://proxybot.io/documentation?step=scraping)

Inspecting a book element gives us the following HTML markup 🕵.


{% gist https://gist.github.com/princePeterHansen/b65f14a90f4d59842e4f73edf4c8ef15 %}


**NB!** We will use the above HTML for creating request objects.

----------

## 3) Send HTTP POST request to API service 🚀

In the below example we will extract title and link for all the books found on the page.

**Request URL**  :  
`https://proxybot.io/api/v1/API_KEY?url=http://books.toscrape.com`

**Request BODY:**


{% gist https://gist.github.com/princePeterHansen/1aa7366acdc835af3b063498fabc75be %}

We specify CSS selector for book’s  **title**  and request to get its value as text. However, for the  **link’s**  value, we need to instruct service to get the value from the  **href**  attribute.

The response will contain titles and links of all books on the page.

**Response:**

{% gist https://gist.github.com/princePeterHansen/cfd38c70d582e96669c27e2e5f70437a %}

This is already super cool!

If you are interested in only specific data then this type of response might be already good enough.

However, I would like to have a formatted response, let's see how we can achieve that.

**Request BODY for formatted response:**

{% gist https://gist.github.com/princePeterHansen/f64cc0f2601a5d933b7df4b329bb9d47 %}

We need to ask to return  **“json”**  and provide an array with selectors in “**extract**” property.


Additionally, we can specify  **“as”**  property which will be used for formatting the response object.

The above request will result in the following response

**Formatted response:**

{% gist https://gist.github.com/princePeterHansen/19b885cdf8d37d0036b0e1d40e726bb3 %}

Wow! We can specify the format of an object we want to get back! How cool is that?

----------

Congratulations 🥳 Now you know how to scrape websites without coding. As you can see it is pretty simple. I hope this article was interesting and useful. 

In case you looking for a proxy providers, here you can find a list with [TOP 7 proxy providers in 2021](https://dev.to/princepeterhansen/top-7-residential-proxies-providers-31f9 "hand-picked proxy providers").