Introduction to web scraping with goquery
Tags: go, scraping
Published: 2023-03-24 09:13:52

---
title: Introduction to web scraping with goquery
published: true
date: 2021-04-17 16:27:40 UTC
tags: go, scraping
canonical_url: https://pelle.link/en/introduction-webscraping-goquery/
cover_image: https://pelle.link/en/introduction-webscraping-goquery/scraper.jpg
---

The use of an API is the preferred solution to retrieve data, but when it does not exist, is incomplete, or is subject to limitations that are too restrictive, _scraping_ the associated website can be a good alternative.

What we call _web scraping_ is the collection (via the writing of bots called _scrapers_) of targeted data present on a set of web pages. A _scraper_ is designed specifically for a given site, unlike a _crawler_ which will be more generalist (_Google Shopping_ uses _scrapers_ when _Google_ (aka the search engine) uses _crawlers_).

The uses are multiple (aggregation of ads from several job boards, competitive intelligence, …), and are not necessarily reserved to the professional world.

We can indeed find several situations where writing a personal _scraper_ can have its interest:

- Sending an email as soon as the homework has been updated on the children’s school blog
- Automatically setting the alarm clock 15 minutes earlier when a weather site forecasts ice the next morning
- Receive a notification when the price of an item on an e-commerce site falls below a certain threshold

Let’s illustrate this last use case.

## Looking for a cheap Raspberry Pi 4

Let’s take this [Raspberry Pi 4B](https://www.kubii.fr/cartes-raspberry-pi/2955-raspberry-pi-4-modele-b-8gb-0765756931199.html) on [www.kubii.fr](https://www.kubii.fr), it is on sale for 83.99€ but we would like to wait until it drops below 75€ before ordering. As a price drop can quickly lead to a stockout, we might as well be warned quickly to avoid missing a good deal.

The information we are looking for here is the price of the item:

![raspberry-pi-4b-price](https://pelle.link/en/introduction-webscraping-goquery/raspberry-pi-4b-price.png)

Let’s see how it looks in the HTML:

```html
<p class="our_price_display" itemprop="offers" itemscope="" itemtype="https://schema.org/Offer">
  <link itemprop="availability" href="https://schema.org/InStock">
  <span id="our_price_display" class="price" itemprop="price" content="83.99">83,99 €</span> TTC
  <meta itemprop="priceCurrency" content="EUR">
</p>
```

From there, we need to find an _anchor point_, that is to say a criterion that will allow us to find the price (i.e. the attribute `content` of the `span` tag) for sure and without duplication.

After analyzing the complete page, it turns out that the `itemprop="price"` attribute is a good candidate (we only find one `span` tag meeting this criterion).

Our _scraper_ (in go) can now take shape:

```go
package main

import (
	"github.com/PuerkitoBio/goquery"
	"log"
	"net/http"
	"strconv"
	"time"
)

func main() {
	goodPrice := false

	// Stay in the loop as long as we have not passed below our threshold

	for !goodPrice {

		// Loading of the product page
		res, err := http.Get("https://www.kubii.fr/cartes-raspberry-pi/2955-raspberry-pi-4-modele-b-8gb-0765756931199.html")
		if err != nil {
			log.Printf("Unable to load the product page, we will try again later: %v", err)
			time.Sleep(30 * time.Minute)
			continue
		}
		if res.StatusCode != 200 {
			res.Body.Close()
			log.Printf("Unable to load the product page, we will try again later, status code: %d", res.StatusCode)
			time.Sleep(3 * time.Hour)
			continue
		}

		// HTML Parsing through goquery
		doc, err := goquery.NewDocumentFromReader(res.Body)
		res.Body.Close()
		if err != nil {
			log.Panicf("Unable to parse HTML content: %v", err)
		}

		// Price retrieval
		var priceStr string
		var ok bool

		if priceStr, ok = doc.Find("span[itemprop='price']").Attr("content"); !ok {
			log.Panicf("Price not found, scraper should be updated")
		}

		// Price conversion
		price, err := strconv.ParseFloat(priceStr, 64)
		if err != nil {
			log.Panicf("Unable to read the price, scraper should be updated")
		}

		if price >= 75 {
			log.Printf("%0.2f € it's still too expensive!", price)
		} else {
			log.Printf("Only %0.2f €, go go go!!!", price)
			// TODO: Sending an email, an SMS, calling a discord webhook, triggering the alarm, flashing the connected bulbs, etc ...
			goodPrice = true
		}

		// We try again in 30 minutes
		time.Sleep(30 * time.Minute)

	}
    
    log.Printf("Done!")

}

```

The logic is quite simple

- Every 30 minutes, we retrieve the price until it falls below 75€, if necessary we raise an alert.
- A basic (and perfectible!) management of the unavailability of the product page is applied, whether they are
  - due to us (network failure, … ) => new attempt after 30 minutes
  - or due to the site (maintenance, anti-bot defense, …) => new attempt after 3 hours (no need to be stubborn)
- If the price is no longer retrievable, we assume that the site must have been updated and that our _anchor point_ must have jumped at the same time, so we must find a new one and update the _scraper_ accordingly.

Note that the search for the famous `<span>` tag containing our price is easily done via the excellent [goquery](https://github.com/PuerkitoBio/goquery) library, which allows, as in javascript, to use CSS selectors to locate DOM elements.

## Finally

A small VPS, a NAS, an old Raspberry PI (hence the purchase of a new one!) or simply your PC (if you’re the kind of person who keeps it on 24 hours a day), and you’re ready to deploy your first _scraper_.

Feel free to adapt this example to your needs but, word to the wise, be sure to stay reasonable on the frequency of requests, some sites do not like bots that monopolize too many resources on their server to the detriment of _real_ visitors. So you can understand them if they end up blacklisting your IP because you try to retrieve a price every 2 seconds instead of 30 minutes…