Building a Go Web Scraper Step-by-Step: The Beginners Guide to Colly
Tags: webdev, javascript, go, python
Published: 2022-09-01 00:24:59

<!-- wp:image -->
<figure class="wp-block-image"><img src="https://www.scraperapi.com/wp-content/uploads/2022/08/golang-scraper.jpg" alt=""/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Go allows developers to create complicated projects with a simpler syntax than C, but with almost the same efficiency and control.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Its simplicity and efficiency is why we decided to add Golang to our web scraping beginners series and show you how to use it to extract data at scale.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Why Use Go Over Python or JavaScript for Web Scraping?</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>If you’ve followed our series, you’ve seen how easy web scraping is with languages like Python and JavaScript, so why should you give Go a shot?</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>There are three main reasons for choosing Go over other languages:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>Go is a&nbsp;<a href="https://www.geeksforgeeks.org/what-is-a-typed-language/" target="_blank" rel="noreferrer noopener">statically typed language</a>, making it easier to find errors without running your program.&nbsp;<a href="https://www.codecademy.com/article/what-is-an-ide" target="_blank" rel="noreferrer noopener">Integrated Development Environments</a>&nbsp;(IDEs) will highlight errors immediately and even show you suggestions to fix them.</li><li>IDEs can be more helpful the more they understand the code, and because we declare the data types in Go, there’s less ambiguity in the code. Thus, IDEs can provide better auto-complete features and suggestions than in other languages.</li><li>Unlike Python or JavaScript, Go is a compiled language that outputs machine code directly, making it faster than Python. In an&nbsp;<a href="https://medium.com/@arnesh07/how-golang-can-save-you-days-of-web-scraping-72f019a6de87#:~:text=From%20this%20graph%2C%20it%20is,the%20same%20number%20of%20URLs." target="_blank" rel="noreferrer noopener">experiment run by Arnesh Agrawal</a>, Python (using the Beautiful Soup library) took almost 40 minutes to scrape 2000 URLs, while Go (using the Goquery package) took less than 20 minutes.</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>In summary, Go is an excellent option if you need to optimize scraping speed or if you’re looking for a statically typed language to transition to.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Web Scraping with Go</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>For this project, we’ll scrape the<a href="https://www.jackjones.com/nl/en/jj/shoes/" target="_blank" rel="noreferrer noopener">&nbsp;<em>Jack and Jones</em>&nbsp;shoe category</a>&nbsp;page to extract product names, prices, and URLs before exporting the data into a JSON file using Go’s web scraping library Colly.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>A consideration:</strong>&nbsp;We’ll try to explain every step of the way as detailed as possible, so even if you don’t have experience with Go, you’ll be able to follow along. However, if you still feel lost while reading, here’s a great<a href="https://www.youtube.com/watch?v=yyUHQIec83I" target="_blank" rel="noreferrer noopener">&nbsp;introduction to Go</a>&nbsp;to watch beforehand.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>1. Setting Up Our Project</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>We first need to head to<a href="https://go.dev/dl/" target="_blank" rel="noreferrer noopener">&nbsp;https://go.dev/dl/</a>&nbsp;and download the right version of Go based on our operating system. In our case, we’ll pick the ARM64 version as we’re using a MacBook Air M1.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":1553} -->
<figure class="wp-block-image aligncenter"><img src="https://www.scraperapi.com/wp-content/uploads/2022/08/golang-scraper-img1-1024x242.png" alt="" class="wp-image-1553"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p><strong>Note:</strong>&nbsp;You can also use<a href="https://formulae.brew.sh/formula/go#default" target="_blank" rel="noreferrer noopener">&nbsp;Homebrew</a>&nbsp;in MacOS or<a href="https://chocolatey.org/" target="_blank" rel="noreferrer noopener">&nbsp;Chocolatey</a>&nbsp;in Windows to install Go.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":1554} -->
<figure class="wp-block-image aligncenter"><img src="https://www.scraperapi.com/wp-content/uploads/2022/08/golang-scraper-img2.png" alt="" class="wp-image-1554"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Once the download is complete, follow the instructions to install it on your machine. With it installed, let’s create a new directory named go-scraper and open it on VScode or your preferred IDE.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>Note:</strong>&nbsp;Open the terminal and enter the&nbsp;go version&nbsp;command. If everything goes well, it will log the version like this:</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":1555} -->
<figure class="wp-block-image aligncenter"><img src="https://www.scraperapi.com/wp-content/uploads/2022/08/golang-scraper-img3.png" alt="" class="wp-image-1555"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>For this tutorial, we’ll be using VScode to write our code, so we’ll also download Go’s VScode extension for a better experience.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":1556} -->
<figure class="wp-block-image aligncenter"><img src="https://www.scraperapi.com/wp-content/uploads/2022/08/golang-scraper-img4.png" alt="" class="wp-image-1556"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Next, open the terminal and type the following command to create or initialize the project:&nbsp;<code>go mod init go-scraper</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":1557} -->
<figure class="wp-block-image aligncenter"><img src="https://www.scraperapi.com/wp-content/uploads/2022/08/golang-scraper-img5.png" alt="" class="wp-image-1557"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>In Go, “a<a href="https://go.dev/blog/using-go-modules" target="_blank" rel="noreferrer noopener">&nbsp;module</a>&nbsp;is a collection of&nbsp;<a href="https://go.dev/ref/spec#Packages" target="_blank" rel="noreferrer noopener">Go packages</a>&nbsp;stored in a file tree with a&nbsp;<em>go.mod</em>&nbsp;file at its root.” The command above –&nbsp;<code>go mod init</code>&nbsp;– tells Go that the directory we’re specifying (<em>go-scraper</em>) is the module’s root.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Without leaving the terminal, let’s create a new&nbsp;<em>jack-scraper.go</em>&nbsp;file using:&nbsp;touch jack-scraper.go&nbsp;and – using the command found on Colly’s documentation –&nbsp;<code>go get -u github.com/gocolly/colly/...</code>&nbsp;to install Colly.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":1558} -->
<figure class="wp-block-image aligncenter"><img src="https://www.scraperapi.com/wp-content/uploads/2022/08/golang-scraper-img6-1024x589.png" alt="" class="wp-image-1558"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>All the dependencies downloaded were added to the&nbsp;<em>go.mod</em>&nbsp;file and a new&nbsp;<em>go.sum</em>&nbsp;file – which may contain hashes for multiple versions of a module- was created.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Now we can consider our environment set!</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>2. Sending HTTP Requests with Colly</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>At the top of our <em>jack-scraper.go</em> file, we’ll add our package’s name, which as a convention, we’ll name main:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code class="">package main</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>And then import the dependencies to the project:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code class="">import (
   "github.com/gocolly/colly"
)</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p><strong>Note:</strong>&nbsp;Due to Go’s typed nature, VScode will tell us there’s an error with your import. Basically, it’s telling us that we imported a dependency, but we’re not using it. This is one of the advantages of using Go, as we don’t need to run our code to find errors.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Something particular to Go is that we need to provide a starting point for the code to run. So we’ll create the main function, and all our logic will be inside it. For now, we’ll print “Function is working.”</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code class="">func main() {
  
   fmt.Println("Function is working")
}</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>If everything went well it should return this:</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":1559} -->
<figure class="wp-block-image aligncenter"><img src="https://www.scraperapi.com/wp-content/uploads/2022/08/golang-scraper-img7.png" alt="" class="wp-image-1559"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>You might have also noticed that a new package was imported. That’s because Go can tell we’re trying to use the&nbsp;.Println()&nbsp;function from the&nbsp;fmt&nbsp;package, so it automatically imported it. Handy, right?</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To handle our request and deal with all callbacks necessary to extract the data, Colly uses a <a href="http://go-colly.org/docs/introduction/start/" target="_blank" rel="noreferrer noopener">collector</a> object. Initializing the collector is as simple as calling the .NewCollector() method from Colly and passing the domains we want to allow Colly to visit:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code class="">c := colly.NewCollector(
    colly.AllowedDomains("www.jackjones.com"),
)</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>On our new c instance, we can now call the <code>.Visit()</code> method to make colly send our request.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code class="">c.Visit("https://www.jackjones.com/nl/en/jj/shoes/"</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>However, there are a few more things we need to add to our code before sending our&nbsp;collector&nbsp;out into the world. Let’s see what’s happening behind the scenes.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>First, we’ll create a callback to print out the URL Colly is navigating to – this will become more useful as we scale our scraper from one page to multiple pages.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code class="">c.OnRequest(func(r *colly.Request) {
    fmt.Println("Scraping:", r.URL)
})</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>And then a callback to print out the status of the request.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code class="">c.OnResponse(func(r *colly.Response) {
    fmt.Println("Status:", r.StatusCode)
})</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>As we said before, the&nbsp;<code>collector</code>&nbsp;object is responsible for the callbacks attached to a&nbsp;<code>collector</code>&nbsp;job. So each of these functions will be called at various stages in the&nbsp;<code>collector</code>&nbsp;job cycle.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>In our example, the&nbsp;<code>.OnRequest()</code>&nbsp;function is called right before the collector makes the HTTP request, while the&nbsp;<code>.OnRespond()</code>&nbsp;function will be called after a response is received.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Just for good measure, let’s also create an error handler before running the scraper.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code class="">c.OnError(func(r *colly.Response, err error) {
    fmt.Println("Request URL:", r.Request.URL, "failed with response:", r, "\nError:", err)
})</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>To run our code, open a terminal and use the&nbsp;<code>go run jack-scraper.go</code>&nbsp;command.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":1560} -->
<figure class="wp-block-image aligncenter"><img src="https://www.scraperapi.com/wp-content/uploads/2022/08/golang-scraper-img8.png" alt="" class="wp-image-1560"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Awesome, we got a beautiful 200 (successful) status code! We’re ready for phase 2, the extraction.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>3. Inspecting the Target Website</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>To extract data from an HTML file, we need more than just access to the site. Each website has its own structure and we must understand it in order to scrape specific elements without adding noise to our dataset.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The good news is that we can quickly look at the site’s HTML structure by right-clicking on the page and selecting&nbsp;<em>Inspect</em>&nbsp;from the menu. It’ll open the Inspector tools, and we can start hovering on elements to see their positions in the HTML and attributes.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":1561} -->
<figure class="wp-block-image aligncenter"><img src="https://www.scraperapi.com/wp-content/uploads/2022/08/golang-scraper-img9-1024x490.png" alt="" class="wp-image-1561"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>We use&nbsp;<a href="https://www.scraperapi.com/blog/css-selectors-cheat-sheet/">CSS selectors in web scraping</a>&nbsp;(classes, IDs, etc.) to tell our scrapers where to locate the elements we want them to extract for us. Fortunately for us, Colly is based on the&nbsp;<a href="https://pkg.go.dev/github.com/PuerkitoBio/goquery" target="_blank" rel="noreferrer noopener">Goquery package</a>, which provides Colly with a JQuery-like syntax to target these selectors.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Let’s try to grab the first shoe’s product name using the DevTools console to test the ground.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>4. Using DevTools to Test Our CSS Selectors</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>If we inspect the element further, we can see that the name of the product is in a&nbsp;<code>&lt;a&gt;</code>&nbsp;tag with the class “<code>product-tile__name__link</code>”, wrapped between&nbsp;<code>&lt;header&gt;</code>&nbsp;tags.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":1562} -->
<figure class="wp-block-image aligncenter"><img src="https://www.scraperapi.com/wp-content/uploads/2022/08/golang-scraper-img10.png" alt="" class="wp-image-1562"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>On our <a href="https://developer.chrome.com/docs/devtools/console/" target="_blank" rel="noreferrer noopener">browser’s console</a>, let’s use the <code>document.querySelectorAll()</code> method to find the that element.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code class="">document.querySelectorAll("a.product-tile__name__link.js-product-tile-link")</code></pre>
<!-- /wp:code -->

<!-- wp:image {"align":"center","id":1563} -->
<figure class="wp-block-image aligncenter"><img src="https://www.scraperapi.com/wp-content/uploads/2022/08/golang-scraper-img11.png" alt="" class="wp-image-1563"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Yes! It returns 44 elements that match perfectly with the number of elements on the page.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The product’s URL is inside the same element, so we can use the same selector for it. On the other hand, after some testing, we can use “<code>em.value__price</code>” to pick the price.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>5. Scraping All Product Names on the Page</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>If you’ve read some of our previous articles, you know we love testing. So, before extracting all our target elements, we’re first going to test our scraper by extracting all product names from the page.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To do so, Colly has a <code>.OnHTML()</code> function to handle the HTML from the response (there’s also an <code>.OnXML()</code> function you can use if you’re not sure if the response will be HTML or XML content).</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code class="">c.OnHTML("a.product-tile__name__link.js-product-tile-link", func(h *colly.HTMLElement) {
    fmt.Println(h.Text)
})</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Here’s a breakdown of what’s happening:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li>We pass the main element we want to work with&nbsp;<code>a.product-tile__name__link.js-product-tile-link</code>&nbsp;as the first argument of the&nbsp;<code>OnHTML()</code>&nbsp;function.</li><li>The second argument is a function that will run when Colly finds the element we specified.</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p>Inside that function, we told Colly what to do with the&nbsp;h&nbsp;object. In our case, it’s printing the text of the element.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":1564} -->
<figure class="wp-block-image aligncenter"><img src="https://www.scraperapi.com/wp-content/uploads/2022/08/golang-scraper-img12.png" alt="" class="wp-image-1564"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>And yes, it’s working great so far. However, there’s a lot of empty space around our text, adding noise to the data.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To solve this problem, let’s get even more specific by adding the <code>&lt;header></code> element as our main selector and then looking for the text using the <code>.ChildText()</code> function.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code class="">c.OnHTML("header.product-tile__name", func(h *colly.HTMLElement) {
    fmt.Println(h.ChildText("a.product-tile__name__link.js-product-tile-link"))
})</code></pre>
<!-- /wp:code -->

<!-- wp:image {"align":"center","id":1565} -->
<figure class="wp-block-image aligncenter"><img src="https://www.scraperapi.com/wp-content/uploads/2022/08/golang-scraper-img13.png" alt="" class="wp-image-1565"/></figure>
<!-- /wp:image -->

<!-- wp:heading {"level":3} -->
<h3>6. Extracting All HTML Elements</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>By now, we have a good grasp of the logic behind the&nbsp;<code>OnHTML()</code>&nbsp;function, so let’s scale it a little bit more and extract the rest of the data.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>We begin by changing the main selector to the one that contains all the data we want:</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":1566} -->
<figure class="wp-block-image aligncenter"><img src="https://www.scraperapi.com/wp-content/uploads/2022/08/golang-scraper-img14-1024x200.png" alt="" class="wp-image-1566"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Then, we go down the hierarchy to extract the name, price, and URL of each product:</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":1567} -->
<figure class="wp-block-image aligncenter"><img src="https://www.scraperapi.com/wp-content/uploads/2022/08/golang-scraper-img15.png" alt="" class="wp-image-1567"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Here’s how it translates to code:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code class="">c.OnHTML("div.product-tile__content-wrapper", func(h *colly.HTMLElement) {
    name := h.ChildText("a.product-tile__name__link.js-product-tile-link")
    price := h.ChildText("em.value__price")
    url := h.ChildAttr("a.product-tile__name__link.js-product-tile-link", "href")
    fmt.Println(name, price, url)
})</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>For this function, we’re storing each element inside a variable to make it easier to work with later. For example, printing all the elements scraped to the console.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>Note:</strong>&nbsp;When using the&nbsp;<code>.ChildAttr()</code>&nbsp;callback function, we need to pass the selectors of the element as a first argument<code>&nbsp;(</code><code>a.product-tile__name__link.js-product-tile-link</code><code>)</code>&nbsp;and the name of the attribute as the second&nbsp;<code>(</code><code>href)</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":1568} -->
<figure class="wp-block-image aligncenter"><img src="https://www.scraperapi.com/wp-content/uploads/2022/08/golang-scraper-img16-1024x127.png" alt="" class="wp-image-1568"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>We got a beautiful list of elements printed on the console but not quite the best format for analysis. Let’s solve that!</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>7. Export Data to a JSON File in Colly</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Saving our data to a JSON file will be more useful than having everything on the terminal, and it’s not that hard to do, thanks to Go’s built-in JSON modules.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":4} -->
<h4>Creating a Structure</h4>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Outside the main function, we need to create a structure (struct) to group every data set (name, price, URL) into a single type.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code class="">type products struct {
   Name  string
   Price string
   URL   string
}</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Structures let you combine different data types, so we have to define the data type for each element. We could also define the JSON field if we want, but it’s not mandatory, as in some cases we don’t know what these will be.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code class="">type products struct {
   Name  string `json:"name"`
   Price string `json:"price"`
   URL   string `json:"url"`
}</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p><strong>TIP:</strong> for a cleaner look, we can group all fields sharing the same type in a single line of code like:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code class="">type products struct {
   Name, Price, URL string
}</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Of course, there’s going to be an “unused” error message on our code because we haven’t, well, used the <code>products struct</code> anywhere. So, we’ll assign each scraped element to one of our fields in the <code>struct</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code class="">c.OnHTML("div.product-tile__content-wrapper", func(h *colly.HTMLElement) {
    products := products{
        Name:  h.ChildText("a.product-tile__name__link.js-product-tile-link"),
        Price: h.ChildText("em.value__price"),
        URL:   h.ChildAttr("a.product-tile__name__link.js-product-tile-link", "href"),
    }
    
    fmt.Println(products)
})</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>If we run our application, our data is now grouped, making it possible to pass each element as an individual “item” to the empty list we’ll later turn into a JSON file.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":1569} -->
<figure class="wp-block-image aligncenter"><img src="https://www.scraperapi.com/wp-content/uploads/2022/08/golang-scraper-img17-1024x128.png" alt="" class="wp-image-1569"/></figure>
<!-- /wp:image -->

<!-- wp:heading {"level":4} -->
<h4>Adding Scraped Elements to a Slice</h4>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>With our structure holding the product data, we’ll next send all of them to an empty&nbsp;<a href="https://go.dev/tour/moretypes/7" target="_blank" rel="noreferrer noopener">Slice</a>&nbsp;(instead of an Array like we would in other languages) to create a list of items we’ll export to the JSON file.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To collect the empty slice, add this code after the collector initiation code:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code class="">var allProducts []products</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Inside the <code>.OnHTML()</code> function, instead of printing our structure, let’s instead append all the items inside products to the Slice.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code class="">allProducts = append(allProducts, products)</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>If we print the Slice out now, here’s the result:</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":1570} -->
<figure class="wp-block-image aligncenter"><img src="https://www.scraperapi.com/wp-content/uploads/2022/08/golang-scraper-img18-1024x152.png" alt="" class="wp-image-1570"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>You can see that each product information set is inside curly braces ({…}), and the entire Slice is inside brackets ([…]).</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":4} -->
<h4>Writing the Slice into a JSON File</h4>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>We already did all the heavy lifting we needed to do. From here on, Go has a very easy-to-use JSON module that’ll handle the writing for us:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code class="">   //We pass the data to Marshal
   content, err := json.Marshal(allProducts)
   //We need to handle someone the potential error
   if err != nil {
       fmt.Println(err.Error())
   }
   //We write a new file passing the file name, data, and permission
   os.WriteFile("jack-shoes.json", content, 0644)
}</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Save it, and the necessary dependencies will update:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code class="">import (
   "encoding/json"
   "fmt"
   "os"
  
   "github.com/gocolly/colly"
)</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Let’s run our code and see what it returns:</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":1571} -->
<figure class="wp-block-image aligncenter"><img src="https://www.scraperapi.com/wp-content/uploads/2022/08/golang-scraper-img19-1024x269.png" alt="" class="wp-image-1571"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p><strong>Note:</strong>&nbsp;When you open your file, it has all the data in a single line. To display your document as the image above, right-click on the window and select&nbsp;<em>format document</em>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>We could also print the length allProducts after creating the JSON file for testing purposes:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code class="">fmt.Println(len(allProducts))</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>But we should be getting everything we want.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>That said, a single page isn’t enough for most projects. In fact, in almost all projects like this, we’ll need to scrape multiple pages to gather as much information as possible. Luckily, we can scale our project with just a few lines of code.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>8. Scraping Multiple Pages</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>If we scroll down to the bottom of the product list, we can see that J&amp;J is using a numbered pagination on their category page.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":1572} -->
<figure class="wp-block-image aligncenter"><img src="https://www.scraperapi.com/wp-content/uploads/2022/08/golang-scraper-img20.png" alt="" class="wp-image-1572"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>We could try to figure out how they construct their URLs and see if we can mimic it with a loop, but Colly has a more elegant solution, similar to <a href="https://www.scraperapi.com/blog/how-to-deal-with-pagination-in-python-step-by-step-guide-full-code/">how Scrapy navigates paginations</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code class="">c.OnHTML("a.paging-controls__next.js-page-control", func(p *colly.HTMLElement) {
    nextPage := p.Request.AbsoluteURL(p.Attr("data-href"))
    c.Visit(nextPage)
})</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>In a new&nbsp;OnHTML()&nbsp;function, we’re targeting the next button on the menu.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":1573} -->
<figure class="wp-block-image aligncenter"><img src="https://www.scraperapi.com/wp-content/uploads/2022/08/golang-scraper-img21-1024x182.png" alt="" class="wp-image-1573"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>On the internal function, we’re grabbing the value inside&nbsp;data-href&nbsp;(which contains the URL), storing it into a new variable (nextPage), and then telling our scraper to visit the page.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Running the code now, we’ll bring all product data from every page in the pagination.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":1574} -->
<figure class="wp-block-image aligncenter"><img src="https://www.scraperapi.com/wp-content/uploads/2022/08/golang-scraper-img22.png" alt="" class="wp-image-1574"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>As you can see, our Slice now contains 102 items.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>9. Avoid Getting Your Colly Scraper Blocked</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>One thing we always have to consider when scraping the web is that most sites don’t like web scrapers because many developers have no regard for the websites they extract data from.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Imagine that you want to scrape thousands of pages from a site. Every request you send takes resources away from the real users, creating more expenses for the site’s owner and possibly hurting the user experience with slow load times.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>That’s why we should always use&nbsp;<a href="https://www.scraperapi.com/blog/web-scraping-best-practices/">web scraping best practices</a>&nbsp;to ensure we’re not hurting our target sites.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>However, there’s another side of the story. To prevent scrapers from accessing the site, more and more websites implement anti-scraping systems designed to identify and block scrapers.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Although scraping a few pages won’t raise any flags – in most cases – scraping multiple pages will definitely put your IP and scraper at risk.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To avoid these measures, we would have to create a function that changes our IP address, have access to a pool of IP addresses for our script to rotate between, create some way to deal with CAPTCHAs, and handle javascript pages – which are becoming more common.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Or we could just send our HTTP request through ScraperAPI’s server and let them handle everything automatically:</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>1. First, we’ll only need to <a href="https://www.scraperapi.com/signup">create a free ScraperAPI account to redeem 5000 free API credits</a> and get access to our API key from the dashboard.<br>2. For simplicity, we’ll delete the <code>colly.AllowedDomains("www.jackjones.com")</code> setting from the <code>collector</code>.<br>3. We’ll add the ScraperAPI endpoint to our initial <code>.Visit()</code> function like this:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code class="">c.Visit("http://api.scraperapi.com?api_key={yourApiKey}&amp;url=https://www.jackjones.com/nl/en/jj/shoes/")</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>4. And make a similar change for how we visit the next page:<br></p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code class="">c.Visit("http://api.scraperapi.com?api_key={yourApiKey}&amp;url=" + nextPage)</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>5. For it to work properly and avoid errors, we’ll need to change Colly’s 10 second default timeout to at least 60 seconds to give our scraper enough time to handle any headers, CAPTCHAs, etc. We’ll use the sample code from Colly’s documentation – but change the timeout from 30 to 60:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code class="">c.WithTransport(&amp;http.Transport{
    DialContext: (&amp;net.Dialer{
        Timeout:   60 * time.Second,
        KeepAlive: 30 * time.Second,
        DualStack: true,
    }).DialContext,
    MaxIdleConns:          100,
    IdleConnTimeout:       90 * time.Second,
    TLSHandshakeTimeout:   10 * time.Second,
    ExpectContinueTimeout: 1 * time.Second,
})</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Add this code after creating the collector.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Running our code will bring back the same data as before. With the difference being that ScraperAPI will rotate our IP address for each request sent, look for the best “proxy + headers” combination to ensure a successful request, and handle any other complexities our scraper could encounter.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":1575} -->
<figure class="wp-block-image aligncenter"><img src="https://www.scraperapi.com/wp-content/uploads/2022/08/golang-scraper-img23-1024x168.png" alt="" class="wp-image-1575"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Colly will save all the data into a formatted JSON file that we can use in other applications or projects.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>With just a few small changes to the code, you can scrape any website you need, as long as the information is live in the HTML file.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>However, we can also configure our ScraperAPI endpoint to render JavaScript content before returning the HTML document. So unless the content is behind an event (like clicking a button), you should also be able to grab dynamic content without a problem.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Wrapping Up: Full Colly Web Scraper Code</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Congratulations, you created your first Colly web scraper! If you’ve followed along, here’s how your codebase should look:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code class="">package main
  
import (
   "encoding/json"
   "fmt"
   "net"
   "net/http"
   "os"
   "time"
  
   "github.com/gocolly/colly"
)
  
type products struct {
   Name  string `json:"name"`
   Price string `json:"price"`
   URL   string `json:"url"`
}
  
func main() {
   c := colly.NewCollector()
   c.WithTransport(&amp;http.Transport{
       DialContext: (&amp;net.Dialer{
           Timeout:   60 * time.Second,
           KeepAlive: 30 * time.Second,
           DualStack: true,
       }).DialContext,
       MaxIdleConns:          100,
       IdleConnTimeout:       90 * time.Second,
       TLSHandshakeTimeout:   10 * time.Second,
       ExpectContinueTimeout: 1 * time.Second,
   })
  
   var allProducts []products
  
   c.OnRequest(func(r *colly.Request) {
       fmt.Println("Scraping:", r.URL)
   })
  
   c.OnResponse(func(r *colly.Response) {
       fmt.Println("Status:", r.StatusCode)
   })
  
   c.OnHTML("div.product-tile__content-wrapper", func(h *colly.HTMLElement) {
       products := products{
           Name:  h.ChildText("a.product-tile__name__link.js-product-tile-link"),
           Price: h.ChildText("em.value__price"),
           URL:   h.ChildAttr("a.product-tile__name__link.js-product-tile-link", "href"),
       }
  
       allProducts = append(allProducts, products)
   })
  
   c.OnHTML("a.paging-controls__next.js-page-control", func(p *colly.HTMLElement) {
       nextPage := p.Request.AbsoluteURL(p.Attr("data-href"))
       c.Visit("http://api.scraperapi.com?api_key={yourApiKey}&amp;url=" + nextPage)
   })
  
   c.OnError(func(r *colly.Response, err error) {
       fmt.Println("Request URL:", r.Request.URL, "failed with response:", r, "\nError:", err)
   })
  
   c.Visit("http://api.scraperapi.com?api_key={yourApiKey}&amp;url=https://www.jackjones.com/nl/en/jj/shoes/")
  
   content, err := json.Marshal(allProducts)
   if err != nil {
       fmt.Println(err.Error())
   }
   os.WriteFile("jack-shoes.json", content, 0644)
   fmt.Println("Total products: ", len(allProducts))
}</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Web scraping is one of the most powerful tools to have in your data collection arsenal. However, remember that every website is built somewhat differently. Focus on the fundamentals of website structure, and you’ll be able to solve any problem that comes your way.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Until next time, happy scraping!</p>
<!-- /wp:paragraph -->