Building a Web Scraper in Golang: Complete Tutorial
Tags: webscraping, tutorial, beginners, scraping
Published: 2022-11-07 07:20:45

<span class="c14 c25 c65"></span></p><p class="c20"><span class="c71 c73 c54">Ever wondered how to build a web scraper in Golang? Check out this practical tutorial. </span></p><p class="c20"><span class="c71 c73 c54">Golang, or Go, is designed to leverage the static typing and run-time efficiency of C and usability of Python and JavaScript, with added features of high-performance networking and multiprocessing. It&rsquo;s also compiled and excels in concurrency, making it quick.</span></p><p class="c20"><span class="c71 c73 c54">This article will guide you through the step-by-step process of writing a fast and efficient Golang web scraper that can extract public data from a target website.</span></p><h2 class="c67" id="h.613fo2vbcbb2"><span class="c53 c14 c54">Installing Go</span></h2><p class="c20"><span class="c54">To start, head over to the </span><span class="c52"><a class="c10" href="https://www.google.com/url?q=https://go.dev/dl/&amp;sa=D&amp;source=editors&amp;ust=1667206257780327&amp;usg=AOvVaw2KTQbzmVYlzdbNHgSzzEJ-">Go downloads</a></span><span class="c71 c73 c54">&nbsp;page. Here you can download all of the common installers, such as Windows MSI installer, macOS Package, and Linux tarball. Go is open-source, meaning that if you wish to compile Go on your own, you can download the source code as well.</span></p><p class="c20"><span class="c71 c73 c54">A package manager facilitates working with first-party and third-party libraries by helping you to define and download project dependencies. The manager pins down version changes, allowing you to upgrade your dependencies without fear of breaking the established infrastructure.</span></p><h3 class="c2" id="h.qtas58bmodz"><span class="c13">Installing Go on macOS</span></h3><p class="c20"><span class="c54">If you prefer package managers, you can use </span><span class="c52"><a class="c10" href="https://www.google.com/url?q=https://brew.sh/&amp;sa=D&amp;source=editors&amp;ust=1667206257780847&amp;usg=AOvVaw2PH6-HbjUjyZgr6Z0Q_REV">Homebrew</a></span><span class="c71 c73 c54">&nbsp;on macOS. Open the terminal and enter the following:</span></p><p class="c20"><span class="c71 c15 c73">brew install go</span></p><h3 class="c2" id="h.hwjmd1kbtemz"><span class="c13">Installing Go on Windows</span></h3><p class="c20"><span class="c54">On Windows, you can use the </span><span class="c52"><a class="c10" href="https://www.google.com/url?q=https://chocolatey.org/&amp;sa=D&amp;source=editors&amp;ust=1667206257781339&amp;usg=AOvVaw3yydZ6NgsoAV-MMiR8oye_">Chocolatey</a></span><span class="c50 c14">&nbsp;package manager. Open the command prompt and enter the following:</span></p><p class="c20"><span class="c50 c14 c51">choco install golang</span></p><h3 class="c2" id="h.pox7ocvlkzyb"><span class="c13">Installing Go on Linux</span></h3><p class="c20"><span class="c71 c54 c73">Installing Go on Linux requires five simple steps: </span></p><p class="c20"><span class="c50 c14">1. Remove previous Go installations (if any) using the following command:</span></p><p class="c20"><span class="c15">rm </span><span class="c11">-</span><span class="c15">rf </span><span class="c11">/</span><span class="c15">usr</span><span class="c11">/</span><span class="c15">local</span><span class="c11">/</span><span class="c50 c14 c51">go </span></p><p class="c20"><span class="c25 c31">2. Download the </span><span class="c54 c31 c44">GO for Linux</span><span class="c25 c31">&nbsp;package; head over to the </span><span class="c52 c31"><a class="c10" href="https://www.google.com/url?q=https://go.dev/dl/&amp;sa=D&amp;source=editors&amp;ust=1667206257782096&amp;usg=AOvVaw2c9E0-ZzM9RkSk4YXo85-D">Go downloads</a></span><span class="c14 c26 c25">&nbsp;page, or use:</span></p><p class="c20"><span class="c15">wget https</span><span class="c11">://</span><span class="c15">go</span><span class="c36">.</span><span class="c15">dev</span><span class="c11">/</span><span class="c15">dl</span><span class="c11">/</span><span class="c15">go1</span><span class="c36">.</span><span class="c34">19.2</span><span class="c36">.</span><span class="c15">linux</span><span class="c11">-</span><span class="c15">amd64</span><span class="c36">.</span><span class="c15">tar</span><span class="c36">.</span><span class="c71 c15 c73">gz</span></p><p class="c20"><span class="c25 c31">3. Once the </span><span class="c42 c44">.tar.gz</span><span class="c42">&nbsp;</span><span class="c25 c31">file is downloaded, extract the archive in the </span><span class="c42">/usr/local</span><span class="c14 c26 c25">directory through:</span></p><p class="c20"><span class="c15">tar </span><span class="c11">-</span><span class="c34">C</span><span class="c15">&nbsp;</span><span class="c11">/</span><span class="c15">usr</span><span class="c11">/</span><span class="c15">local </span><span class="c11">-</span><span class="c15">xzf go1</span><span class="c36">.</span><span class="c34">19.2</span><span class="c36">.</span><span class="c15">linux</span><span class="c11">-</span><span class="c15">amd64</span><span class="c36">.</span><span class="c15">tar</span><span class="c36">.</span><span class="c71 c15 c73">gz</span></p><p class="c20"><span class="c25 c31">4. Add the Go path to the </span><span class="c42">PATH</span><span class="c25 c31">&nbsp;environment variable by adding the following line into </span><span class="c42">$HOME/.profile</span><span class="c25 c31">&nbsp;file, or for a system-wide installation, add it in </span><span class="c42">/etc/profile</span><span class="c14 c26 c25">&nbsp;file: </span></p><p class="c20"><span class="c51 c59">export</span><span class="c15">&nbsp;</span><span class="c34">PATH</span><span class="c11">=</span><span class="c15">$</span><span class="c34">PATH</span><span class="c11">:/</span><span class="c15">usr</span><span class="c11">/</span><span class="c15">local</span><span class="c11">/</span><span class="c15">go</span><span class="c11">/</span><span class="c50 c14 c51">bin</span></p><p class="c20"><span class="c25">5. Use the source </span><span class="c42">$HOME/.profile </span><span class="c25">command to apply changes in the environment variable of the</span><span class="c42">&nbsp;.profile </span><span class="c22 c14">file.</span></p><p class="c20"><span class="c25">Now, you can use the </span><span class="c42">go version</span><span class="c22 c14">&nbsp;command to verify that the Go version is installed.</span></p><p class="c20"><span class="c22 c14">Once Go is installed, you can use any code editor or an integrated development environment (IDE) that supports Go.</span></p><h3 class="c2" id="h.a3zqnubjuv6a"><span class="c13 c48">How to install Golang in Visual Studio Code?</span></h3><p class="c20"><span class="c22 c14">While you can use virtually any code editor to write a Go program, one of the most commonly used ones is Visual Studio Code. </span></p><p class="c43"><span class="c25">For Golang to be supported, you&rsquo;ll need to install the Go extension. To do that, select the Extensions icon on the left side, type in </span><span class="c25 c44">Go</span><span class="c22 c14">&nbsp;in the search bar, and simply click Install :</span></p>
![intalling golang](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ax8gn51lqmenjahzvw9u.png)

<p class="c20"><span style="overflow: hidden; display: inline-block; margin: 0.00px 0.00px; border: 0.00px solid #000000; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px); width: 624.00px; height: 226.67px;"><img alt="" src="images/image3.png" style="width: 624.00px; height: 226.67px; margin-left: 0.00px; margin-top: 0.00px; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px);" title=""></span></p><p class="c70"><span class="c71 c57 c44 c48">Go extension for Visual Studio Code</span></p><p class="c20 c1"><span class="c22 c14"></span></p><p class="c20"><span class="c25">Once you&rsquo;ve finished installing the Go</span><span class="c25 c44">&nbsp;</span><span class="c22 c14">extension, you&rsquo;ll need to update Go tools. </span></p><p class="c20"><span class="c25">Press Ctrl+Shift+P to open the Show All Commands window and search for </span><span class="c42">Go: Install/Update tools</span><span class="c22 c14">.</span></p><p class="c43"><span class="c22 c14">Take a look at the image below to see how it looks:</span>
![nstalled golang](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/e1w4vect1p2od3x7ntvn.png)


</p><p class="c20"><span style="overflow: hidden; display: inline-block; margin: 0.00px 0.00px; border: 0.00px solid #000000; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px); width: 624.00px; height: 161.33px;"><img alt="" src="images/image1.png" style="width: 624.00px; height: 161.33px; margin-left: 0.00px; margin-top: 0.00px; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px);" title=""></span></p><p class="c70"><span class="c57 c44 c48 c71">Go tools for Visual Studio Code</span></p><p class="c20 c1"><span class="c22 c14"></span></p><p class="c20"><span class="c22 c14">After selecting all the available Go tools, click on the OK button to install.</span></p><p class="c20"><span class="c25">We can also use a separate IDE (e.g., </span><span class="c32"><a class="c10" href="https://www.google.com/url?q=https://www.jetbrains.com/go/download/%23section%3Dmac&amp;sa=D&amp;source=editors&amp;ust=1667206257784959&amp;usg=AOvVaw2ZmVtXKQrps5H8vBHpu6nx">GoLand</a></span><span class="c22 c14">) to write, debug, compile, and run the Go projects. Both Visual Studio Code and GoLand are available for Windows, macOS, and Linux.</span></p><h2 class="c67" id="h.8mzatsxfgxtv"><span class="c53 c14 c25">Web scraping frameworks</span></h2><p class="c20"><span class="c25">Go offers a wide selection of frameworks. Some are simple packages with core functionality, while others, such as </span><span class="c32"><a class="c10" href="https://www.google.com/url?q=https://github.com/MontFerret/ferret&amp;sa=D&amp;source=editors&amp;ust=1667206257785513&amp;usg=AOvVaw3U5wzFg87uxPdm20Yf5mV7">Ferret</a></span><span class="c25">, </span><span class="c32"><a class="c10" href="https://www.google.com/url?q=https://github.com/PuerkitoBio/gocrawl&amp;sa=D&amp;source=editors&amp;ust=1667206257785681&amp;usg=AOvVaw1SgZUbZ7tsSIdpf2ipBctR">Gocrawl</a></span><span class="c25">, </span><span class="c32"><a class="c10" href="https://www.google.com/url?q=https://github.com/anaskhan96/soup&amp;sa=D&amp;source=editors&amp;ust=1667206257785834&amp;usg=AOvVaw1oGjwjKoAsFyzsuBlcgKlR">Soup</a></span><span class="c25">, and </span><span class="c32"><a class="c10" href="https://www.google.com/url?q=https://github.com/hakluke/hakrawler&amp;sa=D&amp;source=editors&amp;ust=1667206257785981&amp;usg=AOvVaw3p_LG2VMSplH05nyCrIrGC">Hakrawler</a></span><span class="c22 c14">, provide a complete web scraping infrastructure to simplify data extraction. Let&rsquo;s have a brief overview of these frameworks. </span></p><h3 class="c2" id="h.culjbsxw11ts"><span class="c13 c48">Ferret</span></h3><p class="c20"><span class="c22 c14">Ferret is a fast, portable, and extensible framework for designing Go web scrapers. It&rsquo;s pretty easy to use as the user simply needs to write a declarative query expressing which data to extract. Ferret handles the HTML retrieving and parsing part by itself.</span></p><h3 class="c2" id="h.a5967kzhzoqx"><span class="c13 c48">Gocrawl</span></h3><p class="c20"><span class="c25">Gocrawl is a web scraping framework written in Go language. It gives complete control to visit, inspect, and query different URLs using </span><span class="c32"><a class="c10" href="https://www.google.com/url?q=https://github.com/PuerkitoBio/goquery&amp;sa=D&amp;source=editors&amp;ust=1667206257786473&amp;usg=AOvVaw2JS26vSj86oQ5DExUvf16e">goquery</a></span><span class="c22 c14">. This framework allows concurrent execution as it applies goroutines. </span></p><h3 class="c2" id="h.d6bdjugj40oo"><span class="c13 c48">Soup</span></h3><p class="c20"><span class="c22 c14">Soup is a small web scraping framework that can be used to implement a Go web scraper. It provides an API for retrieving and parsing the content. </span></p><h3 class="c2" id="h.kpo9b8x4iiyi"><span class="c13 c48">Hakrawler</span></h3><p class="c20"><span class="c25">Hakrawler is a simple and fast web crawler available with Go language. It&rsquo;s a simplified version of the most popular Golang web scraping framework &ndash; </span><span class="c32"><a class="c10" href="https://www.google.com/url?q=http://go-colly.org/&amp;sa=D&amp;source=editors&amp;ust=1667206257786958&amp;usg=AOvVaw3l9N04aV9YeuTi19SclJ_3">GoColly</a></span><span class="c22 c14">. It&rsquo;s mainly used to extract URLs and JavaScript file locations.</span></p><h3 class="c2" id="h.vfqv1sy387ys"><span class="c13 c48">GoQuery</span></h3><p class="c20"><span class="c25">GoQuery is a framework that provides functionalities similar to </span><span class="c32"><a class="c10" href="https://www.google.com/url?q=https://jquery.com/&amp;sa=D&amp;source=editors&amp;ust=1667206257787287&amp;usg=AOvVaw22DkolvmSUUlOrfCEYoHwW">jQuery</a></span><span class="c25">&nbsp;in Golang. &nbsp;It uses two basic Go packages &ndash; </span><span class="c32"><a class="c10" href="https://www.google.com/url?q=https://pkg.go.dev/golang.org/x/net/html&amp;sa=D&amp;source=editors&amp;ust=1667206257787458&amp;usg=AOvVaw1CXgqmbXoheoyqVdE1hdun">net/html</a></span><span class="c25">&nbsp;(a Golang HTML parser) and </span><span class="c32"><a class="c10" href="https://www.google.com/url?q=https://github.com/andybalholm/cascadia&amp;sa=D&amp;source=editors&amp;ust=1667206257787651&amp;usg=AOvVaw1L5Cjv_w7flUvnBhZjt1p4">cascadia</a></span><span class="c14 c22">&nbsp;(a CSS Selector). &nbsp;</span></p><h3 class="c2" id="h.brsqxlwm00m8"><span class="c13 c48">Colly</span></h3><p class="c20"><span class="c25">The most popular framework for writing web scrapers in Go is </span><span class="c32"><a class="c10" href="https://www.google.com/url?q=https://github.com/gocolly/colly&amp;sa=D&amp;source=editors&amp;ust=1667206257788035&amp;usg=AOvVaw0Twdx4Heh4QEHH-fFO61X4">Colly</a></span><span class="c22 c14">. </span></p><p class="c20"><span class="c25">Colly is a fast scraping framework that can be used to write any kind of crawler, scraper, or spider. If you want to know more about differentiating a scraper from a crawler, check </span><span class="c32"><a class="c10" href="https://www.google.com/url?q=https://oxylabs.io/blog/crawling-vs-scraping&amp;sa=D&amp;source=editors&amp;ust=1667206257788325&amp;usg=AOvVaw2ms46_iHtFOs5yTkFykrzM">this article</a></span><span class="c22 c14">.</span></p><p class="c20"><span class="c22 c14">Colly has a clean API, handles cookies and sessions automatically, supports caching and robots.txt, and, most importantly, it&rsquo;s fast. Colly offers distributed scraping, HTTP request delays, and concurrency per domain.</span></p><p class="c20"><span class="c25">In this Golang Colly tutorial, we&rsquo;ll be using Colly to scrape </span><span class="c32"><a class="c10" href="https://www.google.com/url?q=https://books.toscrape.com/&amp;sa=D&amp;source=editors&amp;ust=1667206257788689&amp;usg=AOvVaw3E6_Q3gyt20-sguC4rp1me">books.toscrape.com</a></span><span class="c22 c14">. The website is a dummy book store for practicing web scraping.</span></p><h3 class="c2" id="h.l4nhwzdpd8x3"><span class="c13 c48">How to import a package in Golang?</span></h3><p class="c20"><span class="c25">As the name suggests, the </span><span class="c42">import</span><span class="c25 c44">&nbsp;</span><span class="c25">directive imports different packages into a Golang program. For example, the </span><span class="c42">fmt</span><span class="c25">&nbsp;package has definitions of formatted I/O library functions and can be imported using the </span><span class="c42">import</span><span class="c22 c14">&nbsp;preprocessor directive, as shown in the following snippet:</span></p><p class="c20"><span class="c12">package</span><span class="c8">&nbsp;main</span></p><p class="c20"><span class="c12">import</span><span class="c37 c15">&nbsp;</span><span class="c17">&quot;fmt&quot;</span></p><p class="c20"><span class="c37 c15">func </span><span class="c5">main</span><span class="c9">()</span><span class="c37 c15">&nbsp;</span><span class="c9">{</span></p><p class="c20"><span class="c37 c15">&nbsp; &nbsp; fmt</span><span class="c9">.</span><span class="c5">Println</span><span class="c9">(</span><span class="c17">&quot;Hello World&quot;</span><span class="c9">)</span></p><p class="c20"><span class="c9">}</span></p><p class="c20"><span class="c25 c31">The code above first imports the </span><span class="c42 c31">fmt</span><span class="c25 c31">&nbsp;package and then uses its </span><span class="c42 c31">Println </span><span class="c25 c31">function to display the </span><span class="c42 c31">Hello World</span><span class="c25 c31 c44">&nbsp;</span><span class="c14 c26 c25">text in the console. </span></p><p class="c20"><span class="c25 c31">We can also import multiple packages using a single </span><span class="c42 c31">import</span><span class="c25 c31 c44">&nbsp;</span><span class="c14 c26 c25">directive, as you can see from the example below:</span></p><p class="c20"><span class="c12">package</span><span class="c8">&nbsp;main</span></p><p class="c20"><span class="c12">import</span><span class="c37 c15">&nbsp;</span><span class="c9">(</span></p><p class="c20"><span class="c37 c15">&nbsp; &nbsp; </span><span class="c17">&quot;fmt&quot;</span></p><p class="c20"><span class="c37 c15">&nbsp; &nbsp; </span><span class="c17">&quot;math/rand&quot;</span></p><p class="c20"><span class="c9">)</span></p><p class="c20"><span class="c37 c15">func </span><span class="c5">main</span><span class="c9">()</span><span class="c37 c15">&nbsp;</span><span class="c9">{</span></p><p class="c20"><span class="c37 c15">&nbsp; &nbsp; fmt</span><span class="c9">.</span><span class="c5">Println</span><span class="c9">(</span><span class="c17">&quot;Hello World&quot;</span><span class="c9">)</span></p><p class="c20"><span class="c37 c15">&nbsp; &nbsp; fmt</span><span class="c9">.</span><span class="c5">Println</span><span class="c9">(</span><span class="c37 c15">rand</span><span class="c9">.</span><span class="c5">Intn</span><span class="c9">(</span><span class="c27">25</span><span class="c9">))</span></p><p class="c20"><span class="c9">}</span></p><h2 class="c67" id="h.ff11qdvp1mt4"><span class="c14 c25 c53">Parsing HTML with Colly</span></h2><p class="c20"><span class="c14 c26 c25">To easily extract structured data from the URLs and HTML, the first step is to create a project and install Colly.</span></p><p class="c20"><span class="c14 c26 c25">Create a new directory and navigate there using the terminal. From this directory, run the following command:</span></p><p class="c20"><span class="c37 c15">go mod init oxylabs</span><span class="c9">.</span><span class="c37 c15">io</span><span class="c37 c11">/</span><span class="c37 c15">web</span><span class="c37 c11">-</span><span class="c37 c15">scraping</span><span class="c37 c11">-</span><span class="c12">with</span><span class="c37 c11">-</span><span class="c37 c15">go</span></p><p class="c20"><span class="c25 c31">This will create a </span><span class="c42 c31">go.mod</span><span class="c14 c26 c25">&nbsp;file that contains the following lines with the name of the module and the version of Go. In this case, the version of Go is 1.17:</span></p><p class="c20"><span class="c37 c15">module oxylabs</span><span class="c9">.</span><span class="c37 c15">io</span><span class="c37 c11">/</span><span class="c37 c15">web</span><span class="c37 c11">-</span><span class="c37 c15">scraping</span><span class="c37 c11">-</span><span class="c12">with</span><span class="c37 c11">-</span><span class="c8">go</span></p><p class="c20"><span class="c37 c15">go </span><span class="c27">1.17</span></p><p class="c20"><span class="c50 c14">Next, run the following command to install Colly and its dependencies:</span></p><p class="c20"><span class="c37 c15">go </span><span class="c12">get</span><span class="c37 c15">&nbsp;github</span><span class="c9">.</span><span class="c37 c15">com</span><span class="c37 c11">/</span><span class="c37 c15">gocolly</span><span class="c37 c11">/</span><span class="c37 c15">colly</span></p><p class="c20"><span class="c54">This command will also update the </span><span class="c42">go.mod</span><span class="c54">&nbsp;file with all the required dependencies as well as create a </span><span class="c42">go.sum</span><span class="c71 c73 c54">&nbsp;file.</span></p><p class="c20"><span class="c54">We are now ready to write the web scraper code file. Create a new file, save it as </span><span class="c42">books.go</span><span class="c50 c14">&nbsp;and enter the following code:</span></p><p class="c20"><span class="c12">package</span><span class="c8">&nbsp;main</span></p><p class="c20 c1"><span class="c8"></span></p><p class="c20"><span class="c12">import</span><span class="c37 c15">&nbsp;</span><span class="c9">(</span></p><p class="c20"><span class="c37 c15">&nbsp; &nbsp;</span><span class="c17">&quot;encoding/csv&quot;</span></p><p class="c20"><span class="c37 c15">&nbsp; &nbsp;</span><span class="c17">&quot;fmt&quot;</span></p><p class="c20"><span class="c37 c15">&nbsp; &nbsp;</span><span class="c17">&quot;log&quot;</span></p><p class="c20"><span class="c37 c15">&nbsp; &nbsp;</span><span class="c17">&quot;os&quot;</span></p><p class="c20 c1"><span class="c8"></span></p><p class="c20"><span class="c37 c15">&nbsp; &nbsp;</span><span class="c17">&quot;github.com/gocolly/colly&quot;</span></p><p class="c20"><span class="c9">)</span></p><p class="c20"><span class="c37 c15">func </span><span class="c5">main</span><span class="c9">()</span><span class="c37 c15">&nbsp;</span><span class="c9">{</span></p><p class="c20"><span class="c37 c15">&nbsp; &nbsp;</span><span class="c37 c51 c64">// Scraping code here</span></p><p class="c20"><span class="c37 c15">&nbsp; &nbsp;fmt</span><span class="c9">.</span><span class="c5">Println</span><span class="c9">(</span><span class="c17">&quot;Done&quot;</span><span class="c9">)</span></p><p class="c20"><span class="c9">}</span></p><p class="c20"><span class="c71 c73 c54">The first line is the name of the package. Next, there are some built-in packages being imported as well as Colly itself.</span></p><p class="c20"><span class="c54">The </span><span class="c42">main()</span><span class="c14 c50">&nbsp;function is going to be the entry point of the program. This is where we&rsquo;ll write the code for the web scraper.</span></p><h3 class="c2" id="h.wdftotajiiwc"><span class="c13">Sending HTTP requests with Colly</span></h3><p class="c20"><span class="c71 c73 c54">The fundamental component of a Colly web scraper is the Collector. The Collector makes HTTP requests and traverses HTML pages.</span></p><p class="c20"><span class="c71 c73 c54">The Collector exposes multiple events. We can hook custom functions that execute when these events are raised. These functions are anonymous and pass as a parameter.</span></p><p class="c20"><span class="c50 c14">First, to create a new Collector using default settings, enter this line in your code:</span></p><p class="c20"><span class="c27">c</span><span class="c37 c15">&nbsp;</span><span class="c37 c11">:=</span><span class="c37 c15">&nbsp;colly</span><span class="c9">.</span><span class="c5">NewCollector</span><span class="c9">()</span></p><p class="c0"><span class="c14 c26 c25">There are many other parameters that can be used to control the behavior of the Collector. In this example, we are going to limit the allowed domains. Change the line as follows:</span></p><p class="c0 c1"><span class="c14 c26 c15"></span></p><p class="c0"><span class="c27">c</span><span class="c37 c15">&nbsp;</span><span class="c37 c11">:=</span><span class="c37 c15">&nbsp;colly</span><span class="c9">.</span><span class="c5">NewCollector</span><span class="c9">(</span></p><p class="c0"><span class="c37 c15">&nbsp; &nbsp;colly</span><span class="c9">.</span><span class="c5">AllowedDomains</span><span class="c9">(</span><span class="c17">&quot;books.toscrape.com&quot;</span><span class="c9">),</span></p><p class="c0"><span class="c9 c14">)</span></p><p class="c0 c1"><span class="c14 c37 c48 c57"></span></p><p class="c20"><span class="c25 c31">Once the instance is available, the </span><span class="c42 c31">Visit()</span><span class="c14 c26 c25">&nbsp;function can be called to start the scraper. However, before doing so, it&rsquo;s important to hook up to a few events.</span></p><p class="c20"><span class="c25 c31">The </span><span class="c42 c31">OnRequest</span><span class="c14 c26 c25">&nbsp;event is raised when an HTTP request is sent to a URL. This event is used to track which URL is being visited. Simple use of an anonymous function that prints the URL being requested is as follows: </span></p><p class="c0"><span class="c37 c15">c</span><span class="c9">.</span><span class="c5">OnRequest</span><span class="c9">(</span><span class="c5">func</span><span class="c9">(</span><span class="c37 c15">r </span><span class="c37 c11">*</span><span class="c37 c15">colly</span><span class="c9">.</span><span class="c37 c15">Request</span><span class="c9">)</span><span class="c37 c15">&nbsp;</span><span class="c9">{</span></p><p class="c0"><span class="c37 c15">&nbsp; &nbsp;fmt</span><span class="c9">.</span><span class="c5">Println</span><span class="c9">(</span><span class="c17">&quot;Visiting&quot;</span><span class="c9">,</span><span class="c37 c15">&nbsp;r</span><span class="c9">.</span><span class="c27">URL</span><span class="c9">)</span></p><p class="c20"><span class="c9 c14">})</span></p><p class="c20"><span class="c14 c26 c25">Note that the anonymous function being sent as a parameter here is a callback function. It means that this function will be called when the event is raised.</span></p><p class="c20"><span class="c25 c31">Similarly, </span><span class="c42 c31">OnResponse</span><span class="c14 c26 c25">&nbsp;can be used to examine the response. The following is one such example:</span></p><p class="c20"><span class="c37 c15">c</span><span class="c9">.</span><span class="c5">OnResponse</span><span class="c9">(</span><span class="c5">func</span><span class="c9">(</span><span class="c37 c15">r </span><span class="c37 c11">*</span><span class="c37 c15">colly</span><span class="c9">.</span><span class="c37 c15">Response</span><span class="c9">)</span><span class="c37 c15">&nbsp;</span><span class="c9">{</span></p><p class="c20"><span class="c37 c15">&nbsp; &nbsp;fmt</span><span class="c9">.</span><span class="c5">Println</span><span class="c9">(</span><span class="c37 c15">r</span><span class="c9">.</span><span class="c37 c15">StatusCode</span><span class="c9">)</span></p><p class="c20"><span class="c9">})</span></p><p class="c20"><span class="c25 c31">The </span><span class="c42 c31">OnHTML</span><span class="c14 c26 c25">&nbsp;event can be used to take action when a specific HTML element is found.</span></p><h3 class="c2" id="h.1hy62rlpxxl8"><span class="c13 c48">Locating HTML elements via CSS selector</span></h3><p class="c20"><span class="c25 c31">The </span><span class="c42 c31">OnHTML</span><span class="c14 c26 c25">&nbsp;event can be hooked using the CSS selector and a function that executes when the HTML elements matching the selector are found.</span></p><p class="c20"><span class="c25 c31">For example, the following function executes when a </span><span class="c42 c31">title</span><span class="c14 c26 c25">&nbsp;tag is encountered:</span></p><p class="c20"><span class="c37 c15">c</span><span class="c9">.</span><span class="c5">OnHTML</span><span class="c9">(</span><span class="c17">&quot;title&quot;</span><span class="c9">,</span><span class="c37 c15">&nbsp;</span><span class="c5">func</span><span class="c9">(</span><span class="c37 c15">e </span><span class="c37 c11">*</span><span class="c37 c15">colly</span><span class="c9">.</span><span class="c37 c15">HTMLElement</span><span class="c9">)</span><span class="c37 c15">&nbsp;</span><span class="c9">{</span></p><p class="c20"><span class="c37 c15">&nbsp; &nbsp;fmt</span><span class="c9">.</span><span class="c5">Println</span><span class="c9">(</span><span class="c37 c15">e</span><span class="c9">.</span><span class="c37 c15">Text</span><span class="c9">)</span></p><p class="c20"><span class="c9">})</span></p><p class="c20"><span class="c25 c31">This function extracts the text inside the </span><span class="c42 c31">title</span><span class="c25 c31">&nbsp;tag and prints it. Putting together all we have gone through so far, the </span><span class="c42 c31">main()</span><span class="c14 c26 c25">&nbsp;function is as follows:</span></p><p class="c20"><span class="c6">func </span><span class="c3">main</span><span class="c19">()</span><span class="c6">&nbsp;</span><span class="c19">{</span></p><p class="c20"><span class="c6">&nbsp; &nbsp;</span><span class="c30">c</span><span class="c6">&nbsp;</span><span class="c37 c56">:=</span><span class="c6">&nbsp;colly</span><span class="c19">.</span><span class="c3">NewCollector</span><span class="c19">(</span></p><p class="c20"><span class="c6">colly</span><span class="c19">.</span><span class="c3">AllowedDomains</span><span class="c19">(</span><span class="c37 c62">&quot;books.toscrape.com&quot;</span><span class="c19">),</span></p><p class="c20"><span class="c6">&nbsp; &nbsp;</span><span class="c19">)</span></p><p class="c20 c1"><span class="c6 c14"></span></p><p class="c20"><span class="c6">&nbsp; &nbsp;c</span><span class="c19">.</span><span class="c3">OnHTML</span><span class="c19">(</span><span class="c37 c62">&quot;title&quot;</span><span class="c19">,</span><span class="c6">&nbsp;</span><span class="c3">func</span><span class="c19">(</span><span class="c6">e </span><span class="c37 c56">*</span><span class="c6">colly</span><span class="c19">.</span><span class="c6">HTMLElement</span><span class="c19">)</span><span class="c6">&nbsp;</span><span class="c19">{</span></p><p class="c20"><span class="c6">&nbsp; &nbsp; &nbsp; fmt</span><span class="c19">.</span><span class="c3">Println</span><span class="c19">(</span><span class="c6">e</span><span class="c19">.</span><span class="c6">Text</span><span class="c19">)</span></p><p class="c20"><span class="c6">&nbsp; &nbsp;</span><span class="c19">})</span></p><p class="c20 c1"><span class="c6 c14"></span></p><p class="c20"><span class="c6">&nbsp; &nbsp;c</span><span class="c19">.</span><span class="c3">OnResponse</span><span class="c19">(</span><span class="c3">func</span><span class="c19">(</span><span class="c6">r </span><span class="c37 c56">*</span><span class="c6">colly</span><span class="c19">.</span><span class="c6">Response</span><span class="c19">)</span><span class="c6">&nbsp;</span><span class="c19">{</span></p><p class="c20"><span class="c6">&nbsp; &nbsp; &nbsp; fmt</span><span class="c19">.</span><span class="c3">Println</span><span class="c19">(</span><span class="c6">r</span><span class="c19">.</span><span class="c6">StatusCode</span><span class="c19">)</span></p><p class="c20"><span class="c6">&nbsp; &nbsp;</span><span class="c19">})</span></p><p class="c20 c1"><span class="c6 c14"></span></p><p class="c20"><span class="c6">&nbsp; &nbsp;c</span><span class="c19">.</span><span class="c3">OnRequest</span><span class="c19">(</span><span class="c3">func</span><span class="c19">(</span><span class="c6">r </span><span class="c37 c56">*</span><span class="c6">colly</span><span class="c19">.</span><span class="c6">Request</span><span class="c19">)</span><span class="c6">&nbsp;</span><span class="c19">{</span></p><p class="c20"><span class="c6">&nbsp; &nbsp; &nbsp; fmt</span><span class="c19">.</span><span class="c3">Println</span><span class="c19">(</span><span class="c37 c62">&quot;Visiting&quot;</span><span class="c19">,</span><span class="c6">&nbsp;r</span><span class="c19">.</span><span class="c30">URL</span><span class="c19">)</span></p><p class="c20"><span class="c6">&nbsp; &nbsp;</span><span class="c19">})</span></p><p class="c20 c1"><span class="c6 c14"></span></p><p class="c20"><span class="c6">&nbsp; &nbsp;c</span><span class="c19">.</span><span class="c3">Visit</span><span class="c19">(</span><span class="c37 c62">&quot;https://books.toscrape.com/&quot;</span><span class="c19">)</span></p><p class="c20"><span class="c19">}</span></p><p class="c20"><span class="c71 c73 c54">This file can be run from the terminal as follows:</span></p><p class="c0"><span class="c37 c15">go run books</span><span class="c9">.</span><span class="c37 c15">go</span></p><p class="c0 c1"><span class="c9 c14"></span></p><p class="c0"><span class="c71 c73 c76">The output will be as follows:</span></p><p class="c0 c1"><span class="c9 c14"></span></p><p class="c0"><span class="c37 c15">Visiting https</span><span class="c37 c11">:/</span><span class="c37 c51 c63">/books.toscrape.com/</span></p><p class="c0"><span class="c27">200</span></p><p class="c0"><span class="c37 c15">All products </span><span class="c37 c11">|</span><span class="c37 c15">&nbsp;Books to Scrape </span><span class="c37 c11">-</span><span class="c8">&nbsp;Sandbox </span></p><h3 class="c2" id="h.zc7nonqp3xbz"><span class="c13 c49">Extracting the HTML elements</span></h3><p class="c20"><span class="c41">Now that we know how Colly works let&rsquo;s modify </span><span class="c33 c49">OnHTML</span><span class="c35 c14 c49">&nbsp;to extract the book titles and prices.</span></p><p class="c43"><span class="c35 c14 c49">The first step is to understand the HTML structure of the page.</span>
![web scraping html structure](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fm1svsvx2zo35li01y3n.png)

</p><p class="c0"><span style="overflow: hidden; display: inline-block; margin: 0.00px 0.00px; border: 0.00px solid #000000; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px); width: 624.00px; height: 348.00px;"><img alt="" src="images/image2.png" style="width: 624.00px; height: 348.00px; margin-left: 0.00px; margin-top: 0.00px; transform: rotate(0.00rad) translateZ(0px); -webkit-transform: rotate(0.00rad) translateZ(0px);" title=""></span></p><p class="c70"><span class="c7">The books are in the &lt;article&gt; tags</span></p><p class="c20"><span class="c41">Each book is contained in an </span><span class="c33 c49">article</span><span class="c41">&nbsp;tag that has a </span><span class="c33 c49">product_pod</span><span class="c41">&nbsp;class. The CSS selector would be </span><span class="c33 c49">.product_pod</span><span class="c35 c14 c49">.</span></p><p class="c20"><span class="c41">Next, the complete book title is found in the thumbnail image as an </span><span class="c33 c49">alt</span><span class="c41">attribute value. The CSS selector for the book title would be </span><span class="c33 c49">.image_container img</span><span class="c35 c14 c49">.</span></p><p class="c20"><span class="c41">Finally, the CSS selector for the book price would be </span><span class="c33 c49">.price_color</span><span class="c35 c14 c49">.</span></p><p class="c20"><span class="c41">The </span><span class="c33 c49">OnHTML</span><span class="c35 c14 c49">&nbsp;can be modified as follows:</span></p><p class="c20"><span class="c23">c</span><span class="c16">.</span><span class="c24">OnHTML</span><span class="c16">(</span><span class="c28">&quot;.product_pod&quot;</span><span class="c16">,</span><span class="c23">&nbsp;</span><span class="c24">func</span><span class="c16">(</span><span class="c23">e </span><span class="c37 c40">*</span><span class="c23">colly</span><span class="c16">.</span><span class="c23">HTMLElement</span><span class="c16">)</span><span class="c23">&nbsp;</span><span class="c16">{</span></p><p class="c20"><span class="c23">&nbsp; &nbsp;</span><span class="c37 c60 c69">title</span><span class="c23">&nbsp;</span><span class="c37 c40">:=</span><span class="c23">&nbsp;e</span><span class="c16">.</span><span class="c24">ChildAttr</span><span class="c16">(</span><span class="c28">&quot;.image_container img&quot;</span><span class="c16">,</span><span class="c23">&nbsp;</span><span class="c28">&quot;alt&quot;</span><span class="c16">)</span></p><p class="c20"><span class="c23">&nbsp; &nbsp;</span><span class="c37 c60 c69">price</span><span class="c23">&nbsp;</span><span class="c37 c40">:=</span><span class="c23">&nbsp;e</span><span class="c16">.</span><span class="c24">ChildText</span><span class="c16">(</span><span class="c28">&quot;.price_color&quot;</span><span class="c16">)</span></p><p class="c20"><span class="c16">})</span></p><p class="c20"><span class="c35 c14 c49">This function will execute every time a book is found on the page.</span></p><p class="c20"><span class="c41">Note the use of the </span><span class="c33 c49">ChildAttr</span><span class="c41">&nbsp;function that takes two parameters: the CSS selector and the name of the attribute &ndash; it isn&rsquo;t subtle. A better idea would be to create a data structure to hold this information. In this case, we can use </span><span class="c33 c49">struct</span><span class="c35 c14 c49">&nbsp;as follows:</span></p><p class="c0"><span class="c23">type Book struct </span><span class="c16">{</span></p><p class="c0"><span class="c23 c14">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Title string</span></p><p class="c0"><span class="c23 c14">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Price string</span></p><p class="c0"><span class="c16">}</span></p><p class="c0 c1"><span class="c8"></span></p><p class="c20"><span class="c45">The </span><span class="c33">OnHTML</span><span class="c35 c14">&nbsp;will be modified as follows:</span></p><p class="c20"><span class="c23">c</span><span class="c16">.</span><span class="c24">OnHTML</span><span class="c16">(</span><span class="c28">&quot;.product_pod&quot;</span><span class="c16">,</span><span class="c23">&nbsp;</span><span class="c24">func</span><span class="c16">(</span><span class="c23">e </span><span class="c37 c40">*</span><span class="c23">colly</span><span class="c16">.</span><span class="c23">HTMLElement</span><span class="c16">)</span><span class="c23">&nbsp;</span><span class="c16">{</span></p><p class="c20"><span class="c23">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="c37 c60 c69">book</span><span class="c23">&nbsp;</span><span class="c37 c40">:=</span><span class="c23">&nbsp;Book</span><span class="c16">{}</span></p><p class="c20"><span class="c23">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;book</span><span class="c16">.</span><span class="c23">Title </span><span class="c37 c40">=</span><span class="c23">&nbsp;e</span><span class="c16">.</span><span class="c24">ChildAttr</span><span class="c16">(</span><span class="c28">&quot;.image_container img&quot;</span><span class="c16">,</span><span class="c23">&nbsp;</span><span class="c28">&quot;alt&quot;</span><span class="c16">)</span></p><p class="c20"><span class="c23">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;book</span><span class="c16">.</span><span class="c23">Price </span><span class="c37 c40">=</span><span class="c23">&nbsp;e</span><span class="c16">.</span><span class="c24">ChildText</span><span class="c16">(</span><span class="c28">&quot;.price_color&quot;</span><span class="c16">)</span></p><p class="c20"><span class="c23">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;fmt</span><span class="c16">.</span><span class="c24">Println</span><span class="c16">(</span><span class="c23">book</span><span class="c16">.</span><span class="c23">Title</span><span class="c16">,</span><span class="c23">&nbsp;book</span><span class="c16">.</span><span class="c23">Price</span><span class="c16">)</span></p><p class="c20"><span class="c16 c14">})</span></p><p class="c20"><span class="c35 c14">For now, this web scraper is simply printing the information to the console, which isn&rsquo;t particularly useful. We&rsquo;ll revisit this function when it&rsquo;s time to save the data to a CSV file.</span></p><h2 class="c67" id="h.x4xb9a7q5zu1"><span class="c53 c14 c54">Handling pagination</span></h2><p class="c20"><span class="c45">First, we need to locate the &ldquo;next&rdquo; button and create a CSS selector. For this particular site, the CSS selector is </span><span class="c33">.next &gt; a</span><span class="c45">. Using the selector, a new function can be added to the </span><span class="c33">OnHTML</span><span class="c45">&nbsp;event. In this function, we&rsquo;ll convert a relative URL to an absolute URL. Then, we&rsquo;ll call the </span><span class="c33">Visit()</span><span class="c35 c14">&nbsp;function to crawl the converted URL:</span></p><p class="c20"><span class="c29">c</span><span class="c37 c38">.</span><span class="c37 c46">OnHTML</span><span class="c37 c38">(</span><span class="c37 c77 c75">&quot;.next &gt; a&quot;</span><span class="c37 c38">,</span><span class="c29">&nbsp;</span><span class="c37 c46">func</span><span class="c37 c38">(</span><span class="c29">e </span><span class="c37 c61">*</span><span class="c29">colly</span><span class="c37 c38">.</span><span class="c29">HTMLElement</span><span class="c37 c38">)</span><span class="c29">&nbsp;</span><span class="c37 c38">{</span></p><p class="c20"><span class="c29">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="c37 c69 c75">nextPage</span><span class="c29">&nbsp;</span><span class="c37 c61">:=</span><span class="c29">&nbsp;e</span><span class="c37 c38">.</span><span class="c29">Request</span><span class="c37 c38">.</span><span class="c37 c46">AbsoluteURL</span><span class="c37 c38">(</span><span class="c29">e</span><span class="c37 c38">.</span><span class="c37 c46">Attr</span><span class="c37 c38">(</span><span class="c37 c75 c77">&quot;href&quot;</span><span class="c37 c38">))</span></p><p class="c20"><span class="c29">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;c</span><span class="c37 c38">.</span><span class="c37 c46">Visit</span><span class="c37 c38">(</span><span class="c29">nextPage</span><span class="c37 c38">)</span></p><p class="c20"><span class="c37 c38">})</span></p><p class="c20"><span class="c35 c14">The existing function that scrapes the book information will be called on all of the resulting pages as well. No additional code is needed.</span></p><p class="c20"><span class="c35 c14">Now that we have the data from all of the pages, it&rsquo;s time to save it to a CSV file.</span></p><h2 class="c67" id="h.ch3fhbhotyrl"><span class="c53 c14 c54">Writing data to a CSV file</span></h2><p class="c20"><span class="c35 c14">The built-in CSV library can be used to save the structure to CSV files. If you want to save the data in JSON format, you can use the JSON library as well.</span></p><p class="c20"><span class="c35 c14">To create a new CSV file, enter the following code before creating the Colly collector:</span></p><p class="c20"><span class="c4">file</span><span class="c16">,</span><span class="c4">&nbsp;</span><span class="c37 c60 c69">err</span><span class="c4">&nbsp;</span><span class="c37 c40">:=</span><span class="c4">&nbsp;os</span><span class="c16">.</span><span class="c24">Create</span><span class="c16">(</span><span class="c28">&quot;export.csv&quot;</span><span class="c16">)</span></p><p class="c20"><span class="c37 c60 c59">if</span><span class="c4">&nbsp;err </span><span class="c37 c40">!=</span><span class="c4">&nbsp;nil </span><span class="c16">{</span></p><p class="c20"><span class="c4">&nbsp; &nbsp;log</span><span class="c16">.</span><span class="c24">Fatal</span><span class="c16">(</span><span class="c4">err</span><span class="c16">)</span></p><p class="c20"><span class="c16">}</span></p><p class="c20"><span class="c4">defer file</span><span class="c16">.</span><span class="c24">Close</span><span class="c16 c14">()</span></p><p class="c20"><span class="c45">This will create </span><span class="c33">export.csv</span><span class="c35 c14">&nbsp;and delay closing the file until the program completes its cycle.</span></p><p class="c20"><span class="c35 c14">Next, add these two lines to create a CSV writer:</span></p><p class="c0"><span class="c27">writer</span><span class="c4">&nbsp;</span><span class="c37 c11">:=</span><span class="c4">&nbsp;csv</span><span class="c9">.</span><span class="c5">NewWriter</span><span class="c9">(</span><span class="c4">file</span><span class="c9">)</span></p><p class="c0"><span class="c4">defer writer</span><span class="c9">.</span><span class="c5">Flush</span><span class="c9">()</span><span class="c4 c14">&nbsp;</span></p><p class="c0 c1"><span class="c4 c14"></span></p><p class="c20"><span class="c35 c14 c58">Now, it&rsquo;s time to write the headers:</span></p><p class="c0"><span class="c37 c58 c69">headers</span><span class="c4">&nbsp;</span><span class="c37 c58 c72">:=</span><span class="c4">&nbsp;</span><span class="c37 c55">[]</span><span class="c4">string</span><span class="c37 c55">{</span><span class="c18">&quot;Title&quot;</span><span class="c37 c55">,</span><span class="c4">&nbsp;</span><span class="c18">&quot;Price&quot;</span><span class="c37 c55">}</span></p><p class="c0"><span class="c4">writer</span><span class="c37 c55">.</span><span class="c37 c42">Write</span><span class="c37 c55">(</span><span class="c4">headers</span><span class="c37 c55">)</span></p><p class="c0 c1"><span class="c4 c14"></span></p><p class="c20"><span class="c45">Finally, modify the </span><span class="c33">OnHTML</span><span class="c14 c35">&nbsp;function to write each book as a single row:</span></p><p class="c0"><span class="c4">c</span><span class="c9">.</span><span class="c5">OnHTML</span><span class="c9">(</span><span class="c17">&quot;.product_pod&quot;</span><span class="c9">,</span><span class="c4">&nbsp;</span><span class="c5">func</span><span class="c9">(</span><span class="c37 c15">e </span><span class="c37 c11">*</span><span class="c15 c37">colly</span><span class="c9">.</span><span class="c37 c15">HTMLElement</span><span class="c9">)</span><span class="c4">&nbsp;</span><span class="c9">{</span></p><p class="c0"><span class="c4">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="c27">book</span><span class="c4">&nbsp;</span><span class="c37 c11">:=</span><span class="c4">&nbsp;Book</span><span class="c9">{}</span></p><p class="c0"><span class="c4">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;book</span><span class="c9">.</span><span class="c4">Title </span><span class="c37 c11">=</span><span class="c4">&nbsp;e</span><span class="c9">.</span><span class="c5">ChildAttr</span><span class="c9">(</span><span class="c17">&quot;.image_container img&quot;</span><span class="c9">,</span><span class="c4">&nbsp;</span><span class="c17">&quot;alt&quot;</span><span class="c9">)</span></p><p class="c0"><span class="c4">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;book</span><span class="c9">.</span><span class="c4">Price </span><span class="c37 c11">=</span><span class="c4">&nbsp;e</span><span class="c9">.</span><span class="c5">ChildText</span><span class="c9">(</span><span class="c17">&quot;.price_color&quot;</span><span class="c9">)</span></p><p class="c0"><span class="c4">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="c27">row</span><span class="c4">&nbsp;</span><span class="c37 c11">:=</span><span class="c4">&nbsp;</span><span class="c9">[]</span><span class="c4">string</span><span class="c9">{</span><span class="c4">book</span><span class="c9">.</span><span class="c4">Title</span><span class="c9">,</span><span class="c4">&nbsp;book</span><span class="c9">.</span><span class="c4">Price</span><span class="c9">}</span></p><p class="c0"><span class="c4">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;writer</span><span class="c9">.</span><span class="c5">Write</span><span class="c9">(</span><span class="c4">row</span><span class="c9">)</span></p><p class="c0"><span class="c9 c14">})</span></p><p class="c0 c1"><span class="c14 c37 c57"></span></p><p class="c20"><span class="c35 c14">That&rsquo;s all! The code for the Golang web scraper is now complete.</span></p><p class="c20"><span class="c35 c14">Run the file by entering the following in the terminal:</span></p><p class="c0"><span class="c4">go run books</span><span class="c9">.</span><span class="c4 c14">go</span></p><p class="c0 c1"><span class="c4 c14"></span></p><p class="c20"><span class="c45">This will create an </span><span class="c33">export.csv</span><span class="c35 c14">&nbsp;file with 1,000 rows of data.</span></p><h2 class="c67" id="h.3j8sc6cjpko7"><span class="c53 c14 c54">Scheduling tasks with GoCron</span></h2><p class="c20"><span class="c35 c14">For some tasks, you might want to schedule a web scraper to extract data periodically or at a specific time. You can do that by using your OS&#39;s schedulers or a high-level scheduling package usually available with the language you&#39;re using.</span></p><p class="c20"><span class="c45">To schedule a Go scraper, you can use OS tools like </span><span class="c52 c68"><a class="c10" href="https://www.google.com/url?q=https://oxylabs.io/blog/automated-web-scraper-cron&amp;sa=D&amp;source=editors&amp;ust=1667206257819893&amp;usg=AOvVaw1bjoDrjzTM-vtS99y8RUoo">Cron</a></span><span class="c45">&nbsp;or </span><span class="c52 c68"><a class="c10" href="https://www.google.com/url?q=https://oxylabs.io/blog/automated-web-scraper-windows-task-scheduler&amp;sa=D&amp;source=editors&amp;ust=1667206257820238&amp;usg=AOvVaw31lg0qNybF92zp65JOb7Zb">Windows Task Scheduler</a></span><span class="c45">. Alternatively, you can equip a high-level </span><span class="c52 c68"><a class="c10" href="https://www.google.com/url?q=https://github.com/go-co-op/gocron&amp;sa=D&amp;source=editors&amp;ust=1667206257820422&amp;usg=AOvVaw1uRDubwXpea6OST_y-ytIE">GoCron task scheduling package</a></span><span class="c35 c14">&nbsp;available with Golang. It&#39;s essential to keep in mind that scheduling a scraper through OS-provided schedulers limits the portability of the code. However, the GoCron task scheduler package solves this problem and works well with almost all operating systems. </span></p><p class="c20"><span class="c45">GoCron is a task scheduling package available in Golang for running specific codes at a particular time. It offers similar functionalities as Python&#39;s job scheduling module named </span><span class="c52 c68"><a class="c10" href="https://www.google.com/url?q=https://schedule.readthedocs.io/en/stable/&amp;sa=D&amp;source=editors&amp;ust=1667206257820722&amp;usg=AOvVaw3Yq71bfY4W_Bltqn5Pl0Wm">schedule</a></span><span class="c35 c14">.</span></p><p class="c20"><span class="c35 c14">Scheduling a task with GoCron requires a package to be installed with Golang, which you can do by using the following command: </span></p><p class="c43"><span class="c60 c54">go </span><span class="c59 c60">get</span><span class="c60 c54">&nbsp;github</span><span class="c60 c57">.</span><span class="c60 c54">com</span><span class="c40">/</span><span class="c60 c54">go</span><span class="c40">-</span><span class="c60 c54">co</span><span class="c40">-</span><span class="c60 c54">op</span><span class="c40">/</span><span class="c14 c39">gocron</span></p><p class="c0 c1"><span class="c50 c14 c58"></span></p><p class="c0"><span class="c25 c31">The next step is to write a GoCron script to schedule our code. Let&#39;s look at the following code example to understand how GoCron scheduler works:</span></p><p class="c0 c1"><span class="c4 c14"></span></p><p class="c0"><span class="c12">package</span><span class="c4 c14">&nbsp;main</span></p><p class="c0 c1"><span class="c4 c14"></span></p><p class="c0"><span class="c12">import</span><span class="c4">&nbsp;</span><span class="c9">(</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; </span><span class="c17">&quot;fmt&quot;</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; </span><span class="c17">&quot;time&quot;</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; </span><span class="c17">&quot;github.com/go-co-op/gocron&quot;</span></p><p class="c0"><span class="c9">)</span></p><p class="c0 c1"><span class="c4 c14"></span></p><p class="c0"><span class="c4">func </span><span class="c5">My_Task_1</span><span class="c9">()</span><span class="c4">&nbsp;</span><span class="c9">{</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; fmt</span><span class="c9">.</span><span class="c5">Println</span><span class="c9">(</span><span class="c17">&quot;Hello Task 1&quot;</span><span class="c9">)</span></p><p class="c0"><span class="c9">}</span></p><p class="c0 c1"><span class="c4 c14"></span></p><p class="c0"><span class="c4">func </span><span class="c5">main</span><span class="c9">()</span><span class="c4">&nbsp;</span><span class="c9">{</span></p><p class="c0 c1"><span class="c4 c14"></span></p><p class="c0"><span class="c4">&nbsp; &nbsp; my_scheduler </span><span class="c37 c11">:=</span><span class="c4">&nbsp;gocron</span><span class="c9">.</span><span class="c5">NewScheduler</span><span class="c9">(</span><span class="c4">time</span><span class="c9">.</span><span class="c27">UTC</span><span class="c9">)</span></p><p class="c0 c1"><span class="c4 c14"></span></p><p class="c0"><span class="c4">&nbsp; &nbsp; my_scheduler</span><span class="c9">.</span><span class="c5">Every</span><span class="c9">(</span><span class="c27">5</span><span class="c9">).</span><span class="c5">Seconds</span><span class="c9">().</span><span class="c5">Do</span><span class="c9">(</span><span class="c4">My_Task_1</span><span class="c9">)</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; my_scheduler</span><span class="c9">.</span><span class="c5">StartAsync</span><span class="c9">()</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; my_scheduler</span><span class="c9">.</span><span class="c5">StartBlocking</span><span class="c9">()</span></p><p class="c0"><span class="c9 c14">}</span></p><p class="c0 c1"><span class="c9 c14"></span></p><p class="c20"><span class="c41">The code above schedules the </span><span class="c33 c49">My_task_1</span><span class="c41 c44">&nbsp;</span><span class="c35 c14 c49">function to run every 5 seconds. Moreover, we can start the GoCron scheduler in two modes: asynchronous mode and blocking mode. </span></p><p class="c20"><span class="c33 c49">StartAsync()</span><span class="c41">&nbsp;will start the scheduler asynchronously, while the </span><span class="c33 c49">StartBlocking()</span><span class="c35 c14 c49">&nbsp;method will start the scheduler in blocking mode by blocking the current execution path.</span></p><p class="c20"><span class="c35 c14 c49">Side note: The above code example starts the GoCron scheduler in both the asynchronous and the blocking modes. However, we can choose either of these as per our requirements.</span></p><p class="c20"><span class="c35 c14 c49">Let&rsquo;s schedule our Golang web scraper code example using the GoCron scheduling module.</span></p><p class="c0"><span class="c12">package</span><span class="c4 c14">&nbsp;main</span></p><p class="c0 c1"><span class="c4 c14"></span></p><p class="c0"><span class="c12">import</span><span class="c4">&nbsp;</span><span class="c9">(</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; </span><span class="c17">&quot;encoding/csv&quot;</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; </span><span class="c17">&quot;fmt&quot;</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; </span><span class="c17">&quot;log&quot;</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; </span><span class="c17">&quot;os&quot;</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; </span><span class="c17">&quot;time&quot;</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; </span><span class="c17">&quot;github.com/go-co-op/gocron&quot;</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; </span><span class="c17">&quot;github.com/gocolly/colly&quot;</span></p><p class="c0"><span class="c9">)</span></p><p class="c0"><span class="c4 c14">&nbsp;</span></p><p class="c0"><span class="c4">type Book struct </span><span class="c9">{</span></p><p class="c0"><span class="c4 c14">&nbsp; &nbsp; Title string</span></p><p class="c0"><span class="c4 c14">&nbsp; &nbsp; Price string</span></p><p class="c0"><span class="c9">}</span></p><p class="c0"><span class="c4 c14">&nbsp;</span></p><p class="c0"><span class="c4">func </span><span class="c5">BooksScraper</span><span class="c9">()</span><span class="c4">&nbsp;</span><span class="c9">{</span></p><p class="c0"><span class="c4 c14">&nbsp;</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; fmt</span><span class="c9">.</span><span class="c5">Println</span><span class="c9">(</span><span class="c17">&quot;Start scraping&quot;</span><span class="c9">)</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; file</span><span class="c9">,</span><span class="c4">&nbsp;</span><span class="c27">err</span><span class="c4">&nbsp;</span><span class="c37 c11">:=</span><span class="c4">&nbsp;os</span><span class="c9">.</span><span class="c5">Create</span><span class="c9">(</span><span class="c17">&quot;export.csv&quot;</span><span class="c9">)</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; </span><span class="c12">if</span><span class="c4">&nbsp;err </span><span class="c37 c11">!=</span><span class="c4">&nbsp;nil </span><span class="c9">{</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; &nbsp; &nbsp; log</span><span class="c9">.</span><span class="c5">Fatal</span><span class="c9">(</span><span class="c4">err</span><span class="c9">)</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; </span><span class="c9">}</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; defer file</span><span class="c9">.</span><span class="c5">Close</span><span class="c9">()</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; </span><span class="c27">writer</span><span class="c4">&nbsp;</span><span class="c37 c11">:=</span><span class="c4">&nbsp;csv</span><span class="c9">.</span><span class="c5">NewWriter</span><span class="c9">(</span><span class="c4">file</span><span class="c9">)</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; defer writer</span><span class="c9">.</span><span class="c5">Flush</span><span class="c9">()</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; </span><span class="c27">headers</span><span class="c4">&nbsp;</span><span class="c37 c11">:=</span><span class="c4">&nbsp;</span><span class="c9">[]</span><span class="c4">string</span><span class="c9">{</span><span class="c17">&quot;Title&quot;</span><span class="c9">,</span><span class="c4">&nbsp;</span><span class="c17">&quot;Price&quot;</span><span class="c9">}</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; writer</span><span class="c9">.</span><span class="c5">Write</span><span class="c9">(</span><span class="c4">headers</span><span class="c9">)</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; </span><span class="c27">c</span><span class="c4">&nbsp;</span><span class="c37 c11">:=</span><span class="c4">&nbsp;colly</span><span class="c9">.</span><span class="c5">NewCollector</span><span class="c9">(</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; &nbsp; &nbsp; colly</span><span class="c9">.</span><span class="c5">AllowedDomains</span><span class="c9">(</span><span class="c17">&quot;books.toscrape.com&quot;</span><span class="c9">),</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; </span><span class="c9">)</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; c</span><span class="c9">.</span><span class="c5">OnHTML</span><span class="c9">(</span><span class="c17">&quot;.product_pod&quot;</span><span class="c9">,</span><span class="c4">&nbsp;</span><span class="c5">func</span><span class="c9">(</span><span class="c37 c15">e </span><span class="c37 c11">*</span><span class="c37 c15">colly</span><span class="c9">.</span><span class="c37 c15">HTMLElement</span><span class="c9">)</span><span class="c4">&nbsp;</span><span class="c9">{</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c27">book</span><span class="c4">&nbsp;</span><span class="c37 c11">:=</span><span class="c4">&nbsp;Book</span><span class="c9">{}</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; &nbsp; &nbsp; book</span><span class="c9">.</span><span class="c4">Title </span><span class="c37 c11">=</span><span class="c4">&nbsp;e</span><span class="c9">.</span><span class="c5">ChildAttr</span><span class="c9">(</span><span class="c17">&quot;.image_container img&quot;</span><span class="c9">,</span><span class="c4">&nbsp;</span><span class="c17">&quot;alt&quot;</span><span class="c9">)</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; &nbsp; &nbsp; book</span><span class="c9">.</span><span class="c4">Price </span><span class="c37 c11">=</span><span class="c4">&nbsp;e</span><span class="c9">.</span><span class="c5">ChildText</span><span class="c9">(</span><span class="c17">&quot;.price_color&quot;</span><span class="c9">)</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; &nbsp; &nbsp; </span><span class="c27">row</span><span class="c4">&nbsp;</span><span class="c37 c11">:=</span><span class="c4">&nbsp;</span><span class="c9">[]</span><span class="c4">string</span><span class="c9">{</span><span class="c4">book</span><span class="c9">.</span><span class="c4">Title</span><span class="c9">,</span><span class="c4">&nbsp;book</span><span class="c9">.</span><span class="c4">Price</span><span class="c9">}</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; &nbsp; &nbsp; writer</span><span class="c9">.</span><span class="c5">Write</span><span class="c9">(</span><span class="c4">row</span><span class="c9">)</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; </span><span class="c9">})</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; c</span><span class="c9">.</span><span class="c5">OnResponse</span><span class="c9">(</span><span class="c5">func</span><span class="c9">(</span><span class="c37 c15">r </span><span class="c37 c11">*</span><span class="c37 c15">colly</span><span class="c9">.</span><span class="c37 c15">Response</span><span class="c9">)</span><span class="c4">&nbsp;</span><span class="c9">{</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; &nbsp; &nbsp; fmt</span><span class="c9">.</span><span class="c5">Println</span><span class="c9">(</span><span class="c4">r</span><span class="c9">.</span><span class="c4">StatusCode</span><span class="c9">)</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; </span><span class="c9">})</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; c</span><span class="c9">.</span><span class="c5">OnRequest</span><span class="c9">(</span><span class="c5">func</span><span class="c9">(</span><span class="c37 c15">r </span><span class="c37 c11">*</span><span class="c37 c15">colly</span><span class="c9">.</span><span class="c37 c15">Request</span><span class="c9">)</span><span class="c4">&nbsp;</span><span class="c9">{</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; &nbsp; &nbsp; fmt</span><span class="c9">.</span><span class="c5">Println</span><span class="c9">(</span><span class="c17">&quot;Visiting&quot;</span><span class="c9">,</span><span class="c4">&nbsp;r</span><span class="c9">.</span><span class="c27">URL</span><span class="c9">)</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; </span><span class="c9">})</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; c</span><span class="c9">.</span><span class="c5">Visit</span><span class="c9">(</span><span class="c17">&quot;https://books.toscrape.com/&quot;</span><span class="c9">)</span></p><p class="c0"><span class="c9">}</span></p><p class="c0"><span class="c4 c14">&nbsp;</span></p><p class="c0"><span class="c4">func </span><span class="c5">main</span><span class="c9">()</span><span class="c4">&nbsp;</span><span class="c9">{</span></p><p class="c0"><span class="c4 c14">&nbsp;</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; </span><span class="c27">my_scheduler</span><span class="c4">&nbsp;</span><span class="c37 c11">:=</span><span class="c4">&nbsp;gocron</span><span class="c9">.</span><span class="c5">NewScheduler</span><span class="c9">(</span><span class="c4">time</span><span class="c9">.</span><span class="c27">UTC</span><span class="c9">)</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; my_scheduler</span><span class="c9">.</span><span class="c5">Every</span><span class="c9">(</span><span class="c27">2</span><span class="c9">).</span><span class="c5">Minute</span><span class="c9">().</span><span class="c5">Do</span><span class="c9">(</span><span class="c4">BooksScraper</span><span class="c9">)</span></p><p class="c0"><span class="c4">&nbsp; &nbsp; my_scheduler</span><span class="c9">.</span><span class="c5">StartBlocking</span><span class="c9">()</span></p><p class="c0"><span class="c9 c14">}</span></p><p class="c0 c1"><span class="c9 c14"></span></p><h2 class="c67" id="h.nddwm7cprpx"><span class="c53 c14 c54">Summary </span></h2><p class="c20"><span class="c45">The code used in this article ran in less than 12 seconds. Executing the same task in Scrapy, which is one of the most optimized modern frameworks for Python, took 22 seconds. If speed is what you prioritize for your web scraping tasks, it&rsquo;s a good idea to consider Golang in tandem with a modern framework such as Colly. You can </span><span class="c52 c68"><a class="c10" href="https://www.google.com/url?q=https://github.com/oxylabs/golang-web-scraper&amp;sa=D&amp;source=editors&amp;ust=1667206257834240&amp;usg=AOvVaw0GIYox-OFqdZ3NOuNqVPZn">click here</a></span><span class="c35 c14">&nbsp;to find the complete code used in this article for your convenience. </span></p><p class="c0 c1"><span class="c9 c14"></span></p></body></html>