Crawling websites with elixir and crawly
Tags: elixir
Published: 2020-08-13 13:21:51

---
title: Crawling websites with elixir and crawly
published: true
date: 2020-08-13 12:00:00 UTC
tags: Elixir
canonical_url: https://www.codegram.com/blog/crawling-websites-with-elixir-and-crawly/
---

In this post, I'd like to introduce [crawly](https://github.com/oltarasenko/crawly) which is an elixir library for [crawling](https://en.wikipedia.org/wiki/Web_crawler) websites.

My idea for this post is to be a quick introduction to how we can use crawly. For this little example we're going to extract the latest posts titles on our website and write it to a file, so let's do it.

First of all, we need elixir installed, in case you don't have it installed you can check this [guide](https://elixir-lang.org/install.html).

Once we have elixir installed let's create our elixir application with built-in supervisor

```
mix new crawler --sup
```

In order to add the crawly dependencies to our project, we are going to change the deps function in the file `mix.exs` and it should look like this

```
defp deps do
    [
        {:crawly, "~> 0.10.0"},
        {:floki, "~> 0.26.0"} # used to parse html
    ]
end
```

We need to install the dependencies that we just added running the command below

```
mix deps.get
```

Let's create a spider file `lib/crawler/blog_spider.ex` that is going to make a request to our blog, query the HTML response to get the post titles, and then returns a `ParsedItem` which contains items and requests. We are not going to leave requests as an empty list to keep it simple.

```
defmodule BlogSpider do
  use Crawly.Spider

  @impl Crawly.Spider
  def base_url(), do: "https://www.codegram.com"

  @impl Crawly.Spider
  def init(), do: [start_urls: ["https://www.codegram.com/blog"]] # urls that are going to be parsed

  @impl Crawly.Spider
  def parse_item(response) do
    {:ok, document} = Floki.parse_document(response.body)

    items =
      document
      |> Floki.find("h5.card-content__title") # query h5 elements with class card-content__title
      |> Enum.map(&Floki.text/1)
      |> Enum.map(fn title -> %{title: title} end)

    %Crawly.ParsedItem{items: items, requests: []}
  end
end
```

Now that we have our spider created it would be nice to save what we're extracting into some file. To do this we can use a pipeline provided by crawly called `Crawly.Pipelines.WriteToFile`. For that, we need a `config` folder and a `config.exs` file:

```
mkdir config # creates the config directory
touch config/config.exs # creates an empty file called config.exs inside the config folder
mkdir -p priv/output # creates output folder inside priv where we are going to store our files
```

Now let's create the configuration to save the response from our spider into a file.

```
use Mix.Config

config :crawly,
  pipelines: [
    Crawly.Pipelines.JSONEncoder, # encode each item into json
    {Crawly.Pipelines.WriteToFile, folder: "priv/output/", extension: "jl"} # stores the items into a file inside the folder specified
  ]
```

Now that we are good to go, we can open the elixir repl

```
iex -S mix
```

And then we can execute our spider

```
Crawly.Engine.start_spider(BlogSpider)
```

The spider is going to be executed by a supervisor and then we should see a new file inside priv/output folder. In my case the lasts posts showing in the first page are

```
{"title":"\"High tech, high touch\": A communication toolkit for virtual team"}
{"title":"My learning experience in a fully remote company as a Junior Developer"}
{"title":"Finding similar documents with transformers"}
{"title":"UX… What?"}
{"title":"Slice Machine from Prismic"}
{"title":"Stop (ab)using z-index"}
{"title":"Angular for Junior Backend Devs"}
{"title":"Jumping into the world of UX 🦄"}
{"title":"Gettin' jiggy wit' Git - Part 1"}
```

This is just a simple example of what is possible to do using `crawly`. I hope you enjoyed this introduction and remember to be responsible when extracting data from websites.