#GopherDiggingRuby: Make a dev.to image link fetcher in Ruby
Tags: ruby, codenewbie, webdev, tutorial
Published: 2022-01-18 05:59:16

---
title: #GopherDiggingRuby: Make a dev.to image link fetcher in Ruby
published: true
description: 
tags: #ruby, #codenewbie, #webdev, #tutorial
---

I write a lot of blog posts on dev.to and really like using Markdown for blogging. But one thing that would make the blogging process even more convenient, is if I had all the image links I've made on previous dev.to posts in one place for reusing images.

So in this tutorial, I will show how we can use Ruby packages to make a CSV file of every image link in your blog posts from the command line, using the [dev.to/Forem API](https://docs.forem.com/api/).

We will:

* 💻 Send an authenticated request to the dev.to API with the standard library `net/http` package and the Forem API
* 🐈 Deserialize JSON into objects of a custom class with the standard library `json` package
* 💎 Retrieve the image links using the `commonmarker` gem
* 💼 Serialize objects into CSV files with the standard library `csv` package

This tutorial assumes you're familiar with JSON, HTTP requests, and the basics of Ruby.

## Contacting the dev.to API

The first thing you need in order to talk to a web API is an API **client**. An API client is some code that sends authenticated HTTP requests to the web API. We can get a client in one of two ways:

1. Search for a Ruby Gem of a client for the dev.to/Forem APIs; a lot of people have made code repositories for talking to websites' APIs so that other people can use them.
2. Make our own API client from scratch

Since we're only talking to one HTTP endpoint on dev.to's API in this tutorial, we'll go with the second option. So let's jump to dev.to's [API documentation](https://developers.forem.com/)!

The endpoint we're talking to is the [User's Articles endpoint](https://developers.forem.com/api#operation/getUserArticles), which lists all articles belonging to the user calling that endpoint as JSON. Looking at the request and response samples on the right, we will find that:

* The request is a GET request to the endpoint `/articles/me`
* The response is an array of JSON objects
  * Each JSON object in the array contains many fields for all the data representing the article, namely `title` for the title of the article, and `body_markdown` for the Markdown of the entire text of the article
* The sample cURL request contacting that endpoint contains the header `api-key`, which tells dev.to who you are and gives you access to your own account's data.

So we need a client class that can:

1. Send an HTTP request to the dev.to API's `/articles/me` endpoint, with an API key as authentication
2. Deserialize the JSON response to Ruby objects, giving us access to the fields we are interested in.

If you're following along, make a folder for your Ruby app and write this code to `app.rb` in that folder:

```ruby
require 'net/http'
require 'json'

class DevToClient
  def initialize(api_key)
    @api_key = api_key
  end

  def get_my_articles
    # [TODO] Send HTTP request to /articles/me and deserialize
    # the JSON response
  end
end
```

We have a `DevToClient` class with two methods:

* `initialize` as its constructor. In the constructor, we pass in our API key, and it gets stored in the instance variable `@api_key`
* `get_my_articles`, which will send an HTTP request to dev.to's `/articles/me` endpoint that uses the `@api_key` and deserialize the response

Now for getting the HTTP response. Looking at the documentation for `net/http`, there is the method [HTTP.get_response](https://ruby-doc.org/stdlib-3.0.1/libdoc/net/http/rdoc/Net/HTTP.html#method-c-get) for sending a GET request to the URL passed in and getting back an HTTP response.

From the `get_response` method's documentation, in addition to a URL, we also are able to optionally pass in a hash of any request headers we want to pass in. So we can pass in an `api-key` header with this code:

```ruby
  def get_my_articles
    Net::HTTP.get_response(
      URI('https://dev.to/api/articles/me'),
      { 'api-key': @api_key }
    )
  end
```

We send the request to dev.to's `/api/articles/me` endpoint with an `api-key` header containing the value of the `DevToClient`'s `@api_key` instance variable.

To run this, you are going to need to get your own API key for the dev.to API. To do that, first, make a dev.to account if you don't have one already. Then, you can get your API key by following [the Authentication instructions on the dev.to API docs](https://developers.forem.com/api#section/Authentication).

> ⚠️WARNING!⚠️ For any web API you are working with, DO NOT share your API key or other forms of authentication with anyone; don't post it online or email it to your friends, and also don't commit it in your code! If someone else gets ahold of your API key, they will be able to impersonate you on that API and access your account data! If you suspect that someone has obtained one of your authentication keys or secrets, you should have that key/secret invalidated and then a fresh API key/secret created to protect your account.

Now, your save your API key to the environment variable `DEVTO_API_KEY`. Then, at the bottom of `app.rb`, add this code:

```ruby
api_key = ENV['DEVTO_API_KEY']
puts DevToClient.new(api_key).get_my_articles
```

Run the code with `ruby app.rb` and you should see terminal output like this:

```
#<Net::HTTPOK:0x00007fc20a92f800>
```

We got back a response of class [HTTPOK](https://ruby-doc.org/stdlib-3.0.1/libdoc/net/http/rdoc/Net/HTTPOK.html) (which inherits from [HTTPSuccess](https://ruby-doc.org/stdlib-3.0.1/libdoc/net/http/rdoc/Net/HTTPSuccess.html) and in turn inherits from [HTTPResponse](https://ruby-doc.org/stdlib-3.0.1/libdoc/net/http/rdoc/Net/HTTPResponse.html)). So now we have our response, so let's parse it to make Ruby objects for each article.

## JSON parsing in Ruby

In addition to `net/http`, the Ruby standard library has a [json](https://ruby-doc.org/stdlib-3.0.1/libdoc/json/rdoc/JSON.html) package for serializing deserializing JSON, and inside that package there is the method [JSON.parse](https://ruby-doc.org/stdlib-3.0.1/libdoc/json/rdoc/JSON.html#method-i-parse). So if we did Ruby code like

```ruby
sloth_json = <<EOF
{
    "sci_name":    "Bradypus",
    "common_name": "Three-toed sloth",
    "claw_count":  3
}
EOF

sloth = JSON.parse(sloth_json)
sci_name = sloth["sci_name"]
common_name = sloth["common_name"]
claw_count = sloth["claw_count"]
puts "The #{sci_name} (#{common_name}) has #{claw_count} claws"
```

and then ran `ruby app.rb`, we would get output like this:

```
The Bradypus (Three-toed sloth) has 3 claws
```

The object returned from `JSON.parse` is a Ruby hash, with its field names becoming the hash's keys.

Let's try `JSON.parse` to return a Ruby object from `DevToClient#get_my_articles`:

```ruby
  def get_my_articles
    res = Net::HTTP.get_response(
      URI('https://dev.to/api/articles/me'),
      { 'api-key': @api_key }
    )

    if res.code.to_i > 299 || res.code.to_i < 200
      raise "got status code #{res.code}"
    end
    JSON.parse res.body
  end
```

Now, if we got a status code besides 2xx, we raise an error. Otherwise, we return the result of parsing the response body.

If you then ran code like

```ruby
puts DevToClient.new(api_key).get_my_articles
```

you will see that the Ruby object that the response body deserialized to was an array of Ruby hashes. So we could do something like this:

```ruby
DevToClient.new(api_key).get_my_articles.each do |article|
  md = article["body_markdown"]

  # now use a Markdown file to find every image link in
  # the article
end
```

But what if we wanted a `DevToArticle` class that handles digging for all the image links, and we wanted to deserialize our JSON to an array of `DevToArticles` instead of hashes?

Let's start by making a `DevToArticle` class:

```ruby
class DevToArticle
  attr_accessor :id, :title, :body_markdown, :url

  def initialize
    # [TODO] Add deserialization logic here
  end

  def get_article_images
    # [TODO] Add Markdown parsing for the article's
    # @body_markdown
  end
end
```

Since Ruby doesn't directly know if that the JSON it's getting is supposed to be a `DevToArticle`, calling `JSON.parse` will return an array of Ruby hashes. So we will need just a bit of extra logic for converting those hashes to `DevToArticle`s.

I wasn't sure how to do this at first; in Go, the main programming language I work with, I would be doing this using code like this:

```go
type DevToArticle struct {
    ID           int    `json:"id"`
    Title        string `json:"title"`
    BodyMarkdown string `json:"body_markdown"`
    URL          string `json:"url"`
}

func (d *DevToClient) GetMyArticles() ([]DevToArticle, error) {
    // get the HTTP response for the "user's articles" API
    // endpoint here

    var articles []DevToArticle
    if err := json.NewDecoder(res).Decode(&articles); err != nil {
        return nil, err
    }
    return articles, nil
}
```

I searched for how to deserialize to a custom class rather than a hash, and after asking about that on Twitter, Jamie Gaskins told me that there isn't really a standardized way in Ruby to deserialize to a class, but you are able to [give your Ruby class an initialize method that takes in a hash](https://twitter.com/jamie_gaskins/status/1483130085716725761). So based on that advice, in our `DevToArticle#initialize` class, the deserialization logic would look like this:

```ruby
  def initialize(attributes)
    @id = attributes['id']
    @title = attributes['title']
    @body_markdown = attributes['body_markdown']
    @url = attributes['url']
  end
```

For each field we want an instance variable for, we just pull it out of the `attributes` hash passed in.

Note, by the way, that this also gives us control of the casing scheme for the deserialized objects. In Ruby, the standardized casing for instance variables is snake_case, and that's the casing the Forem API uses, but what if Forem was a camelCase API instead? `@body_markdown` still is able to be snake_case even if `bodyMarkdown` in the hash is camelCase:

```ruby
    @body_markdown = attributes['bodyMarkdown']
```

Now, to have `DevToClient#get_my_articles` return an array of `DevToArticle`s instead of an array of hashes, we can do this:

```ruby
    articles = JSON.parse(res.body)
    articles.map { |article| DevToArticle.new article }
```

By passing that block into `articles.map`, we get back an array of `DevToArticle`s created from each hash in the `articles` array, so now `get_my_articles` returns the type we want: an array of `DevToArticle`s. Now let's jump into the Markdown of those articles all their image links!

## Markdown parsing with commonmarker

Unlike HTTP and JSON, the standard library in Ruby doesn't have a Markdown package, so we can either write our own Markdown parser, or use a Markdown-parsing Ruby Gem.

And it turns out that there's a popular Ruby Gem that lets us parse a Markdown file and then walk over its nodes (nodes as in text, links, images, etc): [CommonMarker](https://github.com/gjtorikian/commonmarker)! To get it, first run `bundle init` to create a Gemfile, then in the Gemfile, add the line

```ruby
gem "commonmarker"
```

Then run `bundle install`. If CommonMarker it successfully installs, you should be able to use it in your Ruby code.

To start, add `require 'commonmarker'` to the top of `app.rb`, then in `DevToArticle#get_article_images`, add this code:

```ruby
  def get_article_images
    doc = CommonMarker.render_doc(@body_markdown, :DEFAULT)
    puts doc
  end
```

If you run that function in `app.rb`, you will get output for an article like:

```
#<CommonMarker::Node:0x00007fca8718f228>
```

indicating that we were able to parse the Markdown in `@body_markdown`, converting it to a `CommonMarker::Node` object.

Following [this example in the CommonMarker documentation](https://www.gjtorikian.com/commonmarker/#label-Generating+a+document), we can walk over all the nodes in the document with code like this:

```ruby
  def get_article_images
    doc = CommonMarker.render_doc(@body_markdown, :DEFAULT)
    doc.walk do |node|
      puts node.type
    end
  end
```

Now in the `do` block, we are looking at each `Node` and seeing what type of node it is. So if you run this code, you might see output like this:

```
text
code
text
paragraph
image
text
text
paragraph
```

We're only interested in `image` nodes, so we'll add an if statement to check the node's type, which according to the documentation, is the Ruby symbol `:image` according to the [new(p1) docs](https://www.gjtorikian.com/commonmarker/CommonMarker/Node.html#method-c-new).

```ruby
    doc.walk do |node|
      if node.type == :image
        # [TODO] retrieve the node's content
      end
    end
```

Now we're only processing image links. And a Markdown image link has two parts: descriptive alt text, which screen reader software reads when viewing images, and the URL of the image. So we need to find ways to get both of those.

Looking at the CommonMarker documentation, the `Node` method for getting the alt text is [to_plaintext](https://www.gjtorikian.com/commonmarker/CommonMarker/Node.html#method-i-to_plaintext), and the `Node` method for getting the URL of the image is [url](https://www.gjtorikian.com/commonmarker/CommonMarker/Node.html#method-i-url). So now, we can return the parts to the image link:

```ruby
  def get_article_images
    doc = CommonMarker.render_doc(@body_markdown, :DEFAULT)
    image_links = []
    doc.walk do |node|
      if node.type == :image
        image_links.push [
          node.to_plaintext.delete_suffix("\n"), node.url
        ]
      end
    end

    image_links
  end
```

So now, we have all the data we'll need for serializing to a CSV file!

## Serializing your image links to a CSV file

The Ruby standard library also comes with a `csv` package for parsing CSV files, or generating them from arrays of data. Each row will be one image link, including:

* The alt text of the image link
* The URL of the image
* The ID of the article that the image link came from
* The title of the article that the image link came from
* The URL of the article that the image link came from

So we will want CSV header text like:

```csv
Alt Text,Image URL,Article ID,Article Title,Article URL
```

And for each row in the CSV, we will want an `DevToImageLink` Ruby class to represent all the data in that row

```ruby
class DevToImageLink
  def initialize(article, image_alt, image_url)
    @article = article
    @image_alt = image_alt
    @image_url = image_url
  end

  def to_csv_row
    [@image_alt, @image_url, @article.id, @article.title, @article.url]
  end
end
```

In the `initialize` method we pass in the `DevToArticle` for the image link, and the alt text and URL of the image, to become instance variables. And in the `to_csv_row` method, all of these fields are put into a Ruby array.

Heading back to the `DevToArticle` class, now that we have the `DevToImageLink` class defined, let's have `DevToArticle#get_article_images` return an array of `DevToImageLink`s, rather than an array of arrays:

```diff
      if node.type == :image
-       image_links.push [node.to_plaintext, node.url]
+       image_alt = node.to_plaintext
+       image_url = node.url
+       image_links.push DevToImageLink.new(self, image_alt, image_url)
      end
```

Now that that's all set, let's add a top-level `get_image_links_csv` function that will convert our article to a CSV.

```ruby
def get_image_links_csv(api_key)
  CSV.generate do |csv|
    csv << [
      'Alt Text','Image URL','Article ID','Article Title','Article URL'
    ]

    DevToClient.new(api_key).get_my_articles.each do |article|
      article.get_article_images.each do |image_link|
        csv << image_link.to_csv_row
      end
    end
  end
end
```

The function `CSV.generate` takes in a block and returns the CSV string generated in that block.

In the first line inside the block, we pass in our CSV headers with the CSV's [<< method](https://ruby-doc.org/stdlib-3.0.1/libdoc/csv/rdoc/CSV.html#method-i-3C-3C), so they will serve as the first line of the CSV.

Now, we loop over the image links in each of the articles returned by `DevToClient#get_my_articles`. For each image link, we call the `DevToImageLink#to_csv_row` method, and then load the returned array into the CSV.

Finally, the return value of `CSV#generate` is a string in CSV format. So now, we can use that code like this:

```ruby
puts get_image_links_csv(api_key)
```

Using three standard library packages and a gem, we were able to make a convenient script for getting all our dev.to image links and converting them to a CSV. In my next Ruby tutorial, I will be looking at using a gem for giving this script a better user experience so it's easier to search the CSV for the image you want.