Interacting with the Dev.to Article API - A Simple Retry System
Tags: go, api, atlg
Published: 2019-03-23 19:12:14

---
title: Interacting with the Dev.to Article API - A Simple Retry System
published: true
description: "Let's hack in a simple system to retry grabbing articles."
tags: go, golang, api, atlg,
cover_image: https://dev.shindakun.net/img/attemptinggo_w.png
series: "ATLG Sidebar"
---

# ATLG Sidebar

## But What About Errors

As always it seems like I'm running short of time! This week I wanted to put a simple retry system in place. This will allow us to make a second attempt to download and removes a few of the `panic()`'s we had.

***

## Updates

I'm not going to do a full code walkthrough this time around since we aren't making too many changes. Instead, we'll look at a before and after on the bits that saw some work.

The first thing we want to do is set up a struct to hold the article IDs for the ones we want to retry.

```go
type Retry struct {
  IDs []int32
}
```

Perfect! We could extend it our if we say wanted to retry 3 times or something. In fact, we don't need it at all in my experience. I have been able to pull every public article from the API without any panics at all. But, I'm sure it may not run that smooth every time.

So you don't have to swap back to the [first part](https://dev.to/shindakun/interacting-with-the-devto-article-api-4g34), here is our original `main()`.

```go
func main() {
  dtc := New("https://dev.to/api/", nil)
  doit := true
  c := 1

  for doit {
    req, err := dtc.FormatPagedRequest("page", fmt.Sprintf("%d", c))
    if err != nil {
      panic(err)
    }
    resp, err := dtc.Client.Do(req)
    if err != nil {
      panic(err)
    }
    defer resp.Body.Close()

    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
      panic(err)
    }

    var wg sync.WaitGroup
    var articles Articles

    json.Unmarshal(body, &articles)
    wg.Add(len(articles))

    for i := range articles {
      go getArticle(dtc, articles[i].ID, &wg)
    }
    wg.Wait()

    if string(body) != "[]" {
      c++
      continue
    }
    doit = false
  }
}
```

And now our updated version, with changes noted.

```go
func main() {
  dtc := New("https://dev.to/api/", nil)
  doit := true
  c := 1
```

We're adding in `retries` and `report`. The first will hold the article IDs that we want to attempt to get a second time. The second will hold any IDs that failed the second time around which we'll output to the console. We didn't remove any of the panics in `main()` this time. I think we could extend the retry system to cover it at some point.

```go
  retries := Retry{}
  report := Retry{}

  for doit {
    req, err := dtc.FormatPagedRequest("page", fmt.Sprintf("%d", c))
    if err != nil {
      panic(err)
    }
    resp, err := dtc.Client.Do(req)
    if err != nil {
      panic(err)
    }
```

As was pointed out on [Twitter by VirgileMathieu](https://twitter.com/VirgileMathieu/status/1109609005125652480) we're currently using `defer` inside our `for` loop. This may not be the best idea and could lead to unintended consequences. We have a couple of options to deal with this. First, we could remove the `defer` and just `.Close()` or we could wrap the entire section inside of an anonymous function.


```go
    defer resp.Body.Close()

    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
      panic(err)
    }

    var wg sync.WaitGroup
    var articles Articles
    err = json.Unmarshal(body, &articles)
    if err != nil {
      panic(err)
    }

    wg.Add(len(articles))
```

We're going to pass in a pointer to `retries` so we can keep it up to date as we try to `getArticle()`.

```go
    for i := range articles {
      getArticle(dtc, articles[i].ID, &retries, &wg)
    }
    wg.Wait()

    if string(body) != "[]" {
      c++
      continue
    }
    doit = false
  }
```

Once our main loop has ended we're going to set up a second `WaitGroup`. We'll then attempt to grab any articles we may have missed. It might be worth setting up a loop here to tackle them in batches of 10 or so at a time. I'll do that and update the post. We should also wrap this section in an `if` no point going into it if `retries` is empty.

```go
  // Lets try to get the ones we couldn't before
  var wg sync.WaitGroup
  wg.Add(len(retries.IDs))

  for i := range retries.IDs {
    getArticle(dtc, retries.IDs[i], &report, &wg)
  }
  wg.Wait()

  fmt.Printf("Unable to grab the following articles: %v\n", report)
}
```

***

## Get Those Articles

Now let's look at our original `getArticles()`. It wasn't too bad - it got the job done! We want to get rid of the panics and allow the program to continue on even if we hit an error.

```go
func getArticle(dtc *DevtoClient, i int32, wg *sync.WaitGroup) {
  defer wg.Done()
  r, err := dtc.FormatArticleRequest(i)
  if err != nil {
    panic(err)
  }

  resp, err := dtc.Client.Do(r)
  if err != nil {
    panic(err)
  }
  defer resp.Body.Close()

  body, err := ioutil.ReadAll(resp.Body)
  if err != nil {
    panic(err)
  }
  fileName := fmt.Sprintf("%d.json", i)
  ioutil.WriteFile("./out/"+fileName, body, 0666)
}
```

We'll need to update the signature to since we're now passing in a pointer `*Retry`. Second, we'll update each of our `err` checks to update the retries struct and return to `main()`.

```go
func getArticle(dtc *DevtoClient, i int32, retries *Retry, wg *sync.WaitGroup) {
  defer wg.Done()
  r, err := dtc.FormatArticleRequest(i)
  if err != nil {
    retries.IDs = append(retries.IDs, i)
    return
  }
```

Note that we are adding a secondary check to see if we hit a `statusCode` over 399. This will cause us to add that article ID for any article that returns a [client or server error](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status).

```go
  resp, err := dtc.Client.Do(r)
  if err != nil || resp.StatusCode > 399 {
    retries.IDs = append(retries.IDs, i)
    return
  }

  defer resp.Body.Close()

  _, err = ioutil.ReadAll(resp.Body)
  if err != nil {
    retries.IDs = append(retries.IDs, i)
    return
  }
}
```

***

## Next time

That's all for this time around. We've gone pretty far with this code example so I'll be looking for something else to work with. Have any ideas? Let me know in the comments!

***

You can find the code for this and most of the other Attempting to Learn Go posts in the repo on GitHub.

{% github shindakun/atlg %}

***

| Enjoy this post? |
|:----:|
| [How about buying me a coffee?](https://ko-fi.com/shindakun) |

***