Don't Panic: Catching Panics in Errgroup Goroutines
Tags: go
Published: 2021-01-16 16:04:37

---
title: Don't Panic: Catching Panics in Errgroup Goroutines
published: true
description: How to gracefully avoid crashing in production 
tags: go
---
<img src ="https://upload.wikimedia.org/wikipedia/commons/0/00/The_Hitchhiker%27s_Guide_to_the_Galaxy%2C_english.svg" width="400" />

*Image by [Nicosmos](https://commons.wikimedia.org/wiki/File:The_Hitchhiker%27s_Guide_to_the_Galaxy,_english.svg), Public domain, via Wikimedia Commons*

### TL;DR version
[StevenACoffman/errgroup](https://github.com/StevenACoffman/errgroup) is a drop-in alternative to Go's wonderful [`sync/errgroup`](https://pkg.go.dev/golang.org/x/sync/errgroup) but it converts goroutine panics to errors.

### Why you want this

While `net/http` installs a panic handler with each request-serving goroutine,
goroutines **do not** and **cannot** inherit panic handlers from parent goroutines,
so a `panic()` in one of the child goroutines will kill the whole program.

So in production, whenever you use an `sync.errgroup`, you have to have the discipline to always remember to add a
deferred `recover()` to the beginning of **every** new goroutine, and convert any panics to errors. 

```go
			defer func() {
				if rec := recover(); rec != nil {
					err = FromPanicValue(rec)
				}
			}()
```
You can see this in [action in the Go Playground here](https://play.golang.org/p/WNgEPAQgbWg). You also want to be careful not to lose the stack trace. The `CollectStack` function I used here is overly simplified, so it adds a little noise because it doesn't the skip the `FromPanicValue` and `CollectStack` frames. 🤷‍♂️

```go
func FromPanicValue(i interface{}) error {
	switch value := i.(type) {
	case nil:
		return nil
	case string:
		return fmt.Errorf("panic: %v\n%s", value, CollectStack())
	case error:
		return fmt.Errorf("panic in errgroup goroutine %w\n%s", value, CollectStack())
	default:
		return fmt.Errorf("unknown panic: %+v\n%s", value, CollectStack())
	}
}

func CollectStack() []byte {
	buf := make([]byte, 64<<10)
	buf = buf[:runtime.Stack(buf, false)]
	return buf
}

```

A co-worker of mine co-worker [Ben Kraft](https://github.com/benjaminjkraft), wrote some handy wrapper code around `sync/errgroup` to *avoid* that boilerplate (and *required* discipline). With his permission, I [lightly modified it](https://github.com/StevenACoffman/errgroup) to
lift it out of our private work repository for the more general Go community.

`StevenACoffman/errgroup` is a drop-in alternative to Go's wonderful
[`sync/errgroup`](https://pkg.go.dev/golang.org/x/sync/errgroup) with the difference that it converts goroutine panics to errors. 

You can [see it in use in the playground](https://play.golang.org/p/S8Gmr_sWZIi) or here:

```go
package main

import (
	"fmt"

	"github.com/StevenACoffman/errgroup"
)

func main() {
	g := new(errgroup.Group)
	var urls = []string{
		"http://www.golang.org/",
		"http://www.google.com/",
		"http://www.somestupidname.com/",
	}
	for i := range urls {
		// Launch a goroutine to fetch the URL.
		i := i // https://golang.org/doc/faq#closures_and_goroutines
		g.Go(func() error {

			// deliberate index out of bounds triggered
			fmt.Println("Fetching:", i, urls[i+1])

			return nil
		})
	}
	// Wait for all HTTP fetches to complete.
	err := g.Wait()
	if err == nil {
		fmt.Println("Successfully fetched all URLs.")
	} else {
		fmt.Println(err)
	}
}
```

### Counterpoint
There is [an interesting discussion](https://github.com/oklog/run/issues/10) which has an alternative view that,
with few exceptions, panics **should** crash your program. I'm ok with that in development and testing, but would rather sleep soundly at night.

### Prior Art
With only a cursory search, I found a few existing open source examples.

#### [Kratos](https://github.com/go-kratos/kratos) errgroup 

Kratos Go framework for microservices has a similar [errgroup](https://github.com/go-kratos/kratos/blob/master/pkg/sync/errgroup/errgroup.go)
solution.

#### PanicGroup by Sergey Alexandrovich

In the article [Errors in Go:
From denial to acceptance](https://evilmartians.com/chronicles/errors-in-go-from-denial-to-acceptance), 
(which advocates panic based flow control 😱), they have a `PanicGroup` that's roughly equivalent:

```go
type PanicGroup struct {
  wg      sync.WaitGroup
  errOnce sync.Once
  err     error
}

func (g *PanicGroup) Wait() error {
  g.wg.Wait()
  return g.err
}

func (g *PanicGroup) Go(f func()) {
  g.wg.Add(1)

  go func() {
    defer g.wg.Done()
    defer func(){
      if r := recover(); r != nil {
        if err, ok := r.(error); ok {
          // We need only the first error, sync.Once is useful here.
          g.errOnce.Do(func() {
            g.err = err
          })
        } else {
          panic(r)
        }
      }
    }()

    f()
  }()
}
```

I would appreciate feedback or suggestions for improvement!