Semaphored Wait Group
Tags: go, concurrency, channels, semaphores
Published: 2020-04-29 13:19:59

One of the first things you might like to try when starting your journey with Go are the concurrency patterns.
You will probably start using _[goroutines](https://tour.golang.org/concurrency/1)_ to run things "in the background",
and you will also get to know _[channels](https://tour.golang.org/concurrency/2)_ that allow for safe communication between the sub-processes.

When you'll want to spawn many goroutines, you will surely get to know [WaitGroups](https://gobyexample.com/waitgroups) to wait for them to finish.
There will also be the _[defer](https://tour.golang.org/flowcontrol/12)_ statement which helps you to not forget to clean up after a function has finished running (and remember, deferred calls are executed in [reverse](https://tour.golang.org/flowcontrol/13) order).

Let's start with a simple example:
```go
package main

import (
	"log"
	"sync"
	"time"
)

func worker(id int, wg *sync.WaitGroup) {
	defer wg.Done()
	defer log.Printf("#%d done", id)
	log.Printf("#%d starting", id)
	time.Sleep(time.Second)
}

func main() {
	var wg sync.WaitGroup
	for i := 1; i <= 100; i++ {
		wg.Add(1)
		go worker(i, &wg)
	}
	wg.Wait()
	log.Printf("all done")
}
```

This works well, but what if we had a million items to process, and the actual work would be a memory-heavy operation? 
We don't want to start a million goroutines. 
We need something to limit the number of goroutines being run at the same time.

This is where a semaphore will come in handy.
We'll define how many concurrent workers we want, and the semaphore will not allow starting new goroutines until a slot is free.

## Using a Channel to Create a Simple Semaphore

Let's say we want a maximum of 5 workers running at any time.
We'll need a buffered channel, and we do not really care what type of values it holds.
```go
sem := make(chan bool, 5)
```
Whenever we want to start a goroutine, we'll push a value to the channel:
```go
		wg.Add(1)
		sem <- true
		go worker(i, &wg, sem) // we need to pass the channel to the worker
```
When we're done - we'll take one value out of the channel:
```go
func worker(id int, wg WaitGroup, sem <-chan bool) {
	defer wg.Done()
	defer log.Printf("#%d done", id)
	log.Printf("#%d starting", id)
	time.Sleep(time.Second)
	<-sem
}
```

> Please notice, that wen we added the `sem` argument to the worker, we also converted the semaphore to a read-only channel using `<-chan` syntax.

But wait..., we do not want to change the code too much. 
Can we just create our own implementation of a wait-group that will have the desired feature?

Yes we can, and Go's interfaces will help us do it. 

# Go's Implicit Interfaces - The Ultimate Decoupling

In most other languages the _interface_ has to be defined before it can be implemented. 
In Go - it is the other way around. 
We can create interfaces that satisfy our needs and if anything (internal or external) has the methods with matching signatures, 
Go will treat it as a type that implements the interface.

> [A Tour of Go](https://tour.golang.org/methods/10): Implicit interfaces decouple the definition of an interface from its implementation, which could then appear in any package without prearrangement.

So, even though we cannot change the `sync.WaitGroup` type, we can extract an interface that matches our current requirements. We'll have:
```go
type WaitGroup interface {
	Add(delta int)
	Done()
	Wait()
}
```
and change `worker` function's signature to:
```go
func worker(id int, wg WaitGroup)
```

Now, we can create our own implementation of **our** `WaitGroup` interface with a built-in semaphore.
```go
type SemaphoredWaitGroup struct {
	sem chan bool
	wg   sync.WaitGroup
}

func (s *SemaphoredWaitGroup) Add(delta int) {
	s.wg.Add(delta)
	s.sem <- true
}

func (s *SemaphoredWaitGroup) Done() {
	<-s.sem
	s.wg.Done()
}

func (s *SemaphoredWaitGroup) Wait() {
	s.wg.Wait()
}
```

Finally, instead of using the default `sync.WaitGroup` we can use our own `SemaphoredWaitGroup`:
```go
	wg := SemaphoredWaitGroup{sem: make(chan bool, 5)}
```
and we have ourselves a limited pool of workers. 

> The complete example is available as a [GitHub Gist](https://gist.github.com/pawzar/b05924b67d289247fd3259e8b961645b)
> and on [Go Playground](https://play.golang.org/p/jW-cO39WGC5).

Original post at: https://www.tegh.net/knowledge/semaphored-wait-group/