Implemented a bench marker to compare Go's HTTP Router
Tags: go, webdev, programming, opensource
Published: 2022-12-20 13:28:46

This article is a translation of [bmf-tech.com - GoのHTTP Routerを比較するベンチマーカーを実装した](https://bmf-tech.com/posts/Go%e3%81%aeHTTP%20Router%e3%82%92%e6%af%94%e8%bc%83%e3%81%99%e3%82%8b%e3%83%99%e3%83%b3%e3%83%81%e3%83%9e%e3%83%bc%e3%82%ab%e3%83%bc%e3%82%92%e5%ae%9f%e8%a3%85%e3%81%97%e3%81%9f).

This is day 5 of [Makuake Advent Calendar 2022](https://adventar.org/calendars/8496)!

# Overview
Bench markers were implemented to compare the performance of Go's HTTP Router.

[bmf-san/go-router-benchmark](https://github.com/bmf-san/go-router-benchmark)

The following HTTP Routers are currently being compared.

- [bmf-san/goblin](https://github.com/bmf-san/goblin)
- [julienschmidt/httprouter](https://github.com/julienschmidt/httprouter)
- [go-chi/chi](https://github.com/go-chi/chi)
- [gin-gonic/gin](https://github.com/gin-gonic/gin)
- [uptrace/bunrouter](https://github.com/uptrace/bunrouter)
- [dimfeld/httptreemux](https://github.com/dimfeld/httptreemux)
- [beego/mux](https://github.com/beego/mux)
- [gorilla/mux](https://github.com/gorilla/mux)
- [nissy/bon](https://github.com/nissy/bon)
- [naoina/denco](https://github.com/naoina/denco)
- [labstack/echo](https://github.com/labstack/echo/v4)
- [gocraft/web](https://github.com/gocraft/web)
- [vardius/gorouter](https://github.com/vardius/gorouter)
- [go-ozzo/ozzo-routing](https://github.com/go-ozzo/ozzo-routing)
- [lkeix/techbook13-sample](https://github.com/lkeix/techbook13-sample)

In some test cases, the Go standard [net/http#ServeMux](https://pkg.go.dev/net/http#ServeMux) is also covered.

# Motivation
I have created my own HTTP Router called [bmf-san/goblin](https://github.com/bmf-san/goblin).

[bmf-san/goblin](https://github.com/bmf-san/goblin) is a simple HTTP Router based on the Trie tree with minimal functionality.

By comparing the performance of [bmf-san/goblin](https://github.com/bmf-san/goblin) with other HTTP Routers, [bmf-san/goblin](https://github.com/bmf-san/) goblin), the motivation is to get hints on how to improve [bmf-san/goblin]().

Another reason is that I have my own bench marker that can replace [julienschmidt/go-http-routing-benchmark](https://github.com/julienschmidt/go-http-routing-benchmark). Another motivation was to be able to maintain

The [julienschmidt/go-http-routing-benchmark](https://github.com/julienschmidt/go-http-routing-benchmark) is the [julienschmidt/httprouter]( https://github.com/julienschmidt/httprouter), but maintenance seemed to have stopped in recent years, so I decided to create my own benchmarker and implement it. I decided to implement bench markers.

# About the test design of the bench marker
As a premise, I would like to state that [bmf-san/go-router-benchmark](https://github.com/bmf-san/go-router-benchmark) is not a complete comparison of HTTP Router performance .

The reasons are as follows.

- Each HTTP Router has different functions and specifications, and it is not realistic to prepare a test case that perfectly covers all of them, so only some of the specifications are taken into account in the comparison.
- Because each HTTP Router has its own data structure and algorithms, which may or may not be good or bad, and therefore the defined routing test cases may not fully evaluate the performance of each HTTP Router.

Therefore, the benchmark test will be a test case for a specific function or specification of HTTP Router only, but it can measure a certain difference in performance.

In [bmf-san/go-router-benchmark](https://github.com/bmf-san/go-router-benchmark), the performance is measured for the processing part of routing.
Specifically, it tests the `ServeHTTP` function in [http#Handler](https://pkg.go.dev/net/http#Handler).

[benchmark_test.go#L21](https://github.com/bmf-san/go-router-benchmark/blob/main/benchmark_test.go#L21)

The part that defines routing is not tested.
The process of defining routing is the process of registering data necessary for the routing process.

```go
package main

import (
	"fmt"
	"net/http" )
)

func handler(w http.ResponseWriter, r *http.Request) {
	fmt.Fprint(w, "Hello World")
}

func main() {
	mux := http.NewServeMux()
	mux.HandleFunc("/", handler) // here
	ListenAndServe(":8080", mux)
}
```

The following two test cases are available for the routing process.

- Static routes
- Routes with path parameters

Each test case is described below.

## Static routes
A static route is a route without variable parameters such as `/foo/bar`.

The following four input patterns are available for testing this route.

- `/`
- `/foo`
- `/foo/bar/baz/qux/quux`
- `/foo/bar/baz/qux/quux/corge/grault/garply/waldo/fred`

This test case also compares the Go standard [net/http#ServeMux](https://pkg.go.dev/net/http#ServeMux).

## Routes with path parameters
A route with path parameters is a route with variable parameters such as `/foo/:bar`.

In this route test, the following three input patterns are provided.

- `/foo/:bar`.
- `/foo/:bar/:baz/:qux/:quux/:corge`.
- `/foo/:bar/:baz/:qux/:quux/:corge/:grault/:garply/:waldo/:fred/:plugh`

Since some HTTP Routers have different syntaxes for variable parameters, we also take into account the corresponding syntaxes for each.

ex.[pathparam.go#L15](https://github.com/bmf-san/go-router-benchmark/blob/main/pathparam.go#L15)

# Benchmark test results
Benchmark test results are available at [<Public>go-router-benchmark](https://docs.google.com/spreadsheets/d/1DrDNGJXfquw_PED3-eMqWqh7qbCTVBWZtqaKPXtngxg/ edit#gid=1913830192).

The benchmark test environment is as follows

- go version: go1.19
- goos: darwin
- goarch: amd64
- pkg: github.com/go-router-benchmark
- cpu: VirtualApple @ 2.50GHz

The benchmark results are as follows
- time
  - Number of function executions
  - The higher the number of executions, the better the performance.
- ns/op
  - Time required per function execution
  - The smaller the time, the better the performance
- B/op
  - Size of memory allocated per function execution
  - The smaller the size, the better the performance
- allocs/op
  - Number of memory allocations made per function execution
  - The fewer the number of memory allocations made per execution of the allocs/op function, the better the performance


cf. [dev.to - Improving the performance of your code starting with Go](https://dev.to/bmf_san/improving-the-performance-of-your-code-starting-with-go-522)

Describe the results of each test case.

## Static routes
For static routes, one point of comparison seems to be whether the performance is better than or equal to the standard [net/http#ServeMux](https://pkg.go.dev/net/http#ServeMux).
HTTP Router, which is touted for its better performance, is still better than the standard.

### time
|       time        | static-routes-root | static-routes-1 | static-routes-5 | static-routes-10 |
| ----------------- | ------------------ | --------------- | --------------- | ---------------- |
| servemux          | 24301910           | 22053468        | 13324357        | 8851803          |
| goblin            | 32296879           | 16738813        | 5753088         | 3111172          |
| httprouter        | 100000000          | 100000000       | 100000000       | 72498970         |
| chi               | 5396652            | 5350285         | 5353856         | 5415325          |
| gin               | 34933861           | 34088810        | 34136852        | 33966028         |
| bunrouter         | 63478486           | 54812665        | 53564055        | 54345159         |
| httptreemux       | 6669231            | 6219157         | 5278312         | 4300488          |
| beegomux          | 22320199           | 15369320        | 1000000         | 577272           |
| gorillamux        | 1807042            | 2104210         | 1904696         | 1869037          |
| bon               | 72425132           | 56830177        | 59573305        | 58364338         |
| denco             | 90249313           | 92561344        | 89325312        | 73905086         |
| echo              | 41742093           | 36207878        | 23962478        | 12379764         |
| gocraftweb        | 1284613            | 1262863         | 1000000         | 889360           |
| gorouter          | 21622920           | 28592134        | 15582778        | 9636147          |
| ozzorouting       | 31406931           | 34989970        | 24825552        | 19431296         |
| techbook13-sample | 8176849            | 6349896         | 2684418         | 1384840          |

<iframe width="467" height="741" seamless frameborder="0" scrolling="no" src="https://docs.google.com/spreadsheets/d/e/2PACX-1vRiWBjJim4v_XyoN45s4VVQXD-PIBHKjyVfOv5OX37376SZ9GvL5bmqQegLl-5arBpD-3hhTKTEgkIj/pubchart?oid=800028423&amp;format=interactive"></iframe>

[Graph - time](https://docs.google.com/spreadsheets/d/e/2PACX-1vRiWBjJim4v_XyoN45s4VVQXD-PIBHKjyVfOv5OX37376SZ9GvL5bmqQegLl-5arBpD-3hhTKTEgkIj/pubchart?oid=800028423&format=interactive)

### nsop
|       nsop        | static-routes-root | static-routes-1 | static-routes-5 | static-routes-10 |
| ----------------- | ------------------ | --------------- | --------------- | ---------------- |
| servemux          | 50.44              | 54.97           | 89.81           | 135.2            |
| goblin            | 36.63              | 69.9            | 205.2           | 382.7            |
| httprouter        | 10.65              | 10.74           | 10.75           | 16.42            |
| chi               | 217.2              | 220.1           | 216.7           | 221.5            |
| gin               | 34.53              | 34.91           | 34.69           | 35.04            |
| bunrouter         | 18.77              | 21.78           | 22.41           | 22               |
| httptreemux       | 178.8              | 190.9           | 227.2           | 277.7            |
| beegomux          | 55.07              | 74.69           | 1080            | 2046             |
| gorillamux        | 595.7              | 572.8           | 626.5           | 643.3            |
| bon               | 15.75              | 20.17           | 18.87           | 19.16            |
| denco             | 14                 | 13.03           | 13.4            | 15.87            |
| echo              | 28.17              | 32.83           | 49.82           | 96.77            |
| gocraftweb        | 929.4              | 948.8           | 1078            | 1215             |
| gorouter          | 55.16              | 37.64           | 76.6            | 124.1            |
| ozzorouting       | 42.62              | 34.22           | 48.12           | 61.6             |
| techbook13-sample | 146.1              | 188.4           | 443.5           | 867.8            |

<iframe width="467" height="741" seamless frameborder="0" scrolling="no" src="https://docs.google.com/spreadsheets/d/e/2PACX-1vRiWBjJim4v_XyoN45s4VVQXD-PIBHKjyVfOv5OX37376SZ9GvL5bmqQegLl-5arBpD-3hhTKTEgkIj/pubchart?oid=1691114342&amp;format=interactive"></iframe>

[Graph - nsop](https://docs.google.com/spreadsheets/d/e/2PACX-1vRiWBjJim4v_XyoN45s4VVQXD-PIBHKjyVfOv5OX37376SZ9GvL5bmqQegLl-5arBpD-3hhTKTEgkIj/pubchart?oid=1691114342&format=interactive)

### bop
|        bop        | static-routes-root | static-routes-1 | static-routes-5 | static-routes-10 |
| ----------------- | ------------------ | --------------- | --------------- | ---------------- |
| servemux          | 0                  | 0               | 0               | 0                |
| goblin            | 0                  | 16              | 80              | 160              |
| httprouter        | 0                  | 0               | 0               | 0                |
| chi               | 304                | 304             | 304             | 304              |
| gin               | 0                  | 0               | 0               | 0                |
| bunrouter         | 0                  | 0               | 0               | 0                |
| httptreemux       | 328                | 328             | 328             | 328              |
| beegomux          | 32                 | 32              | 32              | 32               |
| gorillamux        | 720                | 720             | 720             | 720              |
| bon               | 0                  | 0               | 0               | 0                |
| denco             | 0                  | 0               | 0               | 0                |
| echo              | 0                  | 0               | 0               | 0                |
| gocraftweb        | 288                | 288             | 352             | 432              |
| gorouter          | 0                  | 0               | 0               | 0                |
| ozzorouting       | 0                  | 0               | 0               | 0                |
| techbook13-sample | 304                | 308             | 432             | 872              |

<iframe width="467" height="741" seamless frameborder="0" scrolling="no" src="https://docs.google.com/spreadsheets/d/e/2PACX-1vRiWBjJim4v_XyoN45s4VVQXD-PIBHKjyVfOv5OX37376SZ9GvL5bmqQegLl-5arBpD-3hhTKTEgkIj/pubchart?oid=675738282&amp;format=interactive"></iframe>

[Graph - bop](https://docs.google.com/spreadsheets/d/e/2PACX-1vRiWBjJim4v_XyoN45s4VVQXD-PIBHKjyVfOv5OX37376SZ9GvL5bmqQegLl-5arBpD-3hhTKTEgkIj/pubchart?oid=675738282&format=interactive)

### allocs
|      allocs       | static-routes-root | static-routes-1 | static-routes-5 | static-routes-10 |
| ----------------- | ------------------ | --------------- | --------------- | ---------------- |
| servemux          | 0                  | 0               | 0               | 0                |
| goblin            | 0                  | 1               | 1               | 1                |
| httprouter        | 0                  | 0               | 0               | 0                |
| chi               | 2                  | 2               | 2               | 2                |
| gin               | 0                  | 0               | 0               | 0                |
| bunrouter         | 0                  | 0               | 0               | 0                |
| httptreemux       | 3                  | 3               | 3               | 3                |
| beegomux          | 1                  | 1               | 1               | 1                |
| gorillamux        | 7                  | 7               | 7               | 7                |
| bon               | 0                  | 0               | 0               | 0                |
| denco             | 0                  | 0               | 0               | 0                |
| echo              | 0                  | 0               | 0               | 0                |
| gocraftweb        | 6                  | 6               | 6               | 6                |
| gorouter          | 0                  | 0               | 0               | 0                |
| ozzorouting       | 0                  | 0               | 0               | 0                |
| techbook13-sample | 2                  | 3               | 11              | 21               |

<iframe width="467" height="741" seamless frameborder="0" scrolling="no" src="https://docs.google.com/spreadsheets/d/e/2PACX-1vRiWBjJim4v_XyoN45s4VVQXD-PIBHKjyVfOv5OX37376SZ9GvL5bmqQegLl-5arBpD-3hhTKTEgkIj/pubchart?oid=1054975173&amp;format=interactive"></iframe>

[Graph - allocs](https://docs.google.com/spreadsheets/d/e/2PACX-1vRiWBjJim4v_XyoN45s4VVQXD-PIBHKjyVfOv5OX37376SZ9GvL5bmqQegLl-5arBpD-3hhTKTEgkIj/pubchart?oid=1054975173&format=interactive)

## Routes with path parameters
For routes with path parameters, we split the performance into two groups: one with a large degradation in performance as the number of parameters increases, and the other with a modest degradation.

### time
|       time        | pathparam-routes-1 | pathparam-routes-5 | pathparam-routes-10 |
| ----------------- | ------------------ | ------------------ | ------------------- |
| goblin            | 1802690            | 492392             | 252274              |
| httprouter        | 25775940           | 10057874           | 6060843             |
| chi               | 4337922            | 2687157            | 1772881             |
| gin               | 29479381           | 15714673           | 9586220             |
| bunrouter         | 37098772           | 8479642            | 3747968             |
| httptreemux       | 2610324            | 1550306            | 706356              |
| beegomux          | 3177818            | 797472             | 343969              |
| gorillamux        | 1364386            | 470180             | 223627              |
| bon               | 6639216            | 4486780            | 3285571             |
| denco             | 20093167           | 8503317            | 4988640             |
| echo              | 30667137           | 12028713           | 6721176             |
| gocraftweb        | 921375             | 734821             | 466641              |
| gorouter          | 4678617            | 3038450            | 2136946             |
| ozzorouting       | 27126000           | 12228037           | 7923040             |
| techbook13-sample | 3019774            | 917042             | 522897              |

<iframe width="455" height="739" seamless frameborder="0" scrolling="no" src="https://docs.google.com/spreadsheets/d/e/2PACX-1vRiWBjJim4v_XyoN45s4VVQXD-PIBHKjyVfOv5OX37376SZ9GvL5bmqQegLl-5arBpD-3hhTKTEgkIj/pubchart?oid=1039813866&amp;format=interactive"></iframe>

[Graph - time](https://docs.google.com/spreadsheets/d/e/2PACX-1vRiWBjJim4v_XyoN45s4VVQXD-PIBHKjyVfOv5OX37376SZ9GvL5bmqQegLl-5arBpD-3hhTKTEgkIj/pubchart?oid=1039813866&format=interactive)

### nsop
|       nsop        | pathparam-routes-1 | pathparam-routes-5 | pathparam-routes-10 |
| ----------------- | ------------------ | ------------------ | ------------------- |
| goblin            | 652.4              | 2341               | 4504                |
| httprouter        | 45.73              | 117.4              | 204.2               |
| chi               | 276.4              | 442.8              | 677.6               |
| gin               | 40.21              | 76.39              | 124.3               |
| bunrouter         | 32.52              | 141.1              | 317.2               |
| httptreemux       | 399.7              | 778.5              | 1518                |
| beegomux          | 377.2              | 1446               | 3398                |
| gorillamux        | 850.3              | 2423               | 5264                |
| bon               | 186.5              | 269.6              | 364.4               |
| denco             | 60.47              | 139.4              | 238.7               |
| echo              | 39.36              | 99.6               | 175.7               |
| gocraftweb        | 1181               | 1540               | 2280                |
| gorouter          | 256.4              | 393                | 557.6               |
| ozzorouting       | 43.66              | 99.52              | 150.4               |
| techbook13-sample | 380.7              | 1154               | 2150                |

<iframe width="460" height="739" seamless frameborder="0" scrolling="no" src="https://docs.google.com/spreadsheets/d/e/2PACX-1vRiWBjJim4v_XyoN45s4VVQXD-PIBHKjyVfOv5OX37376SZ9GvL5bmqQegLl-5arBpD-3hhTKTEgkIj/pubchart?oid=1534246873&amp;format=interactive"></iframe>

[Graph - nsop](https://docs.google.com/spreadsheets/d/e/2PACX-1vRiWBjJim4v_XyoN45s4VVQXD-PIBHKjyVfOv5OX37376SZ9GvL5bmqQegLl-5arBpD-3hhTKTEgkIj/pubchart?oid=1534246873&format=interactive)

### bop
|        bop        | pathparam-routes-1 | pathparam-routes-5 | pathparam-routes-10 |
| ----------------- | ------------------ | ------------------ | ------------------- |
| goblin            | 409                | 962                | 1608                |
| httprouter        | 32                 | 160                | 320                 |
| chi               | 304                | 304                | 304                 |
| gin               | 0                  | 0                  | 0                   |
| bunrouter         | 0                  | 0                  | 0                   |
| httptreemux       | 680                | 904                | 1742                |
| beegomux          | 672                | 672                | 1254                |
| gorillamux        | 1024               | 1088               | 1751                |
| bon               | 304                | 304                | 304                 |
| denco             | 32                 | 160                | 320                 |
| echo              | 0                  | 0                  | 0                   |
| gocraftweb        | 656                | 944                | 1862                |
| gorouter          | 360                | 488                | 648                 |
| ozzorouting       | 0                  | 0                  | 0                   |
| techbook13-sample | 432                | 968                | 1792                |

<iframe width="460" height="739" seamless frameborder="0" scrolling="no" src="https://docs.google.com/spreadsheets/d/e/2PACX-1vRiWBjJim4v_XyoN45s4VVQXD-PIBHKjyVfOv5OX37376SZ9GvL5bmqQegLl-5arBpD-3hhTKTEgkIj/pubchart?oid=73824357&amp;format=interactive"></iframe>

[Graph - bop](https://docs.google.com/spreadsheets/d/e/2PACX-1vRiWBjJim4v_XyoN45s4VVQXD-PIBHKjyVfOv5OX37376SZ9GvL5bmqQegLl-5arBpD-3hhTKTEgkIj/pubchart?oid=73824357&format=interactive)

### allocs
|      allocs       | pathparam-routes-1 | pathparam-routes-5 | pathparam-routes-10 |
| ----------------- | ------------------ | ------------------ | ------------------- |
| goblin            | 6                  | 13                 | 19                  |
| httprouter        | 1                  | 1                  | 1                   |
| chi               | 2                  | 2                  | 2                   |
| gin               | 0                  | 0                  | 0                   |
| bunrouter         | 0                  | 0                  | 0                   |
| httptreemux       | 6                  | 9                  | 11                  |
| beegomux          | 5                  | 5                  | 6                   |
| gorillamux        | 8                  | 8                  | 9                   |
| bon               | 2                  | 2                  | 2                   |
| denco             | 1                  | 1                  | 1                   |
| echo              | 0                  | 0                  | 0                   |
| gocraftweb        | 9                  | 12                 | 14                  |
| gorouter          | 4                  | 4                  | 4                   |
| ozzorouting       | 0                  | 0                  | 0                   |
| techbook13-sample | 10                 | 33                 | 59                  |

<iframe width="460" height="739" seamless frameborder="0" scrolling="no" src="https://docs.google.com/spreadsheets/d/e/2PACX-1vRiWBjJim4v_XyoN45s4VVQXD-PIBHKjyVfOv5OX37376SZ9GvL5bmqQegLl-5arBpD-3hhTKTEgkIj/pubchart?oid=344550080&amp;format=interactive"></iframe>

[Graph - allocs](https://docs.google.com/spreadsheets/d/e/2PACX-1vRiWBjJim4v_XyoN45s4VVQXD-PIBHKjyVfOv5OX37376SZ9GvL5bmqQegLl-5arBpD-3hhTKTEgkIj/pubchart?oid=344550080&format=interactive)

# Conclusion.
It can be seen that the HTTP Router with better performance shows less performance degradation in each test case.
This seems to be a clear trend indicating that the implementations are optimized.

When examining some of the better performing HTTP Router implementations, we find that they employ a more sophisticated tree structure.
For example, Echo, gin, httprouter, bon, and chi use the Radix tree (Patricia trie), while denco uses a double array.

Regarding [bmf-san/goblin](https://github.com/bmf-san/goblin), we found that it is a proprietary extension of the tri-tree, which is not very optimized and has poor performance compared to other HTTP Routers. (We will do our best to improve it...)

On the other hand, it seemed possible that the multifunctionality of some of the HTTP Routers that seemed to perform poorly may have reduced performance.

We felt that we could further obtain performance trends for each HTTP Router by adding test cases, which we will try to address if we have time.