Experiments with h3 clients + Envoy
Tags: envoyproxy, http3, quic
Published: 2021-04-13 03:07:13

I've been experimenting with HTTP/3 (h3) support in Envoy Proxy. I now have both upstream and downstream working

Inspired by the paper referred to in {% twitter 1361566348887920640 %} I set out to expand the number of h3 clients for testing



Thanks to [@howardjohn](https://github.com/howardjohn) at Google for helping with the initial h3 config. Thanks to [@triplewy1](https://twitter.com/triplewy1) for helping me sort out the correct parameters to pass to proxygen.

Big thanks to the Envoy team who have helped with configs, testing, ideas etc. In particular [@alyssawilk](https://github.com/alyssawilk), [@danzh2010](https://github.com/danzh2010) + [@mattklein123](https://github.com/mattklein123)

# Envoy

I'm building Envoy from source (main branch) on Linux with a limited set of extensions

# Clients

I've ended up building and testing 7 h3 clients:

| #. | Client                                                                               | Language      | Compilation                   |
| -- | -------------                                                                        |-------------  | -------                       |
| 1. | [curl/curl](https://github.com/curl/curl/blob/master/docs/HTTP3.md)                  | C             | cloudflare/quiche + BoringSSL |
| 2. | [hyperium/h3](https://github.com/hyperium/h3)                                        | Rust          | musl static                   |
| 3. | [proxygen/hq](https://github.com/facebook/proxygen)                                  | C++           |                               |
| 4. | [mozilla/neqo](https://github.com/mozilla/neqo/blob/main/docs/linux_build.md)        | Rust          | NSS                           |
| 5. | [istio/quic-go](https://github.com/istio/istio/tree/master/pkg/test/echo/cmd/client) | Go            |                               |
| 6. | [cloudflare/quiche](https://github.com/cloudflare/quiche)                            | Rust          | musl static                   |
| 7. | [quinn-rs/quinn](https://github.com/quinn-rs/quinn)                                  | Rust          | musl static                   |

# Testing

## hyperfine

I used the excellent [hyperfine](https://github.com/sharkdp/hyperfine) for testing. Please note that benchmarking is hard and this is in no way a proper benchmark. This is more for fun, learning how to build and use new h3 clients and working out how to configure h3 / QUIC for Envoy. Please take all results with a huge grain of salt

tl;dr - neqo generally slightly quicker followed closely by proxygen, quic-go + h3 (not always in that order). Then quinn, curl and cloudflare/quiche. I'm surprised by cloudflare/quiche being so slow however I believe it has not been optimized at this point

{% gist https://gist.github.com/moderation/1dd5477b1e20c099e97ca6d60f987652 %} 

## h3spec

I've also tested using the excellent [h3spec](https://github.com/kazu-yamamoto/h3spec). We found one crashing bug using this test suite which has subsequently been fixed

{% gist https://gist.github.com/moderation/66d699bd89d973fbf9dbf5448888b176 %}

45 examples, 13 failures. This suite has been great for catching crashes but it should be noted the goal is not to attain 100% as there are a number of performance trade-offs to consider

# Config

## Downstream h3 with local direct responses + h2 upstream

The first config shows how to set up a TCP + UDP listener on the same port, testing JSON structed logging, an Envoy direct response on `/local`, `alt-svc` headers on h2 responses

I use CUE for all of these configs and these are exported YAML. The process is started on the Fish shell with:

`<path to>/envoy --concurrency 1 --log-level debug --config-path (cue export downstream_httpbin_org.cue | psub)`

Note that for h3 to worked today you'll need to set `concurrency` to 1

{% gist https://gist.github.com/moderation/7878a0a4e56965a3548547795123eb85 %}

## Downstream h1 with h3 upstream

This is a simpler config with a stock h1 listener but talks h3 to the upstream service

{% gist https://gist.github.com/moderation/7004f3fd47d54af5f52713528331a91b %}

# Future

* Would be fun to test this with things like dynamic forward proxies
* The testing above is done on an Envoy proxy with a the runtime value `envoy.reloadable_features.prefer_quic_kernel_bpf_packet_routing: true` set and Linux Capabilities of `sudo setcap cap_bpf+ep <path to>/envoy` on a kernel >= 5.8.x. However as per the following issue it is not sure what effect this has {% github https://github.com/envoyproxy/envoy/issues/15845 %} 

# Updates

* quinn removed support for their h3 test client in https://github.com/quinn-rs/quinn/pull/1155