Serving compressed static assets with HTTP in Go 1.16
Tags: go, static, http
Published: 2021-01-17 12:27:21

The upcoming Go 1.16 release introduces a new way to manage static assets with [`embed`](https://tip.golang.org/pkg/embed/) package and `//go:embed` compiler directives.

Before there were multiple 3rd party tools that achieved embedding by explicitly generating `.go` files with assets data organized as strings or byte slices.

New `embed` package improves dev experience by eliminating the need for an explicit step to convert assets to Go code. Embedding is performed transparently during `go build`.

Another advantage is that embedded data is placed into a read-only TEXT section of a binary and it does not occupy process residential memory unless necessary.

To use `embed` package for static assets you can create a `.go` file in directory with asset files:

```
package static

import "embed"

//go:embed *
var Assets embed.FS
```

and then use `static.Assets` in your application.

You can create [`http.Handler`](https://golang.org/pkg/net/http/#Handler) to serve assets with

```go
http.FileServer(http.FS(static.Assets))
``` 
and mount it into your HTTP server.

This would work to serve your `.js`, `.css`, `.png` and other files out of the box with just one limitation: it won't help you with `Content-Encoding` of those assets. 

Many web resources (html, css, js to name a few types) allow high compression ratio due to their textual nature. Serving those resources compressed can make a big difference on web application latency and bandwidth consumption.

For dynamic data compression is usually applied dynamically, your web application can have an HTTP middleware to compress responses or it can be done externally with another web server (for example `nginx`). Dynamic compression takes extra CPU and memory, but in case of relatively small and dynamic payloads that cost is usually justified by improved overall performance.

On the other hand static assets don't change during application life cycle and compressing them over and over again for each request is a waste of resource. A better solution is to compress them once and serve compressed copies to many requests.

Nginx has a [`ngx_http_gzip_static_module`](http://nginx.org/en/docs/http/ngx_http_gzip_static_module.html) module to serve pre-compressed data using `.gz` file extensions to map compressed resources to requests. 

I thought this approach can work well for Go embedded files too and implemented [`github.com/vearutop/statigz`](https://github.com/vearutop/statigz): a file server that can serve compressed version of an asset if it is available in embedded file system.

You need to use 
```go
statigz.FileServer(static.Assets)
```
instead of 

```go
http.FileServer(http.FS(static.Assets))
```
to enable statically compressed assets.

## How it works?

Assume file server receives a request for `script.js`.

First, it checks if request accepts compressed encoding of response. If yes, it checks if `script.js.gz` (or `script.js.br` for [`brotli`](https://github.com/google/brotli)) is available in embedded file system and serves contents of `script.js.gz`.

If request does not accept available encodings (which should be unlikely in case of browsers), file server checks if `script.js` is available in file system and serves its content.

If `script.js` is not available, it checks for `script.js.gz` and decodes its content into response.

If neither `script.js` nor `script.js.gz|br` are available file server would respond with status `404 Not found`.

So, you can embed only compressed resources and serve them to clients that accept compression and to those that don't.

Files are served with `ETag` that is a hash of their contents to allow cache on client side.

Brotli is a more efficient than `gzip` compression format, but it has limitations (see [this](https://bugs.chromium.org/p/chromium/issues/detail?id=452335) and [this](https://bugzilla.mozilla.org/show_bug.cgi?id=1218924)) outside of HTTPS. At the same time its support adds ~260KB to binary size, so it is not enabled by default but available as an option.

```go
// Add import "github.com/vearutop/statigz/brotli".
statigz.FileServer(static.Assets, brotli.AddEncoding)
```

Recommended way of embedding assets is to compress them before the build, so that binary includes `*.gz` or `*.br` files. This can be inconvenient in some cases, there is [`EncodeOnInit`](https://pkg.go.dev/github.com/vearutop/statigz@v1.1.0#EncodeOnInit) option to compress assets in runtime when creating file server. 
```go
statigz.FileServer(static.Assets, brotli.AddEncoding, statigz.EncodeOnInit)
```
Once compressed, assets will be served directly without additional dynamic compression.

Files with extensions `.gz`, `.br`, `.gif`, `.jpg`, `.png`, `.webp` are excluded from runtime encoding by default.

> NOTE: Compressing assets in runtime can degrade startup performance and increase memory usage to prepare and store compressed data.