Reverse HTTP proxy over WebSocket in Go (Part 1)
Tags: go, webdev, websocket
Published: 2021-12-14 10:07:16

## Key takeaways

- A reverse HTTP proxy over WebSocket is a type of proxy server which uses the WebSocket protocol as a "tunnel" to pass TCP communication from server to client.
- In Go project, [gorilla/websocket](https://github.com/gorilla/websocket) is widely used to implement WebSocket.
- WebSocket is designed to work over HTTP. To achieve compatibility with HTTP, the WebSocket handshake uses the HTTP [Upgrade](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Upgrade) header in order to change from the HTTP protocol to the WebSocket protocol.
- Authentication of WebSocket is a difficult design problem, and there are many options.

## Reverse HTTP proxy over WebSocket

There are two types of proxies, forward proxy and reverse proxy. 

A forward proxy (or gateway, or tunnel, or just "proxy") provides proxy services to a client or a group of clients. It allow users to hide their IP address while accessing a Web service or an API server. In contract, a [reverse proxy](https://en.wikipedia.org/wiki/Reverse_proxy) is the opposite of what a forward proxy does. It is a type of proxy server that retrieves resources on behalf of a client from servers.

Further, this post focuses on a reverse HTTP proxy **over WebSocket**, in brief, it uses the WebSocket protocol as a "tunnel" to pass TCP communication from server to client.

![A network diagram for reverse proxy over WebSocket](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iupaoeq283lyeqpkg049.png)

## A case study in Go

There are not that many examples of implementation reverse proxy over WebSocket. In JavaScript, [mhzed/wstunnel](https://github.com/mhzed/wstunnel) is well known, in Haskell, [erebe/wstunnel](https://github.com/erebe/wstunnel) is. 

In Go, [inconshreveable/ngrok](https://github.com/inconshreveable/ngrok) and [coyove/goflyway](https://github.com/coyove/goflyway) is well known, especially [ngrok](https://ngrok.com/) is popular among developers as a SaaS service.

In this post, we will focus on understanding the basic concepts and read prototypical and simpler one [root-gg/wsp](https://github.com/root-gg/wsp). wsp is developed by [root-gg](https://github.com/root-gg), which is a reverse HTTP proxy over WebSocket, whose aim is to securely make call to internal APIs from outside. It is difficult to use in production, but it is a good learning material to explain the design of reverse proxy over WebSocket.

However, maintenance has stopped since the days of Go 1.6, so I'll proceed with this post based on code [hgsgtk/wsp](https://github.com/hgsgtk/wsp) that I forked and modified for the Go situation in 2021 (Thank you [root-gg](https://github.com/root-gg)).

{% github hgsgtk/wsp no-readme %}

## Demonstration of wsp

It's easier to get a general idea of how it works if you see it in action, so here's a brief demo. In the demonstration, the following four system component are introduced.

1. Internal API(*http://localhost:8081*): the server you want to request via the reverse proxy
2. wsp client: running in a network with the internal API
3. wsp server(*http://localhost:8080*): running outside the network
4. App: starting point for communication to the internal API

![A diagram of system components quoted from README in root-gg/wsp](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bh86d3jt53euo8dcwpj8.png)

Try to send a request to the internal API via WebSocket communication to the endpoint `/hello`.

```bash
$ curl -H 'X-PROXY-DESTINATION: http://localhost:8081/hello' http://127.0.0.1:8080/request
```

HTTP communication was relayed by the following route.

    app -> wsp server -(WebSocket)-> wsp client -> internal API
 
Here is a terminal image when sending a HTTP request from app.

![A terminal image when sending a HTTP request from app](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nxdm25yw7vsxg3tzpdhz.png)
 
![A terminal image when wsp client got a message from wsp server](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t1nqmlepsmps0u1j41mz.png)

The internal network restricts incoming requests from outside or does not have any global IPs, so that it is not possible for external server to connect to a client started in an internal network. For example, if the client is launched on your local PC, it is not possible to send a request to the client from an external server.

![A diagram describing the internal network restricts requests from outside](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g46nm13lirz81yxbpnyp.png)

Therefore, the starting point is wsp client in the internal network. In this design, it starts with a request for a WebSocket connection from wsp client to wsp server.

![A diagram describing to establish WebSocket connection from client](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6h0wc2gvgyer36krwjrz.png)

The following points to implement this design will be explained in this post.

- Start a WebSocket server

I will explain the rest points in part 2 and beyond.

- Establish a WebSocket connection
- Keep a established connection
- Relay TCP connection from "App" to the peer of WebSocket
- Relay TCP connection in WebSocket data to "internal API"
 
## Start a WebSocket server 

The first step is to start up a WebSocket server which waits for requests from WebSocket clients. 

As a side note, WebSocket is a mechanism for low-cost, [full-duplex](https://en.wikipedia.org/wiki/Duplex_(telecommunications)#FULL-DUPLEX) communication on Web, which protocol was standardized as [RFC 6455](https://datatracker.ietf.org/doc/html/rfc6455). The following diagram, quoted by [Wikipedia](https://en.wikipedia.org/wiki/WebSocket), describe a communication using WebSocket between client and server.

![A diagram describing a connection using WebSocket](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z7qiia95sgoly2rya998.png)

In the case of wsp, it starts like this.

```bash
$ ./wsp_server -config examples/wsp_server.cfg
{
  "Host": "127.0.0.1",
  "Port": 8080,
  "Timeout": 1000,
  "IdleTimeout": 60000,
  "Whitelist": [],
  "Blacklist": [],
  "SecretKey": ""
}
```

Let's read the Go code. First is the main function ([cmd/wsp_server/main.go](https://github.com/hgsgtk/wsp/blob/3594d04fdadaddfa2e206339ebad63a127b050af/cmd/wsp_server/main.go)), which is the entry point.

```go
package main

import (
	// (omit)
)

func main() {
	// (omit)

	server := server.NewServer(config)
	server.Start()

	// (omit: shutdown)
}
```

[server.NewServer](https://github.com/hgsgtk/wsp/blob/ec9cc7a1921d2f9a753f0019ac21ab3e3d7abd0e/server/server.go#L52) function returns a pointer of [Server](https://github.com/hgsgtk/wsp/blob/ec9cc7a1921d2f9a753f0019ac21ab3e3d7abd0e/server/server.go#L21) struct which represents a reverse HTTP proxy over WebSocket.

```go
type Server struct {
	Config *Config

	upgrader websocket.Upgrader

	pools []*Pool
	lock  sync.RWMutex
	done  chan struct{}

	dispatcher chan *ConnectionRequest

	server *http.Server
}
```

An important field is `upgrader` whose type is [websocket.Upgrader](https://pkg.go.dev/github.com/gorilla/websocket#Upgrader) to specify parameters for upgrading an HTTP connection to a WebSocket connection. In Go project, [gorilla/websocket](https://github.com/gorilla/websocket) is widely used to implement WebSocket, and many samples using this library are available on the internet.

WebSocket is designed to work over HTTP. To achieve compatibility with HTTP, the WebSocket handshake uses the HTTP [Upgrade](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Upgrade) header in order to change from the HTTP protocol to the WebSocket protocol.

Fields of [Upgrader](https://pkg.go.dev/github.com/gorilla/websocket#Upgrader) struct is as follow:

```go
type Upgrader struct {
	HandshakeTimeout time.Duration
	ReadBufferSize, WriteBufferSize int
	Subprotocols []string
	Error func(w http.ResponseWriter, r *http.Request, status int, reason error)
	CheckOrigin func(r *http.Request) bool
	EnableCompression bool
}
```

If you want to know more about the WebSocket parameters or gorilla/websocket internal implementation, refer to this post [Learn WebSocket handshake protocol with gorilla/websocket server](https://dev.to/hgsgtk/learn-websocket-handshake-protocol-with-gorillawebsocket-server-10k9).

In `NewServer` function, it creates a value whose type is [websocket.Upgrader](https://pkg.go.dev/github.com/gorilla/websocket#Upgrader) with default parameters.

```go
func NewServer(config *Config) (server *Server) {
	// (omit)

	server.upgrader = websocket.Upgrader{}

	// (omit)
	return
}
```

Next, call `server.Start` function to start a WebSocket server.

```go
func (server *Server) Start() {
	// (omit)

	r := http.NewServeMux()
	r.HandleFunc("/request", server.request)

	// (omit)

	server.server = &http.Server{
		Addr:    server.Config.GetAddr(),
		Handler: r,
	}
	go func() { log.Fatal(server.server.ListenAndServe()) }()
```

In this code, an HTTP server that exposes an endpoint `/request` is set up at a given address. The handler mapped to `/request` is as follow ([code](https://github.com/hgsgtk/wsp/blob/ec9cc7a1921d2f9a753f0019ac21ab3e3d7abd0e/server/server.go#L254)):

```go
func (server *Server) register(w http.ResponseWriter, r *http.Request) {
	secretKey := r.Header.Get("X-SECRET-KEY")
	if secretKey != server.Config.SecretKey {
		wsp.ProxyErrorf(w, "Invalid X-SECRET-KEY")
		return
	}

	ws, err := server.upgrader.Upgrade(w, r, nil)
	if err != nil {
		wsp.ProxyErrorf(w, "HTTP upgrade error : %v", err)
		return
	}

	// (omit)
}
```

There are several knowledges to implement WebSocket server, so I'll explain it line by line.

## Authentication of WebSocket

The first line is:

```go
secretKey := r.Header.Get("X-SECRET-KEY")
if secretKey != server.Config.SecretKey {
	wsp.ProxyErrorf(w, "Invalid X-SECRET-KEY")
	return
}
```

The WebSocket protocol does not handle authorization or authentication. However, it is a problem that anyone can access the WebSocket server without authentication just because the protocol is not prepared. 

Business requirements often require authenticating the connecting WebSocket client and associating it with a registered customer record.

There are several patterns to implement authentication, and [Heroku Dev Center - WebSocket Security](https://devcenter.heroku.com/articles/websocket-security) describes one pattern - "ticket" -based authentication system.

1. The client-side contacts the server-side to obtain an authorization "ticket".
2. The server stores the ticket and returns it to the client
3. The client opens the WebSocket connection, and sends along this "ticket" as part of an initial handshake.
4. The server can compare this ticket, check an user's condition (i.e. source IPs)

See Authorization and IPs chapter in Armin Ronacher's post [Websockets 101](https://lucumr.pocoo.org/2012/9/24/websockets-101/) for detail.

There are several options for where and how to tell the "ticket" from the client to the server. A commonly used method for simple authentication is to set the relevant ticket in the HTTP header. For example, [Amazon API Gateway](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-api-key-source.html) provides the option to require API clients to pass the API key as the `X-API-Key` header.

The sample code implements a similar concept. It validates the value in the header `X-SECRET-KEY` at initial handshake.

```go
secretKey := r.Header.Get("X-SECRET-KEY")
```

Also, depending on your business requirements, there are several options for how to issue the "ticket". A idea is to issue "ticket" (or access key, or access token) in advance on their administration UI. Or if the process of asking where the WebSocket server to connect contains before handshake, it is also good idea to issue a one-time "ticket" by "access token" which is issued in advance. 

In "ticket" -based authentication system, the server authenticates only at initial handshake, but there is also an idea to authenticate every message. See Klee Thomas's post [Authenticating Websockets](https://dev.to/kleeut/authenticating-websockets-21lo) more detail.

## Handshake (HTTP Upgrade)

After authentication, upgrade an HTTP connection to an WebSocket connection.

```go
ws, err := server.upgrader.Upgrade(w, r, nil)
if err != nil {
	wsp.ProxyErrorf(w, "HTTP upgrade error : %v", err)
	return
}
```

[websocket.Upgrader.Upgrade](https://pkg.go.dev/github.com/gorilla/websocket#Upgrader.Upgrade) function does it. To put it simply, it checks if the handshake request from the client is valid and returns a successful handshake response if it is.

i.e. a valid handshake request

    GET /register HTTP/1.1
    Upgrade: websocket
    Connection: Upgrade
    Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
    Origin: http://example.com
    Sec-WebSocket-Version: 13

i.e. a successful handshake response

    HTTP/1.1 101 Switching Protocols
    Upgrade: websocket
    Connection: Upgrade
    Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=

If you want to know the internal implementation of `Upgrader.Upgrade` function, see [Learn WebSocket handshake protocol with gorilla/websocket server](https://dev.to/hgsgtk/learn-websocket-handshake-protocol-with-gorillawebsocket-server-10k9).

## Conclusion

This post explained how to start a WebSocket server and WebSocket specification and security considerations to establish a WebSocket connection.

I will explain the rest points in part 2 and beyond.

- Establish a WebSocket connection ([Part 2](https://dev.to/hgsgtk/reverse-http-proxy-over-websocket-in-go-part-2-o4a))

{% link https://dev.to/hgsgtk/reverse-http-proxy-over-websocket-in-go-part-2-o4a %}

- Keep a established connection
- Relay TCP connection from "App" to the peer of WebSocket
- Relay TCP connection in WebSocket data to "internal API"