Reverse HTTP proxy over WebSocket in Go (Part 3)
Tags: go, webdev, websocket
Published: 2021-12-16 08:09:13

## Series introduction

In my [previous post](https://dev.to/hgsgtk/reverse-http-proxy-over-websocket-in-go-part-2-o4a) I talked about how to establish a WebSocket connection in Go.

{% link https://dev.to/hgsgtk/reverse-http-proxy-over-websocket-in-go-part-2-o4a %}

In this post, I will be starting to talk about how to relay TCP connection from "App" to the peer of WebSocket in Go.

- Start a WebSocket server ([Part 1](https://dev.to/hgsgtk/reverse-http-proxy-over-websocket-in-go-part-1-13n4))
- Establish a WebSocket connection ([Part 2](https://dev.to/hgsgtk/reverse-http-proxy-over-websocket-in-go-part-2-o4a))
- Relay TCP connection from "App" to the peer of WebSocket ([Part 3](https://dev.to/hgsgtk/reverse-http-proxy-over-websocket-in-go-part-3-2ggd) | Part 4)
- Relay TCP connection in WebSocket data to "internal API"
- Keep a established connection

## Reverse HTTP proxy over WebSocket

A reverse HTTP proxy over WebSocket is a type of proxies, which retrieves resources on behalf on a client from servers and uses the WebSocket protocol as a "tunnel" to pass TCP communication from server to client.

![A network diagram for reverse proxy over WebSocket](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iupaoeq283lyeqpkg049.png)

I'll use [root-gg/wsp](https://github.com/root-gg/wsp) as a sample code to explain it. I'll use the forked code to explain because maintenance has stopped and the Go language and libraries version needed to be updated.

{% github hgsgtk/wsp no-readme %}

## Relay TCP connection to the peer of WebSocket

In the [part 1](https://dev.to/hgsgtk/reverse-http-proxy-over-websocket-in-go-part-1-13n4), I presented a demonstration.

![A terminal image when sending a HTTP request from app](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nxdm25yw7vsxg3tzpdhz.png)

HTTP communication is relayed by the following route.

    app -[1]-> wsp server -[2](WebSocket)-> wsp client -> internal API

As a pre requirement, a WebSocket connection has already been established between wsp server and wsp client before relaying. 

The way to establish WebSocket connection is explained in [part 2](https://dev.to/hgsgtk/reverse-http-proxy-over-websocket-in-go-part-2-o4a), but next we need to pool the connection on the server side and use it when relaying.

These flow are divided into three parts to explain it.

1. Receive requests to be proxied (`[1]` in the relay flow)
2. Pool the WebSocket connection on the server for relaying
3. Relay TCP connection to the peer WebSocket (`[2]` in the relay flow)

This post describes the 1st and 2nd flow. In part 4, I'll explain the 3rd flow.

## 1. Receive requests to be proxied

To receive requests to be proxied to "internal API" finally, there are mainly two kinds of API interfaces.

1. Expose an endpoint (i.e. `/requests`) that accept HTTP requests
2. Work as a HTTP proxy used by "app" (i.e. `curl -x {wsp server's address} http://localhost:8081`)

In the demo example, wsp server (WebSocket server) chose 1st option and exposes the endpoint whose path is `/requests`.

```bash
$ curl -H 'X-PROXY-DESTINATION: http://localhost:8081/hello' http://127.0.0.1:8080/request
```

Let's read the HTTP handler code in Go ([code](https://github.com/hgsgtk/wsp/blob/8d7d7b4eb6aa700950c41756608177514b935507/server/server.go#L189)), which waits the request to `/requests/`.

```go
func (s *Server) request(w http.ResponseWriter, r *http.Request) {
	// Parse destination URL
	dstURL := r.Header.Get("X-PROXY-DESTINATION")
	if dstURL == "" {
		wsp.ProxyErrorf(w, "Missing X-PROXY-DESTINATION header")
		return
	}
	URL, err := url.Parse(dstURL)
	if err != nil {
		wsp.ProxyErrorf(w, "Unable to parse X-PROXY-DESTINATION header")
		return
	}
	r.URL = URL

	// (omit)
}
```

wsp defined the custom HTTP header `X-PROXY-DESTINATION` to specify the final destination URL. For example, when you want to access http://localhost:8081/hello, add a HTTP header `X-PROXY-DESTINATION: http://localhost:8081/hello` in your request.

## 2. Pool the WebSocket connection on the server for relaying

We need to make sure that the http handler can recognize and use the connected WebSocket connections.

In this code, an HTTP server that exposes an endpoint `/request` is set up at a given address. The handler mapped to `/request` is as follow ([code](https://github.com/hgsgtk/wsp/blob/59a39b293b70d916023c35e8af880dc8fe98bcc2/server/server.go#L242)):

```go
func (s *Server) Register(w http.ResponseWriter, r *http.Request) {
	// - 1. Upgrade a received HTTP request to a WebSocket connection
	// (omit)
	// - 2. Wait a greeting message from the peer and parse it
	// (omit)

	// 3. Register the connection into server pools.
	// s.lock is for exclusive control of pools operation.
	s.lock.Lock()
	defer s.lock.Unlock()

	var pool *Pool
	// There is no need to create a new pool,
	// if it is already registered in current pools.
	for _, p := range s.pools {
		if p.id == id {
			pool = p
			break
		}
	}
	if pool == nil {
		pool = NewPool(s, id)
		s.pools = append(s.pools, pool)
	}
	// update pool size
	pool.size = size

	// Add the WebSocket connection to the pool
	pool.Register(ws)
}
```

First, the server needs to have a WebSocket connection pooled so that it can be used for relay. One design idea to achieve this is to prepare three entities `Server`, `Pool`, and `Connection` as shown below.

![A Miro image describing the relationship map](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b2ow33vm886u9cg7svxz.png)
 
In this code, the [Server struct](https://github.com/hgsgtk/wsp/blob/59a39b293b70d916023c35e8af880dc8fe98bcc2/server/server.go#L21) have a field `pools` which is a pointer of the slice of Pool struct.

```go
type Server struct {
	// (omit)

	pools []*Pool

	// (omit)
}
```

The [Pool struct](https://github.com/hgsgtk/wsp/blob/36c3c8bd07d3e61af25f4f76e7ccd2216ab29ea8/server/pool.go#L12) represents the connection from ws client. The definition is as follow:

```go
type Pool struct {
	server *Server
	id     PoolID

	size int

	connections []*Connection
	idle        chan *Connection

	done bool
	lock sync.RWMutex
}
```

The [Connection struct](https://github.com/hgsgtk/wsp/blob/50cee519e566bcb0fda34c4f8eec2246bf596d88/server/connection.go#L30) manages a single WebSocket connection because wsp supports multiple connections from a single wsp client at the same time.

```go
type Connection struct {
	pool         *Pool
	ws           *websocket.Conn
	status       ConnectionStatus
	idleSince    time.Time
	lock         sync.Mutex
	nextResponse chan chan io.Reader
}
```

Then, going back to the handler code, we can see that the server checks whether requesting connection is already registered, and create a new pool if it's not registered in current pools.

```go
func (s *Server) Register(w http.ResponseWriter, r *http.Request) {
	// (omit)

	var pool *Pool
	// There is no need to create a new pool,
	// if it is already registered in current pools.
	for _, p := range s.pools {
		if p.id == id {
			pool = p
			break
		}
	}
	if pool == nil {
		pool = NewPool(s, id)
		s.pools = append(s.pools, pool)
	}

	// (omit)
}
```

Each ws client issues an unique ID, which is a common design when you want to pool and recognize external devices. By issuing an ID, the server is able to recognize meta information such as ws client source IP and so on.

At the end of the handler function, the server registers a new connection to the created pool.

```go
func (s *Server) Register(w http.ResponseWriter, r *http.Request) {
	// (omit)

	pool.Register(ws)
}
```

[Pool.Register](https://github.com/hgsgtk/wsp/blob/50cee519e566bcb0fda34c4f8eec2246bf596d88/server/pool.go#L38) is as follow:

```go
func (pool *Pool) Register(ws *websocket.Conn) {
	// (omit)

	log.Printf("Registering new connection from %s", pool.id)
	connection := NewConnection(pool, ws)
	pool.connections = append(pool.connections, connection)
}
```

This handler process registered an idle connection that can be used for relay. [NewConnection()](https://github.com/hgsgtk/wsp/blob/f14b5f4d2f9172a2b15e8156f8376070eaa56747/server/connection.go) function mark a new connection as a usable connection for relay in the pool and start a thread (technically, start a new [goroutine](https://go.dev/tour/concurrency/1) running) that keeps reading messages over the connected WebSocket connection from the wsp client.

```go
func NewConnection(pool *Pool, ws *websocket.Conn) *Connection {
	// Initialize a new Connection
	c := new(Connection)
	c.pool = pool
	c.ws = ws
	c.nextResponse = make(chan chan io.Reader)
	c.status = Idle

	// Mark that this connection is ready to use for relay
	c.Release()

	// Start to listen to incoming messages over the WebSocket connection
	go c.read()

	return c
}
```

At initialization, a new Connection is created by state `Idle`.

## Enum in Go

The status `Idle` means it is opened but not working now. Here is a state diagram describing the behavior of `Connection`.

![A state diagram of Connection](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iroqa3b17nb6psu2meg6.png)

In this repository, I made the defined type [ConnectionStatus](https://github.com/hgsgtk/wsp/blob/ea4902a8e11f820268e52a6245092728efeffd7f/server/connection.go#L19) and constants `Idle`, `Busy`, and `Closed`.

```go
// ConnectionStatus is an enumeration type which represents the status of WebSocket connection.
type ConnectionStatus int

const (
	// Idle state means it is opened but not working now.
	// The default value for Connection is Idle, so it is ok to use zero-value(int: 0) for Idle status.
	Idle ConnectionStatus = iota
	Busy
	Closed
)
```

By the way, if you want not to define ConnectionStatus with [Zero-Value](https://go.dev/ref/spec#The_zero_value), you can skip the zero-value as follows:

```go
// Pattern1: Assign zero-value to blank
const (
	_ State = iota 
	A = iota
	B
)

// Pattern2: Start from 1
const (
	C State = iota + 1
	D
)
```

## Conclusion

This post explained how to relay TCP connection from "App" to the peer of WebSocket, especially implementation to receive requests to be proxied and to pool the WebSocket connection on the server for relaying.

I will explain the rest points in part 3 and beyond.

- Relay TCP connection from "App" to the peer of WebSocket (rest)
- Relay TCP connection in WebSocket data to "internal API"
- Keep a established connection