Using Nginx as a proxy to multiple Unix sockets
Tags: nginx, go, unixsocket
Published: 2022-08-08 08:59:00

TL;DR _Listening port may be a contended resource on a busy shared machine, unix sockets are virtually unlimited. Nginx can expose them with a single port and prefixed URLs._

In some situations you may want to run many (instances of) applications on a single machine. Each instance may need to provide internal information (e.g. Prometheus `/metrics`, profiling/debug handlers) over restricted HTTP.

When number of instances grows it becomes a burden to provision listening ports without conflicts. In contrast, using Unix sockets allows for more transparency (readable filenames) and scalability (easy to come up with unique name).

Here is a small demo program written in Go that would serve trivial HTTP service with Unix socket.

```go
package main

import (
	"context"
	"flag"
	"io/fs"
	"log"
	"net"
	"net/http"
	"os"
	"os/signal"
)

func main() {
	var socketPath string

	flag.StringVar(&socketPath, "socket", "./soc1", "Path to unix socket.")
	flag.Parse()

	if socketPath == "" {
		flag.Usage()
		return
	}

	listener, err := net.Listen("unix", socketPath)
	if err != nil {
		log.Println(err.Error())
		return
	}

	// By default, unix socket would only be available to same user.
	// If we want access it from Nginx, we need to loosen permissions.
	err = os.Chmod(socketPath, fs.ModePerm)
	if err != nil {
		log.Println(err)
		return
	}

	httpServer := http.Server{
		Handler: http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
			log.Println(request.URL.String())
			if _, err := writer.Write([]byte(request.URL.String())); err != nil {
				log.Println(err.Error())
			}
		}),
	}

	// Setting up graceful shutdown to clean up Unix socket.
	go func() {
		sigint := make(chan os.Signal, 1)
		signal.Notify(sigint, os.Interrupt)
		<-sigint
		if err := httpServer.Shutdown(context.Background()); err != nil {
			log.Printf("HTTP Server Shutdown Error: %v", err)
		}
	}()

	log.Printf("Service is listening on socket file %s", socketPath)
	err = httpServer.Serve(listener)
	if err != nil {
		log.Println(err.Error())
		return
	}
}
```

Now let's run a couple of instances in separate shells.

```
./soc -socket /home/ubuntu/soc1
```

```
./soc -socket /home/ubuntu/soc2
```

Here is a minimal Nginx config to serve those instances with URL prefixes. It would receive `http://my-host/soc1/foo/bar`, strip path prefix `/soc1` and pass `/foo/bar` to `soc1`.

```
server {
    listen 80 default;

    location /soc1/ {
        proxy_pass http://soc1/;
    }
    location /soc2/ {
        proxy_pass http://soc2/;
    }
}

upstream soc1 {
    server unix:/home/ubuntu/soc1;
}

upstream soc2 {
    server unix:/home/ubuntu/soc2;
}
```

Every Unix socket is defined as `upstream` and has `/location` statement in `server`.

It is also possible to use Unix sockets directly in `/location`, like in

```
    location /soc1/ {
        proxy_pass http://unix:/home/ubuntu/soc1;
    }
```
however it has an unwanted limitation that you can not add trailing `/` to `proxy_pass`. And this means that URL will be passed as is, e.g. `soc1` will receive `/soc1/foo` instead of `/foo`.

To avoid such limitation we can use named upstream and add trailing `/` to `proxy_pass`.
```
    location /soc1/ {
        proxy_pass http://soc1/; # Mind trailing "/".
    }
```