[Go] Understanding net/http - How to run a server like a Pro?
Tags: go, webdev, beginners
Published: 2024-03-01 17:23:07

*image credit to [Renee French](https://reneefrench.blogspot.com/), the [source is the one of the official Go blog posts](https://go.dev/blog/gopher)

# TL; DR
- Use `ServeMux` as the multiplexer
- Use `Serve` struct for detailed configurations
- It’s okay with using `Serve::ListenAndServe`, but `Listen` and `Serve` can be called separately(`Listen` is in the [net](https://pkg.go.dev/net) package)

# Introduction - Stop relying on default functions
At a glance, It looks simple to write an HTTP server in Go, using the standard [net/http](https://pkg.go.dev/net/http) package. Many of the tutorials and blog posts usually don’t go beyond using default functions like `http.HandleFunc` or `http.ListenAndServe`.  And there don’t seems to be many issues with it. Who cares as long as it works well?

Well, if you’re a professional software engineer, you would highly disagree with this. We need to take control over every aspect of what we’re wielding as our mighty weapon, if possible. Relying on default functions provided by the package may obscure the details under the hood. As I spend more time learning Go, I wanted to know what’s behind this “default layer”, and be able to write an HTTP server in a more sophisticated way. 

In this short article, I want to discuss the ways of starting a server in net/http package, from simple ones to more advanced ones. Before diving into the main part of the article, let me point out the following three points:
- This article is motivated by a [blog post by an engineer at Grafana](https://grafana.com/blog/2024/02/09/how-i-write-http-services-in-go-after-13-years/), which has been quite popular in the Go community recently.
- I won’t consider any of the third-party libraries like [Gin](https://gin-gonic.com/) or [Echo](https://echo.labstack.com/). It is not because they are not good, but because I want to dig into what is considered as “standard” in Go language and by its developers. 
- Also, it could mean so many things when I say “How to write a server in Go”, because there are a myriads of things to consider - middleware, router, security, network protocols, concurrency issues, to name a few. Therefore, this article only aims to explore how to write and run a "server instance" rather a "web server application".  

# Various ways to run a server using net/http package

## 1. Start simple - `DefaultServeMux`

Many learning materials for beginners in Go(such as [Go by Example](https://gobyexample.com/http-server) or [Go Web Examples](https://gowebexamples.com/http-server) shows how to run an HTTP server in a simple example like:

```go
package main

import (
	"fmt"
	"log"
	"net/http"
)

func main() {
	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(200)
		fmt.Fprint(w, "Hello, Go HTTP Server!")
	})

	log.Fatal(http.ListenAndServe(":8000", nil))
}
```

In a nutshell, this method simply registers a plain function of two parameters `(w http.ResponseWriter, r *http.Request)` such that the function is called whenever a client requests the targeted endpoint. In this way, there is no need to fully understand the interface `Handler`; you just need to write logic to process incoming requests.

However, there is an interesting point to look at here. What is the relationship between `http.HandleFunc` and `http.ListenAndServe`, as these two functions have nothing in common. But, as a matter of fact, they **DO** share a global variable called `DefaultServeMux`.

If you read the [documentation](https://pkg.go.dev/net/http), it says

for `HandlerFunc`:
 
> HandleFunc registers the handler function for the given pattern in DefaultServeMux

and for `ListenAndServe`:

> The handler is typically nil, in which case DefaultServeMux is used.

Thus, a function registered by `HandleFunc` is stored in the global variable `DefaultServeMux` of the package, which is then used by `ListenAndServe`. So the key now is to understand `ServeMux` in general.

### (**REMARK**) What is `ServeMux`?

Let’s look at the documentation once again:

> ServeMux is an HTTP request multiplexer. It matches the URL of each incoming request against a list of registered patterns and calls the handler for the pattern that most closely matches the URL.

As a [multiplexer](https://en.wikipedia.org/wiki/Multiplexing), a `ServeMux` instance receives multiple HTTP requests and hands each of them over to an appropriate handler. Thus, `ListenAndServe` starts a server that **listens** to requests headed to the specified address(in this case, it is "http://localhost:8000"), and then delegates them to its `(Default)ServeMux` in order to handle(note that `ServeMux` implements the interface `http.Handler`). 

## 2. More sophisticated usage - make your own `ServeMux`

But why is it not a good practice to use `DefaultServeMux`? Mainly because it is a global object. According to [Practical Go by Amit Saha](https://www.amazon.com/Practical-Go-Building-Non-Network-Applications/dp/1119773814), many unnoticed packages could also access the object, such that related security or concurrency problems could occur. Therefore, it is better to create our own `ServeMux` instances using `http.NewServeMux`.

```go
func main() {
	mux := http.NewServeMux()
	
	mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(200)
		fmt.Fprint(w, "Hello, Go HTTP Server!")
	})

	log.Fatal(http.ListenAndServe(":8000", mux))
}	
```

Of course, since `ListenAndServe` accepts the `http.Handler` interface in general, we can wrap `mux` with middlewares(the technique is introduced in [one of my previous article](https://dev.to/uponthesky/go-understanding-nethttp-package-handler-family-2422))

```go
func someMiddleware(h http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
  		// custom logic
		h.ServeHTTP(w, r)
		// custom logic
	}

func main() {
    // …
    mux := http.NewServeMux()
    mux = someMiddleware(mux)
    // …
}
```

# 3. Writing your own `Server`

But `ServeMux` is only a multiplexer that distributes incoming requests, and the server is bootstrapped at `http.ListenAndServe`.  Obviously, it doesn’t accept any configuration for the server application it sets up. Therefore, if we need to tweak a few configurations for a server, we use [`http.Server` struct](https://pkg.go.dev/net/http#hdr-Servers) in the package. 

As you see the [documentation](https://pkg.go.dev/net/http#Server), the configurable variables are related to fairly low-level concepts. But if we’re considering a production environment, complicated configuration is inevitable.

Now we have more freedom in writing a web server in Go, only using the net/http package:

```go
func main() {
	addr := os.Getenv("SERVER_ADDRESS")

	if addr == "" {
   		addr = ":8080"
       }
	
	// step 1: make your own `ServeMux`
        mux := http.NewServeMux()

        // add handlers & middlewares
 	// …

	// step 2: make your own `Server`
	srv := &http.Server{Addr: addr, Handler: mux, ReadHeaderTimeout: time.Minute}

	log.Fatal(srv.ListenAndServe())
}
```

Also, we may go deeper into setting up the TLS support using `ListenAndServeTLS` instead of plain `ListenAndServe`, with necessary certificate files. For simplicity, we will keep using `ListenAndServe` for the rest of the article.

## 4. Advanced - Splitting `ListenAndServe` into `Listen` and `Serve`
Many of the learning materials and blog posts on writing Go server with net/http simply use the `ListenAndServe` method. In most of the cases it is fine 
since it uses [TCP connections under the hood](https://pkg.go.dev/net/http#Server.ListenAndServe). 

So we break down the function to `Listen` and `Serve`, instead of the last line `log.Fatal(s.ListenAndServe())` in the above code chunk like this:

```go
// step 3: Listen to TCP connections
ln, err := net.Listen("tcp", addr)

if err != nil {
  log.Fatal(err)
} 

defer ln.Close()

log.Fatal(srv.Serve(ln))
```

As you might have already expected, `Listen` function is a wrapper of the [system call `listen` at the OS kernel level](https://cs.opensource.google/go/go/+/master:src/syscall/zsyscall_linux_amd64.go;drc=45b641ce15159e29fa4494b837493042d1e10384;l=1212). And `Serve` [reads request data from the socket buffer](https://cs.opensource.google/go/go/+/refs/tags/go1.22.0:src/net/http/request.go;l=1051) and [hands over the information(with pre-generated response instance) to its handler](https://cs.opensource.google/go/go/+/refs/tags/go1.22.0:src/net/http/server.go;l=2039). We can go deeper here but it is beyond the scope of this post. However, savor the beauty of the naming "Listen"(the incoming connection from a socket) and "Serve"(the request from the listened connection with enrolled handler) here - how well they are made!

# Conclusion
In effect, we have break down "convenient" default functions in the net/http package into several components that we can configure directly. As we go deeper, it goes down to the fundamental part of the network programming - TCP socket programming, as we discussed at [Advanced - Splitting `ListenAndServe` into `Listen` and `Serve`](#4-advanced---splitting-listenandserve-into-listen-and-serve). I am not sure whether we need to configure at this level. Nevertheless, I believe it is important to understand how the mechanism works when we write a server program. 

Still, it is of course not writing a server instance "as a Pro". There are more details uncovered yet in this post. But "like a Pro", we mimic the way a professional software engineer writes code. 

# references
- Go by Example: https://gobyexample.com/http-server
- Go Web Examples: https://gowebexamples.com/http-server
- net/http package: https://pkg.go.dev/net/http
- Learn Go with Tests: https://quii.gitbook.io/learn-go-with-tests/build-an-application/http-server
- "How I write HTTP services in Go after 13 years" by Mat Ryer: https://grafana.com/blog/2024/02/09/how-i-write-http-services-in-go-after-13-years/ 
- Network Programming with Go by Adam Woodcock, Ch9: https://www.amazon.com/Network-Programming-Go-Adam-Woodbeck/dp/1718500882
- Practical Go by Amit Saha, Ch5~7: https://www.amazon.com/Practical-Go-Building-Non-Network-Applications/dp/1119773814