Serving Plug: Building an Elixir HTTP server from scratch
Tags: elixir, programming
Published: 2019-01-22 14:06:38

---
title: Serving Plug: Building an Elixir HTTP server from scratch
layout: layout_elixir_alchemy
description: It’s time for another Elixir Alchemy! We'll learn by building today. We're implementing an HTTP server with a Plug adapter, which parses requests and builds responses so it can run Plug apps.
cover_image: https://blog.appsignal.com/images/blog/2019-01/elixir-plug-octopus.png
tags: elixir, programming
published: true
canonical_url: https://blog.appsignal.com/2019/01/22/serving-plug-building-an-elixir-http-server.html
---

Welcome back to another edition of Elixir Alchemy! In our continued quest to find out what’s happening under the hood, we'll take a deep dive into HTTP servers in Elixir.

To understand how HTTP servers work, we'll implement a minimal example of one that can run a Plug application. We’ll learn about decoding requests and encoding responses, as well as how Plug interacts with the web server by building a minimal subset of an HTTP server that can accept HTTP requests and run a Plug application.

-> The HTTP server we’re building is for educational purposes. We won’t build a production-ready HTTP server. If you're looking for one that is, please try [cowboy](https://github.com/ninenines/cowboy), which is the default choice of HTTP server in Elixir applications.

## HTTP over TCP

We’ll start with the fundamentals. HTTP is a protocol that commonly uses [TCP](https://en.wikipedia.org/wiki/Transmission_Control_Protocol) to transport requests and responses between an HTTP client—like a web browser—and a web server.

Erlang provides the `:gen_tcp` module that can be used to start a TCP socket that receives and transmits data. We'll use that library as the basis of our server.

```elixir
# lib/http.ex
defmodule Http do
  require Logger

  def start_link(port: port) do
    {:ok, socket} = :gen_tcp.listen(port, active: false, packet: :http_bin, reuseaddr: true)
    Logger.info("Accepting connections on port #{port}")

    {:ok, spawn_link(Http, :accept, [socket])}
  end

  def accept(socket) do
    {:ok, request} = :gen_tcp.accept(socket)

    spawn(fn ->
      body = "Hello world! The time is #{Time.to_string(Time.utc_now())}"

      response = """
      HTTP/1.1 200\r
      Content-Type: text/html\r
      Content-Length: #{byte_size(body)}\r
      \r
      #{body}
      """

      send_response(request, response)
    end)

    accept(socket)
  end

  def send_response(socket, response) do
    :gen_tcp.send(socket, response)
    :gen_tcp.close(socket)
  end

  def child_spec(opts) do
    %{id: Http, start: {Http, :start_link, [opts]}}
  end
end
```

In this example, we’ve created a TCP server that responds to every request with the current time in an HTTP response.

We start a socket to listen to the passed in port in `start_link/1`, and we'll spawn `accept/1` in a new process that waits until a request comes in over the socket by calling `:gen_tcp.accept/1`.

When it does, we put it into the `request` variable and create a response to send to the client. In this case, we'll be sending a response that shows the current time.

>  <h3>Building HTTP responses</h3>
>
>  <p><pre><code>HTTP/1.1 200\r\nContent-Type: text/html\r\n\r\n\Hello world! The time is 14:45:49.058045</code></pre></p>
>
>  <p>An HTTP response contains a couple of parts:</p>
>
>  <ul>
>    <li>A status line, with the protocol version (<code>HTTP/1.1</code>) and a response code (<code>200</code>)</li>
>    <li>A <a href="https://en.wikipedia.org/wiki/Carriage_return">carriage return</a>, followed by a <a href="https://en.wikipedia.org/wiki/Newline">line feed</a> (<code>\r\n</code>) to split the status line from the rest of the response</li>
>    <li>(Optional) header lines (<code>Content-Type: text/html</code>), separated by CRLFs</li>
>    <li>A double CRLF, to separate the headers from the response body</li>
>    <li>The response body that will be shown in the browser (<code>Hello world! The time is 14:45:49.058045</code>)</li>
>  </ul>

We pass the body we built to `send_response/2`, which takes care of sending the response over the socket and finally closing the socket connection.

We spawn a process for each request so the server can call `accept/1` again to accept new requests. This way, we can respond to requests in parallel, instead of subsequent requests having to wait for previous ones to complete being processed.

## Running the Server

Let's try it out. We'll use a [supervisor](https://blog.appsignal.com/2017/08/10/elixir-alchemy-supervisors-building-fault-tolerant-elixir-applications.html) to run our HTTP server to make sure it's restarted immediately when if it fails.

Our server implementation has a `child_spec/1` function that specifies how it should be started. It states that it should call the `Http.start/1` function with the passed options, which will return the newly spawned process' ID.

```elixir
# lib/http.ex
defmodule Http do
  # ...

  def child_spec(opts) do
    %{id: Http, start: {Http, :start_link, [opts]}}
  end
end
```

Because of that, we can add it to the list of children managed by the supervisor in `lib/http/application.ex`.

```elixir
# lib/http/application.ex
defmodule Http.Application do
  @moduledoc false
  use Application

  def start(_type, _args) do
    children = [
      {Http, port: 8080}
    ]

    opts = [strategy: :one_for_one, name: Http.Supervisor]
    Supervisor.start_link(children, opts)
  end
end
```

We pass `{Http, port: 8080}` as one of our supervisor's children to start the server at port 8080 when the application is started.

```
$ mix run --no-halt
19:39:29.454 [info]  Accepting connections on port 8080
```

If we start the server and use our browser to send a request, we can see that it indeed returns the current time.

![A screenshot of our Elixir HTTP server displaying the current time.](https://blog.appsignal.com/images/blog/2019-01/http_1.png)

## Plug

Now that we know how a web server works, let's take it to the next level. Our current implementation has the response hard-coded into the server. To allow our web server to run different apps, we'll move the app out into a separate [Plug](https://github.com/elixir-plug/plug) module.

```elixir
# lib/current_time.ex
defmodule CurrentTime do
  import Plug.Conn

  def init(options), do: options

  def call(conn, _opts) do
    conn
    |> put_resp_content_type("text/html")
    |> send_resp(200, "Hello world! The time is #{Time.to_string(Time.utc_now())}")
  end
end
```

The `CurrentTime` module defines a `call/2` function that takes the passed in `%Plug.Conn` struct. It then sets the response content type to "text/html" before sending the "Hello world!" message—together with the current time—back as a response.

Our new module behaves the same as the web server example, but it's detached from the web server. Because of Plug, we could swap out servers without having to change the application's code, and we can also change the application without having to touch the web server.

### Writing a Plug Adapter

To make sure our web server can communicate with our web application, we need to build a `%Plug.Conn{}` struct to pass to `CurrentTime.call/2`. We'll also need to turn the sent response into a string that our web server can send back over the socket.

To do this, we'll create an adapter that handles the communication between our Plug app and our web server.

```elixir
# lib/http/adapter.ex
defmodule Http.PlugAdapter do
  def dispatch(request, plug) do
    %Plug.Conn{
      adapter: {Http.PlugAdapter, request},
      owner: self()
    }
    |> plug.call([])
  end

  def send_resp(socket, status, headers, body) do
    response = "HTTP/1.1 #{status}\r\n#{headers(headers)}\r\n#{body}"

    Http.send_response(socket, response)
    {:ok, nil, socket}
  end

  def child_spec(plug: plug, port: port) do
    Http.child_spec(port: port, dispatch: &dispatch(&1, plug))
  end

  defp headers(headers) do
    Enum.reduce(headers, "", fn {key, value}, acc ->
      acc <> key <> ": " <> value <> "\n\r"
    end)
  end
end
```

Instead of responding directly from `Http.accept/2`, we'll use our adapter's `dispatch/2` function to build a `%Plug.Conn{}` struct and pass that to our plug's `call/2` function.

In the `%Plug.Conn{}`, we'll set the `:adapter` to link to our Adapter module, and then pass the socket the response that should be sent over. This ensures that the Plug app knows which module to call `send_resp/4` on.

Our adapter's `send_resp/4` takes the socket connection, the response status, a list of headers and a body, which are all prepared by the Plug application. It uses the passed in arguments to build the response and calls out to `Http.send_response/2` that we've implemented before.

The `child_spec/1` for our adapter returns the `child_spec/1` for the `Http` module. This causes the web server to start when we supervise our adapter. We'll pass the dispatch function as the dispatch so that it can be called by our web server when it receives a response.

```elixir
# lib/http/application.ex
defmodule Http.Application do
  @moduledoc false
  use Application

  def start(_type, _args) do
    children = [
      {Http.PlugAdapter, plug: CurrentTime, port: 8080}
    ]

    opts = [strategy: :one_for_one, name: Http.Supervisor]
    Supervisor.start_link(children, opts)
  end
end
```

Instead of starting `Http` in our application module, we'll start `Http.PlugAdapter`, which will take care of setting the plug, preparing the dispatch function and starting the web server.

```elixir
# lib/http.ex
defmodule Http do
  require Logger

  def start_link(port: port, dispatch: dispatch) do
    {:ok, socket} = :gen_tcp.listen(port, active: false, packet: :http_bin, reuseaddr: true)
    Logger.info("Accepting connections on port #{port}")

    {:ok, spawn_link(Http, :accept, [socket, dispatch])}
  end

  def accept(socket, dispatch) do
    {:ok, request} = :gen_tcp.accept(socket)

    spawn(fn ->
      dispatch.(request)
    end)

    accept(socket, dispatch)
  end

  # ...
end
```

Since we now handle requests in our Plug, we can remove most of the code in `Http.accept/2`. The `Http.start_link/2` function will now receive the dispatch function from the adapter, which is used to send the request to in `Http.accept/2`.

```
$ mix run --no-halt
19:39:29.454 [info]  Accepting connections on port 8080
```

Running the server again, everything still works exactly as before. However, our HTTP server, web application and Plug adapter are now three separate modules.

![A screenshot of our Elixir HTTP server displaying the current time.](https://blog.appsignal.com/images/blog/2019-01/http_2.png)

## Swapping out Plug Applications

Because our server is now separate from our adapter and web application, we can swap the Plug out to run another application on the server. Let's give that a shot.

```elixir
# mix.exs
defmodule Http.MixProject do
  # ...

  defp deps do
    [
      {:plug_octopus, github: "jeffkreeftmeijer/plug_octopus"},
      {:plug, "~> 1.7"}
    ]
  end
end
```

In our `mix.exs` file, we add `:plug_octopus` as a dependency.

```elixir
# lib/http/application.ex
defmodule Http.Application do
  @moduledoc false
  use Application

  def start(_type, _args) do
    children = [
      {Http.PlugAdapter, plug: Plug.Octopus, port: 8080}
    ]

    opts = [strategy: :one_for_one, name: Http.Supervisor]
    Supervisor.start_link(children, opts)
  end
end
```

We then swap `CurrentTime` for `Plug.Octopus` in our `Http.Application` module. Starting the server and visiting <http://localhost:8080> now shows an octopus!

![A screenshot of our Elixir HTTP server running Plug.Octopus.](https://blog.appsignal.com/images/blog/2019-01/octopus_1.png)

However, clicking the *flip!* and *crash!* buttons doesn't do anything and any URL that's called shows the same page. That's because we skipped over parsing the requests altogether. Since we don't read the requests, we'll always pass the same response back. Let's fix that.

## Parsing Requests

To read requests, we'll need to read the response from the socket. Thanks to the `http_bin` option that we're passing when calling `:gen_tcp.listen/2`, the request is returned in a format we can pattern match on.

```elixir
# lib/http.ex
defmodule Http do
  # ...

  def read_request(request, acc \\ %{headers: []}) do
    case :gen_tcp.recv(request, 0) do
      {:ok, {:http_request, :GET, {:abs_path, full_path}, _}} ->
        read_request(request, Map.put(acc, :full_path, full_path))

      {:ok, :http_eoh} ->
        acc

      {:ok, {:http_header, _, key, _, value}} ->
        read_request(
          request,
          Map.put(acc, :headers, [{String.downcase(to_string(key)), value} | acc.headers])
        )

      {:ok, line} ->
        read_request(request, acc)
    end
  end

  # ...
end
```

The `Http.read_request/2` function takes a socket connection and will be called from the dispatch function. It will keep calling `:gen_tcp.recv/2` to accept lines from the request until it receives an `:http_eoh` response, indicating the end of the requests headers.

We match on the `:http_request` line, which includes the full request path. We'll use that to extract the path and URL parameters later. We'll also match on all `:http_header` lines, which we convert to a list we can pass to our Plug application later.

```elixir
# lib/http/adapter.ex
defmodule Http.PlugAdapter do
  def dispatch(request, plug) do
    %{full_path: full_path} = Http.read_request(request)

    %Plug.Conn{
      adapter: {Http.PlugAdapter, request},
      owner: self(),
      path_info: path_info(full_path),
      query_string: query_string(full_path)
    }
    |> plug.call([])
  end

  # ...

  defp headers(headers) do
    Enum.reduce(headers, "", fn {key, value}, acc ->
      acc <> key <> ": " <> value <> "\n\r"
    end)
  end

  defp path_info(full_path) do
    [path | _] = String.split(full_path, "?")
    path |> String.split("/") |> Enum.reject(&(&1 == ""))
  end

  defp query_string([_]), do: ""
  defp query_string([_, query_string]), do: query_string

  defp query_string(full_path) do
    full_path
    |> String.split("?")
    |> query_string
  end
end
```

We call `Http.read_request/1` from `Http.PlugAdapter.dispatch/2`. Having the `full_path`, we can extract the `path_info` (a list of path segments), and `query_string` (all URL parameters after the "?"). We add these to the `%Plug.Conn{}` to have our Plug app handle the rest.

Restarting the server, we can now flip and crash the lobster.

![A screenshot of our Elixir HTTP server running Plug.Octopus and accepting URL parameters.](https://blog.appsignal.com/images/blog/2019-01/octopus_2.png)

## A Minimal Web Server Example that flips and crashes

With everyone in place, and no screws on the floor, our project to look into HTTP servers in Elixir is done. We've implemented a web server that extracts data from requests and used it to send a request to a Plug application. It even has concurrency included: since each request spawns a separate process, our web server can handle multiple concurrent users.

There's more to HTTP servers than we've shown in this article, but we hope implementing one from the ground up gave you some insight into how a web server could work. 

Check out [the finished project](https://github.com/jeffkreeftmeijer/elixir-http-server) if you'd like to review the code, and don't forget to [subscribe to the mailing list](https://appsignal.com/elixir-alchemy) if you'd like to read more Elixir Alchemy!