Clustering with Phoenix 1.7
Tags: pubsub, phoenix, 2023, flyio
Published: 2023-08-08 08:17:55

There several good articles about how to setup a Phoenix cluster with PubSub messaging through the nodes but I found them to be incomplete or slightly out of date so in this article I plan to give step by step instructions on how to create a clustered Phoenix 1.7 application in 2023.

The goal of this article will be to run multiple instances of our Phoenix which seamlessly can do message passing between nodes.

The target production environment will be Fly.io for this article, but I'll also show how to setup your project for local development as well so you can make sure you are subscribing and broadcasting correctly.

## Simplest Demo Project


We're going to create a stripped down Phoenix app that does only two things:
- Exposes a live view that displays messages received
- Has an API end-point to receive external messages via HTTP


# Create project

_Note_ - we will build this project step by step but feel free to check out the [completed project here](https://github.com/byronsalty/talkie)

## Setup Phoenix app
Phoenix new with most things removed:
```
mix phx.new talkie --no-ecto --no-mailer
```

Update the `config/dev.exs` file to allow passing in a `PORT` since we'll need to run two instances:
```
# Before
config :talkie, TalkieWeb.Endpoint,
	http: [ip: {127, 0, 0, 1}, port: 4000],
	...

# After
port = String.to_integer(System.get_env("PORT") || "5000")

config :talkie, TalkieWeb.Endpoint,
	http: [ip: {127, 0, 0, 1}, port: port],
	...
```

## Create the `ping` API endpoint:
```
# Add this controller
defmodule TalkieWeb.APIController do
	use TalkieWeb, :controller
	
	def ping(conn, _params) do
		# Not doing anything but responding so far
		json(conn, %{pong: true})
	end
end
```

Add the `ping` route to your `router.ex` inside the existing `/api` scope:
```
scope "/api", TalkieWeb do
	pipe_through :api
	get "/ping", APIController, :ping
end
```

Test it:
```
curl http://127.0.0.1:5000/api/ping

# {"pong": true}
```


## Create the Liveview viewer

Add a LiveView module like this
```
defmodule TalkieWeb.ViewerLive.Index do
	use TalkieWeb, :live_view

	@impl true
	def render(assigns) do
		~H"""
		<h1>Messages</h1>
		<%= for msg <- @messages do %>
			<span><%= msg %></span>
		<% end %>
		"""
	end

	@impl true
	def mount(_params, _session, socket) do
		{:ok, assign(socket, :messages, [])}
	end

	@impl true
	def handle_info({:message, msg}, socket) do
		{:noreply, assign(socket, 
			:messages, [msg | socket.assigns.messages])}
	end
	
	@impl true
	def handle_info(
		%Phoenix.Socket.Broadcast{
			topic: "messages",
			event: "ping",
			payload: {:message, msg}
		}, socket) do
		handle_info({:message, msg}, socket)
	end
end
```

And add this line to your `router.ex`:
```
	# in the browser scope
	live "/viewer", ViewerLive.Index
```

Test it by pointing your browser at (http://127.0.0.1:5000/viewer)


# Listen for messages on One instance

Now that we have those components. Let's wire them together with a single instance.

We're going to have our Viewer listen for messages, and our API broadcast whenever the `/ping` api is hit.

You "listen" by _subscribing_ to a topic. Update the liveview mount to look like this now:
```
def mount(_params, _session, socket) do
	TalkieWeb.Endpoint.subscribe("messages")
	{:ok, assign(socket, :messages, ["test message..."])}
end
```
And you broadcast by adding the broadcast to the ping function this the API Controller:
```
def ping(conn, _params) do
	Phoenix.PubSub.broadcast!(Talkie.PubSub, "messages",
		%Phoenix.Socket.Broadcast{
			topic: "messages",
			event: "ping",
			payload: {:message, "ping"}
		}
	)
	json(conn, %{pong: true})
end
```

### Test it

Start the server:
```
PORT=5000 mix phx.server
```

Open a browser to: (http://127.0.0.1:5000/viewer)

Hit the ping api again:
```
curl http://127.0.0.1:5000/api/ping
```

This should cause the Viewer to display a "ping" message instantly.


# Clustering for local dev 

Now let's see clustering NOT work first. Open up two instances like this:
```
PORT=5000 elixir --name a@127.0.0.1 -S mix phx.server
PORT=5001 elixir --name b@127.0.0.1 -S mix phx.server
```

Bring up a Viewer pointed to both servers:
(http://127.0.0.1:5000/viewer)
(http://127.0.0.1:5001/viewer)

And if you run the `curl` on the ping API you'll see that only one viewer will update. 

### Add `libcluster` to your dependencies in `mix.exs` and rerun `mix deps.get`
```
{:libcluster, "~> 3.3"},
```

### Add the libcluster config to your `config/dev.exs`:
```
config :libcluster,
	topologies: [
		example: [
			strategy: Cluster.Strategy.Epmd,
			config: [hosts: [:"a@127.0.0.1", :"b@127.0.0.1"]],
			connect: {:net_kernel, :connect_node, []},
			disconnect: {:erlang, :disconnect_node, []},
			list_nodes: {:erlang, :nodes, [:connected]},
		]
	]
```

### Update your `application.ex` 
You will define a `topologies` variable in your start() function and add a `Cluster.Supervisor` to the list of children.

It will look like:
```
def start(_type, _args) do
	topologies = Application.get_env(:libcluster, :topologies) || []
	
	children = [
		...
		{Cluster.Supervisor, [topologies, [name: Talkie.ClusterSupervisor]]}
	]

	...
```

Now if you restart both of the instances and test again you SHOULD see both Viewers update when you ping either one of the instances. 

Pretty cool right!


# Clustering with Fly.io

First, deploy the app as-is to Fly and test that you see it not working as expected again.

What you should see (assuming you default to 2 instances) is a 50/50 chance that you'll receive the ping message if you pull up a `/viewer` page and then hit the ping API. A rough test is to bring up 2-4 separate windows and to the `/viewer` path and test a ping to see how many of the windows update.

### Add `libcluster` configuration to prod in the `config/runtime.exs` file
```
app_name = System.get_env("FLY_APP_NAME") || "talkie"

config :libcluster,
	debug: true,
	topologies: [
		fly6pn: [
			strategy: Cluster.Strategy.DNSPoll,
			config: [
				polling_interval: 5_000,
				query: "#{app_name}.internal",
				node_basename: app_name
			]
		]
	]
```

### Change how the RELEASE_NODE value is set in the `rel/env.sh.eex`

Note if you don't have this file, then you just haven't completed `fly deploy` yet. Do that first.

```
#Before
export RELEASE_NODE="${FLY_APP_NAME}-${FLY_IMAGE_REF##*-}@${FLY_PRIVATE_IP}"

#After
ip=$(grep fly-local-6pn /etc/hosts | cut -f 1)
export RELEASE_NODE="${FLY_APP_NAME}@${ip}"
```

See more from [Fly.io guide](https://fly.io/docs/elixir/the-basics/clustering/)


# See the full project

Check out the [Github Repo](https://github.com/byronsalty/talkie)for the project used to test this article.


# Other Resources

There are some good examples here:

Well [written article](https://www.poeticoding.com/distributed-phoenix-chat-with-pubsub-pg2-adapter/) from Alvise Susmel that helped me a lot