Connect Nerves devices forming an Erlang cluster
Tags: elixir, nerves, iot, erlang
Published: 2021-10-23 19:16:14

I want to enable my [Nerves] devices to talk to each other using [Distributed Erlang]. Here is my memo.

[Nerves]: https://www.nerves-project.org/
[Distributed Erlang]: https://erlang.org/doc/reference_manual/distributed.html

## Preparation

### About Nerves

If you are not familiar with Nerves, [this conference talk by a Nerves co-author Frank Hunleth](https://youtu.be/dZzM-rLu-zo) will introduce it to you.

### Find hostname or IP address for each device

Normally when shelling into a [Nerves] device, we will see IP address and host name printed by [nerves_motd].

![](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/97xt3gij2f3d6afcm53f.png)

[nerves_motd]: https://github.com/nerves-project/nerves_motd

### DNS Bridge configuration

According to [mdns_lite - DNS Bridge configuration](https://hexdocs.pm/mdns_lite/readme.html#dns-bridge-configuration) documentation, we need to let Erlang/OTP's built-in DNS resolver know about mDNS. [Underjord's YouTube video explaining DNS Bridge configuration] was helpful.

[Underjord's YouTube video explaining DNS Bridge configuration]: https://youtu.be/ZdtAVlzFf6Q?t=1503
[mdns_lite - DNS Bridge configuration]: https://hexdocs.pm/mdns_lite/readme.html#dns-bridge-configuration

## Start Erlang nodes

- [nerves_pack - Erlang distribution] documentation explains about how to form an Erlang cluster.
- Let's say we have a device with host name `nerves-mn00.local` and anoter with host name `nerves-mn02.local`.
- Alternatively IP addresses can be used instead of host names
- Make sure that the same [Erlang magic cookie] is used

```elixir
❯ ssh nerves-mn00.local

# make sure the epmd OS process is running by calling epmd -daemon
iex> System.cmd("epmd", ["-daemon"])

# start a node.
iex> Node.start(:"nerves@nerves-mn00.local")

# check current node. `node/0` does the same.
iex(nerves@nerves-mn00.local)> Node.self()

# configure the magic cookie to form this cluster.
iex(nerves@nerves-mn00.local)> Node.set_cookie(:securecookie)
```

```elixir
❯ ssh nerves-mn02.local

iex> System.cmd("epmd", ["-daemon"])
iex> Node.start(:"nerves@nerves-mn02.local")
iex(nerves@nerves-mn02.local)> Node.set_cookie(:securecookie)
```

[nerves_pack - Erlang distribution]: https://hexdocs.pm/nerves_pack/readme.html#erlang-distribution
[Erlang magic cookie]: https://erlang.org/doc/reference_manual/distributed.html#security

## Connect a node to another

- Let's connect `nerves-mn02.local` to `nerves-mn00.local`

```elixir
iex(nerves@nerves-mn02.local)> Node.connect(:"nerves@nerves-mn00.local")
true

iex(nerves@nerves-mn02.local)> Node.list()
[:"nerves@nerves-mn00.local"]
```

## Invoke a function in another node

- From `nerves-mn02.local` invoke a function that is defined in `nerves-mn00.local` using [`Node.spawn/2`] or something similar
- Let's access to the other device and peek into its firmware infomation by invoking [`Toolshed.Nerves.uname/0`] that is normally available in our Nerves IEx shell.

```elixir
Node.spawn(
  # the node name we want to invoke a function in
  :"nerves@nerves-mn00.local",
  # the function we want to invoke
  fn -> Toolshed.Nerves.uname() end
)
```

```elixir
iex(nerves@nerves-mn02.local)> Node.spawn(:"nerves@nerves-mn00.local", fn -> Toolshed.Nerves.uname() end)
Nerves nerves-mn00 hello_nerves 0.1.0 (240d82e1-1e6c-5800-0ef9-63cba9efc212) arm
#PID<51667.6840.0>
```

Yay, we were able to at least do something in another device through the [Distributed Erlang]. Isn't it cool?

I put together what I learned as [`kantan_cluster`] Elixir package. It allows us to form an Erlang cluster automatically. Please take a look if you like.

## Resources

- [Elixir Node](https://hexdocs.pm/elixir/1.12/Node.html)
- [iex - Man Page](https://www.mankier.com/1/iex)
- [Forming an Erlang cluster of Pi Zeros by underjord](https://youtu.be/ZdtAVlzFf6Q)
- [nerves_pack - Erlang distribution](https://hexdocs.pm/nerves_pack/readme.html#erlang-distribution)
- [mdns_lite - DNS Bridge configuration](https://hexdocs.pm/mdns_lite/readme.html#dns-bridge-configuration)
- [nerves_pack（vintage_net含む）を使ってNerves起動時に`Node.connect()`するようにした by nishiuchikazuma](https://qiita.com/nishiuchikazuma/items/f68d2661959197d0765c)
- [はじめてNerves(8) 単一ホストで動くシステムを複数ホストに分散する by kikuyuta](https://qiita.com/kikuyuta/items/4b8326d0584fe26083a6)

<!-- Links -->

[`Toolshed.Nerves.uname/0`]: https://hexdocs.pm/toolshed/Toolshed.Nerves-function-uname.html#uname/0
[`Node.spawn/2`]: https://hexdocs.pm/elixir/1.12/Node.html#spawn/2
[`kantan_cluster`]: https://github.com/mnishiguchi/kantan_cluster