YugabyteDB in a Micro VM with no network
Tags: yugabytedb, microvm, firecracker, dns
Published: 2022-08-30 12:41:35

[In a previous post](
https://dev.to/yugabyte/fastest-start-of-yugabytedb-for-a-small-lab-283a) I mentioned how to start [YugabyteDB](https://www.yugabyte.com/yugabytedb/) fast for a quick short-life lab. This can run on Micro VM (like firecracker) and in some cases you have no network interfaces with it. You don't need it if you connect with `ysqlsh` from the MicroVM but you may encounter name resolution issues. Note that you can reproduce it with `docker run --network none`.

This post is about the following errors and their resolution:
```
ysqlsh: FATAL:  Timed out: OpenTable RPC (request call id 2) to 0.0.0.0:9100 timed out after 120.000s
```
or, in the `yb-tserver.ERROR`:
```
E0830 11:32:44.518383    66 async_initializer.cc:100] Failed to initialize client: Network error (yb/util/net/net_util.cc:447): Could not determine local host names: Unable to lookup FQDN (699cdfc74424), getaddrinfo returned -11 (EAI_SYSTEM): Connection refused (system error 111)
```

This has to do with name resolution. YugabyteDB is a [distributed SQL database](https://www.yugabyte.com/tech/distributed-sql/) and must have a matching hostname and IP address to identify the nodes. The last message is explicit: YugabyteDB calls the Linux `getaddrinfo()` system call to get the IP of `699cdfc74424` which is my container hostname

Basically, it does the same as:
```sh
sh-4.2# getent ahosts  $(hostname)

127.0.0.2       STREAM 699cdfc74424
127.0.0.2       DGRAM
127.0.0.2       RAW
::1             STREAM
::1             DGRAM
::1             RAW
```
But why `127.0.0.2`? I'm trying to start YugabyteDB on `127.0.0.1` which is the only interface I have on my no-network container:
```sh

/home/yugabyte/bin/yb-master --fs_data_dirs=/var/tmp --master_addresses=127.0.0.1:7100 --replication_factor=1 --default_memory_limit_to_ram_ratio=0.30 &
/home/yugabyte/bin/yb-tserver --fs_data_dirs=/var/tmp --tserver_master_addrs=127.0.0.1:7100 --default_memory_limit_to_ram_ratio=0.30 &

```

The reason is in `/etc/nsswitch.conf`:
```sh
sh-4.2# grep hosts: /etc/nsswitch.conf

#hosts:     db files nisplus nis dns
hosts:      files dns myhostname
sh-4.2#
```

The host name resolution goes first to `files`, which in this case is `/etc/hosts`:
```sh
sh-4.2# cat /etc/hosts

127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
```
My hostname is unknown there.

Then it goes to `dns` but I've no DNS in my no-network container (reason for the `Connection refused` error I guess)

Then it goes to `myhostname` which resolves the hostname, but to `127.0.0.2`. It is used to avoid patching the `/etc/hosts` but in my case cannot be used because `127.0.0.2` is not known.
I tried to start `yb-master --master_addresses=127.0.0.2:7100` but it fails with `None of the local addresses are present in master_addresses 127.0.0.2:7100`

Then, my solution is simply to add the hostname to the `/etc/hosts`:

```sh
echo "127.0.0.1 localhost $(hostname)"  > /etc/hosts
```

With this before starting `yb-master--fs_data_dirs=/var/tmp --master_addresses=127.0.0.1:7100 --replication_factor=1` and `yb-tserver --fs_data_dirs=/var/tmp --master_addresses=127.0.0.1:7100 --replication_factor=1` you should be able to connect with a simple `ysqlsh` which connects by default to `localhost`

Of course, a distributed SQL database with no network makes no sense except for a small container used for continuous integration for example.