IOT Stack on Raspberry Pi: Full Upgrade Procedure
Tags: raspberrypi, iot
Published: 2023-02-16 06:50:31

My Raspberry PI based IOT stack, introduced in an [earlier article](https://admantium.com/blog/iot01_installing_iot_stack/), is productive and online for more than one year. It’s time to make a full upgrade of the operating system, the Docker containers, and all associated configurations.

This article completely covers my approach, including regrettable mistakes that I disclose to help other preventing them or finding a quick solution. Learn about compatibility problems of Alpine Linux with Raspberry Pi OS based on Debian Buster, and feature changes in the IOT Stack itself.

_The technical context of this article is to start an upgrade from Home Assistant 2022.11 and ESPHome 2022.11.3, but it should work with older and newer versions as well._

_This article originally appeared at my blog [admantium.com](https://admantium.com/blog/iot09_how_to_update_iot_stack/)_.

## Recommended Update Procedure

The [official IOT Stack documentation](https://sensorsiot.github.io/IOTstack/Updates/) explains why updates can be difficult and things can break: The Docker container use ephemeral storage for runtime, and they are versioned projects themselves, with specific configuration options passed as environment variables, config files, or flags to the binary. This versioned configuration complexity is very well handled in each commit version of the master branch. But with each passing commit, config changes accumulate, and it some point in time, changes will not be fully backward compatible. In addition, newer Docker containers might require newer libraries of the base operating system. And the final catch is that a new [Raspberry Pi OS based on Debian Bullseye](https://www.raspberrypi.com/news/raspberry-pi-os-debian-bullseye/) was released in November 2021.

This is a complex situation. The documentation therefore recommends:

- Make a complete disc image of your Raspberry Pi as a backup
- Upgrade the Raspberry Pi OS libraries
- Upgrade only the Docker container

If you want to push even further, you can also:

- Upgrade to the next Raspberry Pi release
- Update the IoT Stack project files

Knowing this, let’s get started.

## Context

My IOT stack installation was initially completed in June 2021, on these components:

The Raspberry Pi version is based on the older Debian Buster release.

```bash
Linux nexus 5.10.103-v7l+ #1529 SMP Tue Mar 8 12:24:00 GMT 2022 armv7l GNU/Linux

RETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs
```

The IoT Stack version that I used since the inception is a commit in 2021-06 on the master branch. Essentially, I have a docker-compose file that references versioned Docker images and list the options required to run them (env vars, ports, container names for inter-container communication). In this file, only a few images were versions pinned, several just used the `latest` tag. My assumption was that this means every time a container is restarted it will also pull the latest image. But this was wrong! Running a `docker image ls`, it showed that I still used images that were over 15 months old!

```bash
REPOSITORY                                          TAG         IMAGE ID       CREATED         SIZE
ghcr.io/home-assistant/raspberrypi4-homeassistant   stable      6573960f8063   14 months ago   1.05GB
esphome/esphome                                     2021.9.1    76da910d1982   14 months ago   942MB


iotstack_mosquitto                                  latest      df0853b69834   15 months ago   11.7MB
influxdb                                            1.8         69a9438650d8   15 months ago   246MB
telegraf                                            latest      a721ac170fad   15 months ago   273MB
iotstack_nodered                                    latest      74a0b2f9d520   15 months ago   428MB

grafana/grafana                                     latest      7914b5231e0b   15 months ago   178MB
portainer/portainer-ce                              latest      bd3c978cdaec   15 months ago   157MB

pihole/pihole                                       2021.12.1   d463557f7f81   11 months ago   236MB
```

Now let’s see how to upgrade the stack systematically.

## Update Raspberry Pi OS Libraries

The first step is to perform al full upgrade of the Raspberry Pi OS itself. Two commands are sufficient:

```yaml
sudo apt-get update
sudo apt-get full-upgrade

The following packages will be upgraded:
  base-files bind9-host bluez containerd.io distro-info-data dnsutils docker-ce docker-ce-cli docker-ce-rootless-extras
  ffmpeg folder2ram git git-man isc-dhcp-client isc-dhcp-common libarchive13 libavcodec58 libavdevice58 libavfilter7
...
```

This update went smoothly. After a reboot, all things worked as before.

## Update Docker Images

### ESPHome

ESP Home is an essential part of my IOT stack: A highly usable, sophisticated framework to quickly add esp8266/esp32 boards and hundreds of sensors into a managed fleet complete with over-the-air update functionality. I'm still utterly fascinated by this project.

To update it, I researched the official Docker hub for tags, and then slowly upgraded the version information in the `docker-compose.yml` file:

```bash
2021.9.1
2021.12.3
2022.3.2
2022.6.3
2022.11.3
```

Every time I changed the version information in the docker-compose file, I also used the following command to enforce the creation of a new container:

```bash
docker-compose up --force-recreate --build -d esphome
```

In between, I always checked that the UI was working. With the final update complete, let’s try updating the software that runs on a sensor:

```bash
INFO Reading configuration /config/esp8266-01.yaml...
INFO Generating C++ source...
INFO Core config or version changed, cleaning build files...
INFO Compiling app...
Processing esp8266-01 (board: d1_mini; framework: arduino; platform: platformio/espressif8266 @ 3.2.0)
--------------------------------------------------------------------------------
Platform Manager: Installing platformio/espressif8266 @ 3.2.0
INFO Installing platformio/espressif8266 @ 3.2.0
...
INFO framework-arduinoespressif8266@3.30002.0 has been installed!
```

Then, connect to the sensor and see its streaming log files:

```bash
[19:31:17][C][mqtt_subscribe.sensor:031]: MQTT Subscribe 'Configure Sleep Time'
[19:31:17][C][mqtt_subscribe.sensor:031]:   State Class: ''
[19:31:17][C][mqtt_subscribe.sensor:031]:   Unit of Measurement: 'm'
[19:31:17][C][mqtt_subscribe.sensor:031]:   Accuracy Decimals: 0
[19:31:17][C][mqtt_subscribe.sensor:032]:   Topic: esp/sleep_time
[19:31:17][C][mqtt.sensor:027]: MQTT Sensor 'esp8266-01.voltage':
[19:31:17][C][mqtt.sensor:031]:   State Topic: 'nodes/esp8266-01/sensor/esp8266-01voltage/state'
[19:31:17][C][mqtt.sensor:027]: MQTT Sensor 'WiFi Signal esp8266-01':
[19:31:17][C][mqtt.sensor:031]:   State Topic: 'nodes/esp8266-01/sensor/wifi_signal_esp8266-01/state'
[19:31:17][C][mqtt.binary_sensor:022]: MQTT Binary Sensor 'ESP01 Motion':
[19:31:17][C][mqtt.binary_sensor:023]:   State Topic: 'nodes/esp8266-01/binary_sensor/esp01_motion/state'
[19:31:42][D][sensor:126]: 'VCC': Sending state 2.96582 V with 2 decimals of accuracy
[19:31:56][D][sensor:126]: 'WiFi Signal esp8266-01': Sending state -70.00000 dBm with 0 decimals of accuracy
```

That’s good. The first update process worked.

### PiHole

The next image to be updated is PiHole, a handy utility for removing DNS queries to undesired external services. I used the very same strategy as before: upgrade tag by tag. The intended upgrade order was as follows:

```bash
2022.02.1
2022.04.3
2022.05
```

But very soon into the upgrade process, I noticed that the dashboard did not work anymore:

![](https://admantium.com/images/blog/iot09_broken_dashboard.png)

So, I went back the versions, and was eventually stuck with `2022.02.1`.

The hunt for the root error was surprising: Two different threads came to the same conclusion.

First, I searched the Internet about PiHole, Docker and Raspberry Pi and stumbled over [this thread](https://github.com/pi-hole/docker-pi-hole/discussions/1021) that explained that the Debian version of the Raspberry Pi is missing an more up-to-date version of the library `libseccomp`.

Second, I looked into logs from another Docker container that I wanted to update, and saw these messages:

```bash
s6-linux-init-hpr: fatal: unable to reboot(): Operation not permitted
s6-svscan: warning: unable to iopause: Operation not permitted
s6-svscan: warning: executing into .s6-svscan/crash
s6-svscan crashed. Killing everything and exiting.
```

Searching the internet again for these error messages, I stumbled upon a [github issue](https://github.com/oznu/docker-homebridge/issues/434#issuecomment-1111958145) in which one solution was also to upgrade to a newer version of `libseccomp`.

And then those two threads came together in this [blog post about libseccomp and Alpine3.13](https://blog.samcater.com/fix-workaround-rpi4-docker-libseccomp2-docker-20/), the very Docker images that several IoT stack images are built upon. Alpine based images after version 3.13 cannot run on a Raspberry Pi with Buster Debian, because the `libseccomp` version is just too old.

In a nutshell, the solution is to get an up-to-date `libseccomp` from the next upstream Raspberry Pi OS release (which is now based on bullsye). The concrete commands to get this done are these:

```bash
echo 'deb http://deb.debian.org/debian buster-backports main contrib non-free' | sudo tee -a /etc/apt/sources.list.d/debian-backports.list
sudo apt update
sudo apt install libseccomp2 -t buster-backports
```

After this, I could update the PiHole container to the latest version successfully, and the logfiles indicate a successful start:

```bash
Pi-hole version is v5.14.2 (Latest: v5.14.2)
AdminLTE version is v5.17 (Latest: v5.17)
FTL version is v5.19.2 (Latest: v5.19.2)
Container tag is: 2022.11.2
```

### Update all other Docker Containers

The `libseccomp` problem was the root cause that prevented all the other updates as well!

Once solved, I could upgrade all remaining images to the following versions:

```bash
✅ esphome/esphome:2022.11.3
✅ homeassistant/home-assistant:2022.11
✅ pihole/pihole:2022.11.2
✅ mosquitto:2022.12 (IOT Stack Template)
✅ influxdb:1.8.10
✅ telegraf:1.24.4
✅ nodered:2022.12 (IOT Stack Template)
✅ portainer:linux-arm-2.16.2
✅ grafana/grafana:9.3.1
```

The final proof is the Home Assistant dashboard, showing the sensor data just as before:

![](https://admantium.com/images/blog/iot09_home_assistant_dashboard.png)

## Version Updates & Migrations

Although all applications could be updated successfully, some changes needed to be applied nevertheless. Here is a short summary:

### ESPHome: FastLED Driver is not supported anymore

During updating the ESPHome code on the sensors, I saw this error:

```bash
Failed config

light.fastled_clockless: [source /config/esp8266-dht.yaml:60]

  This feature requires framework version 2.7.4 or lower. Please see note on documentation for FastLED.
```

The [fastled module](https://esphome.io/components/light/fastled.html) is simply not supported anymore. Changing the config to this one worked:

```yml
esp8266:
  framework:
    version: 2.7.4
  board: d1_mini

esphome:
  name: esp8266-dht
```

### Home Assistent: Config File Update

The Home Assistant update included a heavy database migration as seen in the first log lines in the newly started container:

```bash
2022-12-07 11:16:50.466 WARNING (Recorder) [homeassistant.components.recorder.migration] Database is about to upgrade from schema version: 22 to: 30
2022-12-07 11:16:50.495 WARNING (Recorder) [homeassistant.components.recorder.migration] Adding columns name to table statistics_meta. Note: this can take several minutes on large databases and slow computers. Please be patient!
2022-12-07 11:16:50.825 WARNING (Recorder) [homeassistant.components.recorder.migration] Adding index `ix_statistics_statistic_id_start` to database. Note: this can take several minutes on large databases and slow computers. Please be patient!
2022-12-07 11:16:50.893 WARNING (Recorder) [homeassistant.components.recorder.migration] Adding index `ix_statistics_short_term_statistic_id_start` to database. Note: this can take several minutes on large databases and slow computers. Please be patient!
2022-12-07 11:16:51.554 WARNING (Recorder) [homeassistant.components.recorder.migration] Adding columns attributes_id to table states. Note: this can take several minutes on large databases and slow computers. Please be patient!
```

Additional container logs showed several errors related to new features of the most recent versions. Particularly following:

```bash
2022-12-07 11:16:52.554 WARNING (MainThread) [homeassistant.components.mqtt] The 'broker' option near /config/configuration.yaml:34 is deprecated, please remove it from your configuration
2022-12-07 11:16:53.001 WARNING (MainThread) [homeassistant.components.mqtt] The 'password' option near /config/configuration.yaml:34 is deprecated, please remove it from your configuration
2022-12-07 11:16:53.401 WARNING (MainThread) [homeassistant.components.mqtt] The 'username' option near /config/configuration.yaml:34 is deprecated, please remove it from your configuration
2022-12-07 11:16:53.998 WARNING (MainThread) [homeassistant.components.mqtt.mixins] Manually configured MQTT sensor(s) found under platform key 'sensor', please move to the mqtt integration key, see <https://www.home-assistant.io/integrations/sensor.mqtt/#new_format>
2022-12-07 11:16:54.456 ERROR (MainThread) [homeassistant.components.camera] The mjpeg platform for the camera integration does not support platform setup. Please remove it from your config.
```

I updated the config file as suggested by these logfiles. And for the camera, I switched to the [new MJPG IP camera config](https://www.home-assistant.io/integrations/mjpeg).

## Conclusion

IOT stack is an amazing project that bundles essential applications for the setup and deployment of sensors, combined with data extraction, visualization, and dashboards to build automation. This blog post showed how to update a 15 month old installation to the newest version without breaking things. The essential steps are these: a) Update all Raspberry Pi packages to their latest version, b) Install the most recent `libseccomp` library from the next upstream Raspberry Pi OS, c) force recreate the Docker images, climbing from version to version, d) follow recommended config updates for the Docker containers. These steps resulted in an up-to-date versioned stack, and only some minor configuration changes in the config files were required.