Implementing Non-Trivial Containerized Systems - Part 2: Containerizing With Docker
Tags: containers, tutorial, cloud, docker
Published: 2021-06-04 17:45:18

This is the second part of a multi-part series on designing and building non-trivial containerized solutions. We're making a radio station using off-the-shelf components and some home-spun software, all on top of Docker, Docker Compose, and eventually, Kubernetes.

In this part, we're going to take the architecture we cooked up in the last part, and see if we can't get it running in containers, using nothing but Docker itself.
Today, we're going to build a radio station. We drew up the architecture in [*our last post*](https://starkandwayne.com/blog/implementing-non-trivial-containerized-systems-part-1-picking-components/):
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/oebipl4fmwdjyk02nrpz.png)
Roll up your sleeves, we're gonna start building some containers.

##Sourcing Audio - youtube-dl##
Before we can broadcast audio, we have to have audio, and we're going to get our audio from YouTube. This, then, is our first container.

We're going to start with Ubuntu 20.04 (Focal Fossa) base image. I like this image not only because it's familiar and homey; but also for access to `apt` for package management, and (more importantly) all those sweet, sweet Debian packages. Case in point, we need Python (youtube-dl is written in Python):
```
FROM ubuntu:20.04
RUN apt-get update \
 && apt-get install -y python3 python3-pip \
 && pip3 install youtube-dl
```
We're going to set up an `ENTRYPOINT` in our image, so that when we `docker run` it, we can pass it arguments, and use the container as if it were youtube-dl itself.
```
FROM ubuntu:20.04
RUN apt-get update \
 && apt-get install -y python3 python3-pip \
 && pip3 install youtube-dl

WORKDIR /radio
ENTRYPOINT ["youtube-dl"]
```
Let's run what we've got so far:
```
$ docker build -t youtube-dl .
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM ubuntu:20.04
 ---> f63181f19b2f
Step 2/2 : RUN apt-get update  && apt-get install -y python3 python3-pip  && pip3 install youtube-dl
 ---> Running in cbc3f65f6028
<a whole bunch of apt-get output...>
Collecting youtube-dl
  Downloading youtube_dl-2021.3.14-py2.py3-none-any.whl (1.9 MB)
Installing collected packages: youtube-dl
Successfully installed youtube-dl-2021.3.14
Removing intermediate container cbc3f65f6028
 ---> 66648c847e34
Successfully built 66648c847e34
Successfully tagged youtube-dl:latest

$ docker run youtube-dl --version
2021.03.14
```
Since we used an `ENTRYPOINT`, the `--version` argument we gave to the container got passed through youtube-dl. This is important, since this will be our primary means of interacting with youtube-dl.

Let's try to download some back episodes of my podcast, __Rent / Buy / Build__. You can find those [*here*](https://www.youtube.com/watch?v=D7hxDYBoHvQ&list=PLGDWYlG32yKoYAuREw9eSLFIdWJZ_3rW8).
```
$ docker run youtube-dl https://youtu.be/D7hxDYBoHvQ?list=PLGDWYlG32yKoYAuREw9eSLFIdWJZ_3rW8
[youtube:tab] Downloading playlist PLGDWYlG32yKoYAuREw9eSLFIdWJZ_3rW8 - add --no-playlist to just download video D7hxDYBoHvQ
[youtube:tab] PLGDWYlG32yKoYAuREw9eSLFIdWJZ_3rW8: Downloading webpage
[youtube:tab] PLGDWYlG32yKoYAuREw9eSLFIdWJZ_3rW8: Downloading webpage
[download] Downloading playlist: Rent / Buy / Build Episodes
[youtube:tab] playlist Rent / Buy / Build Episodes: Downloading 1 videos
[download] Downloading video 1 of 1
[youtube] D7hxDYBoHvQ: Downloading webpage
[youtube] D7hxDYBoHvQ: Downloading MPD manifest
[download] Destination: Episode 0 - What's All This Then-D7hxDYBoHvQ.mp4
[download] 100% of 12.76MiB in 00:0296MiB/s ETA 00:003
[download] Finished downloading playlist: Rent / Buy / Build Episodes
```
Looks like we got the file, but where did it go? Into the ether. When our container process (youtube-dl) exits (i.e. finishes downloading and transcoding the video), the root filesystem for the container is wiped away. This is a feature, honest. If we want to keep any of the files we create in the container, we're going to need to bind-mount in some persistent(-ish) storage. If you recall, we set our `WORKDIR` to `/radio`, so if we can mount a directory from *outside* of the container, we can keep those files around longer.
```
$ mkdir radio
$ docker run -v $PWD/radio:/radio youtube-dl https://youtu.be/D7hxDYBoHvQ?list=PLGDWYlG32yKoYAuREw9eSLFIdWJZ_3rW8
[youtube:tab] Downloading playlist PLGDWYlG32yKoYAuREw9eSLFIdWJZ_3rW8 - add --no-playlist to just download video D7hxDYBoHvQ
[youtube:tab] PLGDWYlG32yKoYAuREw9eSLFIdWJZ_3rW8: Downloading webpage
[youtube:tab] PLGDWYlG32yKoYAuREw9eSLFIdWJZ_3rW8: Downloading webpage
[download] Downloading playlist: Rent / Buy / Build Episodes
[youtube:tab] playlist Rent / Buy / Build Episodes: Downloading 1 videos
[download] Downloading video 1 of 1
[youtube] D7hxDYBoHvQ: Downloading webpage
[youtube] D7hxDYBoHvQ: Downloading MPD manifest
[download] Destination: Episode 0 - What's All This Then-D7hxDYBoHvQ.mp4
[download] 100% of 12.76MiB in 00:0218MiB/s ETA 00:004
[download] Finished downloading playlist: Rent / Buy / Build Episodes

$ ls -l radio
total 13312
-rw-r--r-- 1 jhunt staff 13383936 Mar 22 14:30 "Episode 0 - What's All This Then-D7hxDYBoHvQ.mp4"
```
Yay! Files! Too bad they're video files, not audio files. To fix that, let's throw ffmpeg into our container image.
```
FROM ubuntu:20.04
RUN apt-get update \
 && DEBIAN_FRONTEND=noninteractive apt-get install -y python3 python3-pip ffmpeg \
 && pip3 install youtube-dl

WORKDIR /radio
ENTRYPOINT ["youtube-dl"]
```
*(Note: I had to add that `DEBIAN_FRONTEND=noninteractive` bit to keep the package manager from bothering me with terminal prompts about timezone data.)*

With ffmpeg, we can take the files that come out of youtube-dl, drop the video components, and convert the audio into a common format (which will be useful when we get to LiquidSoap programming).
```
$ docker run -v $PWD/radio:/radio --entrypoint ffmpeg youtube-dl \
  -i "Episode 0 - What's All This Then-D7hxDYBoHvQ.mp4" \
  -vn -c:a libopus \
  "Episode 0 - What's All This Then-D7hxDYBoHvQ.opus"
```
That's a fair bit of domain expertise regarding file formats and conversion, and it's all right out there, in the open. It's also a two step process. This is a problem that can be fixed by a small shell script.
```
#!/bin/sh
set -eu

mkdir -p /tmp/ytdl.$$
cd /tmp/ytdl.$$
for url in "$@"; do
  youtube-dl -x "$url"
  for f in *; do
    ffmpeg -i "$f" -vn -c:a libopus "$f.opus"
    rm "$f"
    mv "$f.opus" /radio
  done
done
cd /
rm -rf /tmp/ytdl.$$
ls -1 /radio/*.opus > /radio/playlist.m3u
```
There is one new feature in this shell script: __playlist management__. After we've ingested new files into `/radio`, we rebuild a `.m3u` playlist. This is a simple file: one file path per line, read sequentially. Piping the `ls` output through `sort` ensures that we have a stable playlist order; we'll let other systems randomize playlist ordering if they wish.

We can copy that script (which I've unimaginatively named `entrypoint`) into our docker image via the Dockerfile:
```
FROM ubuntu:20.04
RUN apt-get update \
 && DEBIAN_FRONTEND=noninteractive apt-get install -y python3 python3-pip ffmpeg \
 && pip install youtube-dl

WORKDIR /radio

COPY entrypoint /usr/bin/entrypoint
RUN chmod 0755 /usr/bin/entrypoint
ENTRYPOINT ["entrypoint"]
```
With this change in place, we go back to invoking our container (after a fresh build!) with just the URL we want downloaded and transcoded:
```
$ docker run -v $PWD/radio:/radio youtube-dl https://youtu.be/D7hxDYBoHvQ?list=PLGDWYlG32yKoYAuREw9eSLFIdWJZ_3rW8
[youtube:tab] Downloading playlist PLGDWYlG32yKoYAuREw9eSLFIdWJZ_3rW8 - add --no-playlist to just download video D7hxDYBoHvQ
[youtube:tab] PLGDWYlG32yKoYAuREw9eSLFIdWJZ_3rW8: Downloading webpage
[youtube:tab] PLGDWYlG32yKoYAuREw9eSLFIdWJZ_3rW8: Downloading webpage
[download] Downloading playlist: Rent / Buy / Build Episodes
[youtube:tab] playlist Rent / Buy / Build Episodes: Downloading 1 videos
[download] Downloading video 1 of 1
[youtube] D7hxDYBoHvQ: Downloading webpage
[youtube] D7hxDYBoHvQ: Downloading MPD manifest
[download] Destination: Episode 0 - What's All This Then-D7hxDYBoHvQ.m4a
[download] 100% of 9.73MiB in 00:0123MiB/s ETA 00:005
[ffmpeg] Correcting container in "Episode 0 - What's All This Then-D7hxDYBoHvQ.m4a"
[ffmpeg] Post-process file Episode 0 - What's All This Then-D7hxDYBoHvQ.m4a exists, skipping
[download] Finished downloading playlist: Rent / Buy / Build Episodes
ffmpeg version 4.2.4-1ubuntu0.1 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-10ubuntu2)
  configuration: --prefix=/usr --extra-version=1ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'Episode 0 - What's All This Then-D7hxDYBoHvQ.m4a':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2mp41
    encoder         : Lavf58.29.100
  Duration: 00:10:30.10, start: 0.000000, bitrate: 129 kb/s
    Stream #0:0(eng): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      handler_name    : ISO Media file produced by Google Inc.
Stream mapping:
  Stream #0:0 -> #0:0 (aac (native) -> opus (libopus))
Press [q] to stop, [?] for help
[libopus @ 0x560757a9e280] No bit rate set. Defaulting to 96000 bps.
Output #0, opus, to 'Episode 0 - What's All This Then-D7hxDYBoHvQ.m4a.opus':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2mp41
    encoder         : Lavf58.29.100
    Stream #0:0(eng): Audio: opus (libopus), 48000 Hz, stereo, flt, 96 kb/s (default)
    Metadata:
      handler_name    : ISO Media file produced by Google Inc.
      encoder         : Lavc58.54.100 libopus
      major_brand     : isom
      minor_version   : 512
      compatible_brands: isomiso2mp41
size=    6139kB time=00:10:30.11 bitrate=  79.8kbits/s speed=72.5x
video:0kB audio:6085kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.876994%

$ ls -l radio
total 6144
-rw-r--r-- 1 jhunt staff 6286163 Mar 22 14:52 "Episode 0 - What's All This Then-D7hxDYBoHvQ.m4a.opus"
-rw-r--r-- 1 jhunt staff      61 Mar 22 14:52  playlist.m3u
```
We can even do multiple!
```
$ docker run -v $PWD/radio:/radio youtube-dl \
  ... \
  ... \
  ...
```
##You're On The Air With Icecast##
Time to turn up the transmitters and start broadcasting! For that, we need Icecast. As before, we'll start with an Ubuntu 20.04 base and add the packages we need.
```
FROM ubuntu:20.04
RUN apt-get update \
 && apt-get install -y icecast2
```
Icecast won't let just anybody stream music to it. That would be pure chaos! This is a *broadcast* radio station, not a peer-to-peer free-for-all. We chose the tunes. We pick the commercials. We write the jingles. To keep out the riff-raff, Icecast uses speakeasy authentication. Each source driver is required to provide the secret password before we'll accept its audio stream.

The Icecast core team definitely has a sense of humor, because in the default configuration, the source password is "hackme". We need to change that.

It's important to remember that Icecast was born when XML was all the rage and came of age before it was fashionable to configure things via environment variables. To change the source driver password, we need to modify the configuration file: `/etc/icecast2/icecast.xml`.
```
FROM ubuntu:20.04
ARG PASSWORD
RUN apt-get update \
 && apt-get install -y icecast2 \
 && sed -i -e "s/>hackme</>$PASSWORD</" /etc/icecast2/icecast.xml
```
This sed command finds all instances of `">hackme<"` and replaces it with `">$PASSWORD<"`. The new `ARG` directive sets up a Docker *build argument*, which is a bit like an environment variable that only exists when the container image is *built*. We specify it thusly:
```
$ docker build --build-arg PASSWORD=sekrit -t icecast .
```
Let's take it for a spin and make sure everything is ship shape.
```
$ docker run icecast icecast2 -c /etc/icecast2/icecast.xml
[2021-03-22  19:04:15] WARN CONFIG/_parse_root Warning,  not configured, using default value "localhost". This will cause problems, e.g. with YP directory listings.
[2021-03-22  19:04:15] WARN CONFIG/_parse_root Warning,  not configured, using default value "Earth".
[2021-03-22  19:04:15] WARN CONFIG/_parse_root Warning,  contact not configured, using default value "icemaster@localhost".
[2021-03-22  19:04:15] WARN fserve/fserve_recheck_mime_types Cannot open mime types file /etc/mime.types
ERROR: You should not run icecast2 as root
Use the changeowner directive in the config file
```
Drat.

Icecast doesn't want to run as root, so we're going to need to provision a user account *inside* the container.
```
FROM ubuntu:20.04
ARG PASSWORD
RUN apt-get update \
 && apt-get install -y icecast2 \
 && sed -i -e "s/>hackme</>$PASSWORD</" /etc/icecast2/icecast.xml \
 && useradd radio \
 && chown -R radio:radio /etc/icecast2 /var/log/icecast2
USER radio
ENTRYPOINT ["icecast2"]
CMD ["-c", "/etc/icecast2/icecast.xml"]
```
The `useradd` command updates the files inside the container (notably, `/etc/passwd`), while the `USER` directive tells Docker that it should switch to that effective UID when the container starts up.

I also took the liberty of setting an entrypoint and some arguments. When you specify both of them, you get the ability to provide "default" arguments (CMD) to the base command of the docker image (ENTRYPOINT).

That's all there is to the Icecast image. Running it requires a little bit more effort, but not much. Notably, we need to provide a port so that listeners can find the radio station, and so that source drivers can stream in their audio. We do this by forwarding ports from the Docker host *into* the container, like this:
```
$ docker run -p 8000:8000 icecast
[2021-03-22  19:11:48] WARN CONFIG/_parse_root Warning,  not configured, using default value "localhost". This will cause problems, e.g. with YP directory listings.
[2021-03-22  19:11:48] WARN CONFIG/_parse_root Warning,  not configured, using default value "Earth".
[2021-03-22  19:11:48] WARN CONFIG/_parse_root Warning,  contact not configured, using default value "icemaster@localhost".
[2021-03-22  19:11:48] WARN fserve/fserve_recheck_mime_types Cannot open mime types file /etc/mime.types
```
Inside the container namespaces, Icecast is binding all interfaces on port 80. Outside, Docker (with a little help from `iptables`) is forwarding traffic bound for the *host* on port 8000 into the container, on port 8000. (Incidentally, this is *usually* done via a Linux bridge and two ends of a *veth* pair, but that's a topic for a different, more nerdy post.)

You can see the port assignments in both the docker ps output:
```
$ docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED        STATUS                  PORTS                  NAMES
52993e74a1b3   icecast   "icecast2 -c /etc/ic…"   1 second ago   Up Less than a second   0.0.0.0:8000->8000/tcp   friendly_albattani
```
and also via netstat or lsof (pick your favorite!):
```
# lsof -nP -i TCP
com.docke 29389 jhunt   41u     IPv6  0xb3ce8809472e12f         0t0                 TCP *:8000 (LISTEN)
```
Let's leave Icecast to its thoughts, and move onto setting up our audio source.

##The Magic That Is LiquidSoap##
What is [LiquidSoap](https://www.liquidsoap.info/)? MAGIC.
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/oam6fwbrats63fwb31m5.png)
I truly, and fervently mean that. Their website is more modest:

LiquidSoap is a powerful and flexible language for describing audio and video streams. It offers a rich collection of operators that you can combine at will, giving you more power than you need for creating or transforming streams.

__more power than you need__ – that about sums up what we're going to use it for. We have audio files that we ripped right out of some Internet videos, and we have a radio broadcasting engine just waiting to be fed some juicy audio streams. We need a way to hook those up. And we're going to use LiquidSoap.

Here's the humble .liq script we're going to use:
```
output.icecast(%opus,
  host = "10.128.0.56",
  port = 8000,
  password = "sekrit",
  mount = "pirate-radio.opus",
  playlist.safe(reload=120,"/radio/playlist.m3u"))
```
But before we get ahead of ourselves, let's build out our Docker image for running that little gem.

You know the drill; start with Ubuntu and add the packages you need. As we've done for every image so far, we're going to set an `ENTRYPOINT` so that we can pretend our container *is* the `liquidsoap` command.
```
FROM ubuntu:20.04
RUN apt-get update \
 && apt-get install -y liquidsoap
USER nobody
ENTRYPOINT ["liquidsoap"]
```
As with Icecast, we set a `USER`, so that Liquidsoap doesn't get angry about us being root and all. This time, however, we're just going to use the default `nobody` account that comes with Ubuntu.

Convinced that the image is fit and fine, we can revisit that `.liq` script. Here's a version with some more comments.
```
# radio.liq
#
# This script streams audio from files on-disk,
# in an m3u playlist, to our Icecast server in
# opus format.
#
output.icecast(%opus,
  host = "10.128.0.56",        # where is Icecast?
  port = 8000,                 # what port is it bound on?
  password = "sekrit",         # what's the secret pass phrase?
  mount = "pirate-radio.opus", # what should we call this stream?
  
  # create an infallible playlist (hence 'safe')
  # using the files on-disk, in /radio.
  #
  playlist.safe(reload=120,"/radio/playlist.m3u"))
```
(Remember, LiquidSoap is a fully-fledged programming language. You can do all sorts of crazy things with it.)

Notably, we're going to output a stream to our Icecast server. Because the container has it's own network stack (containers!) I've specified the IP of the Docker host *itself* as the Icecast host endpoint. This causes packets to route out of the container and eventually find their way through our port-forwarding rules and into the other container. In the next part of this series, when we bring Docker Compose into the picture, we'll sand that particular rough patch down sufficiently.

Remember that *m3u* playlist we made while we were ingesting new audio files via youtube-dl? That's what this LiquidSoap script is going to consume while it builds its audio stream. I want to point out something that may not be obvious at first: the paths specified *in* the playlist must be valid when our script goes to consume them. Keeping everything always mounted at `/radio` suffices, but it's not the only way.

This is it, the final piece of the puzzle. Let's spin up that docker container and see if it works.
```
$ docker run -d --rm --name icecast -p 8000:8000 icecast
$ docker run --name liquid -v $PWD/radio:/radio 'output.icecast(%opus, host=...'
...
2021/03/22 20:09:17 [playlist(dot)m3u:3] Loading playlist...
2021/03/22 20:09:17 [playlist(dot)m3u:3] No mime type specified, trying autodetection.
2021/03/22 20:09:17 [playlist(dot)m3u:3] Playlist treated as format application/x-mpegURL
2021/03/22 20:09:17 [decoder:3] Method "OGG" accepted "/radio/Episode 0 - What's All This Then-D7hxDYBoHvQ.m4a.opus".
2021/03/22 20:09:17 [playlist(dot)m3u:3] Successfully loaded a playlist of 1 tracks.
2021/03/22 20:09:17 [decoder:3] Method "OGG" accepted "/radio/Episode 0 - What's All This Then-D7hxDYBoHvQ.m4a.opus".
2021/03/22 20:09:17 [playlist(dot)m3u:3] Prepared "/radio/Episode 0 - What's All This Then-D7hxDYBoHvQ.m4a.opus" (RID 2).
2021/03/22 20:09:17 [pirate-radio(dot)opus:3] Connecting mount pirate-radio.opus for source@192.168.88.225...
2021/03/22 20:09:17 [pirate-radio(dot)opus:3] Connection setup was successful.
2021/03/22 20:09:17 [clock.wallclock_main:3] Streaming loop starts, synchronized with wallclock.
```
The logs for the LiquidSoap container clearly show that the source was able to connect and start streaming in audio. Now we can visit the web interface (http://<docker host>:8000), which should look something like this:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lq5eew67hyyr7cdoqe2y.png)
Congratulations. You're officially an Internet radio DJ. Now, click that play button, enjoy the show, and go brainstorm a better radio handle.

In the next part of the series, we'll gather up all of the port forwarding rules, commands, and volume bind-mounts, and wrap it all up in a neat little package using Docker Compose. This will make our lives easier as we iterate on the whole system.