How to Deploy More Complex Microservice Apps with Managed Kubernetes
Tags: kubernetes, tutorial, microservices
Published: 2020-04-02 15:13:15

In the following tutorial, the Platform9 technical team shows you how to deploy a more complex microservice. The idea is to help you gain familiarity with a managed Kubernetes service and to show you how you can leverage Platform9 Managed Kubernetes for more advanced scenarios.

In this example, we are going to see the deployment of:

 - A Redis master
 - Multiple Redis slaves
 - A sample guestbook application that uses Redis as a store

We assume that you have already set up a Platform9 cluster with at least one node, and the cluster is ready.

Let’s start with the Redis parts.

##Deploying and exposing a Redis Cluster##
In this tutorial, we are going to expand our examples with deploying a more complex microservice. The idea is to make you more comfortable with the platform and to show you how you can leverage it for more advanced scenarios.

Redis is a key-value in-memory store that is used mainly as a cache service. In order to set up Clustering for Data Replication, we need a Redis instance that acts as Master, together with additional instances as slaves. Then the guestbook application can use this instance to store data. The Redis master will propagate the writes to the slave nodes.

We can initiate a Redis Master deployment in a few different ways: either using the kubectl tool, the Platform9 UI or the Kubernetes UI. For convenience, we use the kubectl tool as it’s the most commonly understood in tutorials.

First, we need to create a Redis Cluster Deployment. Looking at their [documentation here](https://redis.io/topics/cluster-tutorial), to set up a cluster, we need some configuration properties. We can leverage kubernetes configmaps to store and reference them in the deployment spec.

We need to save a script and a redis.conf file that is going to be used to configure the master and slave nodes.

Create the following config redis-cluster.config.yml

With these values:
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="s">$ cat redis-cluster.config.yml</span>
<span class="nn">---</span>
<span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">ConfigMap</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">redis-cluster-config</span>
<span class="na">data</span><span class="pi">:</span>
  <span class="s">update-ip.sh</span><span class="pi">:</span> <span class="pi">|</span>
    <span class="s">#!/bin/sh</span>
    <span class="s">sed -i -e "/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${IP}/" /data/nodes.conf</span>
    <span class="s">exec "$@"</span>
  <span class="s">redis.conf</span><span class="pi">:</span> <span class="pi">|+</span>
    <span class="s">cluster-enabled yes</span>
    <span class="s">cluster-config-file /data/nodes.conf</span>
    <span class="s">appendonly yes</span>
</code></pre></div></div>

We define a script that will insert an IP value to the node.conf file. This is to fix an issue with Redis as [referenced here](https://github.com/antirez/redis/issues/4645). We use this script every time we deploy a new redis image.

Then we have the redis.conf, which applies the minimal cluster configuration.

Apply this spec into the cluster:
<code><span class="nv">$ </span>kubectl apply <span class="nt">-f</span> redis-cluster.config.yml
</code>

Then verify that it exists in the list of configmaps:

<code><span class="nv">$ </span>kubectl get configmaps

</code>

Next we need to define a spec for the redis cluster instances. We can use a Deployment or a StatefulSet to define 3 instances:

Here is the spec: redis-cluster.statefulset.yml

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="s">$ cat redis-cluster.statefulset.yml</span>
<span class="nn">---</span>
<span class="na">apiVersion</span><span class="pi">:</span> <span class="s">apps/v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">StatefulSet</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">redis-cluster</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">serviceName</span><span class="pi">:</span> <span class="s">redis-cluster</span>
  <span class="na">replicas</span><span class="pi">:</span> <span class="m">6</span>
  <span class="na">selector</span><span class="pi">:</span>
    <span class="na">matchLabels</span><span class="pi">:</span>
      <span class="na">app</span><span class="pi">:</span> <span class="s">redis-cluster</span>
  <span class="na">template</span><span class="pi">:</span>
    <span class="na">metadata</span><span class="pi">:</span>
      <span class="na">labels</span><span class="pi">:</span>
        <span class="na">app</span><span class="pi">:</span> <span class="s">redis-cluster</span>
    <span class="na">spec</span><span class="pi">:</span>
      <span class="na">containers</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">redis</span>
        <span class="na">image</span><span class="pi">:</span> <span class="s">redis:5.0.7-alpine</span>
        <span class="na">ports</span><span class="pi">:</span>
        <span class="pi">-</span> <span class="na">containerPort</span><span class="pi">:</span> <span class="m">6379</span>
          <span class="na">name</span><span class="pi">:</span> <span class="s">client</span>
        <span class="pi">-</span> <span class="na">containerPort</span><span class="pi">:</span> <span class="m">16379</span>
          <span class="na">name</span><span class="pi">:</span> <span class="s">gossip</span>
        <span class="na">command</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">/conf/update-ip.sh"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">redis-server"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">/conf/redis.conf"</span><span class="pi">]</span>
        <span class="na">env</span><span class="pi">:</span>
        <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">IP</span>
          <span class="na">valueFrom</span><span class="pi">:</span>
            <span class="na">fieldRef</span><span class="pi">:</span>
              <span class="na">fieldPath</span><span class="pi">:</span> <span class="s">status.podIP</span>
        <span class="na">volumeMounts</span><span class="pi">:</span>
        <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">conf</span>
          <span class="na">mountPath</span><span class="pi">:</span> <span class="s">/conf</span>
          <span class="na">readOnly</span><span class="pi">:</span> <span class="no">false</span>
        <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">data</span>
          <span class="na">mountPath</span><span class="pi">:</span> <span class="s">/data</span>
          <span class="na">readOnly</span><span class="pi">:</span> <span class="no">false</span>
      <span class="na">volumes</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">conf</span>
        <span class="na">configMap</span><span class="pi">:</span>
          <span class="na">name</span><span class="pi">:</span> <span class="s">redis-cluster-config</span>
          <span class="na">defaultMode</span><span class="pi">:</span> <span class="m">0755</span>
  <span class="na">volumeClaimTemplates</span><span class="pi">:</span>
  <span class="pi">-</span> <span class="na">metadata</span><span class="pi">:</span>
      <span class="na">name</span><span class="pi">:</span> <span class="s">data</span>
    <span class="na">spec</span><span class="pi">:</span>
      <span class="na">accessModes</span><span class="pi">:</span> <span class="pi">[</span> <span class="s2">"</span><span class="s">ReadWriteOnce"</span> <span class="pi">]</span>
      <span class="na">resources</span><span class="pi">:</span>
        <span class="na">requests</span><span class="pi">:</span>
          <span class="na">storage</span><span class="pi">:</span> <span class="s">1Gi</span>
</code></pre></div></div>

In the above step we defined a few things:

 - An IP environment variable that we need in the update-ip.sh script that we defined in the configmap earlier. This is the pod-specific IP address using the Downward API.
 - Some shared volumes including the configmap that we defined earlier.
 - Two container ports – 6379 and 16379 – for the gossip protocol.

With this spec we can deploy the Redis cluster instances:

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl apply <span class="nt">-f</span> redis-cluster.statefulset.yml
</code></pre></div></div>

Once we verify that we have the deployment ready, we need to perform the last step, which is bootstrapping the cluster. Consulting the documentation here for [creating the cluster](https://redis.io/topics/cluster-tutorial#creating-the-cluster), we need to ssh into one of the instances and run the redis-cli cluster create command. For example taken from the docs:

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>redis-cli <span class="nt">--cluster</span> create 127.0.0.1:7000 127.0.0.1:7001 <span class="se">\</span>
127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 <span class="se">\</span>
<span class="nt">--cluster-replicas</span> 1
</code></pre></div></div>

To do that in our case, we need to get the local pod IPs of the instances and feed them to that command.

We can query the IP using this command:

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl get pods <span class="nt">-l</span> <span class="nv">app</span><span class="o">=</span>redis-cluster <span class="nt">-o</span> <span class="nv">jsonpath</span><span class="o">=</span><span class="s1">'{range.items[*]}{.status.podIP}:6379 '</span>
</code></pre></div></div>

So if we save them in a variable or a file, we can pipe them at the end of the redis-cli command:

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>POD_IPS <span class="o">=</span> <span class="si">$(</span>kubectl get pods <span class="nt">-l</span> <span class="nv">app</span><span class="o">=</span>redis-cluster <span class="nt">-o</span> <span class="nv">jsonpath</span><span class="o">=</span><span class="s1">'{range.items[*]}{.status.podIP}:6379 '</span><span class="si">)</span>
</code></pre></div></div>

Then we can run the following command:

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl <span class="nb">exec</span> <span class="nt">-it</span> redis-cluster-0 <span class="nt">--</span> redis-cli <span class="nt">--cluster</span> create <span class="nt">--cluster-replicas</span> 1 <span class="nv">$POD_IPS</span>
</code></pre></div></div>

If everything is OK, you will see the following prompt. Enter ‘yes’ to accept and continue:

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Can I <span class="nb">set </span>the above configuration? <span class="o">(</span><span class="nb">type</span> <span class="s1">'yes'</span> to accept<span class="o">)</span>: <span class="nb">yes</span>
<span class="o">&gt;&gt;&gt;</span> Nodes configuration updated
<span class="o">&gt;&gt;&gt;</span> Assign a different config epoch to each node
<span class="o">&gt;&gt;&gt;</span> Sending CLUSTER MEET messages to <span class="nb">join </span>the cluster
Waiting <span class="k">for </span>the cluster to <span class="nb">join</span>
........

<span class="o">[</span>OK] All nodes agree about slots configuration.
<span class="o">&gt;&gt;&gt;</span> Check <span class="k">for </span>open slots...
<span class="o">&gt;&gt;&gt;</span> Check slots coverage...
<span class="o">[</span>OK] All 16384 slots covered.
</code></pre></div></div>

Then we can verify the cluster state by running the cluster info command:

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl <span class="nb">exec</span> <span class="nt">-it</span> redis-cluster-0 <span class="nt">--</span> redis-cli cluster info

cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:28
cluster_stats_messages_pong_sent:34
cluster_stats_messages_sent:62
cluster_stats_messages_ping_received:29
cluster_stats_messages_pong_received:28
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:62
</code></pre></div></div>

Before we continue deploying the guestbook app, we need to offer a unified service frontend for the Redis Cluster so that it’s easily discoverable in the cluster.

Here is the service spec: redis-cluster.service.yml

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="s">$ cat redis-cluster.service.yml</span>

<span class="nn">---</span>
<span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Service</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">redis-master</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">type</span><span class="pi">:</span> <span class="s">ClusterIP</span>
  <span class="na">ports</span><span class="pi">:</span>
  <span class="pi">-</span> <span class="na">port</span><span class="pi">:</span> <span class="m">6379</span>
    <span class="na">targetPort</span><span class="pi">:</span> <span class="m">6379</span>
    <span class="na">name</span><span class="pi">:</span> <span class="s">client</span>
  <span class="pi">-</span> <span class="na">port</span><span class="pi">:</span> <span class="m">16379</span>
    <span class="na">targetPort</span><span class="pi">:</span> <span class="m">16379</span>
    <span class="na">name</span><span class="pi">:</span> <span class="s">gossip</span>
  <span class="na">selector</span><span class="pi">:</span>
    <span class="na">app</span><span class="pi">:</span> <span class="s">redis-cluster</span>
</code></pre></div></div>

We expose the cluster as redis-master here, as the guestbook app will be looking for a host service to connect to with that name.

Once we apply this service spec, we can move on to deploying and exposing the Guestbook Application:

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl apply <span class="nt">-f</span> redis-cluster.service.yml
</code></pre></div></div>

##Deploying and exposing a GuestBook Application##
The guestbook application is a simple php script that shows a form to submit a message. Initially it will attempt to connect to either the redis-master host or the redis-slave hosts. It needs the **GET_HOSTS_FROM** environment variable set pointing to the file with the following variables: **REDIS_MASTER_SERVICE_HOST**: of the master **REDIS_SLAVE_SERVICE_HOST**: of the slave

First, let’s define the deployment spec bellow:

php-guestbook.deployment.yml

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="s">$ cat php-guestbook.deployment.yml</span>

<span class="nn">---</span>
<span class="na">apiVersion</span><span class="pi">:</span> <span class="s">apps/v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Deployment</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">guestbook</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">replicas</span><span class="pi">:</span> <span class="m">1</span>
  <span class="na">selector</span><span class="pi">:</span>
    <span class="na">matchLabels</span><span class="pi">:</span>
      <span class="na">app</span><span class="pi">:</span> <span class="s">guestbook</span>
  <span class="na">template</span><span class="pi">:</span>
    <span class="na">metadata</span><span class="pi">:</span>
      <span class="na">labels</span><span class="pi">:</span>
        <span class="na">app</span><span class="pi">:</span> <span class="s">guestbook</span>
    <span class="na">spec</span><span class="pi">:</span>
      <span class="na">containers</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">php-redis</span>
        <span class="na">image</span><span class="pi">:</span> <span class="s">gcr.io/google-samples/gb-frontend:v6</span>
        <span class="na">resources</span><span class="pi">:</span>
          <span class="na">requests</span><span class="pi">:</span>
            <span class="na">cpu</span><span class="pi">:</span> <span class="s">150m</span>
            <span class="na">memory</span><span class="pi">:</span> <span class="s">150Mi</span>
        <span class="na">env</span><span class="pi">:</span>
        <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">GET_HOSTS_FROM</span>
          <span class="na">value</span><span class="pi">:</span> <span class="s">env</span>
        <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">REDIS_MASTER_SERVICE_HOST</span>
          <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">redis-master"</span>
        <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">REDIS_SLAVE_SERVICE_HOST</span>
          <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">redis-master"</span>
        <span class="na">ports</span><span class="pi">:</span>
        <span class="pi">-</span> <span class="na">containerPort</span><span class="pi">:</span> <span class="m">80</span>
</code></pre></div></div>

The code of the gb-frontend image is [located here](https://github.com/kubernetes/examples/blob/master/guestbook/php-redis/guestbook.php).

Next is the associated service spec:

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span>
<span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Service</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">guestbook-lb</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">type</span><span class="pi">:</span> <span class="s">NodePort</span>
  <span class="na">ports</span><span class="pi">:</span>
  <span class="pi">-</span> <span class="na">port</span><span class="pi">:</span> <span class="m">80</span>
  <span class="na">selector</span><span class="pi">:</span>
      <span class="na">app</span><span class="pi">:</span> <span class="s">guestbook</span>

</code></pre></div></div>

Note: **NodePort** will assign a random port over the public IP of the Node. In either case, we get a public host:port pair where we can inspect the application. Here is a screenshot of the app after we deployed it:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/sdhdxvj6n1liv6611olw.png)

##Cleaning up##
Once we have finished experimenting with the application, we can clean up the resources and all the servers by issuing kubectl delete statements. A convenient way is to delete by labels. For example:

<code><span class="nv">$ </span>kubectl delete statefulset redis-cluster
<span class="nv">$ </span>kubectl delete service redis-master
<span class="nv">$ </span>kubectl delete deployment guestbook
<span class="nv">$ </span>kubectl delete service guestbook-lb
<span class="nv">$ </span>kubectl delete configmap redis-cluster-config
<span class="nv">$ </span>kubectl delete pvc <span class="nt">-l</span> <span class="nv">app</span><span class="o">=</span>redis-cluster
</code>

##Next Steps##
Deploying multi-container systems in Kubernetes with Platform9 is no different than usual. The big difference is that you gain a quality of server and a maintenance-free Kubernetes experience with first-class support for troubleshooting issues. That’s not to mention that you can host the cluster on your own bare-metal server or AWS, eliminating any vendor lock-ins. For more information visit [Platform9 Managed Kubernetes page](https://bit.ly/2V7NnFV).