How to Deploy Hashicorp Nomad Cluster on Vultr
Tags: docker, infrastructureascode, kubernetes, cloudcomputing
Published: 2024-06-18 07:43:30

## Introduction

Hashicorp Nomad is a workload orchestrator to deploy and manage applications across large clusters of servers. Nomad is a single binary application which can run in both server and client modes. The server manages the cluster state and applications are deployed in the client machine.

Nomad support running various type of deployments using docker, binary files, java jar files, and Linux VMs using QEMU driver.

This article demonstrates step-by-step process to build a Nomad cluster on Vultr.

## Prerequisites

Before you begin, you must have basic knowledge of Linux and Vultr services.

* [Vultr VPC 2.0](https://docs.vultr.com/introduction-to-vultr-virtual-private-cloud-2-0)
* [Vultr Startup Script](https://docs.vultr.com/vultr-startup-scripts-quickstart-guide)
* [Vultr Load Balancer](https://docs.vultr.com/vultr-load-balancers)

## Example Nomad Cluster

The cluster consists of one Nomad server operating in a VM and three Nomad clients, each running on separate VMs, all forming a single compute plane. After deployment, the server will be accessible in the public IP on port 4646. To access client application, a load balancer will be attached to all clients on port 80. The entire cluster is located inside a VPC with a private IP range.

[![nomad-diagram.png](https://i.postimg.cc/qqQgH71f/nomad-diagram.png)](https://postimg.cc/N29Q1Qhb)
  
## Set Up VPC 2.0

1. Open the [Vultr Customer Portal](https://my.vultr.com/)
2. Navigate to **Network** and click **VPC 2.0** on the main navigation menu.
3. Choose Add **VPC 2.0 Network**  

    [![create-vpc2-0.png](https://i.postimg.cc/httTjkmb/create-vpc2-0.png)](https://postimg.cc/w7n74b07)

4. Select a location.
5. Configure an IP range in the VPC, for example, 10.1.0.0/20.
6. Name the VPC 2.0 and click **Add Network**  

    [![choose-vpc2-0-options.png](https://i.postimg.cc/qq2Xs2Bs/choose-vpc2-0-options.png)](https://postimg.cc/N2fHY9HF)

This creates a vultr VPC 2.0 where the Nomad cluster will be deployed.

## How to use Startup Script to Install Nomad and Docker

The bash script given below install Hashicorp Nomad and Docker in a Ubuntu VM.
A startup Script in Vultr allows the script to reuse by multiple VMs.
  ```bash
    #!/usr/bin/env bash
    set -e

    # Disable interactive apt prompts
    export DEBIAN_FRONTEND=noninteractive

    NOMAD_VERSION="${NOMAD_VERSION:-1.7.3}"

    # Update packages
    sudo apt-get -y update
    # Install software-properties-common
    sudo apt-get install -y software-properties-common

    # Add HashiCorp GPG key
    curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
    # Add HashiCorp repository
    sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
    # Update packages again
    sudo apt-get -y update

    # Install Nomad
    sudo apt-get install -y nomad="${NOMAD_VERSION}"-1


    # Disable the firewall
    sudo ufw disable || echo "ufw not installed"

    # Install Docker and associated dependencies
    sudo apt-get -y update
    sudo apt-get -y install \
        ca-certificates \
        curl \
        gnupg
    # Add Docker’s official GPG key
    sudo install -m 0755 -d /etc/apt/keyrings
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
    sudo chmod a+r /etc/apt/keyrings/docker.gpg
    echo \
        "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
        "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" |
        sudo tee /etc/apt/sources.list.d/docker.list >/dev/null  
    sudo apt-get -y update

    # Install Docker and required packages
    sudo apt-get -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
          
    # Create daemon.json if it doesn't exist
    if [ ! -f /etc/docker/daemon.json ]; then
        sudo touch /etc/docker/daemon.json
    fi

    # Restart Docker
    sudo systemctl restart docker

    # Add the current user to the docker group
    sudo usermod -aG docker ubuntu
  ```

1. Open the [Vultr Customer Portal](https://my.vultr.com/)
2. Navigate to **Orchestration** and click **Scripts** on the main navigation menu.
3. Choose Add **Startup Script**

    [![create-startup-script.png](https://i.postimg.cc/tRvrpNwc/create-startup-script.png)](https://postimg.cc/PNZbMYtb)

4. Give the script a **name** and choose Type as **Boot**
5. Copy the above given bash script and paste it in the **Script** location
6. Click Add Script

    [![choose-startup-script-options.png](https://i.postimg.cc/pVsSyHhq/choose-startup-script-options.png)](https://postimg.cc/0rJnHhCw)

This creates a Vultr startup script that can be used to deploy multiple VMs.

## Nomad server User Data

Nomad uses a config file to run the server with its required configuration. The default file location is /etc/nomad.d/nomad.hcl. The bash script shown below will add the Nomad server configuration to the config file and run the server as a systemd service.

  ```bash
    #!/usr/bin/env bash
    set -Eeuo pipefail

    # Add Nomad client configuration to /etc/nomad.d/ folder

    cat <<EOF >/etc/nomad.d/nomad.hcl
    datacenter = "dc1"
    data_dir   = "/opt/nomad/data"
    bind_addr = "0.0.0.0"
    log_level = "INFO"

    advertise {
      http = "{{ GetInterfaceIP \"enp8s0\" }}"
      rpc  = "{{ GetInterfaceIP \"enp8s0\" }}"
      serf = "{{ GetInterfaceIP \"enp8s0\" }}"
    }

    server {
      enabled          = true
      bootstrap_expect = "1"
      encrypt          = "z8geXx7U+JPk6u/vlBRDhh81h5W12AXBN+7AUo5eXMI="
      server_join {
        retry_join = ["127.0.0.1"]
      }
    }

    acl {
      enabled = false
    }
    EOF

    sudo systemctl enable --now nomad
    sudo systemctl restart nomad
  ```

* The Vultr **Cloud-Init User-Data** is a script that can be used to launch the server when it is booted for the first time.
* Add the script to the **Cloud-Init User-Data** during the server launch.

## Deploying Nomad server

1. Open the [Vultr Customer Portal](https://my.vultr.com/)
2. Navigate to **Compute** on the main navigation menu.
3. Choose **Deploy Server**

    [![deploy-server.png](https://i.postimg.cc/dtJfV0V7/deploy-server.png)](https://postimg.cc/ThH0kfmG)

4. Select the **Type** of Server
5. Select Location same as VPC
6. Select **Ubuntu 22.04 LTS** as the Operating System
7. Select a **Plan**
8. In the **Additional Features** section select **Virtual Private Cloud 2.0** and **Cloud-Init User-Data**

    [![server-config-1.png](https://i.postimg.cc/JhRCwQHF/server-config-1.png)](https://postimg.cc/75tW00V1)

9. Copy the Nomad server User Data from the previous section and add it in the user data input section
10. Select the VPC name created in the previous section

    [![server-vpc-config.png](https://i.postimg.cc/D0M1hmMJ/server-vpc-config.png)](https://postimg.cc/fSYVjWZD)

11. In the **Server Settings** section, choose the **nomad-startup-script** created.
12. Add **Server Hostname & Label**
13. Click **Deploy Now** to launch the Nomad server

    [![server-config-2.png](https://i.postimg.cc/ydkZPcjJ/server-config-2.png)](https://postimg.cc/mzWkZc0R)

The Nomad server is now deployed in the VPC.

## Fetching the Private IP of Nomad server

To connect Nomad client with server, the private IP of Nomad server is required.

1. Select **VPC 2.0** in the navigation menu and click the VPC ID

    [![private-IP-1.png](https://i.postimg.cc/tTSsnJHX/private-ip-1.png)](https://postimg.cc/vgnYpQBj)

2. Note down the private IP of the Nomad server from the **Attached Nodes**

    [![private-IP-2.png](https://i.postimg.cc/Y0dz9Mdg/private-ip-2.png)](https://postimg.cc/7JT2t88Y)

## Nomad client User Data

The given bash script run a Nomad client as a systemd service along with its configuration during a server boot. Update the **private IP** of the Nomad server in the **server_join** block as shown below to make the script connect with the server.

Add the modified script to the **Cloud-Init User-Data** during the server launch.

  ```bash
    #!/usr/bin/env bash
    set -Eeuo pipefail

    # Add Nomad server configuration to /etc/nomad.d/ folder

    cat <<EOF >/etc/nomad.d/nomad.hcl
    datacenter = "dc1"
    data_dir  = "/opt/nomad/data"
    bind_addr = "0.0.0.0"
    log_level = "INFO"
        
    # add nomad advertise address
    advertise {
      http = "{{ GetInterfaceIP \"enp8s0\" }}"
      rpc  = "{{ GetInterfaceIP \"enp8s0\" }}"
      serf = "{{ GetInterfaceIP \"enp8s0\" }}"
    }


    # Enable the client
    client {
      enabled = true
      options {
        "driver.raw_exec.enable"    = "1"
        "docker.privileged.enabled" = "true"
      }
      # Configure the server_join option
      server_join {
        retry_join = [ "10.1.0.3" ]
      }
      
      # Configure the network interface
      network_interface = "enp8s0"

    }

    # Enable the docker plugin
    plugin "docker" {
      config {
        endpoint = "unix:///var/run/docker.sock"

        volumes {
          enabled      = true
          selinuxlabel = "z"
        }
        allow_privileged = true
      }
    }
    EOF

    # Enables nomad systemd service
    sudo systemctl enable --now nomad

    # Runs nomad service
    sudo systemctl restart nomad
  ```

## Deploying Nomad clients

1. Open the [Vultr Customer Portal](https://my.vultr.com/)
2. Navigate to **Compute** on the main navigation menu.
3. Choose **Deploy Server**

    [![deploy-server.png](https://i.postimg.cc/dtJfV0V7/deploy-server.png)](https://postimg.cc/ThH0kfmG)

4. Select the **Type** of Server
5. Select Location same as VPC
6. Select **Ubuntu 22.04 LTS** as the Operating System
7. Select a **Plan**
8. In the **Additional Features** section select **Virtual Private Cloud 2.0** and **Cloud-Init User-Data**

    [![server-config-1.png](https://i.postimg.cc/JhRCwQHF/server-config-1.png)](https://postimg.cc/75tW00V1)

9. Copy the **Nomad client User Data** from the previous section and add it in the user data input section
10. Select the VPC name created in the previous section

    [![server-vpc-config.png](https://i.postimg.cc/D0M1hmMJ/server-vpc-config.png)](https://postimg.cc/fSYVjWZD)

11. In the **Server Settings** section, choose the **nomad-startup-script** created.
12. Increase the **Server Qty** to 3
13. Add **Server Hostname & Label** for each server
14. Click **Deploy Now** to launch all 3 Nomad clients

    [![nomad-client-deploy.png](https://i.postimg.cc/nLx4whTG/nomad-client-deploy.png)](https://postimg.cc/G4MT4rW4)

After both the server and clients are deployed in the VPC, a cluster is formed between the server and clients with the help of the  **server_join** configuration in Nomad.

[![vultr-servers.png](https://i.postimg.cc/764GrSPQ/vultr-servers.png)](https://postimg.cc/qtmM8h38)

## Accessing Nomad UI

Fetch the public IP of **Nomad server** from Vultr UI and browse <http://public-ip:4646>. The Nomad UI is accessible on port 4646, as shown in the screenshot below.

[![nomad-ui-front.png](https://i.postimg.cc/9MgxnXC0/nomad-ui-front.png)](https://postimg.cc/MMRbcScJ)

Navigate to the **Clients** and **Server** page in sidebar to view connected clients and servers.

* Nomad server

[![nomad-server.png](https://i.postimg.cc/2STSJgbM/nomad-server.png)](https://postimg.cc/RW6mJsTL)

* Nomad clients

[![nomad-ui-clients.png](https://i.postimg.cc/PxTvZ1Dn/nomad-ui-clients.png)](https://postimg.cc/VSZNPr4K)

## Attach Vultr Load Balancer to Nomad Clients

1. Open the [Vultr Customer Portal](https://my.vultr.com/)
2. Select **Load Balancers** on the main navigation menu and select **Add Load Balancer** option.

    [![create-load-balancer.png](https://i.postimg.cc/q7CxQcgt/create-load-balancer.png)](https://postimg.cc/hJcmtdLg)

3. Select the same location as the VPC 2.0
4. Use the default **Load Balancer Configuration** and **Forwarding Rules**
5. Select the VPC Network created in the previous section
6. Choose **Add Load Balancer**
7. After the Load Balancer is created, add the nomad clients as targets.

    [![load-balancer-attach-instances.png](https://i.postimg.cc/gkC9Ns2R/load-balancer-attach-instances.png)](https://postimg.cc/VdWHNjML)

You have attached a Load Balancer to the Nomad clients.

## Deploying Sample Web Application

You can deploy web application in a Nomad cluster using Nomad Job files. A Job file must include a **.nomad** extension written in the HCL (Hashicorp configuration language).

A sample nomad job file is shown below, which deploys a web api on the Nomad cluster.
The api is deployed on port 80 and uses a docker image called **traefik/whoami**, which returns the client IP address and port number when invoked.
  ```HCL
    job "webapp" {
      datacenters = ["dc1"]

      type = "service"

      group "webapp" {
        count = 3

        network {
           mode = "host"
           port "http" {
             to = 80
             static = 80
           }
        }

        task "server" {
          env {
            WHOAMI_PORT_NUMBER = "${NOMAD_PORT_http}"
          }

          driver = "docker"

          config {
            image = "traefik/whoami"
            ports = ["http"]
          }
        }
      }
    }
  ```

1. Open Nomad UI
2. Navigate to **Jobs** and select **Run Job**

    [![nomad-job.png](https://i.postimg.cc/d0ysHgJ2/nomad-job.png)](https://postimg.cc/d7JcDHk3)

3. Add the above job file in text input and click **Plan**

    [![nomad-job-plan.png](https://i.postimg.cc/3JQvX5HH/nomad-job-plan.png)](https://postimg.cc/7bXh4R6B)

4. Click **Run**

    [![nomad-job-run.png](https://i.postimg.cc/90RH07zG/nomad-job-run.png)](https://postimg.cc/Lgpb0htX)

5. After the job is in running state, Open the browser and navigate to load balancer IP on port 80, and you can see the response from the application container. Refresh the page to see the load balancer in action.

    [![nomad-job-running.png](https://i.postimg.cc/kgDYmm6b/nomad-job-running.png)](https://postimg.cc/ZWtx8XLb)

    [![final-response.png](https://i.postimg.cc/9XKv1bx2/final-response.png)](https://postimg.cc/3WjSxX5f)

## Conclusion

You have created a Nomad cluster in the VPC and deployed a web application with load balancer. You can now access the Nomad UI and deploy various applications on it.

## More Information

For more information, please see:

* [Hashicorp Nomad documentation](https://developer.hashicorp.com/nomad/intro)
* [Hashicorp Nomad website](https://www.nomadproject.io/)