Apache Superset and Azure - multi-container application deployment
Tags: superset, azure, python, datascience
Published: 2022-05-10 17:13:24

In this article you’ll learn the steps to deploy Apache Superset on Azure using Azure Kubernetes Service. 
    &nbsp;

## Background
I work on improving the Python Developer Experience on Microsoft Azure and I've spoken with lots of Python developers from around the world. One thing I've heard repeatedly is that **many Python devs like to run production code in containers**. And another thing I've learned is many apps require multiple containers.

I also like to do some data analysis on the side and recently ran across [Apache Superset](https://github.com/apache/superset) which describes itself as a "modern data exploration and data visualization platform". Coincidentally, **Superset has a lot of Python code and can be deployed in containers** (nine of them at current count!)
![Superset image from https://github.com/apache/superset](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xo3ozpzn8u0ce9qd34uw.png)

There are several options for running container-based apps on Azure. Because **Superset requires several containers** (9 at current count) I decided to use Azure Kubernetes Service (AKS). In a future post I plan to share how you can use Azure's new managed service for containers, Azure Container Apps, instead.

While I did not find an existing document or post to help me complete my journey in a straightforward manner, here are several docs that helped:

- [Superset - Running on Kubernetes](https://superset.apache.org/docs/installation/running-on-kubernetes)
- [Quickstart: Develop on Azure Kubernetes Service (AKS) with Helm](https://docs.microsoft.com/en-us/azure/aks/quickstart-helm)
- [Install existing applications with Helm in Azure Kubernetes Service (AKS)](https://docs.microsoft.com/en-us/azure/aks/kubernetes-helm)
- [Quickstart: Deploy an Azure Kubernetes Service cluster using the Azure CLI](https://docs.microsoft.com/en-us/azure/aks/learn/quick-kubernetes-deploy-cli)

&nbsp;

## The set up
Some of the syntax below may vary with your choice of OS or shell.

**Prerequisites** you will need:
- [Git](https://git-scm.com/downloads)
- [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli)
- [Helm CLI](https://helm.sh/docs/intro/install/)
- An Azure subscription. If you don't already have one, you can [create one](https://azure.microsoft.com/free) for free.
- [Docker Desktop](https://www.docker.com/products/docker-desktop/)

&nbsp;
## Get the Superset app and create your AKS Cluster
1. **Clone the Apache Superset repo**
    ```powershell
    git clone https://github.com/apache/superset
    
    cd superset
    ```
    &nbsp;
1. **Sign in to the Azure CLI**
You can sign in by using the [az login](https://docs.microsoft.com/en-us/cli/azure/reference-index?view=azure-cli-latest#az-login) command
    ```powershell
    az login
    ```
    &nbsp;
1. **Create a new AKS cluster with ACR integration**
Azure Container Registry (ACR) is kind of like Docker Hub. It's a place to store your container images so you can run your application in your Azure Kubernetes Service (AKS) cluster. We will use it to store the Superset container images.
Use `az acr create` to create an ACR named _[yourname]supersetacr_ in a resource group called _supersetrg_. 
Replace `[yourname]` below with a name of your choice.
    ```powershell
    # Create a Resource Group to hold your ACR and AKS resources
    # Feel free to use a location closer to you
    az group create supersetrg --location westus2
    
    # Create an Azure Container Registry
    az acr create -n [yourname]supersetacr -g supersetrg --sku basic

    # Create an AKS cluster with ACR integration
    az aks create -n supersetaks -g supersetrg --generate-ssh-keys --attach-acr [yourname]supersetacr
    ```
    &nbsp;

    When you create the ACR, you will see a blob of JSON. Here are a couple of **important values to notice**:
    ```text
    ...
    "location": "westus2",
    "loginServer": "[yourname]supersetacr.azurecr.io",
    "name": "[yourname]2supersetacr",
    ...
    "provisioningState": "Succeeded",
    "publicNetworkAccess": "Enabled",
    "resourceGroup": "supersetrg",
    "sku": {
      "name": "Basic",
      "tier": "Basic"
    },
    ...
    ```
    &nbsp;

    It will take a few minutes to create the AKS cluster. When it's done you will see an even larger blob of JSON. Here are a couple of **important values to notice** in this one:
    ```text
    ...
      "osDiskSizeGb": 128,
      "osDiskType": "Managed",
      "osSku": "Ubuntu",
      "osType": "Linux",
    ...
      "provisioningState": "Succeeded",
    ...
      "vmSize": "Standard_DS2_v2",
    ...
    "azurePortalFqdn": "supersetak-supersetrg-2223f9-06aacbbd.portal.hcp.westus2.azmk8s.io",
    ...
    "fqdn": "supersetak-supersetrg-2223f9-06aacbbd.hcp.westus2.azmk8s.io",
    ...
    "kubernetesVersion": "1.22.6",
    ...
    "nodeResourceGroup": "MC_supersetrg_supersetaks_westus2",
    ...
    ```
    &nbsp;
    Additional ways to integrate ACR with AKS: [3 Ways to integrate ACR with AKS](https://www.thorsten-hans.com/3-ways-to-integrate-acr-with-aks/)
    &nbsp;

1. **Push the Superset container images into your ACR**
Superset uses two container images, which you can see in the superset repo in the `/helm/superset/values.yaml` file:
    ```yaml
    image: 
      repository: apache/superset
    ...
    initImage: 
      repository: busybox
    ```
    &nbsp;

    While ACR can technically pull images directly from Docker Hub, the throttling that Docker Hub has recently implemented means it could take a while (sometimes a *long* while) for the images to get into your ACR. Instead **we're going to pull the images to your machine and then push them to ACR**. Don't forget to replace `[yourname]` with the name you chose previously.
    ```powershell
    # First - login to your ACR so Docker can push to it
    az acr login -n [yourname]supersetacr.azurecr.io
    
    # Pull the Superset image
    docker pull apache/superset
    # Tag the image using your ACR login server name 
    docker tag apache/superset [yourname]supersetacr.azurecr.io/superset
    # Push the image to ACR
    docker push [yourname]supersetacr.azurecr.io/superset
    
    # Pull the Busybox image
    docker pull busybox
    # Tag the image using your ACR login server name
    docker tag busybox [yourname]supersetacr.azurecr.io/busybox
    # Push the image to ACR
    docker push [yourname]supersetacr.azurecr.io/busybox
    ```
    &nbsp;

1. **Create a my_values.yaml file to override defaults**
In the previous step we gave the images new tags and now we need to create override the defaults in /helm/superset/values.yaml so our new images will be used.
    Add the following to a new file called `my_values.yaml`, being sure to replace `[yourname]` with the name you chose previously:
    ```yaml
    image:
      repository: [yourname]supersetacr.azurecr.io/superset
    
    initImage:
      repository: [yourname]supersetacr.azurecr.io/busybox
    ```
    &nbsp;

    While we're in the my_values.yaml file, in order to **expose the Superset website**, we need to change service type from `ClusterIP` to `LoadBalancer` and to make it easier to browse we will set the port to `80`. Add these lines to `my_values.yaml` as well:
    ```yaml
    # Set type to 'LoadBalancer' so we can browse it
    service:
      type: LoadBalancer
      port: 80
    ```
    &nbsp;

    And unless you're planning on deploying to production, I recommend that you **load up some examples** to play with. Just make one more addition to the `my_values.yaml` file:
    ```yaml
    # Load Superset Examples
    init:
      loadExamples: true
    ```
    &nbsp;

    Here is the entire `my_values.yaml` file:
    ```yaml
    image:
      repository: [yourname]supersetacr.azurecr.io/superset
    
    initImage:
      repository: [yourname]supersetacr.azurecr.io/busybox

    # Set type to 'LoadBalancer' so we can browse it
    service:
      type: LoadBalancer
      port: 80

    # Load Superset Examples
    init:
      loadExamples: true
    ```
    &nbsp;

## Deploy Superset to your AKS cluster
1. **Get the credentials to your AKS cluster so helm deploy to it**
Use the `az aks get-credentials` command to download credentials for your AKS cluster:
    ```powershell
    az aks get-credentials -n supersetaks -g supersetrg
    ```
    &nbsp;

1. **Deploy to AKS using helm**
First update your helm chart dependencies using the `helm dependency update` command:
    ```powershell
    helm dependency update helm/superset
    ```
    &nbsp;

    Then install your helm chart using the `helm install` command:
    ```powershell
    helm upgrade --install --values my_values.yaml superset helm/superset
    ```
    &nbsp;

## Check it out!
1. **View your AKS cluster in the Azure Portal and open Superset**
Some Azure services make it very easy for you to quickly jump from your CLI to view the resource in the Azure web portal; fortunately, AKS is one of those services.

    Open the portal to your AKS cluster with the `az aks browse -n supersetaks -g supersetrg` command.
    This will take you to the _Workloads_ page where you will see the status of your Kubernetes workloads. Give it five minutes or so for everything to get deployed and to turn green. 
    ![AKS Workloads page](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/10q0fy8unuckeikuf227.png)
    
    Then click on `Services and ingresses`.

    On the _Services and ingresses_ page, you will find the `External IP` for your Superset website:
    ![AKS Services and ingresses page](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hq7i3wojuf59tqnkx71n.png)
    
    When you click on the External IP **and add the port number :8088**, if everything worked perfectly, you will see the **Superset login page** below where you can use `admin` and `admin` to login.
    ![Superset login page](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dejv5fg7u16fc8ktsvfl.png)&nbsp;

## Conclusion
While I haven't tried out everything yet, I have connected to an external database and created charts and dashboards. I'm happy to say that **performance has been great** so far and I'm looking forward to seeing what I can create!
Here's one of the dashboards you can explore when you load the examples mentioned above:

![USA Births Names Dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/74mo0x9occ5ke4exbs03.png)
 
## Cleaning up the resources
Since AKS is not free, when you are done testing Superset you may want to **delete the resources**.

When you created your ACR, you created a resource group called `supersetrg`. And when you created your AKS cluster another resource group was also created with a name similar to this: `MC_supersetrg_supersetaks_westus2`. 

You will need to delete both of these resource groups to prevent incurring additional costs.   
```powershell
# View all of your resource groups
az group list -o table

# Delete the resource groups for your AKS cluster
az group delete -g supersetrg
az group delete -g MC_supersetrg_supersetaks_westus2
```
&nbsp;
 
## Next steps
- Consider [creating an Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/flexible-server/quickstart-create-server-cli) and [connecting it to Superset](https://superset.apache.org/docs/databases/postgres)
- Once you have a database connected, you can [upload some data to Superset from a CSV or Excel file](https://superset.apache.org/docs/creating-charts-dashboards/exploring-data/#loading-csv-data)
- Learn how to [Explore Data in Superset](https://superset.apache.org/docs/creating-charts-dashboards/exploring-data)
- Learn how to [Create your First Dashboard](https://superset.apache.org/docs/creating-charts-dashboards/creating-your-first-dashboard)