Run Apache APISIX on Microsoft Azure Container Instance
Tags: azure, apisix, opensource, devops
Published: 2022-06-14 14:06:27

## Introduction

[Apache APISIX](https://apisix.apache.org/) is an **open-source Microservice API gateway** and platform designed for managing microservices requests of high availability, fault tolerance, and distributed system. You can install **Apache APISIX** by the [different methods](https://apisix.apache.org/docs/apisix/how-to-build) ([Docker](https://www.docker.com/), [Helm](https://helm.sh/), or [RPM](https://rpm.org/)) and run it in the various public cloud providers because of its cloud-native behavior. In this post, you will learn how easily run Apache APISIX API Gateway in [Azure Container Instances](https://azure.microsoft.com/en-us/services/container-instances/) with multiple containers (Apisix and etcd) straight from Docker CLI. 
 
☝️Alternatively, there are a bunch of options you can also deploy APISIX in Azure:

[Azure Kubernetes Service](https://azure.microsoft.com/en-us/services/kubernetes-service/), if you want a complete orchestration solution and if you need to manage your deployment and infrastructure.
[Azure Service Fabric](https://azure.microsoft.com/en-us/services/service-fabric/) for containers and services orchestration.
[Azure Compute](https://azure.microsoft.com/en-us/product-categories/compute/), to build your own solution.

## In this walkthrough, you will

✔️ Create an Azure resource group.
✔️ Configure Azure Container Instances.
✔️ Create an Azure context for Docker to offload Apisix and etcd containers execution to ACI.
✔️ Get [Apache APISIX example source code](https://github.com/apache/apisix-docker) for Docker from GitHub.
✔️ Modify Docker compose file there.
✔️ Setup volumes in [Azure Storage Account](https://docs.microsoft.com/en-us/azure/storage/common/storage-account-create?tabs=azure-portal) with [Azure File Share] (https://docs.microsoft.com/en-us/azure/storage/files/storage-files-introduction).
✔️ Add APISIX config files to Azure File Share.
✔️ Bring up APISIX in Azure Container Instances.
✔️ Verify APISIX running.
✔️ Cleanup after you finish.

## Prerequisites

➡️ **Azure subscription** - create a [free account](https://azure.microsoft.com/en-gb/free/) before you begin.
➡️ **Azure CLI** - you must have Azure CLI installed on your local computer hence we are going to use Azure CLI to interact with all Azure resources. See [Install the Azure CLI] (https://docs.microsoft.com/en-us/cli/azure/install-azure-cli) how to set up.
➡️ **Docker Desktop** - you need also [Docker desktop](https://www.docker.com/products/docker-desktop/) installed locally to complete this tutorial. It is available for [Windows](https://desktop.docker.com/win/edge/Docker%20Desktop%20Installer.exe) or [macOS](https://desktop.docker.com/mac/edge/Docker.dmg). Or install the [Docker ACI Integration CLI for Linux](https://docs.docker.com/engine/context/aci-integration/#install-the-docker-aci-integration-cli-on-linux). 

We use [Docker Compose](https://docs.docker.com/compose/) to define and deploy two containers for **Apisix** and [**etcd**](https://github.com/etcd-io/etcd) as a [container group](https://docs.microsoft.com/en-us/azure/container-instances/container-instances-container-groups) in Azure Container Instances.

> APISIX uses etcd to save and synchronize configuration. Before installing APISIX, you need to install etcd on your machine. It would be installed automatically if you choose the Docker compose, so it is a perfect fit for this example.

💁Run containers in Azure Container Instances on-demand when you develop cloud-native apps like APISIX with Docker and you want to switch seamlessly from local development to cloud deployment. This excellent capability is enabled by [integration between Docker and Azure](https://docs.docker.com/cloud/aci-integration/)

## Create a resource group

Before you create and manage your APISIX container instance, you need a resource group to deploy it to. A resource group is a logical collection into which all Azure resources are deployed and managed.

Create a resource group with the `az group create` command. In the following example, a resource group named _apisix_ is created in the _centralus_ region:

``` bash
az group create --name apisix --location centralus
```

## Configure Azure Container Instances

You can now run the Azure CLI with the `az` command from any command-line interface.

> Note that I used Windows OS with Docker desktop installed on my machine for this demo and I run `az` commands from either Windows Command Prompt or PowerShell.

We use Docker commands to run containers in Azure Container Instances, so the first thing we need to log into Azure by running the following command:

``` bash
docker login azure
```
You can also log in using a Service Principal (SP). Provide the id and password of the SP using --client-id and --client-secret arguments when calling `docker login azure`.

## Create an Azure context

Once logged in, you will create an ACI context by running `docker context create aci`. The context is responsible for associating requests issued by Docker CLI to nodes or clusters in ACI. For example, to create a context called _apisixacicontext_ you can use:

``` bash
docker context create aci apisixacicontext
```

`docker context create` is an interactive command. It guides you through the process of configuring a new Docker context for our existing Azure resource group.

Run `docker context ls` to confirm that you added the ACI context to your Docker contexts:

``` bash
docker context ls
```
Output:

![docker context ls output](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/e6o9h1dp8i7kh4aubhmg.png)

Next, change to the ACI context. Subsequent Docker commands run in this context.

``` bash
docker context use apisixacicontext
```

## Get Apache APISIX from GitHub

In this demo, we are using [Apache APISIX Docker repo](https://github.com/apache/apisix-docker) and it contains an example `docker-compose.yaml` file and other config files that show how to start APISIX using docker compose. We try out this example:

Use [git](https://git-scm.com/downloads) to clone the repository and cd into the _example_ folder.

``` bash
git clone 'https://github.com/apache/apisix-docker'
cd apisix-docker/example
```

## Modify Docker compose file

Next, open docker-compose.yaml in a text editor. The [example docker compose file](https://github.com/apache/apisix-docker/blob/master/example/docker-compose.yml) defines several services: `apisix-dashboard, apisix, etcd, web1, web2, prometheus, and grafana`:

- `apisix-dashboard, apisix, etcd` are the essential services required for starting apisix-dashboard, apisix, and etcd.
- `web1, web2` are sample backend services used for testing purposes. They use nginx-alpine image.
- `prometheus, grafana` are services used for exposing metrics of the running services.

For the sake of simplicity, we are going to use and run only APISIX and etcd services in this demo. We can simply do the following changes by removing other services and defining volumes like etcd-data and apisix-data. In the next step, we use Azure file share as volumes. 

``` yaml
version: "3"

services:
  apisix:
    image: apache/apisix:2.13.1-alpine
    restart: always
    volumes:
      - apisix-data:/apisix/conf/
    depends_on:
      - etcd
    ports:
      - "9080:9080/tcp"
      - "9091:9091/tcp"
      - "9443:9443/tcp"
      - "9092:9092/tcp"
    networks:
      apisix:

  etcd:
    image: bitnami/etcd:3.4.15
    restart: always
    volumes:
      - etcd-data:/bitnami/etcd
    environment:
      ETCD_ENABLE_V2: "true"
      ALLOW_NONE_AUTHENTICATION: "yes"
      ETCD_ADVERTISE_CLIENT_URLS: "http://0.0.0.0:2379"
      ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
    ports:
      - "2379:2379/tcp"
    networks:
      apisix:

networks:
  apisix:
    driver: bridge

volumes:
  etcd-data:
    driver: azure_file
    driver_opts:
      share_name: etcdshare
      storage_account_name: apisixstorage
  apisix-data:
    driver: azure_file
    driver_opts:
      share_name: apisixshare
      storage_account_name: apisixstorage
```

## Setup volumes using Azure file share

Apache APISIX has to persist etcd state and mount external configuration files like _/apisix_conf/conf.yaml_ (defines the configs for apisix) in the repo folder onto the containers. You can store persistent data outside of the container-filesystem in [ACI using Azure file share](https://docs.docker.com/cloud/aci-integration/#using-azure-file-share-as-volumes-in-aci-containers:~:text=must%20be%20available.-,Using%20Azure%20file%20share%20as%20volumes%20in%20ACI%20containers,-%F0%9F%94%97) since Azure Container Instances are stateless. If the container is restarted, crashes, or stops, all of its state is lost.

:information_source: More about [how to mount an Azure file share in Azure Container Instances](https://docs.microsoft.com/en-us/azure/container-instances/container-instances-volume-azure-files)

As you may notice, we declared two volumes in `docker-compose.yaml` file and set the driver to _azure_file_. Before using an Azure file share with Azure Container Instances, you must create a new Azure Storage account to host the file share and add a file share to it. To create Azure storage with the name, for example, _apisixstorage_ :

``` bash
az storage account create --resource-group apisix --name apisixstorage --location centralus --sku Standard_LRS
```

Run the following two commands to create two Azure file shares combined with the required Azure Storage Account respectively using `docker volume create`.

For the first volume _apisixshare_:

``` bash
docker volume create apisixshare --storage-account apisixstorage
```

For the second volume _etcdshare_:

``` bash
docker volume create etcdshare --storage-account apisixstorage
```

You can see created the storage with two files shares in the Azure portal too.

![Storage with 2 file shares](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/73myc0270dp27kjus2t0.png)
 
Having the volume in place, we can run stateful APISIX in ACI as shown in the next section.

## Add APISIX config files to Azure File Share

Now we need to upload manually Apache APISIX config files to Azure File Share. You can simply use the Azure portal to do so. Or you can always use [az storage file upload](https://docs.microsoft.com/en-us/azure/storage/files/storage-how-to-use-files-portal?tabs=azure-cli) Azure CLI command.

1. Find the **File shares** from the navigation bar of **apisixstorage** storage we created in the previous step and select **apisixshare** file-share to open. The fileshare panel opens.

![apisixshare](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5k41r5yufntaxoswe0us.png)
 
2. In the menu at the top, select **Upload**. The **Upload files** panel opens. Download and add all files including directories from [Apache APISIX conf folder](https://github.com/apache/apisix/tree/master/conf). The final list of config files in **apisixshare** file-share should match Apache APISIX conf folder. Similar to the output below:

 ![apisixshare with all config files](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/q8dgnm8ai4z6mnz1166d.png)

> Here one more thing notice whatever the files you will add/update in file share it will add/update in container as well after you mount the file share as volume to the container. However, there are some limitations to this like _you can only mount the whole share and not the subfolders or single file within it_ that's why we define the mount path to be `/apisix/conf` in the volume property of `docker-compose.yaml` file. Read more about [other limitations](https://docs.microsoft.com/en-us/azure/container-instances/container-instances-volume-azure-files#limitations). 

``` yaml
...
volumes:
 - apisix-data:/apisix/conf/
...
```
 
Behind the scene, ACI copies all the above config files from `apisixshare` file-share to `/apisix/conf/` folder in Linux APISIX container.

![Apisix container bash](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xf49ti2ee7x2h87swcrc.png)
 

## Bring up APISIX in ACI

Finally, now we can deploy APISIX with etcd to Azure Container Instances. Execute `docker compose up` to create the container group in Azure Container Instances.

``` bash
docker compose up
```

Wait until the container group is deployed. Then, you can also verify container instances are created in the Azure portal. 

![Apisix container instances](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cl9888g6est6m256jrat.png)

> You can also assign the value _apisixaci_ to the _domainname_ property under `apisix` service in the `docker-compose.yaml` file which results a custom DNS name in ACI. In other words, you instruct Azure to link a subdomain of `azurecontainer.io` to the public IP address of the exposed container. The FQDN (the fully qualified domain name) follows the schema and will expose APISIX instance on `apisixaci.centralus.azurecontainer.io` serving the proxy on port `9080`. However, with the current configuration, we can access our APISIX with its the public IP address.
 
Next, you run `docker ps` to see the running containers and the IP address assigned to the container group.
 
![Apisix is running in ACI](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t1c1gtqttr2pbx4nvevt.png)
 
 
To see the logs of the APISIX, run the `docker logs` command. For example:

``` bash
docker logs example_apisix
```

Sample output:

``` bash
...
2022/06/11 19:02:45 [warn] 211#211: *4 [lua] plugin.lua:223: load_stream(): new plugins: {"limit-conn":true,"ip-restriction":true,"mqtt-proxy":true}, context: 
...
```

## Verify Apache APISIX running

To verify if Apache APISIX is running in the cloud, we run the below `curl` command and check the response from APISIX's [REST Admin API](https://apisix.apache.org/docs/apisix/admin-api). You need to replace `ACI_PUBLIC_IP_ADDRESS` to your container instance group's the public IP address or FQDN. 

``` bash
curl "http://{ACI_PUBLIC_IP_ADDRESS}:9080/apisix/admin/services/" -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1'
```

The response indicates that APISIX is running successfully:

``` bash
{
  "count":0,
  "action":"get",
  "node":{
    "key":"/apisix/services",
    "nodes":[],
    "dir":true
  }
}
```

Here we go, Apache APISIX is up and running in Azure Container Instance group and responding to your requests 👏💪. 

### Troubleshoot😕

When you are requesting Apache APISIX Admin API, you may get `403 Forbidden` HTTP status error. The reason is that your client IP address might be not whitelisted in the [APISIX config file](https://github.com/apache/apisix/blob/master/conf/config.yaml). The REST Admin API to control Apache APISIX, which only allows `127.0.0.1` access by default, you can modify the `allow_admin` field in `conf/config.yaml` to specify a list of IPs that are allowed to call the Admin API. Also, note that the Admin API uses key auth to verify the identity of the caller. The `admin_key` field in `conf/config.yaml` needs to be modified before deployment to ensure security. 

``` yaml
  allow_admin:                  # http://nginx.org/en/docs/http/ngx_http_access_module.html#allow
    - 0.0.0.0/0  # We need to restrict ip access rules for security. 0.0.0.0/0 is for test.
    - YOUR_IP_ADDRESS

  admin_key:
    - name: "admin"
      key: YOUR_ADMIN_API_KEY
      role: admin                 # admin: manage all configuration data
```

## Cleanup after you finish

When you finish trying the application, stop the application and containers with `docker compose down` inside the `apisix-docker/compose` folder.

``` bash
docker compose down
```
This command deletes all containers _apisix_ and _etcd_ in Azure Container Instances.

You can also remove the Docker context created during this demo, use the `docker context rm apisixacicontext` command after switching back to the default context:

``` bash
docker context use default
```

Then, you run

``` bash
docker context rm apisixacicontext
```

## Conclusion

Up to now, we learn how to deploy Apache APISIX to Azure Cloud with Docker Compose to switch from running a multi-container APISIX locally to running in Azure Container Instances. From this stage, you can create a [route](https://apisix.apache.org/docs/apisix/getting-started#:~:text=Copy-,Create%20a%20Route,-%23), [upstream](https://apisix.apache.org/docs/apisix/getting-started#:~:text=10%20by%20APISIX.-,Abstracting%20to%20Upstream,-%23) and manage the traffic to your backend services with the available [built-in plugins](https://apisix.apache.org/plugins) if you want to take advantage of more APISIX's features. You can provision other services with APISIX Docker compose like prometheus, grafana as well. 

## Recommended content

➔ Watch Video Tutorial [Getting Started with Apache APISIX](https://youtu.be/dUOjJkb61so)  

➔ Read the blog post [Overview of Apache APISIX API Gateway Plugins](https://dev.to/apisix/overview-of-apache-apisix-api-gateway-plugins-2m8o)

➔ Read the blog post [Centralized Authentication with Apache APISIX Plugins](https://dev.to/apisix/centralized-authentication-with-apache-apisix-plugins-30fo)
 
➔ Read the blog post [API Security with OIDC by using Apache APISIX and Microsoft Azure AD](https://dev.to/apisix/api-security-with-oidc-by-using-apache-apisix-and-microsoft-azure-ad-50h3) 

Community⤵️

🙋 [Join the Apache APISIX Community](https://apisix.apache.org/docs/general/community) 
🐦 [Follow us on Twitter](https://twitter.com/ApacheAPISIX)
📝 [Find us on Slack](https://join.slack.com/t/the-asf/shared_invite/zt-vlfbf7ch-HkbNHiU_uDlcH_RvaHv9gQ)
📧 [Mail to us](dev@apisix.apache.org) with your questions