Running Elasticsearch on Kubernetes from Azure Kubernetes Service
Tags: elasticsearch, kubernetes, azure, observability
Published: 2020-10-22 11:39:57

It's safe to say that Kubernetes is the de facto standard for orchestrating containers and the applications running in them. As the standard, a variety of managed services and orchestration options are available to choose from. In this blog post, we're going to take a look at running the Elastic Stack on Azure Kubernetes Service (AKS) using Elastic Cloud on Kubernetes (ECK) as the operator.

[Elastic Cloud on Kubernetes](https://www.elastic.co/elastic-cloud-kubernetes), is the official operator for running the Elastic Stack on Kubernetes. ECK helps to manage, scale, upgrade, and deploy the Elastic Stack securely. In the steps below, we will deploy ECK on AKS and then use that deployment to collect logs, metrics, security events from a virtual machine on Azure.

Here's what we'll do:

1. Create an Azure Kubernetes Service cluster
2. Install Elastic Cloud on Kubernetes
3. Create an Elasticsearch cluster
4. Deploy Kibana
5. Create an Azure VM for us to monitor
6. Deploy Metric beat to collect VM metrics and events

## Deploying AKS, ECK, Elasticsearch, and Kibana

> Note: You need to have [Azure Account](https://azure.microsoft.com/en-us/free/?WT.mc_id=elastic-blog-arsaha) and the [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli) for Microsoft Azure installed to run some platform-specific commands. This helps you to create your cluster using this Azure CLI command.

### Step 1: Create an AKS cluster

```
az aks create --resource-group resourceGroupName --name clusterName --node-count 3 --generate-ssh-keys
```

### Step 2: Connect to the AKS cluster

```
az aks get-credentials --resource-group resourceGroupName --name clusterName
```

### Step 3: Install the ECK operator

```
kubectl apply -f https://download.elastic.co/downloads/eck/1.1.2/all-in-one.yamlkubectl -n elastic-system logs -f statefulset.apps/elastic-operator
```

### Step 4: Create an Elasticsearch cluster with an external IP
We're using the default load balancer that is available with Azure Kubernetes Service.

```
cat <<EOF | kubectl apply -f -apiVersion: elasticsearch.k8s.elastic.co/v1 
kind: Elasticsearch 
metadata: 
  name: quickstart 
spec: 
  version: 7.9.2 #Make sure you use the version of your choice 
  http: 
    service: 
      spec: 
        type: LoadBalancer #Adds a External IP 
  nodeSets: 
  - name: default 
    count: 1 
    config: 
      node.master: true 
      node.data: true 
      node.ingest: true 
      node.store.allow_mmap: false 
EOF
```

### Step 5: Monitor the cluster creation

```
kubectl get elasticsearch 
```

```
kubectl get pods -w
```

### Step 6: Check the logs of the pod created

```
kubectl logs -f quickstart-es-default-0 
```

```
kubectl get service quickstart-es-http
```

### Step 7: Retrieve the password of Elasticsearch cluster

```
PASSWORD=$(kubectl get secret quickstart-es-elastic-user -o=jsonpath='{.data.elastic}' | base64 --decode)
```

```
curl -u "elastic:$PASSWORD" -k "https://<IP_ADDRESS>:9200"
```

> Note: The public IP address of Elasticsearch can be picked by running: ```
kubectl get svc quickstart-es-http
```

### Step 8: Deploy Kibana

```
cat <<EOF | kubectl apply -f - 
apiVersion: kibana.k8s.elastic.co/v1 
kind: Kibana 
metadata: 
  name: quickstart 
spec: 
  version: 7.9.2 #Make sure Kibana and Elasticsearch are on the same version. 
  http: 
    service: 
      spec: 
        type: LoadBalancer #Adds a External IP 
  count: 1 
  elasticsearchRef: 
    name: quickstart 
EOF
```

### Step 9: Monitor the Kibana deployment

```
kubectl get kibana
```

## Ingesting and analyzing Azure metrics

Now that we have created an Elasticsearch cluster with Kibana in AKS, let's go ahead and ingest some observability data from Azure Cloud itself. Filebeat and Metricbeat make this easy by coming with out-of-the-box an Azure Module, helping to easily gather logs (activity, sign in, audit) and metrics (vm, container registry, billing) from Azure Cloud Platform.

In this tutorial, we will install Metricbeat on an Azure VM and enable the Azure cloud module. Before that, we also need to have credentials to authenticate with Azure Monitor REST API which uses Azure Resource Manager authentication model.

We need to have `client_id`, `client_secret`, `subscription_id`, `tenant_id` which can be obtained by creating an Azure Active Directory App. You can use this guide to Azure AD application and service principal that can access resources.

### Step 1: Create a Azure VM and SSH into the VM

```
az vm create \ 
  --resource-group myResourceGroup \ 
  --name myVM \ 
  --image UbuntuLTS \ 
  --admin-username azureuser \ 
  --generate-ssh-keys ssh azureuser@<IP_ADDRESS>
```

### Step 2: Install Metricbeat 

```
wget https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.9.2-amd64.deb
```

### Step 3: Configure Elasticsearch and Kibana credentials in Metricbeat

This helps us to ship the data to Elasticsearch Cluster created on AKS as well as load dashboards in Kibana.

```
vim /etc/metricbeat/metricbeat.yml
```

```
 setup.kibana: 
   host: "https://<public_ip_addr>:5601"
```

> Note: The public IP address of Kibana can be picked by running: ```
kubectl get "kubectl get svc quickstart-kb-http"
```

```
vim /etc/metricbeat/modules.d/azure.yml.disabled
```

Replace the client_id, client_secret, subscription_id, tenant_id for all the metricsets listed in the yml file. A sample might look like this.

```
- module: azure 
  metricsets: 
  - monitor 
  enabled: true 
  period: 300s 
  client_id: '8dec1ab1-1691-48a6-af43-f87de68e971b' 
  client_secret: '~fwL-MhOcguaD2yK1e_.OWHhhqwdp-p974' 
  tenant_id: 'aa40685b-417d-4664-b4ec-8f7640719adb' 
  subscription_id: '70bd6e77-4b1e-4835-8896-db77b8eef364' 
  refresh_list_interval: 600s 
  resources: 
  - resource_query: "resourceType eq 'Microsoft.DocumentDb/databaseAccounts'" 
    metrics: 
    - name: ["DataUsage", "DocumentCount", "DocumentQuota"] 
      namespace: "Microsoft.DocumentDb/databaseAccounts"
```

### Step 4: Enable Azure Module and start Metricbeat

```
cd /usr/bin/
```

```
./metricbeat modules enable azure
./metricbeat setup --dashboards
./metricbeat -e
```

### Step 5: Monitor Metrics of Azure in Kibana

Log into Kibana and head over to Dashboards. Search for "Azure" to look at several preconfigured dashboards regarding storage, database, billing. Here is what a sample monitoring dashboard would look like:


![azure-metrics](https://dev-to-uploads.s3.amazonaws.com/i/vujdb1u6f7xlw75h8ebe.jpg)


## Wrapping up

And that's that! You have successfully built a secure Elastic Stack deployment on a managed Kubernetes service. You can also deploy other applications like [Elastic APM](https://www.elastic.co/apm) or [Elastic Workplace Search](https://www.elastic.co/workplace-search). In addition to that, you can also enable [cross-cluster search](https://www.elastic.co/guide/en/elasticsearch/reference/7.9/modules-cross-cluster-search.html) and [replication](https://www.elastic.co/guide/en/elasticsearch/reference/7.9/xpack-ccr.html), which enables you to deploy an Elastic Stack on Kubernetes cluster across regions to serve users.

We encourage you to [try ECK for yourself] (https://www.elastic.co/downloads/elastic-cloud-kubernetes) (on any Kubernetes service), and if you have further questions related to this blog post, you can always reach us out on our [community discussion forums](http://discuss.elastic.co/) or [Slack workspace] (https://ela.st/slack).

If you'd like to learn more about how Elastic can help with Kubernetes observability, check out our Best of Elastic [Observability webinars series](https://www.elastic.co/webinars/best-of-elastic-observability)

You can follow me at [Aravind Putrevu](https://twitter.com/intent/follow?ref_src=twsrc%5Etfw&screen_name=aravindputrevu&tw_p=followbutton)