How to use Azure Go SDK to manage Azure Data Explorer clusters
Tags: azure, go, bigdata, tutorial
Published: 2020-07-30 12:52:14

[Getting started with Azure Data Explorer using the Go SDK](https://dev.to/azure/getting-started-with-azure-data-explorer-using-the-go-sdk-30jm) covered how to use the [Azure Data Explorer Go SDK](https://github.com/Azure/azure-kusto-go) to ingest and query data from azure data explorer to ingest and query data. In this blog you will the [Azure Go SDK](https://docs.microsoft.com/azure/developer/go/?WT.mc_id=devto-blog-abhishgu) to manage Azure Data Explorer clusters and databases.

[Azure Data Explorer](https://docs.microsoft.com/azure/data-explorer/?WT.mc_id=devto-blog-abhishgu) (also known as **Kusto**) is a fast and scalable data exploration service for analyzing large volumes of diverse data from any data source, such as websites, applications, IoT devices, and more. This data can then be used for diagnostics, monitoring, reporting, machine learning, and additional analytics capabilities.

In case you're wondering, we are talking about [two different SDKs here](https://docs.microsoft.com/azure/data-explorer/kusto/api/golang/kusto-golang-client-library?WT.mc_id=devto-blog-abhishgu). The one covered in this blog is for resource administration (also known as the control plane SDK) and the the one I used in the [other post](https://dev.to/azure/getting-started-with-azure-data-explorer-using-the-go-sdk-30jm) is data plane SDK for interacting with the Azure Data Explorer service itself (ingestion, query etc.)

**What's covered ?**

A simple CLI application is used as an example to demonstrate how to use the Go SDK. We'll try out the application first and go through how to:

- Create and list Azure Data Explorer clusters
- Create and list databases in that cluster
- Delete the database and cluster

```
$ ./goadx --help
CLI to test sample program for Azure Data Explorer

Usage:
  goadx [command]

Available Commands:
  cluster     create, list and delete Azure Data Explorer clusters
  db          create, list and delete databases in an Azure Data Explorer cluster
  help        Help about any command

Flags:
  -h, --help   help for goadx

Use "goadx [command] --help" for more information about a command.
```

Once that's done, we'll walk through the sample code to understand what's going on

> The code is available on GitHub https://github.com/abhirockzz/azure-go-sdk-for-dataexplorer

Please note that this CLI based example is just meant to showcase how to use the Azure Go SDK (in the context of Azure Data Explorer) as a part of a larger application. It is not supposed to replace/substitute the [Azure CLI which can be used to manage Azure Data Explorer resources](https://docs.microsoft.com/azure/data-explorer/create-cluster-database-cli?WT.mc_id=devto-blog-abhishgu)

## Pre-requisites

[Install Go 1.13 or above](https://golang.org/dl/)

You will need a [Microsoft Azure account](https://docs.microsoft.com/azure/?WT.mc_id=devto-blog-abhishgu). Go ahead and [sign up for a free one!](https://azure.microsoft.com/free/?WT.mc_id=devto-blog-abhishgu)

Install the [Azure CLI](https://docs.microsoft.com/cli/azure/install-azure-cli?view=azure-cli-latest&WT.mc_id=devto-blog-abhishgu) if you don't have it already (should be quick!)

## Run the CLI application

Get the code and build it:

```shell
git clone https://github.com/abhirockzz/azure-go-sdk-for-dataexplorer
cd azure-go-sdk-for-dataexplorer
go build -o goadx

//to confirm
chmod a+x goadx && ./goadx
```

To see the details for individual commands, e.g. for cluster creation:

```shell
./goadx cluster create --help

//output

Creates 1 instance of compute type DevNoSLAStandardD11V2 in Basic tier

Usage:
  goadx cluster create [flags]

Flags:
  -h, --help          help for create
      --loc string    ADX cluster location
      --name string   ADX cluster name

Global Flags:
      --rg string    Azure resource group
      --sub string   Azure subscription
```

### Few more steps before creating the cluster...

**Create a resource group ....**

... using [az group create](https://docs.microsoft.com/cli/azure/group?view=azure-cli-latest&WT.mc_id=devto-blog-abhishgu#az-group-create) CLI command

```
az group create -l <region> -n <name>

e.g. az group create -l southeastasia -n my-adx-rg
```

We need a [Service Principal](https://docs.microsoft.com/azure/active-directory/develop/app-objects-and-service-principals?WT.mc_id=devto-blog-abhishgu#service-principal-object) for the Go SDK to authenticate with Azure Data Explorer service to execute cluster and database operations.

**Create a Service Principal...**

...using [az ad sp create-for-rbac](https://docs.microsoft.com/cli/azure/ad/sp?view=azure-cli-latest&WT.mc_id=devto-blog-abhishgu#az-ad-sp-create-for-rbac)

```
az ad sp create-for-rbac -n "test-datax-sp"
```

You will get a JSON response as such

```json
{
  "appId": "fe7280c7-5705-4789-b17f-71a472340429",
  "displayName": "test-datax-sp",
  "name": "http://test-datax-sp",
  "password": "29c719dd-f2b3-46de-b71c-4004fb6116ee",
  "tenant": "42f988bf-86f1-42af-91ab-2d7cd011db42"
}
```

Set the service principal details as environment variables. During the code walk-through, you will see why these specific variables are required

```shell
export AZURE_CLIENT_ID="<appId>"
export AZURE_CLIENT_SECRET="<password>"
export AZURE_TENANT_ID="<tenant>"
```

You are all set!

### Create an Azure Data Explorer cluster

```shell
./goadx cluster create --name <name of the cluster you want to create> --loc <azure region for the cluster> --rg <name of the resource group> --sub <azure subscription id>
```

For example:

```shell
./goadx cluster create --name MyAdxCluster --loc "Southeast Asia" --rg my-adx-rg --sub 9a42a42f-ae42-4242-b6a7-eea0ea42d342
```

This will create a single instance cluster with the `DevNoSLAStandardD11V2` VM 

> this is hard coded for sake of simplicity/ease-of-use

The cluster creation will take some time, and the code blocks for that time period. Please be patient ;) Grab some coffee, check Twitter or do whatever keeps you busy for about ~10-15 mins. Once the cluster is created, you will see this message:

```
created cluster MyAdxCluster with ID /subscriptions/9a42a42f-ae42-4242-b6a7-eea0ea42d342/resourceGroups/my-adx-rg/providers/Microsoft.Kusto/Clusters/MyAdxCluster and type Microsoft.Kusto/Clusters
```

### List down all the clusters

You just created a cluster, let's make sure you can get its info (including any other cluster you might already have in your resource group)

```shell
./goadx cluster list --rg <name of the resource group> --sub <azure subscription id>
```

for example

```shell
./goadx cluster list --rg my-adx-rg --sub 9a42a42f-ae42-4242-b6a7-eea0ea42d342
```

You will get a tabular output as such:

```shell
+---------------+---------+----------------+-----------+-------------------------------------------------------+
|     NAME      |  STATE  |    LOCATION    | INSTANCES |                          URI                          |
+---------------+---------+----------------+-----------+-------------------------------------------------------+
| MyAdxCluster | Running | Southeast Asia |         1 | https://MyAdxCluster.southeastasia.kusto.windows.net |
+---------------+---------+----------------+-----------+-------------------------------------------------------+
```

### You have the cluster, its time to create a database

To create a database in the cluster

```shell
./goadx db create --name <name of the database you want to create> --cluster <nname of the adx cluster> --loc <azure region> --rg <resource group name> --sub <azure subcription id>
```

for example

```shell
./goadx db create --name testadxdb --cluster MyAdxCluster --loc "Southeast Asia" --rg my-adx-rg --sub 9a42a42f-ae42-4242-b6a7-eea0ea42d342
```

It shouldnt take a too long. Once the DB is created, you will see this message:

```
created DB MyAdxCluster/testadxdb with ID /subscriptions/9a42a42f-ae42-4242-b6a7-eea0ea42d342/resourceGroups/my-adx-rg/providers/Microsoft.Kusto/Clusters/MyAdxCluster/Databases/testadxdb and type Microsoft.Kusto/Clusters/Databases
```

### To check the DB you just created ...

```shell
./goadx db list --cluster <name of the adx cluster> --rg <resource group name> --sub <azure subscription id>
```

for example

```shell
./goadx db list --cluster MyAdxCluster --rg my-adx-rg --sub 9a42a42f-ae42-4242-b6a7-eea0ea42d342
```

You will get a tabular output as such:

```shell
+---------------------------+-----------+----------------+------------------------------------+
|           NAME            |   STATE   |    LOCATION    |                TYPE                |
+---------------------------+-----------+----------------+------------------------------------+
| MyAdxCluster/testadxdb     | Succeeded | Southeast Asia | Microsoft.Kusto/Clusters/Databases |
+---------------------------+-----------+----------------+------------------------------------+
```

You can continue experimenting further and create more clusters and/or databases

### Time to clean up ...

You can delete the cluster directly as such:

```shell
./goadx cluster delete --name <name of adx cluster> --rg <resource group name> --sub <azure subcription id>
```

example:

```shell
./goadx cluster delete --name MyAdxCluster --rg my-adx-rg --sub 9a42a42f-ae42-4242-b6a7-eea0ea42d342
```

You should see a confirmation message indicating that the cluster has been deleted:

```
deleted ADX cluster MyAdxCluster from resource group my-adx-rg
```

Or you could just delete the database itself

```shell
./goadx db delete --name <database name> --cluster <adx cluster name> --rg <resource group name> --sub <azure subscription id>
```

For example

```shell
./goadx db delete --name testadxdb --cluster MyAdxCluster --rg my-adx-rg --sub 9a42a42f-ae42-4242-b6a7-eea0ea42d342
```

You should see a confirmation message indicating that the database has been deleted:

```
deleted DB testadxdb from cluster MyAdxCluster
```

Alright, now that you what can be done, it's time to see how its done!

## Code walk through

At a high level, the application consists of two parts:

- Azure Data Explorer operations (create, read, delete) - part of the [`ops` package](https://github.com/abhirockzz/azure-go-sdk-for-dataexplorer/tree/master/ops)
- CLI commands - part of the [`cli` package](https://github.com/abhirockzz/azure-go-sdk-for-dataexplorer/tree/master/cli)

To be honest, the CLI bit is not important (at-least in the context of the blog). It uses the (ubiquitous) [Cobra](https://github.com/spf13/cobra) package to implement the cluster and cluster (top-level) commands and the sub-commands - `create`, `list` and `delete`. If you're interested, take a look at the following:

- [root.go](https://github.com/abhirockzz/azure-go-sdk-for-dataexplorer/blob/master/cli/root.go) - defines the root command for CLI entry point
- [cluster.go](https://github.com/abhirockzz/azure-go-sdk-for-dataexplorer/blob/master/cli/cluster.go) - defines and implements sub-commands for cluster operations
- [db.go](https://github.com/abhirockzz/azure-go-sdk-for-dataexplorer/blob/master/cli/db.go) - defines and implements sub-commands for database operations

The CLI part simply invokes CRUD oeprations on Azure Data Explorer resources which is the important part. So, lets go through them

### Authentication


This is handled in [ops/client.go](https://github.com/abhirockzz/azure-go-sdk-for-dataexplorer/blob/master/ops/client.go). To get a handle for the client components to execute the required operations, we need to authenticate. For e.g. to get the [`kusto.ClustersClient`](https://godoc.org/github.com/Azure/azure-sdk-for-go/services/kusto/mgmt/2020-02-15/kusto#ClustersClient):

```go
func getClustersClient(subscription string) kusto.ClustersClient {
    client := kusto.NewClustersClient(subscription)
    authR, err := auth.NewAuthorizerFromEnvironment()
    if err != nil {
        log.Fatal(err)
    }
    client.Authorizer = authR

    return client
}
```

The part to note is the [auth.NewAuthorizerFromEnvironment()](https://godoc.org/github.com/Azure/go-autorest/autorest/azure/auth#NewAuthorizerFromEnvironment) which looks for a set of pre-defined environment variables - this is [User environment-based authentication](https://docs.microsoft.com/en-us/azure/developer/go/azure-sdk-authorization#use-environment-based-authentication)

### Cluster operations

[ops/cluster.go](https://github.com/abhirockzz/azure-go-sdk-for-dataexplorer/blob/master/ops/cluster.go) is where the cluster create, list and delete is handled. Here is the snippet for create:

```go
func CreateCluster(sub, clusterName, location, rgName string) {
    ...
    result, err := client.CreateOrUpdate(ctx, rgName, clusterName, kusto.Cluster{Location: &location, Sku: &kusto.AzureSku{Name: kusto.DevNoSLAStandardD11V2, Capacity: &numInstances, Tier: kusto.Basic}})
    err = result.WaitForCompletionRef(context.Background(), client.Client)
    r, err := result.Result(client)
    ...
}
```

[CreateOrUpdate](https://godoc.org/github.com/Azure/azure-sdk-for-go/services/kusto/mgmt/2020-02-15/kusto#ClustersClient.CreateOrUpdate) is used to initiate the cluster creation. Since this is a long running op, it returns a [`ClustersCreateOrUpdateFuture`](https://godoc.org/github.com/Azure/azure-sdk-for-go/services/kusto/mgmt/2020-02-15/kusto#ClustersCreateOrUpdateFuture) which we them wait on (using [`WaitForCompletionRef`](https://godoc.org/github.com/Azure/go-autorest/autorest/azure#Future.WaitForCompletionRef))

> Note that I have used `context.Background()`, you can use a different context e,g one with timeout or one which can be cancelled. 

Once that's one, we check the future for the [`Result`](https://godoc.org/github.com/Azure/azure-sdk-for-go/services/kusto/mgmt/2020-02-15/kusto#ClustersCreateOrUpdateFuture.Result)

Listing all the clusters in a resource group is quite simple and can be done using [`ListByResourceGroup`](https://godoc.org/github.com/Azure/azure-sdk-for-go/services/kusto/mgmt/2020-02-15/kusto#ClustersClient.ListByResourceGroup)

```go
func ListClusters(sub, rgName string) kusto.ClusterListResult {
    ctx := context.Background()

    result, err := getClustersClient(sub).ListByResourceGroup(ctx, rgName)
    if err != nil {
        log.Fatal(err.Error())
    }
    return result
}
```

`Delete` is similar to `Create` in the sense that it is a long running operation as well. Its the same drill - initiate the deletion ([using Delete](https://godoc.org/github.com/Azure/azure-sdk-for-go/services/kusto/mgmt/2020-02-15/kusto#ClustersClient.Delete)), wait for it to finish and then confirm the result

```go
func DeleteCluster(sub, clusterName, rgName string) {
    ...
    result, err := client.Delete(ctx, rgName, clusterName)
    err = result.WaitForCompletionRef(context.Background(), client.Client)
    r, err := result.Result(client)
    ...
}
```

### Database operations

[ops/db.go](https://github.com/abhirockzz/azure-go-sdk-for-dataexplorer/blob/master/ops/db.go) is where the database create, list and delete is handled. Here is the snippet for create:

```go
func CreateDatabase(sub, rgName, clusterName, location, dbName string) {
    ...
    future, err := client.CreateOrUpdate(ctx, rgName, clusterName, dbName, kusto.ReadWriteDatabase{Kind: kusto.KindReadWrite, Location: &location})
    err = future.WaitForCompletionRef(context.Background(), client.Client)

    r, err := future.Result(client)
    kdb, _ := r.Value.AsReadWriteDatabase()
    ...
}
```

Listing is straightforward and taken care of by [`ListByCluster`](https://godoc.org/github.com/Azure/azure-sdk-for-go/services/kusto/mgmt/2020-02-15/kusto#DatabasesClient.ListByCluster)

```go
func ListDBs(sub, rgName, clusterName string) kusto.DatabaseListResult {
    ctx := context.Background()
    result, err := getDBClient(sub).ListByCluster(ctx, rgName, clusterName)
    if err != nil {
        log.Fatal("failed to get databases in cluster", err)
    }
    return result
}
```

And finally, the deletion which is handled using [`Delete`](https://godoc.org/github.com/Azure/azure-sdk-for-go/services/kusto/mgmt/2020-02-15/kusto#DatabasesClient.Delete)

```go
func DeleteDB(sub, rgName, clusterName, dbName string) {
    ...
    future, err := getDBClient(sub).Delete(ctx, rgName, clusterName, dbName)
    err = future.WaitForCompletionRef(context.Background(), client.Client)
    r, err := future.Result(client)
    ...
}
```

That's it for the code walkthrough!

## Conclusion

Hopefully this was helpful in demonstrating how to use the Azure Go SDK to work with Azure Data Explorer resources. You can check the documentation for [Python](https://docs.microsoft.com/azure/data-explorer/create-cluster-database-python?WT.mc_id=devto-blog-abhishgu), [.NET](https://docs.microsoft.com/azure/data-explorer/create-cluster-database-csharp?WT.mc_id=devto-blog-abhishgu) and other examples. Happy exploring!