Accessing Grafana via SSH Tunneling
Tags: prometheus, grafana, kubernetes, remote
Published: 2020-05-09 13:31:17


Here is a quick guide "how-to" for configuring Prometheus and Grafana on Kubernetes Cluster that is behind firewalls and accessing the Grafana UI from a remote machine (your laptop at home) using SSH tunneling. 

## Setup Helm 3

You can use Helm 2 as well or skip this step if you already have helm installed. 

#### Step 1: Install Helm 3
```
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh
```

#### Step 2: Add Stable Charts
```
helm repo add stable https://kubernetes-charts.storage.googleapis.com/
```

## Setup Prometheus and Grafana using Prometheus Operator

This is the easiest way to setup Prometheus and Grafana, and have the Grafana configured to use Prometheus as a data source. 

#### Step 3: Install Prometheus Operator

In my case, the Kubernetes cluster is behind a firewall. I am configuring the Granfana to be accessible via NodePort, as I need to access the Grafana UI using ssh tunnel. 

```
kubectl create namespace prometheus-operator
helm install prometheus-operator stable/prometheus-operator -n prometheus-operator --set prometheusOperator.createCustomResource=false,grafana.service.type=NodePort
```


#### Step 4: Verify

```
kubectl get pods -n prometheus-operator
```

The above commands should show that all promtheus operator, prometheus, node exporter and grafana pods are running. 

```
NAME                                                     READY   STATUS    RESTARTS   AGE
alertmanager-prometheus-operator-alertmanager-0          2/2     Running   0          30m
prometheus-operator-grafana-cf6954699-5rcgl              2/2     Running   0          30m
prometheus-operator-kube-state-metrics-5fdcd78bc-sckjv   1/1     Running   0          30m
prometheus-operator-operator-5dd8f8f568-52qk8            2/2     Running   0          30m
prometheus-operator-prometheus-node-exporter-p8pm8       1/1     Running   0          30m
prometheus-operator-prometheus-node-exporter-trlhp       1/1     Running   0          30m
prometheus-operator-prometheus-node-exporter-wsm4n       1/1     Running   0          30m
prometheus-prometheus-operator-prometheus-0              3/3     Running   1          30m
```

```
kubectl get svc -n prometheus-operator
```

Note that Grafana alone is running on NodePort

```
NAME                                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
alertmanager-operated                          ClusterIP   None            <none>        9093/TCP,9094/TCP,9094/UDP   31m
prometheus-operated                            ClusterIP   None            <none>        9090/TCP                     30m
prometheus-operator-alertmanager               ClusterIP   10.102.104.48   <none>        9093/TCP                     31m
prometheus-operator-grafana                    NodePort    10.96.160.172   <none>        80:31409/TCP                 31m
prometheus-operator-kube-state-metrics         ClusterIP   10.105.92.154   <none>        8080/TCP                     31m
prometheus-operator-operator                   ClusterIP   10.99.15.245    <none>        8080/TCP,443/TCP             31m
prometheus-operator-prometheus                 ClusterIP   10.109.75.138   <none>        9090/TCP                     31m
prometheus-operator-prometheus-node-exporter   ClusterIP   10.98.128.115   <none>        9100/TCP                     31m
```

## Configure SSH Tunnesl to access Grafana UI

You can skip this step, if you direct access to the Kubernetes Worker node IP from your machines. 

#### Windows using PuTTY

- Get the Kubernetes Worker Node IP and the Grafana Node Port. 
- Get the SSH server using which, Kubernetes Worker Node IP is accessible. Say this is Landing IP. 
- Configure the PuTTY as follows:
  - Create a new Session with Landing IP, Landing Port
  - Create a Connection -> SSH -> Tunnels  
    - Source Port = Grafana NodePort
    - Destination = Kubernetes Worker Node IP:Grafana Node Port
  - Open the PuTTY session. Enter SSH user name and passowrd for the Landing IP. 
- Now you can access Grafana UI at the following URL. Default login and password ( admin/prom-operator )
  ```
  http://localhost:<Grafana-Node-Port>/
  ``` 


#### Linux using SSH

- Get the Kubernetes Worker Node IP and the Grafana Node Port. 
- Get the SSH server using which, Kubernetes Worker Node IP is accessible. Say this is Landing IP. 
- Open SSH tunnel using the following command. 
  ```
  ssh -NL <Grafana-Node-Port>:<k8s-worker-node-IP>:<Grafana-Node-Port> <landing-machine-user>@<landing-machine-ip> -p <landing-machine-ssh-port>
  ```
- Now you can access Grafana UI at `http://localhost:<Grafana-Node-Port>/`. Default login and password ( admin/prom-operator )
  
## Verify Granafa Dashboard

- Login to Granfa UI 
- Click on Settings -> Data Source. You must see a Default Prometheus data source for `http://prometheus-operator-prometheus:9090/`
- Click on Dashboards -> Manage Dashboards. You must see a list of dashboards. Click on any of them like: `kubernetes-compute-resources-cluster`
- You must see some colors like:

![Grafana](https://github.com/kmova/bootstrap/blob/master/prometheus_operator/grafana_prom_operator.png?raw=true)


## References: 
- https://helm.sh/docs/intro/install/
- https://github.com/helm/charts/issues/19452
- https://www.linode.com/docs/networking/ssh/using-ssh-on-windows/#ssh-tunneling-port-forwarding
- https://github.com/helm/charts/tree/master/stable/prometheus-operator
- https://github.com/helm/charts/tree/master/stable/grafana#configuration