Grafana Cloud PDC agent and Istio
Tags: istio, pdc, kubernetes, grafanacloud
Published: 2023-08-13 17:18:18

When you need to setup **G**rafana **P**rivate **D**ata source **C**onnect in Kubernetes you need to apply some tricks in order to make it work.

Here I will describe what I did in order to use it.

Used links:
* [Official guide](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/configure-private-datasource-connect/#set-up-a-private-data-source-connection)

I created my custom helm chart that contains next `deployment.yaml`:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: {{ .Release.Name }}
    name: {{ .Release.Name }}
  name: {{ .Release.Name }}
spec:
  replicas: {{ .Values.minReplicas }}
  selector:
    matchLabels:
      name: {{ .Release.Name }}
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: {{ .Release.Name }}
      annotations:
        proxy.istio.io/config: |
          holdApplicationUntilProxyStarts: true
    spec:
      containers:
        - name: {{ .Release.Name }}
          env:
            - name: CLUSTER
              valueFrom:
                secretKeyRef:
                  key: cluster
                  name: {{ .Release.Name }}
            - name: HOSTED_GRAFANA_ID
              valueFrom:
                secretKeyRef:
                  key: hostedGrafanaId
                  name: {{ .Release.Name }}
            - name: TOKEN
              valueFrom:
                secretKeyRef:
                  key: token
                  name: {{ .Release.Name }}
          args:
            - -cluster
            - "$(CLUSTER)"
            - -gcloud-hosted-grafana-id
            - "$(HOSTED_GRAFANA_ID)"
            - -token
            - "$(TOKEN)"
            - -ssh-key-file
            - "/home/pdc/.ssh/grafana_pdc_v3"
          image: grafana/pdc-agent:{{ .Values.version }}
          imagePullPolicy: Always
          resources:
            limits:
              cpu: 1024m
              memory: 1Gi
            requests:
              cpu: 1024m
              memory: 1Gi
          securityContext:
            allowPrivilegeEscalation: false
            privileged: false
            runAsNonRoot: true
            capabilities:
              drop:
                - all
      securityContext:
        runAsUser: 30000
        runAsGroup: 30000
        fsGroup: 30000
      topologySpreadConstraints:
        - labelSelector:
            matchLabels:
              app: {{ .Release.Name }}
          maxSkew: 1
          minDomains: {{ .Values.minReplicas }}
          topologyKey: "kubernetes.io/hostname"
          whenUnsatisfiable: DoNotSchedule
          matchLabelKeys:
            - pod-template-hash
          nodeAffinityPolicy: Honor
          nodeTaintsPolicy: Honor
        - labelSelector:
            matchLabels:
              app: {{ .Release.Name }}
          maxSkew: 1
          minDomains: {{ .Values.minReplicas }}
          topologyKey: "topology.kubernetes.io/zone"
          whenUnsatisfiable: DoNotSchedule
          matchLabelKeys:
            - pod-template-hash
          nodeAffinityPolicy: Honor
          nodeTaintsPolicy: Honor
```

## Nuances
### Istio
#### Sidecar
Set `holdApplicationUntilProxyStarts: true` for the pods, so they will not start until istio sidecar not starts.

#### Access (optional)
If you not allow outbound traffic - set `ServiceEntry` that [will allow several urls](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/configure-private-datasource-connect/#before-you-begin).
What I have for `API` access:
```yaml
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: {{ .Values.name }}-api
spec:
  hosts:
    - private-datasource-connect-api-<cluster>.grafana.net
  location: MESH_EXTERNAL
  ports:
    - name: https
      number: 443
      protocol: HTTPS
  resolution: DNS
```

What I have for `ssh` access:
```yaml
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: {{ .Values.name }}-ssh
spec:
  hosts:
    - private-datasource-connect-<cluster>.grafana.net
  location: MESH_EXTERNAL
  ports:
    - name: tcp
      number: 22
      protocol: TCP
  resolution: DNS
```

### Grafana PDC config
#### Key Pair force regeneration
I set `-ssh-key-file` to `/home/pdc/.ssh/grafana_pdc_v3` because if there already host in allowed list(for ssh access) - then it not starts and fail in constant restarts.
This should be addressed in [GitHub issue](https://github.com/grafana/pdc-agent/issues/21)

#### Log level
Currently, in PDC agent log level set to `debug` level.
Unfortunately, as of today, when you use `-ssh-key-file` parameter you cannot change it.
This should be addressed in [GitHub issue](https://github.com/grafana/pdc-agent/issues/20)