Grafana dashboard: Loki:NamespaceLogs
Description: Loki dashboards for promtail metrics

# grafana  + MySQL + InfluxDB+ telegraf + Loki + promtail 
![telegraf UI](https://github.com/AbeYuki/monitoring-k8s/raw/main/docs/ui-telegraf-network.png)
![telegraf UI](https://github.com/AbeYuki/monitoring-k8s/raw/main/docs/ui-telegraf-resources.png)
![loki UI](https://github.com/AbeYuki/monitoring-k8s/raw/main/docs/ui-loki.png)

# My Dashbards
After installing git monitoring-k8s,All settings are complete.  
If you want to use git monitoring-k8s without installing it, delete the regular expression described in the variable.  
git の monitoring-k8s をインストールすると、全ての設定が完了します。  
git の monitoring-k8s インストールしないで利用する場合は、変数に記載している正規表現を削除してください。  
[Loki:AccessLogs](https://grafana.com/grafana/dashboards/16226)  
[Loki:NamespaceLogs](https://grafana.com/grafana/dashboards/16227)  
[Loki:SystemLogs](https://grafana.com/grafana/dashboards/16228)  
[Telegraf:KubernetesResources](https://grafana.com/grafana/dashboards/16229)  
[Telegraf:SystemResources](https://grafana.com/grafana/dashboards/16230)  

# setup
```
git clone https://github.com/AbeYuki/monitoring-k8s.git
```


## はじめに
README.md ファイルがある場所へ移動  
シークレットを含んだファイル(★add, ★fix)の追加、修正を行い deploy する流れ  
cat リダイレクトでファイル作成例としているが、エディタでの作成を推奨  
```
.
├── README.md
├── grafana
│   ├── deployment-backend-grafana-db01.yaml
│   ├── deployment-frontend-grafana-app01.yaml
│   └── grafana.ini ★add
├── influxdb
│   ├── configmap-influxdb-init01.yaml
│   └── deployment-backend-influxdb-db01.yaml
├── kustomization.yaml ★fix
├── label-transformer.yaml
├── loki
│   ├── config.yaml
│   └── deployment-frontend-loki-app01.yaml
├── namespace.yaml
├── password.txt ★add
├── promtail
│   ├── config.yaml
│   └── daemonset-backend-agent01.yaml
├── rbac
│   ├── rbac-influxdb.yaml
│   ├── rbac-prometheus.yaml
│   ├── rbac-promtail.yaml
│   └── rbac-telegraf.yaml
├── telegraf
│   ├── daemonset-backend-agent01.yaml
│   └── telegraf.conf ★add
└── token.txt ★add
```


## kustomize.yaml setup
secretGenerator で作成するパスワード、トークンファイル作成
```bash
echo -n 'password' > password.txt
```
```bash
echo -n 'token' > token.txt
```


## grafana setup
kustomization.yaml で指定したパスワードに修正し、grafana/grafana.ini ファイル作成

```
[database]  
  password = 修正
  mysql://grafana:修正@monitoring-backend-grafana-db01-001:3306/grafana

[session]  
  provider_config = `grafana:修正@tcp(monitoring-backend-grafana-db01-001:3306)/grafana` 
```

password に "#" または ";" が含まれている場合は三重引用符にする必要がある  
例)  
```
[database]
  password = #password; -> 誤
[database]
  password = """#password;""" -> 正
```

```conf
cat <<'EOF'> grafana/grafana.ini
[server]
  protocol = http
  http_port = 3000
[database]
  type = mysql
  host = monitoring-backend-grafana-db01-001:3306
  name = grafana
  user = grafana
  password = password
  ssl_mode = disable
  url = mysql://grafana:password@monitoring-backend-grafana-db01-001:3306/grafana
[session]
  provider_config = `grafana:password@tcp(monitoring-backend-grafana-db01-001:3306)/grafana` 
  provider = mysql
[analytics]
  reporting_enabled = false
  check_for_updates = true
[log]
  mode = console
  level = info
[paths]
  data         = /var/lib/grafana/data
  logs         = /var/log/grafana
  plugins      = /var/lib/grafana/plugins
  provisioning = /etc/grafana/provisioning
[unified_alerting]
  enabled = true
[alerting]
[annotations.api]
[annotations.dashboard]
[annotations]
[auth.anonymous]
[auth.azuread]
[auth.basic]
[auth.generic_oauth]
[auth.github]
[auth.gitlab]
[auth.google]
[auth.grafana_com]
[auth.jwt]
[auth.ldap]
[auth.okta]
[auth.proxy]
[auth]
[aws]
[azure]
[dashboards]
[dataproxy]
[datasources]
[date_formats]
[emails]
[enterprise]
[explore]
[expressions]
[external_image_storage.azure_blob]
[external_image_storage.gcs]
[external_image_storage.local]
[external_image_storage.s3]
[external_image_storage.webdav]
[external_image_storage]
[feature_toggles]
[geomap]
[grafana_com]
[live]
[log.console]
[log.file]
[log.frontend]
[log.syslog]
[metrics.environment_info]
[metrics.graphite]
[metrics]
[panels]
[plugin.grafana-image-renderer]
[plugins]
[quota]
[remote_cache]
[rendering]
[security]
[smtp]
[snapshots]
[tracing.jaeger]
[unified_alerting]
[users]
EOF
```



## telegraf setup
kustomization.yaml で指定したトークンに修正、disk 等の監視対象の調整を行い telegraf/telegraf.conf ファイル作成  
```
[[outputs.influxdb_v2]]  
  token = "修正"  
[[inputs.disk]]  
  fstype = [ "ext4", "xfs" ]  
  path = [ "/", "/backup", "/var/lib/longhorn" ]  
```
```bash
cat <<'EOF'> telegraf/telegraf.conf
[agent]
  interval = "60s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = ""
  hostname = "$HOSTNAME"
  omit_hostname = false
[[outputs.influxdb_v2]]
  urls = ["http://monitoring-backend-influxdb-db01-001:8086"]
  token = "修正"
  organization = "monitoirng"
  bucket = "monitoring"
  timeout = "5s"    
[[inputs.cpu]]
  percpu = true
  totalcpu = true
  fielddrop = ["time_*"]
[[inputs.system]]
[[inputs.disk]]
  ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs"]
  mount_points = ["/", "/backup", "/var/lib/longhorn"]
[inputs.disk.tagpass]
  fstype = [ "ext4", "xfs" ]
  path = [ "/", "/backup", "/var/lib/longhorn" ]
[[inputs.diskio]]
  devices = ["sd*"]
[[inputs.kernel]]
[[inputs.mem]]
[[inputs.processes]]
[[inputs.swap]]
[[inputs.system]]
[[inputs.net]]
[[inputs.netstat]]
[[inputs.interrupts]]
[[inputs.linux_sysctl_fs]]
[[inputs.docker]]
  endpoint = "unix:///var/run/docker.sock"
[[inputs.kubernetes]]
  url = "https://$HOSTIP:10250"
  bearer_token = "/run/secrets/kubernetes.io/serviceaccount/token"
  insecure_skip_verify = true
EOF
```



## kustomization.yaml の patchesStrategicMerge を修正して storageclass,resources の調整
```
patchesStrategicMerge:
- |-
  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: frontend-grafana-app01
  spec:
    accessModes:
      - ReadWriteMany
    resources:
      requests:
        storage: 調整
    storageClassName: 調整
```

## docker.sock の権限変更
docker metrics を収集する場合、 docker.sock の権限を 666 に変更する
```
sudo chmod 666 /var/run/docker.sock
```
```
vagrant@ubuntu2004:~$ ls -l /var/run/docker.sock
srw-rw---- 1 root docker 0 May  9 05:29 /var/run/docker.sock
vagrant@ubuntu2004:~$ sudo chmod 666 /var/run/docker.sock
vagrant@ubuntu2004:~$ ls -l /var/run/docker.sock
srw-rw-rw- 1 root docker 0 May  9 05:29 /var/run/docker.sock
vagrant@ubuntu2004:~$ 
```


# deploy 
## deploy namespace
```
kubectl apply -f namespace.yaml
```

## deploy resource
```
kubectl apply -k ./
```




# Grafana datasource setting
## Influxdb setting
- Query Language
  - Flux
- url
  - http://monitoring-backend-influxdb-db01-001:8086
- Access
  - Server(default)
- InfluxDB Details
  - Organization
    - monitoring
  - Token
    - token
  - Default Bucket
    - monitoring

![datasource-influxdb](https://github.com/AbeYuki/monitoring-k8s/raw/main/docs/datasource-influxdb.png)




## Loki setting
- HTTP
  - URL
    - http://monitoring-frontend-loki-app01-001.monitoring.svc.cluster.local:3100

![datasource-influxdb](https://github.com/AbeYuki/monitoring-k8s/raw/main/docs/datasource-loki.png)

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "datasource",
          "uid": "grafana"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "target": {
          "limit": 100,
          "matchAny": false,
          "tags": [],
          "type": "dashboard"
        },
        "type": "dashboard"
      }
    ]
  },
  "description": "Loki dashboards for promtail metrics",
  "editable": true,
  "fiscalYearStartMonth": 0,
  "gnetId": 16227,
  "graphTooltip": 0,
  "id": 2,
  "links": [],
  "liveNow": false,
  "panels": [
    {
      "datasource": {
        "type": "loki",
        "uid": "Vn8F1nDVk"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "area"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 4,
        "w": 24,
        "x": 0,
        "y": 0
      },
      "id": 5,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "single",
          "sort": "none"
        }
      },
      "pluginVersion": "8.4.2",
      "targets": [
        {
          "datasource": {
            "type": "loki",
            "uid": "Vn8F1nDVk"
          },
          "expr": "count_over_time( {namespace=\"$namespace\"} |~ \"info\" [5m])",
          "hide": false,
          "queryType": "range",
          "refId": "A"
        }
      ],
      "title": "info",
      "transformations": [],
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "loki",
        "uid": "Vn8F1nDVk"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "area"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 4,
        "w": 24,
        "x": 0,
        "y": 4
      },
      "id": 11,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "single",
          "sort": "none"
        }
      },
      "pluginVersion": "8.4.2",
      "targets": [
        {
          "datasource": {
            "type": "loki",
            "uid": "Vn8F1nDVk"
          },
          "expr": "count_over_time( {namespace=\"$namespace\"} |~ \"warning\" [5m])",
          "hide": false,
          "queryType": "range",
          "refId": "A"
        }
      ],
      "title": "warning",
      "transformations": [],
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "loki",
        "uid": "Vn8F1nDVk"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "area"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 4,
        "w": 24,
        "x": 0,
        "y": 8
      },
      "id": 6,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "single",
          "sort": "none"
        }
      },
      "pluginVersion": "8.4.2",
      "targets": [
        {
          "datasource": {
            "type": "loki",
            "uid": "Vn8F1nDVk"
          },
          "expr": "count_over_time( {namespace=\"$namespace\"} |~ \"alert\" [5m])",
          "hide": false,
          "queryType": "range",
          "refId": "A"
        }
      ],
      "title": "alert",
      "transformations": [],
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "loki",
        "uid": "Vn8F1nDVk"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "area"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 4,
        "w": 24,
        "x": 0,
        "y": 12
      },
      "id": 4,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "single",
          "sort": "none"
        }
      },
      "pluginVersion": "8.4.2",
      "targets": [
        {
          "datasource": {
            "type": "loki",
            "uid": "Vn8F1nDVk"
          },
          "expr": "count_over_time( {namespace=\"$namespace\"} |~ \"error\" [5m])",
          "hide": false,
          "queryType": "range",
          "refId": "A"
        }
      ],
      "title": "error",
      "transformations": [],
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "loki",
        "uid": "Vn8F1nDVk"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "area"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 4,
        "w": 24,
        "x": 0,
        "y": 16
      },
      "id": 9,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "single",
          "sort": "none"
        }
      },
      "pluginVersion": "8.4.2",
      "targets": [
        {
          "datasource": {
            "type": "loki",
            "uid": "Vn8F1nDVk"
          },
          "expr": "count_over_time( {namespace=\"$namespace\"} |~ \"crit\" [5m])",
          "hide": false,
          "queryType": "range",
          "refId": "A"
        }
      ],
      "title": "crit",
      "transformations": [],
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "loki",
        "uid": "Vn8F1nDVk"
      },
      "gridPos": {
        "h": 6,
        "w": 24,
        "x": 0,
        "y": 20
      },
      "id": 2,
      "options": {
        "dedupStrategy": "numbers",
        "enableLogDetails": true,
        "prettifyLogMessage": false,
        "showCommonLabels": false,
        "showLabels": false,
        "showTime": true,
        "sortOrder": "Descending",
        "wrapLogMessage": false
      },
      "targets": [
        {
          "datasource": {
            "type": "loki",
            "uid": "Vn8F1nDVk"
          },
          "expr": "{namespace=\"$namespace\"} |~ \"info\"",
          "refId": "A"
        }
      ],
      "title": "info_Logs",
      "type": "logs"
    },
    {
      "datasource": {
        "type": "loki",
        "uid": "Vn8F1nDVk"
      },
      "gridPos": {
        "h": 6,
        "w": 24,
        "x": 0,
        "y": 26
      },
      "id": 13,
      "options": {
        "dedupStrategy": "numbers",
        "enableLogDetails": true,
        "prettifyLogMessage": false,
        "showCommonLabels": false,
        "showLabels": false,
        "showTime": true,
        "sortOrder": "Descending",
        "wrapLogMessage": false
      },
      "targets": [
        {
          "datasource": {
            "type": "loki",
            "uid": "Vn8F1nDVk"
          },
          "expr": "{namespace=\"$namespace\"} |~ \"warning\"",
          "refId": "A"
        }
      ],
      "title": "warning_Logs",
      "type": "logs"
    },
    {
      "datasource": {
        "type": "loki",
        "uid": "Vn8F1nDVk"
      },
      "gridPos": {
        "h": 6,
        "w": 24,
        "x": 0,
        "y": 32
      },
      "id": 12,
      "options": {
        "dedupStrategy": "numbers",
        "enableLogDetails": true,
        "prettifyLogMessage": false,
        "showCommonLabels": false,
        "showLabels": false,
        "showTime": true,
        "sortOrder": "Descending",
        "wrapLogMessage": false
      },
      "targets": [
        {
          "datasource": {
            "type": "loki",
            "uid": "Vn8F1nDVk"
          },
          "expr": "{namespace=\"$namespace\"} |~ \"alert\"",
          "refId": "A"
        }
      ],
      "title": "alart_Logs",
      "type": "logs"
    },
    {
      "datasource": {
        "type": "loki",
        "uid": "Vn8F1nDVk"
      },
      "gridPos": {
        "h": 6,
        "w": 24,
        "x": 0,
        "y": 38
      },
      "id": 8,
      "options": {
        "dedupStrategy": "numbers",
        "enableLogDetails": true,
        "prettifyLogMessage": false,
        "showCommonLabels": false,
        "showLabels": false,
        "showTime": true,
        "sortOrder": "Descending",
        "wrapLogMessage": false
      },
      "targets": [
        {
          "datasource": {
            "type": "loki",
            "uid": "Vn8F1nDVk"
          },
          "expr": "{namespace=\"$namespace\"} |~ \"error\"",
          "refId": "A"
        }
      ],
      "title": "error_Logs",
      "type": "logs"
    },
    {
      "datasource": {
        "type": "loki",
        "uid": "Vn8F1nDVk"
      },
      "gridPos": {
        "h": 6,
        "w": 24,
        "x": 0,
        "y": 44
      },
      "id": 10,
      "options": {
        "dedupStrategy": "numbers",
        "enableLogDetails": true,
        "prettifyLogMessage": false,
        "showCommonLabels": false,
        "showLabels": false,
        "showTime": true,
        "sortOrder": "Descending",
        "wrapLogMessage": false
      },
      "targets": [
        {
          "datasource": {
            "type": "loki",
            "uid": "Vn8F1nDVk"
          },
          "expr": "{namespace=\"$namespace\"} |~ \"crit\"",
          "refId": "A"
        }
      ],
      "title": "crit_Logs",
      "type": "logs"
    },
    {
      "datasource": {
        "type": "loki",
        "uid": "Vn8F1nDVk"
      },
      "gridPos": {
        "h": 15,
        "w": 24,
        "x": 0,
        "y": 50
      },
      "id": 7,
      "options": {
        "dedupStrategy": "numbers",
        "enableLogDetails": true,
        "prettifyLogMessage": false,
        "showCommonLabels": false,
        "showLabels": false,
        "showTime": true,
        "sortOrder": "Descending",
        "wrapLogMessage": false
      },
      "targets": [
        {
          "datasource": {
            "type": "loki",
            "uid": "Vn8F1nDVk"
          },
          "expr": "{namespace=\"$namespace\"}",
          "refId": "A"
        }
      ],
      "title": "Logs",
      "type": "logs"
    }
  ],
  "refresh": false,
  "schemaVersion": 37,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": [
      {
        "current": {
          "selected": false,
          "text": "argocd",
          "value": "argocd"
        },
        "datasource": {
          "type": "loki",
          "uid": "Vn8F1nDVk"
        },
        "definition": "label_values(namespace)",
        "hide": 0,
        "includeAll": false,
        "multi": false,
        "name": "namespace",
        "options": [],
        "query": "label_values(namespace)",
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 0,
        "type": "query"
      }
    ]
  },
  "time": {
    "from": "now-12h",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "",
  "title": "Loki:NamespaceLogs",
  "uid": "K8jJysL7k",
  "version": 2,
  "weekStart": ""
}