Repository: risingwavelabs/risingwave
File: grafana/dashboard/dev/cluster_alerts.py (Python)

Code before:
from ..common import *
from . import section

@section
def _(outer_panels: Panels):
    panels = outer_panels
    return [
        outer_panels.row_collapsed(
            "Cluster Alerts",
            [
                panels.subheader(
                    "Streaming Alerts",
"""[Alert Reference]
- Recovery Triggered: cluster recovery is triggered. Check 'Errors by Type' / 'Node Count' panels to find the root cause. Check the error logs as well.
- Too Many Barriers: there are too many uncommitted barriers generated. This means the streaming graph is stuck.
  Check the following panels to follow-up:
  - Streaming Backfill: Check if there's any throughput in the panels, if yes, backfill is in progress. If throughput is high, it could lead to additional pressure on the stream graph.
  - Storage Alerts: Look at the alerts in the storage section, specifically `Write Stall`. That will cause backpressure and the stream graph being stuck.
  - Cluster Resource: If Relative CPU or Memory usage is high, it can lead to the stream graph being stuck.
  - Barrier Latency: Get the latency of the barrier, it should be high.
  - Streaming Relations: Look at the TopN relations by CPU usage and busy rate. These relations are likely to be the bottleneck.
  - Streaming Operators by Operator: Look at the alerts in the streaming operators by operator section, the following panels are more likely to be the bottleneck:
    - Merger Barrier Align: If the merger barrier align is high, it means the merger is not able to align the barriers in time.
    - Join Amplification: If the join amplification is high, it means the join is not able to process the data in time.
""",
                    height=5,
                ),
                panels.timeseries_count(
                    "Streaming Alerts",
                    "",
                    [
                        panels.target(
                            alert_when(
                                f"sum(rate({metric('recovery_latency_count')}[$__rate_interval])) by (recovery_type) + "
                                + f"sum(rate({metric('recovery_failure_cnt')}[$__rate_interval])) by (recovery_type)"
                            ),
                            "Recovery Triggered {{recovery_type}}",
                        ),
                        panels.target(
                            alert_threshold(metric("all_barrier_nums"), 200),
                            "Too Many Barriers {{database_id}}",
                        ),
                    ],
                    ["last"],

Review comment (suggestion):
The documentation uses inconsistent terminology. Line 15 says "the streaming graph is stuck" while line 18 says "the stream graph being stuck". Consider using consistent terminology throughout, preferably "streaming graph" to match the established terminology in line 15.
```suggestion
  - Streaming Backfill: Check if there's any throughput in the panels, if yes, backfill is in progress. If throughput is high, it could lead to additional pressure on the streaming graph.
  - Storage Alerts: Look at the alerts in the storage section, specifically `Write Stall`. That will cause backpressure and the streaming graph being stuck.
  - Cluster Resource: If Relative CPU or Memory usage is high, it can lead to the streaming graph being stuck.
```

Code after:
from ..common import *
from . import section

@section
def _(outer_panels: Panels):
    panels = outer_panels
    return [
        outer_panels.row_collapsed(
            "Cluster Alerts",
            [
                panels.subheader(
                    "Streaming Alerts",
"""[Alert Reference]
- Recovery Triggered: cluster recovery is triggered. Check `Errors by Type` / `Node Count` panels to find the root cause. Check the error logs as well.
- Too Many Barriers: there are too many uncommitted barriers generated. This means the streaming graph is stuck.
  Check the following panels to follow-up:
  - `Streaming Backfill`: Check if there's any throughput in the panels, if yes, backfill is in progress. If throughput is high, it could lead to additional pressure on the streaming graph.
  - `Storage Alerts`: Look at the alerts in the storage section, specifically `Write Stall`. That will cause backpressure and the streaming graph being stuck.
  - `Cluster Resource Alerts`: If Relative CPU or Memory usage is high, it can lead to the streaming graph being stuck.
  - `Barrier Latency`: Get the latency of the barrier, it should be high.
  - `Streaming Relations`: Look at the TopN relations by CPU usage and busy rate. These relations are likely to be the bottleneck.
  - `Streaming Operators by Operator`: Look at the alerts in the streaming operators by operator section, the following panels are more likely to be the bottleneck:
    - Merger Barrier Align: If the merger barrier align is high, it means the merger is not able to align the barriers in time.
    - Join Amplification: If the join amplification is high, it means the join is not able to process the data in time.
""",
                    height=9,
                ),
                panels.timeseries_count(
                    "Streaming Alert Signals",
                    "",
                    [
                        panels.target(
                            alert_when(
                                f"sum(rate({metric('recovery_latency_count')}[$__rate_interval])) by (recovery_type) + "
                                + f"sum(rate({metric('recovery_failure_cnt')}[$__rate_interval])) by (recovery_type)"
                            ),
                            "Recovery Triggered {{recovery_type}}",
                        ),
                        panels.target(
                            alert_threshold(metric("all_barrier_nums"), 200),
                            "Too Many Barriers {{database_id}}",
                        ),
                    ],
                    ["last"],