Repository: risingwavelabs/risingwave
File: grafana/dashboard/dev/cluster_alerts.py (Python)

Code before:
                        ),
                        panels.target(
                            alert_when(
                                "changes(("
                                + 'kube_pod_container_status_last_terminated_timestamp{cluster=~"$cluster",namespace=~"$namespace",pod=~"$pod"} '
                                + "* on (namespace,pod,container) group_left (reason) "
                                + 'kube_pod_container_status_last_terminated_reason{cluster=~"$cluster",namespace=~"$namespace",pod=~"$pod",reason!~"Completed"}'
                                + ")[$__rate_interval:])"
                            ),
                            "[{{reason}}] {{container}} {{pod}}",
                        ),
                    ],
                    ["last"],
                ),
                panels.subheader(
                    "Storage Alerts",
                    """[Alert Reference]
- Lagging Version: the checkpointed or pinned version id is lagging behind the current version id. Check 'Hummock Manager' section in dev dashboard.
- Lagging Compaction: there are too many ssts in L0. This can be caused by compactor failure or lag of compactor resource. Check 'Compaction' section in dev dashboard, and take care of the type of 'Commit Flush Bytes' and 'Compaction Throughput', whether the throughput is too low.
- Lagging Vacuum: there are too many stale files waiting to be cleaned. This can be caused by compactor failure or lag of compactor resource. Check 'Compaction' section in dev dashboard.
- Abnormal Meta Cache Memory: the meta cache memory usage is too large, exceeding the expected 10 percent.
- Abnormal Block Cache Memory: the block cache memory usage is too large, exceeding the expected 10 percent.
- Abnormal Uploading Memory Usage: uploading memory is more than 70 percent of the expected, and is about to spill.
- Write Stall: Compaction cannot keep up. Stall foreground write, Check 'Compaction' section in dev dashboard.
- Abnormal Version Size: the size of the version is too large, exceeding the expected 300MB. Check 'Hummock Manager' section in dev dashboard.
- Abnormal Delta Log Number: the number of delta logs is too large, exceeding the expected 5000. Check 'Hummock Manager' and `Compaction` section in dev dashboard and take care of the type of 'Compaction Success Count', whether the number of trivial-move tasks spiking.
- Abnormal Pending Event Number: the number of pending events is too large, exceeding the expected 10000000. Check 'Hummock Write' section in dev dashboard and take care of the 'Event handle latency', whether the time consumed exceeds the barrier latency.
- Abnormal Object Storage Failure: object storage failures are occurring. Check 'Object Storage' section in dev dashboard and take care of the 'Object Storage Failure Rate', whether the rate is too high.
""",
                    height=10,
                ),
                panels.timeseries_count(
                    "Storage Alerts",
                    "",
                    [
                        panels.target(
                            alert_threshold(
                                f"{metric('storage_current_version_id')} - {metric('storage_checkpoint_version_id')}",
                                100,
                            ),
                            "Lagging Version (checkpoint)",
                        ),
                        panels.target(
                            alert_threshold(
                                f"{metric('storage_current_version_id')} - {metric('storage_min_pinned_version_id')}",
                                100,
                            ),
                            "Lagging Version (pinned)",
                        ),
                        panels.target(
                            alert_threshold(

Review comment (suggestion):
The phrase "Check 'Hummock Manager' and `Compaction` section" uses inconsistent quote styles. It should use consistent single quotes for both section names: "Check 'Hummock Manager' and 'Compaction' section" to maintain consistency with other references in the documentation.
```suggestion
- Abnormal Delta Log Number: the number of delta logs is too large, exceeding the expected 5000. Check 'Hummock Manager' and 'Compaction' section in dev dashboard and take care of the type of 'Compaction Success Count', whether the number of trivial-move tasks spiking.
```

Code after:
                                ">",
                            ),
                            "CPU Saturation (k8s limit) - {{namespace}}/{{pod}}",
                        ),
                        panels.target(
                            alert_when(
                                "changes(("
                                + 'kube_pod_container_status_last_terminated_timestamp{cluster=~"$cluster",namespace=~"$namespace",pod=~"$pod"} '
                                + "* on (namespace,pod,container) group_left (reason) "
                                + 'kube_pod_container_status_last_terminated_reason{cluster=~"$cluster",namespace=~"$namespace",pod=~"$pod",reason!~"Completed"}'
                                + ")[$__rate_interval:])"
                            ),
                            "[{{reason}}] {{container}} {{pod}}",
                        ),
                    ],
                    ["last"],
                ),
                panels.subheader(
                    "Storage Alerts",
                    """[Alert Reference]
- Lagging Version: the checkpointed or pinned version id is lagging behind the current version id. Check `Hummock Manager` section in dev dashboard.
- Lagging Compaction: there are too many ssts in L0. This can be caused by compactor failure or lag of compactor resource. Check `Compaction` section in dev dashboard, and take care of the type of `Commit Flush Bytes` and `Compaction Throughput`, whether the throughput is too low.
- Lagging Vacuum: there are too many stale files waiting to be cleaned. This can be caused by compactor failure or lag of compactor resource. Check `Compaction` section in dev dashboard.
- Abnormal Meta Cache Memory: the meta cache memory usage is too large, exceeding the expected 10 percent.
- Abnormal Block Cache Memory: the block cache memory usage is too large, exceeding the expected 10 percent.
- Abnormal Uploading Memory Usage: uploading memory is more than 70 percent of the expected, and is about to spill.
- Write Stall: Compaction cannot keep up. Stall foreground write, Check `Compaction` section in dev dashboard.
- Abnormal Version Size: the size of the version is too large, exceeding the expected 300MB. Check `Hummock Manager` section in dev dashboard.
- Abnormal Delta Log Number: the number of delta logs is too large, exceeding the expected 5000. Check `Hummock Manager` and `Compaction` section in dev dashboard and take care of the type of `Compaction Success Count`, whether the number of trivial-move tasks spiking.
- Abnormal Pending Event Number: the number of pending events is too large, exceeding the expected 10000000. Check `Hummock Write` section in dev dashboard and take care of the `Event handle latency`, whether the time consumed exceeds the barrier latency.
- Abnormal Object Storage Failure: object storage failures are occurring. Check `Object Storage` section in dev dashboard and take care of the `Object Storage Failure Rate`, whether the rate is too high.
""",
                    height=12,
                ),
                panels.timeseries_count(
                    "Storage Alert Signals",
                    "",
                    [
                        panels.target(
                            alert_threshold(
                                f"{metric('storage_current_version_id')} - {metric('storage_checkpoint_version_id')}",
                                100,
                            ),
                            "Lagging Version (checkpoint)",
                        ),
                        panels.target(
                            alert_threshold(
                                f"{metric('storage_current_version_id')} - {metric('storage_min_pinned_version_id')}",
                                100,
                            ),
                            "Lagging Version (pinned)",