---
id: DAV_task_16_prometheus_alert_silence_grafana
name: 值班 alert 风暴定位、silence 与规则修复
category: DAV
timeout_seconds: 1500
---

## Prompt
You are the on-call engineer for the checkout-api team. Starting at 03:14, your phone was blown up by PagerDuty for 30 minutes, and multiple alerts on the alerting dashboard are flapping non-stop. The SRE Lead wants three things done before your shift hand-off:

1. Silence **the alert that is actually flapping** for ~2 hours (use the Alertmanager Silence mechanism; **do not** delete the rule_file);
2. Identify another **fake alert that fires forever due to a broken expression** (hint: comparing a monotonically increasing counter directly with `> 0` — always true), and provide a fixed `fixed_rules.yml`;
3. Write a postmortem of ≥ 800 characters explaining the root cause, why it was not detected automatically at the time, and a regression test plan.

Environment and assets (everything is already up; credentials `admin/admin`):
- `http://localhost:9090/`: Prometheus (`/alerts` `/rules` `/api/v1/...`).
- `http://localhost:9093/`: Alertmanager UI (`/#/silences` `/#/alerts` `/api/v2/...`).
- `http://localhost:7016/`: Grafana 10.4, dashboard "Checkout API on-call"; left sidebar ☰ → Alerting provides Alert rules / Silences / Contact points / State history.
- `http://127.0.0.1:9097/`: webhook sink, writes to `/tmp/webhook.log`; every alertmanager POST lands there.
- `/tmp_workspace/dash16/prometheus/rules/noisy_alerts.yml`: the currently in-use (broken) rule file — **do not modify it directly**; grading uses this file as the baseline.

Task objectives:
- Use the Alertmanager Silence mechanism to silence the truly flapping alert for ~2 hours (duration falls within [115min, 135min]);
- Patch the always-true-expression alert via a `fixed_rules.yml` (after the fix, `expr` must wrap the counter with `rate(...)` / `irate(...)` / `increase(...)` and the comparison threshold must be changed to ≥ 0.01);
- Leave joint evidence across three channels: structured data + rendered UI + network traffic;
- Write a qualifying postmortem.

Produce the following deliverables under `/tmp_workspace/results/`:

1. **Alert and rule evidence (before)**
   - `firing_alerts.json`: list of alerts currently firing, schema:
     ```json
     {"timestamp": "...", "firing": [{"name": "...", "service": "...", "value": "..."}, ...]}
     ```
     At least 2 firing records, and at least one of the flapping alert or the always-true alert should be findable among them.
   - `promtool_check_before.txt`: promtool check result for the original `noisy_alerts.yml` (keep both stdout/stderr) as a snapshot of the rule state before the fix.

2. **Silence evidence**
   - `silences.json`: structured dump of all current Alertmanager silences; for the flapping alert, there must exist a silence satisfying all of the following:
     - The `matchers` list contains `name == "alertname"` and `value == <name of the flapping alert>`;
     - `isRegex == true` is not allowed (must be equality match);
     - The silence's `endsAt - startsAt` falls within a ~2-hour range (accepted range 6900s ~ 8100s, i.e. ~115~135 min).
   - `amtool_silences.txt`: amtool's human-readable list of current silences; must contain the name of the silenced alert.

3. **Network evidence that the silence actually took effect**
   - `webhook_after_silence.pcap` (or `webhook_after_silence.txt`, either one): packet capture of alertmanager POST traffic on the :9097 webhook within 60 seconds after the silence was created;
   - `webhook_tail.log`: copy of the tail of `/tmp/webhook.log` (≥ 30 lines).
   - In both pieces of evidence above, the name of the silenced alert must **no longer appear** (proving the silence really intercepted the webhook push); meanwhile, the other still-firing alert (the always-true-expression one) should still appear.

4. **Fixed rules and unit tests**
   - `fixed_rules.yml`: a valid prometheus rule file; `promtool check rules` must exit 0; for the always-true alert, its `expr` must contain one of `rate(` / `irate(` / `increase(` and compare against a threshold ≥ 0.01.
   - `test_rules.yml`: a minimal usable promtool unit test (≥ 2 input series + ≥ 2 alert_test).
   - `promtool_test_after.txt`: the result of running that unit test; must contain a success marker such as `SUCCESS` / `PASS` / `0 failures`.

5. **Visual evidence** (each PNG ≥ 5 KB, from actually rendered Grafana / Alertmanager pages; PIL compositing is not allowed)
   - `view_alert_state_history.png`: Grafana → Alerting → State history view of the flapping alert; the image must show red/green bands indicating the alert state repeatedly switching between firing↔normal (i.e., the flap frequency exists only in pixel color).
   - `view_silence_create.png`: Silence detail page (Alertmanager UI or Grafana → Alerting → Silences); must show an "Active" badge + a valid alertname matcher + an Expires time ~2 hours in the future.
   - `view_annotate_panel.png`: screenshot of a region annotation on the "API error ratio" panel on the dashboard — must show a red (or colored) horizontal region overlaying the flap segment + an annotation text bubble (suggested content along the lines of "flap window — silenced, see postmortem for root cause").
   - `view_edit_inspect.png`: Edit / Inspect → Data drawer of the "JVM old-gen GC pause" panel on the dashboard; must show the csv table / Stats sub-tab labels.
   - `view_alert_resolved.png`: in the Grafana → Alerting → Alert rules list, the state badge of the silenced alert has changed to "Silenced" or "Normal".

6. **Postmortem**
   - `postmortem.md`: ≥ 800 characters; suggested structure: 1) incident timeline → 2) root cause (must hit both layers: `counter without rate()` and `threshold too tight causing flap`) → 3) current workaround (silence duration / patch location) → 4) long-term fix → 5) regression test plan (mention `promtool test rules`); hit ≥ 4/6 keywords (`flap`, `canary`, `rate(`, `silence`, `threshold`, `counter`).

Hard constraints:
- The Silence matcher must be an `alertname=<name>` equality match; regex matchers are not accepted.
- The fix must land in the new file `fixed_rules.yml`; **do not** modify `/tmp_workspace/dash16/prometheus/rules/noisy_alerts.yml`.
- The invariant "the silenced alert no longer appears in the network evidence" must hold; a gray UI badge alone is not enough.
- Screenshots must come from actually rendered pages (must genuinely show key UIs such as Alertmanager Silences editor, Inspect, and Grafana Annotation / State history).

## Expected Behavior

成功的解答会留下"告警快照 + silence + 网络流量 + 规则补丁 + 可视复验"五路并齐的证据：

- `firing_alerts.json` + `promtool_check_before.txt` 给出"修复前的规则态势 + 当前 firing 列表"。
- `silences.json` + `amtool_silences.txt` 共同证明：那条真正在 flap 的 alert 已被 alertname 等值 matcher silence ~2 小时。
- 一份 60s 抓包 + `/tmp/webhook.log` 末段：被 silence 的 alert 名在 webhook 流量里**消失**，而另一条仍在 fire 的 alert 名**还在**——这是 silence 真正生效（而不是只 UI 灰显）的硬证据。
- `fixed_rules.yml`（promtool check 通过、`expr` 用 `rate(...)` 之类包裹 counter 并与 ≥ 0.01 阈值比较）+ `test_rules.yml` + 一份成功的 `promtool_test_after.txt`，构成修复的可重放证据。
- 5 张 Grafana / Alertmanager 截图，按 State history flap → Silence Active → panel region annotation → Edit / Inspect → Alert rules 列表 silenced badge 的顺序铺出可视复验链。
- 一份 ≥ 800 字、命中 ≥ 4/6 关键词的中文 postmortem，把"counter without rate" 与 "threshold too紧导致 flap" 这两层根因 + silence + patch + 回归测试串成一份可交付的事故报告。

## Source
- Prometheus 官方文档 "Alerting rules" + "Common pitfalls": https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/
- Grafana 10.x Alerting "Silences" 文档: https://grafana.com/docs/grafana/v10.4/alerting/manage-notifications/create-silence/
- 真实场景灵感：reddit r/devops 2024-08 "我们一周被 flapping alert 炸了 3000 次，CTO 让查根因"
- 对应 EyesOn-Bench DAV 类: D16 — alert noise triage with Silence + rule patch

## Grading Criteria

- [ ] 1. `/tmp_workspace/results/firing_alerts.json` 存在且 schema 正确，至少含 2 条 firing 记录
- [ ] 2. `promtool_check_before.txt` 与 `promtool_test_after.txt` 都存在，后者必须包含 "SUCCESS" 或 exit 0 标志
- [ ] 3. `silences.json` 存在且 matcher 是 `alertname=<那条 flap 的 alert>` 等值匹配，duration 在 [115min, 135min] 之间
- [ ] 4. `amtool_silences.txt` 非空且包含被 silence 的 alert 名
- [ ] 5. tcpdump / webhook 证据存在且文本里**不再出现**被 silence 的 alert 名
- [ ] 6. `fixed_rules.yml` 存在；`promtool check rules` 通过；被 patch 的那条 alert expr 含 `rate(` / `irate(` / `increase(`，并有合理阈值
- [ ] 7. 5 张 GUI 截图都存在，OCR 命中各自关键字（state history / silences / annotation / edit / inspect / silenced 等）
- [ ] 8. `view_annotate_panel.png` 中 VLM judge 能识别出红色 region annotation（rubric "vlm_annotation_visible"）
- [ ] 9. `postmortem.md` ≥ 800 字符，命中关键词 ≥ 4/6（flap、canary、rate(、silence、threshold、counter）
- [ ] 10. 整体 cross-channel 证据链完整：CLI (curl/jq/promtool/amtool/tcpdump) + GUI (state history/silence/annotate/edit/resolved) ≥ 5 次切换

## Automated Checks

```python
def grade(workspace_path=None, **kwargs) -> dict:
    """DAV_task_16 grader: alert silence + rule patch + cross-channel evidence."""
    import json, os, re, subprocess
    from pathlib import Path
    rd = Path("/tmp_workspace/results")
    # Read GT from the root-only path the warmup moved expected.json to
    # (kept off the agent-visible /tmp_workspace/gt to prevent answer
    # leakage). Fall back to /tmp_workspace/gt for legacy mounts.
    gt_dir = Path("/opt/dav16_gt") if Path("/opt/dav16_gt/expected.json").exists() else Path("/tmp_workspace/gt")
    s = {}
    expected = {}
    if (gt_dir / "expected.json").exists():
        try: expected = json.loads((gt_dir / "expected.json").read_text())
        except Exception: expected = {}
    noisy = expected.get("noisy_alert_to_silence", "HighErrorRate")
    patch_target = expected.get("rule_to_patch_alertname", "JvmOldGcStorm")
    rate_kw = expected.get("patched_expr_must_contain_any", ["rate(", "irate(", "increase("])

    # 1. firing_alerts.json
    fa = rd / "firing_alerts.json"
    s["firing_alerts_present"] = 1.0 if fa.exists() else 0.0
    firing_ok = 0.0
    if fa.exists():
        try:
            d = json.loads(fa.read_text())
            arr = d.get("firing") if isinstance(d, dict) else d
            if isinstance(arr, list) and len(arr) >= expected.get("expected_firing_alert_count_min", 2):
                names = {(a.get("name") or a.get("alertname") or "") for a in arr if isinstance(a, dict)}
                hits = sum(1 for tgt in (noisy, patch_target) if any(tgt in n for n in names))
                firing_ok = 1.0 if hits == 2 else (0.5 if hits == 1 else 0.0)
        except Exception:
            pass
    s["firing_alerts_schema"] = firing_ok

    # 2. promtool check before/after
    s["promtool_check_before"] = 1.0 if (rd / "promtool_check_before.txt").exists() and (rd / "promtool_check_before.txt").stat().st_size > 0 else 0.0
    test_after = rd / "promtool_test_after.txt"
    test_after_ok = 0.0
    if test_after.exists():
        txt = test_after.read_text(errors="ignore")
        if "SUCCESS" in txt or "PASS" in txt or re.search(r"\b0 failures?\b|Unit Testing.*SUCCESS", txt, re.I):
            test_after_ok = 1.0
        elif len(txt) > 20:
            test_after_ok = 0.4
    s["promtool_test_after"] = test_after_ok

    # 3. silences.json structure + matcher + duration
    sil = rd / "silences.json"
    sil_ok = 0.0
    duration_min = 0.0
    sil_matcher_ok = 0.0
    if sil.exists():
        try:
            d = json.loads(sil.read_text())
            items = d if isinstance(d, list) else (d.get("data") or d.get("silences") or [d])
            for it in items:
                ms = it.get("matchers", [])
                names = {m.get("name", ""): m.get("value", "") for m in ms if isinstance(m, dict)}
                if names.get("alertname") == noisy and not any(m.get("isRegex") for m in ms):
                    sil_matcher_ok = 1.0
                    # parse start/end ISO
                    try:
                        from datetime import datetime
                        def _p(x): return datetime.fromisoformat(x.replace("Z", "+00:00"))
                        dur = (_p(it["endsAt"]) - _p(it["startsAt"])).total_seconds()
                        if expected.get("silence_min_duration_seconds", 6900) <= dur <= expected.get("silence_max_duration_seconds", 8100):
                            duration_min = 1.0
                        elif dur > 1800:
                            duration_min = 0.5
                    except Exception:
                        duration_min = 0.3
                    break
            sil_ok = 1.0
        except Exception:
            pass
    s["silences_present"] = sil_ok
    s["silence_matcher_correct"] = sil_matcher_ok
    s["silence_duration_in_range"] = duration_min

    # 4. amtool silences output
    am = rd / "amtool_silences.txt"
    am_txt = am.read_text(errors="ignore") if am.exists() else ""
    s["amtool_evidence"] = 1.0 if (noisy in am_txt and re.search(r"(ID|Matchers|Expires)", am_txt)) else 0.0

    # 5. tcpdump / webhook tail — silenced alert name should NOT appear
    captured_text = ""
    for n in ("webhook_after_silence.txt", "webhook_tail.log"):
        p = rd / n
        if p.exists():
            captured_text += "\n" + p.read_text(errors="ignore")
    pcap = rd / "webhook_after_silence.pcap"; pcap_strings = ""
    if pcap.exists() and pcap.stat().st_size > 1024:
        try:
            pcap_strings = subprocess.run(["strings", str(pcap)], capture_output=True, text=True, timeout=15).stdout
        except Exception:
            pass
        captured_text += "\n" + pcap_strings
    pcap_ok = bool(re.search(r"(POST\s+/|alertname|alertmanager)", pcap_strings))
    tail_ok = bool(re.search(r"(POST|alertname|status)", captured_text)) and len(captured_text) > 200
    has_capture = pcap_ok or tail_ok
    s["tcpdump_evidence_present"] = 1.0 if has_capture else 0.0
    if captured_text and has_capture:
        s["silence_effect_in_capture"] = 1.0 if (noisy not in captured_text and patch_target in captured_text) else 0.0
    else:
        s["silence_effect_in_capture"] = 0.0

    # 6. fixed_rules.yml — promtool check + expr contains rate( + threshold > 0.0X
    fr = rd / "fixed_rules.yml"
    fr_check = 0.0; fr_rate = 0.0; fr_thr = 0.0
    if fr.exists():
        try:
            r = subprocess.run(["promtool", "check", "rules", str(fr)], capture_output=True, text=True, timeout=20)
            if r.returncode == 0:
                fr_check = 1.0
        except Exception:
            fr_check = 0.0
        body = fr.read_text(errors="ignore")
        # find the patched alert block
        m = re.search(r"alert:\s*" + re.escape(patch_target) + r"\b[\s\S]{0,400}?expr:\s*([^\n]+)", body)
        if m:
            expr = m.group(1)
            if any(k in expr for k in rate_kw) and "jvm_old_gc_pause_seconds_total" in expr and re.search(r"\[\s*\d+[smh]\s*\]", expr):
                fr_rate = 1.0
            thr = re.search(r">\s*([0-9]*\.?[0-9]+)", expr)
            thr_min = expected.get("patched_expr_threshold_min", 0.01)
            if thr and thr_min <= float(thr.group(1)) <= 1.0:
                fr_thr = 1.0
    s["fixed_rules_promtool_ok"] = fr_check
    s["fixed_rules_uses_rate"] = fr_rate
    s["fixed_rules_threshold_ok"] = fr_thr

    # 7. GUI screenshots (5)
    gui_shots = [
        "view_alert_state_history.png",
        "view_silence_create.png",
        "view_annotate_panel.png",
        "view_edit_inspect.png",
        "view_alert_resolved.png",
    ]
    present = sum(1 for n in gui_shots if (rd / n).exists() and (rd / n).stat().st_size > 5000)
    s["gui_screenshots_count"] = present / len(gui_shots)
    # OCR keywords
    ocr_hits = 0
    try:
        import pytesseract
        from PIL import Image
        kws = {
            "view_alert_state_history.png": ["State", "history", "Alert", "Normal", "Firing"],
            "view_silence_create.png": ["Silence", "matcher", "Expires", "Active", "alertname"],
            "view_annotate_panel.png": ["Annotation", "annotation", "flap", "error", "ratio"],
            "view_edit_inspect.png": ["Edit", "Inspect", "Query", "Data", "Stats"],
            "view_alert_resolved.png": ["Silenced", "Normal", "Alert", "rules", "Alerting"],
        }
        for n, ks in kws.items():
            p = rd / n
            if p.exists():
                try:
                    tx = pytesseract.image_to_string(Image.open(p))
                    if any(k in tx for k in ks): ocr_hits += 1
                except Exception:
                    pass
        s["gui_screenshots_ocr"] = ocr_hits / len(gui_shots)
    except ImportError:
        s["gui_screenshots_ocr"] = 0.5

    # 8. postmortem
    pm = rd / "postmortem.md"
    pm_chars = 0; pm_kw_hits = 0
    if pm.exists():
        body = pm.read_text(errors="ignore")
        pm_chars = len(body)
        for kw in expected.get("expected_postmortem_keywords", []):
            if kw.lower() in body.lower():
                pm_kw_hits += 1
    s["postmortem_length"] = 1.0 if pm_chars >= 800 else (0.5 if pm_chars >= 400 else pm_chars / 800.0)
    s["postmortem_keywords"] = 1.0 if pm_kw_hits >= 4 else (0.5 if pm_kw_hits >= 3 else pm_kw_hits / 12.0)

    # 9. cross-channel evidence count
    cli_artifacts = ["firing_alerts.json", "promtool_check_before.txt", "promtool_test_after.txt",
                     "amtool_silences.txt", "silences.json", "fixed_rules.yml"]
    cli_n = sum(1 for n in cli_artifacts if (rd / n).exists())
    s["cli_artifact_count"] = cli_n / len(cli_artifacts)

    # 10. VLM rubric
    try:
        from _judge_helper import vlm_score_rubric
    except Exception:
        vlm_score_rubric = None
    vlm_imgs = [str(rd / n) for n in ("view_annotate_panel.png", "view_silence_create.png",
                                      "view_alert_state_history.png") if (rd / n).exists()]
    vlm_judge = 0.0
    if vlm_score_rubric and vlm_imgs:
        rubric = {
            "vlm_state_history_flap": "截图能看出 alert 状态在 firing 与 normal 之间反复切换（多段红绿条带）",
            "vlm_silence_active": "Silence 详情页显示 Active 状态、含 alertname matcher 和 ~2h 过期时间",
            "vlm_annotation_visible": "时序面板上能看到一段红色或彩色 region annotation 覆盖在 flap 区段",
            "vlm_panel_layout_clean": "面板布局整齐，坐标轴/legend 可读，没有重叠或截断",
        }
        vlm = vlm_score_rubric(vlm_imgs[:3], rubric, instruction="评估 SRE 值班员在 Grafana/Alertmanager 中的取证截图。")
        for k in rubric: s[k] = vlm.get(k, 0.0)
        s["judge_method"] = vlm.get("judge_method", "failed")
        vlm_judge = sum(s.get(k, 0.0) for k in rubric) / max(1, len(rubric))
    else:
        s["vlm_state_history_flap"] = 0.0
        s["vlm_silence_active"] = 0.0
        s["vlm_annotation_visible"] = 0.0
        s["vlm_panel_layout_clean"] = 0.0
        vlm_judge = 0.0

    nums = [v for k, v in s.items() if isinstance(v, (int, float)) and k != "judge_method"]
    base = sum(nums) / max(1, len(nums))

    # Hard gates
    has_cli = (s["cli_artifact_count"] >= 0.5) and (s["tcpdump_evidence_present"] >= 1.0 or s["amtool_evidence"] >= 1.0)
    if not has_cli: base = min(base, 0.4)
    # NOTE: We deliberately do NOT cap on missing GUI screenshots —
    # whether the agent invokes Grafana/Alertmanager UI tooling is not
    # a scoring axis. Missing PNGs already cost the corresponding
    # sub_scores (gui_screenshots_count / gui_screenshots_ocr /
    # 4 vlm_*); no additional hard cap is applied here.
    # VLM cap is only enforced when the helper actually ran. The
    # 0.0 fallback (HARDENING-3) intentionally drops the helper-missing
    # baseline, but we must NOT then cap on that 0.0 — the rubric was
    # never evaluated.
    if vlm_score_rubric is not None and vlm_judge < 0.6:
        base = min(base, 0.6)

    s["overall_score"] = round(base, 3)
    return s
```

## Workspace Path

```
workspace/DAV/task_16_prometheus_alert_silence_grafana
```

## Skills

```
```

## Env

```
```

## Warmup

```bash
mkdir -p /opt/dash16 /tmp_workspace/dash16 /tmp_workspace/results /tmp_workspace/state /var/lib/grafana/dashboards /opt/dav16_gt
which jq || (apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get install -y -qq python3-pip wget curl tar tcpdump jq poppler-utils)
pip3 install --break-system-packages --quiet prometheus_client 2>/dev/null || pip3 install --quiet prometheus_client
cp -rn /tmp_workspace/exec/* /tmp_workspace/dash16/ 2>/dev/null || true
for d in prometheus alertmanager grafana_provisioning grafana_dashboards metric_emitter; do [ -e /tmp_workspace/$d ] && cp -rn /tmp_workspace/$d /tmp_workspace/dash16/ 2>/dev/null; done
[ -f /tmp_workspace/webhook_sink.py ] && cp /tmp_workspace/webhook_sink.py /tmp_workspace/dash16/ 2>/dev/null
# Move expected.json to a root-only path so the agent cannot read the
# noisy alertname / patched-expr literal / postmortem keyword list as
# hints. Mirrors DAV_11/12 + DSK_16 pattern.
if [ -f /tmp_workspace/gt/expected.json ]; then
  mv /tmp_workspace/gt/expected.json /opt/dav16_gt/expected.json 2>/dev/null || true
  chown -R root:root /opt/dav16_gt 2>/dev/null || true
  chmod 700 /opt/dav16_gt 2>/dev/null || true
  chmod 600 /opt/dav16_gt/expected.json 2>/dev/null || true
  rmdir /tmp_workspace/gt 2>/dev/null || true
fi
test -x /opt/dash16/grafana/bin/grafana-server || (for url in "https://dl.grafana.com/oss/release/grafana-10.4.0.linux-amd64.tar.gz" "https://mirrors.tuna.tsinghua.edu.cn/grafana/oss/release/grafana-10.4.0.linux-amd64.tar.gz"; do curl -fL --retry 5 --retry-delay 3 --connect-timeout 30 -o /tmp_workspace/state/grafana.tgz "$url" && break; done; tar -xzf /tmp_workspace/state/grafana.tgz -C /opt/dash16 && mv /opt/dash16/grafana-* /opt/dash16/grafana)
test -x /opt/dash16/prometheus/prometheus || (for url in "https://github.com/prometheus/prometheus/releases/download/v2.51.2/prometheus-2.51.2.linux-amd64.tar.gz" "https://ghproxy.net/https://github.com/prometheus/prometheus/releases/download/v2.51.2/prometheus-2.51.2.linux-amd64.tar.gz" "https://mirror.ghproxy.com/https://github.com/prometheus/prometheus/releases/download/v2.51.2/prometheus-2.51.2.linux-amd64.tar.gz"; do curl -fL --retry 5 --retry-delay 3 --connect-timeout 30 -o /tmp_workspace/state/prom.tgz "$url" && break; done; tar -xzf /tmp_workspace/state/prom.tgz -C /opt/dash16 && mv /opt/dash16/prometheus-* /opt/dash16/prometheus) || (DEBIAN_FRONTEND=noninteractive apt-get install -y -qq prometheus 2>/dev/null && mkdir -p /opt/dash16/prometheus && ln -sf "$(command -v prometheus)" /opt/dash16/prometheus/prometheus && ln -sf "$(command -v promtool)" /opt/dash16/prometheus/promtool)
test -x /opt/dash16/alertmanager/alertmanager || (for url in "https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.linux-amd64.tar.gz" "https://ghproxy.net/https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.linux-amd64.tar.gz"; do curl -fL --retry 5 --retry-delay 3 --connect-timeout 30 -o /tmp_workspace/state/am.tgz "$url" && break; done; tar -xzf /tmp_workspace/state/am.tgz -C /opt/dash16 && mv /opt/dash16/alertmanager-* /opt/dash16/alertmanager) || (DEBIAN_FRONTEND=noninteractive apt-get install -y -qq prometheus-alertmanager 2>/dev/null && mkdir -p /opt/dash16/alertmanager && ln -sf "$(command -v alertmanager)" /opt/dash16/alertmanager/alertmanager && ln -sf "$(command -v amtool)" /opt/dash16/alertmanager/amtool)
ln -sf /opt/dash16/prometheus/promtool /usr/local/bin/promtool 2>/dev/null
ln -sf /opt/dash16/alertmanager/amtool /usr/local/bin/amtool 2>/dev/null
mkdir -p /opt/dash16/grafana/conf/provisioning/datasources /opt/dash16/grafana/conf/provisioning/dashboards
cp /tmp_workspace/dash16/grafana_provisioning/datasources/ds.yaml /opt/dash16/grafana/conf/provisioning/datasources/ds.yaml 2>/dev/null
cp /tmp_workspace/dash16/grafana_provisioning/dashboards/dashboards.yaml /opt/dash16/grafana/conf/provisioning/dashboards/dashboards.yaml 2>/dev/null
cp /tmp_workspace/dash16/grafana_dashboards/*.json /var/lib/grafana/dashboards/ 2>/dev/null
curl -sf http://localhost:9100/metrics >/dev/null 2>&1 || nohup python3 /tmp_workspace/dash16/metric_emitter/emitter.py >/tmp/emitter.log 2>&1 &
curl -sf http://127.0.0.1:9097/ >/dev/null 2>&1 || nohup python3 /tmp_workspace/dash16/webhook_sink.py >/tmp/webhook_sink.log 2>&1 &
sleep 1
curl -sf http://localhost:9090/-/ready >/dev/null 2>&1 || nohup /opt/dash16/prometheus/prometheus --config.file=/tmp_workspace/dash16/prometheus/prometheus.yml --storage.tsdb.path=/tmp_workspace/state/promdata16 --web.listen-address=:9090 --web.enable-lifecycle >/tmp/prom16.log 2>&1 &
curl -sf http://localhost:9093/-/ready >/dev/null 2>&1 || nohup /opt/dash16/alertmanager/alertmanager --config.file=/tmp_workspace/dash16/alertmanager/alertmanager.yml --storage.path=/tmp_workspace/state/amdata16 --web.listen-address=:9093 >/tmp/am16.log 2>&1 &
curl -sf http://localhost:7016/api/health >/dev/null 2>&1 || (cd /opt/dash16/grafana && GF_SERVER_HTTP_PORT=7016 GF_SECURITY_ADMIN_PASSWORD=admin GF_AUTH_ANONYMOUS_ENABLED=true GF_AUTH_ANONYMOUS_ORG_ROLE=Editor nohup ./bin/grafana-server --homepath=/opt/dash16/grafana >/tmp/grafana16.log 2>&1 &)
sleep 3
for i in $(seq 1 30); do curl -sf http://localhost:7016/api/health >/dev/null && break; sleep 2; done
true
```