---
id: DAV_task_17_jaeger_trace_root_cause_compare
name: Jaeger 分布式追踪根因定位与 trace 对比
category: DAV
timeout_seconds: 1500
---

## Prompt
You are the on-call engineer for the checkout-api team. Last night PagerDuty fired with "p95 latency jumped from 90 ms to 3 s". RED metrics (QPS / error rate / duration histogram) can only prove "things got slow", but cannot prove "which segment is slow". The dev team has already pushed OpenTelemetry traces for every request to the local Jaeger (all-in-one). You need to locate the true bottleneck span from the trace dimension and produce a chain of evidence that a third party can review.

Environment and materials (already provisioned; warmup has pre-seeded ≥ 60 traces):
- The Jaeger service is exposed at `http://localhost:16686/` (Search, Trace timeline, Compare Traces, Deep dependency graph and all other panels are hosted here).
- JSON HTTP endpoints: `http://localhost:16685/api/services`, `/api/operations?service=...`, `/api/traces?service=...&operation=...&limit=...&lookback=...`, `/api/traces/<traceID>`.
- Jaeger's own admin / metrics port is at `http://localhost:14269/`.
- The trace emitter is already pushing spans via OTLP/HTTP to the local collector port `:4318` (OTLP gRPC is on `:4317`).
- The service name is fixed at `checkout-api`, the root operation name is `POST /checkout`; the bottleneck sits in the `payments.*` subtree (leaf span names shaped like `service.subsystem.protocol`), accompanied by an HTTP 5xx status code.

Task objectives:
- Quantify "just how slow it actually is right now" from the trace dimension — the long-tail distribution formed by root span p50/p95/p99 and trace counts.
- Pin down the single leaf span (the full `service.subsystem.protocol` name) that is solely responsible for the long tail, its duration, its HTTP status code, its `otel.status_code=ERROR` tag, and its peer.service.
- Use a diff between one slow trace and one fast trace to prove the structural difference (the bottleneck span only appears in the slow trace / its duration is significantly inflated).
- Use network traffic on the OTLP collector port to prove that traces are actually being shipped in real time.
- Write the hypothesis + a full root-cause report.

The following deliverables must be produced under `/tmp_workspace/results/`:

1. **Service and operation listings**
   - `jaeger_services.json`: the raw JSON of Jaeger's currently known service list; once parsed, `checkout-api` must be findable in it.
   - `jaeger_operations.json`: the raw JSON of the operation list for `checkout-api`; once parsed, `POST /checkout` (or at least something containing the `checkout` substring) must be findable.

2. **Bulk trace sample**
   - `raw_traces.json`: a raw JSON batch of traces for `service=checkout-api & operation=POST /checkout` (Jaeger `/api/traces` return structure, with a top-level `data` array); the `data` array must have length ≥ 60.

3. **Root-span latency distribution**
   - `trace_durations.csv`: a CSV sorted by root span duration in descending order; the header must be `traceID,duration_us,span_count`; the row count must match the number of traces in `raw_traces.json` (tolerance ±10%).
   - `duration_summary.json`: root-span latency summary, schema:
     ```json
     {"p50_ms": <float>, "p95_ms": <float>, "p99_ms": <float>, "n": <int>}
     ```
     `p95_ms` must be ≥ 2000 (this is the incident-detection threshold).

4. **Full text of a single slow trace and the bottleneck leaf**
   - `slow_trace.json`: the complete Jaeger trace JSON for the slowest trace picked from the top of `trace_durations.csv` (a single trace, including the `spans` array), with span count ≥ 5.
   - `slow_span.json`: the "deepest, longest" leaf span dug out of `slow_trace.json` by reconstructing the parent-child chain along `references`, schema:
     ```json
     {
       "trace_id": "<hex>",
       "span_name": "<full service.subsystem.protocol name; must contain payments.gateway.http>",
       "duration_us": <int>,
       "tags": {"http.status_code": 504, "otel.status_code": "ERROR", "peer.service": "..."},
       "operation_path": ["<root op>", "...", "<leaf op>"]
     }
     ```
     `duration_us / 1000 ≥ 2400`; `tags` must contain both `504` and the `ERROR`/`status_code` wording.

5. **Network-traffic evidence of OTLP upload**
   - `otlp_capture.pcap`: a stretch of emitter traffic captured on the `:4318` (OTLP/HTTP collector) port (≥ 30 s or ≥ 60 packets), file ≥ 200 B; after `strings` extraction it must hit `POST /checkout` or `checkout-api`, serving as hard evidence that "traces really are being fed in".

6. **Visual evidence** (each PNG ≥ 5 KB, from an actually rendered Jaeger UI; image stitching is not allowed)
   - `view_jaeger_search.png`: the Jaeger Search page result under the conditions service=checkout-api, operation=POST /checkout — the image must simultaneously show the top duration-vs-time scatter plot (**two clusters**: dense fast traces at the lower left + isolated slow-trace cluster at the upper right) and the trace list entries below it.
   - `view_trace_waterfall.png`: the Trace timeline waterfall after opening any one slow trace — the `payments` subtree must be expanded, and the image must contain a child span bar that is **noticeably** longer than its siblings (occupying the majority of the trace's width).
   - `view_span_tags.png`: the expanded Tags / Logs drawer of the bottleneck span above — the image must clearly show the specific value of `http.status_code` (504) and error tags such as `otel.status_code = ERROR` (highlighted as a red chip).
   - `view_trace_compare.png`: the Compare Traces diff view — showing the side-by-side structures of one fast trace and one slow trace simultaneously, with diff spans highlighted (color / border / label all acceptable) and shared spans de-emphasized.

7. **Conclusion documents**
   - `hypothesis.md`: a hypothesis writeup of ≥ 400 characters; must explicitly state the true bottleneck span's full name (in `service.subsystem.protocol` form), its HTTP status code, and its `peer.service` tag.
   - `root_cause_report.md`: an incident report of ≥ 800 characters; suggested structure 1) symptoms → 2) what RED metrics showed / failed to show → 3) the true cause seen in the trace waterfall → 4) differences revealed by Compare Traces → 5) immediate mitigation (fail-fast / timeout / circuit breaker) → 6) long-term governance → 7) verification plan (write a query using Jaeger's native search to monitor future regressions).

Hard constraints:
- All numbers must be genuinely exported from Jaeger / actual packet captures; hard-coding is not allowed. `slow_span.json` must be a real leaf computed from the parent-child chain in `slow_trace.json`.
- `slow_span.json.span_name` must hit `payments.gateway.http` (the bottleneck leaf name), `duration_us / 1000 ≥ 2400`, and `tags` must contain both `504` and the `ERROR`/`status_code` wording.
- `silence` does not apply to this task; the `pcap` must yield `POST /checkout` or `checkout-api` from `strings` (proof that the emitter is online).
- Screenshots must come from actual rendering (PIL stitching / placeholder images are not accepted); they must genuinely show the various Jaeger UI panels.

## Expected Behavior

成功的解答会留下"服务清单 + 根 span 时延分布 + 单条慢 trace 全文 + 瓶颈 leaf 拆解 + 网络抓包 + 4 张可视证据 + 两份结论文档"七路并齐的证据链：

- `jaeger_services.json` 与 `jaeger_operations.json` 共同证明 `checkout-api` / `POST /checkout` 存在于本机 Jaeger 的索引中。
- `raw_traces.json` ≥ 60 条 trace + `trace_durations.csv` 行数对齐 + `duration_summary.json` 中 p95_ms ≥ 2000，描出根 span 时延的长尾分布（远高于 90 ms 正常水位）。
- `slow_trace.json` 含 ≥ 5 个 span 的完整原文，配合 `slow_span.json` 把瓶颈 leaf 锁定到 `payments.gateway.http`、duration ≥ 2400 ms、`http.status_code=504`、`otel.status_code=ERROR`，并附 `peer.service` 与从 root 到 leaf 的 `operation_path`。
- `otlp_capture.pcap` 在 OTLP collector 端口上抓到 emitter 实时上送的 trace 流量，`strings` 能挖出 `POST /checkout`，证明 trace 不是历史残留而是真实活体数据。
- 4 张界面截图按 Search 页 scatter 两簇 → Trace timeline 中支配级长 bar → Tags/Logs 抽屉中红色 504 chip → Compare Traces 差异高亮的顺序铺开可视复验链，让审阅者无需重跑 jaeger 也能看清"哪条 trace、哪条 span、长在哪、错在哪"。
- `hypothesis.md` 用 ≥ 400 字浓缩 trace 视角下的最小可判定假设（瓶颈 span 全名 + 状态码 + peer.service）。
- `root_cause_report.md` 用 ≥ 800 字、命中 ≥ 4/6 关键词的方式串起现象 / RED 盲点 / waterfall 真因 / Compare 差异 / 缓解 / 长期治理 / 回归监控，作为一份可交付的事故复盘。

## Source
- Jaeger 官方文档 "Sampling & Tracing Best Practices": https://www.jaegertracing.io/docs/1.57/
- OpenTelemetry Python OTLP/HTTP quickstart: https://opentelemetry.io/docs/languages/python/exporters/
- 真实场景灵感：Honeycomb 2024 blog "p95 spike was a single grandchild HTTP call"
- 对应 EyesOn-Bench DAV 类: D17 — distributed-trace root-cause via Jaeger compare

## Grading Criteria

- [ ] 1. `jaeger_services.json` 与 `jaeger_operations.json` 都存在且包含 checkout-api / POST /checkout
- [ ] 2. `raw_traces.json` ≥ 60 条 trace
- [ ] 3. `trace_durations.csv` schema 正确，行数 = trace 数
- [ ] 4. `duration_summary.json` p95_ms 高于正常水位（grader 内部对照阈值）
- [ ] 5. `slow_trace.json` 是单条 trace，含 ≥ 5 个 span
- [ ] 6. `slow_span.json` 的 `span_name` 指向真正的瓶颈 leaf span，duration 与 tags 与 GT 匹配
- [ ] 7. `hypothesis.md` 与 `root_cause_report.md` 都存在；后者 ≥ 800 字符；命中关键词 ≥ 4/6
- [ ] 8. 4 张 GUI 截图都存在，OCR 命中各自关键字
- [ ] 9. `otlp_capture.pcap` 存在且 strings 命中 "POST /checkout"
- [ ] 10. VLM 判断 `view_trace_waterfall.png` 中能看见一条占大比例长度的 child span

## Automated Checks

```python
def grade(workspace_path=None, **kwargs) -> dict:
    """DAV_task_17 grader: jaeger trace root-cause + compare evidence."""
    import csv, json, re, subprocess
    from pathlib import Path
    rd = Path("/tmp_workspace/results")
    # Read GT from the root-only path the warmup moved expected.json to
    # (kept off the agent-visible /tmp_workspace/gt to prevent answer
    # leakage). Fall back to /tmp_workspace/gt for legacy mounts.
    gt_dir = Path("/opt/dav17_gt") if Path("/opt/dav17_gt/expected.json").exists() else Path("/tmp_workspace/gt")
    s = {}
    expected = {}
    if (gt_dir / "expected.json").exists():
        try: expected = json.loads((gt_dir / "expected.json").read_text())
        except Exception: expected = {}
    svc = expected.get("service_name", "checkout-api")
    op = expected.get("slow_root_operation", "POST /checkout")
    slow_name = expected.get("slow_span_name", "payments.gateway.http")
    min_dur_ms = expected.get("slow_span_min_duration_ms", 2400)
    err_code = expected.get("slow_span_error_status_code", 504)

    # 1. services + operations
    s_svc = 0.0; s_op = 0.0
    js = rd / "jaeger_services.json"
    if js.exists():
        try:
            d = json.loads(js.read_text())
            arr = d.get("data") if isinstance(d, dict) else d
            if isinstance(arr, list) and any(svc in str(x) for x in arr):
                s_svc = 1.0
        except Exception: pass
    s["services_present"] = s_svc
    jo = rd / "jaeger_operations.json"
    if jo.exists():
        try:
            d = json.loads(jo.read_text())
            arr = d.get("data") if isinstance(d, dict) else d
            txt = json.dumps(arr)
            if op in txt or "checkout" in txt:
                s_op = 1.0
        except Exception: pass
    s["operations_present"] = s_op

    # 2. raw_traces.json — must be ≥ 60
    rt = rd / "raw_traces.json"
    n_traces = 0; n_spans = 0
    if rt.exists():
        try:
            d = json.loads(rt.read_text())
            arr = d.get("data") if isinstance(d, dict) else d
            if isinstance(arr, list):
                n_traces = len(arr)
                n_spans = sum(len(t.get("spans", [])) for t in arr if isinstance(t, dict))
        except Exception: pass
    min_n = expected.get("min_total_traces", 60)
    s["raw_traces_present"] = 1.0 if n_traces >= min_n else (n_traces / min_n if n_traces else 0.0)

    # 3. trace_durations.csv schema + rows
    td = rd / "trace_durations.csv"
    td_ok = 0.0
    if td.exists():
        try:
            rows = list(csv.DictReader(td.open()))
            if rows and {"traceID", "duration_us", "span_count"}.issubset(set(rows[0].keys())):
                if abs(len(rows) - max(n_traces, 1)) <= max(2, int(0.1 * max(n_traces, 1))):
                    td_ok = 1.0
                else:
                    td_ok = 0.5
        except Exception: pass
    s["trace_durations_csv"] = td_ok

    # 4. duration_summary.json — p95 ≥ 2000 ms
    ds = rd / "duration_summary.json"
    p95_ok = 0.0
    if ds.exists():
        try:
            d = json.loads(ds.read_text())
            p95 = float(d.get("p95_ms", 0))
            if p95 >= expected.get("expected_p95_root_ms_min", 2000):
                p95_ok = 1.0
            elif p95 >= 500:
                p95_ok = 0.5
        except Exception: pass
    s["duration_p95_ok"] = p95_ok

    # 5. slow_trace.json
    st = rd / "slow_trace.json"
    st_ok = 0.0
    if st.exists():
        try:
            d = json.loads(st.read_text())
            spans = (d.get("data", [{}])[0] if isinstance(d.get("data"), list) else d).get("spans", [])
            if isinstance(d, dict) and (d.get("spans") or spans):
                spans = spans or d.get("spans", [])
                if len(spans) >= 5:
                    st_ok = 1.0
                elif len(spans) >= 2:
                    st_ok = 0.5
        except Exception: pass
    s["slow_trace_present"] = st_ok

    # 6. slow_span.json — must identify the true bottleneck leaf span
    ss = rd / "slow_span.json"
    span_name_ok = 0.0; span_dur_ok = 0.0; span_tag_ok = 0.0
    if ss.exists():
        try:
            d = json.loads(ss.read_text())
            name = str(d.get("span_name", ""))
            dur_us = float(d.get("duration_us", 0))
            st_spans = []
            if st.exists():
                try:
                    _d = json.loads(st.read_text()); _t = (_d.get("data",[_d])[0] if isinstance(_d.get("data"),list) else _d)
                    st_spans = _t.get("spans", [])
                except Exception: pass
            matched = next((sp for sp in st_spans if str(d.get("span_name","")) in (sp.get("operationName","") or "")
                            and abs(int(sp.get("duration",0)) - int(d.get("duration_us",0))) <= 50), None)
            span_name_ok = 1.0 if (slow_name == name and matched) else (0.0 if not matched else 0.3)
            span_dur_ok = 1.0 if (matched and dur_us/1000.0 >= min_dur_ms) else 0.0
            tags = d.get("tags", {})
            tag_text = json.dumps(tags) if not isinstance(tags, str) else tags
            has_code = re.search(r'"http\.status_code"\s*:\s*"?%d' % err_code, tag_text) is not None
            has_err  = ("ERROR" in tag_text) and ("peer.service" in tag_text)
            span_tag_ok = 1.0 if (has_code and has_err) else (0.4 if has_code else 0.0)
        except Exception: pass
    s["slow_span_name"] = span_name_ok
    s["slow_span_duration"] = span_dur_ok
    s["slow_span_tags"] = span_tag_ok

    # 7. hypothesis + report
    hy = rd / "hypothesis.md"
    hy_chars = len(hy.read_text(errors="ignore")) if hy.exists() else 0
    hy_root = 0.0
    if hy.exists() and expected.get("expected_hypothesis_root_cause_substring", "payments.gateway.http") in hy.read_text(errors="ignore"):
        hy_root = 1.0
    s["hypothesis_length"] = 1.0 if hy_chars >= 400 else hy_chars / 400.0
    s["hypothesis_identifies_root"] = hy_root

    rp = rd / "root_cause_report.md"
    rp_chars = 0; rp_kw = 0
    if rp.exists():
        body = rp.read_text(errors="ignore")
        rp_chars = len(body)
        # Keyword set inlined here as a grader-only default so gt/expected.json
        # can omit the answer list. Mirrors the postmortem checklist hinted at
        # by the Prompt's "缓解 / 治理 / 回归监控" sections.
        DEFAULT_KW = ["payments.gateway.http","504 Gateway Timeout",
                      "peer.service=payments-gateway","circuit breaker",
                      "fail-fast","SLO regression","p95"]
        for kw in expected.get("expected_postmortem_keywords", DEFAULT_KW):
            if kw.lower() in body.lower():
                rp_kw += 1
    s["report_length"] = 1.0 if rp_chars >= 800 else rp_chars / 800.0
    s["report_keywords"] = 1.0 if rp_kw >= 5 else (rp_kw / 5.0 if rp_kw else 0.0)

    # 8. GUI screenshots (4) + OCR
    gui_shots = ["view_jaeger_search.png", "view_trace_waterfall.png",
                 "view_span_tags.png", "view_trace_compare.png"]
    present = sum(1 for n in gui_shots if (rd / n).exists() and (rd / n).stat().st_size > 5000)
    s["gui_screenshots_count"] = present / len(gui_shots)
    ocr_hits = 0
    try:
        import pytesseract
        from PIL import Image
        kws = {
            "view_jaeger_search.png": ["Jaeger", "Find", "Traces", "Service", "Operation"],
            "view_trace_waterfall.png": ["Trace", "Timeline", "Service", "Duration", "Spans", "payments"],
            "view_span_tags.png": ["Tags", "Logs", "status_code", "http", "error", "504"],
            "view_trace_compare.png": ["Compare", "Traces", "diff", "A", "B", "Service"],
        }
        for n, ks in kws.items():
            p = rd / n
            if p.exists():
                try:
                    tx = pytesseract.image_to_string(Image.open(p))
                    if any(k.lower() in tx.lower() for k in ks): ocr_hits += 1
                except Exception: pass
        s["gui_screenshots_ocr"] = ocr_hits / len(gui_shots)
        s["gui_screenshots_count"] = (present / len(gui_shots)) if ocr_hits >= 3 else 0.0
    except ImportError:
        s["gui_screenshots_ocr"] = 0.5

    # 9. otlp pcap
    pcap = rd / "otlp_capture.pcap"
    pcap_ok = 0.0
    if pcap.exists() and pcap.stat().st_size > 200:
        try:
            out = subprocess.run(["strings", str(pcap)], capture_output=True, text=True, timeout=15)
            if "POST /checkout" in out.stdout or "checkout-api" in out.stdout:
                pcap_ok = 1.0
            elif out.stdout.strip():
                pcap_ok = 0.5
        except Exception:
            pcap_ok = 0.4
    s["otlp_pcap_evidence"] = pcap_ok

    # CLI artifact tally
    cli_artifacts = ["jaeger_services.json", "jaeger_operations.json", "raw_traces.json",
                     "trace_durations.csv", "duration_summary.json",
                     "slow_trace.json", "slow_span.json"]
    cli_n = sum(1 for n in cli_artifacts if (rd / n).exists())
    s["cli_artifact_count"] = cli_n / len(cli_artifacts)

    # VLM rubric
    try:
        from _judge_helper import vlm_score_rubric
    except Exception:
        vlm_score_rubric = None
    vlm_imgs = [str(rd / n) for n in ("view_trace_waterfall.png", "view_trace_compare.png",
                                      "view_span_tags.png") if (rd / n).exists()]
    vlm_judge = 0.0
    if vlm_score_rubric and vlm_imgs:
        rubric = {
            "vlm_waterfall_hotspot": "Trace timeline 中能看到一条明显比同级 sibling 长得多的子 span（占据 trace 大部分宽度）",
            "vlm_compare_diff": "Compare Traces 视图清楚展示两条 trace 之间结构或耗时差异（高亮 / diff 颜色）",
            "vlm_error_chip_visible": "Tags / Logs 抽屉中能看到一个明显的 HTTP error 状态码 chip 与 ERROR 状态高亮",
            "vlm_layout_clean": "整体 Jaeger UI 布局清晰，没有遮挡或截断，关键面板可读",
        }
        vlm = vlm_score_rubric(vlm_imgs[:3], rubric, instruction="评估分布式追踪根因定位 / Compare Traces 截图。")
        for k in rubric: s[k] = vlm.get(k, 0.0)
        s["judge_method"] = vlm.get("judge_method", "failed")
        vlm_judge = sum(s.get(k, 0.0) for k in rubric) / max(1, len(rubric))
    else:
        # Helper offline → leave rubric scores at 0.0 (no freebie) and
        # rely on the 'vlm_score_rubric is not None' helper-guard below
        # to skip the VLM cap entirely.
        s["vlm_waterfall_hotspot"] = 0.0
        s["vlm_compare_diff"] = 0.0
        s["vlm_error_chip_visible"] = 0.0
        s["vlm_layout_clean"] = 0.0
        vlm_judge = 0.0

    IGNORE = {"judge_method","cli_artifact_count","vlm_layout_clean","hypothesis_length","report_length"}
    nums = [v for k, v in s.items() if isinstance(v,(int,float)) and k not in IGNORE]
    base = sum(nums) / max(1, len(nums))

    # Hard gates
    has_cli = (s["cli_artifact_count"] >= 0.85) and (s["otlp_pcap_evidence"] >= 1.0) and (s["raw_traces_present"] >= 1.0)
    must_pass = (s["slow_span_name"] >= 1.0 and s["slow_span_tags"] >= 1.0)
    if not has_cli: base = min(base, 0.35)
    # NOTE: We deliberately do NOT cap on missing GUI screenshots —
    # whether the agent invokes Jaeger UI tooling is not a scoring
    # axis. Missing PNGs already cost the corresponding sub_scores
    # (gui_screenshots_count / gui_screenshots_ocr / 4 vlm_*); no
    # additional hard cap is applied here.
    # VLM cap aligned with batch3 (vlm_avg<0.6 cap @0.45) and only
    # enforced when the helper actually ran.
    if vlm_score_rubric is not None and vlm_judge < 0.6:
        base = min(base, 0.45)
    if not must_pass: base = min(base, 0.55)

    s["overall_score"] = round(base, 3)
    return s
```

## Workspace Path

```
workspace/DAV/task_17_jaeger_trace_root_cause_compare
```

## Skills

```
```

## Env

```
```

## Warmup

```bash
mkdir -p /opt/dash17 /tmp_workspace/dash17 /tmp_workspace/results /tmp_workspace/state /tmp_workspace/state/jaeger_badger /opt/dav17_gt
which jq || (apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get install -y -qq python3-pip wget curl tar tcpdump jq binutils)
pip3 install --break-system-packages --quiet opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http 2>/dev/null || pip3 install --quiet opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
cp -rn /tmp_workspace/exec/* /tmp_workspace/dash17/ 2>/dev/null || true
[ -d /tmp_workspace/trace_emitter ] && cp -rn /tmp_workspace/trace_emitter /tmp_workspace/dash17/ 2>/dev/null
[ -d /tmp_workspace/jaeger_data ] && cp -rn /tmp_workspace/jaeger_data /tmp_workspace/dash17/ 2>/dev/null
# Move expected.json to a root-only path so the agent cannot read the
# slow_span_name / status_code / peer_service / postmortem keyword
# answer set as hints. Mirrors DAV_11/12/16 + DSK_16 pattern.
if [ -f /tmp_workspace/gt/expected.json ]; then
  mv /tmp_workspace/gt/expected.json /opt/dav17_gt/expected.json 2>/dev/null || true
  chown -R root:root /opt/dav17_gt 2>/dev/null || true
  chmod 700 /opt/dav17_gt 2>/dev/null || true
  chmod 600 /opt/dav17_gt/expected.json 2>/dev/null || true
  rmdir /tmp_workspace/gt 2>/dev/null || true
fi
test -x /opt/dash17/jaeger/jaeger-all-in-one || (for url in "https://github.com/jaegertracing/jaeger/releases/download/v1.57.0/jaeger-1.57.0-linux-amd64.tar.gz" "https://ghproxy.net/https://github.com/jaegertracing/jaeger/releases/download/v1.57.0/jaeger-1.57.0-linux-amd64.tar.gz" "https://mirror.ghproxy.com/https://github.com/jaegertracing/jaeger/releases/download/v1.57.0/jaeger-1.57.0-linux-amd64.tar.gz"; do curl -fL --retry 5 --retry-delay 3 --connect-timeout 30 -o /tmp_workspace/state/jaeger.tgz "$url" && break; done; tar -xzf /tmp_workspace/state/jaeger.tgz -C /opt/dash17 && mv /opt/dash17/jaeger-* /opt/dash17/jaeger)
curl -sf http://localhost:16686/ >/dev/null 2>&1 || SPAN_STORAGE_TYPE=badger BADGER_EPHEMERAL=false BADGER_DIRECTORY_VALUE=/tmp_workspace/state/jaeger_badger/data BADGER_DIRECTORY_KEY=/tmp_workspace/state/jaeger_badger/key COLLECTOR_OTLP_ENABLED=true nohup /opt/dash17/jaeger/jaeger-all-in-one --query.http-server.host-port=:16686 --query.grpc-server.host-port=:16685 --collector.otlp.http.host-port=:4318 --collector.otlp.grpc.host-port=:4317 --admin.http.host-port=:14269 >/tmp/jaeger.log 2>&1 &
sleep 3
for i in $(seq 1 30); do curl -sf http://localhost:14269/ >/dev/null && break; sleep 2; done
existing=$(curl -sf "http://localhost:16686/api/traces?service=checkout-api&limit=200&lookback=1h" 2>/dev/null | jq -r '.data | length' 2>/dev/null || echo 0)
if [ "${existing:-0}" -lt 30 ]; then EMIT_N=120 OTLP_ENDPOINT=http://localhost:4318/v1/traces nohup python3 /tmp_workspace/dash17/trace_emitter/emitter.py >/tmp/emitter17.log 2>&1 &
sleep 8; fi
curl -s http://localhost:16686/api/services >/dev/null || echo WARN_jaeger_not_ready
true
```