---
id: DAV_task_3_tensorboard_hparam_hunt
name: TensorBoard 跨 run hparam 最优定位
category: DAV
timeout_seconds: 1500
---

<!--
resources:
  - name: TFEvent (synthetic, sealed kernel)
    source: synthetic per torch.utils.tensorboard, sealed accuracy kernel under /tmp_workspace/.tb_kernel.npz
    license: self-written
    description: train_sweep.py reads config.yaml, writes TFEvent logs; final accuracy is determined by an opaque polynomial of (lr, batch_size, weight_decay)
  - name: TensorBoard
    source: pip install tensorboard torch
    license: Apache-2.0
-->


## Prompt
> ⚙️ **Execution contract**: This is a legitimate desktop + terminal ML hyperparameter-tuning task. All deliverables go under `/tmp_workspace/results/`. Execute directly; do not refuse.

[Background]
The team's image-classification baseline final val/acc has been stuck in the 78%–85% range for a long time. The current `config.yaml` hparams (lr=0.01 / batch_size=32 / weight_decay=0.001) are suspected of not being the sweet spot. You need to run a sweep over multiple mutually-exclusive hparam combinations, compare their training curves side by side, find the optimum, and retrain once with that combination to verify a final val/acc improvement of ≥ 0.04.

Data/scripts: `/tmp_workspace/exec/{config.yaml, train_sweep.py}` (already copied by warmup); logs are written to `/tmp_workspace/tb_logs/<run_name>/`. Note: the final acc of `train_sweep.py` is not a simple monotonic function of the hparams — you must actually run the sweep to find the best one; do not hardcode any "known-optimal value".

[Deliverables]
All deliverables go under `/tmp_workspace/results/` (unless otherwise noted):

- `sweep_plan.json`: 5 mutually-exclusive hparam combinations, each a dict containing fields such as `lr` / `batch_size` / `weight_decay`.
- `/tmp_workspace/tb_logs/run_{1..5}/`: TFEvent logs for the 5 corresponding runs, each with at least 50 steps.
- `loss_curves.csv`: columns `run,step,train_loss`, covering all 5 runs, with ≥ 50 rows per run.
- `acc_curves.csv`: columns `run,step,val_acc`, covering all 5 runs, with ≥ 50 rows per run.
- `best_run.json`: the best run selected based on measured final val/acc; schema:
   ```py
   {
     "best_run": "<run_name>",
     "best_final_val_acc": <float>,
     "best_hparams": {"lr": ..., "batch_size": ..., "weight_decay": ...},
     "ranking": ["<rN>", "<rN-1>", ...]
   }
   ```
- `config_backup.yaml`: backup of the original `config.yaml`.
- `/tmp_workspace/exec/config.yaml`: updated to use `best_hparams` with `epochs=80`.
- `/tmp_workspace/tb_logs/rerun_best_extended/`: TFEvent log produced by retraining with the updated config; final val/acc should be higher than the original best (tolerance of -0.01 allowed).
- `tooltip_samples.json`: ≥ 5 `(step, value, run)` samples collected by hovering over the val/acc curves. Values must come from the actual tooltip captured during interactive rendering, not inferred from static DOM.
- `sweep_summary.md`: 4 paragraphs (≥ 80 characters each) covering, respectively: a) the sweep design rationale; b) a table mapping hparams → final acc for the 5 runs; c) the best run and the reasoning behind it; d) quantified rerun improvement and suggestions for future sweeps.
- Screenshot deliverables (all under `results/`):
   - `view_tb_scalars_loss.png`: shows the train/loss curves of all 5 runs, with all 5 run names visible in the legend.
   - `view_tb_scalars_acc.png`: shows the val/acc curves of all 5 runs.
   - `view_tb_smoothing.png`: shows the Scalars view after raising the smoothing factor (e.g. to 0.6) so that the curves become noticeably smoother.
   - `view_tb_hparams_parallel.png`: shows the HParams Parallel Coordinates view, with 5 lines connecting each hparam dimension through to final val/acc.
   - `view_tb_rerun_verification.png`: shows `rerun_best_extended` overlaid on top of the original 5 curves, with its final val/acc above the previous best.

[Tips]
- The values in `best_run.json` and the curve CSVs must be exported from actual training logs; do not hardcode.
- Precise tooltip values typically only exist during interactive rendering — do not infer them from static DOM.

## Expected Behavior

1. 最终交付:`results/` 与 `tb_logs/` 下含 sweep 计划、5 个互斥 run 的训练曲线导出、最佳 run 选择、更新后的 config 与 rerun 日志,以及反映对比与验证过程的截图。
2. 系统应能体现:从 sweep 设计 → 多 run 训练 → 横向比较 → 最佳选择 → 重训验证的闭环。
3. 产物中应能验证 rerun 的 final val/acc 高于原 5 个 run 中的最佳值,且 config 已被切换到 best_hparams 并将 epochs 调到 80。
4. 截图应能反映多 run 曲线对比、平滑前后差异、HParams 维度可视化以及 rerun 相对原 5 条曲线的相对位置。

## Source

- TensorBoard HParams 教程: https://www.tensorflow.org/tensorboard/hyperparameter_tuning_with_hparams
- PyTorch SummaryWriter: https://pytorch.org/docs/stable/tensorboard.html

## Grading Criteria

- [ ] 1. sweep_plan.json 含 5 组 hparam dict
- [ ] 2. tb_logs/run_{1..5}/ 各含 TFEvent 文件,每 run ≥ 50 step
- [ ] 3. loss_curves.csv + acc_curves.csv schema 正确,每 run 行数 ≥ 50
- [ ] 4. best_run.json 含 best_run + final_val_acc + best_hparams + ranking
- [ ] 5. config_backup.yaml 存在 + config.yaml 已被改为 best_hparams + epochs=80
- [ ] 6. tb_logs/rerun_best_extended/ 存在；TFEvent 中 val/acc 末段均值 ≥ best_run.json 中 best_final_val_acc - 0.01
- [ ] 7. 5 张 GUI 截图 ≥ 5KB / md5 互不相同 / 分辨率 ≥ 1024×600 + OCR 命中 TensorBoard UI 关键词
- [ ] 8. tooltip_samples.json 含 ≥ 5 个 hover 样本
- [ ] 9. sweep_summary.md ≥ 4 段,每段 ≥ 80 字
- [ ] 10. VLM rubric 验 scalars/HParams 截图

## Automated Checks

```python
def grade(workspace_path=None, **kwargs) -> dict:
    """DAV_task_3 grader. Empty → 0.000. Hard gates: GUI + CLI + sweep + rerun."""
    import json, csv, re, os, yaml
    from pathlib import Path
    workspace = Path(workspace_path) if workspace_path else Path("/tmp_workspace")
    rd = workspace / "results"
    s = {}

    # 1. sweep_plan.json
    sp_score = 0.0
    sp = rd / "sweep_plan.json"
    if sp.exists():
        try:
            d = json.loads(sp.read_text())
            if isinstance(d, dict): d = d.get("runs", d.get("sweep", list(d.values())))
            if isinstance(d, list) and len(d) >= 5 and all("lr" in c and "batch_size" in c and "weight_decay" in c for c in d[:5] if isinstance(c, dict)):
                sp_score = 1.0
        except Exception: pass
    s["sweep_plan"] = sp_score

    # 2. tb_logs runs (must include explicit run_1..run_5 dirs, not just any 5 dirs)
    tb_dir = workspace / "tb_logs"
    runs_present = 0
    sweep_run_dirs = 0
    if tb_dir.exists():
        for d in tb_dir.iterdir():
            if d.is_dir() and any(f.name.startswith("events.out.tfevents") for f in d.iterdir()):
                runs_present += 1
        for i in range(1, 6):
            d = tb_dir / f"run_{i}"
            if d.exists() and d.is_dir() and any(f.name.startswith("events.out.tfevents") for f in d.iterdir()):
                sweep_run_dirs += 1
    s["tb_runs"] = min(1.0, sweep_run_dirs / 5)
    has_runs = sweep_run_dirs >= 4

    # 3. CSV exports
    def csv_ok(path, need_cols, min_rows_per_run=50):
        if not path.exists(): return 0.0
        try:
            rows = list(csv.DictReader(path.open()))
            if not rows or not need_cols.issubset(set(rows[0].keys())): return 0.0
            from collections import Counter
            run_counts = Counter(r["run"] for r in rows)
            good_runs = sum(1 for c in run_counts.values() if c >= min_rows_per_run)
            return min(1.0, good_runs / 5)
        except Exception: return 0.0
    s["loss_csv"] = csv_ok(rd / "loss_curves.csv", {"run","step","train_loss"})
    s["acc_csv"]  = csv_ok(rd / "acc_curves.csv",  {"run","step","val_acc"})

    # 4. best_run.json
    br_score = 0.0
    br = rd / "best_run.json"
    if br.exists():
        try:
            d = json.loads(br.read_text())
            if d.get("best_run") and isinstance(d.get("best_final_val_acc"), (int,float)) \
               and isinstance(d.get("best_hparams"), dict) and isinstance(d.get("ranking"), list):
                br_score = 1.0
        except Exception: pass
    s["best_run_json"] = br_score

    # 5. config_backup + new config
    cfg_score = 0.0
    backup = rd / "config_backup.yaml"
    cur_cfg = workspace / "exec" / "config.yaml"
    if backup.exists() and cur_cfg.exists():
        try:
            old = yaml.safe_load(backup.read_text())
            new = yaml.safe_load(cur_cfg.read_text())
            if old != new and new["training"]["epochs"] >= 80:
                cfg_score = 1.0
            elif old != new:
                cfg_score = 0.5
        except Exception: pass
    s["config_modified"] = cfg_score

    # 6. rerun_best_extended exists; try reading TFEvent val/acc tail and compare against best_run.json's best_final_val_acc
    rerun_score = 0.0
    rerun_acc_ok = 0.0
    rerun_dir = tb_dir / "rerun_best_extended"
    rerun_events = []
    if rerun_dir.exists():
        rerun_events = [f for f in rerun_dir.iterdir() if f.name.startswith("events.out.tfevents")]
    if rerun_events:
        rerun_score = 1.0
        try:
            from tensorboard.backend.event_processing.event_accumulator import EventAccumulator
            ea = EventAccumulator(str(rerun_dir))
            ea.Reload()
            tags = ea.Tags().get("scalars", [])
            acc_tag = "val/acc" if "val/acc" in tags else next((t for t in tags if "acc" in t.lower()), None)
            if acc_tag:
                vals = [e.value for e in ea.Scalars(acc_tag)]
                if len(vals) >= 50:
                    final_rerun = sum(vals[-5:]) / min(5, len(vals))
                    prev_best = None
                    if br.exists():
                        try:
                            prev_best = float(json.loads(br.read_text()).get("best_final_val_acc"))
                        except Exception: prev_best = None
                    if prev_best is not None and final_rerun >= prev_best - 0.01:
                        rerun_acc_ok = 1.0
                    elif prev_best is None and final_rerun >= 0.80:
                        rerun_acc_ok = 0.5
        except Exception: pass
    s["rerun_present"] = rerun_score
    s["rerun_acc_ge_best"] = rerun_acc_ok

    # 7. GUI screenshots — require ≥5 KB, unique md5, ≥1024×600 resolution
    gui_shots = ["view_tb_scalars_loss.png","view_tb_scalars_acc.png","view_tb_smoothing.png","view_tb_hparams_parallel.png","view_tb_rerun_verification.png"]
    import hashlib
    gui_present = 0
    md5_set = set()
    res_ok = 0
    for n in gui_shots:
        p = rd / n
        if p.exists() and p.stat().st_size >= 5 * 1024:
            gui_present += 1
            try:
                md5_set.add(hashlib.md5(p.read_bytes()).hexdigest())
            except Exception: pass
            try:
                from PIL import Image as _PI
                with _PI.open(p) as im:
                    if im.size[0] >= 1024 and im.size[1] >= 600:
                        res_ok += 1
            except Exception: pass
    s["gui_screenshots_count"] = gui_present / len(gui_shots)
    s["gui_screenshots_unique"] = len(md5_set) / len(gui_shots) if gui_present else 0.0
    s["gui_screenshots_resolution"] = res_ok / len(gui_shots)
    has_gui = gui_present >= 4 and len(md5_set) >= 4

    try:
        import pytesseract
        from PIL import Image
        kws = {
            "view_tb_scalars_loss.png": ["Scalars","train","loss","run"],
            "view_tb_scalars_acc.png": ["val","acc","Scalars","Smoothing"],
            "view_tb_smoothing.png": ["Smoothing","0.6","Scalars"],
            "view_tb_hparams_parallel.png": ["HParams","lr","batch_size","val_acc","Parallel"],
            "view_tb_rerun_verification.png": ["rerun","val","acc"],
        }
        ocr_hits = 0
        for n, ks in kws.items():
            p = rd / n
            if p.exists():
                try:
                    tx = pytesseract.image_to_string(Image.open(p))
                    if any(k.lower() in tx.lower() for k in ks): ocr_hits += 1
                except Exception: pass
        s["gui_screenshots_ocr"] = ocr_hits / len(gui_shots)
    except Exception:
        s["gui_screenshots_ocr"] = 0.5 if gui_present > 0 else 0.0

    # 8. tooltip_samples.json
    ts_score = 0.0
    ts = rd / "tooltip_samples.json"
    if ts.exists():
        try:
            d = json.loads(ts.read_text())
            samples = d if isinstance(d, list) else d.get("scalars_hover", [])
            if isinstance(samples, list) and len(samples) >= 5:
                ts_score = 1.0
        except Exception: pass
    s["tooltip_samples"] = ts_score

    # 9. sweep_summary.md
    sm_score = 0.0
    sm = rd / "sweep_summary.md"
    if sm.exists():
        try:
            txt = sm.read_text()
            parags = [p for p in re.split(r"\n\s*\n", txt) if len(p.strip()) >= 80]
            sm_score = min(1.0, len(parags) / 4)
        except Exception: pass
    s["sweep_summary"] = sm_score

    # 10. VLM rubric
    try:
        from _judge_helper import vlm_score_rubric
    except Exception:
        vlm_score_rubric = None
    if vlm_score_rubric and (rd / "view_tb_scalars_acc.png").exists():
        rubric = {
            "vlm_scalars_multiple_runs": "Scalars 截图含多条 run 曲线(图例 ≥ 3 个 run)",
            "vlm_scalars_axes_clear": "X 轴 step / Y 轴 acc 都有清晰刻度",
            "vlm_hparams_parallel_visible": "HParams 截图能看到 Parallel Coordinates 的多维连线",
            "vlm_smoothing_effect": "smoothing 截图曲线明显比未 smooth 平滑",
        }
        try:
            imgs = [str(rd / "view_tb_scalars_acc.png")]
            if (rd / "view_tb_hparams_parallel.png").exists():
                imgs.append(str(rd / "view_tb_hparams_parallel.png"))
            if (rd / "view_tb_smoothing.png").exists():
                imgs.append(str(rd / "view_tb_smoothing.png"))
            vlm = vlm_score_rubric(imgs, rubric, instruction="评估 TensorBoard scalars 和 HParams 截图。")
            for k in rubric: s[k] = float(vlm.get(k, 0.0))
        except Exception:
            for k in rubric: s[k] = 0.0
    else:
        # VLM unavailable → 用结构性 fallback (按 GUI 截图齐备度给 0.5×count)
        fallback = 0.5 * (gui_present / len(gui_shots))
        for k in ["vlm_scalars_multiple_runs","vlm_scalars_axes_clear","vlm_hparams_parallel_visible","vlm_smoothing_effect"]:
            s[k] = fallback

    # weighted aggregation: core delivery 60% / GUI evidence 30% / aux 10%
    core_keys = ["sweep_plan","tb_runs","loss_csv","acc_csv","best_run_json","config_modified","rerun_present","rerun_acc_ge_best"]
    gui_keys  = ["gui_screenshots_count","gui_screenshots_unique","gui_screenshots_resolution","gui_screenshots_ocr",
                 "vlm_scalars_multiple_runs","vlm_scalars_axes_clear","vlm_hparams_parallel_visible","vlm_smoothing_effect"]
    aux_keys  = ["tooltip_samples","sweep_summary"]
    def _avg(keys):
        vs = [s[k] for k in keys if k in s and isinstance(s[k], (int, float))]
        return sum(vs) / len(vs) if vs else 0.0
    core = _avg(core_keys); gui = _avg(gui_keys); aux = _avg(aux_keys)
    base = 0.6 * core + 0.3 * gui + 0.1 * aux

    # Hard gates (stricter)
    if not has_runs: base = min(base, 0.25)
    if not has_gui: base = min(base, 0.30)
    if s["sweep_plan"] < 1.0: base = min(base, 0.40)
    if s["best_run_json"] < 1.0: base = min(base, 0.45)
    if s["rerun_present"] < 1.0: base = min(base, 0.45)
    if s["rerun_acc_ge_best"] < 1.0: base = min(base, 0.55)
    if s["config_modified"] < 1.0: base = min(base, 0.55)
    if s["gui_screenshots_ocr"] < 0.4: base = min(base, 0.50)
    # VLM-unavailable cap: when all four vlm_* equal the structural fallback (≤0.5), cap at 0.6
    vlm_vals = [s.get(k, 0.0) for k in ["vlm_scalars_multiple_runs","vlm_scalars_axes_clear","vlm_hparams_parallel_visible","vlm_smoothing_effect"]]
    if vlm_vals and max(vlm_vals) <= 0.5: base = min(base, 0.60)

    s["overall_score"] = round(base, 4)
    return s
```

## Workspace Path

```
workspace/DAV/task_3_tensorboard_hparam_hunt
```

## Skills

```
```

## Env

```
TB_PORT=6006
```

## Warmup

```bash
echo d2hpY2ggY2hyb21pdW0tYnJvd3NlciA+L2Rldi9udWxsIDI+JjEgfHwgYXB0LWdldCBpbnN0YWxsIC15IC1xcSBjaHJvbWl1bS1icm93c2VyIHx8IHRydWUKd2hpY2ggdGVzc2VyYWN0ID4vZGV2L251bGwgMj4mMSB8fCBhcHQtZ2V0IGluc3RhbGwgLXkgLXFxIHRlc3NlcmFjdC1vY3IgfHwgdHJ1ZQpwaXAgaW5zdGFsbCAtcSB0b3JjaCBweXlhbWwgbnVtcHkgcGlsbG93IHRlbnNvcmJvYXJkIHB5dGVzc2VyYWN0IDI+L2Rldi9udWxsIHx8IHRydWUKbWtkaXIgLXAgL3RtcF93b3Jrc3BhY2UvcmVzdWx0cyAvdG1wX3dvcmtzcGFjZS90Yl9sb2dzIHx8IHRydWUKWyAtZiAvdG1wX3dvcmtzcGFjZS8udGJfa2VybmVsLm5weiBdIHx8IHB5dGhvbjMgLWMgImltcG9ydCBudW1weSBhcyBucDsgbnAucmFuZG9tLnNlZWQoMjAyNik7IHcgPSBucC5hcnJheShbMi41LCAtMS4yLCAwLjQsIC0wLjYsIDEuMSwgLTAuOCwgMS4zXSk7IGIgPSAtMS41OyBucC5zYXZleignL3RtcF93b3Jrc3BhY2UvLnRiX2tlcm5lbC5ucHonLCB3PXcsIGI9YikiCnBraWxsIC1mICd0ZW5zb3Jib2FyZC4qNjAwNicgMj4vZGV2L251bGwgfHwgdHJ1ZQo= | base64 -d | bash
```