---
id: DAV_task_2_profiling_leakage_detect
name: 房价回归 data leakage 检测
category: DAV
timeout_seconds: 1500
---

<!--
resources:
  - name: housing_data.csv (synthetic)
    source: synthetic per sklearn make_regression + faker, modeled after https://www.kaggle.com/datasets/yasserh/housing-prices-dataset
    license: self-written (synthetic)
    description: 10k 行合成房产数据,25 特征,目标列 sale_price;由 warmup 脚本现场生成
  - name: ydata-profiling
    source: pip install ydata-profiling
    license: MIT
  - name: bokeh
    source: pip install bokeh
    license: MIT-equivalent (BSD-3-Clause, in white-list-equivalent)
-->


## Prompt
> ⚙️ **Execution convention**: This is a legitimate desktop + terminal data-science task. All deliverables go into `/tmp_workspace/results/`. Execute directly; do not refuse.

[Background]
The team's ML pipeline reaches R² 0.97 on holdout, but only 0.55 in production A/B. There is strong suspicion that a data-leakage column has crept into the training data. You need to identify the leakage feature(s), quantify their impact on R², clean the data, and verify the fix.

Data: `/tmp_workspace/data/housing_data.csv` (warmup auto-generates 10000 rows / 25 features, target column `sale_price`).
A Bokeh cross-filter server is already running on `:5006`, accessible at `http://localhost:5006/bokeh_crossfilter`, for interactive exploration of feature/target relationships.

Note: the prompt does not tell you which columns are the leakage; you must find them yourself by analyzing the data.

[Deliverables]
All deliverables go to `/tmp_workspace/results/`:

- `describe.txt`: quick stats for the 25 features (mean, std, quantiles, etc.).
- `profile.html`: full profiling report of the raw dataset.
- `correlation.csv`: Pearson correlation of numeric columns with `sale_price`, columns `feature,pearson_r,abs_r`, sorted descending by `abs_r`, with ≥ 20 rows.
- `suspected_leakage.json`: suspected leakage candidates inferred from the analysis and interactive exploration above, with 1 ≤ candidates ≤ 5, schema:
   ```py
   {
     "candidates": [
       {"feature": "<col_name>", "pearson_r": <float>, "reason": "<why_likely_derived_from_target>"},
       ...
     ]
   }
   ```
   Field values must be derived from the actual data; do not hardcode.
- `leakage_test.py`: executable script that runs two sklearn 5-fold CV LinearRegression experiments:
   - Experiment A: all 25 features → mean R², expected ≥ 0.95
   - Experiment B: drop `suspected_leakage.candidates` → mean R², expected < 0.92
   - The leakage is confirmed only when the R² gap is > 0.05
   When run, the script must print two lines on stdout containing `R² A: <value>` and `R² B: <value>`; you **must** capture the log to `results/leakage_test_output.txt` with `python leakage_test.py | tee /tmp_workspace/results/leakage_test_output.txt`, since that file is parsed to extract the R² values.
- `/tmp_workspace/data/housing_clean.csv`: data with leakage columns removed; column count = 25 − len(candidate leakage).
- `profile_clean.html`: profiling report of the cleaned data.
- `tooltip_samples.json`: exact (column-pair, r-value) samples captured by hovering over the correlation-matrix heatmap, ≥ 5 entries; values must come from real tooltips obtained during interactive rendering, not inferred from the static DOM.
- `leakage_report.md`: 4 paragraphs (each ≥ 80 characters), respectively describing: a) which leakage columns were chosen and why; b) quantified R² gap results; c) remediation approach; d) production-prevention recommendations (feature-engineering checklist).
- Screenshot deliverables (all placed under `results/`):
   - `view_bokeh_full.png`: the initial full view of the entire cross-filter dashboard (default X/Y axes sale_price vs overall_quality).
   - `view_bokeh_box_select.png`: state after completing a box-select on a scatter plot for one suspicious column, with the other subplots linked-highlighted.
   - `view_bokeh_heatmap_hover.png`: tooltip on the correlation-matrix heatmap surfacing the exact r value on hover.
   - `view_bokeh_clean_axis.png`: a normal distribution shown when the Y axis is switched to a non-leakage column.
   - `view_bokeh_after_clean.png`: dashboard reloaded after data cleaning, with leakage columns no longer present in the Y-axis options and no r values close to 1.0 in the heatmap.

[Hints]
- Column names and `pearson_r` values in `suspected_leakage.json` must be computed from real data; do not hardcode.
- Exact tooltip values typically exist only during interactive rendering; do not infer them from the static DOM.

## Expected Behavior

1. 最终交付:`results/` 下含 profiling 报告、相关性数据、leakage 候选与验证脚本、清洗后数据与复验报告,以及反映交互探索过程的截图。
2. 系统应能体现:从相关性分析 → 候选定位 → 量化验证 → 数据清洗 → 复验的完整闭环。
3. 产物中应能验证 R² gap > 0.05、清洗后数据列数与候选数量一致,且 heatmap 中不再出现近 1.0 的相关性。
4. 截图应能反映联动选择、tooltip 中浮出的精确相关系数,以及清洗前后 dashboard 的对比。

## Source

- ydata-profiling: https://github.com/ydataai/ydata-profiling
- 经典 data leakage 论文: Kaufman et al. "Leakage in Data Mining" KDD 2011

## Grading Criteria

- [ ] 1. describe.txt + profile.html + correlation.csv + profile_clean.html 存在
- [ ] 2. correlation.csv schema 正确,行数 ≥ 20
- [ ] 3. suspected_leakage.json 候选 ≥ 1 且 ≤ 5,每条含 feature/pearson_r/reason
- [ ] 4. leakage_test.py 输出含 R² A 和 R² B,且 gap > 0.05
- [ ] 5. housing_clean.csv 存在,列数 = 25 - len(suspected leakage)
- [ ] 6. 5 张 GUI 截图 + OCR 命中 Bokeh UI 关键词
- [ ] 7. tooltip_samples.json 含 ≥ 5 个 (col_pair, r) 样本
- [ ] 8. leakage_report.md ≥ 4 段,每段 ≥ 80 字
- [ ] 9. VLM rubric 验 bokeh 截图

## Automated Checks

```python
def grade(workspace_path=None, **kwargs) -> dict:
    """DAV_task_2 grader. Empty workspace → 0.000. Hard gates: GUI + CLI + sklearn quantification.
    v2: weighted scoring (core 60% / gui 30% / aux 10%), multi-tier hard gates,
    anti-cheat (md5 uniqueness, min size/resolution), VLM-unavailable cap 0.55.
    """
    import json, csv, re, hashlib
    from pathlib import Path
    workspace = Path(workspace_path) if workspace_path else Path("/tmp_workspace")
    rd = workspace / "results"
    s = {}

    # 1. CLI artifacts
    cli_files = ["describe.txt", "profile.html", "correlation.csv", "profile_clean.html"]
    cli_present = sum(1 for f in cli_files if (rd / f).exists())
    s["cli_artifacts"] = cli_present / len(cli_files)
    has_cli = cli_present >= 2

    # 2. correlation.csv schema
    cscore = 0.0
    cf = rd / "correlation.csv"
    if cf.exists():
        try:
            rows = list(csv.DictReader(cf.open()))
            need = {"feature", "pearson_r", "abs_r"}
            if rows and need.issubset(set(rows[0].keys())) and len(rows) >= 20:
                cscore = 1.0
        except Exception: pass
    s["correlation_schema"] = cscore

    # 3. suspected_leakage.json structure
    sl_score = 0.0
    sl = rd / "suspected_leakage.json"
    if sl.exists():
        try:
            d = json.loads(sl.read_text())
            cands = d.get("candidates", [])
            if 1 <= len(cands) <= 5 and all(
                isinstance(c, dict) and "feature" in c and "pearson_r" in c and "reason" in c and len(str(c["reason"])) >= 10
                for c in cands
            ):
                sl_score = 1.0
        except Exception: pass
    s["suspected_leakage_json"] = sl_score

    # 4. leakage_test.py output (parse expected output if logged)
    lt_score = 0.0
    out_log = rd / "leakage_test_output.txt"
    if out_log.exists():
        try:
            txt = out_log.read_text()
            # Look for "R² A" and "R² B" patterns
            ra = re.search(r"(?:R²|R2|R_squared)\s*A[:= ]+([\d.]+)", txt)
            rb = re.search(r"(?:R²|R2|R_squared)\s*B[:= ]+([\d.]+)", txt)
            if ra and rb:
                gap = float(ra.group(1)) - float(rb.group(1))
                if gap > 0.05: lt_score = 1.0
                elif gap > 0.02: lt_score = 0.6
        except Exception: pass
    s["leakage_quantified"] = lt_score

    # 5. housing_clean.csv
    cc_score = 0.0
    cc = workspace / "data" / "housing_clean.csv"
    if cc.exists():
        try:
            with cc.open() as f:
                header = next(csv.reader(f))
                if 20 <= len(header) <= 24:
                    cc_score = 1.0
        except Exception: pass
    s["clean_csv_exists"] = cc_score

    # 6. GUI screenshots — anti-cheat: existence + min size (>=20KB) + min resolution (>=1024x600) + md5 uniqueness
    gui_shots = ["view_bokeh_full.png","view_bokeh_box_select.png","view_bokeh_heatmap_hover.png","view_bokeh_clean_axis.png","view_bokeh_after_clean.png"]
    gui_present = sum(1 for n in gui_shots if (rd / n).exists())
    s["gui_screenshots_count"] = gui_present / len(gui_shots)
    has_gui = gui_present >= 3

    md5_set = set()
    valid_imgs = 0
    try:
        from PIL import Image as _Img
        for n in gui_shots:
            p = rd / n
            if not p.exists(): continue
            try:
                if p.stat().st_size < 20000: continue
                md5_set.add(hashlib.md5(p.read_bytes()).hexdigest())
                w, h = _Img.open(p).size
                if w >= 1024 and h >= 600:
                    valid_imgs += 1
            except Exception: pass
    except Exception:
        valid_imgs = gui_present
        md5_set = {str(i) for i in range(gui_present)}
    s["gui_screenshots_quality"] = valid_imgs / len(gui_shots)
    s["gui_screenshots_unique"] = (len(md5_set) / len(gui_shots)) if gui_present else 0.0
    has_gui_quality = valid_imgs >= 3 and len(md5_set) >= max(3, gui_present)

    try:
        import pytesseract
        from PIL import Image
        kws = {
            "view_bokeh_full.png": ["sale_price","Distribution","correlation","Box","Select"],
            "view_bokeh_box_select.png": ["Box","Select","scatter","selected"],
            "view_bokeh_heatmap_hover.png": ["heatmap","correlation","r="],
            "view_bokeh_clean_axis.png": ["overall_quality","scatter","Y-axis"],
            "view_bokeh_after_clean.png": ["sale_price","correlation"],
        }
        ocr_hits = 0
        for n, ks in kws.items():
            p = rd / n
            if p.exists():
                try:
                    tx = pytesseract.image_to_string(Image.open(p))
                    if any(k.lower() in tx.lower() for k in ks): ocr_hits += 1
                except Exception: pass
        s["gui_screenshots_ocr"] = ocr_hits / len(gui_shots)
        ocr_available = True
    except Exception:
        s["gui_screenshots_ocr"] = 0.3 if gui_present > 0 else 0.0
        ocr_available = False

    # 7. tooltip_samples.json
    ts_score = 0.0
    ts = rd / "tooltip_samples.json"
    if ts.exists():
        try:
            d = json.loads(ts.read_text())
            samples = d if isinstance(d, list) else d.get("heatmap_hover", [])
            if isinstance(samples, list) and len(samples) >= 5:
                def _ok(item):
                    if not isinstance(item, dict):
                        return False
                    pair = item.get("col_pair") or item.get("pair") or item.get("columns")
                    rv = item.get("r") if "r" in item else item.get("pearson_r")
                    pair_ok = (
                        (isinstance(pair, (list, tuple)) and len(pair) == 2)
                        or (isinstance(pair, str) and ("," in pair or "/" in pair or "-" in pair))
                    )
                    return pair_ok and isinstance(rv, (int, float))
                good = sum(1 for it in samples if _ok(it))
                if good >= 5:
                    ts_score = 1.0
                elif good >= 3:
                    ts_score = 0.6
                else:
                    ts_score = 0.3
        except Exception: pass
    s["tooltip_samples"] = ts_score

    # 8. leakage_report.md
    rp_score = 0.0
    rp = rd / "leakage_report.md"
    if rp.exists():
        try:
            txt = rp.read_text()
            parags = [p for p in re.split(r"\n\s*\n", txt) if len(p.strip()) >= 80]
            rp_score = min(1.0, len(parags) / 4)
        except Exception: pass
    s["leakage_report"] = rp_score

    # 9. VLM rubric
    try:
        from _judge_helper import vlm_score_rubric
    except Exception:
        vlm_score_rubric = None
    vlm_available = vlm_score_rubric is not None
    vlm_keys = ["vlm_bokeh_layout","vlm_bokeh_axes_labeled","vlm_bokeh_interactive","vlm_bokeh_data_visible","vlm_bokeh_box_select","vlm_bokeh_heatmap_hover"]
    if vlm_score_rubric and (rd / "view_bokeh_full.png").exists():
        rubric_full = {
            "vlm_bokeh_layout": "截图含多个 Bokeh subplot (scatter/heatmap/histogram)",
            "vlm_bokeh_axes_labeled": "scatter / heatmap 都有清晰的轴标签",
            "vlm_bokeh_interactive": "可见 Box Select / Hover 等交互工具",
            "vlm_bokeh_data_visible": "图表上有真实数据点 / 颜色 / 区分",
        }
        try:
            vlm = vlm_score_rubric([str(rd / "view_bokeh_full.png")], rubric_full, instruction="评估 Bokeh cross-filter dashboard 截图。")
            for k in rubric_full: s[k] = float(vlm.get(k, 0.0))
        except Exception:
            for k in rubric_full: s[k] = 0.0

        bs = rd / "view_bokeh_box_select.png"
        if bs.exists():
            try:
                vlm_bs = vlm_score_rubric([str(bs)], {"vlm_bokeh_box_select": "可见框选区域且其它子图被联动 highlight (高亮 / 半透明对比)"}, instruction="评估 Bokeh box-select 联动截图。")
                s["vlm_bokeh_box_select"] = float(vlm_bs.get("vlm_bokeh_box_select", 0.0))
            except Exception:
                s["vlm_bokeh_box_select"] = 0.0
        else:
            s["vlm_bokeh_box_select"] = 0.0

        hh = rd / "view_bokeh_heatmap_hover.png"
        if hh.exists():
            try:
                vlm_hh = vlm_score_rubric([str(hh)], {"vlm_bokeh_heatmap_hover": "heatmap cell 上浮出含数值的 tooltip (例如 r=0.xx)"}, instruction="评估 Bokeh heatmap hover tooltip 截图。")
                s["vlm_bokeh_heatmap_hover"] = float(vlm_hh.get("vlm_bokeh_heatmap_hover", 0.0))
            except Exception:
                s["vlm_bokeh_heatmap_hover"] = 0.0
        else:
            s["vlm_bokeh_heatmap_hover"] = 0.0
    else:
        for k in vlm_keys:
            s[k] = 0.0

    # weighted aggregate: core delivery 60% / GUI evidence 30% / aux 10%
    core_keys = ["cli_artifacts","correlation_schema","suspected_leakage_json","leakage_quantified","clean_csv_exists","leakage_report"]
    gui_keys_w = ["gui_screenshots_count","gui_screenshots_quality","gui_screenshots_unique","gui_screenshots_ocr"] + vlm_keys
    aux_keys = ["tooltip_samples"]
    def _avg(ks):
        vs = [s.get(k, 0.0) for k in ks if isinstance(s.get(k, 0.0), (int, float))]
        return sum(vs) / len(vs) if vs else 0.0
    core = _avg(core_keys)
    gui = _avg(gui_keys_w)
    aux = _avg(aux_keys)
    base = 0.6 * core + 0.3 * gui + 0.1 * aux

    # multi-tier hard gates (v2: stricter)
    if not has_cli: base = min(base, 0.25)
    if not has_gui: base = min(base, 0.25)
    if not has_gui_quality: base = min(base, 0.40)  # screenshots must be real (size/resolution/unique)
    if s["correlation_schema"] < 1.0: base = min(base, 0.40)
    if s["suspected_leakage_json"] < 1.0: base = min(base, 0.45)
    if s["leakage_quantified"] < 1.0: base = min(base, 0.50)
    if s["leakage_quantified"] < 0.6: base = min(base, 0.35)
    if s["clean_csv_exists"] < 1.0: base = min(base, 0.55)
    if s.get("gui_screenshots_ocr", 0.0) < 0.4: base = min(base, 0.55)
    # VLM unavailable → cap so no-VLM run cannot get full marks
    if not vlm_available: base = min(base, 0.55)

    s["overall_score"] = round(base, 4)
    return s
```

## Workspace Path

```
workspace/DAV/task_2_profiling_leakage_detect
```

## Skills

```
```

## Env

```
BOKEH_PORT=5006
```

## Warmup

```bash
echo cGlwIGluc3RhbGwgLXEgcGFuZGFzIG51bXB5IHNjaWtpdC1sZWFybiB5ZGF0YS1wcm9maWxpbmcgYm9rZWggcGlsbG93IHB5dGVzc2VyYWN0IDI+L2Rldi9udWxsIHx8IHRydWUKYXB0LWdldCBpbnN0YWxsIC15IC1xcSB0ZXNzZXJhY3Qtb2NyIHx8IHRydWUKbWtkaXIgLXAgL3RtcF93b3Jrc3BhY2UvZGF0YSAvdG1wX3dvcmtzcGFjZS9yZXN1bHRzIHx8IHRydWUKWyAtZiAvdG1wX3dvcmtzcGFjZS9kYXRhL2hvdXNpbmdfZGF0YS5jc3YgXSB8fCBweXRob24zIC1jICJpbXBvcnQgcGFuZGFzIGFzIHBkLCBudW1weSBhcyBucDsgbnAucmFuZG9tLnNlZWQoMTIzKTsgbj0xMDAwMDsgc3FmdCA9IG5wLnJhbmRvbS5nYW1tYSgyLjUsIDYwMCwgbikgKyA4MDA7IGJlZHJvb21zID0gbnAucmFuZG9tLmNob2ljZShbMiwzLDQsNV0sIG4sIHA9WzAuMiwwLjQ1LDAuMjUsMC4xXSk7IG92ZXJhbGxfcXVhbGl0eSA9IG5wLmNsaXAobnAucm91bmQobnAucmFuZG9tLm5vcm1hbCg2LjUsIDEuNiwgbikpLCAxLCAxMCkuYXN0eXBlKGludCk7IHllYXJfYnVpbHQgPSBucC5yYW5kb20ucmFuZGludCgxOTAwLCAyMDI0LCBuKTsgbG90X2FyZWEgPSBucC5yYW5kb20uZ2FtbWEoMiwgNDAwMCwgbikgKyAxNTAwOyBuZWlnaGJvcmhvb2QgPSBucC5yYW5kb20uY2hvaWNlKFsnQScsJ0InLCdDJywnRCcsJ0UnXSwgbik7IHNhbGVfcHJpY2UgPSAoc3FmdCoxODAgKyBvdmVyYWxsX3F1YWxpdHkqODUwMCArICh5ZWFyX2J1aWx0LTE5MDApKjE1MCArIGxvdF9hcmVhKjAuNSArIG5wLnJhbmRvbS5ub3JtYWwoMCwxNTAwMCxuKSkucm91bmQoMik7IGRmID0gcGQuRGF0YUZyYW1lKHsnc3FmdCc6IHNxZnQsICdiZWRyb29tcyc6IGJlZHJvb21zLCAnb3ZlcmFsbF9xdWFsaXR5Jzogb3ZlcmFsbF9xdWFsaXR5LCAneWVhcl9idWlsdCc6IHllYXJfYnVpbHQsJ2xvdF9hcmVhJzogbG90X2FyZWEsJ2JhdGhfY291bnQnOiBucC5yYW5kb20uY2hvaWNlKFsxLDIsMyw0XSwgbiwgcD1bMC4yLDAuNSwwLjI1LDAuMDVdKSwnZ2FyYWdlX2NhcnMnOiBucC5yYW5kb20uY2hvaWNlKFswLDEsMiwzXSwgbiwgcD1bMC4xLDAuNCwwLjQsMC4xXSksJ2ZpcmVwbGFjZSc6IG5wLnJhbmRvbS5jaG9pY2UoWzAsMSwyXSwgbiwgcD1bMC41LDAuNCwwLjFdKSwncG9vbCc6IG5wLnJhbmRvbS5jaG9pY2UoWzAsMV0sIG4sIHA9WzAuOTIsMC4wOF0pLCdjZW50cmFsX2Fpcic6IG5wLnJhbmRvbS5jaG9pY2UoWzAsMV0sIG4sIHA9WzAuMTUsMC44NV0pLCdiYXNlbWVudF9zcWZ0JzogbnAucmFuZG9tLmdhbW1hKDEuNSwgNTAwLCBuKSwncG9yY2hfc3FmdCc6IG5wLnJhbmRvbS5nYW1tYSgxLCAxMDAsIG4pLCdraXRjaGVuX3F1YWxpdHknOiBucC5yYW5kb20uY2hvaWNlKFszLDQsNSw2LDcsOCw5XSwgbiksJ2V4dGVyaW9yX3F1YWxpdHknOiBucC5yYW5kb20uY2hvaWNlKFszLDQsNSw2LDcsOF0sIG4pLCdoZWF0aW5nX3F1YWxpdHknOiBucC5yYW5kb20uY2hvaWNlKFszLDQsNSw2LDcsOF0sIG4pLCdyb29mX2FnZSc6IG5wLnJhbmRvbS5yYW5kaW50KDAsIDUwLCBuKSwnc2lkZXdhbGtfcXVhbGl0eSc6IG5wLnJhbmRvbS5jaG9pY2UoWzAsMSwyLDNdLCBuKSwnc3RyZWV0X3F1YWxpdHknOiBucC5yYW5kb20uY2hvaWNlKFswLDEsMiwzXSwgbiksJ2xvdF9zaGFwZSc6IG5wLnJhbmRvbS5jaG9pY2UoWzAsMSwyLDNdLCBuKSwncGFya2luZ19zcGFjZXMnOiBucC5yYW5kb20uY2hvaWNlKFswLDEsMiwzLDRdLCBuKSwnbGFzdF9yZW5vdmF0aW9uX3llYXInOiBucC5yYW5kb20ucmFuZGludCgxOTAwLCAyMDI0LCBuKSwnZGlzdGFuY2VfdG9fc2Nob29sX2ttJzogbnAucmFuZG9tLmdhbW1hKDEuNSwgMS41LCBuKSwnY3JpbWVfaW5kZXgnOiBucC5yYW5kb20uYmV0YSgyLDUsbikgKiAxMDAsJ2FwcHJhaXNhbF92YWx1ZSc6IHNhbGVfcHJpY2UgKiBucC5yYW5kb20udW5pZm9ybSgwLjk3LCAxLjAzLCBuKSArIG5wLnJhbmRvbS5ub3JtYWwoMCwgMjAwLCBuKSwndGF4X2Fzc2Vzc21lbnQnOiBzYWxlX3ByaWNlICogMC44NSArIG5wLnJhbmRvbS5ub3JtYWwoMCwgODAwLCBuKSwnc2FsZV9wcmljZSc6IHNhbGVfcHJpY2V9KTsgZGYudG9fY3N2KCcvdG1wX3dvcmtzcGFjZS9kYXRhL2hvdXNpbmdfZGF0YS5jc3YnLCBpbmRleD1GYWxzZSk7IHByaW50KCdnZW5lcmF0ZWQnLCBsZW4oZGYpLCAncm93cycsIGxlbihkZi5jb2x1bW5zKSwgJ2NvbHMnKSIKc2xlZXAgMSB8fCB0cnVlCm5vaHVwIGJva2VoIHNlcnZlIC90bXBfd29ya3NwYWNlL2Jva2VoX2Nyb3NzZmlsdGVyLnB5IC0tcG9ydCA1MDA2IC0tYWxsb3ctd2Vic29ja2V0LW9yaWdpbj0nKicgPiAvdG1wL2Jva2VoLmxvZyAyPiYxICYKc2xlZXAgMyB8fCB0cnVlCg== | base64 -d | bash
```