Deploy FlakyGym UI + inference updates (minimal upload)

.gitignore

@@ -0,0 +1,32 @@
+# Python caches
+__pycache__/
+*.py[cod]
+*.so
+
+# Virtual environments
+.venv/
+venv/
+openenv/
+
+# Local editor / tooling state
+.vscode/
+.codex/
+.agents/
+
+# Secrets and local config
+.env
+.env.*
+*.local
+
+# Logs and runtime artifacts
+*.log
+outputs/
+
+# Build artifacts
+build/
+dist/
+*.egg-info/
+
+# Dataset/generated outputs
+dataset/__pycache__/
+dataset/py_tasks.csv

.openenv_push_ignore

@@ -0,0 +1,26 @@
+.git/
+.agents/
+.codex/
+.vscode/
+openenv/
+openenv/**
+.venv/
+.venv/**
+venv/
+venv/**
+__pycache__/
+**/__pycache__/
+*.pyc
+*.pyo
+*.pyd
+*.log
+*.tmp
+*.cache
+agent_trace.log
+debug_trace.log
+debug_trace2.log
+debug_trace3.log
+debug_trace4.log
+debug_trace5.log
+outputs/
+.pytest_cache/

Dockerfile

@@ -14,5 +14,4 @@ COPY . .
 
 EXPOSE 8000
 
-ENV ENABLE_WEB_INTERFACE=true
 CMD ["python", "-m", "server.app"]

GRADING.md

@@ -73,8 +73,25 @@ reward = progress
   - else -> `0.01`
 
 ### `search_code`
-- if query contains any flaky-signal tokens (`sleep`, `random`, `time`, `datetime`, `thread`, `asyncio`, `fixture`, `setup`, `teardown`, `global`, `shared`, `singleton`, `os.environ`, `socket`, `timeout`, `retry`, `mock`, `patch`) -> `0.04`
-- otherwise -> `0.01`
+- base reward:
+  - if query contains any flaky-signal tokens (`sleep`, `random`, `time`, `datetime`, `thread`, `asyncio`, `fixture`, `setup`, `teardown`, `global`, `shared`, `singleton`, `os.environ`, `socket`, `timeout`, `retry`, `mock`, `patch`) -> `0.04`
+  - otherwise -> `0.01`
+- spam penalties (all apply, then summed and capped):
+  - repeated same normalized search pattern in episode:
+    - `repeat_penalty = min(0.02 * (pattern_count - 1), 0.12)` for `pattern_count > 1`
+  - repeated same search context (same normalized pattern + same extracted top `.py` hit files):
+    - `context_penalty = min(0.03 * (context_count - 1), 0.15)` for `context_count > 1`
+  - long search-only streak:
+    - `streak_penalty = min(0.02 * (consecutive_searches - 3), 0.20)` for `consecutive_searches > 3`
+  - total spam penalty cap: `min(sum_penalties, 0.35)`
+- final `search_code` progress:
+
+```text
+progress = max(-0.25, base_reward - spam_penalty)
+```
+
+- environment appends `WARNING:` text to tool output when penalties fire.
+- `consecutive_searches` resets on any non-`search_code` action.
 
 ### `run_test`
 - if category is **not** one of `OD`, `OD-Brit`, `OD-Vic` -> `0.05`
@@ -251,3 +268,15 @@ reward = clamp(0.05 + 0.0, 0, 1) = 0.05
 - Timeout does not invoke grader; it only ends the episode.
 - Dataset construction choices (especially `label` and category quality) heavily influence observed score behavior.
 
+## 9) Inference-side controls (not grader formulas)
+
+`inference.py` now includes policy/runtime controls that do not change grader math directly but change agent behavior:
+
+- episode memory injected into every prompt (recent files, search patterns, no-progress streak)
+- explicit loop warning prompt when no-progress/duplicate patterns are detected
+- duplicate `read_file` attempts are overridden to targeted `search_code`
+- conversation compaction controls:
+  - `--history-prune-start-step` (default `12`)
+  - `--history-window-turns` (default `4`)
+  - `--history-max-chars` (default `50000`)
+- detailed tracing options (`--trace-agent`, `--trace-prompts`) for audit/debug

README.md

@@ -15,6 +15,8 @@ tags:
 
 OpenEnv-compatible RL environment for flaky-test investigation in real Python repos.
 
+Flaky tests are dangerous because they make CI results untrustworthy: real regressions can be ignored as "just flaky," while healthy code can fail randomly and block releases, wasting engineering time and eroding confidence in test signals. We are building this Gym-style RL environment so agents can practice flaky-test triage in realistic repositories, learn to separate true failures from nondeterministic noise, and generate faster, more reliable debugging and fix strategies at scale.
+
 ## Setup
 
 ```bash
@@ -58,14 +60,15 @@ curl -s http://localhost:8000/health
 
 ## Run Inference
 
-Recommended (OpenRouter):
+Recommended (HF Router/OpenRouter/OpenAI compatible):
 
 ```bash
-export OPENROUTER_API_KEY=your_openrouter_api_key
-export API_BASE_URL=https://openrouter.ai/api/v1
-export MODEL_NAME=qwen/qwen3.6-plus:free
+export HF_TOKEN=your_hf_token
+# optional:
+# export API_BASE_URL=https://router.huggingface.co/v1
+# export MODEL_NAME=openai/gpt-oss-120b:novita
 
-python inference.py --dataset-path dataset/py_tasks.csv --episodes-per-task 5
+python inference.py --dataset-path dataset/py_tasks.csv --episodes-per-task 2
 ```
 
 ### Run Inference From Space UI
@@ -73,22 +76,41 @@ python inference.py --dataset-path dataset/py_tasks.csv --episodes-per-task 5
 When deployed, the Space homepage serves a UI at `/` (also `/web`) that starts
 `inference.py` in the background and streams logs live.
 
+UI defaults:
+- `episodes_per_task=1`
+- slider range up to `100`
+- live ETA estimator: `selected_tasks × episodes_per_task × 180s`
+- warning when ETA may exceed 20 minutes (hackathon guidance)
+
 ### `inference.py` flags
 
 | Flag | Type | Default | Description |
 |---|---|---|---|
 | `--dataset-path` | `str` | `dataset/py_tasks.csv` | Processed task CSV used by env |
-| `--episodes-per-task` | `int` | `5` | Episodes per selected task type |
+| `--episodes-per-task` | `int` | `2` | Episodes per selected task type |
 | `--task-types` | `str` | `classify,root_cause,fix_proposal` | Comma-separated task types |
 | `--max-steps` | `int` | `20` | Max steps per episode |
+| `--no-progress` | flag | `False` | Disable progress bars in non-compliance mode |
+| `--trace-agent` | flag | `False` | Print detailed action/model/tool trace |
+| `--trace-prompts` | flag | `False` | Include full prompts in trace |
+| `--trace-max-chars` | `int` | `2500` | Clip size for traced prompt/output blocks |
+| `--compliance-stdout` | flag | `True` | Strict `[START]/[STEP]/[END]` logs (default on) |
+| `--no-compliance-stdout` | flag | `False` | Switch to baseline summary/progress output |
 | `--benchmark-name` | `str` | `flakysleuth` | Label printed in `[START]` logs |
+| `--history-prune-start-step` | `int` | `12` | Start compacting history from this step |
+| `--history-window-turns` | `int` | `4` | Keep this many recent assistant/user turns on prune |
+| `--history-max-chars` | `int` | `50000` | Force prune when message history exceeds this size |
 
-Trace to log:
+Detailed trace to log:
 ```bash
 python inference.py \
   --dataset-path dataset/py_tasks.csv \
-  --episodes-per-task 5 \
-  --task-types classify,root_cause > agent_trace.log 2>&1
+  --episodes-per-task 1 \
+  --task-types classify,root_cause \
+  --no-compliance-stdout \
+  --trace-agent \
+  --history-prune-start-step 12 \
+  --history-window-turns 4 > agent_trace.log 2>&1
 ```
 
 ## OpenEnv CLI

env/environment.py

@@ -41,6 +41,9 @@ class FlakySleuthEnv:
         self.cumulative_progress = 0.0
         self.files_read: set[str] = set()
         self.episode_actions: list[FlakySleuthAction] = []
+        self.search_pattern_counts: dict[str, int] = {}
+        self.search_context_counts: dict[str, int] = {}
+        self.consecutive_searches = 0
 
     def reset(self) -> FlakySleuthObservation:
         if self.sandbox:
@@ -61,6 +64,9 @@ class FlakySleuthEnv:
         self.cumulative_progress = 0.0
         self.files_read = set()
         self.episode_actions = []
+        self.search_pattern_counts = {}
+        self.search_context_counts = {}
+        self.consecutive_searches = 0
 
         return self._make_obs()
 
@@ -153,6 +159,8 @@ class FlakySleuthEnv:
 
         progress = 0.0
         output = ""
+        if action.action_type != "search_code":
+            self.consecutive_searches = 0
 
         if action.action_type == "read_file":
             content = self.sandbox.read_file(action.argument)
@@ -168,8 +176,13 @@ class FlakySleuthEnv:
                 progress = self._file_relevance_reward(action.argument)
 
         elif action.action_type == "search_code":
+            self.consecutive_searches += 1
             output = self.sandbox.grep(action.argument)
-            progress = self._search_relevance_reward(action.argument)
+            base_progress = self._search_relevance_reward(action.argument)
+            spam_penalty, warnings = self._search_spam_penalty(action.argument, output)
+            progress = max(-0.25, base_progress - spam_penalty)
+            if warnings:
+                output = f"{output}\n\nWARNING: {' '.join(warnings)}"
 
         elif action.action_type == "run_test":
             output = self.sandbox.run_test(self.current_task.get("test_name", ""))
@@ -198,6 +211,60 @@ class FlakySleuthEnv:
             return 0.04
         return 0.01
 
+    def _search_spam_penalty(self, pattern: str, output: str) -> tuple[float, list[str]]:
+        penalty = 0.0
+        warnings: list[str] = []
+
+        pattern_key = " ".join(pattern.lower().split())
+        if pattern_key:
+            pattern_count = self.search_pattern_counts.get(pattern_key, 0) + 1
+            self.search_pattern_counts[pattern_key] = pattern_count
+            if pattern_count > 1:
+                repeat_penalty = min(0.02 * (pattern_count - 1), 0.12)
+                penalty += repeat_penalty
+                warnings.append(
+                    f"Repeated search pattern ({pattern_count}x) penalty={repeat_penalty:.2f}."
+                )
+
+        context_hits = self._extract_search_hits(output)
+        context_key = f"{pattern_key}::{','.join(context_hits)}"
+        context_count = self.search_context_counts.get(context_key, 0) + 1
+        self.search_context_counts[context_key] = context_count
+        if context_count > 1:
+            context_penalty = min(0.03 * (context_count - 1), 0.15)
+            penalty += context_penalty
+            warnings.append(
+                f"Same search context repeated ({context_count}x) penalty={context_penalty:.2f}."
+            )
+
+        if self.consecutive_searches > 3:
+            streak_penalty = min(0.02 * (self.consecutive_searches - 3), 0.20)
+            penalty += streak_penalty
+            warnings.append(
+                f"Search-only streak={self.consecutive_searches} penalty={streak_penalty:.2f}."
+            )
+
+        return min(penalty, 0.35), warnings
+
+    def _extract_search_hits(self, output: str) -> tuple[str, ...]:
+        files: list[str] = []
+        seen: set[str] = set()
+        for raw_line in output.splitlines():
+            line = raw_line.strip()
+            if not line or line.startswith("No matches found") or line.startswith("Search "):
+                continue
+            filepath = line.split(":", 1)[0].strip()
+            if filepath.startswith("./"):
+                filepath = filepath[2:]
+            if not filepath.endswith(".py"):
+                continue
+            if filepath not in seen:
+                seen.add(filepath)
+                files.append(filepath)
+            if len(files) >= 5:
+                break
+        return tuple(files)
+
     def _make_obs(self, tool_output: str | None = None) -> FlakySleuthObservation:
         if not self.current_task:
             raise RuntimeError("No current task available")

inference.py

@@ -1,23 +1,44 @@
-"""FlakySleuth compliance inference script.
+"""FlakySleuth baseline inference script.
+
+Environment variables:
+  Preferred:
+    HF_TOKEN / HUGGINGFACE_HUB_TOKEN (or OPENROUTER_API_KEY / API_KEY)
+    API_BASE_URL (optional, defaults to https://openrouter.ai/api/v1 for router-style keys)
+    MODEL_NAME (optional, defaults to qwen/qwen3.6-plus:free on OpenRouter)
+
+  Optional fallback:
+    OPENAI_API_KEY
+    API_BASE_URL (defaults to https://api.openai.com/v1 when OpenAI key is used)
+    MODEL_NAME (defaults to gpt-4o-mini for OpenAI)
 """
 
 from __future__ import annotations
 
-import argparse
 import json
 import os
+import argparse
+import time
+from collections import defaultdict
+from pathlib import Path
 from typing import Any
 
 from openai import OpenAI
 
+try:
+    from tqdm import tqdm
+except Exception:  # pragma: no cover
+    tqdm = None
+
 from env.environment import FlakySleuthEnv
 from env.models import FlakySleuthAction, FlakySleuthObservation
 
-HF_TOKEN = os.environ.get("HF_TOKEN") or os.environ.get("HUGGINGFACE_HUB_TOKEN")
 OPENROUTER_API_KEY = os.environ.get("OPENROUTER_API_KEY")
 OPENAI_API_KEY = os.environ.get("OPENAI_API_KEY")
+HF_TOKEN = os.environ.get("HF_TOKEN") or os.environ.get("HUGGINGFACE_HUB_TOKEN")
+# Optional for environments created via from_docker_image(); kept for checklist parity.
+LOCAL_IMAGE_NAME = os.environ.get("LOCAL_IMAGE_NAME")
 RAW_API_KEY = os.environ.get("API_KEY")
-API_KEY = RAW_API_KEY or HF_TOKEN or OPENROUTER_API_KEY or OPENAI_API_KEY or ""
+API_KEY = RAW_API_KEY or OPENROUTER_API_KEY or OPENAI_API_KEY or HF_TOKEN or ""
 
 
 def _looks_like_openrouter_key(key: str | None) -> bool:
@@ -26,14 +47,19 @@ def _looks_like_openrouter_key(key: str | None) -> bool:
 
 DEFAULT_BASE_URL = (
     "https://router.huggingface.co/v1"
-    if (HF_TOKEN and not RAW_API_KEY and not OPENROUTER_API_KEY and not OPENAI_API_KEY)
+    if (
+        HF_TOKEN
+        and not RAW_API_KEY
+        and not OPENROUTER_API_KEY
+        and not OPENAI_API_KEY
+    )
     else (
-        "https://openrouter.ai/api/v1"
-        if (
-            (OPENROUTER_API_KEY and not RAW_API_KEY and not OPENAI_API_KEY)
-            or (_looks_like_openrouter_key(RAW_API_KEY) and not OPENAI_API_KEY)
-        )
-        else "https://api.openai.com/v1"
+    "https://openrouter.ai/api/v1"
+    if (
+        (OPENROUTER_API_KEY and not RAW_API_KEY and not OPENAI_API_KEY)
+        or (_looks_like_openrouter_key(RAW_API_KEY) and not OPENAI_API_KEY)
+    )
+    else "https://api.openai.com/v1"
     )
 )
 API_BASE_URL = os.environ.get("API_BASE_URL", DEFAULT_BASE_URL)
@@ -41,13 +67,17 @@ API_BASE_URL = os.environ.get("API_BASE_URL", DEFAULT_BASE_URL)
 DEFAULT_MODEL = (
     "openai/gpt-oss-120b:novita"
     if API_BASE_URL.startswith("https://router.huggingface.co")
-    else ("qwen/qwen3.6-plus:free" if API_BASE_URL.startswith("https://openrouter.ai") else "gpt-4o-mini")
+    else (
+    "qwen/qwen3.6-plus:free"
+    if API_BASE_URL.startswith("https://openrouter.ai")
+    else "gpt-4o-mini"
+    )
 )
 MODEL_NAME = os.environ.get("MODEL_NAME", DEFAULT_MODEL)
-
-EPISODES_PER_TASK = 5
+# Keep a conservative default to stay under common hackathon runtime limits.
+EPISODES_PER_TASK = 2
 MAX_STEPS = 20
-BENCHMARK_NAME = "flakysleuth"
+MEMORY_MAX_CHARS = 900
 
 client = OpenAI(api_key=API_KEY, base_url=API_BASE_URL)
 
@@ -81,34 +111,44 @@ Rules:
 """
 
 
-def _single_line(text: str) -> str:
+def _to_single_line(text: str) -> str:
     return " ".join(str(text).split())
 
 
-def log_start(task: str, env_name: str, model: str) -> None:
-    print(f"[START] task={task} env={env_name} model={model}", flush=True)
+def _compliance_log_start(task: str, benchmark: str, model: str) -> None:
+    print(f"[START] task={task} env={benchmark} model={model}", flush=True)
 
 
-def log_step(step: int, action: str, reward: float, done: bool, error: str | None) -> None:
-    error_value = _single_line(error) if error else "null"
-    done_value = str(bool(done)).lower()
+def _compliance_log_step(
+    step: int,
+    action: str,
+    reward: float,
+    done: bool,
+    error: str | None,
+) -> None:
+    error_value = _to_single_line(error) if error else "null"
     print(
-        f"[STEP] step={step} action={_single_line(action)} "
-        f"reward={reward:.2f} done={done_value} error={error_value}",
+        f"[STEP] step={step} action={_to_single_line(action)} "
+        f"reward={reward:.2f} done={str(bool(done)).lower()} error={error_value}",
         flush=True,
     )
 
 
-def log_end(success: bool, steps: int, score: float, rewards: list[float]) -> None:
-    rewards_str = ",".join(f"{r:.2f}" for r in rewards)
+def _compliance_log_end(success: bool, steps: int, score: float, rewards: list[float]) -> None:
+    rewards_value = ",".join(f"{r:.2f}" for r in rewards)
     print(
         f"[END] success={str(bool(success)).lower()} steps={steps} "
-        f"score={score:.2f} rewards={rewards_str}",
+        f"score={score:.2f} rewards={rewards_value}",
         flush=True,
     )
 
 
-def obs_to_prompt(obs: FlakySleuthObservation, max_steps: int) -> str:
+def obs_to_prompt(
+    obs: FlakySleuthObservation,
+    *,
+    memory_hint: str | None = None,
+    max_steps: int = MAX_STEPS,
+) -> str:
     tree_preview = "\n".join(obs.file_tree[:40])
     return f"""TASK: {obs.task_description}
 
@@ -125,7 +165,10 @@ Repository file tree:
 {tree_preview}
 
 Last tool output:
-{obs.tool_output or '(No action taken yet)'}
+{obs.tool_output or "(No action taken yet)"}
+
+Episode memory:
+{memory_hint or "(No memory yet.)"}
 
 Return only JSON action."""
 
@@ -157,10 +200,18 @@ def heuristic_action(obs: FlakySleuthObservation) -> FlakySleuthAction:
     )
 
 
-def llm_action(messages: list[dict[str, str]]) -> FlakySleuthAction | None:
+def llm_action(
+    messages: list[dict[str, str]],
+) -> tuple[FlakySleuthAction | None, dict[str, Any]]:
+    meta: dict[str, Any] = {
+        "attempted": False,
+        "raw_output": "",
+        "error": "",
+    }
     if not API_KEY:
-        return None
+        return None, meta
 
+    meta["attempted"] = True
     response = client.chat.completions.create(
         model=MODEL_NAME,
         messages=messages,
@@ -168,80 +219,407 @@ def llm_action(messages: list[dict[str, str]]) -> FlakySleuthAction | None:
         temperature=0.0,
     )
     raw = (response.choices[0].message.content or "").strip()
+    meta["raw_output"] = raw
     cleaned = raw.replace("```json", "").replace("```", "").strip()
     payload = json.loads(cleaned)
-    return FlakySleuthAction.model_validate(payload)
+    return FlakySleuthAction.model_validate(payload), meta
+
+
+def _clip_text(text: str, max_chars: int) -> str:
+    if max_chars <= 0:
+        return text
+    if len(text) <= max_chars:
+        return text
+    remaining = len(text) - max_chars
+    return f"{text[:max_chars]}\n...[truncated {remaining} chars]"
+
+
+def _trace_print(
+    enabled: bool,
+    message: str,
+    *,
+    text: str | None = None,
+    max_chars: int = 0,
+) -> None:
+    if not enabled:
+        return
+    print(message)
+    if text is not None:
+        print(_clip_text(text, max_chars))
+
+
+def _format_duration(seconds: float) -> str:
+    seconds = max(0.0, float(seconds))
+    mins, secs = divmod(int(round(seconds)), 60)
+    hrs, mins = divmod(mins, 60)
+    if hrs > 0:
+        return f"{hrs:d}h {mins:02d}m {secs:02d}s"
+    return f"{mins:02d}m {secs:02d}s"
+
+
+def _build_episode_memory(
+    *,
+    unique_read_files: list[str],
+    zero_gain_read_files: set[str],
+    search_patterns: list[str],
+    blocked_duplicate_reads: int,
+    no_progress_streak: int,
+    max_chars: int,
+) -> str:
+    read_tail = ", ".join(unique_read_files[-8:]) if unique_read_files else "none"
+    zero_tail = ", ".join(sorted(zero_gain_read_files)[-8:]) if zero_gain_read_files else "none"
+    search_tail = ", ".join(search_patterns[-6:]) if search_patterns else "none"
+    loop_warning = (
+        "WARNING: Possible loop detected. Stop repeating similar exploration. "
+        "Switch strategy or take a terminal action."
+        if no_progress_streak >= 3 or blocked_duplicate_reads >= 2
+        else "Status: exploration progress appears normal."
+    )
+    memory = (
+        f"Read files (recent): {read_tail}\n"
+        f"Zero-gain read files: {zero_tail}\n"
+        f"Search patterns (recent): {search_tail}\n"
+        f"Blocked duplicate reads: {blocked_duplicate_reads}\n"
+        f"No-progress streak: {no_progress_streak}\n"
+        f"{loop_warning}\n"
+        "Guidance: Avoid rereading zero-gain files unless there is new evidence. "
+        "Prefer targeted search_code or terminal action when confidence is enough."
+    )
+    return _clip_text(memory, max_chars=max_chars)
+
+
+def _duplicate_read_replacement_pattern(obs: FlakySleuthObservation) -> str:
+    test_hint = obs.test_name.split("::")[-1] if obs.test_name else "test"
+    return (
+        f"{test_hint}|random|sleep|time|timeout|retry|asyncio|thread|"
+        "fixture|global|shared|mock|patch"
+    )
+
+
+def _messages_char_count(messages: list[dict[str, str]]) -> int:
+    # Lightweight size heuristic to avoid unbounded context growth.
+    return sum(len(str(msg.get("content", ""))) + 32 for msg in messages)
+
+
+def _prune_messages_window(
+    messages: list[dict[str, str]],
+    *,
+    step_number: int,
+    prune_start_step: int,
+    window_turns: int,
+    max_chars: int,
+) -> tuple[list[dict[str, str]], dict[str, Any] | None]:
+    if len(messages) <= 2:
+        return messages, None
+
+    current_chars = _messages_char_count(messages)
+    exceeds_step_threshold = step_number >= prune_start_step
+    exceeds_char_budget = current_chars > max_chars
+    if not exceeds_step_threshold and not exceeds_char_budget:
+        return messages, None
+
+    base = messages[:2]  # system + initial prompt
+    tail = messages[2:]
+    keep_tail_items = max(2, window_turns * 2)
+    if len(tail) > keep_tail_items:
+        tail = tail[-keep_tail_items:]
+    pruned = base + tail
+
+    reason = "step_threshold" if exceeds_step_threshold else "char_budget"
+    return pruned, {
+        "reason": reason,
+        "before_messages": len(messages),
+        "after_messages": len(pruned),
+        "before_chars": current_chars,
+        "after_chars": _messages_char_count(pruned),
+        "step": step_number,
+    }
 
 
 def run_episode(
     env: FlakySleuthEnv,
     *,
-    task_name: str,
-    benchmark_name: str,
-    max_steps: int,
-) -> float:
+    print_terminal: bool = True,
+    trace_agent: bool = False,
+    trace_prompts: bool = False,
+    trace_max_chars: int = 2000,
+    episode_label: str = "",
+    compliance_stdout: bool = False,
+    benchmark_name: str = "flakysleuth",
+    compliance_task_name: str | None = None,
+    history_prune_start_step: int = 12,
+    history_window_turns: int = 4,
+    history_max_chars: int = 50000,
+) -> tuple[float, dict[str, Any]]:
     rewards: list[float] = []
     steps_taken = 0
-    score = 0.0
     success = False
-
-    log_start(task=task_name, env_name=benchmark_name, model=MODEL_NAME)
-
+    episode_task_name = (compliance_task_name or episode_label.split(" ", 1)[0].strip() or "unknown")
+    exploration_reward_total = 0.0
+    final_episode_score = 0.0
+    terminal_meta: dict[str, Any] = {}
+    llm_steps = 0
+    heuristic_steps = 0
+    fallback_reasons: dict[str, int] = {}
+    prune_events = 0
+    read_attempt_counts: dict[str, int] = {}
+    unique_read_files: list[str] = []
+    zero_gain_read_files: set[str] = set()
+    search_patterns: list[str] = []
+    blocked_duplicate_reads = 0
+    no_progress_streak = 0
+    memory_hint = _build_episode_memory(
+        unique_read_files=unique_read_files,
+        zero_gain_read_files=zero_gain_read_files,
+        search_patterns=search_patterns,
+        blocked_duplicate_reads=blocked_duplicate_reads,
+        no_progress_streak=no_progress_streak,
+        max_chars=MEMORY_MAX_CHARS,
+    )
+    if compliance_stdout:
+        _compliance_log_start(episode_task_name, benchmark_name, MODEL_NAME)
     try:
         obs = env.reset()
-        messages: list[dict[str, str]] = [
+
+        initial_prompt = obs_to_prompt(obs, memory_hint=memory_hint, max_steps=env.max_steps)
+        messages = [
             {"role": "system", "content": SYSTEM_PROMPT},
-            {"role": "user", "content": obs_to_prompt(obs, max_steps)},
+            {"role": "user", "content": initial_prompt},
         ]
 
-        for step_idx in range(1, max_steps + 1):
+        if not compliance_stdout:
+            _trace_print(
+                trace_agent,
+                (
+                    f"\n[trace] {episode_label} "
+                    f"task={obs.task_type} repo={obs.repo_url} test={obs.test_name}"
+                ).strip(),
+            )
+        if trace_prompts and not compliance_stdout:
+            _trace_print(
+                trace_agent,
+                "[trace] system prompt:",
+                text=SYSTEM_PROMPT,
+                max_chars=trace_max_chars,
+            )
+            _trace_print(
+                trace_agent,
+                "[trace] initial user prompt:",
+                text=initial_prompt,
+                max_chars=trace_max_chars,
+            )
+
+        for step_idx in range(env.max_steps):
+            messages, prune_info = _prune_messages_window(
+                messages,
+                step_number=step_idx + 1,
+                prune_start_step=history_prune_start_step,
+                window_turns=history_window_turns,
+                max_chars=history_max_chars,
+            )
+            if prune_info:
+                prune_events += 1
+                if trace_agent and not compliance_stdout:
+                    print(
+                        "[trace] context_pruned "
+                        f"reason={prune_info['reason']} "
+                        f"step={prune_info['step']} "
+                        f"messages={prune_info['before_messages']}->{prune_info['after_messages']} "
+                        f"chars={prune_info['before_chars']}->{prune_info['after_chars']}"
+                    )
+
+            action: FlakySleuthAction
+            action_source = "heuristic"
+            llm_meta: dict[str, Any] = {"attempted": False, "raw_output": "", "error": ""}
             try:
-                action = llm_action(messages) or heuristic_action(obs)
-            except Exception:
+                candidate, llm_meta = llm_action(messages)
+                if candidate is not None:
+                    action = candidate
+                    action_source = "llm"
+                else:
+                    action = heuristic_action(obs)
+                    if llm_meta.get("attempted"):
+                        llm_meta["error"] = (
+                            "Model response unavailable, using heuristic fallback."
+                        )
+            except Exception as exc:
+                llm_meta["error"] = str(exc)
                 action = heuristic_action(obs)
 
+            if action.action_type == "read_file":
+                prior_reads = read_attempt_counts.get(action.argument, 0)
+                if prior_reads >= 1:
+                    blocked_duplicate_reads += 1
+                    replacement = FlakySleuthAction(
+                        action_type="search_code",
+                        argument=_duplicate_read_replacement_pattern(obs),
+                    )
+                    if trace_agent and not compliance_stdout:
+                        print(
+                            "[trace] action_overridden "
+                            f"reason=duplicate_read file={action.argument} "
+                            f"replacement={replacement.action_type}"
+                        )
+                    action = replacement
+
+            if action_source == "llm":
+                llm_steps += 1
+            else:
+                heuristic_steps += 1
+                if not API_KEY:
+                    reason_key = "no_api_key"
+                elif llm_meta.get("error"):
+                    reason_key = "llm_error"
+                elif llm_meta.get("attempted"):
+                    reason_key = "empty_or_invalid_response"
+                else:
+                    reason_key = "heuristic_default"
+                fallback_reasons[reason_key] = fallback_reasons.get(reason_key, 0) + 1
+
+            if trace_agent and not compliance_stdout:
+                print(f"[trace] step={step_idx + 1} action_source={action_source}")
+                if llm_meta.get("attempted"):
+                    _trace_print(
+                        True,
+                        "[trace] raw model output:",
+                        text=str(llm_meta.get("raw_output", "")),
+                        max_chars=trace_max_chars,
+                    )
+                if llm_meta.get("error"):
+                    print(f"[trace] llm_error={llm_meta['error']}")
+                print(f"[trace] action={action.model_dump_json()}")
+
             obs, reward, done, info = env.step(action)
-            rewards.append(float(reward or 0.0))
-            steps_taken = step_idx
+            rewards.append(reward)
+            steps_taken = step_idx + 1
+
+            if action.action_type == "read_file":
+                read_attempt_counts[action.argument] = read_attempt_counts.get(action.argument, 0) + 1
+                if action.argument not in unique_read_files:
+                    unique_read_files.append(action.argument)
+                if reward <= 0:
+                    zero_gain_read_files.add(action.argument)
+            elif action.action_type == "search_code":
+                if action.argument not in search_patterns:
+                    search_patterns.append(action.argument)
+
+            if done:
+                no_progress_streak = 0
+            elif reward <= 0:
+                no_progress_streak += 1
+            else:
+                no_progress_streak = 0
+
+            memory_hint = _build_episode_memory(
+                unique_read_files=unique_read_files,
+                zero_gain_read_files=zero_gain_read_files,
+                search_patterns=search_patterns,
+                blocked_duplicate_reads=blocked_duplicate_reads,
+                no_progress_streak=no_progress_streak,
+                max_chars=MEMORY_MAX_CHARS,
+            )
 
             step_error: str | None = None
             if isinstance(info, dict):
-                last_action_error = info.get("last_action_error")
-                if last_action_error:
-                    step_error = str(last_action_error)
-
-            log_step(
-                step=step_idx,
-                action=action.model_dump_json(),
-                reward=float(reward or 0.0),
-                done=bool(done),
-                error=step_error,
-            )
+                raw_err = info.get("last_action_error")
+                if raw_err:
+                    step_error = str(raw_err)
+            if not step_error and obs.tool_output and str(obs.tool_output).startswith("ERROR:"):
+                step_error = str(obs.tool_output)
+
+            if compliance_stdout:
+                _compliance_log_step(
+                    step=steps_taken,
+                    action=action.model_dump_json(),
+                    reward=reward,
+                    done=done,
+                    error=step_error,
+                )
+
+            if trace_agent and not compliance_stdout:
+                print(
+                    f"[trace] step_result reward={reward:.3f} done={done} "
+                    f"step_count={obs.step_count}"
+                )
+                if obs.tool_output:
+                    _trace_print(
+                        True,
+                        "[trace] tool_output:",
+                        text=obs.tool_output,
+                        max_chars=trace_max_chars,
+                    )
 
             if done:
-                score = float(reward or 0.0)
+                # Terminal reward already includes cumulative progress + terminal score.
+                final_episode_score = reward
+                terminal_meta = {
+                    "action_type": action.action_type,
+                    "terminal_score": float(info.get("terminal_score", 0) or 0),
+                    "progress_score": float(info.get("progress_score", 0) or 0),
+                    "explore_sum": exploration_reward_total,
+                    "episode_score": final_episode_score,
+                    "llm_steps": llm_steps,
+                    "heuristic_steps": heuristic_steps,
+                    "fallback_reasons": dict(fallback_reasons),
+                    "context_prune_events": prune_events,
+                    "duplicate_read_blocks": blocked_duplicate_reads,
+                }
+                success = final_episode_score > 0.0
+                if print_terminal:
+                    print(
+                        f"  Terminal: {action.action_type}({action.argument[:40]}) "
+                        f"-> terminal={info.get('terminal_score', 0):.2f} "
+                        f"progress={info.get('progress_score', 0):.2f} "
+                        f"explore_sum={exploration_reward_total:.3f} "
+                        f"episode_score={final_episode_score:.3f}"
+                    )
                 break
 
+            exploration_reward_total += reward
             messages.append({"role": "assistant", "content": action.model_dump_json()})
-            messages.append({"role": "user", "content": obs_to_prompt(obs, max_steps)})
-
-        score = min(max(score, 0.0), 1.0)
-        success = score > 0.0
-    except Exception:
-        score = 0.0
+            next_prompt = obs_to_prompt(obs, memory_hint=memory_hint, max_steps=env.max_steps)
+            messages.append({"role": "user", "content": next_prompt})
+            if trace_agent and trace_prompts and not compliance_stdout:
+                _trace_print(
+                    True,
+                    f"[trace] next user prompt (step={step_idx + 1}):",
+                    text=next_prompt,
+                    max_chars=trace_max_chars,
+                )
+    except Exception as exc:
+        terminal_meta["error"] = str(exc)
         success = False
+        if not compliance_stdout:
+            raise
     finally:
-        try:
-            env.close()
-        except Exception:
-            pass
-        log_end(success=success, steps=steps_taken, score=score, rewards=rewards)
+        if compliance_stdout:
+            try:
+                env.close()
+            except Exception:
+                pass
+            _compliance_log_end(
+                success=success,
+                steps=steps_taken,
+                score=min(max(final_episode_score, 0.0), 1.0),
+                rewards=rewards,
+            )
+
+    return final_episode_score, terminal_meta
 
-    return score
+
+def _looks_like_placeholder_dataset(dataset_path: str) -> bool:
+    path = Path(dataset_path)
+    if not path.exists():
+        return False
+    try:
+        text = path.read_text(encoding="utf-8", errors="replace")
+    except Exception:
+        return False
+    return "fixture://" in text
 
 
 def _parse_args() -> argparse.Namespace:
-    parser = argparse.ArgumentParser(description="Run FlakySleuth compliance inference.")
+    parser = argparse.ArgumentParser(description="Run FlakySleuth baseline inference.")
     parser.add_argument(
         "--dataset-path",
         default="dataset/py_tasks.csv",
@@ -264,34 +642,194 @@ def _parse_args() -> argparse.Namespace:
         default=MAX_STEPS,
         help="Max steps per episode.",
     )
+    parser.add_argument(
+        "--no-progress",
+        action="store_true",
+        help="Disable progress bars and print classic per-episode logs.",
+    )
+    parser.add_argument(
+        "--trace-agent",
+        action="store_true",
+        help=(
+            "Print detailed agent trace: model output, chosen action/tool call, and "
+            "step results for every episode."
+        ),
+    )
+    parser.add_argument(
+        "--trace-prompts",
+        action="store_true",
+        help="When tracing, also print full prompts sent to the model.",
+    )
+    parser.add_argument(
+        "--trace-max-chars",
+        type=int,
+        default=2500,
+        help="Max chars per traced text block (prompt/model output/tool output).",
+    )
+    parser.add_argument(
+        "--compliance-stdout",
+        dest="compliance_stdout",
+        action="store_true",
+        help=(
+            "Emit strict compliance logs to stdout using only [START]/[STEP]/[END] lines "
+            "for each episode."
+        ),
+    )
+    parser.add_argument(
+        "--no-compliance-stdout",
+        dest="compliance_stdout",
+        action="store_false",
+        help="Disable strict compliance logs and print baseline summaries/progress.",
+    )
     parser.add_argument(
         "--benchmark-name",
-        default=BENCHMARK_NAME,
-        help="Benchmark label for [START] lines.",
+        default="flakysleuth",
+        help="Benchmark name used in [START] lines when --compliance-stdout is enabled.",
+    )
+    parser.add_argument(
+        "--history-prune-start-step",
+        type=int,
+        default=12,
+        help="Start pruning conversation history only from this step onward.",
     )
+    parser.add_argument(
+        "--history-window-turns",
+        type=int,
+        default=4,
+        help="When pruning is active, keep this many recent assistant/user turns.",
+    )
+    parser.add_argument(
+        "--history-max-chars",
+        type=int,
+        default=50000,
+        help="Approx max chars for messages before forced pruning by size.",
+    )
+    parser.set_defaults(compliance_stdout=True)
     return parser.parse_args()
 
 
 def main() -> None:
+    run_start = time.perf_counter()
     args = _parse_args()
     env = FlakySleuthEnv(dataset_path=args.dataset_path, max_steps=args.max_steps)
-
     allowed_task_types = {"classify", "root_cause", "fix_proposal"}
     task_types = [t.strip() for t in args.task_types.split(",") if t.strip()]
+    invalid = [t for t in task_types if t not in allowed_task_types]
+    if invalid:
+        raise ValueError(
+            f"Invalid task type(s): {invalid}. "
+            "Valid values: classify,root_cause,fix_proposal."
+        )
     if not task_types:
-        return
+        raise ValueError(
+            "No task types selected. Pass --task-types with at least one value."
+        )
+    results: dict[str, list[float]] = defaultdict(list)
+
+    if _looks_like_placeholder_dataset(args.dataset_path) and not args.compliance_stdout:
+        print(
+            "[warning] dataset appears to contain fixture rows (fixture://...). "
+            "Build real dataset from py-data.csv for real evaluation."
+        )
+
+    use_progress = (
+        (tqdm is not None)
+        and (not args.no_progress)
+        and (not args.compliance_stdout)
+        and os.isatty(1)
+    )
+    if args.trace_agent and use_progress and not args.compliance_stdout:
+        print(
+            "[info] --trace-agent enabled, disabling progress bars for readable trace logs."
+        )
+        use_progress = False
+    overall_bar = None
+    if use_progress:
+        overall_bar = tqdm(
+            total=len(task_types) * args.episodes_per_task,
+            desc="All tasks",
+            unit="ep",
+            dynamic_ncols=True,
+        )
 
     for task_type in task_types:
-        if task_type not in allowed_task_types:
-            continue
+        task_start = time.perf_counter()
+        if not args.compliance_stdout:
+            print(f"\n-- Task type: {task_type} --")
         env.loader.force_task_type(task_type)
-        for _ in range(args.episodes_per_task):
-            run_episode(
+        task_bar = None
+        if use_progress:
+            task_bar = tqdm(
+                total=args.episodes_per_task,
+                desc=f"{task_type}",
+                unit="ep",
+                leave=False,
+                dynamic_ncols=True,
+            )
+        for episode in range(args.episodes_per_task):
+            score, meta = run_episode(
                 env,
-                task_name=task_type,
+                print_terminal=(not use_progress) and (not args.compliance_stdout),
+                trace_agent=args.trace_agent,
+                trace_prompts=args.trace_prompts,
+                trace_max_chars=args.trace_max_chars,
+                episode_label=f"{task_type} ep={episode + 1}/{args.episodes_per_task}",
+                compliance_stdout=args.compliance_stdout,
                 benchmark_name=args.benchmark_name,
-                max_steps=args.max_steps,
+                compliance_task_name=task_type,
+                history_prune_start_step=args.history_prune_start_step,
+                history_window_turns=args.history_window_turns,
+                history_max_chars=args.history_max_chars,
             )
+            results[task_type].append(score)
+            if use_progress and task_bar is not None:
+                task_bar.update(1)
+                task_avg = sum(results[task_type]) / len(results[task_type])
+                task_bar.set_postfix(
+                    score=f"{score:.3f}",
+                    avg=f"{task_avg:.3f}",
+                    term=f"{meta.get('terminal_score', 0):.2f}",
+                )
+                if overall_bar is not None:
+                    overall_bar.update(1)
+                    all_scores = [s for values in results.values() for s in values]
+                    overall_avg = sum(all_scores) / len(all_scores)
+                    overall_bar.set_postfix(task=task_type, avg=f"{overall_avg:.3f}")
+            elif not args.compliance_stdout:
+                print(f"  Episode {episode + 1}: {score:.3f}")
+        if task_bar is not None:
+            task_bar.close()
+        task_elapsed = time.perf_counter() - task_start
+        if not args.compliance_stdout:
+            avg_task = sum(results[task_type]) / max(1, len(results[task_type]))
+            print(
+                f"  [time] task={task_type} elapsed={_format_duration(task_elapsed)} "
+                f"avg_ep={task_elapsed / max(1, args.episodes_per_task):.2f}s "
+                f"avg_score={avg_task:.3f}"
+            )
+
+    if overall_bar is not None:
+        overall_bar.close()
+
+    if args.compliance_stdout:
+        return
+
+    total_elapsed = time.perf_counter() - run_start
+    print("\n== BASELINE RESULTS ==")
+    all_scores: list[float] = []
+    for task_type in task_types:
+        scores = results[task_type]
+        avg = sum(scores) / len(scores)
+        all_scores.extend(scores)
+        print(f"  {task_type:12s} avg={avg:.3f} scores={[round(s, 3) for s in scores]}")
+
+    overall = sum(all_scores) / len(all_scores)
+    print(f"  {'OVERALL':12s} avg={overall:.3f}")
+    print(
+        f"  {'RUNTIME':12s} total={_format_duration(total_elapsed)} "
+        f"episodes={len(all_scores)} "
+        f"avg_ep={(total_elapsed / max(1, len(all_scores))):.2f}s"
+    )
 
 
 if __name__ == "__main__":

inference_debug.py

@@ -75,6 +75,7 @@ MODEL_NAME = os.environ.get("MODEL_NAME", DEFAULT_MODEL)
 # Keep a conservative default to stay under common hackathon runtime limits.
 EPISODES_PER_TASK = 2
 MAX_STEPS = 20
+MEMORY_MAX_CHARS = 900
 
 client = OpenAI(api_key=API_KEY, base_url=API_BASE_URL)
 
@@ -140,7 +141,7 @@ def _compliance_log_end(success: bool, steps: int, score: float, rewards: list[f
     )
 
 
-def obs_to_prompt(obs: FlakySleuthObservation) -> str:
+def obs_to_prompt(obs: FlakySleuthObservation, *, memory_hint: str | None = None) -> str:
     tree_preview = "\n".join(obs.file_tree[:40])
     return f"""TASK: {obs.task_description}
 
@@ -159,6 +160,9 @@ Repository file tree:
 Last tool output:
 {obs.tool_output or "(No action taken yet)"}
 
+Episode memory:
+{memory_hint or "(No memory yet.)"}
+
 Return only JSON action."""
 
 
@@ -246,6 +250,85 @@ def _format_duration(seconds: float) -> str:
     return f"{mins:02d}m {secs:02d}s"
 
 
+def _build_episode_memory(
+    *,
+    unique_read_files: list[str],
+    zero_gain_read_files: set[str],
+    search_patterns: list[str],
+    blocked_duplicate_reads: int,
+    no_progress_streak: int,
+    max_chars: int,
+) -> str:
+    read_tail = ", ".join(unique_read_files[-8:]) if unique_read_files else "none"
+    zero_tail = ", ".join(sorted(zero_gain_read_files)[-8:]) if zero_gain_read_files else "none"
+    search_tail = ", ".join(search_patterns[-6:]) if search_patterns else "none"
+    loop_warning = (
+        "WARNING: Possible loop detected. Stop repeating similar exploration. "
+        "Switch strategy or take a terminal action."
+        if no_progress_streak >= 3 or blocked_duplicate_reads >= 2
+        else "Status: exploration progress appears normal."
+    )
+    memory = (
+        f"Read files (recent): {read_tail}\n"
+        f"Zero-gain read files: {zero_tail}\n"
+        f"Search patterns (recent): {search_tail}\n"
+        f"Blocked duplicate reads: {blocked_duplicate_reads}\n"
+        f"No-progress streak: {no_progress_streak}\n"
+        f"{loop_warning}\n"
+        "Guidance: Avoid rereading zero-gain files unless there is new evidence. "
+        "Prefer targeted search_code or terminal action when confidence is enough."
+    )
+    return _clip_text(memory, max_chars=max_chars)
+
+
+def _duplicate_read_replacement_pattern(obs: FlakySleuthObservation) -> str:
+    test_hint = obs.test_name.split("::")[-1] if obs.test_name else "test"
+    return (
+        f"{test_hint}|random|sleep|time|timeout|retry|asyncio|thread|"
+        "fixture|global|shared|mock|patch"
+    )
+
+
+def _messages_char_count(messages: list[dict[str, str]]) -> int:
+    # Lightweight size heuristic to avoid unbounded context growth.
+    return sum(len(str(msg.get("content", ""))) + 32 for msg in messages)
+
+
+def _prune_messages_window(
+    messages: list[dict[str, str]],
+    *,
+    step_number: int,
+    prune_start_step: int,
+    window_turns: int,
+    max_chars: int,
+) -> tuple[list[dict[str, str]], dict[str, Any] | None]:
+    if len(messages) <= 2:
+        return messages, None
+
+    current_chars = _messages_char_count(messages)
+    exceeds_step_threshold = step_number >= prune_start_step
+    exceeds_char_budget = current_chars > max_chars
+    if not exceeds_step_threshold and not exceeds_char_budget:
+        return messages, None
+
+    base = messages[:2]  # system + initial prompt
+    tail = messages[2:]
+    keep_tail_items = max(2, window_turns * 2)
+    if len(tail) > keep_tail_items:
+        tail = tail[-keep_tail_items:]
+    pruned = base + tail
+
+    reason = "step_threshold" if exceeds_step_threshold else "char_budget"
+    return pruned, {
+        "reason": reason,
+        "before_messages": len(messages),
+        "after_messages": len(pruned),
+        "before_chars": current_chars,
+        "after_chars": _messages_char_count(pruned),
+        "step": step_number,
+    }
+
+
 def run_episode(
     env: FlakySleuthEnv,
     *,
@@ -257,6 +340,9 @@ def run_episode(
     compliance_stdout: bool = False,
     benchmark_name: str = "flakysleuth",
     compliance_task_name: str | None = None,
+    history_prune_start_step: int = 12,
+    history_window_turns: int = 4,
+    history_max_chars: int = 50000,
 ) -> tuple[float, dict[str, Any]]:
     rewards: list[float] = []
     steps_taken = 0
@@ -265,12 +351,30 @@ def run_episode(
     exploration_reward_total = 0.0
     final_episode_score = 0.0
     terminal_meta: dict[str, Any] = {}
+    llm_steps = 0
+    heuristic_steps = 0
+    fallback_reasons: dict[str, int] = {}
+    prune_events = 0
+    read_attempt_counts: dict[str, int] = {}
+    unique_read_files: list[str] = []
+    zero_gain_read_files: set[str] = set()
+    search_patterns: list[str] = []
+    blocked_duplicate_reads = 0
+    no_progress_streak = 0
+    memory_hint = _build_episode_memory(
+        unique_read_files=unique_read_files,
+        zero_gain_read_files=zero_gain_read_files,
+        search_patterns=search_patterns,
+        blocked_duplicate_reads=blocked_duplicate_reads,
+        no_progress_streak=no_progress_streak,
+        max_chars=MEMORY_MAX_CHARS,
+    )
     if compliance_stdout:
         _compliance_log_start(episode_task_name, benchmark_name, MODEL_NAME)
     try:
         obs = env.reset()
 
-        initial_prompt = obs_to_prompt(obs)
+        initial_prompt = obs_to_prompt(obs, memory_hint=memory_hint)
         messages = [
             {"role": "system", "content": SYSTEM_PROMPT},
             {"role": "user", "content": initial_prompt},
@@ -299,6 +403,24 @@ def run_episode(
             )
 
         for step_idx in range(MAX_STEPS):
+            messages, prune_info = _prune_messages_window(
+                messages,
+                step_number=step_idx + 1,
+                prune_start_step=history_prune_start_step,
+                window_turns=history_window_turns,
+                max_chars=history_max_chars,
+            )
+            if prune_info:
+                prune_events += 1
+                if trace_agent and not compliance_stdout:
+                    print(
+                        "[trace] context_pruned "
+                        f"reason={prune_info['reason']} "
+                        f"step={prune_info['step']} "
+                        f"messages={prune_info['before_messages']}->{prune_info['after_messages']} "
+                        f"chars={prune_info['before_chars']}->{prune_info['after_chars']}"
+                    )
+
             action: FlakySleuthAction
             action_source = "heuristic"
             llm_meta: dict[str, Any] = {"attempted": False, "raw_output": "", "error": ""}
@@ -317,6 +439,36 @@ def run_episode(
                 llm_meta["error"] = str(exc)
                 action = heuristic_action(obs)
 
+            if action.action_type == "read_file":
+                prior_reads = read_attempt_counts.get(action.argument, 0)
+                if prior_reads >= 1:
+                    blocked_duplicate_reads += 1
+                    replacement = FlakySleuthAction(
+                        action_type="search_code",
+                        argument=_duplicate_read_replacement_pattern(obs),
+                    )
+                    if trace_agent and not compliance_stdout:
+                        print(
+                            "[trace] action_overridden "
+                            f"reason=duplicate_read file={action.argument} "
+                            f"replacement={replacement.action_type}"
+                        )
+                    action = replacement
+
+            if action_source == "llm":
+                llm_steps += 1
+            else:
+                heuristic_steps += 1
+                if not API_KEY:
+                    reason_key = "no_api_key"
+                elif llm_meta.get("error"):
+                    reason_key = "llm_error"
+                elif llm_meta.get("attempted"):
+                    reason_key = "empty_or_invalid_response"
+                else:
+                    reason_key = "heuristic_default"
+                fallback_reasons[reason_key] = fallback_reasons.get(reason_key, 0) + 1
+
             if trace_agent and not compliance_stdout:
                 print(f"[trace] step={step_idx + 1} action_source={action_source}")
                 if llm_meta.get("attempted"):
@@ -334,6 +486,32 @@ def run_episode(
             rewards.append(reward)
             steps_taken = step_idx + 1
 
+            if action.action_type == "read_file":
+                read_attempt_counts[action.argument] = read_attempt_counts.get(action.argument, 0) + 1
+                if action.argument not in unique_read_files:
+                    unique_read_files.append(action.argument)
+                if reward <= 0:
+                    zero_gain_read_files.add(action.argument)
+            elif action.action_type == "search_code":
+                if action.argument not in search_patterns:
+                    search_patterns.append(action.argument)
+
+            if done:
+                no_progress_streak = 0
+            elif reward <= 0:
+                no_progress_streak += 1
+            else:
+                no_progress_streak = 0
+
+            memory_hint = _build_episode_memory(
+                unique_read_files=unique_read_files,
+                zero_gain_read_files=zero_gain_read_files,
+                search_patterns=search_patterns,
+                blocked_duplicate_reads=blocked_duplicate_reads,
+                no_progress_streak=no_progress_streak,
+                max_chars=MEMORY_MAX_CHARS,
+            )
+
             step_error: str | None = None
             if isinstance(info, dict):
                 raw_err = info.get("last_action_error")
@@ -373,6 +551,11 @@ def run_episode(
                     "progress_score": float(info.get("progress_score", 0) or 0),
                     "explore_sum": exploration_reward_total,
                     "episode_score": final_episode_score,
+                    "llm_steps": llm_steps,
+                    "heuristic_steps": heuristic_steps,
+                    "fallback_reasons": dict(fallback_reasons),
+                    "context_prune_events": prune_events,
+                    "duplicate_read_blocks": blocked_duplicate_reads,
                 }
                 success = final_episode_score > 0.0
                 if print_terminal:
@@ -387,7 +570,7 @@ def run_episode(
 
             exploration_reward_total += reward
             messages.append({"role": "assistant", "content": action.model_dump_json()})
-            next_prompt = obs_to_prompt(obs)
+            next_prompt = obs_to_prompt(obs, memory_hint=memory_hint)
             messages.append({"role": "user", "content": next_prompt})
             if trace_agent and trace_prompts and not compliance_stdout:
                 _trace_print(
@@ -483,6 +666,24 @@ def _parse_args() -> argparse.Namespace:
         default="flakysleuth",
         help="Benchmark name used in [START] lines when --compliance-stdout is enabled.",
     )
+    parser.add_argument(
+        "--history-prune-start-step",
+        type=int,
+        default=12,
+        help="Start pruning conversation history only from this step onward.",
+    )
+    parser.add_argument(
+        "--history-window-turns",
+        type=int,
+        default=4,
+        help="When pruning is active, keep this many recent assistant/user turns.",
+    )
+    parser.add_argument(
+        "--history-max-chars",
+        type=int,
+        default=50000,
+        help="Approx max chars for messages before forced pruning by size.",
+    )
     return parser.parse_args()
 
 
@@ -550,6 +751,9 @@ def main() -> None:
                 compliance_stdout=args.compliance_stdout,
                 benchmark_name=args.benchmark_name,
                 compliance_task_name=task_type,
+                history_prune_start_step=args.history_prune_start_step,
+                history_window_turns=args.history_window_turns,
+                history_max_chars=args.history_max_chars,
             )
             results[task_type].append(score)
             if use_progress and task_bar is not None:

server/app.py

@@ -28,7 +28,7 @@ class FlakySleuthState(BaseModel):
 
 class InferenceRunRequest(BaseModel):
     dataset_path: str = Field(default="dataset/py_tasks.csv")
-    episodes_per_task: int = Field(default=1, ge=1, le=50)
+    episodes_per_task: int = Field(default=1, ge=1, le=100)
     task_types: str = Field(default="classify,root_cause,fix_proposal")
     max_steps: int = Field(default=20, ge=1, le=100)
     benchmark_name: str = Field(default="flakysleuth")

server/inference_runner.py

@@ -48,8 +48,8 @@ class InferenceRunner:
 
         if not dataset_rel:
             raise ValueError("dataset_path must not be empty.")
-        if episodes < 1 or episodes > 50:
-            raise ValueError("episodes_per_task must be between 1 and 50.")
+        if episodes < 1 or episodes > 100:
+            raise ValueError("episodes_per_task must be between 1 and 100.")
         if max_steps < 1 or max_steps > 100:
             raise ValueError("max_steps must be between 1 and 100.")
         if not task_types:

server/ui.py

@@ -8,7 +8,7 @@ def render_home_page() -> str:
 <head>
   <meta charset="utf-8" />
   <meta name="viewport" content="width=device-width, initial-scale=1" />
-  <title>FlakySleuth Run Studio</title>
+  <title>FlakyGym Control Center</title>
   <link rel="preconnect" href="https://fonts.googleapis.com" />
   <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
   <link href="https://fonts.googleapis.com/css2?family=Space+Grotesk:wght@500;600;700&family=IBM+Plex+Mono:wght@400;500&display=swap" rel="stylesheet" />
@@ -123,6 +123,60 @@ def render_home_page() -> str:
       letter-spacing: -0.01em;
     }
 
+    .brief-grid {
+      display: grid;
+      grid-template-columns: repeat(2, minmax(0, 1fr));
+      gap: 12px;
+    }
+
+    .brief-card {
+      border: 1px solid rgba(15, 139, 99, 0.22);
+      border-radius: 12px;
+      background: rgba(255, 255, 255, 0.84);
+      padding: 10px;
+      display: grid;
+      gap: 8px;
+    }
+
+    .brief-card h3 {
+      margin: 0;
+      font-size: 0.95rem;
+      letter-spacing: -0.01em;
+    }
+
+    .brief-card p {
+      margin: 0;
+      font-size: 12px;
+      color: #36564a;
+      line-height: 1.45;
+    }
+
+    .brief-list {
+      margin: 0;
+      padding-left: 16px;
+      font-size: 12px;
+      color: #2f4f43;
+      line-height: 1.45;
+      display: grid;
+      gap: 5px;
+    }
+
+    .header-chips {
+      display: flex;
+      flex-wrap: wrap;
+      gap: 6px;
+    }
+
+    .header-chips code {
+      background: #eef8f3;
+      border: 1px solid rgba(15, 139, 99, 0.26);
+      border-radius: 999px;
+      padding: 4px 8px;
+      font: 500 11px/1.1 var(--mono);
+      color: #20443a;
+      white-space: nowrap;
+    }
+
     .form-grid {
       display: grid;
       gap: 12px;
@@ -230,6 +284,61 @@ def render_home_page() -> str:
       line-height: 1.4;
     }
 
+    .slider-wrap {
+      display: grid;
+      gap: 8px;
+    }
+
+    .slider-value-row {
+      display: flex;
+      justify-content: space-between;
+      gap: 10px;
+      font: 500 12px/1.3 var(--mono);
+      color: #3c5950;
+    }
+
+    input[type="range"] {
+      width: 100%;
+      padding: 0;
+      accent-color: var(--accent);
+      cursor: pointer;
+    }
+
+    .eta-box {
+      border: 1px solid rgba(15, 139, 99, 0.22);
+      border-radius: 12px;
+      background: rgba(255, 255, 255, 0.85);
+      padding: 10px;
+      display: grid;
+      gap: 6px;
+    }
+
+    .eta-value {
+      font: 700 17px/1 var(--display);
+      letter-spacing: -0.01em;
+    }
+
+    .eta-warning {
+      font: 500 12px/1.45 var(--mono);
+      border-radius: 8px;
+      padding: 6px 8px;
+      display: none;
+    }
+
+    .eta-warning.warn {
+      display: block;
+      color: #752f1b;
+      background: #fff0d7;
+      border: 1px solid rgba(213, 114, 65, 0.45);
+    }
+
+    .eta-warning.ok {
+      display: block;
+      color: #20443a;
+      background: #e8f7ef;
+      border: 1px solid rgba(15, 139, 99, 0.28);
+    }
+
     .actions {
       margin-top: 6px;
       display: flex;
@@ -373,6 +482,10 @@ def render_home_page() -> str:
         grid-template-columns: 1fr;
       }
 
+      .brief-grid {
+        grid-template-columns: 1fr;
+      }
+
       .field.span-2 {
         grid-column: span 1;
       }
@@ -386,12 +499,37 @@ def render_home_page() -> str:
 <body>
   <main class="shell">
     <section class="hero">
-      <span class="eyebrow"><span class="dot"></span>FlakySleuth Space</span>
-      <h1>Run Inference From The Browser</h1>
-      <p>This evaluator console runs benchmark episodes for <strong>classification</strong>, <strong>root-cause identification</strong>, and <strong>fix proposal</strong>. Use it to review logs, score trends, and reproducible run settings while judging submission quality.</p>
+      <span class="eyebrow"><span class="dot"></span>FlakyGym Space</span>
+      <h1>FlakyGym Control Center</h1>
+      <p>This console runs flaky-test benchmark episodes and streams live logs. Use it to configure runs, estimate runtime, and review grader outcomes quickly.</p>
     </section>
 
     <section class="panel-grid">
+      <div class="panel">
+        <h2>Quick Brief: Dataset + Graders</h2>
+        <div class="brief-grid">
+          <div class="brief-card">
+            <h3>Dataset: <code>dataset/py_tasks.csv</code></h3>
+            <p>Each row is one flaky-test investigation task created from <code>py-data.csv</code> (repo + SHA + target test + labels + optional known fix diff).</p>
+            <p class="field-note">Headers:</p>
+            <div class="header-chips">
+              <code>repo_url</code><code>sha</code><code>test_name</code><code>test_file</code>
+              <code>category</code><code>label</code><code>status</code><code>pr_link</code>
+              <code>task_types</code><code>test_code</code><code>known_fix_diff</code>
+            </div>
+          </div>
+
+          <div class="brief-card">
+            <h3>3 Graders (short)</h3>
+            <ul class="brief-list">
+              <li><strong>Task 1 (`classify`):</strong> exact-match flaky vs stable.</li>
+              <li><strong>Task 2 (`root_cause`):</strong> category similarity matrix (partial credit allowed).</li>
+              <li><strong>Task 3 (`fix_proposal`):</strong> weighted score from pattern match, patch applicability, and LLM judge.</li>
+            </ul>
+          </div>
+        </div>
+      </div>
+
       <div class="panel">
         <h2>Run Configuration</h2>
         <form id="run-form" class="form-grid">
@@ -402,12 +540,24 @@ def render_home_page() -> str:
 
           <div class="field">
             <label for="episodes_per_task">Episodes Per Task</label>
-            <input id="episodes_per_task" name="episodes_per_task" type="number" min="1" max="50" value="1" />
+            <div class="slider-wrap">
+              <input id="episodes_per_task" name="episodes_per_task" type="range" min="1" max="100" step="1" value="1" />
+              <div class="slider-value-row">
+                <span><strong id="episodes_per_task_value">1</strong> episode(s)</span>
+                <span>1-100</span>
+              </div>
+            </div>
           </div>
 
           <div class="field">
             <label for="max_steps">Max Steps</label>
-            <input id="max_steps" name="max_steps" type="number" min="1" max="100" value="20" />
+            <div class="slider-wrap">
+              <input id="max_steps" name="max_steps" type="range" min="1" max="100" step="1" value="20" />
+              <div class="slider-value-row">
+                <span><strong id="max_steps_value">20</strong> step(s)</span>
+                <span>1-100</span>
+              </div>
+            </div>
           </div>
 
           <div class="field span-2">
@@ -430,6 +580,15 @@ def render_home_page() -> str:
             <input id="benchmark_name" name="benchmark_name" value="flakysleuth" />
           </div>
 
+          <div class="field span-2">
+            <label>Runtime ETA</label>
+            <div class="eta-box">
+              <div class="eta-value" id="eta-value">~09m 00s</div>
+              <p class="field-note" id="eta-detail">3 task(s) × 1 episode(s) × 180s/episode</p>
+              <div class="eta-warning" id="eta-warning"></div>
+            </div>
+          </div>
+
           <div class="field span-2">
             <label for="api_base_url">API Base URL (optional)</label>
             <input id="api_base_url" name="api_base_url" placeholder="https://api.openai.com/v1 or provider endpoint" />
@@ -494,6 +653,13 @@ def render_home_page() -> str:
     const taskChipsEl = document.getElementById("task-chips");
     const taskSelectEl = document.getElementById("task-type-select");
     const taskAddButton = document.getElementById("btn-add-task");
+    const episodesInput = document.getElementById("episodes_per_task");
+    const episodesValueEl = document.getElementById("episodes_per_task_value");
+    const maxStepsInput = document.getElementById("max_steps");
+    const maxStepsValueEl = document.getElementById("max_steps_value");
+    const etaValueEl = document.getElementById("eta-value");
+    const etaDetailEl = document.getElementById("eta-detail");
+    const etaWarningEl = document.getElementById("eta-warning");
 
     const TASK_TYPE_ORDER = ["classify", "root_cause", "fix_proposal"];
     const TASK_TYPE_LABELS = {
@@ -501,6 +667,35 @@ def render_home_page() -> str:
       root_cause: "Root Cause",
       fix_proposal: "Fix Proposal",
     };
+    const ETA_SECONDS_PER_EPISODE = 180;
+    const HACKATHON_LIMIT_SECONDS = 20 * 60;
+
+    function clampInt(raw, min, max, fallback) {
+      const num = Number(raw);
+      if (!Number.isFinite(num)) return fallback;
+      return Math.max(min, Math.min(max, Math.trunc(num)));
+    }
+
+    function formatDuration(totalSeconds) {
+      const seconds = Math.max(0, Math.round(totalSeconds));
+      const mins = Math.floor(seconds / 60);
+      const secs = seconds % 60;
+      if (mins >= 60) {
+        const hrs = Math.floor(mins / 60);
+        const remMins = mins % 60;
+        return `${hrs}h ${String(remMins).padStart(2, "0")}m ${String(secs).padStart(2, "0")}s`;
+      }
+      return `${String(mins).padStart(2, "0")}m ${String(secs).padStart(2, "0")}s`;
+    }
+
+    function refreshSliderValues() {
+      const episodes = clampInt(episodesInput.value, 1, 100, 1);
+      const maxSteps = clampInt(maxStepsInput.value, 1, 100, 20);
+      episodesInput.value = String(episodes);
+      maxStepsInput.value = String(maxSteps);
+      episodesValueEl.textContent = String(episodes);
+      maxStepsValueEl.textContent = String(maxSteps);
+    }
 
     function parseTaskTypes(raw) {
       const tokens = String(raw || "")
@@ -580,6 +775,7 @@ def render_home_page() -> str:
       taskInput.value = selectedTaskTypes.join(",");
       renderTaskChips();
       renderTaskSelect();
+      updateRuntimeEstimate();
     }
 
     function addSelectedTaskType() {
@@ -591,12 +787,58 @@ def render_home_page() -> str:
       syncTaskTypes();
     }
 
+    function updateRuntimeEstimate() {
+      const episodes = clampInt(episodesInput.value, 1, 100, 1);
+      const maxSteps = clampInt(maxStepsInput.value, 1, 100, 20);
+      const taskCount = selectedTaskTypes.length;
+      const totalEpisodes = taskCount * episodes;
+      const etaSeconds = totalEpisodes * ETA_SECONDS_PER_EPISODE;
+
+      etaValueEl.textContent = `~${formatDuration(etaSeconds)}`;
+      etaDetailEl.textContent =
+        `${taskCount} task(s) × ${episodes} episode(s) × ${ETA_SECONDS_PER_EPISODE}s/episode`;
+
+      const notes = [];
+      if (episodes > 2) {
+        notes.push("Recommended: keep episodes per task at 1-2 for faster hackathon runs.");
+      }
+      if (etaSeconds > HACKATHON_LIMIT_SECONDS) {
+        notes.push("Warning: ETA exceeds 20 minutes, which may violate hackathon runtime guidance.");
+      }
+      if (maxSteps > 20) {
+        notes.push("Higher max steps can increase runtime beyond this ETA estimate.");
+      }
+      if (taskCount === 0) {
+        notes.push("Add at least one task chip to run inference.");
+      }
+
+      etaWarningEl.classList.remove("warn", "ok");
+      if (!notes.length) {
+        etaWarningEl.textContent = "Runtime looks within limits for a quick benchmark run.";
+        etaWarningEl.classList.add("ok");
+        return;
+      }
+
+      etaWarningEl.textContent = notes.join(" ");
+      if (etaSeconds > HACKATHON_LIMIT_SECONDS || episodes > 2 || taskCount === 0) {
+        etaWarningEl.classList.add("warn");
+      } else {
+        etaWarningEl.classList.add("ok");
+      }
+    }
+
     function readFormPayload() {
+      const episodes = clampInt(episodesInput.value, 1, 100, 1);
+      const maxSteps = clampInt(maxStepsInput.value, 1, 100, 20);
+      episodesInput.value = String(episodes);
+      maxStepsInput.value = String(maxSteps);
+      refreshSliderValues();
+      updateRuntimeEstimate();
       return {
         dataset_path: form.dataset_path.value.trim(),
-        episodes_per_task: Number(form.episodes_per_task.value),
+        episodes_per_task: episodes,
         task_types: form.task_types.value.trim(),
-        max_steps: Number(form.max_steps.value),
+        max_steps: maxSteps,
         benchmark_name: form.benchmark_name.value.trim(),
         api_base_url: form.api_base_url.value.trim() || null,
         model_name: form.model_name.value.trim() || null,
@@ -707,8 +949,18 @@ def render_home_page() -> str:
         addSelectedTaskType();
       }
     });
+    episodesInput.addEventListener("input", () => {
+      refreshSliderValues();
+      updateRuntimeEstimate();
+    });
+    maxStepsInput.addEventListener("input", () => {
+      refreshSliderValues();
+      updateRuntimeEstimate();
+    });
 
+    refreshSliderValues();
     syncTaskTypes();
+    updateRuntimeEstimate();
 
     fetchStatus();
     window.setInterval(fetchStatus, 2200);