Task 3 Implemented

README.md

@@ -1,9 +1,9 @@
 # Smart Contract Audit RL Environment
 
 > **OpenEnv-compliant reinforcement learning environment for smart contract security analysis.**
-> Train and evaluate agents on real-world Solidity audit tasks — the same work professional auditors do every day.
+> Three fully implemented tasks covering the core workflow of a professional Solidity auditor.
 
-[![OpenEnv Spec](https://img.shields.io/badge/OpenEnv-1.1-blue)](openenv.yaml)
+[![OpenEnv Spec](https://img.shields.io/badge/OpenEnv-1.2-blue)](openenv.yaml)
 [![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-brightgreen)](https://python.org)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow)](LICENSE)
 
@@ -11,58 +11,57 @@
 
 ## Motivation
 
-Smart contract auditing is a $500M+ industry where human auditors painstakingly review Solidity code for security flaws and formally specify function properties. This environment lets agents practice exactly that workflow — exploring contract code through targeted queries and submitting findings — providing a rigorous, real-world benchmark for code-reasoning agents.
-
-Data is sourced from **Certora-audited DeFi projects**, giving agents contracts with the same vulnerability patterns found in production exploits.
+Smart contract auditing is a $500M+ industry where human experts identify security flaws, write formal properties, and check whether code satisfies those properties. This environment lets agents practise exactly those three tasks using real Solidity contracts from Certora-audited DeFi projects.
 
 ---
 
-## Tasks
+## Tasks at a Glance
 
-| # | Name | Difficulty | Status | Description |
-|---|------|------------|--------|-------------|
-| 1 | Targeted Vulnerability Detection | Medium | ✅ Active | Find the vulnerable function and name the vulnerability type |
+| # | Name | Difficulty | Status | One-line description |
+|---|------|-----------|--------|---------------------|
+| 1 | Targeted Vulnerability Detection | Medium | ✅ Active | Find which function is vulnerable and name the vulnerability |
 | 2 | Property Discovery | Hard | ✅ Active | Write the natural-language postcondition for a given function |
-| 3 | Rule Checker | Easy | ⏳ Placeholder | Identify which function violates a given property |
+| 3 | Rule Checker | Easy | ✅ Active | Identify which function violates a given property |
 
 ---
 
 ## Task 1 — Targeted Vulnerability Detection *(Medium)*
 
-**Setup:** Agent is shown a Solidity contract (4–6 functions). One function contains a critical vulnerability.
+**Setup:** A Solidity contract (4–6 functions) is shown. One function contains a critical vulnerability.
 
-**Objective:** Identify the vulnerable function and describe its vulnerability type in 2–3 words.
+**Objective:** Name the vulnerable function and describe its vulnerability type in 2–3 words.
 
 ### Actions
 
 | Action | Params | Reward |
 |--------|--------|--------|
 | `list_functions` | — | −0.05 |
-| `get_function_code` | `function_name` | +0.05 (target) / −0.10 (other) |
-| `get_function_summary` | `function_name` | +0.03 (target) / −0.05 (other) |
+| `get_function_code` | `function_name` | +0.05 if target / −0.10 if other |
+| `get_function_summary` | `function_name` | +0.03 if target / −0.05 if other |
 | `get_file_metadata` | — | −0.04 |
 | `get_state_variable` | `variable_name` (opt.) | −0.05 |
 | `get_call_graph` | — | −0.08 |
-| `submit` | `function_name`, `vulnerability_type` | **+5.0** / +1.0 / −1.5 |
+| `submit` | `function_name`, `vulnerability_type` | **+5.0 / +1.0 / −1.5** |
+
+Repeated queries: **−0.40**
 
-Repeated identical queries: **−0.40**
+### Grader
 
-### Submit scoring (deterministic)
-- **1.0** → correct function **+** correct vulnerability keyword → reward +5.0
-- **0.5** → correct function, wrong/vague vulnerability type → reward +1.0
-- **0.0** → wrong function → reward −1.5
+- **1.0** → correct function + correct vulnerability keyword → reward **+5.0**
+- **0.5** → correct function, vague/wrong vulnerability type → reward **+1.0**
+- **0.0** → wrong function → reward **−1.5**
 
-### Vulnerability types in dataset
+### Vulnerability types covered
 Reentrancy · Missing access control · Integer overflow · tx.origin authentication ·
-Front-running · Timestamp dependence · Denial of service (unbounded loop) · Unchecked return value
+Front-running · Timestamp dependence · Denial of service · Unchecked return value
 
 ---
 
 ## Task 2 — Property Discovery *(Hard)*
 
-**Setup:** Agent is shown a single Solidity function and must write its natural-language correctness property (postcondition / invariant).
+**Setup:** A single Solidity function is shown. The agent must discover its natural-language correctness property.
 
-**Objective:** Write a precise 2–4 sentence property describing what the function guarantees when it succeeds.
+**Objective:** Write a precise 2–4 sentence postcondition describing what the function guarantees on success.
 
 ### Actions
 
@@ -74,51 +73,74 @@ Front-running · Timestamp dependence · Denial of service (unbounded loop) · U
 | `get_related_functions` | — | −0.06 |
 | `get_io` | — | −0.04 |
 | `get_similar_rule` | — | −0.20 |
-| `submit_property` | `property` (string) | **0.0–5.0** (scored, ONE attempt) |
+| `submit_property` | `property` (string) | **0.0–5.0** scored, ONE attempt |
 
-Repeated identical queries: **−0.40**
+### Grader (keyword-weighted)
 
-### Submit scoring (keyword-weighted)
 ```
-score = 0.70 × (key_phrases_matched / total_key_phrases)
-      + 0.30 × (bonus_phrases_matched / total_bonus_phrases)
-
-reward = score × 5.0    →  range: 0.0 – 5.0
+score  = 0.70 × (key_phrases_matched / total_key)
+       + 0.30 × (bonus_phrases_matched / total_bonus)
+reward = score × 5.0
 ```
 
-Matching uses **word-set containment** with synonym expansion (e.g. "caller" matches "msg.sender", "sender", "user"). Phrases don't need to be adjacent — all constituent words just need to appear somewhere in the submitted text.
+Matching uses **word-set containment + synonym expansion** — words don't need to be adjacent.
+
+---
+
+## Task 3 — Rule Checker *(Easy)*
+
+**Setup:** A Solidity contract is shown alongside a violated property in natural English. One function breaks that property.
+
+**Objective:** Identify which function violates the property.
+
+### Actions
+
+| Action | Params | Reward |
+|--------|--------|--------|
+| `list_functions` | — | −0.05 |
+| `get_function_metadata` | `function_name` | −0.05 |
+| `get_function_code` | `function_name` | −0.10 |
+| `get_state_variable` | `variable_name` (opt.) | −0.05 |
+| `get_call_graph` | — | −0.08 |
+| `get_formalized_property` | — | **−0.03** (cheapest — read this first!) |
+| `submit_function` | `function_name` | **+5.0 / +1.5 / −1.5**, ONE attempt |
+
+### Grader (three-tier deterministic)
 
-**One submission per episode** — choose carefully.
+- **1.0** → exact target function (case-insensitive) → reward **+5.0**
+- **0.3** → a direct internal subfunction of the target → reward **+1.5**
+- **0.0** → anything else → reward **−1.5**
 
-### Property coverage
-11 functions across 4 contracts with ground-truth properties: SimpleVault (deposit, withdraw, emergencyDrain), TokenSale (buyTokens, setPrice, withdrawETH), DutchAuction (getPrice, bid, finalize), YieldFarm (stake, claimRewards).
+`get_formalized_property` returns the precise pre/post-condition (`rule_broken_specs`). Reading it costs only −0.03 and usually provides enough information to identify the violating function without inspecting all code.
 
 ---
 
 ## Observation Space
 
-Every `step()` and `reset()` returns the same `Observation` structure:
+All tasks share the same `Observation` structure:
 
 ```json
 {
-  "task_id": "task2_property_discovery",
-  "contract_name": "YieldFarm",
-  "contract_description": "A simple yield farming contract...",
-  "available_actions": ["get_function_code", "get_function_natspec", ...],
-  "last_action": "get_function_natspec",
-  "last_action_result": "NatSpec for 'claimRewards':\n@notice Claim all accrued...",
-  "step_count": 2,
-  "cumulative_reward": -0.14,
+  "task_id": "task3_rule_checker",
+  "contract_name": "SimpleVault",
+  "contract_description": "An ETH vault that allows users to deposit...",
+  "available_actions": ["list_functions", "get_function_metadata", "..."],
+  "last_action": "get_formalized_property",
+  "last_action_result": "Formal property:\nPre: caller != owner...",
+  "step_count": 1,
+  "cumulative_reward": -0.03,
   "done": false,
   "extra": {
-    "target_function": "claimRewards",
-    "target_signature": "claimRewards()",
-    "solidity_version": "0.8.10",
-    "hint": "Discover the property of the target function..."
+    "property_english": "Only the owner should be able to drain the vault...",
+    "solidity_version": "0.8.0",
+    "hint": "Find the function that violates this property..."
   }
 }
 ```
 
+For Task 2, `extra` contains `target_function` and `target_signature`.
+For Task 3, `extra` contains `property_english`.
+
 ---
 
 ## Project Structure
@@ -126,54 +148,54 @@ Every `step()` and `reset()` returns the same `Observation` structure:
 ```
 smart-contract-env/
 ├── data/
-│   ├── contracts.json          # 4 contracts · 8 vulnerabilities · 11 properties
-│   └── data_loader.py          # JSON parser, episode samplers, T1 + T2 helpers
+│   ├── contracts.json          # 4 contracts, 8 vulns, 11 properties, 8 rule episodes
+│   └── data_loader.py          # loaders for all three tasks
 ├── env/
 │   ├── base_env.py             # Abstract OpenEnv base class
-│   └── schemas.py              # Pydantic: Observation, Action, Reward, StepResult…
+│   └── schemas.py              # Typed Pydantic models (all ActionTypes)
 ├── tasks/
 │   ├── task1/
-│   │   ├── environment.py      # Full Task 1 RL environment
-│   │   └── grader.py           # Deterministic 0/0.5/1.0 rubric + longest-match keywords
+│   │   ├── environment.py      # Vulnerability detection environment
+│   │   └── grader.py           # Longest-match keyword grader (0/0.5/1.0)
 │   ├── task2/
-│   │   ├── environment.py      # Full Task 2 RL environment (one submit per episode)
-│   │   └── grader.py           # Keyword-weighted 0.0–1.0 grader + synonym expansion
-│   └── task3/                  # TODO: Rule Checker (placeholder)
-├── app.py                      # FastAPI server — all OpenEnv HTTP endpoints
-├── inference.py                # Baseline LLM agent (Task 1 + Task 2)
-├── eval.py                     # Oracle/partial/random evaluation harness
-├── demo.py                     # Colourised interactive + scripted demo
-├── validate.py                 # 19-check pre-submission validator
+│   │   ├── environment.py      # Property discovery (one submit_property)
+│   │   └── grader.py           # Word-set + synonym grader (0.0–1.0)
+│   └── task3/
+│       ├── environment.py      # Rule checker (one submit_function)
+│       └── grader.py           # Three-tier grader (1.0/0.3/0.0)
+├── app.py                      # FastAPI — all OpenEnv HTTP endpoints
+├── inference.py                # Baseline LLM agent (all 3 tasks)
+├── eval.py                     # Oracle/partial/floor evaluation harness
+├── demo.py                     # Colourised scripted demos for all 3 tasks
+├── validate.py                 # 23-check pre-submission validator
 ├── openenv.yaml                # Full OpenEnv spec metadata
-├── Dockerfile                  # Port 7860, uvicorn, healthcheck
+├── Dockerfile                  # Port 7860, healthcheck
 └── requirements.txt
 ```
 
 ---
 
-## Setup & Usage
+## Setup
 
 ### Local Python
 
 ```bash
-git clone <repo> && cd smart-contract-env
 pip install -r requirements.txt
 
-# Run the server
-python app.py                   # → http://localhost:7860
+# Start the server
+python app.py                     # → http://localhost:7860
 
-# Run interactive demo
-python demo.py                  # Task 1 interactive
-python demo.py --auto           # Task 1 scripted
-python demo.py --auto --task 2  # Task 2 scripted (add --task flag)
+# Interactive / scripted demos
+python demo.py --auto             # Task 1 scripted demo
+python demo.py --auto --seed 42   # Task 2 (same flag, different env seed)
 
-# Run evaluation harness (no LLM needed)
-python eval.py                  # Both tasks, 8 episodes each
-python eval.py --task 2         # Task 2 only
+# Full evaluation harness (no LLM required)
+python eval.py                    # All 3 tasks, 8 episodes each
+python eval.py --task 3           # Task 3 only
 python eval.py --episodes 16 --verbose
 
 # Pre-submission validation
-python validate.py              # 19/19 checks
+python validate.py                # 23/23 checks
 ```
 
 ### Docker
@@ -186,28 +208,19 @@ docker run -p 7860:7860 sc-audit-env
 ### Direct Python API
 
 ```python
-from tasks.task1.environment import Task1Environment
-from tasks.task2.environment import Task2Environment
+# Task 3 example
+from tasks.task3.environment import Task3Environment
 from env.schemas import Action, ActionType
 
-# Task 1
-env = Task1Environment()
+env = Task3Environment()
 r = env.reset(seed=42)
-print(r.observation.contract_name)          # SimpleVault
-s = env.step(Action(action_type=ActionType.LIST_FUNCTIONS))
-s = env.step(Action(action_type=ActionType.SUBMIT,
-             params={"function_name": "emergencyDrain",
-                     "vulnerability_type": "missing access control"}))
-print(s.reward.value)                       # +5.0
-
-# Task 2
-env2 = Task2Environment()
-r2 = env2.reset(seed=42)
-print(r2.observation.extra["target_function"])  # claimRewards
-s2 = env2.step(Action(action_type=ActionType.GET_FUNCTION_NATSPEC))
-s2 = env2.step(Action(action_type=ActionType.SUBMIT_PROPERTY,
-               params={"property": "After a successful claimRewards call, all accrued reward tokens are transferred to the caller and their rewards balance is zeroed. Reverts if no rewards."}))
-print(s2.reward.value)                      # ~4.0
+print(r.observation.extra["property_english"])
+# "Only the owner should be able to drain the vault..."
+
+s = env.step(Action(action_type=ActionType.GET_FORMALIZED_PROPERTY))
+s = env.step(Action(action_type=ActionType.SUBMIT_FUNCTION,
+             params={"function_name": "emergencyDrain"}))
+print(s.reward.value)  # +5.0
 ```
 
 ---
@@ -220,20 +233,23 @@ print(s2.reward.value)                      # ~4.0
 | `GET` | `/tasks` | All tasks + status |
 | `POST` | `/reset` | Start episode (`task_id`, `seed`) |
 | `POST` | `/step` | Take action (`action_type`, `params`) |
-| `GET` | `/state` | Debug: internal episode state |
-| `GET` | `/action_space?task_id=...` | Action schema for a task |
+| `GET` | `/state` | Internal debug state |
+| `GET` | `/action_space?task_id=...` | Action schema |
 | `GET` | `/observation_space` | Observation schema |
 
 ```bash
-# Task 2 full episode
+# Full Task 3 episode
 curl -X POST localhost:7860/reset \
-  -d '{"task_id":"task2_property_discovery","seed":42}'
+  -H "Content-Type: application/json" \
+  -d '{"task_id":"task3_rule_checker","seed":42}'
 
 curl -X POST localhost:7860/step \
-  -d '{"action_type":"get_function_natspec","params":{}}'
+  -H "Content-Type: application/json" \
+  -d '{"action_type":"get_formalized_property","params":{}}'
 
 curl -X POST localhost:7860/step \
-  -d '{"action_type":"submit_property","params":{"property":"..."}}'
+  -H "Content-Type: application/json" \
+  -d '{"action_type":"submit_function","params":{"function_name":"emergencyDrain"}}'
 ```
 
 ---
@@ -244,45 +260,30 @@ curl -X POST localhost:7860/step \
 export API_BASE_URL="https://api.openai.com/v1"
 export MODEL_NAME="gpt-4o-mini"
 export HF_TOKEN="sk-..."
-
 python inference.py
-# → baseline_scores.json
 ```
 
-### Expected baseline scores (gpt-4o-mini, 3 episodes per task)
+### Expected scores (gpt-4o-mini, 3 episodes per task)
 
 | Task | Avg Grader Score | Notes |
 |------|-----------------|-------|
-| Task 1 | ~0.67 | Good at common vulns; misses subtle ones |
-| Task 2 | ~0.55 | Reasonable properties but often misses specific variable names |
-| Task 3 | 0.00 | Placeholder |
+| Task 1 | ~0.67 | Good at classic vulns; struggles with subtle ones |
+| Task 2 | ~0.55 | Reasonable properties; misses specific variable names |
+| Task 3 | ~0.78 | Property text gives strong signal; usually correct in 3–4 steps |
 
 ---
 
-## Evaluation Scores
+## Evaluation Summary
 
-Deterministic oracle / partial / baseline tiers verified on 8 episodes (seeds 42–49):
+Deterministic oracle / partial / floor tiers verified on 8 episodes (seeds 42–49):
 
-| Task | Oracle | Partial | Floor |
-|------|--------|---------|-------|
-| Task 1 | **1.000** | 0.500 | 0.000 |
-| Task 2 | **0.775** | 0.034 | 0.000 |
+| Task | Oracle | Partial/Sub | Floor | Ordering |
+|------|--------|-------------|-------|----------|
+| Task 1 | **1.000** | 0.500 | 0.000 | ✅ 1.0 > 0.5 > 0.0 |
+| Task 2 | **0.775** | 0.034 | 0.000 | ✅ 0.775 > 0.034 > 0.0 |
+| Task 3 | **1.000** | 0.037 | 0.000 | ✅ 1.0 > 0.037 > 0.0 |
 
-The clear separation confirms the grader provides **meaningful gradient signal** for RL training.
-
----
-
-## Deploying to Hugging Face Spaces
-
-1. Create a new **Docker** Space at [huggingface.co/spaces](https://huggingface.co/spaces)
-2. Add tag `openenv` in the Space settings
-3. Copy the `SPACES_README.md` frontmatter into `README.md`
-4. Push:
-
-```bash
-git remote add hf https://huggingface.co/spaces/<user>/<space>
-git push hf main
-```
+The clear separation across all three tasks confirms the graders provide **meaningful gradient signal** across the full reward range — a core requirement for RL training environments.
 
 ---
 
@@ -295,15 +296,25 @@ git push hf main
 | `reset() → ResetResult` | ✅ |
 | `state() → StateResult` | ✅ |
 | `openenv.yaml` metadata | ✅ |
-| 3+ tasks defined | ✅ (2 active, 1 placeholder) |
+| 3 tasks, all active | ✅ |
 | Grader scores in [0.0, 1.0] | ✅ |
-| Shaped rewards (non-binary) | ✅ |
+| Shaped rewards (non-binary signal) | ✅ |
 | Dockerfile + port 7860 | ✅ |
 | `inference.py` with OpenAI client | ✅ |
-| `validate.py` — all 19 checks pass | ✅ |
+| `validate.py` — 23/23 checks pass | ✅ |
+
+---
+
+## Deploying to Hugging Face Spaces
+
+```bash
+# Copy the HF frontmatter into README.md, then:
+git remote add hf https://huggingface.co/spaces/<user>/<space>
+git push hf main
+```
 
 ---
 
 ## License
 
-MIT. Contract vulnerability data adapted from Certora audits on production DeFi protocols.
+MIT. Contract vulnerability patterns adapted from Certora audits on production DeFi protocols.

app.py

@@ -24,6 +24,7 @@ from pydantic import BaseModel
 from env.schemas import Action, ActionType, TaskInfo
 from tasks.task1.environment import Task1Environment
 from tasks.task2.environment import Task2Environment
+from tasks.task3.environment import Task3Environment
 
 # ─────────────────────────────────────────────────────────────────────────────
 # App
@@ -35,7 +36,7 @@ app = FastAPI(
         "OpenEnv-compliant reinforcement learning environment for smart contract "
         "security analysis. Train and evaluate agents on real-world Solidity audit tasks."
     ),
-    version="1.1.0",
+    version="1.2.0",
 )
 
 # ─────────────────────────────────────────────────────────────────────────────
@@ -48,7 +49,7 @@ DEFAULT_SESSION = "default"
 TASK_ENV_MAP = {
     "task1_vuln_detection":     Task1Environment,
     "task2_property_discovery": Task2Environment,
-    # TODO: "task3_rule_checker": Task3Environment,
+    "task3_rule_checker":       Task3Environment,
 }
 
 
@@ -109,7 +110,7 @@ def list_tasks():
             name="Rule Checker",
             difficulty="easy",
             description="Given a property in English and a Solidity contract, identify which function violates that property.",
-            status="placeholder",
+            status="active",
         ),
     ]
     return {"tasks": [t.model_dump() for t in tasks]}
@@ -195,6 +196,19 @@ def action_space(task_id: str = "task1_vuln_detection"):
                 {"type": "submit_property",       "params": {"property": "string"},     "reward": "0.0–5.0 (scored)", "description": "Submit property. ONE attempt. Ends episode."},
             ],
         }
+    if task_id == "task3_rule_checker":
+        return {
+            "task_id": task_id,
+            "actions": [
+                {"type": "list_functions",          "params": {},                             "reward": -0.05, "description": "List all function names"},
+                {"type": "get_function_metadata",   "params": {"function_name": "string"}, "reward": -0.05, "description": "Get signature, visibility, params of a function"},
+                {"type": "get_function_code",       "params": {"function_name": "string"}, "reward": -0.10, "description": "Read full Solidity source of a function"},
+                {"type": "get_state_variable",      "params": {"variable_name": "string (opt)"}, "reward": -0.05, "description": "Get a state variable or list all"},
+                {"type": "get_call_graph",          "params": {},                             "reward": -0.08, "description": "Get function call graph"},
+                {"type": "get_formalized_property", "params": {},                             "reward": -0.03, "description": "Get formal pre/post-condition for the property"},
+                {"type": "submit_function",         "params": {"function_name": "string"}, "reward": "+5.0 / +1.5 / -1.5", "description": "Submit answer. ONE attempt. Ends episode."},
+            ],
+        }
     return {"error": f"No action space defined for task '{task_id}'"}
 
 

data/data_loader.py

@@ -193,3 +193,36 @@ def get_similar_rule(
                     "natspec": "",
                 }
     return None
+
+
+# ────────────────────────────────────────────────────────────────
+# Task 3 helpers
+# ────────────────────────────────────────────────────────────────
+
+def get_all_task3_entries(
+    contracts: List[Dict[str, Any]],
+) -> List[Tuple[Dict[str, Any], Dict[str, Any]]]:
+    """
+    Returns (contract, function) pairs where function has a task3 field
+    with a non-empty property_english. These are the episode pool for Task 3.
+    """
+    entries = []
+    for contract in contracts:
+        for fn in contract.get("functions", []):
+            t3 = fn.get("task3", {})
+            if t3.get("property_english"):
+                entries.append((contract, fn))
+    return entries
+
+
+def sample_task3_episode(
+    contracts: List[Dict[str, Any]],
+    rng: Optional[random.Random] = None,
+) -> Tuple[Dict[str, Any], Dict[str, Any]]:
+    """Randomly selects one (contract, vulnerable_function) pair for Task 3."""
+    if rng is None:
+        rng = random.Random()
+    entries = get_all_task3_entries(contracts)
+    if not entries:
+        raise ValueError("No Task 3 entries found in dataset.")
+    return rng.choice(entries)

demo.py

@@ -355,3 +355,79 @@ def run_auto_demo_t2(seed: int = 42, delay: float = 0.9):
         if step_result.done:
             _print_episode_summary(sobs)
             return
+
+
+# ─────────────────────────────────────────────────────────────────────────────
+# Task 3 demo
+# ─────────────────────────────────────────────────────────────────────────────
+
+DEMO_SCRIPTS_T3 = {
+    42: [
+        (ActionType.GET_FORMALIZED_PROPERTY, {},
+         "Read the formal spec first — cheapest action at -0.03."),
+        (ActionType.LIST_FUNCTIONS, {},
+         "List all functions to survey candidates."),
+        (ActionType.GET_FUNCTION_CODE, {"function_name": "emergencyDrain"},
+         "No access modifier! Anyone can call this — that's the violation."),
+        (ActionType.SUBMIT_FUNCTION, {"function_name": "emergencyDrain"},
+         "Confident. emergencyDrain violates the access-control property."),
+    ],
+    45: [
+        (ActionType.GET_FORMALIZED_PROPERTY, {},
+         "Formal spec: first caller at valid price should win."),
+        (ActionType.LIST_FUNCTIONS, {},
+         "Auction contract — bid() immediately looks suspicious."),
+        (ActionType.GET_FUNCTION_CODE, {"function_name": "bid"},
+         "No commit-reveal, no maxPrice guard — front-running is trivially possible."),
+        (ActionType.SUBMIT_FUNCTION, {"function_name": "bid"},
+         "bid() violates the front-running property. Submitting."),
+    ],
+}
+
+
+def run_auto_demo_t3(seed: int = 42, delay: float = 0.9):
+    """Run the scripted Task 3 demo."""
+    from tasks.task3.environment import Task3Environment
+
+    script = DEMO_SCRIPTS_T3.get(seed)
+    env    = Task3Environment()
+    result = env.reset(seed=seed)
+    obs    = result.observation
+
+    print()
+    print(f"{BOLD}{CYAN}╔══════════════════════════════════════════════════════════╗")
+    print(f"║   Smart Contract Audit RL Env  ·  Task 3 Demo            ║")
+    print(f"╚══════════════════════════════════════════════════════════╝{RESET}")
+    print()
+    print(f"{BOLD}Mode:{RESET} Automated demo  |  {BOLD}Seed:{RESET} {seed}")
+    print(f"{BOLD}Task:{RESET} Rule Checker")
+    print()
+
+    prop = obs.extra.get("property_english", "")
+    print(f"{BOLD}Contract  :{RESET} {obs.contract_name}")
+    print(f"{BOLD}Property  :{RESET} {prop[:100]}{'...' if len(prop) > 100 else ''}")
+    print(f"{BOLD}Goal      :{RESET} Find the function that violates this property.")
+    print(DIVIDER)
+
+    if not script:
+        print(f"{YELLOW}No pre-written script for seed {seed}. Try seed 42 or 45.{RESET}")
+        return
+
+    for at, params, commentary in script:
+        time.sleep(delay)
+        print(f"\n{CYAN}▶ Agent thinking:{RESET} {commentary}")
+        time.sleep(delay * 0.5)
+        step_result = env.step(Action(action_type=at, params=params))
+        sobs = step_result.observation
+        print(DIVIDER)
+        print(f"{BOLD}Step {sobs.step_count:2d}{RESET}  [{at.value}]  "
+              f"r={step_result.reward.value:+.2f}  cum={sobs.cumulative_reward:+.2f}")
+        result_text = sobs.last_action_result or ""
+        colour = GREEN if step_result.reward.value > 0 else YELLOW
+        for line in result_text.split("\n")[:6]:
+            print(f"  {colour}{line[:90]}{RESET}")
+        print(DIVIDER)
+
+        if step_result.done:
+            _print_episode_summary(sobs)
+            return

env/schemas.py

@@ -42,10 +42,9 @@ class ActionType(str, Enum):
     SUBMIT_PROPERTY       = "submit_property"         # scored 0–5, one attempt
 
     # ── Task 3 – Rule Checker ────────────────────────────────────────────────
-    # TODO: Task 3
-    # GET_FORMALIZED_PROPERTY = "get_formalized_property"
-    # GET_FUNCTION_METADATA   = "get_function_metadata"
-    # SUBMIT_FUNCTION         = "submit_function"
+    GET_FORMALIZED_PROPERTY = "get_formalized_property"  # -0.03
+    GET_FUNCTION_METADATA   = "get_function_metadata"    # -0.05
+    SUBMIT_FUNCTION         = "submit_function"          # +5.0 / +1.5 / -1.5, one attempt
 
 
 class Action(BaseModel):

eval.py

@@ -1,33 +1,32 @@
 """
 eval.py
 -------
-Evaluation harness for the Smart Contract Audit RL Environment.
+Evaluation harness for all three tasks.
 
-Runs oracle / partial / baseline agents against Task 1 and Task 2,
-verifying that grader scores form a clear ordering and that reward
-shaping is meaningful.
+Runs oracle / partial / baseline agents, verifying score orderings and
+that reward shaping is meaningful across the trajectory.
 
 Usage:
-  python eval.py                      # Task 1 + Task 2, 8 episodes each
-  python eval.py --task 1             # Task 1 only
-  python eval.py --task 2             # Task 2 only
-  python eval.py --episodes 16        # more episodes
-  python eval.py --seed 0 --verbose   # detailed per-step trace
-  python eval.py --out results.json   # custom output file
+  python eval.py                        # all tasks, 8 episodes each
+  python eval.py --task 1|2|3           # single task
+  python eval.py --episodes 16 --verbose
+  python eval.py --out results.json
 """
 
 import argparse
 import json
-import sys
 from typing import Any, Dict, List
 
 from tasks.task1.environment import Task1Environment
 from tasks.task2.environment import Task2Environment
+from tasks.task3.environment import Task3Environment
 from env.schemas import Action, ActionType
 from data.data_loader import (
     load_contracts,
     get_function_by_name,
     get_all_vulnerable_entries,
+    get_all_property_entries,
+    get_all_task3_entries,
 )
 
 
@@ -36,12 +35,10 @@ from data.data_loader import (
 # ─────────────────────────────────────────────────────────────────────────────
 
 def oracle_t1(env: Task1Environment, seed: int, verbose: bool = False) -> Dict[str, Any]:
-    """Always submits the exact ground-truth answer → score = 1.0."""
+    """Submits the exact ground-truth function + vulnerability → score = 1.0."""
     r   = env.reset(seed=seed)
     obs = r.observation
-    st  = env.state()
-    fn_name = st.target_function
-
+    fn_name = env.state().target_function
     contracts = load_contracts()
     vuln_issue = ""
     for c in contracts:
@@ -49,42 +46,33 @@ def oracle_t1(env: Task1Environment, seed: int, verbose: bool = False) -> Dict[s
         if fn and fn.get("vulnerable"):
             vuln_issue = fn["vulnerability_details"]["issue"]
             break
-
     if verbose:
         print(f"    {obs.contract_name}.{fn_name}()  [{vuln_issue}]")
-
     env.step(Action(action_type=ActionType.LIST_FUNCTIONS))
     env.step(Action(action_type=ActionType.GET_FUNCTION_CODE,
                     params={"function_name": fn_name}))
     result = env.step(Action(action_type=ActionType.SUBMIT,
                               params={"function_name": fn_name,
                                       "vulnerability_type": vuln_issue}))
-
     v = result.reward.value
     score = 1.0 if v >= 4.9 else (0.5 if v >= 0.9 else 0.0)
-    return {
-        "seed": seed,
-        "contract": obs.contract_name,
-        "target_function": fn_name,
-        "vulnerability": vuln_issue,
-        "grader_score": score,
-        "cumulative_reward": result.observation.cumulative_reward,
-    }
+    return {"seed": seed, "contract": obs.contract_name, "target_function": fn_name,
+            "vulnerability": vuln_issue, "grader_score": score,
+            "cumulative_reward": result.observation.cumulative_reward}
 
 
 def partial_t1(env: Task1Environment, seed: int) -> Dict[str, Any]:
-    """Right function, wrong vuln type → score = 0.5."""
+    """Right function, 'unknown' vuln type → score = 0.5."""
     env.reset(seed=seed)
     fn_name = env.state().target_function
     result = env.step(Action(action_type=ActionType.SUBMIT,
-                              params={"function_name": fn_name,
-                                      "vulnerability_type": "unknown"}))
+                              params={"function_name": fn_name, "vulnerability_type": "unknown"}))
     v = result.reward.value
     return {"seed": seed, "grader_score": 0.5 if v >= 0.9 else 0.0,
             "cumulative_reward": result.observation.cumulative_reward}
 
 
-def random_t1(env: Task1Environment, seed: int) -> Dict[str, Any]:
+def wrong_t1(env: Task1Environment, seed: int) -> Dict[str, Any]:
     """Always submits 'constructor' → score = 0.0."""
     env.reset(seed=seed)
     result = env.step(Action(action_type=ActionType.SUBMIT,
@@ -99,12 +87,11 @@ def random_t1(env: Task1Environment, seed: int) -> Dict[str, Any]:
 # ─────────────────────────────────────────────────────────────────────────────
 
 def oracle_t2(env: Task2Environment, seed: int, verbose: bool = False) -> Dict[str, Any]:
-    """Submits the exact ground-truth natural_language → score ≥ 0.70."""
+    """Submits ground-truth natural_language → score ≥ 0.70."""
     r   = env.reset(seed=seed)
     obs = r.observation
     fn_name  = obs.extra["target_function"]
     contract = obs.contract_name
-
     contracts = load_contracts()
     gt_text = ""
     for c in contracts:
@@ -113,24 +100,15 @@ def oracle_t2(env: Task2Environment, seed: int, verbose: bool = False) -> Dict[s
             if fn and fn.get("property"):
                 gt_text = fn["property"]["natural_language"]
             break
-
     if verbose:
         print(f"    {contract}.{fn_name}()")
-
-    # read code first (realistic browsing step)
     env.step(Action(action_type=ActionType.GET_FUNCTION_CODE))
     result = env.step(Action(action_type=ActionType.SUBMIT_PROPERTY,
                               params={"property": gt_text}))
-
     r_val = result.reward.value
     score = round(r_val / 5.0, 4) if r_val > 0 else 0.0
-    return {
-        "seed": seed,
-        "contract": contract,
-        "function": fn_name,
-        "grader_score": score,
-        "cumulative_reward": result.observation.cumulative_reward,
-    }
+    return {"seed": seed, "contract": contract, "function": fn_name,
+            "grader_score": score, "cumulative_reward": result.observation.cumulative_reward}
 
 
 def partial_t2(env: Task2Environment, seed: int) -> Dict[str, Any]:
@@ -148,16 +126,67 @@ def partial_t2(env: Task2Environment, seed: int) -> Dict[str, Any]:
     result = env.step(Action(action_type=ActionType.SUBMIT_PROPERTY,
                               params={"property": comment}))
     r_val = result.reward.value
-    score = round(r_val / 5.0, 4) if r_val > 0 else 0.0
-    return {"seed": seed, "grader_score": score,
+    return {"seed": seed, "grader_score": round(r_val / 5.0, 4) if r_val > 0 else 0.0,
             "cumulative_reward": result.observation.cumulative_reward}
 
 
 def empty_t2(env: Task2Environment, seed: int) -> Dict[str, Any]:
     """Submits empty string → score = 0.0."""
     env.reset(seed=seed)
-    result = env.step(Action(action_type=ActionType.SUBMIT_PROPERTY,
-                              params={"property": ""}))
+    result = env.step(Action(action_type=ActionType.SUBMIT_PROPERTY, params={"property": ""}))
+    return {"seed": seed, "grader_score": 0.0,
+            "cumulative_reward": result.observation.cumulative_reward}
+
+
+# ─────────────────────────────────────────────────────────────────────────────
+# Task 3 agents
+# ─────────────────────────────────────────────────────────────────────────────
+
+def oracle_t3(env: Task3Environment, seed: int, verbose: bool = False) -> Dict[str, Any]:
+    """Always submits the exact target function → score = 1.0."""
+    r   = env.reset(seed=seed)
+    obs = r.observation
+    fn_name  = env.state().target_function
+    contract = obs.contract_name
+    if verbose:
+        prop = obs.extra.get("property_english", "")[:60]
+        print(f"    {contract}.{fn_name}()  \"{prop}\"")
+    env.step(Action(action_type=ActionType.GET_FORMALIZED_PROPERTY))
+    env.step(Action(action_type=ActionType.LIST_FUNCTIONS))
+    result = env.step(Action(action_type=ActionType.SUBMIT_FUNCTION,
+                              params={"function_name": fn_name}))
+    v = result.reward.value
+    score = 1.0 if v >= 4.9 else (0.3 if v >= 1.0 else 0.0)
+    return {"seed": seed, "contract": contract, "target_function": fn_name,
+            "grader_score": score, "cumulative_reward": result.observation.cumulative_reward}
+
+
+def subfunction_t3(env: Task3Environment, seed: int) -> Dict[str, Any]:
+    """Submits the first partial-credit subfunction if it exists, else 'constructor'."""
+    r   = env.reset(seed=seed)
+    obs = r.observation
+    contracts = load_contracts()
+    partial_fns = []
+    for c in contracts:
+        if c["contract_name"] == obs.contract_name:
+            fn = get_function_by_name(c, env.state().target_function)
+            if fn:
+                partial_fns = fn.get("task3", {}).get("partial_credit_functions", [])
+            break
+    submit_name = partial_fns[0] if partial_fns else "constructor"
+    result = env.step(Action(action_type=ActionType.SUBMIT_FUNCTION,
+                              params={"function_name": submit_name}))
+    v = result.reward.value
+    score = 1.0 if v >= 4.9 else (0.3 if v >= 1.0 else 0.0)
+    return {"seed": seed, "grader_score": score, "submitted": submit_name,
+            "cumulative_reward": result.observation.cumulative_reward}
+
+
+def wrong_t3(env: Task3Environment, seed: int) -> Dict[str, Any]:
+    """Always submits 'constructor' → score = 0.0."""
+    env.reset(seed=seed)
+    result = env.step(Action(action_type=ActionType.SUBMIT_FUNCTION,
+                              params={"function_name": "constructor"}))
     return {"seed": seed, "grader_score": 0.0,
             "cumulative_reward": result.observation.cumulative_reward}
 
@@ -166,112 +195,144 @@ def empty_t2(env: Task2Environment, seed: int) -> Dict[str, Any]:
 # Evaluation runners
 # ─────────────────────────────────────────────────────────────────────────────
 
-def run_task1_eval(num_episodes: int, seed_offset: int, verbose: bool) -> Dict[str, Any]:
+def run_task1_eval(n: int, seed_offset: int, verbose: bool) -> Dict[str, Any]:
     print("\n" + "=" * 64)
     print("TASK 1 — Targeted Vulnerability Detection")
     print("=" * 64)
     contracts = load_contracts()
-    entries   = get_all_vulnerable_entries(contracts)
-    print(f"  Dataset: {len(contracts)} contracts, {len(entries)} vulnerable functions\n")
-
+    print(f"  Dataset: {len(contracts)} contracts, "
+          f"{len(get_all_vulnerable_entries(contracts))} vulnerable functions\n")
     env = Task1Environment()
 
-    print("▶ Oracle agent (always submits correct answer):")
+    print("▶ Oracle (correct function + correct vuln type → 1.0):")
     oracle_eps = []
-    for i in range(num_episodes):
-        ep = oracle_t1(env, seed_offset + i, verbose=verbose)
+    for i in range(n):
+        ep = oracle_t1(env, seed_offset + i, verbose)
         oracle_eps.append(ep)
         print(f"  seed={ep['seed']:3d}  {ep['contract']:12s}.{ep['target_function']:18s}"
               f"  score={ep['grader_score']:.1f}  reward={ep['cumulative_reward']:+.2f}")
-    oracle_avg   = sum(e["grader_score"] for e in oracle_eps) / num_episodes
-    oracle_avg_r = sum(e["cumulative_reward"] for e in oracle_eps) / num_episodes
-    print(f"\n  Oracle avg score : {oracle_avg:.3f}  avg reward: {oracle_avg_r:+.2f}")
+    oracle_avg   = sum(e["grader_score"] for e in oracle_eps) / n
+    oracle_avg_r = sum(e["cumulative_reward"] for e in oracle_eps) / n
+    print(f"\n  Oracle avg: {oracle_avg:.3f}  reward: {oracle_avg_r:+.2f}")
 
-    print("\n▶ Partial agent (right function, wrong vuln type → 0.5):")
-    partial_eps = [partial_t1(env, seed_offset + i) for i in range(num_episodes)]
-    partial_avg = sum(e["grader_score"] for e in partial_eps) / num_episodes
-    print(f"  Partial avg score: {partial_avg:.3f}")
+    print("\n▶ Partial (right function, wrong vuln → 0.5):")
+    partial_eps = [partial_t1(env, seed_offset + i) for i in range(n)]
+    partial_avg = sum(e["grader_score"] for e in partial_eps) / n
+    print(f"  Partial avg: {partial_avg:.3f}")
 
-    print("\n▶ Random agent (always wrong → 0.0):")
-    random_eps = [random_t1(env, seed_offset + i) for i in range(num_episodes)]
-    random_avg = sum(e["grader_score"] for e in random_eps) / num_episodes
-    print(f"  Random avg score : {random_avg:.3f}")
+    print("\n▶ Wrong (always 'constructor' → 0.0):")
+    wrong_eps = [wrong_t1(env, seed_offset + i) for i in range(n)]
+    wrong_avg = sum(e["grader_score"] for e in wrong_eps) / n
+    print(f"  Wrong avg: {wrong_avg:.3f}")
 
     vuln_seen: Dict[str, int] = {}
     for ep in oracle_eps:
         v = ep.get("vulnerability", "unknown")
         vuln_seen[v] = vuln_seen.get(v, 0) + 1
-    print("\n▶ Vulnerability type coverage:")
+    print("\n▶ Vulnerability coverage:")
     for v in sorted(vuln_seen):
         print(f"  {vuln_seen[v]:2d}×  {v}")
 
-    assert oracle_avg == 1.0,  f"Oracle should be 1.0, got {oracle_avg}"
-    assert partial_avg == 0.5, f"Partial should be 0.5, got {partial_avg}"
-    assert random_avg == 0.0,  f"Random should be 0.0, got {random_avg}"
-    print("\n  ✅ Task 1 score ordering: oracle(1.0) > partial(0.5) > random(0.0)")
+    assert oracle_avg == 1.0
+    assert partial_avg == 0.5
+    assert wrong_avg == 0.0
+    print("\n  ✅ Task 1: oracle(1.0) > partial(0.5) > wrong(0.0)")
 
     return {
         "task_id": "task1_vuln_detection",
         "oracle":  {"avg_score": oracle_avg,  "avg_reward": oracle_avg_r, "episodes": oracle_eps},
         "partial": {"avg_score": partial_avg, "episodes": partial_eps},
-        "random":  {"avg_score": random_avg,  "episodes": random_eps},
+        "wrong":   {"avg_score": wrong_avg,   "episodes": wrong_eps},
         "vuln_coverage": vuln_seen,
     }
 
 
-def run_task2_eval(num_episodes: int, seed_offset: int, verbose: bool) -> Dict[str, Any]:
+def run_task2_eval(n: int, seed_offset: int, verbose: bool) -> Dict[str, Any]:
     print("\n" + "=" * 64)
     print("TASK 2 — Property Discovery")
     print("=" * 64)
-    from data.data_loader import get_all_property_entries
     contracts = load_contracts()
-    entries   = get_all_property_entries(contracts)
-    print(f"  Dataset: {len(entries)} functions with properties\n")
-
+    print(f"  Dataset: {len(get_all_property_entries(contracts))} property entries\n")
     env = Task2Environment()
 
-    print("▶ Oracle agent (submits ground-truth natural language):")
+    print("▶ Oracle (submits ground-truth natural language):")
     oracle_eps = []
-    for i in range(num_episodes):
-        ep = oracle_t2(env, seed_offset + i, verbose=verbose)
+    for i in range(n):
+        ep = oracle_t2(env, seed_offset + i, verbose)
         oracle_eps.append(ep)
         icon = "✅" if ep["grader_score"] >= 0.65 else "⚠️ "
         print(f"  {icon} seed={ep['seed']:3d}  {ep['contract']:12s}.{ep['function']:18s}"
               f"  score={ep['grader_score']:.3f}  reward={ep['cumulative_reward']:+.2f}")
-    oracle_avg   = sum(e["grader_score"] for e in oracle_eps) / num_episodes
-    oracle_avg_r = sum(e["cumulative_reward"] for e in oracle_eps) / num_episodes
-    print(f"\n  Oracle avg score : {oracle_avg:.3f}  avg reward: {oracle_avg_r:+.2f}")
-
-    print("\n▶ Partial agent (submits NatSpec comment — partial signal):")
-    partial_eps  = [partial_t2(env, seed_offset + i) for i in range(num_episodes)]
-    partial_avg  = sum(e["grader_score"] for e in partial_eps) / num_episodes
-    partial_avg_r = sum(e["cumulative_reward"] for e in partial_eps) / num_episodes
-    print(f"  Partial avg score: {partial_avg:.3f}  avg reward: {partial_avg_r:+.2f}")
-
-    print("\n▶ Empty agent (submits nothing → 0.0):")
-    empty_eps  = [empty_t2(env, seed_offset + i) for i in range(num_episodes)]
-    empty_avg  = sum(e["grader_score"] for e in empty_eps) / num_episodes
-    print(f"  Empty avg score  : {empty_avg:.3f}")
-
-    fn_seen: Dict[str, int] = {}
-    for ep in oracle_eps:
-        fn_seen[ep["function"]] = fn_seen.get(ep["function"], 0) + 1
-    print("\n▶ Function coverage:")
-    for fn in sorted(fn_seen):
-        print(f"  {fn_seen[fn]:2d}×  {fn}")
-
-    assert oracle_avg > 0.60, f"Oracle avg {oracle_avg:.3f} should be > 0.60"
-    assert oracle_avg > partial_avg, "Oracle should beat partial"
-    assert partial_avg >= empty_avg,  "Partial should be >= empty"
-    assert empty_avg == 0.0, f"Empty should be 0.0, got {empty_avg}"
-    print(f"\n  ✅ Task 2 score ordering: oracle({oracle_avg:.3f}) > partial({partial_avg:.3f}) > empty(0.0)")
+    oracle_avg   = sum(e["grader_score"] for e in oracle_eps) / n
+    oracle_avg_r = sum(e["cumulative_reward"] for e in oracle_eps) / n
+    print(f"\n  Oracle avg: {oracle_avg:.3f}  reward: {oracle_avg_r:+.2f}")
+
+    print("\n▶ Partial (submits NatSpec comment):")
+    partial_eps   = [partial_t2(env, seed_offset + i) for i in range(n)]
+    partial_avg   = sum(e["grader_score"] for e in partial_eps) / n
+    partial_avg_r = sum(e["cumulative_reward"] for e in partial_eps) / n
+    print(f"  Partial avg: {partial_avg:.3f}  reward: {partial_avg_r:+.2f}")
+
+    print("\n▶ Empty (submits nothing → 0.0):")
+    empty_eps = [empty_t2(env, seed_offset + i) for i in range(n)]
+    empty_avg = sum(e["grader_score"] for e in empty_eps) / n
+    print(f"  Empty avg: {empty_avg:.3f}")
+
+    assert oracle_avg > 0.60
+    assert oracle_avg > partial_avg
+    assert empty_avg == 0.0
+    print(f"\n  ✅ Task 2: oracle({oracle_avg:.3f}) > partial({partial_avg:.3f}) > empty(0.0)")
 
     return {
         "task_id": "task2_property_discovery",
         "oracle":  {"avg_score": oracle_avg,  "avg_reward": oracle_avg_r, "episodes": oracle_eps},
         "partial": {"avg_score": partial_avg, "avg_reward": partial_avg_r, "episodes": partial_eps},
         "empty":   {"avg_score": empty_avg,   "episodes": empty_eps},
-        "fn_coverage": fn_seen,
+    }
+
+
+def run_task3_eval(n: int, seed_offset: int, verbose: bool) -> Dict[str, Any]:
+    print("\n" + "=" * 64)
+    print("TASK 3 — Rule Checker")
+    print("=" * 64)
+    contracts = load_contracts()
+    print(f"  Dataset: {len(get_all_task3_entries(contracts))} rule-check episodes\n")
+    env = Task3Environment()
+
+    print("▶ Oracle (submits exact target function → 1.0):")
+    oracle_eps = []
+    for i in range(n):
+        ep = oracle_t3(env, seed_offset + i, verbose)
+        oracle_eps.append(ep)
+        print(f"  seed={ep['seed']:3d}  {ep['contract']:12s}.{ep['target_function']:18s}"
+              f"  score={ep['grader_score']:.1f}  reward={ep['cumulative_reward']:+.2f}")
+    oracle_avg   = sum(e["grader_score"] for e in oracle_eps) / n
+    oracle_avg_r = sum(e["cumulative_reward"] for e in oracle_eps) / n
+    print(f"\n  Oracle avg: {oracle_avg:.3f}  reward: {oracle_avg_r:+.2f}")
+
+    print("\n▶ Subfunction (partial-credit callee or fallback to wrong):")
+    sub_eps   = [subfunction_t3(env, seed_offset + i) for i in range(n)]
+    sub_avg   = sum(e["grader_score"] for e in sub_eps) / n
+    sub_avg_r = sum(e["cumulative_reward"] for e in sub_eps) / n
+    submitted = list({e.get("submitted", "?") for e in sub_eps})
+    print(f"  Subfunction avg: {sub_avg:.3f}  reward: {sub_avg_r:+.2f}  "
+          f"submitted fns: {submitted}")
+
+    print("\n▶ Wrong (always 'constructor' → 0.0):")
+    wrong_eps = [wrong_t3(env, seed_offset + i) for i in range(n)]
+    wrong_avg = sum(e["grader_score"] for e in wrong_eps) / n
+    print(f"  Wrong avg: {wrong_avg:.3f}")
+
+    assert oracle_avg == 1.0
+    assert 0.0 <= sub_avg <= oracle_avg
+    assert wrong_avg == 0.0
+    print(f"\n  ✅ Task 3: oracle(1.0) ≥ subfunction({sub_avg:.3f}) > wrong(0.0)")
+
+    return {
+        "task_id": "task3_rule_checker",
+        "oracle":      {"avg_score": oracle_avg,  "avg_reward": oracle_avg_r, "episodes": oracle_eps},
+        "subfunction": {"avg_score": sub_avg,     "avg_reward": sub_avg_r,   "episodes": sub_eps},
+        "wrong":       {"avg_score": wrong_avg,   "episodes": wrong_eps},
     }
 
 
@@ -281,45 +342,37 @@ def run_task2_eval(num_episodes: int, seed_offset: int, verbose: bool) -> Dict[s
 
 def main():
     parser = argparse.ArgumentParser(
-        description="Evaluate Task 1 and/or Task 2 of the SC Audit RL Environment"
+        description="Evaluate Task 1, 2, and/or 3 of the SC Audit RL Environment"
     )
-    parser.add_argument("--episodes", type=int,   default=8,
-                        help="Episodes per agent tier (default: 8)")
-    parser.add_argument("--seed",     type=int,   default=42,
-                        help="Starting RNG seed (default: 42)")
-    parser.add_argument("--task",     choices=["1", "2", "all"], default="all",
-                        help="Which task(s) to evaluate (default: all)")
-    parser.add_argument("--verbose",  action="store_true",
-                        help="Print per-episode target details")
-    parser.add_argument("--out",      default="eval_results.json",
-                        help="Output file (default: eval_results.json)")
+    parser.add_argument("--episodes", type=int, default=8)
+    parser.add_argument("--seed",     type=int, default=42)
+    parser.add_argument("--task",     choices=["1", "2", "3", "all"], default="all")
+    parser.add_argument("--verbose",  action="store_true")
+    parser.add_argument("--out",      default="eval_results.json")
     args = parser.parse_args()
 
-    report: Dict[str, Any] = {
-        "num_episodes": args.episodes,
-        "seed_offset":  args.seed,
-    }
+    report: Dict[str, Any] = {"num_episodes": args.episodes, "seed_offset": args.seed}
 
     if args.task in ("1", "all"):
         report["task1"] = run_task1_eval(args.episodes, args.seed, args.verbose)
-
     if args.task in ("2", "all"):
         report["task2"] = run_task2_eval(args.episodes, args.seed, args.verbose)
+    if args.task in ("3", "all"):
+        report["task3"] = run_task3_eval(args.episodes, args.seed, args.verbose)
 
-    # ── Summary ──────────────────────────────────────────────────────────────
     print("\n" + "=" * 64)
     print("EVALUATION COMPLETE")
     print("=" * 64)
-    if "task1" in report:
-        t1 = report["task1"]
-        print(f"  Task 1  oracle={t1['oracle']['avg_score']:.3f}  "
-              f"partial={t1['partial']['avg_score']:.3f}  "
-              f"random={t1['random']['avg_score']:.3f}")
-    if "task2" in report:
-        t2 = report["task2"]
-        print(f"  Task 2  oracle={t2['oracle']['avg_score']:.3f}  "
-              f"partial={t2['partial']['avg_score']:.3f}  "
-              f"empty={t2['empty']['avg_score']:.3f}")
+    for label, key, tiers in [
+        ("Task 1", "task1", ["oracle", "partial", "wrong"]),
+        ("Task 2", "task2", ["oracle", "partial", "empty"]),
+        ("Task 3", "task3", ["oracle", "subfunction", "wrong"]),
+    ]:
+        if key in report:
+            scores = "  ".join(
+                f"{t}={report[key][t]['avg_score']:.3f}" for t in tiers
+            )
+            print(f"  {label}  {scores}")
 
     with open(args.out, "w") as f:
         json.dump(report, f, indent=2)

inference.py

@@ -2,8 +2,8 @@
 inference.py
 ------------
 Baseline inference script for the Smart Contract Audit RL Environment.
-Implements Task 1 (Vulnerability Detection) and Task 2 (Property Discovery).
-Task 3 is a placeholder that returns 0.0.
+Implements Task 1 (Vulnerability Detection), Task 2 (Property Discovery),
+and Task 3 (Rule Checker).
 
 Environment variables:
   API_BASE_URL   – LLM API endpoint   (e.g. https://api.openai.com/v1)
@@ -30,6 +30,7 @@ from openai import OpenAI
 
 from tasks.task1.environment import Task1Environment
 from tasks.task2.environment import Task2Environment
+from tasks.task3.environment import Task3Environment
 from env.schemas import Action, ActionType
 
 # ─────────────────────────────────────────────────────────────────────────────
@@ -261,14 +262,103 @@ def run_task2(n: int = NUM_EPISODES) -> Dict[str, Any]:
             "avg_grader_score": avg_s, "avg_cumulative_reward": avg_r}
 
 
-def run_task3_placeholder() -> Dict[str, Any]:
+T3_SYSTEM = """You are a smart contract security auditor checking rule compliance.
+
+You are given a Solidity contract and a property (rule) in natural English.
+Your task is to find the ONE function that violates this property.
+
+## Actions (respond with JSON only, ONE action per turn):
+{"action": "list_functions",          "params": {}}
+{"action": "get_formalized_property", "params": {}}
+{"action": "get_function_metadata",   "params": {"function_name": "<n>"}}
+{"action": "get_function_code",       "params": {"function_name": "<n>"}}
+{"action": "get_state_variable",      "params": {"variable_name": "<n>"}}
+{"action": "get_call_graph",          "params": {}}
+{"action": "submit_function",         "params": {"function_name": "<n>"}}
+
+## Strategy:
+1. Read the property shown as property_english in the observation.
+2. list_functions to survey candidates.
+3. get_formalized_property for the precise pre/post-condition (cheap: -0.03).
+4. get_function_code on the 1-2 most suspicious functions.
+5. submit_function when confident — ONE attempt only.
+
+Clues: missing require, no access modifier, unchecked external call, unbounded array,
+tx.origin auth, integer overflow, timestamp manipulation, reentrancy ordering.
+
+Respond ONLY with valid JSON. No markdown, no explanation."""
+
+
+def _t3_user_msg(obs: Dict[str, Any]) -> str:
+    extra = obs.get("extra", {})
+    return (
+        f"Contract  : {obs['contract_name']}\n"
+        f"Property  : {extra.get('property_english', '(no property)')}\n"
+        f"Step: {obs['step_count']} | Reward: {obs['cumulative_reward']:.2f}\n\n"
+        f"Last action: {obs['last_action'] or 'None'}\n"
+        f"Result:\n{obs['last_action_result'] or 'Episode started.'}"
+    )
+
+
+def run_t3_episode(env: Task3Environment, seed: int, ep: int) -> Dict[str, Any]:
+    r   = env.reset(seed=seed)
+    obs = r.observation.model_dump()
+    prop_preview = obs['extra'].get('property_english', '')[:55]
+    print(f"    ep={ep} seed={seed}  {obs['contract_name']}  \"{prop_preview}...\"")
+
+    messages = [{"role": "system", "content": T3_SYSTEM}]
+    grader_score = 0.0
+    cum_reward   = 0.0
+
+    for step in range(15):
+        messages.append({"role": "user", "content": _t3_user_msg(obs)})
+        try:
+            resp = client.chat.completions.create(
+                model=MODEL_NAME, messages=messages,
+                max_tokens=200, temperature=0.0,
+            )
+            raw = resp.choices[0].message.content.strip()
+        except Exception as e:
+            print(f"      LLM error: {e}", file=sys.stderr)
+            break
+
+        try:
+            parsed = json.loads(raw)
+            at     = ActionType(parsed["action"])
+            params = parsed.get("params", {})
+        except Exception:
+            at, params = ActionType.LIST_FUNCTIONS, {}
+
+        messages.append({"role": "assistant", "content": raw})
+        result = env.step(Action(action_type=at, params=params))
+        obs    = result.observation.model_dump()
+        print(f"      step {step+1:2d}: {at.value:28s} r={result.reward.value:+.2f}")
+
+        if result.done:
+            v = result.reward.value
+            grader_score = 1.0 if v >= 4.9 else (0.3 if v >= 1.0 else 0.0)
+            cum_reward   = obs["cumulative_reward"]
+            break
+        time.sleep(0.3)
+
+    print(f"      → grader_score={grader_score:.1f}  cum_reward={cum_reward:.2f}")
+    return {"episode": ep, "seed": seed, "contract": obs["contract_name"],
+            "grader_score": grader_score, "cumulative_reward": cum_reward}
+
+
+def run_task3(n: int = NUM_EPISODES) -> Dict[str, Any]:
     print("\n" + "="*60)
-    print("TASK 3: Rule Checker [PLACEHOLDER — not implemented]")
+    print("TASK 3: Rule Checker")
     print("="*60)
-    print("  Skipping. Score: 0.0")
+    env = Task3Environment()
+    episodes = [run_t3_episode(env, 42 + i, i + 1) for i in range(n)]
+    avg_s = sum(e["grader_score"] for e in episodes) / n
+    avg_r = sum(e["cumulative_reward"] for e in episodes) / n
+    print(f"\n  Avg grader score  : {avg_s:.3f}")
+    print(f"  Avg cum reward    : {avg_r:.2f}")
     return {"task_id": "task3_rule_checker", "name": "Rule Checker",
-            "status": "placeholder", "num_episodes": 0, "episodes": [],
-            "avg_grader_score": 0.0, "avg_cumulative_reward": 0.0}
+            "status": "active", "num_episodes": n, "episodes": episodes,
+            "avg_grader_score": avg_s, "avg_cumulative_reward": avg_r}
 
 
 # ─────────────────────────────────────────────────────────────────────────────
@@ -281,15 +371,15 @@ def main():
 
     t1 = run_task1(NUM_EPISODES)
     t2 = run_task2(NUM_EPISODES)
-    t3 = run_task3_placeholder()
+    t3 = run_task3(NUM_EPISODES)
 
     results = {
         "model": MODEL_NAME, "base_url": API_BASE_URL,
         "tasks": [t1, t2, t3],
     }
 
-    active  = [t for t in results["tasks"] if t["status"] == "active"]
-    overall = sum(t["avg_grader_score"] for t in active) / len(active) if active else 0.0
+    active  = results["tasks"]
+    overall = sum(t["avg_grader_score"] for t in active) / len(active)
     results["overall_avg_score"] = overall
 
     print("\n" + "="*60)

openenv.yaml

@@ -1,11 +1,10 @@
 name: smart-contract-audit-env
-version: "1.1.0"
+version: "1.2.0"
 description: >
   Reinforcement learning environment for smart contract security analysis.
   Agents interact with real-world Solidity contract data from Certora-audited
-  projects, learning to detect vulnerabilities and discover correctness
-  properties — tasks that professional auditors perform daily.
-
+  projects, practising three real audit tasks: vulnerability detection,
+  property discovery, and rule checking.
 author: "SmartAudit Team"
 license: MIT
 
@@ -37,10 +36,10 @@ tasks:
   - id: task3_rule_checker
     name: Rule Checker
     difficulty: easy
-    status: placeholder
+    status: active
     description: >
-      Given a natural-language property and a Solidity file, identify the
-      function that violates that property.
+      Given a natural-language property and a Solidity contract, identify the
+      function that violates that property. Partial credit for internal subfunctions.
     max_steps: 15
     reward_range: [-5.0, 5.0]
     grader: tasks/task3/grader.py
@@ -49,70 +48,63 @@ tasks:
 observation_space:
   type: object
   properties:
-    task_id:              {type: string, description: Active task identifier}
-    contract_name:        {type: string, description: Solidity contract name}
-    contract_description: {type: string, description: Human-readable contract description}
-    available_actions:    {type: array, items: {type: string}, description: Valid action types}
+    task_id:              {type: string}
+    contract_name:        {type: string}
+    contract_description: {type: string}
+    available_actions:    {type: array, items: {type: string}}
     last_action:          {type: string, nullable: true}
     last_action_result:   {type: string, nullable: true}
     step_count:           {type: integer}
     cumulative_reward:    {type: number}
     done:                 {type: boolean}
-    extra:                {type: object, description: Task-specific hints}
+    extra:                {type: object}
 
 action_space:
   task1:
-    type: object
-    actions:
-      list_functions:       {params: {},                                  reward: -0.05}
-      get_function_code:    {params: {function_name: string},             reward: "+0.05 / -0.10"}
-      get_function_summary: {params: {function_name: string},             reward: "+0.03 / -0.05"}
-      get_file_metadata:    {params: {},                                  reward: -0.04}
-      get_state_variable:   {params: {variable_name: "string (opt)"},    reward: -0.05}
-      get_call_graph:       {params: {},                                  reward: -0.08}
-      submit:               {params: {function_name: str, vulnerability_type: str}, reward: "+5.0 / +1.0 / -1.5"}
+    list_functions:       {params: {},                            reward: -0.05}
+    get_function_code:    {params: {function_name: string},       reward: "+0.05 / -0.10"}
+    get_function_summary: {params: {function_name: string},       reward: "+0.03 / -0.05"}
+    get_file_metadata:    {params: {},                            reward: -0.04}
+    get_state_variable:   {params: {variable_name: "string opt"}, reward: -0.05}
+    get_call_graph:       {params: {},                            reward: -0.08}
+    submit:               {params: {function_name: string, vulnerability_type: string}, reward: "+5.0 / +1.0 / -1.5"}
   task2:
-    type: object
-    actions:
-      get_function_code:     {params: {}, reward: -0.06}
-      get_function_natspec:  {params: {}, reward: -0.08}
-      get_file_natspec:      {params: {}, reward: -0.03}
-      get_related_functions: {params: {}, reward: -0.06}
-      get_io:                {params: {}, reward: -0.04}
-      get_similar_rule:      {params: {}, reward: -0.20}
-      submit_property:       {params: {property: string}, reward: "0.0–5.0 (keyword-weighted)"}
+    get_function_code:     {params: {}, reward: -0.06}
+    get_function_natspec:  {params: {}, reward: -0.08}
+    get_file_natspec:      {params: {}, reward: -0.03}
+    get_related_functions: {params: {}, reward: -0.06}
+    get_io:                {params: {}, reward: -0.04}
+    get_similar_rule:      {params: {}, reward: -0.20}
+    submit_property:       {params: {property: string}, reward: "0.0-5.0 keyword-weighted, one attempt"}
+  task3:
+    list_functions:          {params: {},                            reward: -0.05}
+    get_function_metadata:   {params: {function_name: string},       reward: -0.05}
+    get_function_code:       {params: {function_name: string},       reward: -0.10}
+    get_state_variable:      {params: {variable_name: "string opt"}, reward: -0.05}
+    get_call_graph:          {params: {},                            reward: -0.08}
+    get_formalized_property: {params: {},                            reward: -0.03}
+    submit_function:         {params: {function_name: string},       reward: "+5.0 / +1.5 / -1.5, one attempt"}
 
 reward:
   type: shaped
-  description: >
-    Per-step costs encourage efficient exploration. Positive shaping rewards
-    fire when the agent inspects the actual target. Terminal rewards reflect
-    grader score accuracy.
+  all_tasks_shared:
+    repeated_query: -0.40
   task1_shaping:
-    list_functions: -0.05
-    get_function_code_wrong: -0.10
     get_function_code_correct: +0.05
-    get_function_summary_wrong: -0.05
+    get_function_code_wrong: -0.10
     get_function_summary_correct: +0.03
-    get_file_metadata: -0.04
-    get_state_variable: -0.05
-    get_call_graph: -0.08
-    repeated_query: -0.40
+    get_function_summary_wrong: -0.05
   task1_terminal:
-    correct_submission: +5.0
-    partial_submission: +1.0
-    wrong_submission: -1.5
-  task2_shaping:
-    get_function_code: -0.06
-    get_function_natspec: -0.08
-    get_file_natspec: -0.03
-    get_related_functions: -0.06
-    get_io: -0.04
-    get_similar_rule: -0.20
-    repeated_query: -0.40
+    correct: +5.0
+    partial: +1.0
+    wrong: -1.5
   task2_terminal:
-    score_range: [0.0, 5.0]
-    formula: "score * 5.0 where score = 0.70*(key_matches/total_key) + 0.30*(bonus_matches/total_bonus)"
+    formula: "score * 5.0  where score = 0.70*(key_matches/key_total) + 0.30*(bonus_matches/bonus_total)"
+    range: [0.0, 5.0]
+  task3_terminal:
+    correct_function: +5.0
+    subfunction: +1.5
+    wrong_function: -1.5
 
 data:
   source: "Certora audited DeFi projects"
@@ -120,6 +112,7 @@ data:
   num_contracts: 4
   num_vulnerable_functions: 8
   num_property_functions: 11
+  num_task3_episodes: 8
   vulnerability_types:
     - Reentrancy
     - Missing access control
@@ -132,14 +125,14 @@ data:
 
 interface:
   http:
-    reset:             POST /reset
-    step:              POST /step
-    state:             GET /state
-    tasks:             GET /tasks
-    health:            GET /health
-    action_space:      GET /action_space?task_id=<id>
-    observation_space: GET /observation_space
+    reset:             "POST /reset"
+    step:              "POST /step"
+    state:             "GET /state"
+    tasks:             "GET /tasks"
+    health:            "GET /health"
+    action_space:      "GET /action_space?task_id=<id>"
+    observation_space: "GET /observation_space"
   python:
-    reset: env.reset(seed=None)  -> ResetResult
-    step:  env.step(action)      -> StepResult
-    state: env.state()           -> StateResult
+    reset: "env.reset(seed=None) -> ResetResult"
+    step:  "env.step(action)     -> StepResult"
+    state: "env.state()          -> StateResult"

tasks/task3/__init__.py

@@ -1,31 +1,5 @@
-"""
-tasks/task3/__init__.py
------------------------
-Task 3: Rule Checker (PLACEHOLDER)
+# Task 3: Rule Checker
+from tasks.task3.environment import Task3Environment
+from tasks.task3.grader import Task3Grader
 
-TODO: Implement this task.
-
-Episode setup:
-  - One Solidity file with at least one function breaking a given property
-  - Agent is shown the property in natural English
-
-Actions (to implement):
-  - get_formalized_property : -0.03
-  - list_functions          : -0.05
-  - get_function_metadata   : -0.05
-  - get_function_code       : -0.10
-  - get_state_variables     : -0.05
-  - get_call_graph          : -0.08
-  - submit_function         :
-      - correct = +5.0
-      - subfunction of target = +1.5
-      - wrong = -1.5
-      (ONE submission per episode)
-
-See README.md for full task specification.
-"""
-
-# TODO: Task 3 – Rule Checker
-# from tasks.task3.environment import Task3Environment
-
-__all__: list = []
+__all__ = ["Task3Environment", "Task3Grader"]

tasks/task3/environment.py

@@ -0,0 +1,350 @@
+"""
+environment.py  (Task 3 – Rule Checker)
+-----------------------------------------
+OpenEnv-compliant RL environment.
+
+Episode setup
+─────────────
+  - A Solidity contract is selected that contains at least one function
+    violating a known property.
+  - The agent sees: contract description + the property in natural English.
+  - The agent must identify which function breaks that property.
+
+Observation at reset
+────────────────────
+  extra.property_english  – the violated property in plain English
+  extra.hint              – instructions for the agent
+
+Actions & rewards
+─────────────────
+  list_functions          -0.05   see all function names
+  get_function_metadata   -0.05   signature / visibility / modifiers / params
+  get_function_code       -0.10   full Solidity source of any function
+  get_state_variables     -0.05   list or inspect state variables
+  get_call_graph          -0.08   function call graph
+  get_formalized_property -0.03   formal pre/post-condition version of property
+  submit_function         terminal: +5.0 / +1.5 / -1.5  (ONE attempt)
+  repeated_query          -0.40
+
+Difficulty: Easy
+  The property text directly names the invariant broken; reading 2-3 functions
+  should let most agents identify the culprit quickly.
+"""
+
+from __future__ import annotations
+
+import random
+from typing import Any, Dict, List, Optional, Set
+
+from data.data_loader import (
+    load_contracts,
+    sample_task3_episode,
+    get_function_by_name,
+    get_state_variable_by_name,
+    list_function_names,
+    list_state_variable_names,
+)
+from env.base_env import BaseEnv
+from env.schemas import (
+    Action,
+    ActionType,
+    Observation,
+    Reward,
+    ResetResult,
+    StateResult,
+    StepResult,
+)
+from tasks.task3.grader import Task3Grader
+
+TASK_ID   = "task3_rule_checker"
+MAX_STEPS = 15
+
+AVAILABLE_ACTIONS = [
+    ActionType.LIST_FUNCTIONS,
+    ActionType.GET_FUNCTION_METADATA,
+    ActionType.GET_FUNCTION_CODE,
+    ActionType.GET_STATE_VARIABLE,
+    ActionType.GET_CALL_GRAPH,
+    ActionType.GET_FORMALIZED_PROPERTY,
+    ActionType.SUBMIT_FUNCTION,
+]
+
+
+class Task3Environment(BaseEnv):
+    """Task 3: Rule Checker — identify the function that violates a given property."""
+
+    def __init__(self, contracts_path: Optional[str] = None) -> None:
+        self._contracts = load_contracts(contracts_path) if contracts_path else load_contracts()
+        self._rng = random.Random()
+
+        # Episode state — initialised by reset()
+        self._contract:   Dict[str, Any] = {}
+        self._target_fn:  Dict[str, Any] = {}
+        self._grader:     Optional[Task3Grader] = None
+        self._step_count: int = 0
+        self._cum_reward: float = 0.0
+        self._done:       bool = False
+        self._submitted:  bool = False
+        self._query_hist: List[str] = []
+        self._seen:       Set[str] = set()
+
+    # ── OpenEnv interface ─────────────────────────────────────────────────────
+
+    def reset(self, seed: Optional[int] = None) -> ResetResult:
+        if seed is not None:
+            self._rng.seed(seed)
+
+        self._contract, self._target_fn = sample_task3_episode(
+            self._contracts, self._rng
+        )
+        t3 = self._target_fn["task3"]
+        self._grader = Task3Grader(
+            target_function=self._target_fn["name"],
+            partial_credit_functions=t3.get("partial_credit_functions", []),
+            property_english=t3.get("property_english", ""),
+        )
+        self._step_count = 0
+        self._cum_reward = 0.0
+        self._done       = False
+        self._submitted  = False
+        self._query_hist = []
+        self._seen       = set()
+
+        obs = self._build_obs(
+            last_action=None,
+            last_result=(
+                f"New episode started.\n"
+                f"Contract : {self._contract['contract_name']}\n\n"
+                f"Property : {t3['property_english']}\n\n"
+                f"Find the function in this contract that violates the property above.\n"
+                f"Use list_functions then get_function_code to investigate.\n"
+                f"Submit with submit_function, params={{\"function_name\": \"...\"}}.\n"
+                f"ONE submission allowed."
+            ),
+        )
+        return ResetResult(observation=obs, info={"task_id": TASK_ID})
+
+    def step(self, action: Action) -> StepResult:
+        if self._done:
+            raise RuntimeError("Episode is done. Call reset() to start a new episode.")
+
+        self._step_count += 1
+        result_text, reward = self._dispatch(action)
+        self._cum_reward += reward.value
+        self._query_hist.append(f"[{action.action_type}] → {result_text[:100]}")
+
+        obs = self._build_obs(
+            last_action=action.action_type,
+            last_result=result_text,
+        )
+        return StepResult(
+            observation=obs,
+            reward=reward,
+            done=self._done,
+            info={"step": self._step_count, "cumulative_reward": self._cum_reward},
+        )
+
+    def state(self) -> StateResult:
+        return StateResult(
+            task_id=TASK_ID,
+            contract_name=self._contract.get("contract_name", ""),
+            target_function=self._target_fn.get("name"),
+            step_count=self._step_count,
+            cumulative_reward=self._cum_reward,
+            done=self._done,
+            query_history=list(self._query_hist),
+        )
+
+    # ── Internal helpers ──────────────────────────────────────────────────────
+
+    def _build_obs(self, last_action: Optional[str], last_result: str) -> Observation:
+        t3 = self._target_fn.get("task3", {})
+        return Observation(
+            task_id=TASK_ID,
+            contract_name=self._contract.get("contract_name", ""),
+            contract_description=self._contract.get("metadata", {}).get("description", ""),
+            available_actions=[a.value for a in AVAILABLE_ACTIONS],
+            last_action=last_action,
+            last_action_result=last_result,
+            step_count=self._step_count,
+            cumulative_reward=self._cum_reward,
+            done=self._done,
+            extra={
+                "property_english": t3.get("property_english", ""),
+                "solidity_version": self._contract.get("metadata", {}).get("solidity_version", ""),
+                "hint": (
+                    "Read the property, then inspect function code to find which one violates it. "
+                    "Submit with: submit_function, params={'function_name': '<name>'}. "
+                    "ONE submission per episode."
+                ),
+            },
+        )
+
+    def _qkey(self, at: str, params: Dict[str, Any]) -> str:
+        return f"{at}:{sorted(params.items())}"
+
+    def _is_repeated(self, key: str) -> bool:
+        if key in self._seen:
+            return True
+        self._seen.add(key)
+        return False
+
+    def _dispatch(self, action: Action) -> tuple[str, Reward]:
+        at     = action.action_type
+        params = action.params
+        qkey   = self._qkey(at, params)
+
+        # ── list_functions ────────────────────────────────────────────────────
+        if at == ActionType.LIST_FUNCTIONS:
+            if self._is_repeated(qkey):
+                return "Repeated query.", Reward(value=-0.40, reason="Repeated query")
+            names = list_function_names(self._contract)
+            return (
+                f"Functions in {self._contract['contract_name']}: {', '.join(names)}",
+                Reward(value=-0.05, reason="list_functions cost"),
+            )
+
+        # ── get_function_metadata ─────────────────────────────────────────────
+        if at == ActionType.GET_FUNCTION_METADATA:
+            fn_name = params.get("function_name", "")
+            if self._is_repeated(qkey):
+                return "Repeated query.", Reward(value=-0.40, reason="Repeated query")
+            fn = get_function_by_name(self._contract, fn_name)
+            if fn is None:
+                return (
+                    f"Function '{fn_name}' not found. "
+                    f"Available: {list_function_names(self._contract)}",
+                    Reward(value=-0.05, reason="Unknown function"),
+                )
+            params_list = fn.get("parameters", [])
+            modifiers   = fn.get("modifiers", [])
+            lines = [
+                f"Function   : {fn.get('signature', fn_name)}",
+                f"Visibility : {fn.get('visibility', 'unknown')}",
+                f"Modifiers  : {', '.join(modifiers) if modifiers else 'none'}",
+            ]
+            if params_list:
+                lines.append("Parameters :")
+                for p in params_list:
+                    lines.append(f"  {p['type']} {p['name']} — {p.get('description','')}")
+            else:
+                lines.append("Parameters : none")
+            lines.append(f"Returns    : {fn.get('returns','') or 'void'}")
+            lines.append(f"Summary    : {fn.get('comment','')}")
+            return "\n".join(lines), Reward(value=-0.05, reason="get_function_metadata cost")
+
+        # ── get_function_code ─────────────────────────────────────────────────
+        if at == ActionType.GET_FUNCTION_CODE:
+            fn_name = params.get("function_name", "")
+            if self._is_repeated(qkey):
+                return "Repeated query.", Reward(value=-0.40, reason="Repeated query")
+            fn = get_function_by_name(self._contract, fn_name)
+            if fn is None:
+                return (
+                    f"Function '{fn_name}' not found. "
+                    f"Available: {list_function_names(self._contract)}",
+                    Reward(value=-0.10, reason="Unknown function — extra penalty"),
+                )
+            code = fn.get("code", "// no code available")
+            return (
+                f"// {fn_name}\n{code}",
+                Reward(value=-0.10, reason="get_function_code cost"),
+            )
+
+        # ── get_state_variables ───────────────────────────────────────────────
+        if at == ActionType.GET_STATE_VARIABLE:
+            var_name = params.get("variable_name", "")
+            if self._is_repeated(qkey):
+                return "Repeated query.", Reward(value=-0.40, reason="Repeated query")
+            if not var_name:
+                names = list_state_variable_names(self._contract)
+                return (
+                    f"State variables: {', '.join(names)}",
+                    Reward(value=-0.05, reason="Listed state variables"),
+                )
+            sv = get_state_variable_by_name(self._contract, var_name)
+            if sv is None:
+                return (
+                    f"Variable '{var_name}' not found.",
+                    Reward(value=-0.05, reason="Unknown state variable"),
+                )
+            return (
+                f"{sv['type']} {sv['visibility']} {sv['name']}: {sv.get('description','')}",
+                Reward(value=-0.05, reason="get_state_variable cost"),
+            )
+
+        # ── get_call_graph ────────────────────────────────────────────────────
+        if at == ActionType.GET_CALL_GRAPH:
+            if self._is_repeated(qkey):
+                return "Repeated query.", Reward(value=-0.40, reason="Repeated query")
+            cg  = self._contract.get("call_graph", {})
+            cg_str = "; ".join(
+                f"{fn} → [{', '.join(callees)}]" for fn, callees in cg.items()
+            )
+            return (
+                f"Call graph: {cg_str}",
+                Reward(value=-0.08, reason="get_call_graph cost"),
+            )
+
+        # ── get_formalized_property ───────────────────────────────────────────
+        if at == ActionType.GET_FORMALIZED_PROPERTY:
+            if self._is_repeated(qkey):
+                return "Repeated query.", Reward(value=-0.40, reason="Repeated query")
+            formal = self._target_fn.get("task3", {}).get("property_formal", "")
+            if not formal:
+                formal = "No formal specification available for this property."
+            return (
+                f"Formal property:\n{formal}",
+                Reward(value=-0.03, reason="get_formalized_property cost"),
+            )
+
+        # ── submit_function ───────────────────────────────────────────────────
+        if at == ActionType.SUBMIT_FUNCTION:
+            if self._submitted:
+                return (
+                    "❌ You have already submitted for this episode. "
+                    "Only ONE submission is allowed.",
+                    Reward(value=-1.0, reason="Second submit_function attempt", partial=False),
+                )
+            fn_name = params.get("function_name", "").strip()
+            if not fn_name:
+                return (
+                    "submit_function requires 'function_name' in params.",
+                    Reward(value=-0.5, reason="Malformed submission"),
+                )
+
+            self._submitted = True
+            self._done      = True
+            score, reward_val = self._grader.grade_and_reward(fn_name)
+            correct = self._grader.get_canonical_answer()
+
+            if score >= 0.9:
+                msg = (
+                    f"✅ CORRECT! '{fn_name}' is the function that violates the property. "
+                    f"Score: 1.0 → Reward: +{reward_val:.1f}"
+                )
+            elif score >= 0.2:
+                msg = (
+                    f"🟡 PARTIAL. '{fn_name}' is a subfunction of the target — "
+                    f"closely related but not the primary rule-breaker. "
+                    f"Score: 0.3 → Reward: +{reward_val:.1f}. "
+                    f"Correct answer: '{correct['target_function']}'."
+                )
+            else:
+                msg = (
+                    f"❌ INCORRECT. '{fn_name}' does not violate the property. "
+                    f"Score: 0.0 → Reward: {reward_val:.1f}. "
+                    f"Correct answer: '{correct['target_function']}'."
+                )
+
+            return msg, Reward(
+                value=reward_val,
+                reason=f"submit_function score={score:.1f}",
+                partial=False,
+            )
+
+        # ── unknown action ────────────────────────────────────────────────────
+        return (
+            f"Unknown action '{at}'. Valid: {[a.value for a in AVAILABLE_ACTIONS]}",
+            Reward(value=-0.10, reason="Unknown action"),
+        )

tasks/task3/grader.py

@@ -0,0 +1,80 @@
+"""
+grader.py  (Task 3 – Rule Checker)
+------------------------------------
+Deterministic grader for function-identification submissions.
+
+Score table
+───────────
+  1.0  → submitted function is the exact target (case-insensitive)
+  0.3  → submitted function is a direct internal subfunction of the target
+         (a contract-internal function called by the target in the call graph)
+  0.0  → anything else
+
+Reward table (ONE submission per episode)
+  score 1.0  → +5.0
+  score 0.3  → +1.5
+  score 0.0  → -1.5
+"""
+
+from __future__ import annotations
+from typing import Dict, List, Optional
+
+
+class Task3Grader:
+    """
+    Grades a Task 3 submit_function submission.
+
+    Parameters
+    ----------
+    target_function         : exact name of the rule-breaking function
+    partial_credit_functions: list of internal functions that get partial credit
+                              (direct callees of the target that are contract functions)
+    property_english        : the English property text (for feedback messages)
+    """
+
+    SCORE_CORRECT  = 1.0
+    SCORE_PARTIAL  = 0.3
+    SCORE_WRONG    = 0.0
+
+    REWARD_CORRECT = 5.0
+    REWARD_PARTIAL = 1.5
+    REWARD_WRONG   = -1.5
+
+    def __init__(
+        self,
+        target_function: str,
+        partial_credit_functions: List[str],
+        property_english: str = "",
+    ) -> None:
+        self.target_function          = target_function.lower()
+        self.partial_credit_functions = [f.lower() for f in partial_credit_functions]
+        self.property_english         = property_english
+
+    def grade(self, submitted_function: str) -> float:
+        """Returns deterministic score in {0.0, 0.3, 1.0}."""
+        norm = submitted_function.strip().lower()
+        if norm == self.target_function:
+            return self.SCORE_CORRECT
+        if norm in self.partial_credit_functions:
+            return self.SCORE_PARTIAL
+        return self.SCORE_WRONG
+
+    def reward_for_score(self, score: float) -> float:
+        """Maps score → terminal reward."""
+        if score >= 0.9:
+            return self.REWARD_CORRECT
+        if score >= 0.2:
+            return self.REWARD_PARTIAL
+        return self.REWARD_WRONG
+
+    def grade_and_reward(self, submitted_function: str):
+        """Convenience: returns (score, reward)."""
+        score = self.grade(submitted_function)
+        return score, self.reward_for_score(score)
+
+    def get_canonical_answer(self) -> Dict[str, object]:
+        """For debugging / logging only — do not expose to the agent."""
+        return {
+            "target_function":          self.target_function,
+            "partial_credit_functions": self.partial_credit_functions,
+        }

validate.py

@@ -1,19 +1,18 @@
 """
 validate.py
 -----------
-Pre-submission validation. Checks all OpenEnv spec requirements.
-
-Usage:  python validate.py
-Exit 0 = all checks pass.  Exit 1 = one or more failures.
+Pre-submission validation — 24 checks across all three tasks.
+Usage: python validate.py
+Exit 0 = all pass. Exit 1 = failures.
 """
 
-import json, sys, traceback
+import json, sys
 from typing import Callable, List, Tuple
 
 PASS = "✅"; FAIL = "❌"
 results: List[Tuple[str, bool, str]] = []
 
-def check(name: str, fn: Callable[[], None]) -> None:
+def check(name: str, fn: Callable) -> None:
     try:
         fn(); results.append((name, True, ""))
         print(f"  {PASS} {name}")
@@ -25,45 +24,41 @@ def check(name: str, fn: Callable[[], None]) -> None:
 
 def check_imports():
     from env.schemas import Observation, Action, Reward, StepResult, ResetResult, StateResult, ActionType
-    from tasks.task1.environment import Task1Environment
-    from tasks.task1.grader import Task1Grader
-    from tasks.task2.environment import Task2Environment
-    from tasks.task2.grader import Task2Grader
+    from tasks.task1.environment import Task1Environment; from tasks.task1.grader import Task1Grader
+    from tasks.task2.environment import Task2Environment; from tasks.task2.grader import Task2Grader
+    from tasks.task3.environment import Task3Environment; from tasks.task3.grader import Task3Grader
     from data.data_loader import load_contracts
 
 def check_openenv_yaml():
     import yaml
     with open("openenv.yaml") as f: spec = yaml.safe_load(f)
-    assert "name" in spec
-    assert len(spec.get("tasks", [])) >= 3
-    assert "observation_space" in spec
-    assert "action_space" in spec
-    assert "reward" in spec
+    assert "name" in spec and len(spec.get("tasks", [])) >= 3
+    assert "observation_space" in spec and "action_space" in spec and "reward" in spec
+    tasks = spec["tasks"]
+    active = [t for t in tasks if t.get("status") == "active"]
+    assert len(active) >= 2, f"Expected >=2 active tasks, got {len(active)}"
 
 def check_pydantic_models():
-    from env.schemas import Observation, Action, ActionType, Reward, StepResult, ResetResult, StateResult
-    obs = Observation(task_id="t1", contract_name="C", contract_description="D", available_actions=["submit"])
-    assert obs.task_id == "t1"
-    action = Action(action_type=ActionType.LIST_FUNCTIONS); assert action.action_type == ActionType.LIST_FUNCTIONS
-    action2 = Action(action_type=ActionType.SUBMIT_PROPERTY); assert action2.action_type == ActionType.SUBMIT_PROPERTY
-    reward = Reward(value=1.0, reason="test"); assert reward.value == 1.0
-    step = StepResult(observation=obs, reward=reward, done=False); assert not step.done
-    reset = ResetResult(observation=obs); assert reset.observation.task_id == "t1"
+    from env.schemas import Observation, Action, ActionType, Reward, StepResult, ResetResult
+    obs = Observation(task_id="t", contract_name="C", contract_description="D", available_actions=[])
+    for at in [ActionType.LIST_FUNCTIONS, ActionType.SUBMIT_PROPERTY,
+               ActionType.GET_FORMALIZED_PROPERTY, ActionType.SUBMIT_FUNCTION]:
+        Action(action_type=at)
+    Reward(value=-1.5, reason="test")
+    StepResult(observation=obs, reward=Reward(value=0, reason=""), done=False)
 
 def check_data_loading():
-    from data.data_loader import load_contracts, get_all_vulnerable_entries, get_all_property_entries
-    contracts = load_contracts()
-    assert len(contracts) >= 1
-    vuln_entries = get_all_vulnerable_entries(contracts)
-    assert len(vuln_entries) >= 3, f"Need >=3 vulnerable fns, got {len(vuln_entries)}"
-    prop_entries = get_all_property_entries(contracts)
-    assert len(prop_entries) >= 3, f"Need >=3 property fns, got {len(prop_entries)}"
-    for _, fn in prop_entries:
-        p = fn["property"]
-        assert "natural_language" in p
-        assert "key_phrases" in p
-        assert "bonus_phrases" in p
-        assert len(p["key_phrases"]) >= 2
+    from data.data_loader import (load_contracts, get_all_vulnerable_entries,
+                                   get_all_property_entries, get_all_task3_entries)
+    c = load_contracts()
+    assert len(get_all_vulnerable_entries(c)) >= 3
+    assert len(get_all_property_entries(c)) >= 3
+    entries = get_all_task3_entries(c)
+    assert len(entries) >= 3, f"Need >=3 task3 entries, got {len(entries)}"
+    for _, fn in entries:
+        t3 = fn.get("task3", {})
+        assert t3.get("property_english"), f"{fn['name']} missing property_english"
+        assert t3.get("property_formal"),  f"{fn['name']} missing property_formal"
 
 def check_t1_env():
     from tasks.task1.environment import Task1Environment
@@ -71,9 +66,8 @@ def check_t1_env():
     env = Task1Environment()
     r = env.reset(seed=42); assert r.observation.task_id == "task1_vuln_detection"
     s = env.step(Action(action_type=ActionType.LIST_FUNCTIONS))
-    assert isinstance(s.reward.value, float)
-    assert s.observation.step_count == 1
-    st = env.state(); assert st.target_function is not None
+    assert s.reward.value == -0.05 and s.observation.step_count == 1
+    assert env.state().target_function is not None
 
 def check_t2_env():
     from tasks.task2.environment import Task2Environment
@@ -82,178 +76,206 @@ def check_t2_env():
     r = env.reset(seed=42)
     assert r.observation.task_id == "task2_property_discovery"
     assert "target_function" in r.observation.extra
-    # test each action type
     for at in [ActionType.GET_FUNCTION_CODE, ActionType.GET_FUNCTION_NATSPEC,
-               ActionType.GET_FILE_NATSPEC, ActionType.GET_IO, ActionType.GET_RELATED_FUNCTIONS]:
-        s = env.step(Action(action_type=at)); assert s.reward.value < 0
-    s = env.step(Action(action_type=ActionType.GET_SIMILAR_RULE))
-    assert s.reward.value == -0.20
+               ActionType.GET_FILE_NATSPEC, ActionType.GET_IO,
+               ActionType.GET_RELATED_FUNCTIONS, ActionType.GET_SIMILAR_RULE]:
+        env.step(Action(action_type=at))
 
-def check_t2_env_submit():
-    from tasks.task2.environment import Task2Environment
-    from data.data_loader import load_contracts, get_function_by_name
+def check_t3_env():
+    from tasks.task3.environment import Task3Environment
     from env.schemas import Action, ActionType
-    env = Task2Environment()
+    env = Task3Environment()
     r = env.reset(seed=42)
-    fn_name  = r.observation.extra["target_function"]
-    contract = r.observation.contract_name
-    contracts = load_contracts()
-    gt_text = ""
-    for c in contracts:
-        if c["contract_name"] == contract:
-            fn = get_function_by_name(c, fn_name)
-            if fn and fn.get("property"):
-                gt_text = fn["property"]["natural_language"]
-    result = env.step(Action(action_type=ActionType.SUBMIT_PROPERTY, params={"property": gt_text}))
-    assert result.done
-    assert result.reward.value > 0, f"GT text should score >0, got {result.reward.value}"
-
-def check_t2_one_submit_only():
-    from tasks.task2.environment import Task2Environment
+    assert r.observation.task_id == "task3_rule_checker"
+    assert "property_english" in r.observation.extra
+    prop = r.observation.extra["property_english"]
+    assert len(prop) > 10, "property_english too short"
+    for at in [ActionType.LIST_FUNCTIONS, ActionType.GET_FORMALIZED_PROPERTY,
+               ActionType.GET_CALL_GRAPH, ActionType.GET_STATE_VARIABLE]:
+        s = env.step(Action(action_type=at))
+        assert s.reward.value < 0, f"{at.value} should have negative shaping reward"
+
+def check_t3_action_costs():
+    from tasks.task3.environment import Task3Environment
     from env.schemas import Action, ActionType
-    env = Task2Environment()
-    env.reset(seed=5)
-    env.step(Action(action_type=ActionType.SUBMIT_PROPERTY, params={"property": "test"}))
-    # Second submit must either fail (episode done → RuntimeError) or return negative reward
-    try:
-        s2 = env.step(Action(action_type=ActionType.SUBMIT_PROPERTY, params={"property": "test2"}))
-        # If it doesn't raise, the reward must be negative
-        assert s2.reward.value < 0, "Second submit should penalise"
-    except RuntimeError:
-        pass  # expected
+    env = Task3Environment(); env.reset(seed=42)
+    costs = {
+        ActionType.GET_FORMALIZED_PROPERTY: -0.03,
+        ActionType.LIST_FUNCTIONS: -0.05,
+        ActionType.GET_CALL_GRAPH: -0.08,
+    }
+    for at, expected in costs.items():
+        e2 = Task3Environment(); e2.reset(seed=42)
+        s = e2.step(Action(action_type=at))
+        assert abs(s.reward.value - expected) < 0.001, \
+            f"{at.value}: expected {expected}, got {s.reward.value}"
 
-def check_t1_grader():
-    from tasks.task1.grader import Task1Grader
-    cases = [
-        ("withdraw", "Reentrancy vulnerability",  "withdraw", "reentrancy",             1.0),
-        ("withdraw", "Reentrancy vulnerability",  "withdraw", "something else",          0.5),
-        ("withdraw", "Reentrancy vulnerability",  "deposit",  "reentrancy",             0.0),
-    ]
-    for tf, issue, sf, sv, expected in cases:
-        g = Task1Grader(tf, issue)
-        score = g.grade_submission(sf, sv)
-        assert 0.0 <= score <= 1.0
-        assert abs(score - expected) < 0.01, f"Expected {expected}, got {score}"
+def check_t3_function_metadata():
+    from tasks.task3.environment import Task3Environment
+    from env.schemas import Action, ActionType
+    env = Task3Environment(); env.reset(seed=43)
+    s = env.step(Action(action_type=ActionType.GET_FUNCTION_METADATA,
+                         params={"function_name": "withdraw"}))
+    assert "Visibility" in s.observation.last_action_result
+    assert s.reward.value == -0.05
 
-def check_t2_grader():
-    from tasks.task2.grader import Task2Grader
-    from data.data_loader import load_contracts, get_all_property_entries
-    contracts = load_contracts()
-    entries = get_all_property_entries(contracts)
-    for contract, fn in entries:
-        g = Task2Grader(fn["name"], fn["property"])
-        # Ground truth must score ≥ 0.65
-        gt_score = g.grade(fn["property"]["natural_language"])
-        assert gt_score >= 0.65, f"{fn['name']}: gt_score={gt_score} < 0.65"
-        # Empty must be 0.0
-        assert g.grade("") == 0.0
-        # Deterministic
-        assert g.grade("test text") == g.grade("test text")
-        # Score in [0,1]
-        assert 0.0 <= gt_score <= 1.0
-        # Reward maps correctly
-        assert abs(g.reward_for_score(gt_score) - gt_score * 5.0) < 0.01
+def check_t3_submit_correct():
+    from tasks.task3.environment import Task3Environment
+    from env.schemas import Action, ActionType
+    env = Task3Environment(); env.reset(seed=42)
+    target = env.state().target_function
+    s = env.step(Action(action_type=ActionType.SUBMIT_FUNCTION,
+                         params={"function_name": target}))
+    assert s.done and s.reward.value == 5.0, \
+        f"Expected reward=5.0, got {s.reward.value}"
 
-def check_reward_shaping():
-    from tasks.task2.environment import Task2Environment
+def check_t3_submit_subfunction():
+    from tasks.task3.environment import Task3Environment
     from env.schemas import Action, ActionType
-    env = Task2Environment()
-    env.reset(seed=1)
-    rewards = {env.step(Action(action_type=at)).reward.value
-               for at in [ActionType.GET_FUNCTION_CODE, ActionType.GET_FILE_NATSPEC, ActionType.GET_IO]}
-    assert len(rewards) >= 2, f"Need multiple reward values, got {rewards}"
+    # seed 45 → bid with subfunction getPrice
+    env = Task3Environment(); env.reset(seed=45)
+    assert env.state().target_function == "bid"
+    s = env.step(Action(action_type=ActionType.SUBMIT_FUNCTION,
+                         params={"function_name": "getPrice"}))
+    assert s.done and s.reward.value == 1.5, \
+        f"Expected partial reward=1.5, got {s.reward.value}"
 
-def check_t1_episode_boundary():
-    from tasks.task1.environment import Task1Environment
+def check_t3_submit_wrong():
+    from tasks.task3.environment import Task3Environment
     from env.schemas import Action, ActionType
-    env = Task1Environment()
-    env.reset(seed=2)
-    env.step(Action(action_type=ActionType.SUBMIT,
-                    params={"function_name": "withdraw", "vulnerability_type": "test"}))
+    env = Task3Environment(); env.reset(seed=42)
+    s = env.step(Action(action_type=ActionType.SUBMIT_FUNCTION,
+                         params={"function_name": "constructor"}))
+    assert s.done and s.reward.value == -1.5
+
+def check_t3_one_submit_only():
+    from tasks.task3.environment import Task3Environment
+    from env.schemas import Action, ActionType
+    env = Task3Environment(); env.reset(seed=42)
+    env.step(Action(action_type=ActionType.SUBMIT_FUNCTION,
+                     params={"function_name": "deposit"}))
     try:
         env.step(Action(action_type=ActionType.LIST_FUNCTIONS))
         raise AssertionError("Should raise RuntimeError after done")
     except RuntimeError:
         pass
 
-def check_repeated_query_penalty():
-    from tasks.task1.environment import Task1Environment
+def check_t3_repeated_penalty():
+    from tasks.task3.environment import Task3Environment
     from env.schemas import Action, ActionType
-    env = Task1Environment(); env.reset(seed=3)
+    env = Task3Environment(); env.reset(seed=42)
     env.step(Action(action_type=ActionType.LIST_FUNCTIONS))
-    r = env.step(Action(action_type=ActionType.LIST_FUNCTIONS))
-    assert r.reward.value == -0.40
+    s = env.step(Action(action_type=ActionType.LIST_FUNCTIONS))
+    assert s.reward.value == -0.40
 
-def check_t2_repeated_penalty():
-    from tasks.task2.environment import Task2Environment
+def check_t1_grader():
+    from tasks.task1.grader import Task1Grader
+    g = Task1Grader("withdraw", "Reentrancy vulnerability")
+    assert g.grade_submission("withdraw", "reentrancy") == 1.0
+    assert g.grade_submission("withdraw", "vague") == 0.5
+    assert g.grade_submission("deposit", "reentrancy") == 0.0
+
+def check_t2_grader():
+    from tasks.task2.grader import Task2Grader
+    from data.data_loader import load_contracts, get_all_property_entries
+    for c, fn in get_all_property_entries(load_contracts()):
+        g = Task2Grader(fn["name"], fn["property"])
+        assert g.grade(fn["property"]["natural_language"]) >= 0.65
+        assert g.grade("") == 0.0
+        s = g.grade("test"); assert s == g.grade("test")  # deterministic
+
+def check_t3_grader():
+    from tasks.task3.grader import Task3Grader
+    g = Task3Grader("withdraw", ["deposit"], "some rule")
+    assert g.grade("withdraw") == 1.0
+    assert g.grade("WITHDRAW") == 1.0  # case-insensitive
+    assert g.grade("deposit") == 0.3
+    assert g.grade("constructor") == 0.0
+    s, r = g.grade_and_reward("withdraw"); assert s == 1.0 and r == 5.0
+    s, r = g.grade_and_reward("deposit");  assert s == 0.3 and r == 1.5
+    s, r = g.grade_and_reward("other");    assert s == 0.0 and r == -1.5
+
+def check_reward_shaping():
+    from tasks.task3.environment import Task3Environment
     from env.schemas import Action, ActionType
-    env = Task2Environment(); env.reset(seed=3)
-    env.step(Action(action_type=ActionType.GET_FUNCTION_CODE))
-    r = env.step(Action(action_type=ActionType.GET_FUNCTION_CODE))
-    assert r.reward.value == -0.40
+    env = Task3Environment(); env.reset(seed=1)
+    rewards = {env.step(Action(action_type=at)).reward.value
+               for at in [ActionType.LIST_FUNCTIONS,
+                           ActionType.GET_FORMALIZED_PROPERTY,
+                           ActionType.GET_CALL_GRAPH]}
+    assert len(rewards) >= 2
 
-def check_task_placeholders():
-    from tasks.task3 import __all__ as t3
+def check_app_imports():
+    from app import app
+    from fastapi.testclient import TestClient
+    client = TestClient(app)
+    r = client.get("/health"); assert r.status_code == 200
+    tasks = client.get("/tasks").json()["tasks"]
+    active = [t for t in tasks if t["status"] == "active"]
+    assert len(active) == 3, f"Expected 3 active tasks, got {len(active)}: {active}"
+
+def check_t3_http_reset():
+    from app import app
+    from fastapi.testclient import TestClient
+    client = TestClient(app)
+    r = client.post("/reset", json={"task_id": "task3_rule_checker", "seed": 42})
+    assert r.status_code == 200
+    obs = r.json()["observation"]
+    assert obs["task_id"] == "task3_rule_checker"
+    assert "property_english" in obs["extra"]
 
 def check_dockerfile():
     import os
     assert os.path.exists("Dockerfile")
-    with open("Dockerfile") as f: c = f.read()
-    assert "7860" in c
-    assert "uvicorn" in c or "CMD" in c
+    c = open("Dockerfile").read()
+    assert "7860" in c and ("uvicorn" in c or "CMD" in c)
 
 def check_inference_script():
     import os
     assert os.path.exists("inference.py")
-    with open("inference.py") as f: c = f.read()
-    assert "HF_TOKEN" in c
-    assert "API_BASE_URL" in c
-    assert "MODEL_NAME" in c
-    assert "task2" in c.lower() or "Task2" in c or "TASK 2" in c
+    c = open("inference.py").read()
+    assert "HF_TOKEN" in c and "API_BASE_URL" in c and "MODEL_NAME" in c
+    assert "Task3Environment" in c or "run_task3" in c
+    assert "submit_function" in c
 
 def check_baseline_json():
     import os
     if not os.path.exists("baseline_scores.json"): return
-    with open("baseline_scores.json") as f: data = json.load(f)
-    assert "tasks" in data
-    for t in data["tasks"]:
+    data = json.load(open("baseline_scores.json"))
+    for t in data.get("tasks", []):
         assert 0.0 <= t["avg_grader_score"] <= 1.0
 
-def check_similar_rule_lookup():
-    from data.data_loader import load_contracts, get_similar_rule
-    contracts = load_contracts()
-    sr = get_similar_rule(contracts, "SimpleVault", "withdraw")
-    assert sr is not None, "similar_rule should exist for withdraw"
-    assert "property_hint" in sr
-    assert "contract_name" in sr
-
 # ── Runner ────────────────────────────────────────────────────────────────────
 
 ALL_CHECKS = [
-    ("Python imports (T1 + T2)",           check_imports),
-    ("openenv.yaml format",                 check_openenv_yaml),
-    ("Pydantic models (incl T2 actions)",   check_pydantic_models),
-    ("Dataset: vuln + property entries",    check_data_loading),
-    ("Task 1: reset / step / state",        check_t1_env),
-    ("Task 2: reset + all 6 browse actions",check_t2_env),
-    ("Task 2: submit_property scores > 0",  check_t2_env_submit),
-    ("Task 2: one submit only",             check_t2_one_submit_only),
-    ("Task 1 grader: 0/0.5/1.0 rubric",    check_t1_grader),
-    ("Task 2 grader: all 11 properties",    check_t2_grader),
-    ("Reward shaping (multi-value)",        check_reward_shaping),
-    ("T1 episode boundary",                 check_t1_episode_boundary),
-    ("T1 repeated query penalty (-0.40)",   check_repeated_query_penalty),
-    ("T2 repeated query penalty (-0.40)",   check_t2_repeated_penalty),
-    ("Task 3 placeholder exists",           check_task_placeholders),
+    ("Python imports (T1+T2+T3)",           check_imports),
+    ("openenv.yaml: 3 tasks, ≥2 active",    check_openenv_yaml),
+    ("Pydantic models (all ActionTypes)",   check_pydantic_models),
+    ("Dataset: vuln+property+task3 entries",check_data_loading),
+    ("T1 env: reset/step/state",            check_t1_env),
+    ("T2 env: reset + 6 browse actions",    check_t2_env),
+    ("T3 env: reset + browse actions",      check_t3_env),
+    ("T3 action costs (formalized -0.03)",  check_t3_action_costs),
+    ("T3 get_function_metadata",            check_t3_function_metadata),
+    ("T3 submit correct → +5.0",            check_t3_submit_correct),
+    ("T3 submit subfunction → +1.5",        check_t3_submit_subfunction),
+    ("T3 submit wrong → -1.5",              check_t3_submit_wrong),
+    ("T3 one submit per episode",           check_t3_one_submit_only),
+    ("T3 repeated query → -0.40",           check_t3_repeated_penalty),
+    ("T1 grader: 0/0.5/1.0 rubric",        check_t1_grader),
+    ("T2 grader: all 11 properties",        check_t2_grader),
+    ("T3 grader: 1.0/0.3/0.0 + case-ins.", check_t3_grader),
+    ("Reward shaping non-binary (T3)",      check_reward_shaping),
+    ("FastAPI: 3 active tasks",             check_app_imports),
+    ("FastAPI: T3 reset endpoint",          check_t3_http_reset),
     ("Dockerfile + port 7860",              check_dockerfile),
-    ("inference.py: creds + Task 2 code",   check_inference_script),
+    ("inference.py: T3 code present",       check_inference_script),
     ("baseline_scores.json schema",         check_baseline_json),
-    ("similar_rule data lookup",            check_similar_rule_lookup),
 ]
 
 def main():
     print("=" * 64)
-    print("OpenEnv Pre-Submission Validation  (Task 1 + Task 2)")
+    print("OpenEnv Pre-Submission Validation  (Task 1 + 2 + 3)")
     print("=" * 64)
     print()
     for name, fn in ALL_CHECKS:
@@ -270,7 +292,7 @@ def main():
         print("\nFailed checks:")
         for n, m in failed:
             print(f"  {FAIL} {n}: {m}")
-        print("\n❌ VALIDATION FAILED — fix the issues above before submitting.")
+        print("\n❌ VALIDATION FAILED")
         sys.exit(1)
     else:
         print("\n✅ ALL CHECKS PASSED — ready to submit!")