Add session checkpoint: v3 launch decision with full context

docs/checkpoints/2026-04-23_v3-launch.md

@@ -0,0 +1,208 @@
+# Session Checkpoint — 2026-04-23 22:30 CEST
+
+## Status: v3 Training Launched (Cell 11)
+
+Probe passed (3 steps). Full training run initiated: 500 steps, ~25h estimated on L4.
+
+---
+
+## Context
+
+### Where we are in the pipeline
+```
+Qwen3-4B-Base → Polygl0t/Tucano2-qwen-3.7B-Base (PT continual pretrain)
+    → Polygl0t/Tucano2-qwen-3.7B-Think (SFT + thinking training)
+        → YOUR SFT adapter (domain e-commerce, 1650 samples)
+            → GRPO v2 (210 steps, early stopped) — +42% over SFT baseline
+                → GRPO v3 (launching now) — all fixes from ADR-001 + thinking control patch
+```
+
+### Training data
+- 1,404 prompts after 15% eval holdout (from ~1,650 total)
+- Distribution: extraction=659, sql_qa=655, insights=114, push=222
+- Using ALL data (v2 used 300 subset → memorization → entropy collapse)
+
+### Hardware
+- NVIDIA L4 (24GB VRAM), Vertex AI Workbench
+- Unsloth 2026.4.8, TRL 0.24.0 (pinned), Transformers 5.5.0
+- Peak VRAM in smoke test: 6.8GB / 23.6GB (massive headroom)
+
+### v2 results (baseline to beat)
+- 210/300 steps, early stopped at eval plateau
+- Validation mean reward: 0.54 (+42% vs SFT calibration 0.38)
+- Strong on insights/analysis (0.50-0.70), broken on extraction (0.12)
+- Critical issues: entropy collapse (clip_ratio=0), completion ceiling (100% at 2048), KL=0.004
+
+---
+
+## Problem
+
+### Problem 1: Think model's `<think>` blocks consume all tokens
+The model generates 2000-3000 tokens of `<think>` content before producing answers. At both 2048 (v2) and 4096 (v3) completion ceilings, extraction tasks never produce JSON — the model is stuck in `<think>` at inference time with low temperature.
+
+**Evidence from v3 calibration (Cell 7, temp=0.7):**
+- 8/8 samples hit 4096 ceiling
+- Both extraction samples: stuck in `<think>`, reward=0.11-0.12
+- Task-aware system prompts ("Não pense em excesso") had ZERO measurable effect
+
+**However**, during GRPO training rollouts (temp=1.0):
+- Smoke test: mean completion=528 tokens, 0% ceiling hits
+- Probe step 2: mean completion=358 tokens, 0% ceiling hits
+- Probe step 3: mean completion=1371 tokens, 25% ceiling hits (1 of 4)
+- High temperature produces diverse SHORT completions — the model doesn't lock into verbose thinking
+
+### Problem 2: Entropy collapse (inherited from v2)
+- v2: clip_ratio=0 on ALL steps, KL=0.004 — policy never moved
+- v3 probe: clip_ratio=0 on all 3 steps — but loss is nonzero (0.041 on step 3)
+- May resolve after warmup; entropy monitor callback will detect if it persists
+
+### Problem 3: Think model has no thinking toggle
+- Checked Polygl0t/Tucano2-qwen-3.7B-Think model files
+- `generation_config.json`: temperature=0.1, max_new_tokens=1024, no thinking control
+- `chat_template.jinja`: always injects `<think>` on last assistant turn, no `enable_thinking` conditional
+- Unlike official Qwen3-4B which has `enable_thinking=True/False` toggle
+- Prompt-level control ("Não pense em excesso") proven ineffective at inference time
+- L1 paper (2503.04697) confirms: untrained models ignore length instructions — need RL reward to learn compliance
+
+---
+
+## Decisions Made
+
+### Decision 1: Proceed with v3 training on Think model despite ceiling issues
+- **Rationale**: Probe shows completions are SHORT during training (temp=1.0). The ceiling problem only manifests at low-temperature inference. GRPO rollouts at temp=1.0 produce 358-528 token completions on average. Training will work even if post-training inference needs tuning.
+- **Risk**: If model learns at temp=1.0 but can't transfer to temp=0.1 inference, we get good training metrics but poor deployment performance.
+
+### Decision 2: Task-aware system prompts (3-change patch)
+Applied and verified:
+- **Cell 3**: 4 task-specific system prompts (extraction, sql_qa, insights, push) + `THINK_BUDGETS` + `get_system_prompt()`
+- **Cell 6**: `reward_think_efficiency()` — penalizes bloated `<think>` blocks per task budget (extraction: 150 tok, push: 100, sql_qa: 400, insights: 800)
+- **Cell 7**: `inject_task_system_prompt()` wired into calibration
+- **Cell 8**: System prompt injection into training data via `prepare_grpo_datasets_v3()`
+- **Cell 13**: Validation uses per-task system prompts
+- **Research basis**: OptimalThinkingBench (2508.13141), Mid-Think (2601.07036), L1 (2503.04697)
+- **Observed effect**: Zero at inference (calibration). Unknown during training — the reward signal may teach compliance over hundreds of steps.
+
+### Decision 3: Plan base model training as next step
+- Literature review conclusive: every canonical GRPO paper starts from base/instruct, not thinking models
+- DeepSeek-R1-Zero proved thinking emerges from RL on base models
+- ThinkJSON (2502.14905) beats R1-671B on JSON extraction using Qwen2.5-1.5B BASE + GRPO
+- `Polygl0t/Tucano2-qwen-3.7B-Base` exists and has Portuguese continual pretraining
+- Will need to re-run SFT (LoRA adapters are model-specific, can't transfer Think→Base)
+
+### Decision 4: Did NOT add load_best_model_at_end
+- Requires native eval loop with `metric_for_best_model` — our eval is a custom callback
+- EvalRewardCallback tracks `best_reward` and `best_step` internally
+- `SAVE_STEPS=10` + `SAVE_TOTAL_LIMIT=5` = 50 steps of checkpoint coverage — sufficient
+
+---
+
+## v3 Config (all changes from v2)
+
+| Parameter | v2 | v3 | Paper Reference |
+|-----------|-----|-----|----------------|
+| Temperature | 0.8 | **1.0** | Skywork-OR1 (2505.22312) |
+| max_completion_length | 2048 | **4096** | Dr. GRPO (2503.20783) |
+| num_generations | 8 | **4** | MC-GRPO (2601.22582) — VRAM tradeoff |
+| learning_rate | 5e-7 | **2e-6** | Dr. GRPO Appendix G |
+| β (KL penalty) | implicit | **0.0** | Dr. GRPO §3.2 |
+| Training data | 300 subset | **ALL ~1400** | Skywork-OR1 §3.1 |
+| Rewards | single composite | **staged (format→partial→task)** | Reasoning-SQL (2503.23157) |
+| System prompts | single generic | **4 task-aware** | OptimalThinkingBench (2508.13141) |
+| Think efficiency reward | none | **reward_think_efficiency()** | L1 (2503.04697) |
+| Zero-advantage groups | included | **noise injection (σ=0.005)** | Skywork-OR1 §3.1 |
+| Entropy monitoring | none | **EntropyMonitorCallback** | Skywork-OR1 §4 |
+| grad_accum | 2 | **1** | Effective batch 4 (was 8) |
+| patience | 10 | **15** | More runway |
+| delta | 0.01 | **0.005** | More sensitive |
+| save_steps | 15 | **10** | Never lose best checkpoint |
+| save_total_limit | 3 | **5** | More checkpoint coverage |
+| eval_temperature | 0.7 | **0.1** | Deterministic eval |
+| eval_max_tokens | 2048 | **4096** | Match training |
+
+---
+
+## Probe Results (3 steps)
+
+| Step | Completion (mean) | Clipped | Reward | reward_std | Loss | clip_ratio |
+|------|:-:|:-:|:-:|:-:|:-:|:-:|
+| 1 | 528 | 0% | 0.419 | 0.049 | -0.0002 | 0 |
+| 2 | 358 | 0% | 0.718 | 0.043 | 0.0001 | 0 |
+| 3 | 1371 (one@4096) | 25% | 0.603 | 0.074 | **0.041** | 0 |
+
+- `frac_reward_zero_std = 0` on all steps — v2's critical bug is fixed
+- Step time: 65-420s depending on completion length. Average 180s/step.
+- Estimated full run: 500 steps × 180s = ~25 hours
+
+---
+
+## Consequences
+
+### What we expect from v3
+- SQL/insights/push should improve — model produces answers, rewards have variance
+- Extraction may or may not improve — depends on whether temp=1.0 rollouts produce enough JSON for the reward to shape behavior
+- clip_ratio=0 may persist — if so, entropy collapse is still the failure mode, even with all fixes
+- Training will be slow (~25h) due to occasional 4096-token completions
+
+### What comes after v3
+1. **Evaluate v3** — run benchmark, compare to v2 validation (mean=0.54)
+2. **Document lessons** — update PROJECT.md with v3 findings
+3. **Base model training** — `Polygl0t/Tucano2-qwen-3.7B-Base` → SFT → GRPO with shorter completions (512-1024)
+4. **Hybrid deployment** — base model for extraction/SQL/push, think model for insights (if v3 insights are strong)
+
+---
+
+## Lessons Learned (this session)
+
+### Technical
+
+1. **Thinking models are incompatible with small completion budgets.** The `<think>` block is not controllable via system prompts on untrained models. L1 paper confirmed: length compliance requires RL training. On a 24GB GPU with 4096 max tokens, the think overhead leaves insufficient room for structured output.
+
+2. **Temperature changes everything for GRPO.** At temp=0.7 (calibration), the model locks into verbose deterministic thinking → 100% ceiling hits. At temp=1.0 (training), the model explores diverse short completions → average 358-528 tokens. This is the single biggest factor determining whether GRPO works on this model.
+
+3. **Calibration at inference temperature ≠ training behavior.** The calibration cell uses temp=0.7 to simulate eval. But GRPO trains at temp=1.0. The calibration results (0.43 mean, 100% ceiling) are misleading — actual training dynamics are much healthier (0.60 mean, 25% ceiling). Future calibration should include a temp=1.0 pass.
+
+4. **Every canonical GRPO paper starts from base/instruct, not thinking models.** DeepSeek-R1-Zero, Dr. GRPO, DAPO, ThinkJSON, Reasoning-SQL, RL-Struct — all start from base. Only Skywork-OR1 starts from a thinking model, and that's for squeezing marginal SOTA gains, not domain adaptation.
+
+5. **LoRA adapters are model-specific.** Can't transfer SFT adapter from Think model to Base model — weights are calibrated to different base weight spaces. Switching to base requires re-running SFT.
+
+6. **Transformers version drift causes warnings.** Unsloth 2026.4.8 pulls Transformers 5.5.0 (v2 had 4.57.6). TRL 0.24.0 was written for the older version → deprecation warnings about `generation_config` kwargs and `AttentionMaskConverter`. Harmless but noisy.
+
+### Process
+
+7. **Prompt engineering research before implementation saves compute.** The literature crawl found 6 papers on thinking control (OptimalThinkingBench, Mid-Think, L1, AdaptThink, TALE, ThoughtTerminator) in one research call. The finding that "Don't overthink" reduces tokens by 23% on Qwen3 was directly applicable — even though the effect was zero on this specific model at inference time, the `reward_think_efficiency()` function may still teach compliance during training.
+
+8. **The model family tree matters.** Discovering that `Polygl0t/Tucano2-qwen-3.7B-Think` → `Polygl0t/Tucano2-qwen-3.7B-Base` → `Qwen/Qwen3-4B-Base` gave us a clean non-thinking alternative with Portuguese pretraining preserved. Without checking the Hub metadata, we might have defaulted to vanilla `Qwen3-4B-Base` and lost the Portuguese specialization.
+
+9. **Log everything to W&B.** Moving W&B init to Cell 3 means all preflight checks (inference test, KV cache, calibration) are logged. When the notebook disconnects mid-calibration, the data survives. This was the user's idea — essential for long-running Vertex AI sessions.
+
+---
+
+## Files in repo
+
+```
+rtferraz/tucano2-commerce/
+├── docs/
+│   ├── PROJECT.md                          # Full project documentation
+│   ├── ADR-001-next-steps.md               # Execution plans (benchmark, comparison, v3)
+│   ├── v3_thinking_control_patch.md        # The 3-change patch spec
+│   └── checkpoints/
+│       └── 2026-04-23_v3-launch.md         # ← THIS FILE
+├── notebooks/
+│   └── grpo_vertex_v3.ipynb                # v3 notebook (patched, running)
+├── scripts/
+│   └── md_to_ipynb.py                      # Markdown → notebook converter
+├── grpo_vertex_v2_ipynb.md                 # v2 reference with outputs
+└── .gitignore
+```
+
+---
+
+## To resume this session
+
+1. Check W&B: `tferrazrafael-self/tucano2-commerce` — look for run `grpo-v3-l4-*`
+2. Check training progress: reward trend, clip_ratio, completion_length
+3. If clip_ratio still 0 after step 50 → entropy collapse, consider stopping early
+4. If completion_length trends toward 4096 → model learned to fill budget, think control failing
+5. If reward improves and completion_length stays <2000 → v3 is working, let it run
+6. After training: run Cell 12 (save), Cell 13 (validation), compare to v2 (mean=0.54)
+7. Then: plan base model SFT + GRPO for extraction-focused training