[Pi Coding Agent + Model] Integrated Evaluation -- FINAL CONSOLIDATED REPORT (English)

Evaluation date: 2026-09-04 (four rounds in one day; see Section 0 for history)
Evaluation focus: Task-completion effectiveness under Pi Coding Agent v0.84.4
System environment: Dynamic probing + adaptive optimization (see Section 1)
Models:
A = MiniCPM5-1B-Q4_K_M (openbmb/MiniCPM5-1B-GGUF, 657 MB, 1.08B params);
B = LFM2.5-8B-A1B-Q4_K_M (LiquidAI/LFM2.5-8B-A1B-GGUF, 4.9 GB, 8.47B params / ~1B active);
C = Qwen3.5-2B-Q4_K_M (unsloth/Qwen3.5-2B-GGUF, 1.2 GB, 1.88B params);
D = Qwen3.5-0.8B-Q4_K_M (unsloth/Qwen3.5-0.8B-GGUF, 508 MB, 752.39M params).

This file consolidates ALL materials of this evaluation program into a single report:
20260904_1.txt (Rev.2, A/B, Chinese), report2.txt (A/B/C, Chinese),
report2_en.txt (A/B/C, English), report3_en.txt (A/B/C/D, English),
plus the underlying raw data (benchmark rows, server-log token sums, per-run quality records,
fixture ground truth, run-area file inventory).
Note: /home/user/app/report.txt (OpenAI-vs-Anthropic financial analysis, pre-existing) is a
separate unrelated artifact and is NOT part of this program -- it is only listed in the document
map (Appendix H) for completeness and is not integrated.

0. Document map and revision history

Supersession rule: where numbers conflict, this file governs. Rev.1 numbers are void (see Appendix D
for why). A/B/C numbers here are identical to report3_en.txt; D numbers are identical to Round 4
measurements.

1. Adaptive system optimization

1.1 Hardware topology (measured, not hard-coded)

1.2 Optimal parameter derivation (step by step)

1. Physical pool P: within node0, keep the smallest CPU number per
   (physical_package_id, core_id) (excluding HT siblings 96-143) -> P = [0..47], 48 cores.
2. Working cores = min(P=48, Q=16) = 16.
3. Threads -t = max(1, 16-1) = 15 (one core reserved for the system).
4. Binding: prefer numactl -N 0 -m 0 -> container does not support it, fall back to
   taskset -c <first Q of P> = taskset -c 0-15. 0-15 is the computed P[0:16], not a placeholder.
5. Context: -c 65536 per the "default 64k" requirement; verified loadable for all four models.
6. Fixed: --jinja (required by Pi Agent tool-call templates).

1.3 64k load verification (taskset -c 0-15 -t 15 --jinja -c 65536)

1.4 Before/after optimization comparison (llama-bench -r 3, same session)

Honest reading: the effect differs by model and metric and sits within run-to-run variance for the
small models (note +/-150.77 / +/-98.23 on D pp). The optimized configuration is chosen for spec
compliance (reserve 1 core, pin physical cores, avoid HT-sibling contention) and generally lower
variance, not for a universal speedup.

2. Pure-generation throughput baseline (reference only, not decisive)

Optimal configuration (taskset -c 0-15 -t 15, -r 3, llama.cpp 0.3.0-dev commit 9a4843c):

C is in A's size class yet ~3x slower (architecture/template overhead); D is the fastest Qwen on pp.
Speed does not predict the Agent verdict (Section 5).

3. Pi Agent task results (core)

3.1 Methodology (identical for all four models)

• Backend: llama-server with optimal parameters (taskset -c 0-15 -t 15 --jinja -c 65536),
  one port per model (A:8080, B:8081, C:8082, D:8083); models.json contextWindow = 65536 for all.
• Client: Pi Agent v0.84.4, --tools write,read,edit,bash, --no-session (independent per task),
  280s timeout per task.
• Standardized fixtures:
  • T1 TODO tool: empty directory; build main.py (argparse add/list/done/delete + todos.json)
    • test_main.py (pytest) + self-run pytest -q to green. Pass = files on disk AND pytest all-pass.
  • T2 fix bugs: given calc.py (3 bugs: mul written as addition, div with no zero guard,
    mean([]) crash) + test_calc.py (verified baseline 3 failed / 2 passed); fix until all pass.
  • T3 log analysis: access.log 196674 bytes / 2448 lines (~192 KB Apache logs, seed=20260904);
    sample with bash (head/wc/awk/sort/uniq, no full cat), write report.md with total lines,
    total bytes, per-status-code counts, Top-5 URLs, 2xx success rate (ground truth in 3.6).
• Token accounting: sums of print_timing lines in server logs (prompt_n / predicted_n).
  Non-zero truncated releases: 0 for all models (64k never overflowed).

3.2 A MiniCPM5-1B (server prompt 4381 / predicted 23922)

3.3 B LFM2.5-8B-A1B (server prompt 4461 / predicted 15830)

3.4 C Qwen3.5-2B (server prompt 15398 / predicted 26811; 33 prompt events = multi-turn tool loop)

3.5 D Qwen3.5-0.8B (server prompt 83327 / predicted 106488; all three tasks hit the 280s timeout)

3.6 T3 ground truth (awk direct computation; only C produced a report to grade)

Total lines 2448, total bytes 196674; status codes 200:1348 / 404:452 / 500:223 / 301:216 / 304:209;
Top-5 paths /health 256 / /static/style.css 253 / /static/app.js 252 / /index.html 247 / /api/login 247;
2xx success rate 1348/2448 = 55.07%.

3.7 Four-model comparison

D is a scaling warning: the smaller sibling of the best model is not "faster and cheaper" -- it never
converges, burns the most tokens, and times out everywhere. T1's bug comparison is informative: D's bug
breaks the harness itself (INTERNALERROR), while C's bugs stay inside assertions (fixable given time).

4. Basic generation quality probes (temperature=0)

Protocol (Appendix F): 3 prompts x 3 runs at max_tokens=300; every all-empty item re-tested at 1024.
All 54 runs were 3/3 deterministic (identical outputs across the three runs). Per-run raw records in
Appendix G.

D is the only model 9/9 empty at both budgets: its thinking never terminates into an answer within
1024 tokens on any probe. Paradoxically its tool-call path still emits file-writing actions (T1 has real
files), so the defect is specific to final-answer convergence. Single-turn quality is inverted vs Agent
performance: the best Agent (C) is the most token-hungry per turn.

5. Summary and recommendation

Which model is best suited as the Pi Agent's brain? -- C Qwen3.5-2B. D ranks last.

1. Hard criteria (files on disk + pytest/file verification): A 0/3, B 0/3, C 1 PASS + 1 PARTIAL,
   D 0/3 with 3 timeouts. C is the only model completing a task end-to-end (T2, 39s, 5 passed) and
   the only one producing a usable report (T3, 4/5 sections correct), and the only one with genuine
   multi-turn tool execution.
2. C's weaknesses are explicit: T1's test-isolation bug survived 280s, T3's success-rate arithmetic is
   wrong (counting and division both wrong), single-turn thinking is verbose (678/340 tok for code/math;
   MoE explanation exceeds 1024), highest non-D token spend (26811).
3. A fails harmfully (empty files, runaway generation to 16384); B fails cleanly (plausibly a tool-schema
   mismatch, zero execution); D fails expensively (~106k tokens, zero passes, every timeout hit, 9/9 empty
   probes at both budgets). Below ~2B, this family loses answer convergence faster than it gains speed.
4. Final ranking: C > B (planning text only) > A > D. For actually running the Pi tool chain, pick
   only C among the four, with per-turn budget >= 1024 and arithmetic delegated to bash/awk.
   D must not be used as an Agent brain at any tested budget. If offline draft text without execution is
   enough, B's planning text remains usable.

6. Recommended run commands (all verified in the current environment)

# A MiniCPM5-1B (:8080)
taskset -c 0-15 /home/user/llama.cpp/build/bin/llama-server \
    -m /home/user/models/MiniCPM5-1B-Q4_K_M.gguf \
    -t 15 --jinja -c 65536 --host 127.0.0.1 --port 8080

# B LFM2.5-8B-A1B (:8081)
taskset -c 0-15 /home/user/llama.cpp/build/bin/llama-server \
    -m /home/user/models/LFM2.5-8B-A1B-Q4_K_M.gguf \
    -t 15 --jinja -c 65536 --host 127.0.0.1 --port 8081

# C Qwen3.5-2B (:8082)
taskset -c 0-15 /home/user/llama.cpp/build/bin/llama-server \
    -m /home/user/models/Qwen3.5-2B-Q4_K_M.gguf \
    -t 15 --jinja -c 65536 --host 127.0.0.1 --port 8082

# D Qwen3.5-0.8B (:8083; 64k verified ~2.4s load; NOT recommended as Agent brain)
taskset -c 0-15 /home/user/llama.cpp/build/bin/llama-server \
    -m /home/user/models/Qwen3.5-0.8B-Q4_K_M.gguf \
    -t 15 --jinja -c 65536 --host 127.0.0.1 --port 8083

# Pi usage (models.json holds 4 providers; CTX 65536 everywhere)
pi --provider llama-cpp-local-qwen --model qwen3.5-2b --tools write,read,edit,bash -p "your task"

~/.pi/agent/models.json providers: llama-cpp-local-minicpm -> 127.0.0.1:8080 / minicpm5-1b;
llama-cpp-local-lfm -> 127.0.0.1:8081 / lfm2.5-8b; llama-cpp-local-qwen -> 127.0.0.1:8082 /
qwen3.5-2b; llama-cpp-local-qwen08 -> 127.0.0.1:8083 / qwen3.5-0.8b;
all contextWindow: 65536, maxTokens: 16384.

Appendices

A. Environment and versions

llama.cpp 0.3.0-dev (build 1, commit 9a4843c, GGML_NATIVE=ON), Pi Coding Agent 0.84.4,
Node v22.23.2, Python 3.12 + pytest 9.1.1, Debian Docker on KVM.
Model files in ~/models/: A 657 MB (.../openbmb/MiniCPM5-1B-GGUF/.../MiniCPM5-1B-Q4_K_M.gguf),
B 4.9 GB (.../LiquidAI/LFM2.5-8B-A1B-GGUF/...), C 1.2 GB (.../unsloth/Qwen3.5-2B-GGUF/...),
D 508 MB (.../unsloth/Qwen3.5-0.8B-GGUF/.../Qwen3.5-0.8B-Q4_K_M.gguf).

B. Raw benchmark rows (llama-bench -r 3)

• A optimized: pp512 1604.24 +/- 36.25 / tg128 127.86 +/- 3.85; baseline: 1707.33 +/- 183.80 / 120.51 +/- 5.38.
• B optimized: pp512 320.40 +/- 16.48 / tg128 65.91 +/- 2.36; baseline: 306.56 +/- 4.15 / 73.85 +/- 0.86.
• C optimized: pp512 526.31 +/- 15.99 / tg128 43.07 +/- 2.15; baseline: 518.74 +/- 22.84 / 34.26 +/- 0.67.
• D optimized: pp512 747.69 +/- 98.23 / tg128 62.97 +/- 10.05; baseline: 716.69 +/- 150.77 / 71.38 +/- 2.02.

C. Reproduction paths

• Fixtures: /home/user/agent_eval/task2_bugfix/{calc.py,test_calc.py} (baseline 3 failed / 2 passed),
  /home/user/agent_eval/task3_log/access.log (196674 B / 2448 lines, seed=20260904).
• Run areas: /home/user/agent_eval/run_{minicpm,lfm,qwen,qwen08}/{task1_todo,task2_bugfix,task3_log}/
  (stdout in agent_out*.txt, written files alongside).
• Server logs: /tmp/srv{A,B,C,D}.log (token-sum source; C/D logs contain later re-test segments).
• Quality raw JSON: /tmp/quality.json (A/B @300), /tmp/quality_qwen.json (C @300),
  /tmp/quality_qwen1024.json (C @1024), /tmp/quality_ab1024.json (A/B code+math @1024),
  /tmp/quality_d.json (D @300), /tmp/quality_d1024.json (D @1024).
• Report files in /home/user/app/: agentcompare.txt (spec), 20260904_1.txt (Rev.2 A/B Chinese),
  report2.txt (A/B/C Chinese), report2_en.txt (A/B/C English), report3_en.txt (A/B/C/D English),
  report_final_en.txt (this file: everything integrated).

D. Rev.1 -> Rev.2 correction record (9 items, carried over from 20260904_1.txt)

1. CTX violation (16384/8192 -> 65536, load verified) fixed.
2. -t derivation miswritten ("working cores" -> 16-1=15) fixed.
3. 0-15 placeholder suspicion (proven as computed P[0:16] with open derivation) fixed.
4. Missing before/after optimization data (added, honestly "varies by model") fixed.
5. Single/ambiguous benchmark runs (baseline + optimized, each -r 3) fixed.
6. Non-standard Agent tasks + char counts + no action chains (standard fixtures + server token sums + pytest gates) fixed.
7. 1-shot quality probes (3x3 at temp=0, determinism verified) fixed.
8. Unsupported conclusion ("recommend B" -> "neither A nor B", later superseded by C evidence) fixed.
9. Contradictory numbers (19.6s/259s unrepeatables voided; Rev.2 re-measured 37/4/267 and 100/13/191) fixed.

E. P derivation (unchanged since Rev.2)

node0 0-47,96-143, deduplicated by (physical_package_id, core_id) keeping the smallest id ->
P=[0..47]; Q=16