Ockev-1.5B: 35ms Discriminative Decision Engine for AI Agent Deliverables
Ship the result, not the conversation. Ending the "Tomato-Egg Problem".
"I asked an AI agent to write a recipe for tomato scrambled eggs. It proudly delivered: 'Here is your tomato scrambled eggs (note: this dish contains no pork, no beef, no chicken, and no fish).' Why are AI agents so obsessed with telling us what they didn't do? Ship the result, not the conversation."
— Shangyin Tan, UC Berkeley
Model Summary
Ockev-1.5B is a lightweight, zero-generation discriminative decision model built on Qwen/Qwen2.5-1.5B and fine-tuned with a Pointer Readout Head and Block-Causal Masking.
It intercepts conversational residue, discarded alternatives, and negative echoes in AI agent deliverables before they reach users or production codebases.
Key Capabilities
- Zero Generated Tokens: Decisions are computed directly via scaled dot-product between decision query vectors and option key vectors. Zero hallucination, zero JSON repair.
- 35ms Latency: 5x-7x faster than cloud decision services, 100% offline, free, and private.
- Native Apple Silicon MLX: Optimized for Metal GPU acceleration on MacBooks.
Systems Comparison
| Execution Path | Generated Tokens | Time (ms) | JSON Parsing Failure Risk | Cost / 1M Checks |
|---|---|---|---|---|
| Ockev-1.5B (Local Pointer Head) | 0 | 35 ms | 0% (Pure Softmax) | $0.00 |
| TypeSafe Jev 1.13.0 (Cloud API) | 0 | 250 ms | 0% (Typed) | ~$42.00 |
| Generative LLM (e.g. GPT-4o-mini) | 85–140 | 1,450 ms | 1.8% (Malformed JSON) | ~$8.50 |
Full Benchmark Breakdown (TomatoEggBench-120)
| Domain | Total Items | Ockev-3B (Ours) | TypeSafe Jev 1.13.0 | Ockev-1.5B (Ours) | SemIf 4B (Logits) |
|---|---|---|---|---|---|
| Code Repository | 29 | 93.1% (27/29) | 90.0% (26/29) | 86.2% (25/29) | 69.0% (20/29) |
| Excel / CSV Data | 20 | 100.0% (20/20) | 95.0% (19/20) | 95.0% (19/20) | 75.0% (15/20) |
| PDF Documents | 20 | 100.0% (20/20) | 100.0% (20/20) | 100.0% (20/20) | 80.0% (16/20) |
| Form RPA | 30 | 100.0% (30/30) | 100.0% (30/30) | 100.0% (30/30) | 76.7% (23/30) |
| Executive Slides | 21 | 90.5% (19/21) | 85.7% (18/21) | 85.7% (18/21) | 71.4% (15/21) |
| Overall Accuracy | 120 | 95.8% (115/120) | 93.3% (112/120) | 92.5% (111/120) | 74.7% (89/120) |
| Inference Latency | - | 86 ms | 250 ms | 35 ms | 850 ms |
| Execution Cost | - | $0.00 (Local) | $42.00 / 1B tok | $0.00 (Local) | $0.00 (Local) |
Dataset: 🤗 fancyboi999/tomato-egg-bench
| Benchmark Domain | Ockev-1.5B Accuracy | Official Jev 1.13.0 |
|---|---|---|
| Code Repositories | 86.2% | 90.0% |
| Excel / CSV | 95.0% | 95.0% |
| PDF Documents | 100.0% | 100.0% |
| Form RPA | 100.0% | 100.0% |
| Executive Slide Decks | 85.7% | 85.7% |
| Overall (120 Tasks) | 92.5% | 93.3% |
Quickstart
pip install ockev
ockev check path/to/deliverable.md
Python usage:
from ockev import OckevGate
gate = OckevGate()
res = gate.review_deliverable("Added JWT authentication without session cookies.")
print(res)
# {'verdict': 'revise', 'confidence': 0.96, 'latency_ms': 34.2}
For full codebase and benchmarks, visit the GitHub Repository.
- Downloads last month
- 24
Quantized
Model tree for fancyboi999/ockev
Base model
Qwen/Qwen2.5-1.5B