Dev (dev-0.4b): 399M Bidirectional Decision Model

Dev is an open-source 399M parameter bidirectional decision model built on ModernBERT-large. It is purpose-built for unstructured-to-structured classification—including ticket routing, yes/no verification, and rating scales—executing in a single forward pass (~28ms on MPS) without token generation.

Following Jev (TypeSafe) and Kev-0.5B (Jared Palmer), Dev tests a fundamental architectural question: What if decision models shouldn't be causal decoders at all, but native bidirectional cross-encoders?

---

Key Performance Results

Evaluated against Jared Palmer's kev-0.5b (built on a frozen Qwen-2.5-0.5B causal backbone + 9.3M LoRA pointer head):

Benchmark / Task	What It Tests	Kev-0.5B (Causal Qwen2.5)	Dev-0.4B (Bidirectional ModernBERT)	Result
MTEB Banking77	77-Way Intent Routing	86.0%	91.33% (Top-3: 98.67%)	🏆 +5.33% Win
Google BoolQ	Reading Verification	75.3%	85.20%	🏆 +9.90% Win
Yelp Reviews	5-Star Rating (Exact / MAE)	55.3%	62.67% (MAE: 0.4017)	🏆 +7.37% Win

Inference Latency: Dev-0.4B executes in 27.6ms on Apple Silicon MPS (M1 Max) and ~10ms on CUDA FP16 SDPA (single forward pass, zero token generation loops).

Dev also reranks Python code retrieval modestly above a BM25 lexical baseline on the CodeSearchNet human-judgment benchmark (0.8203 vs 0.7652 NDCG@10 on test). We treat this as a capability check, not a headline benchmark.

---

Core Architecture

1. Native Bidirectional Attention (ModernBERT-large): Instead of causal decoders with lower-triangular masks, Dev uses unconstrained bidirectional cross-attention across all 28 transformer layers.

   • When choices are listed on the token tape, Option 1 can attend forward to Option 4, enabling true mutual candidate conditioning and eliminating position/recency bias.
   • Runs on hardware-fused SDPA kernels (mask=None) on CUDA and Apple Silicon MPS.
   • Native 8k context window.

2. Three Tasks, One Dynamic Head: Instead of separate heads for classification, verification, and regression, Dev realizes that all three tasks are fundamentally classification:

   • Categories (Routing): Classification over $N$ candidate options in the prompt.
   • Yes / No: Classification over two options: ["No", "Yes"].
   • Rating (1–5 scale): Classification over ordered scale levels (["1 star", ..., "5 stars"]), taking the expected value.

3. Dynamic Choices via the GLiNER Mechanism: Borrowing the core insight from GLiNER, candidate choices are not hardcoded into neural network weights. They are written as natural language text directly inside the prompt. Dev's single 2-layer choice head evaluates whatever choices you provide on the fly.

4. Single Forward Pass: The document and all candidate options are evaluated together in one single forward pass (~28ms on MPS, ~10ms on CUDA), rather than running separate passes per option.

---

Post-Training Calibration: Temperature Scaling (Guo et al. 2017)

Cross-Entropy loss separates classes effectively, but its logarithmic tail pushes logits toward extreme values (±infinity), producing overconfidence. While boolean verification comes out of SFT essentially calibrated (ECE: 0.016), multi-class choice and ordinal scoring are significantly overconfident.

Because Dev uses a single universal choice head, fine-tuning the shared head under Brier loss creates cross-task gradient tension and vanishing gradients ($2(p - y) \cdot p(1 - p) \to 0$).

Instead, Dev applies per-readout temperature scaling (Guo et al., 2017) fit post-hoc on held-out validation data by minimizing NLL:

Readout	Fitted T	Validation ECE (equal-mass)	NLL	Top-1 Accuracy
Noul (boolean)	1.3575	0.016 (already low here)	0.17 → 0.15	0.967 → 0.967 (Invariant)
Choice (categorical)	4.2542	0.189 → 0.083	2.26 → 0.73	0.782 → 0.782 (Invariant)
Score (ordinal)	3.7097	0.332 → 0.117	2.59 → 1.14	0.545 → 0.545 (Invariant)

On the external, unseen benchmarks, the shipped temperatures generalize, accuracy exactly invariant:

Benchmark (readout)	ECE: raw → calibrated	Accuracy
Google BoolQ (noul, n=500)	0.103 → 0.077 (−25%)	0.852 → 0.852
Banking77 (choice, n=300)	0.075 → 0.055 (−27%)	0.913 → 0.913
Yelp (score, n=300)	0.318 → 0.155 (−51%)	exact 0.627 → 0.627

• Choice & Score were badly overconfident out of SFT; their ECE falls by half or more.
• Boolean looked already-calibrated on the validation set (0.016) but is overconfident on external BoolQ; its fitted T = 1.36 cuts BoolQ ECE 0.103 → 0.077. The temperatures are fit on validation and checked on the held-out benchmarks.
• Ordinal point estimate: flattening the score distribution raises Yelp MAE modestly (0.402 → 0.429, still sub-half-star). ECE and MAE trade off smoothly as T grows.
• 100% Accuracy Invariance: temperature scaling is strictly monotonic, so all top-1 accuracies, rankings, and benchmark scores are untouched.

Temperatures are saved in run.json and automatically applied at readout during inference.

---

Quickstart & Usage

Installation

git clone https://github.com/nikhilpujari/dev.git
cd dev
pip install -e .

Python Inference

from dev.inference import Predictor

# Load from Hugging Face Hub or local directory ("runs/dev-0.4b")
predictor = Predictor("mpnikhil/dev-0.4b", device="auto")

# 1. Routing to Categories (~28ms MPS, Calibrated)
result = predictor.answer(
    state="Customer cannot log in. Password reset email is failing with 550 Mailbox Unavailable.",
    questions={
        "route_ticket": {
            "type": "choice",
            "instructions": "Assign this ticket to the appropriate queue.",
            "criteria": [
                "billing_support",
                "email_infrastructure",
                "account_security",
                "general_inquiry"
            ]
        }
    }
)
print(result["route_ticket"])
# Output:
# {
#   'criterion': 'email_infrastructure',
#   'confidence': 0.9987,
#   'calibrated': True
# }

# 2. Yes / No Verification
res_noul = predictor.answer(
    state="ModernBERT uses hardware-fused SDPA kernels and 8k context natively.",
    questions={
        "has_8k": {
            "type": "noul",
            "instructions": "Does the passage state ModernBERT supports 8k context?",
            "criteria": ["No", "Yes"]
        }
    }
)
print(res_noul["has_8k"])
# Output:
# {
#   'criterion': 'Yes',
#   'probability_yes': 0.9942,
#   'calibrated': True
# }

# 3. Rating on an Ordinal Scale (Expected Value + Normalized Variance)
score_result = predictor.answer(
    state="Pull request refactors cache layer, adds 14 unit tests, and passes all CI checks.",
    questions={
        "code_quality": {
            "type": "score",
            "instructions": "Rate pull request quality on a 1-5 scale.",
            "criteria": ["Poor", "Needs Work", "Acceptable", "Good", "Excellent"]
        }
    }
)
print(score_result["code_quality"])
# Output:
# {
#   'value': 3.84,           # Expected score
#   'variance': 0.14,        # Clustered consensus
#   'confidence': 0.965,     # Normalized certitude
#   'calibrated': True
# }

---

Intended Use & Limitations

• Intended Use: High-throughput, low-latency unstructured-to-structured classification (support ticket routing, log triage, assertions, guardrail verification, rubric rating).
• Limitations: Dev is a non-generative decision model. It does not output autoregressive text, stream tokens, or perform Chain-of-Thought (CoT) generative reasoning. For tasks requiring reasoning traces or text generation, use a causal generative LLM.

---

Lineage & Acknowledgments

• TypeSafe: For introducing Jev and demonstrating the power of non-generative decision models.
• Archer Hume: For reverse-engineering Jev's behavioral blueprint across 10,000 API calls in “Jev’s Architecture Unmasked”.
• Jared Palmer: For open-sourcing Kev-0.5B on Hugging Face, establishing the single-pass causal decoder baseline.
• Answer.AI & LightOn: For pretraining ModernBERT (answerdotai/ModernBERT-large), the 399M parameter bidirectional encoder backbone.
• Urchade Zaratiana et al.: For GLiNER, whose candidate-span pooling mechanism directly inspired our dynamic choice head.

---

Citations & References

Foundational Architectures & Calibration

@article{vaswani2017attention,
  title={Attention is All You Need},
  author={Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, {\L}ukasz and Polosukhin, Illia},
  journal={Advances in Neural Information Processing Systems},
  volume={30},
  year={2017},
  url={https://arxiv.org/abs/1706.03762}
}

@article{warner2024modernbert,
  title={Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Long-Context Representation},
  author={Warner, Benjamin and Chaffin, Antoine and Clavi{\'e}, Benjamin and Weller, Orion and Hallstr{\"o}m, Oskar and Taghadouei, Saeed and Aarsen, Tom and Shakir, Nathan and Douze, Matthijs and Lipani, Aldo and others},
  journal={arXiv preprint arXiv:2412.13663},
  year={2024},
  url={https://arxiv.org/abs/2412.13663}
}

@article{zaratiana2023gliner,
  title={GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer},
  author={Zaratiana, Urchade and Tomeh, Nadi and Holat, Pierre and Chaffin, Antoine},
  journal={arXiv preprint arXiv:2311.01079},
  year={2023},
  url={https://arxiv.org/abs/2311.01079}
}

@inproceedings{guo2017calibration,
  title={On Calibration of Modern Neural Networks},
  author={Guo, Chuan and Pleiss, Geoff and Sun, Yu and Weinberger, Kilian Q},
  booktitle={International Conference on Machine Learning},
  pages={1321--1330},
  year={2017},
  organization={PMLR},
  url={https://arxiv.org/abs/1706.04599}
}

Datasets

• PolyAI/banking77: Casanueva, I. et al. (2020). Efficient Intent Detection with Dual Sentence Encoders Applications to Banking. Hugging Face Dataset.
• google/boolq: Clark, C. et al. (2019). BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. Hugging Face Dataset.
• code_search_net: Husain, H. et al. (2019). CodeSearchNet Challenge: Evaluating the State of Semantic Code Search. Hugging Face Dataset.
• yelp_review_full: Zhang, X. et al. (2015). Character-level Convolutional Networks for Text Classification. Hugging Face Dataset.

---

License

Apache 2.0