oaklight/krino-modernbert-base-heads

Krino Adapter: ModernBERT-base

Lightweight decision heads trained on a frozen ModernBERT-base backbone. Outputs typed, calibrated decisions (noul / choice / score) instead of text.

302K trainable params on a 149M frozen backbone. 48.8% aggregate accuracy across 19 benchmarks.

Quick start

from krino import KrinoModel

model = KrinoModel.from_pretrained("oaklight/krino-modernbert-base-heads")

answer = model.predict(
    state="Customer: I ordered a laptop last week and it still hasn't arrived.",
    question={
        "type": "choice",
        "instructions": "Which intent does this message express?",
        "criteria": {
            "track_order": "Wants to know where an order is",
            "cancel_order": "Wants to cancel an order",
            "report_damage": "Received a damaged item",
        }
    }
)
print(answer)

Results

Type	Accuracy
Choice	42.9%
Noul	58.9%
Score	44.3%
Aggregate	48.8%

Per-benchmark

Benchmark	Type	Accuracy
agnews	choice	87.0%
mednli	noul	65.6%
mnli	noul	64.4%
typed_decisions	choice	58.2%
sst2	noul	57.4%
multirc	noul	56.6%
contractnli	noul	55.8%
codesearchnet	choice	54.3%
yelp	score	52.2%
tabfact	noul	51.6%
sst5	score	45.8%
fever	choice	40.6%
banking77	choice	34.4%
swag	choice	34.2%
race	choice	32.6%
arc	choice	31.8%
stsb	score	31.2%
hellaswag	choice	30.4%

Architecture

State -> ModernBERT-base (frozen, ModernBERT encoder) -> hidden states
                                               |
  NoulHead (linear -> sigmoid)                -> P(yes)
  ChoiceHead (cross-attention -> softmax)      -> P(option_k)
  ScoreHead (cross-attention -> expected value) -> score

• Backbone: answerdotai/ModernBERT-base (149M params)
• Heads: 302K trainable params (rank 64 AttentionHead)
• Training: Multi-task on 19 NLU benchmarks, type-balanced sampling, 20 epochs

License

MIT