shreyanbr/system-one-distilled

System One (distilled)

A 70.8M-parameter DeBERTa-v3-xsmall cross-encoder that answers typed decision questions, Choice, Score and Noul, in one batched forward pass with zero generated tokens. It implements the TypeSafe Jev POST /v1/systemone schema and is meant for the routing, classification and gating decisions inside software that do not need an autoregressive model.

Supervision for this checkpoint: Claude Haiku 4.5's own answers to 1,500 training items per task.

Accuracy on the 500-item test splits

question	accuracy	majority floor
intent	0.688	0.010
tool	0.930	0.306
needs_tool	0.788	0.694
priority	0.464	0.464
type	0.682	0.374
is_incident	0.780	0.626

Read these next to the floor column, not on their own. Two of the six questions are at or below a constant predictor for every system measured, Claude Haiku 4.5 included, and that is a finding about the labels rather than about any model. The repository documents a retraction of one of our own earlier headline numbers for the same reason.

Use

from systemone.engine import SystemOneEngine
from systemone.schema import SystemOneRequest

engine = SystemOneEngine('shreyanbr/system-one-distilled')
request = SystemOneRequest.model_validate({
    'state': {'message': 'my card still has not arrived'},
    'questions': {'intent': {'type': 'choice', 'instructions': 'What does the customer want?',
                             'criteria': {'card_arrival': 'a card that has not arrived',
                                          'refund': 'a refund request'}}},
})
print(engine.answer(request).answers['intent'])

calibration.json ships alongside the weights and is required: the raw margins are not probabilities, and the engine reads a per-question temperature and a Platt scaler from it.

Caveats worth reading before you trust a number

• The base checkpoint was trained on banking77, so that row is contaminated. banking77 also has a published label-error rate near 14%, capping achievable accuracy around 0.86.
• triage.priority is close to unlearnable on this data. Every backend sits at or below the floor.
• The tickets dataset is CC-BY-NC-4.0, so the triage supervision is not for commercial use.
• Single seed, single run. Intervals are bootstrap over test items, not over training seeds.

Source, benchmark harness and the full limitations section: https://github.com/shreyanbr/jev-haiku-benchmarking