jev-playground-rlcd

A calibrated decision classifier trained with a proper scoring rule on synthetic typed questions, a local, open reimplementation of the "System One" decision-model contract documented for Jev-class models. Part of jev-playground.

Given a text state and typed questions, Noul (yes/no), Choice (closed set), Score (ordered levels), it returns calibrated probability distributions over request-defined label sets, zero-shot over unseen labels. The full distribution is the artifact; confidence is derived from it, never a learned head.

Results (held-out validation, 500 states / 4536 pairs)

vs the same base model, stock:

Metric	noul	choice	score
accuracy	0.731 → 0.999	0.510 → 0.998	0.898 → 0.978
mean L1 to ground truth	0.273 → 0.085	0.226 → 0.048	0.134 → 0.065

Aggregate val ECE: 0.0139. The model is slightly under-confident, the conservative failure mode, with honest spread-out distributions.

Usage

The checkpoint is a standard NLI sequence classifier (entailment head). The easiest way to serve it is behind the jev-playground contract:

pip install -e ".[server,deberta]"   # from the jev-playground repo
hf download soyrsoyr/jev-playground-rlcd --include "deberta-large/*" --local-dir checkpoints

JEV-PLAYGROUND_BACKEND=deberta JEV-PLAYGROUND_DEBERTA_MODEL=checkpoints/deberta-large jev-playground-serve --port 8000

curl -X POST http://localhost:8000/v1/systemone \
  -H "Content-Type: application/json" \
  -d '{
    "state": "Ticket: printer is on fire on floor 3, production down",
    "questions": {
      "urgent": {"type": "noul", "instructions": "This ticket is urgent (P1 or blocking)."},
      "owner": {"type": "choice", "instructions": "Which team should own this ticket?",
                "criteria": {"hardware": "Physical devices", "software": "Apps and bugs"}}
    }
  }'

Training your own: see the repo, the data engine (training/data.py) emits reference distributions that are exact by construction, and the trainer optimizes a proper scoring rule (log-loss / Brier) against them.

Training details

• Data: soyrsoyr/jev-playground-rlcd-v0, 5000 synthetic states (support tickets, product reviews), 40K training pairs, typed questions with reference distributions exact by construction (crisp/weak/contradictory uncertainty slices at 70/20/10).
• Objective: soft-target cross-entropy (log score) over the entailment head; Brier available (--loss brier).
• Hyperparameters: differential LR (heads 2e-5 / encoder 1e-5), linear warmup + decay, batch 8, max-length 256, fp32, 2 epochs.
• Hardware: AMD RX 7900 XT via WSL2 + ROCm, ~19 minutes end to end. An 82M cross-encoder/nli-distilroberta-base run (800 steps, CPU) is also included under distilroberta/.

Limitations / jaggedness (measured, published)

• Calibrated in-distribution (the synthetic template distribution). On arbitrary out-of-distribution state text it degrades: an ad-hoc phrased P1 ticket scores noul ~0.13 instead of ~0.9, and score distributions flatten toward uniform. Broader LLM-authored state diversity is the fix.
• Slightly under-confident (negative confidence-vs-correctness ECE).
• English templates; the Jev-class jaggedness applies (dates read as text, counting by shape, adversarial state sensitivity).

License

MIT