mobarmg/jev-schema-scorer-deberta-v3-large

Schema-conditioned candidate scorer (DeBERTa-v3-large)

One DeBERTa-v3-large encoder with a single scalar head scores (state, question + candidate) pairs. Deterministic code groups the scalar logits per question and decodes them into three answer primitives:

primitive	input schema	answer
choice	criteria: {option_id: description}	argmax option id + probabilities
noul	optional criteria: {"true": ..., "false": ...}	p(proposition is true)
score	criteria: [level 0 description, level 1, ...]	expected level index + per-level probabilities

The question text, criteria and option ids are read at inference time, never baked into the weights, so the same checkpoint answers new questions over new label sets without retraining.

Try it in the Space: mobarmg/jev-schema-scorer.

How it works

For every candidate of a question the model sees a sentence pair:

sequence_a = the state (free text, or a JSON object serialised)
sequence_b = {"candidate": {"id": "<option id>", "description": "<option description>"},
              "type": "choice", "instructions": "...", "criteria": {...}}

The candidate sits right after [SEP], so the only tokens that differ between a question's candidates are where the encoder attends most easily. Each pair yields one logit; a softmax over the question's candidates gives the answer distribution. Training minimises cross-entropy between that grouped softmax and a target distribution (one-hot for choice / score, [1-p, p] for noul).

Usage

The repo ships schema_scorer.py with the request compiler, decoder and a small adapter.

from huggingface_hub import hf_hub_download
import importlib.util

repo = "mobarmg/jev-schema-scorer-deberta-v3-large"
spec = importlib.util.spec_from_file_location("schema_scorer", hf_hub_download(repo, "schema_scorer.py"))
schema_scorer = importlib.util.module_from_spec(spec); spec.loader.exec_module(schema_scorer)

scorer = schema_scorer.LocalSystemOne(repo)
scorer.system_one(
    "Nine days of silence on a signed quote is a joke. Our launch event is on the 28th and we still "
    "do not have the licence keys your sales team promised. Somebody pick up a phone.",
    {
        "department": {"type": "choice", "instructions": "Which team should handle this?",
                       "criteria": {"billing": "Payment or subscription issues",
                                    "technical": "Bugs or integration problems",
                                    "sales": "Pricing or plan questions"}},
        "frustration": {"type": "score", "instructions": "How frustrated the customer appears",
                        "criteria": ["Calm, just stating facts", "Frustrated but civil", "Very angry, strong language"]},
        "is_urgent": {"type": "noul", "instructions": "The message conveys urgency"},
    },
)
# {'answers': {'department': {'type': 'choice', 'choice': 'sales', 'probabilities': {...}},
#              'frustration': {'type': 'score', 'score': 2.0, 'probabilities': [...], 'legend': {...}},
#              'is_urgent': {'type': 'noul', 'noul': 0.99}}}

Without the helper, it is a plain DebertaV2ForSequenceClassification with num_labels=1: tokenize (state, serialised question + candidate) pairs, take logits[:, 0], and softmax over each question's candidates.

Limits: state + schema + candidate must fit in 512 tokens (trained at 384); every question needs at least two candidates.

Training data

Fine-tuned from microsoft/deberta-v3-large (supervised, grouped-softmax cross-entropy, max 384 tokens) on the union of two synthetic English datasets, 17,170 training questions in total:

• v1 (7,650 train / 1,350 eval questions, 3,000 texts, 30 domains): support triage, moderation, product reviews, email routing, resume screening, banking, insurance claims, telehealth messages, IT helpdesk, dating safety, and more. One choice, one score and one noul question per domain.
• v2 (9,520 train / 1,680 eval questions, 2,400 states, 24 domains): harder task shapes. JSON-object states (orders, transactions, API request logs, candidate profiles), multi-turn dialogue transcripts (support chats, sales calls, tutoring), code snippets and log excerpts, choice questions with 5-7 options including an "other" bucket, 4-5 level score questions, noul labels defined by explicit rules over the state, sarcasm and negation, summary faithfulness, and rubric grading of student answers. Each domain has 4-6 questions.

To make the model read the schema instead of memorising label ids, each training example draws a random instruction wording and criteria wording, shuffles the choice options, and replaces the option ids with opaque ids (opt_a, k2, bravo, ...) half of the time. Eval splits hold out 15% of records per domain.

Evaluation

Measured with bench_dataset.py on the held-out eval splits.

eval split	metric	this checkpoint	previous checkpoint (v1 only)
v2 (1,680 questions)	choice accuracy	0.841	0.687
v1 (1,350 questions)	choice / noul / score	unchanged from the previous checkpoint	0.889 acc / 0.940 acc, 0.052 Brier / 0.183 MAE

Chance on v2 choice is 0.214; on v1 it is 0.255. The earlier revision of this repo (v1-only training) remains available in the commit history.

Known weak spots (near chance): order_record.next_action, summary_faithfulness.error_type, and student_answer grading.

Intended use and caveats

• Intended for experiments with schema-conditioned classification: routing, triage, moderation-style labelling, ordinal scoring, and yes/no propositions over short English texts.
• Trained on synthetic data. Expect degraded accuracy far from the training domains, on long inputs, on non-English text, and on criteria that require world knowledge or reasoning across the text.
• Probabilities are often very peaked (the grouped softmax is trained on one-hot targets); treat them as rankings rather than calibrated confidences.
• Not a safety classifier. Do not use its moderation, health, or security outputs to make consequential decisions without human review.