appsec-router-deberta-r5

A DeBERTa-v3-xsmall cross-encoder (22M backbone parameters) that decides whether a spoken or typed interview answer expresses a given point. It runs in the browser on appsecinterview.com, where it steers which follow-up question comes next. It is not a grader: the site's report is produced by a larger server model, and this model's opinion is never shown to the candidate.

• Base: cross-encoder/nli-deberta-v3-xsmall (Apache 2.0, on Microsoft's MIT-licensed DeBERTa-v3-xsmall), re-headed to two labels and fine-tuned.
• Files: fp32 model.safetensors (278 MB) and the int8 onnx/model_quantized.onnx (83 MB) that the site serves.
• Training data: pratikamin/appsec-router-pairs-r5, fully synthetic; see below.

The input format is part of the model

The model scores a pair: the candidate's answer as the first sequence and a hypothesis about the speaker as the second. Hypotheses are built from a one-sentence description of a point a good answer makes, using exactly this template (hypothesis.js in this repo):

export function hypothesisFor(signal) {
  const d = String(signal.description).replace(/\s+/g, " ").trim();
  // Descriptions are third-person present tense: "Frames XSS as…", "Distinguishes…"
  if (/^[A-Z][a-z]+s\b/.test(d)) return `The speaker ${d.charAt(0).toLowerCase()}${d.slice(1)}`;
  return `The speaker's answer: ${d}`;
}

Example pair, label demonstrated:

answer: Because the CDN keys only on the path, the response with the attacker's host header gets stored and everyone gets it for ten minutes. hypothesis: The speaker explains that the cache key omits an input the application uses, so one request is stored and served to other users.

Score = softmax over the two logits, take the demonstrated index (config.json → id2label). Shipped threshold: 0.30 (threshold.json), chosen on held-out tiers; the dev-optimal 0.10 fires on 13% of uncredited plain-language pairs. Pairs are truncated at 256 tokens in training; keep the whole hypothesis and window the answer if it is long.

Use with Transformers.js

import { AutoTokenizer, AutoModelForSequenceClassification } from "@huggingface/transformers";
const id = "pratikamin/appsec-router-deberta-r5";
const tok = await AutoTokenizer.from_pretrained(id);
const model = await AutoModelForSequenceClassification.from_pretrained(id, { dtype: "q8" });
const inputs = tok(answer, { text_pair: hypothesisFor(signal), truncation: true, max_length: 256 });
const { logits } = await model(inputs);
const [a, b] = Array.from(logits.data, Number);
const p = Math.exp(b) / (Math.exp(a) + Math.exp(b)); // index 1 = demonstrated

With cross-origin isolation (COOP/COEP headers) and four WASM threads a pair scores in about 65 ms on a laptop; single-threaded it is closer to 1.8 s. WebGPU gave no gain for this model.

How it was trained

Five rounds of distillation from a server-side router built on openai/gpt-oss-120b (Apache 2.0), recorded step by step in the training repository's READMEs. Round 5 (this model):

• Interview answers to 86 authored AppSec questions and their 378 follow-ups were generated by gpt-oss-120b in several registers (careful, hedged, terse, plain-language, name-dropping, and answers with one clause reversed), then labelled by the same model against each question's signals, keeping only unanimous labels. Reversed-claim and name-drop answers carry intended negative labels rather than teacher labels. Round 5 added plain-word paraphrases of credited answers.
• 42,616 training pairs from 57 topics; dev and test are 14 and 15 unseen topics.
• 3 epochs, batch 32, lr 2e-5, fp32, on a Colab T4. Threshold chosen on held-out tiers, not dev.

Evaluation

Held-out topics, against the teacher's labels, at the shipped threshold 0.30:

Set	Precision	Recall	F1	Follow-up verdicts agreeing
Round-1 hard test (3,327 pairs)	88%	70%	78%	196 / 228 (86%)
Round-5 test (9,430 pairs)	88%	77%	82%	449 / 514 (87%)

By answer style on the round-5 test (credited pairs caught / uncredited pairs fired): plain-word paraphrases 93%; plain answers 78% / 10%; one-sentence answers 65% / 5%; pure name-drops credited 1%; answers with one clause reversed still credited about 13%.

Limitations, read before using

• Synthetic supervision. Every training answer and label came from a model. Agreement figures are with that teacher, not with human judgement. Do not present its scores as an assessment of a person.
• It matches meaning, loosely. A wrong claim that reuses the right vocabulary is credited roughly one time in eight. A correct claim in unusual phrasing is missed about one time in five, more for one-sentence answers.
• Descriptions are inputs. A description written as a list of several things is scored as a whole; a description phrased "asks who…" does not match an answer that states who. Most misses seen in real transcripts were description faults, not model faults.
• Domain. Application security interview answers in English. Nothing else was tested.
• Not calibrated: 0.30 is an operating point, not a probability.