sugiv/call-gate-risk-score

Call Gate Risk Score

A ten-number logistic regression that turns Jev's answers about a voicemail into one probability: is this robocall junk? It sits between Jev and a safety-first policy in Call Gate, a personal system that keeps toll-free robocalls out of a Google Fi iPhone's voicemail.

It is not a standalone text classifier. It needs Jev's answers to the eight evidence questions in code/worker/src/jev.ts.

How it fits with Jev

voicemail audio → Deepgram transcript → Jev (8 typed questions, one call, ~0.2 s)
                                          │ probabilities
                                          ▼
                        risk score (this model) → policy rules → notify / warn / hold back

Jev is a System One model: it answers typed questions with probabilities instead of generating text. Jev answers narrow questions ("does it quote card digits?", "does it push you to an agent?"). This model learns how much each answer should count on this problem's data. The policy then applies rules no model can override; the main one is that a message claiming to be your bank, pharmacy, doctor or school is never held back.

Why a logistic regression on Jev's evidence

Jev already answers everything, so why train anything? Because a verdict and a decision aren't the same thing.

1. Many steady signals beat one shaky one. On the same real-style bank alert, Jev's overall verdict swung from 0.26 to 0.61 legit across runs and versions, while each evidence answer held within about ±0.03. Jev answers each question independently in one pass, so a model over nine steady signals inherits their stability.
2. Evidence carries over to new wording. The features describe tactics (pushes to an agent, pressure, a deal, no real specifics, no official channel), not words, so unseen scripts that use the same tactics still score as junk.
3. Recalibration to this problem. Jev is calibrated on TypeSafe's data. Here the base rate is lopsided: robots that ignore a keypad challenge are usually junk. The bias and weights learn that. TypeSafe doesn't fine-tune Jev on customer data, so this layer is where your own labels go to work.
4. The data picks the signals. "Insists it's legitimate" seemed like an obvious scam signal; its learned weight is −0.005, because real notices say it too.
5. Auditable. Every score breaks down into per-feature contributions.
6. Cheap to keep learning. Ten parameters, trained in milliseconds, retrainable nightly from one-tap labels.

Out-of-fold separation on the same 534 calls:

	Jev's verdict (P(scam) + P(marketing))	This model
AUC	0.989	0.9999
Most junk-looking legit example	0.94	0.56
Junk scoring above every legit example	289 of 404	399 of 404

Weights

scorer-weights.json, trained Oct 17, 2026 on 534 examples:

Feature (Jev answer)	Weight
bias	+0.693
catJunk = P(scam) + P(marketing)	+1.700
agent	+1.512
pressure	+0.556
sales	+0.508
selfAssert	−0.005
specific	−0.792
protected	−1.107
official	−1.303
catLegit	−1.959

P(junk) = 1 / (1 + e^−(bias + Σ weight × answer)). Hold-back threshold τ = 0.90: the lowest value at which no legit training example would be held back, plus a 0.02 margin, never below 0.90. The STIR attestation and repeat-caller weights are 0 for now: that metadata isn't in the training data.

Training

• Data: sugiv/call-gate-robocalls. 300 FTC robocalls (NC State), 230 DeepSeek-written synthetic calls (130 legit, 100 junk), 4 real labeled calls.
• Features: every transcript was run through the same Jev request the production Worker sends (code/training/featurize.ts imports it). 534 calls took 16 s and cost about $0.02.
• Fit: batch gradient descent with L2 regularization 0.01. Legit examples are weighted 3× because a missed real alert costs more than a delivered robocall.
• Validation: 5-fold cross-validation through the production policy code. Each call is scored by a model that never saw it, and the threshold is re-chosen inside each fold.

Results (5-fold, out-of-sample)

Approach	Legit held back (of 130)	Junk held back (of 404)
Jev verdict alone, threshold 0.90	1	313
Jev verdict alone, threshold 0.97	0	254
Jev verdict + safety rules	0	139
Hand-set weights + safety rules	0	9
This model + safety rules	0	161

How to read this:

• Jev's raw verdict is strong but runs close to the edge. The closest legit example, a calm "Harbor Bank, please call the number on the back of your card, no urgency" message, scored 0.94 scam-or-marketing. A 0.90 threshold drops it; 0.97 misses it by 0.03.
• The safety rules cost catch rate on purpose. Of the junk Jev's verdict would catch at 0.97, 33 claim to be a bank or healthcare provider, and 66 quote specifics, as many FTC scams do ("an order for $999"). The policy refuses to hold those back without stronger evidence.
• The learned weights beat Jev's verdict under the same safety rules (161 vs 139). Loosening the "quotes specifics" guard for non-protected callers reaches 249 with still zero legit held back. That candidate waits for real labels, because only 20 non-protected legit examples exist.
• Junk that isn't held back is still delivered: silently for review, or with an impersonation warning.

Validation: is AUC ≈ 1.00 too good to be true?

A near-perfect AUC on 534 examples should raise three suspicions: memorization (overfitting), leakage, and shortcut learning. The obvious shortcut here is authorship, since every legit example is LLM-written and most junk comes from the FTC. code/training/validate_scorer.py retrains independently with numpy and tries to break the result. Raw output: eval/validation.txt.

#	Test	Failure would look like	Result
1	Training-set AUC vs. 5-fold held-out AUC	Training ≫ held-out	0.9999 vs. 0.9999
2	10 different random 5-fold splits	High variance	0.9998–0.9999
3	Label permutation (20 shuffles, full pipeline)	AUC well above 0.5 means leakage	0.479 mean (0.410–0.559)
4	Synthetic-only subset (same author on both sides), 5-fold	Collapse if separating by authorship	1.000 (Jev verdict alone: 0.998); max legit 0.32
5	Train with no FTC data, score all 300 FTC robocalls	Real robocalls fall below synthetic legit	293/300 above every legit score (Jev verdict alone: 207/300)
6	Leave-one-category-out on each junk category	Unseen tactics score low	bank, gov and retail impersonation, injected, sales pitch: 100% above max legit; FTC 293/300
7	Bootstrap (1,000×) of AUC gain over Jev's verdict	Interval includes 0	+0.0107, 95% CI [+0.0039, +0.0187]

Reading: no memorization (1, 2), no leakage (3), no authorship shortcut (4), and transfer to real robocalls and unseen tactics (5, 6). The AUC is high mainly because Jev's evidence makes the task easy: Jev's verdict alone reaches 0.989, and the learned weights add a small but significant +0.011 (7). Their practical value is at the decision boundary, where the most junk-like legit example falls from 0.94 to 0.56.

Not established: behavior on real legit alerts. All 130 legit examples are synthetic, so no test in this table can substitute for real labels. The production system therefore runs in labels-only mode, and holds messages back in stages only after weeks of real calls with zero overturned decisions.

Also in this repo

• code/worker/src/policy.ts: the full decision policy, including the two-level rule for Jev's "which known script?" question (a reworded call matched a known script at 0.99).
• code/training/: corpus building, featurization and training scripts.
• eval/: per-example out-of-fold predictions and the report.
• ARCHITECTURE.md: system design (personal identifiers redacted).

Limitations

• The legit side is synthetic. Expect real-world false-alarm behavior to differ, and recalibrate on real labels.
• English only, Jev 1.13 only (jev-1.13-20260917). Re-fit after any Jev version change.
• Meant for a human-in-the-loop personal system with staged rollout, not for unattended call blocking at scale.