engram-xenc-v0

Cross-encoder for Engram's substitute-hit gating layer. Given a user query and a candidate knowledge object (KO) body, predicts whether the KO answers the query well enough to substitute for a full LLM call.

Trained by distilling Claude Sonnet's ACCEPT/REJECT judgments on 1,560 (query, KO) candidate pairs sampled from the Bitext customer-support dataset.

Engram context

• Codebase consumer: sidecar/gating_cross_encoder.py — invoked on retrieval candidates that pass an initial cosine-distance cut. A positive score routes the query to substitute; a negative routes to grounded-LLM.
• Fetch via: scripts/fetch_models.sh in the Engram repo (idempotent post-clone install step).
• Reproduce locally: python tools/session8_5_train_eval.py (~15 minutes on M4 Max). Inputs are data/session8_5_labeled.json (1,560 rows; 1,248 train / 312 test deterministic split). Deterministic seed (RANDOM_SEED=42).

Eval metrics (on held-out 20% test split)

Metric	This model	Baseline (1 − cosine)
ROC-AUC	0.961	0.800
Average precision	0.931	0.617
Best F1 (threshold = 0.8)	0.889	—
Inference latency	6.5 ms/pair	—

AUC improvement of +0.161 over the cosine baseline; the model earns its keep specifically in the ambiguous-distance zone where cosine alone over-substitutes.

Numbers from data/session8_5_eval.json at training time (2026-04-30). Stochastic MPS ops mean re-trains land within ~±0.01 AUC, not bit-identical.

Training details

• Base model: cross-encoder/ms-marco-MiniLM-L-12-v2 (12-layer cross-encoder pre-trained on MS MARCO passage ranking).
• Loss: binary cross-entropy on the verifier labels (ACCEPT=1, REJECT=0).
• Hyperparameters: 3 epochs, batch size 16, 100 warmup steps.
• Wall time: 869.6 s on Apple M4 Max via MPS backend.

Intended use

• Best for: customer-support / FAQ retrieval where a finite knowledge base covers most queries and the system needs to decide whether to answer from the KB at all.
• Out of scope: open-domain QA, long-form generation, anything needing more than the top-1 KO. This is a gating signal, not a generator.

Limitations

• Domain: trained on customer-support queries (Bitext). Will transfer poorly to legal, medical, or technical-research domains without re-training. The Engram repo's tools/session8_5_train_eval.py is designed to be re-run per customer corpus.
• Calibration: the 0.8 F1-optimal threshold is corpus-specific. Re-calibrate against your own labels before relying on it.
• No PII / safety filter. Inherits the base model's behavior.

License

Apache 2.0. Derived from cross-encoder/ms-marco-MiniLM-L-12-v2 (Apache 2.0). Training data: Bitext customer-support dataset (verify upstream license at huggingface.co/datasets/bitext/Bitext-customer-support-llm-chatbot-training-dataset before any commercial use).

Citation

@misc{engram-xenc-v0,
  author = {Pascaline / Engram},
  title  = {engram-xenc-v0: cross-encoder for KO substitute-hit gating},
  year   = {2026},
  url    = {https://huggingface.co/jchiang11/engram-xenc-v0}
}