mini-clef

mini-clef is a small System 1 decision model (Qwen 3.5 0.8B, routes consumer complaints). You send a state and typed questions. It reads them once (one forward pass, no text generation) and returns a probability for every allowed option. Plain code then writes the JSON, so the output is always valid and always one of the options you allowed.

It is Qwen/Qwen3.5-0.8B with LoRA (merged) plus a small joint schema head, trained on a Mac (Apple Silicon, PyTorch MPS) to route US consumer-finance complaints to one of 9 financial products.

The head follows the design of Cloudflare's open Clef (Apache-2.0), scaled down from a 27B backbone to 0.8B. This model is not made by, or affiliated with, Cloudflare or TypeSafe. The request and response bodies follow the Jev / SystemOne shape (state + questions in, answers with probabilities out).

What this model is for

This is a teaching model. It was fine-tuned on a very small sample: 800 complaints, for 16 minutes, on a laptop. Production decision models such as Jev or Clef learn from vastly more, and far more varied, data. That is what lets them generalise to new questions and new domains, and this model does not. Its purpose is to show, end to end, how a general LLM built to think and write (a "System 2" model) is turned into a System 1 decision model that answers in one pass, and what the small details are that make this work.

How it works

1. One prompt holds everything. The state comes first, then the schema: each question's instruction and its allowed options, written as text. Because the options are text in the prompt, not fixed output slots, the labels are not baked into the weights.
2. One forward pass through the LLM. Its word head (248k words) is not used. The model keeps the hidden state of every token.
3. Joint schema head (≈ 16M parameters, width 512):
   • a question vector = the mean hidden state over its instruction;
   • an option vector = the mean hidden state over the option's text + the option's own word embeddings + its question;
   • 1 evidence-routing layer: every option attends over the whole prompt to collect its evidence;
   • 2 transformer layers: the questions look at each other and at the prompt, so they are decided jointly;
   • a scorer gives one logit per allowed option; a softmax per question gives the probabilities.
4. systemone() turns the probabilities into the response body (choice, confidence, probabilities).

Use

from miniclef import load_release_model, systemone   # miniclef.py is in this repo

model, tok = load_release_model("kgptalkie/mini-clef", device="mps")  # or "cuda" / "cpu"
request = {
    "state": {"complaint": "My car payment was taken twice this month and the lender won't refund it."},
    "questions": {
        "product": {
            "type": "choice",
            "instructions": "Which financial product is the consumer's `complaint` about?",
            "criteria": {
                "credit_reporting": "credit reports, credit bureaus, credit repair, identity on report",
                "debt_collection": "debt collectors, collection calls, debts owed",
                "credit_card": "credit cards and prepaid cards",
                "bank_account": "checking or savings accounts",
                "mortgage": "home loans and mortgages",
                "money_transfer": "money transfers, wire, payment apps, virtual currency",
                "vehicle_loan": "car loans or vehicle leases",
                "student_loan": "student loans",
                "personal_loan": "payday, title, personal or installment loans",
            },
        }
    },
}
print(systemone(model, tok, request))
# {"answers": {"product": {"type": "choice", "choice": "...", "confidence": ..., "probabilities": {...}}}, "usage": {..., "output_tokens": 0}}

Training

Data	kgptalkie/cfpb-complaint-routing: short CFPB consumer complaints (CC0-1.0), ≤ 200 tokens, 9 products
Train / val	pool of 2,700 / 270 (300 / 30 per product), test complaints removed by id and text; v0.1 saw 800 of them (100 steps × 8)
Trained	LoRA r 16 on the language model (10.2M) + the joint schema head (15.8M); everything else frozen, LoRA merged for release
Objective	cross-entropy on the right option. Half of the rows show only a random subset of the options (always incl. the right one), so the head must read the options
Hardware	MacBook Pro M5 Max, 36 GB, PyTorch MPS, bf16, gradient checkpointing; 7.1 GB allocated on the GPU at the end of training
Steps	100 × batch 8, AdamW (LoRA 2e-4, head 1e-3), cosine
Time	15 min 36 s

Results

Test = the 826 complaints of kgptalkie/cfpb-complaint-routing's test split (≤ 200 tokens, from a fixed 1,800-complaint CFPB benchmark set), 9 products.

Model	Accuracy
Qwen3.5-0.8B, prompted for JSON (no tuning)	43.6 %
Laya (ModernBERT-large + decision head)	54.6 %
This model (v0.1, 100 steps)	73.1 %

CFPB product labels are chosen by the consumers who file the complaints, so some labels are wrong or ambiguous (for example a credit-card line on a credit report). No model will reach 100 % on them.

Updates

This first version is a quick test run on a Mac (100 steps), made to show the technique and a real improvement, not the best possible accuracy. Longer training runs (on an RTX 5090) will be pushed to this repo as new versions, and this table will record each one.

Version	Date	Training	Test accuracy
v0.1	2026-10-17	100 steps × 8 complaints, Mac, 15.6 min	73.1 %

Limits

• Trained on one question type (choice) and one task (routing to 9 products). noul and score questions run, but were not trained.
• English, US consumer-finance complaints, ≤ 200 tokens in training. Longer or other-domain text is untested.
• Speed on a Mac is ≈ 350 ms per complaint (batch 16): without the flash-linear-attention / causal-conv1d kernels (CUDA only), Transformers runs Qwen 3.5's linear-attention layers in plain PyTorch.
• Probabilities are trained with log loss, not calibrated or checked for calibration beyond that.

Credits

• Base model: Qwen/Qwen3.5-0.8B (Apache-2.0).
• Joint schema head design and prompt layout: Cloudflare/clef (Apache-2.0); miniclef.py is a scaled-down, text-only adaptation of its joint_schema_model.py.
• Data: CFPB Consumer Complaint Database via BEE-spoke-data (CC0-1.0).
• Made for the YouTube series LLM Fine-Tuning on Mac by KGP Talkie.