Competence Gate — a metacognitive tool-gate for Qwen3.5-4B

A small (~10 MB) LoRA adapter that adds calibrated tool-gating to Qwen/Qwen3.5-4B: it decides when to answer directly, when to look something up, and when to retrieve from your own documents — based on the model's internal competence signal rather than its (saturated) verbalised confidence. It ships with an orchestration substrate that routes private-flavoured queries to a local retriever instead of public web search.

Scope note (read this). The validated, benchmark-supported claim here is parametric tool-call routing — deciding when to answer from the model's own knowledge vs. defer to a tool — and the privacy-routing result below. The gate does not improve grounded document QA (answering faithfully from a provided passage): on that task it does not help and can push toward answering. Grounded-abstention is a separate construct the gate was not trained for; see Scope & what this does not do below. Claims are kept to what external benchmarks support.

Built by Jon-Paul Cacioli (synthiumjp.github.io). The validity-screening and metacognitive-measurement methods used here are shared with Signal & Thread, an independent AI- measurement lab.

This is an open research release, not a commercial product. It is a capability demonstrator — it shows what the science used to measure AI looks like when applied to build a calibrated system. Nothing here is sold, and it is not part of any measurement service. Apache-2.0.

This is an adapter, not a model. The base Qwen3.5-4B (Apache-2.0) is pulled at load time; only the ~10 MB gate is distributed here. The adapter is useless without the orchestration substrate (included) — the gate makes decisions, the substrate enforces them.

What it does (and the evidence for each claim)

This card is written as an instrument-validation dossier: every claim below is reported with its effect size, confidence interval, or screening control, and its honest limitations. Numbers that came in smaller under tightening are reported at the tightened value, not the first look.

1. Reads competence internally (within-band AUROC 0.868, VRS-Valid)

A linear probe on the model's layer-18 hidden state predicts whether the model will answer a factual question correctly, at within-difficulty-band AUROC 0.868 (difficulty-balanced PopQA, n=420), passing validity screening (VRS). The signal is distributed in the DeltaNet recurrent state (linear-attention component alone matches the full state), a first-reported architectural finding.

2. Beats the base model's own tool-gating (Δd′ 0.46, 95% CI [0.01, 0.89])

Distilled into a behavioural gate, it significantly out-discriminates Qwen3.5-4B's native "call vs answer" behaviour: Δd′ = 0.46, 95% CI [0.01, 0.89], P(Δ>0) = 0.976 (pooled held-out n=420, 5000-sample bootstrap). Of items the gate caught that the base missed, 87% were genuinely wrong — it catches real errors, not just calling more.

3. Reduces privacy leaks (privacy-error 0.22→0.10, 95% CI on reduction [0.02, 0.22])

A two-signal variant routes private-document questions to a local retriever instead of public web search. On held-out private queries (n=60), it significantly reduces the rate at which confidential queries are mis-routed to public search: 0.217 → 0.100, reduction 0.117, 95% CI [0.017, 0.217] (CI excludes zero). This is the clinically load-bearing safety property.

4. Two dissociated metacognitive signals (a research finding)

Retrieval-appropriateness ("this needs an external source") and factual competence ("did I answer correctly") are separable internal signals at different layers: retrieval-appropriateness is decodable from layer 1 (comprehension-time), factual competence from layer 18 (post-answer). Screened at n=126 against three controls — embedding-layer AUROC at chance (0.49), permutation-null clean (0.56), and a margin over a lexically-adversarial bag-of-words control. AUROC≈1.0 indicates in-sample separability, not 1.0 generalisation; pending replication at scale.

Deployment fidelity (GGUF, verified)

The GGUF build reproduces the validated (MLX) gate's decisions at --lora-scaled qwen_gate.gguf:8. On a 24-item probe, agreement is 0.83, with perfect agreement on safety-critical directions: zero false tool-calls on known-answerable questions, zero leak-direction reversals. Residual disagreements are conservative-direction (the GGUF gate answers a few borderline items the MLX gate would look up) — the safe error direction. Do not run at scale 1 (the gate does not fire) or above ~8 (over-scaling breaks known-answer behaviour).

Honest limitations

Serve-time confidence bands are coarse. The distilled gate reads nothing at inference, so the shipped confidence signal is route/grounding-based (grounded / declined / answered). The finer confident/tentative gradation needs probe access (offline/eval), and is not available in the GGUF serving path.
Small-n on two claims. The privacy-leak result is n=60; the dissociation is n=126 hand-authored items. Both are screened/CI'd but would benefit from scale.
Not a substitute for professional judgment. This is a decision-support instrument. In clinical, legal, or other high-stakes settings its output must be verified by a qualified human. It is designed to fail to an audited "I don't know," not a confident guess — but "I don't know" is still its job, not yours.
Base-model bounded. It inherits Qwen3.5-4B's knowledge and biases; the gate governs when to trust that knowledge, not what the knowledge is.

Scope & what this does not do (a benchmark-honest boundary)

The gate was trained and validated for one construct: parametric competence — do I know this from my own weights, or should I defer to a tool? An honest external-benchmark check shows this signal does not transfer to a different metacognitive construct, evidential grounding — is this answer supported by the passage in front of me? These look alike from outside ("knowing when to defer") but are separable signals, and the gate only carries the first. The evidence, reported straight:

On grounded document QA the gate does not help — and can hurt. Given a passage and a question, the gate pushes toward answering rather than deferring (it suppresses the base model's own "not in the passage" instinct). On SQuAD 2.0 unanswerable questions (n=300), fabrication was higher with the gate than without it. The gate is a parametric tool-router, not a grounded-reading- comprehension abstainer.
The base model already handles the easy case. When a question is paired with a passage that plainly does not contain the answer (natural-absent, n=150), base-model fabrication was 0% — Qwen3.5-4B already declines correctly without any firewall. There is little for a grounding layer to add in normal document QA.
The hard case is not cheaply fixable. On adversarial unanswerables (SQuAD 2.0, written to look answerable), an entailment (NLI) firewall can cut fabrication but only by declining a large fraction of genuinely-answerable questions — a specialist refuse-vs-answer trade-off with no threshold that gets both. Purpose- built grounded-abstention models (trained end-to-end for it) are the right tool there; a post-hoc firewall on this gate is not.

Practical takeaway: use this for tool-call routing and privacy-aware retrieval routing, which it does measurably well. Do not rely on it as a "won't fabricate about your documents" guarantee — that is a different construct, it is largely a base-model property in normal use, and this gate does not improve it. A fuller writeup of this construct-specificity finding is on the project site.

Quickstart

# 1. clone + install
git clone https://huggingface.co/synthiumjp/competence-gate-qwen3.5-4b
cd competence-gate-qwen3.5-4b
pip install -r requirements.txt

# 2. configure (copy the example, edit if needed)
cp models.example.yaml models.yaml

# 3. put documents to retrieve from in ./docs (.txt .md .pdf .docx)
mkdir -p docs

# 4. run the substrate (the gate + firewalls + retriever + citations)
from registry import Registry
import orchestrator_qwen as O
from conversation import Conversation
reg = Registry.load("models.yaml")
models = {n: O.Model(s) for n, s in reg.specs.items()}
conv = Conversation(reg, models)

r = conv.ask("What did our Q3 report say was the top revenue region?")
# -> answer cited to the source passage, with a route/grounding band; declines
#    when it can't verify. NOTE: the base model already declines well when a fact
#    is plainly absent; the substrate does not add reliable grounded-abstention
#    beyond that, and should not be relied on for adversarial unanswerable cases.

The base Qwen/Qwen3.5-4B (~8 GB) downloads on first load. For GGUF/Ollama serving, see the Modelfile (serve at --lora-scaled qwen_gate.gguf:8).

Usage

MLX (Apple Silicon)

from mlx_lm import load, generate
model, tok = load("Qwen/Qwen3.5-4B", adapter_path="adapters_qwen_gate")
# ...but use the orchestrator, not raw generate — the gate needs the firewalls:

The substrate (orchestrator + retriever + citations)

from registry import Registry
import orchestrator_qwen as O
from conversation import Conversation
reg = Registry.load("models.yaml")
models = {n: O.Model(s) for n, s in reg.specs.items()}
conv = Conversation(reg, models)
r = conv.ask("What did our Q3 report say was the top revenue region?")
# -> answer cited to the source passage, with a route/grounding band; declines
#    when it can't verify. NOTE: the base model already declines well when a fact
#    is plainly absent; the substrate does not add reliable grounded-abstention
#    beyond that, and should not be relied on for adversarial unanswerable cases.

GGUF / Ollama

ollama create competence-gate -f Modelfile   # Modelfile sets --lora-scaled ...:8

Research foundation

This gate is a constructive application of a published research programme on LLM metacognition (measurement theory for language models). Each design decision traces to a specific result:

Why read competence internally rather than trust verbalised confidence. Verbal confidence at the 3-9B instruct scale is psychometrically Invalid -- all seven models tested hit a confidence ceiling despite carrying internal item-level information (Cacioli, Verbal Confidence Saturation in 3-9B Open-Weight Instruction-Tuned LLMs, arXiv:2604.22215). The gate reads the internal signal precisely because the verbalised one is unreliable at this scale.
Validity screening (VRS) before interpreting any confidence signal. Six indices cross-mapped from the PAI and MMPI-3; three-tier Invalid/Indeterminate/ Valid classification; concurrently validated against selective-prediction AUROC (Cacioli, derivation arXiv:2604.17707; protocol arXiv:2604.17714; criterion validation arXiv:2604.17716). Every probe in this gate is VRS-screened before use.
Distilling a probe signal into behaviour (the gate mechanism). Probe-targeted fine-tuning recovers probe-level discrimination in a shipped model, and activation patching localises confidence to a position-specific routing step (Cacioli, Making LLMs Say What They Know: Probe-Targeted Fine-Tuning, Zenodo:10.5281/zenodo.20436841). The gate distils a frozen probe into a LoRA by the same logic.
Why quantised/GGUF serving needs a fidelity check. Quantisation reshapes metacognitive geometry (M-ratio profiles uncorrelated across formats) while discrimination rankings stay stable (Cacioli, Quantisation Reshapes the Metacognitive Geometry of Language Models, arXiv:2604.08976). This is exactly why GGUF fidelity is verified on decision agreement (stable) rather than assumed.
The know-say gap, in signal-detection terms. Type-2 SDT separates how much a model knows from how well it knows what it knows (Cacioli, Do LLMs Know What They Know?, arXiv:2603.25112).

Full programme and code: synthiumjp.github.io · GitHub · ORCID 0009-0000-7054-2014.

Files

adapters_qwen_gate/ — the validated competence gate (mlx)
adapters_qwen_twosignal/ — the two-signal (privacy-routing) gate (mlx)
qwen_gate.gguf — GGUF adapter (serve at scale 8)
registry.py, orchestrator_qwen.py, retriever.py, conversation.py, confidence.py — the orchestration substrate
Modelfile — Ollama config at the verified scale

Citation

@software{competence_gate_qwen35_4b,
  author = {Cacioli, Jon-Paul},
  title  = {A calibrated metacognitive tool-gate for Qwen3.5-4B},
  note   = {Open research release},
  url    = {https://synthiumjp.github.io/},
  year   = {2026}
}

Apache-2.0. Built on Qwen3.5-4B (Apache-2.0).