ANT-4B — a 2.6 M-parameter decision head on Qwen3.5-4B

ANT-4B is an open-source System 1 decision model from the Semantique Team. It takes a state, a question and a list of candidate answers, and returns a calibrated probability over the candidates plus an execute/escalate verdict. It never generates text.

This repository contains the trained head only (2,632,704 trainable parameters, 10.5 MB): the frozen backbone stays where it belongs, in Qwen/Qwen3.5-4B (Apache-2.0).

Status: work in progress. The checkpoint format and the Python API are usable and the numbers below are reproducible, but benchmarks and production guidance are still being finalized. Read the limitations before shipping this anywhere.

How it works

state + question          candidates (any number, any text)
        ↓                              ↓
   ContextEncoder                 CandidateEncoder
        ↓                              ↓
     z_context                    z_candidate_i
          └────────────┬─────────────────┘
                       ↓
              scoring function  →  s_i = f(z_context, z_candidate_i)
                       ↓
              softmax(s / T)  →  P(candidate_i | state, question)
                       ↓
              confidence metrics → execute / escalate

Two encoders share one frozen Qwen3.5 decoder loaded through AutoModel without the LM head. The decoder is used as an encoder; the only trained parameters are the two projections, the input norms and a temperature. Consequences worth knowing:

• N is variable (2 to 100+). The candidate set is given at inference time, so the same checkpoint routes a ticket to four departments or ranks sixty CVs.
• Permutation invariant. Reordering the candidates does not move the probabilities.
• Cacheable. Static candidate sets can be embedded once and reused, so a follow-up question over the same options only pays for the context encoding.

Results

Frozen backbone, 2,632,704 trainable parameters out of 4.2 B, 3 epochs, batch size 8, lr 2e-4, seed 7, trained on a Kaggle T4. Same dataset and same recipe as semantique-team/ant-08b, so the comparison between the two is between backbones, not between training runs.

Split	n	Accuracy	ECE
Test (held-out situations, 24 use cases)	648	0.739	0.056
OOD (held-out use cases, 5 unseen)	456	0.597	0.123

Best validation accuracy during training: 0.731 (epoch 3).

The escalation policy is fitted on validation data at a target risk of 0.10: at a confidence threshold of 0.769 the model covers 48.4 % of decisions at 9.9 % risk, and the decisions it does take are 90.1 % correct (n = 688). That gap between coverage and automation accuracy is the whole point of the escalation policy: when the model is unsure it hands the decision back instead of guessing.

The 4B model earns its size on the hard cases. Against ANT-08B on the same test rows: bug_component 0.70 vs 0.33, incident_severity 0.78 vs 0.56, candidate_screening 0.69 vs 0.47, project_status 0.50 vs 0.28. It loses on a few simpler ones (meeting_action 0.59 vs 0.72, urgenza_ticket_it 0.34 vs 0.38).

How to use

Install the runtime from GitHub (it is not on PyPI yet) and point it at this repository:

pip install "ant-4b @ git+https://github.com/semantique-team/ant-4b"

from huggingface_hub import snapshot_download
from ant.inference.predictor import ANTPredictor

# downloads this repo: ant_head.pt, calibrator.pt, escalation.json, config.json, ...
CHECKPOINT = snapshot_download("semantique-team/ant-4b")

ant = ANTPredictor.load(CHECKPOINT)   # backbone: Qwen/Qwen3.5-4B, fetched from the Hub

result = ant.predict(
    state="Customer has been charged twice for the same invoice. Refund requested.",
    question="Which department should handle this case?",
    candidates=["Billing", "Technical Support", "Sales", "Legal"],
)

print(result.choice)        # "Billing"
print(result.probabilities) # {"Billing": 0.85, "Legal": 0.10, ...}
print(result.action)        # "execute" | "escalate"

That output is a real run of this checkpoint on the MLX 8-bit backbone (see below) on an Apple Silicon laptop: Billing 0.85, Legal 0.10, Sales 0.05, Technical Support 0.01, verdict execute. The exact probabilities move with the backend and its precision — the fp16 torch backbone and the MLX one do not agree to the third decimal — but the choice and the verdict are stable. ANTPredictor.load reads the frozen backbone straight from Qwen/Qwen3.5-4B, so nothing else has to be downloaded by hand.

Apple Silicon / MLX

The torch path above runs anywhere. On Apple Silicon the MLX backend is the faster one. An 8-bit MLX backbone is already on the Hub, so nothing has to be converted locally:

from ant.backbone.mlx import MlxBackbone
from ant.inference.predictor import ANTPredictor

backbone = MlxBackbone(
    "mlx-community/Qwen3.5-4B-MLX-8bit",
    device="cpu",
    quantization="mlx",
    quantization_bits=8,
    quantization_group_size=64,
)
ant = ANTPredictor.load(CHECKPOINT, backbone=backbone)

Pass device="cpu" to the MLX backbone. The frozen model runs inside MLX on the GPU, but its hidden states come back to PyTorch as CPU tensors; with device="mps" the scoring head is placed on MPS and the first layer_norm fails with "Passed CPU tensor to MPS op".

To convert the backbone yourself instead:

python scripts/mlx_convert.py \
  --hf-path models/Qwen3.5-4B \
  --mlx-path models/Qwen3.5-4B-mlx \
  --no-quantize

Quantized MLX backbones are accepted only when their bits and group size match the request (--bits 8 --group-size 64 for the 8-bit variant). Metal 8-bit quantization is off for this architecture family: measured on ANT-4B it produced all-zero hidden states under fp16 and NaN under bf16, and dropped gate accuracy from 0.725 to 0.325 (40-row gate sample). That is a property of the quantizer on these shapes, not of the loading code — a small locally built Qwen3.5 text model under the same quantizer stayed numerically correct.

Latency on Apple Silicon (M3, 16 GB, torch path)

Warm median encode latency after the optimizations described in the source repository: 372 ms on a 14-token input, 373 ms at 53 tokens, 1,446 ms at 210 tokens, 2,866 ms at 418. The first predict() call adds ~1.3 s because it compiles the length buckets. The floor is the linear attention: 24 of the 32 layers are GatedDeltaNet, and without fused CUDA/Triton kernels on MPS they fall back to a correct but slow reference path. Milliseconds on long inputs are not reachable for a 4B hybrid on a 16 GB machine today.

What is in this repository

File	What it is
ant_head.pt	The trained head: 8 tensors, 10.5 MB. torch.save blob containing state_dict, config and backbone_hidden_size.
calibrator.pt	The fitted temperature calibrator (log_temperature).
config.json	The ANT configuration (backbone, scorer, pooling, dimensions, limits).
escalation.json	The fitted risk/coverage policy: threshold, coverage, risk.
metrics.json	Test and OOD metrics, overall and per use case.
examples.json	Evaluation examples with their gold labels.
run_config.json, training_report.json, card.json	Training arguments, loss/accuracy history, display name.

SHA-256 of the weights:

486675655255b5eaebf9cd0183563763e5d613b6909d12d220978cc1adb713c3  ant_head.pt
e46abbd5319f228775411a4f67b17655f2593dcc713a0cf73cfd5aa1d2181ebb  calibrator.pt

Training data

A synthetic multitask dataset generated by scripts/build_dataset.py (seed 7): 5,952 training rows, 688 validation, 648 test and 456 out-of-distribution rows, across 29 use cases in English and Italian (24 in train/val/test: support routing, incident severity, expense category, bug component, email action, urgenza_ticket_it, approvazione_spese_it, and so on; plus 5 use cases held out entirely as OOD). Every row is a state, a question, its candidate set and the labelled answer.

The dataset is not published on the Hub: it is regenerated deterministically from the repository, so this card carries the metrics as a table rather than as a model-index block (which the Hub ignores when the dataset id does not resolve).

Limitations, stated plainly

1. transaction_risk is near chance — 0.125 accuracy on 8 test rows. The task needs better data, not more parameters.
2. urgenza_ticket_it sits at 0.344 on 32 test rows. Same story, and more parameters did not fix it.
3. OOD calibration degrades — ECE 0.123 out of distribution against 0.056 on the test split. Treat any confidence value on an unseen domain as uncalibrated.
4. Single seed, single run. Every number here comes from one training run. No variance over seeds was measured.
5. Training loss fell to 0.005 while validation loss fell much more slowly. The head has enough capacity to memorize the training rows; the validation curve is the honest one.
6. It is a decision model, not a language model. There is no generation path, no chat, no free-form output. Give it candidates and it ranks them.
7. The backbone is Qwen's, not ours. The weights here are the trained head only; the frozen decoder is downloaded from Qwen/Qwen3.5-4B under its own Apache-2.0 licence.
8. The escalation policy is fitted on this dataset. On a different domain the threshold means something else.
9. Serving it on a 16 GB Mac is tight. The plain bf16 load of the 4B backbone is 8.4 GB, and the quantized paths are not yet validated for this checkpoint.

Licence and attribution

• This model repository: Apache-2.0 (see LICENSE).
• The backbone is Qwen/Qwen3.5-4B, Apache-2.0. The notice travels in NOTICE.
• Source code: github.com/semantique-team/ant-4b.

Provenance

Checkpoint ant-4b-2026-10-24, trained on a Kaggle T4 with Qwen/Qwen3.5-4B frozen and head_only mode (2,632,704 trainable parameters out of 4.2 B). Every number in this card comes from the committed artifacts in this repository (metrics.json, escalation.json, training_report.json) and from the evaluation run described in the source repository's EXPERIMENTS.md.