ANT-08B — a 1 M-parameter decision head on Qwen3.5-0.8B

ANT-08B is an open-source System 1 decision model from the Semantique Team. It takes a state, a question and a list of candidate answers, and returns a calibrated probability over the candidates plus an execute/escalate verdict. It never generates text.

This repository contains the trained head only (1,053,696 trainable parameters, 4.2 MB): the frozen backbone stays where it belongs, in Qwen/Qwen3.5-0.8B (Apache-2.0).

Status: work in progress. The checkpoint format and the Python API are usable and the numbers below are reproducible, but benchmarks and production guidance are still being finalized. Read the limitations before shipping this anywhere.

How it works

state + question          candidates (any number, any text)
        ↓                              ↓
   ContextEncoder                 CandidateEncoder
        ↓                              ↓
     z_context                    z_candidate_i
          └────────────┬─────────────────┘
                       ↓
              scoring function  →  s_i = f(z_context, z_candidate_i)
                       ↓
              softmax(s / T)  →  P(candidate_i | state, question)
                       ↓
              confidence metrics → execute / escalate

Two encoders share one frozen Qwen3.5 decoder loaded through AutoModel without the LM head. The decoder is used as an encoder; the only trained parameters are the two projections, the input norms and a temperature. Consequences worth knowing:

• N is variable (2 to 100+). The candidate set is given at inference time, so the same checkpoint routes a ticket to four departments or ranks sixty CVs.
• Permutation invariant. Reordering the candidates does not move the probabilities.
• Cacheable. Static candidate sets can be embedded once and reused, so a follow-up question over the same options only pays for the context encoding.

Results

Frozen backbone, 1,053,696 trainable parameters out of 753 M, 3 epochs, batch size 8, lr 2e-4, seed 7, trained on a Kaggle T4. Same dataset and same recipe as semantique-team/ant-4b, so the comparison between the two is between backbones, not between training runs.

Split	n	Accuracy	ECE
Test (held-out situations, 24 use cases)	648	0.674	0.061
OOD (held-out use cases, 5 unseen)	456	0.570	0.220

Best validation accuracy during training: 0.757 (epoch 3).

The escalation policy is fitted on validation data at a target risk of 0.10: at a confidence threshold of 0.802 the model covers 44.2 % of decisions at 9.9 % risk, and the decisions it does take are 90.1 % correct (n = 688). That gap between coverage and automation accuracy is the whole point of the escalation policy: when the model is unsure it hands the decision back instead of guessing.

How to use

Install the runtime from GitHub (it is not on PyPI yet) and point it at this repository:

pip install "ant-08b @ git+https://github.com/semantique-team/ant-08b"

from huggingface_hub import snapshot_download
from ant.inference.predictor import ANTPredictor

# downloads this repo: ant_head.pt, calibrator.pt, escalation.json, config.json, ...
CHECKPOINT = snapshot_download("semantique-team/ant-08b")

ant = ANTPredictor.load(CHECKPOINT)   # backbone: Qwen/Qwen3.5-0.8B, fetched from the Hub

result = ant.predict(
    state="Customer has been charged twice for the same invoice. Refund requested.",
    question="Which department should handle this case?",
    candidates=["Billing", "Technical Support", "Sales", "Legal"],
)

print(result.choice)        # "Billing"
print(result.probabilities) # {"Billing": 0.99, ...}
print(result.action)        # "execute" | "escalate"

That output is a real run of this checkpoint with the torch backbone on the Hub: Billing 0.99, Legal 0.02, Sales 0.01, Technical Support 0.001, verdict execute. The exact probabilities move with the backend and its precision, but the choice and the verdict are stable.

ANTPredictor.load reads the frozen backbone straight from Qwen/Qwen3.5-0.8B, so nothing else has to be downloaded by hand.

Apple Silicon / MLX

The torch path above runs anywhere. On Apple Silicon the MLX backend is the faster one. An 8-bit MLX backbone is already on the Hub, so nothing has to be converted locally:

from ant.backbone.mlx import MlxBackbone
from ant.inference.predictor import ANTPredictor

backbone = MlxBackbone(
    "mlx-community/Qwen3.5-0.8B-MLX-8bit",
    device="cpu",
    quantization="mlx",
    quantization_bits=8,
    quantization_group_size=64,
)
ant = ANTPredictor.load(CHECKPOINT, backbone=backbone)

Pass device="cpu" to the MLX backbone. The frozen model runs inside MLX on the GPU, but its hidden states come back to PyTorch as CPU tensors; with device="mps" the scoring head is placed on MPS and the first layer_norm fails with "Passed CPU tensor to MPS op".

To convert the backbone yourself instead:

python scripts/mlx_convert.py \
  --hf-path models/Qwen3.5-0.8B \
  --mlx-path models/Qwen3.5-0.8B-mlx \
  --no-quantize

Quantized MLX backbones are accepted only when their bits and group size match the request (--bits 8 --group-size 64 for the 8-bit variant). Metal 8-bit quantization is off for this architecture family: measured on ANT-4B it produced all-zero hidden states under fp16 and NaN under bf16, and dropped gate accuracy from 0.725 to 0.325 (40-row gate sample). That is a property of the quantizer on these shapes, not of the loading code — a small locally built Qwen3.5 text model under the same quantizer stayed numerically correct.

What is in this repository

File	What it is
ant_head.pt	The trained head: 8 tensors, 4.2 MB. torch.save blob containing state_dict, config and backbone_hidden_size.
calibrator.pt	The fitted temperature calibrator (log_temperature).
config.json	The ANT configuration (backbone, scorer, pooling, dimensions, limits).
escalation.json	The fitted risk/coverage policy: threshold, coverage, risk.
metrics.json	Test and OOD metrics, overall and per use case.
examples.json	Evaluation examples with their gold labels.
run_config.json, training_report.json, card.json	Training arguments, loss/accuracy history, display name.

SHA-256 of the weights:

cb79a147fbf4c5d45850e050275bfc0fa4025a52311255f6d272ec98f52bd859  ant_head.pt
3f3f1f19f3347a9175a6ecec2f16836cfaff19ad050a072b6ac5f8613456faa2  calibrator.pt

Training data

A synthetic multitask dataset generated by scripts/build_dataset.py (seed 7): 5,952 training rows, 688 validation, 648 test and 456 out-of-distribution rows, across 29 use cases in English and Italian (24 in train/val/test: support routing, incident severity, expense category, bug component, email action, urgenza_ticket_it, approvazione_spese_it, and so on; plus 5 use cases held out entirely as OOD). Every row is a state, a question, its candidate set and the labelled answer.

The dataset is not published on the Hub: it is regenerated deterministically from the repository, so this card carries the metrics as a table rather than as a model-index block (which the Hub ignores when the dataset id does not resolve).

Limitations, stated plainly

1. transaction_risk is at chance — 0.000 accuracy on 8 test rows. The task needs better data, not more parameters.
2. urgenza_ticket_it sits at 0.375 on 32 test rows. Same story.
3. bug_component (0.325) and project_status (0.281) are where the 0.8B backbone shows its limit: ANT-4B reaches 0.70 and 0.50 on the same rows.
4. OOD calibration is weak: ECE 0.220 on the out-of-distribution split against 0.061 on the test split. Treat any confidence value on an unseen domain as uncalibrated.
5. Single seed, single run. Every number here comes from one training run. No variance over seeds was measured.
6. Training loss fell to 0.010 while validation loss fell much more slowly. The head has enough capacity to memorize the training rows; the validation curve is the honest one.
7. It is a decision model, not a language model. There is no generation path, no chat, no free-form output. Give it candidates and it ranks them.
8. The backbone is Qwen's, not ours. The weights here are the trained head only; the frozen decoder is downloaded from Qwen/Qwen3.5-0.8B under its own Apache-2.0 licence.
9. The escalation policy is fitted on this dataset. On a different domain the threshold means something else.

Licence and attribution

• This model repository: Apache-2.0 (see LICENSE).
• The backbone is Qwen/Qwen3.5-0.8B, Apache-2.0. The notice travels in NOTICE.
• Source code: github.com/semantique-team/ant-08b.

Provenance

Checkpoint ant-08b-2026-10-24, trained on a Kaggle T4 with Qwen/Qwen3.5-0.8B frozen and head_only mode (1,053,696 trainable parameters out of 753 M). Every number in this card comes from the committed artifacts in this repository (metrics.json, escalation.json, training_report.json) and from the evaluation run described in the source repository's EXPERIMENTS.md.