lothnic/themis-vl-2b

Themis 2B text + vision

A small multimodal model for typed decisions. Give Themis a case — text, up to two images, or both — plus questions and their possible answers. It returns a probability distribution for every question in one forward pass. It supports named choices, yes/no decisions, and ordered scores, without generating or parsing answer text.

This repository contains the complete model weights (the frozen Qwen3.5-2B text decoder and vision tower), tokenizer, image processor, decision encoder, calibration settings, and inference code. No separate base-model download is required. The trained LoRA layers are retained inside the complete checkpoint to preserve the numerical path used for evaluation; this is a self-contained model release, not an adapter-only repository.

Property	Value
Developer	lothnic
Version	Themis v1 / 2B text + vision
Backbone	Qwen3.5-2B text decoder (24 layers) and vision tower, both frozen
Active parameters	2,213,241,664 core + 33,638,400 adapter parameters, plus a 255 × 2,048 answer readout
Inputs	A text state, up to two PIL images, and a mapping of typed questions
Outputs	Per-question answers and named probabilities
Rendered context	4,096 tokens, including the case, questions, options, template and image tokens
Options per question	2–255; constrained by the context limit
Images	0–2 per request, 64–1,024 visual tokens each
Released checkpoint	Step 3,258, selected by validation soft cross-entropy
License	Apache 2.0; see LICENSE and NOTICE

Quick start

Use Python 3.12. The following versions are the tested inference environment:

pip install torch==2.14.1 transformers==5.19.0 peft==0.21.2 tokenizers==0.23.3 safetensors huggingface_hub pillow
# Required for CUDA inference (the reported local evaluations used it):
pip install flash-linear-attention==0.5.2

import torch
from transformers import AutoImageProcessor, AutoModel, AutoTokenizer

repo = "lothnic/themis-vl-2b"
revision = "v1.0.0"
tokenizer = AutoTokenizer.from_pretrained(repo, revision=revision, trust_remote_code=True)
image_processor = AutoImageProcessor.from_pretrained(
    repo, revision=revision, trust_remote_code=True, backend="pil"
)
model = AutoModel.from_pretrained(
    repo,
    revision=revision,
    trust_remote_code=True,
    dtype=torch.bfloat16,
).to("cuda").eval()

result = model.predict(
    state="A customer was charged twice for the same order and requests a refund.",
    questions={
        "route": {
            "type": "choice",
            "instructions": "Which team should handle this document?",
            "criteria": {
                "billing": "Charges, payments, and refunds",
                "technical": "Software faults and account access",
                "sales": "Product information and new purchases",
            },
        },
        "needs_review": {
            "type": "noul",
            "instructions": "Does the document need a human review?",
            "criteria": {"true": "Review needed", "false": "No review needed"},
        },
        "urgency": {
            "type": "score",
            "instructions": "Rate urgency using the following ordered levels.",
            "criteria": ["Low", "Medium", "High"],
        },
    },
    images=[],  # optional PIL images, in order; sizes enter the prompt, pixels enter the tower
    tokenizer=tokenizer,
    image_processor=image_processor,
)
print(result)

trust_remote_code=True loads the decision architecture and encoder shipped here. Review the source and pin revision to a release commit for reproducible use. This model uses a custom typed-request interface: the standard string-label pipeline("text-classification") interface and .generate() are not supported.

For CPU inference, load with dtype=torch.float32 and leave the model on CPU; the convolutional fallback used there requires no extra packages. CPU results can differ slightly from BF16 CUDA results. CUDA BF16 on an RTX 4060 Laptop GPU is the locally verified reference environment. Other accelerators and quantized runtimes have not been validated.

Response contract

The response is a mapping keyed by the original question names. Each entry has answer and probabilities. For example, with illustrative probabilities:

{
  "route":        {"answer": "billing", "probabilities": {"billing": 0.93, "technical": 0.05, "sales": 0.02}},
  "needs_review": {"answer": "true", "probabilities": {"true": 0.81, "false": 0.19}},
  "urgency":      {"answer": "2", "probabilities": {"0": 0.05, "1": 0.22, "2": 0.73}},
}

• choice: criteria is a mapping from caller-defined option names to descriptions. answer is an original option name, and probabilities use those names.
• noul: yes/no. answer is "true" or "false"; probability keys are always "true", then "false".
• score: criteria is a list ordered from lowest to highest. answer is the string index of the most probable level; probability keys are string indices. Convert the index to your own numeric scale explicitly.

Calibration is enabled by default: one fitted positive temperature per decision type (choice 1.2431, noul 1.2455, score 1.1750). Pass calibrated=False to predict() for raw probabilities. Positive temperature scaling preserves each question's most probable answer.

Options and images are supplied at inference time; there is no single global label vocabulary. Requests over 4,096 rendered tokens raise an error instead of silently truncating the evidence. Literal special-token strings in user-supplied text are escaped by the encoder. Images are read only for size (width/height descriptors in the prompt) and pixel content (vision tower); no file paths, URLs or metadata enter the request.

How it works

Themis uses the 24-layer Qwen3.5-2B text decoder: 18 gated linear-attention layers and six full-attention layers. Full-attention layers receive bidirectional attention masks over real tokens; linear-attention layers retain their forward recurrence. Padding is excluded from the full-attention keys. Images are encoded by the native, frozen Qwen vision tower; visual tokens are inserted at the token boundary immediately preceding the rendered case, and native spatial position indices are computed with the pinned Transformers methods.

The encoder renders the case, every question, and all answer descriptions, followed by fixed <answer> placeholders. For each question, the hidden state immediately before its placeholder is scored against the frozen embedding rows for that question's valid answer codes. Softmax is computed over those options only, explicitly in FP32 inside the BF16 context. The tokenizer mapping verifies 255 distinct single-token codes. The <answer> embedding was initialized from the existing embedding mean and is part of the checkpoint. Base embeddings, the vision tower and the readout vectors were frozen during training; LoRA updated the text-decoder linear projections, including the recurrent projections. The release computes no vocabulary-wide language-model logits.

The forward path uses FLA 0.5.2's Triton causal convolution for the recurrent layers on CUDA, exactly as trained and evaluated; install flash-linear-attention for CUDA use. On CPU the reference convolution helper is used instead.

Multiple questions share a rendered request, so their surrounding context can affect one another, and question order is not guaranteed to be irrelevant (measured below). The response probabilities describe the provided alternatives; they are not a guarantee that those alternatives include the right action.

Training and checkpoint selection

Base: Qwen/Qwen3.5-2B, revision 15852e8c16360a2fea060d615a32b45270f8a8fc.

Data: lothnic/themis-decision-mixture-v1, revision 93b15c9fba1bfe96b4403626ea9f5c5f8c2fc2fb. Source datasets, attribution, and source-specific terms are documented in the dataset card.

Split	Decisions	Packed requests
Train	73,247	52,111
Validation	4,336	3,148
Calibration	3,883	2,801
Test	5,177	3,943

Training ran for three epochs, finishing at 4,887 optimizer steps on one H100 at a 4,096-token window, effective batch 32 requests. Only decoder LoRA was trained (rank 32, alpha 64, dropout 0.05, all decoder linear projections). The published model is the earlier checkpoint at step 3,258, which had the lowest validation soft cross-entropy (0.5026). Epoch 3 continued to lower the training loss (0.583 → 0.373 → 0.253) while validation soft CE worsened to 0.5386; end-of-training behavior is documented in the project repository, including one external board where the later checkpoint scores better on a narrow workflow distribution and one where it scores worse.

Evaluation

Every number below belongs to step 3,258. The benchmarks use fixed corpora and different scoring rules. They were scored by the exported predictor directly, using the same calibration settings; a few Decision Bench rows that exceeded 4,096 rendered tokens were re-rendered with the limit raised to 8,192. These results are not all zero-shot or independent of the mixture test split; see the protocol notes below.

In-domain mixed test split, 5,177 decisions:

Metric	Value
Answer agreement	83.93% (82.43% text, 92.94% image + text)
Calibrated soft CE / Brier / hard ECE	0.5274 / 0.0767 / 0.0227

Evaluation	Denominator	Accuracy	Brier	NLL	ECE
Decision Bench bench-v4, all rows	1,071	83.01%	0.2338	0.4058	0.0247
— text rows	949	81.66%
— text + image rows	62	91.94%
— image-only rows	60	95.00%
Typed Decisions official test	2,000 decisions	72.90%	0.0340 (raw soft CE 0.8905)
SysOne Bench v2.1.0, raw	1,240 decisions	81.94%
SysOne Bench v2.1.0, calibrated	1,240 decisions	81.69%

Decision Bench macro category accuracy is 83.19% and macro task F1 is 0.8097 across eleven categories. Uncalibrated Decision Bench metrics (more comparable with step counts than calibrated ones): Brier 0.2389, NLL 0.4190, ECE 0.0477. The Typed Decisions test corpus is generated from public workflow material; the overlap audit and metric definitions are included in evaluation.json.

Benchmark protocol and overlap. The training mixture includes 554 cases from the official Typed Decisions training split, so its test score is a fitted evaluation. Normalized exact-state checks found no official test states in the mixture's training, validation or calibration splits. Of the 400 official test cases, 172 also occur as partial-question records in the mixture's test split, so those two reported evaluations are not independent. Paraphrase overlap and base-model pretraining exposure were not assessed. Decision Bench also draws on public sources; its result is not a claim of a contamination-free evaluation. The SysOne audit found no exact evaluation-state matches in the mixture, plus one generic phrase match. Brier and ECE definitions differ between benchmarks; compare checkpoint results within the same protocol.

Behavior probes on the test split (prediction sensitivity only; they do not fabricate targets or validate counterfactual labels):

Probe	Cases	Mean total variation	Answer flip rate
Choice option permutation	256	0.045	5.9%
Image removed	47	0.601	55.3%
Unrelated image substituted	47	0.717	74.5%
Question reordered among others	50	0.038	10.0%

Intended use and limitations

Themis is a decision-scoring model for structured pipelines: routing, triage, layout and document checks, and similar typed choices where probabilities matter more than generated prose. It expects the caller to provide the full decision options.

• No answer generation, no free-form output, no vocabulary-wide logits.
• 4,096-token rendered limit and at most two images per request; larger requests raise an error.
• Answers describe the provided options only; a correct option that is not among the criteria cannot be returned.
• Image understanding comes from the frozen Qwen vision tower plus a LoRA-adapted decoder; the image counterfactuals above show that predictions are highly image-dependent, which also means a wrong or low-quality image changes answers.
• The model is English-prompted in training and evaluation; other languages, domains and long-context use beyond 4,096 tokens are unvalidated.
• Do not use for decisions about people, safety, medical, legal or financial outcomes without independent review.

Files and reproducibility

• model-*.safetensors (+ model.safetensors.index.json): complete checkpoint, including the frozen backbone, vision tower, LoRA adapter and the 255-row answer readout.
• config.json, modeling_themis_vl.py, configuration_themis_vl.py, codec_themis_vl.py: architecture, configuration and encoder for trust_remote_code=True.
• tokenizer.json, tokenizer_config.json, preprocessor_config.json: text tokenizer and image processor.
• calibration.json: fitted temperatures (one per decision type).
• training.json, evaluation.json, release-verification.json: selected checkpoint, training provenance, evaluated metrics and the packaging check.
• checksums.sha256: SHA-256 of every file in this repository.

The release was verified by reloading the saved snapshot with AutoModel.from_pretrained(..., trust_remote_code=True) and comparing 14 reference predictions — API contract examples plus curated text and image requests, with images embedded in the reference file — against the exported predictor: no missing or unexpected tensors, all parameter dtypes preserved, and a maximum probability difference below 1e-6. The comparison is recorded in release-verification.json; evaluation.json contains the benchmark results and protocol metadata.

A measured long-context preflight accompanies this release: single 32K/128K/256K windows with the trained settings do not fit an H100 80GB (activation checkpointing off), while with checkpointing on a 128K window trains at 7,062 tokens/s and 46.96 GiB peak (≈ US$0.21 per million tokens at conservative rates). Long-context continuation is therefore possible with checkpointing enabled, but no long-context checkpoint has been trained or validated.