lothnic/themis-0.8b

Themis 0.8B

A small, text-only model for typed decisions. Give Themis a case, questions, and their possible answers. It returns a probability distribution for each question in one forward pass. It supports named choices, yes/no decisions, and ordered scores, without generating or parsing answer text.

This repository contains the complete model weights, tokenizer, decision encoder, calibration settings, and inference code. No separate base-model download is required. The trained LoRA layers are retained inside the complete checkpoint to preserve the numerical path used for evaluation; this is a self-contained model release, not an adapter-only repository.

Property	Value
Developer	lothnic
Version	Themis v0 / 0.8B
Backbone	Qwen3.5-0.8B text decoder
Active parameters	752,393,024 base + 21,645,312 adapter parameters
Inputs	A text state and a mapping of typed questions
Outputs	Answers, named probabilities, confidence; expected value for scores
Rendered context	2,048 tokens, including questions, options, and template
Options per question	2–255; constrained by the context limit
Modalities	Text only
Released checkpoint	Step 4,153, selected by validation soft cross-entropy
License	Apache 2.0; see LICENSE and NOTICE

Quick start

Use Python 3.12. The following versions are the tested inference environment:

pip install torch==2.14.1 transformers==5.19.0 peft==0.21.2 tokenizers==0.23.3 safetensors huggingface_hub
# Optional CUDA acceleration used in the reported local evaluations:
pip install flash-linear-attention==0.5.2

import torch
from transformers import AutoModel, AutoTokenizer

repo = "lothnic/themis-0.8b"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModel.from_pretrained(
    repo,
    trust_remote_code=True,
    dtype=torch.bfloat16,
).to("cuda").eval()

result = model.predict(
    state="A customer was charged twice for one order and asks for a refund.",
    questions={
        "route": {
            "type": "choice",
            "instructions": "Which team should handle this request?",
            "criteria": {
                "billing": "Charges, payments, and refunds",
                "technical": "Software faults and account access",
                "sales": "Product information and new purchases",
            },
        },
        "needs_review": {
            "type": "noul",
            "instructions": "Does the reported duplicate charge need investigation?",
            "criteria": {"true": "Investigation needed", "false": "No investigation needed"},
        },
        "urgency": {
            "type": "score",
            "instructions": "Rate urgency using the following ordered levels.",
            "criteria": ["Low", "Medium", "High"],
        },
    },
    tokenizer=tokenizer,
)
print(result)

trust_remote_code=True loads the decision architecture and encoder shipped here. Review the source and pin revision to a release commit for reproducible use. This model uses a custom typed-request interface: the standard string-label pipeline("text-classification") interface and .generate() are not supported.

For CPU inference, load with dtype=torch.float32 and leave the model on CPU. CPU results can differ slightly from BF16 CUDA results. CUDA BF16 on an RTX 4060 Laptop GPU is the locally verified reference environment. Other accelerators and quantized runtimes have not been validated.

Response contract

The response has model, answers, calibrated, and input_tokens fields. answers is keyed by the original question names. Each answer contains type, answer, probabilities, and confidence.

• choice: criteria is a mapping from caller-defined option names to descriptions. answer is an original option name, and probabilities use those names.
• noul: yes/no. answer is a Boolean. Probability keys are always "true", then "false"; the internal distribution is [P(true), P(false)].
• score: criteria is a list ordered from lowest to highest. answer is the most probable zero-based level index, probability keys are string indices, and expected_score is the probability-weighted mean index. Convert this index to your own numeric scale explicitly.

Calibration is enabled by default. Pass calibrated=False to predict() for raw probabilities. Positive temperature scaling preserves each question's most probable answer, but can change an expected score or a threshold-based decision.

Options are supplied at inference time; there is no single global label vocabulary. Requests over 2,048 rendered tokens raise an error instead of silently truncating the evidence. Literal special-token strings in user-supplied text are escaped by the encoder.

How it works

Themis uses the 24-layer Qwen3.5 text decoder: 18 gated linear-attention layers and six full-attention layers. Full-attention layers receive bidirectional attention masks over real tokens; linear-attention layers retain their forward recurrence. Padding is excluded from the full-attention keys.

The encoder renders the case, every question, and all answer descriptions, followed by fixed <answer> placeholders. For each question, the hidden state immediately before its placeholder is scored against the frozen embedding rows for that question's valid answer codes. Softmax is computed over those options only. The tokenizer mapping verifies 255 distinct single-token codes.

The <answer> embedding was initialized from the existing embedding mean and frozen. The base embeddings and output-code vectors were frozen during training; LoRA updated text-decoder projections. The released model uses the same encoding, masks, and readout as training. It computes no vocabulary-wide language-model logits and contains no vision tower or auxiliary multi-token prediction weights.

Multiple questions share a rendered request, so their surrounding context can affect one another. The response probabilities describe the provided alternatives; they are not a guarantee that those alternatives include the right action.

Training and checkpoint selection

Base: Qwen/Qwen3.5-0.8B, revision 2fc06364715b967f1860aea9cf38778875588b17.

Data: lothnic/themis-decision-mixture, revision 92843bdf66de205d6b4238c6a0d70701fd6737a7. Source datasets, attribution, and source-specific terms are documented in the dataset card.

Split	Decisions	Packed requests
Train	44,577	33,573
Validation	3,170	2,275
Calibration	2,606	1,945
Test	3,693	2,781

The training split contains 30,400 choices, 7,292 yes/no decisions, and 6,885 ordered scores; 12,900 decisions have soft probability targets. Labels, categories, and provenance are excluded from model inputs. Choice options and their targets are permuted together during training. Yes/no and ordered-score ordering is preserved.

Training ran for three epochs, finishing at 6,297 optimizer steps on one A100 40 GB. The published model is the earlier checkpoint at step 4,153, which had the lowest validation soft cross-entropy, 0.656687. The final step's validation loss was higher, so completing training did not make the final weights the selected release.

Setting	Value
Precision	BF16 base, FP32 LoRA parameters, BF16 CUDA autocast
LoRA rank / alpha / dropout	32 / 64 / 0.05
Optimizer	AdamW, weight decay 0.01
Learning rate	1e-4, cosine decay to 1e-5, 3% warmup
Effective batch	16 packed requests
Objective	Soft cross-entropy, normalized by decisions per accumulation window
Gradient clipping	1.0
Gradient checkpointing	Enabled
Seed	3407

One positive temperature per decision type was fitted on the calibration split only: choice 1.157726, yes/no 1.107598, score 1.020533. The benchmark evaluations below did not update model weights or refit these temperatures.

Evaluation

All results below use checkpoint 4,153. External benchmark results are local reproductions, not operator-verified leaderboard submissions. The tasks and scoring rules differ, so the percentages should not be treated as directly comparable measures of general intelligence.

Evaluation	Decisions	Result	Protocol
Mixture held-out test	3,693	77.34%	Agreement with target distribution's highest-probability option
Typed Decisions official test	2,000	70.45%	Agreement with stored gold.label; 400 cases, five questions per case
SysOne Bench v2.1.0	1,240	76.85%	953 correct decisions across 952 evaluation cases; official scorer

Mixture held-out test

Metric	Raw	Calibrated
Soft cross-entropy ↓	0.708560	0.704117
KL divergence ↓	0.393148	0.388705
Half-vector Brier score ↓	0.097691	0.096900
Argmax agreement ↑	77.34%	77.34%
ECE on 2,134 one-hot targets, 15 bins ↓	4.21%	2.28%

The Brier score in this table is 0.5 * sum((p - target)^2), averaged over decisions. It must not be directly compared with reports using full-vector Brier or binary positive-class Brier.

On 256 deterministic choice examples, changing option order changed the raw top answer in 16/256 cases (6.25%). Mean total-variation distance was 0.06115. The model is not invariant to option ordering.

Typed Decisions

Dataset: LocalLLaMA/typed-decisions, revision 7189f46b02248bd2569df7325c9df8e3b24e90a7, all/test. All 400 cases and 2,000 questions fit the context limit; no truncation or inference errors occurred. Accuracy uses stored labels rather than recomputing labels from rounded probability distributions.

Workflow	Accuracy
Invoice	83.4%
Customer support	69.4%
Agents	67.0%
Security	62.0%

This is not a zero-shot benchmark claim. The Themis training split includes 1,747 decisions from 554 cases in this benchmark's training data. Exact case-state/ID audits found no official test cases in Themis train, validation, or calibration. However, 352 official-test decisions from 172 cases were already represented in the mixture's test split and had been evaluated previously. This limits the independence of subsequent evaluation and model-development conclusions. These checks do not rule out semantic overlap or base-model pretraining exposure.

Calibration did not improve every out-of-distribution metric: Typed Decisions soft cross-entropy rose from 0.898235 to 0.899606, and 15-bin stored-label ECE rose from 10.33% to 11.87%. Confidence should be assessed on the intended deployment distribution.

SysOne Bench

Source: instax-dutta/sysone-bench, commit 93afcfe1bd450dc052b21768e2b95b20316ebb68, corrected v2.1.0 manifest. The manifest SHA-256 is 0d21a64c6b61b6be7b49050dbd89f89f934912a7b586734e475308e9c9422ea2.

The benchmark has 1,190 cases in total. The headline excludes its 238 calibration cases and uses only 952 evaluation cases / 1,240 decisions. Ordered-score answers use the probability-weighted expected level, with the official nearest-level correctness rule. This differs from modal-level classification. Existing mixture temperatures are used; benchmark-fitted calibration diagnostics do not alter the headline answers.

Suite	Accuracy
Multilingual intent	95.00%
Triage	92.71%
Moderation	91.67%
AG News	81.25%
Guardrails	81.25%
Banking77 subset	80.21%
Emotion	72.40%
MNLI	54.17%
SST-5	33.33%

Normalized whole-state matching found no exact overlap with the mixture splits. A literal-content audit flagged one evaluation case containing “Please delete my account.”, also found inside an unrelated aggression-rating training example. Excluding that case gives 952/1,239 = 76.84%. This is a limited text-overlap check, not proof of zero contamination. Corrected labels use upstream source labels and reviewed emotion labels; no independent double-adjudication claim is made. The upstream ECE implementation differs from standard count-weighted ECE, so its values are not presented here as standard ECE.

Intended use and limitations

Themis is intended for experimentation with routing, triage, classification, rubric-based assessment, and probabilistic decision components. Evaluate it against your own evidence, options, and error costs before relying on its answers.

• Text only: no image, document-image, bounding-box, or region outputs.
• A small decision model can choose confidently from flawed options, misread a rubric, or miss factual and contextual details. A highest-probability option is not a verified fact.
• The 2,048-token limit includes the entire request. Longer contexts and automatic evidence truncation are not validated.
• Confidence is not universally calibrated. Domain shifts, option wording, option order, and other questions in the request can alter predictions.
• This release has no demonstrated fairness, robustness, or safety guarantee. “Security”, “moderation”, and “guardrails” benchmark scores do not make it a complete security or moderation system.
• Its sentiment and natural-language-inference results show substantial room for improvement. Strong scores on short intent tasks should not be generalized to all decision tasks or all languages.
• The released weights are intended for inference. Training resume additionally requires the original optimizer, scheduler, sampler, and RNG states; those are not included here.

Files and reproducibility

• model.safetensors: complete text-backbone and trained LoRA weights; no base-model network fetch at load time.
• config.json: architecture, codec mapping, and fitted temperatures.
• modeling_themis.py, configuration_themis.py, codec_themis.py: custom model and request encoder.
• Tokenizer files: the exact saved tokenizer, including <answer>.
• calibration.json, training.json, evaluation.json: settings, pinned revisions, and aggregate evaluation results.
• release-verification.json, checksums.sha256: local inference-equivalence checks and file integrity hashes.

The architecture is adapted from the Qwen3.5 text backbone, with task-specific encoding, attention behavior, and readout as described above. The base weights remain subject to their Apache 2.0 license and attribution; this release includes the upstream license and a notice identifying the modifications.