Instructions to use torchcast-ai/torchcast-decision-31b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use torchcast-ai/torchcast-decision-31b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="torchcast-ai/torchcast-decision-31b")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("torchcast-ai/torchcast-decision-31b")
model = AutoModelForMultimodalLM.from_pretrained("torchcast-ai/torchcast-decision-31b", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use torchcast-ai/torchcast-decision-31b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "torchcast-ai/torchcast-decision-31b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "torchcast-ai/torchcast-decision-31b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/torchcast-ai/torchcast-decision-31b

SGLang

How to use torchcast-ai/torchcast-decision-31b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "torchcast-ai/torchcast-decision-31b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "torchcast-ai/torchcast-decision-31b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "torchcast-ai/torchcast-decision-31b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "torchcast-ai/torchcast-decision-31b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use torchcast-ai/torchcast-decision-31b with Docker Model Runner:
```
docker model run hf.co/torchcast-ai/torchcast-decision-31b
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Torchcast Decision 31B (v1.0)

A text-based decision model developed by Torchcast AI, based on Gemma-4-31B-it. It returns yes/no probabilities (noul), choice distributions (choice) and ordinal score distributions (score) through a TypeSafe-compatible API. Each decision reads the option letters at the answer position in one forward pass (choice lists with more than 26 options are decided in a few passes over groups of options); nothing is generated.

Use the bundled torchcast_decision serving package with the bundled decision_config.json. Generic Hub text-generation examples and causal-attention runtimes (vLLM, llama.cpp, MLX) do not reproduce the evaluated interface: the global attention layers attend to the whole prompt in both directions (see below).

Serving instructions

Model

Weights: a full fine-tune of google/gemma-4-31B-it (all layers except the input embeddings) on a mix of public datasets (official training splits) and synthetic decision data covering choice, yes/no and score questions. No benchmark test items were used for training.
Attention: hybrid. The sliding-window layers are causal as in the base model; the global layers use no causal mask within a prompt (attention_mode: hybrid_bidir_global), as in training.
Prompt format: torchcast_gemma4_v1 (bundled torchcast_decision/prompt.py), Gemma 4's chat format with thinking off.
Serving settings (decision_config.json): temperature 0.8 (choice), 0.6 (yes/no), 0.6 (score); yes/no commit floor 0.801.

Results

Board results pending: the model has not yet been measured on the JevBench leaderboard.

Artifact

Pinned revision: tag v1.0.0 of this repository.
Weight SHA-256: see SHA256SUMS.
Served model name: torchcast-decision-31b-v1.0.

Serving settings

Serve the model through the bundled torchcast_decision package (Torchcast AI, Apache License 2.0); decision_config.json holds the serving settings. Within a prompt, the global attention layers use no causal mask (as in training), so loading the weights with plain transformers, vLLM, llama.cpp or MLX gives different probabilities. The bf16 weights need one GPU with at least 80 GB: tested on an H100 80 GB (FlashAttention-3 kernel for the sliding-window layers, pinned to a fixed commit and prefetched for offline starts) and an RTX PRO 6000 96 GB (PyTorch SDPA kernel). Prompts longer than 16,384 tokens keep the head and the tail of the state, as in training. At start-up the server captures CUDA graphs for prompts up to 4,096 tokens (about one minute); graph and eager outputs are bit-identical. Repeated runs on the same GPU type give identical probabilities. One request at a time over the public JevBench items, the median latency is 48 ms on the H100 and 74 ms on the RTX PRO 6000; the longest prompt takes 4.1 s and 6.2 s. See NOTICE and the serving instructions on GitHub.

Intended use and limitations

English text-based decision research, such as classification, routing and bounded rubric evaluation. The server returns decisions and does not execute actions. Probabilities can be unreliable on new domains.

Licence and attribution

Model weights: Apache License 2.0 (see LICENSE). Base model: Gemma-4-31B-it by Google, Apache License 2.0 (NOTICE, Gemma 4 licence). Serving package: Apache License 2.0 (torchcast_decision/LICENSE, torchcast_decision/NOTICE). This project is independent of TypeSafe AI and the JevBench maintainers.

Downloads last month: 23

Safetensors

Model size

33B params

Tensor type

BF16

Model tree for torchcast-ai/torchcast-decision-31b

Base model

google/gemma-4-31B

Finetuned

google/gemma-4-31B-it

Finetuned

(333)

this model

Quantizations

1 model