Instructions to use togethercomputer/Tev1-4B-experimental with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use togethercomputer/Tev1-4B-experimental with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="togethercomputer/Tev1-4B-experimental")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("togethercomputer/Tev1-4B-experimental")
model = AutoModelForMultimodalLM.from_pretrained("togethercomputer/Tev1-4B-experimental", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use togethercomputer/Tev1-4B-experimental with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "togethercomputer/Tev1-4B-experimental"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "togethercomputer/Tev1-4B-experimental",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/togethercomputer/Tev1-4B-experimental

SGLang

How to use togethercomputer/Tev1-4B-experimental with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "togethercomputer/Tev1-4B-experimental" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "togethercomputer/Tev1-4B-experimental",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "togethercomputer/Tev1-4B-experimental" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "togethercomputer/Tev1-4B-experimental",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use togethercomputer/Tev1-4B-experimental with Docker Model Runner:
```
docker model run hf.co/togethercomputer/Tev1-4B-experimental
```

Real-world agent-routing results: 40/40, zero confident-wrong, ~120 ms local

by Krioasns - opened about 18 hours ago

Discussion

Krioasns

about 18 hours ago

Thanks for releasing this. We've been using Tev1-4B-experimental (Q8_0 GGUF, locally on an RTX 4070 Ti SUPER) as the pre-turn router in an open-source desktop assistant, and wanted to share real-world numbers in case they're useful.

Task: before each turn, pick which of 6 specialist agents handles it and whether the turn needs tools. We ask one yes/no question per agent and read P(yes) for each.

Results (62 prompts, one run):

Routing: 40/40 correct, adversarial set 9/10, multi-question "surmise" set 12/12
Zero confident-wrong answers: every turn it settled on its own was right
~120 ms median per routing decision, vs ~460 ms for qwen3.5:9b as an LLM router on the same GPU
It settled 30 of 40 routing turns by itself; the rest fell back to the 9B LLM

One thing that helped a lot: we originally required every non-winning option to score ≤0.30. Tev1 often gives the right agent ~0.95 with one distractor around 0.35-0.40, which forced pointless fallbacks. Accepting a "dominant single yes" (winner ≥0.85, all others <0.50, gap ≥0.35) raised the share it settles alone from 30% to 75% with no wrong answers. If its probabilities are calibrated that way by design, that rule may be worth suggesting to other users.

Per-case results, including Tev1's per-question probabilities, are here: https://huggingface.co/datasets/Krioasns/system1-decision-routing-bench

Happy to share more detail if it's useful for the next version.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment