Instructions to use migtissera/Tess-4-27B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use migtissera/Tess-4-27B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="migtissera/Tess-4-27B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("migtissera/Tess-4-27B")
model = AutoModelForMultimodalLM.from_pretrained("migtissera/Tess-4-27B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use migtissera/Tess-4-27B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "migtissera/Tess-4-27B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "migtissera/Tess-4-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/migtissera/Tess-4-27B

SGLang

How to use migtissera/Tess-4-27B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "migtissera/Tess-4-27B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "migtissera/Tess-4-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "migtissera/Tess-4-27B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "migtissera/Tess-4-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use migtissera/Tess-4-27B with Docker Model Runner:
```
docker model run hf.co/migtissera/Tess-4-27B
```

MTP not working?

by henrykrinkle - opened 5 days ago

Discussion

henrykrinkle

5 days ago

Tess-4-27B ships the 15 mtp.* tensors, and they are byte-identical to the Qwen/Qwen3.6-27B base MTP head — the fine-tune reused it rather than retraining. llm-compressor drops MTP during quantisation (transformers v5 doesn't instantiate the submodule), so it was regrafted in bf16 from the source checkpoint and the 15 tensors are present in this repo's safetensors index for vLLM/SGLang to pick up.
Measured with vLLM (method: qwen3_5_mtp, num_speculative_tokens: 1, greedy, 12 mixed chat/code/reasoning prompts): The fine-tune shifted the trunk so far that the inherited head is useless for speculation: 0.0% of drafted tokens were accepted, and speculative decoding is counterproductive (it slows generation, since every rejected draft is wasted compute). Serve without --speculative-config. The mtp.* tensors are still shipped (matching the parent's structure and available for anyone who retrains the head), but do not enable MTP speculative decoding as-is. Positive control: the base Qwen/Qwen3.6-27B MTP head accepts 92.7% on the identical vLLM stack, so the number below is a genuine property of Tess's fine-tuned trunk, not a serving artefact. End-to-end throughput was 35.3 tok/s with speculation on vs 40.6 tok/s off = 0.87×.

https://huggingface.co/huginnfork/Tess-4-27B-FP8#speculative-decoding-mtp

It seems just grafting the MTP head of the base model onto the finetuned version isn't enough. Our community (club 3090) has benchmarked this model locally for agentic workloads and it turned out to be the first finetuned model of Qwen 3.6 27B that has significant improvements over the base model. However, the speed is painfully slow (48 tps compared to 85-110 tps, same fp8 quant served on vLLM).

migtissera

Owner 5 days ago

Damn, yeah I wasn’t paying attention to MTP at all. Also, I trained with 64K context, like legit 64K context length sanitized agentic traces — and it was impossible to find a GPU cluster for that big context. So I used Tinker from Thinking Machines for this finetune. I don’t think they include MTP in the LoRA. I’ll see what I can do, but unfortunately this may be the compromise for the quality gain. I have finetuned the Qwen3.6 35B MoE as well — I can release that if it would help with the speed? Might be lower quality though.

henrykrinkle

5 days ago

Sadly, this is a big compromise. Your model is legit the first one that got us excited after many failed variants (Qwopus, Qwable, Ornith, Thinking Cap etc). All of those are within noise or worse than the base model. And 35b is always worse than 27b. Your model released just after we converged into a high quality, production ready stack (fp8, 85-110 tps, doesn't degrade with long context). Improvements are substantial even with thinking off. We'll wait for your 35b version and see.

migtissera

Owner 5 days ago

That's fair. Do you guys usually fine-tune the MTP head as well? I've been out of the game for a while, been focusing on the application layer AI and only just getting back into fine-tuning.

henrykrinkle

5 days ago

No we aren't doing any fine tuning. We focus on getting the best quantizations and configs to serve different models on consumer GPUs, mostly RTX 3090s but applicable to 4090s and 5090s as well.

AFAIK, the MTP heads can be trained separately: https://docs.vllm.ai/projects/speculators/en/latest/user_guide/tutorials/train_mtp_online/
Any chance you could do that as well?

migtissera

Owner 5 days ago

I'm not sure right now. I think this is upstream of me honestly -- it's Tinker. But GPT-5.6 drops this morning, and my Fable-5 quota resets tonight. I'll see what I can do.

Does the 35B-A3B come with MTP as well? Which of these models do you guys like? https://tinker-docs.thinkingmachines.ai/tinker/models/ -- This is the easiest way for me to get a fine-tune out right now. All my other work is focused on very large MoE fine-tunes and won't be suitable for local LLM community.

henrykrinkle

5 days ago

Yes the 35B-A3B does come with MTP. However, it isn't as impactful because the model is inherently fast. In addition to that, for this model MTP is counterproductive when TP=2 and beyond. It slows down token generation because of the overheads.

The general consensus of the local llm community is that around 30b is the sweet spot for a dense model. Currently Qwen 3.6 27b is the king. Nothing has been close to beat it yet. We do discuss and tinker with other models like Gemma (not as good) or bigger models like Deepseek V4, Hy3, GLM 5.2 but the hardware requirement is too high and the speed is too slow for any meaningful work.

migtissera

Owner 5 days ago

Figured out how to fine-tune the MTP head. It's running right now. Should have an update later today or tomorrow.

henrykrinkle

5 days ago

Glad to hear that! Looking towards the new King!

migtissera

Owner 4 days ago

Hey! Here you go: https://huggingface.co/migtissera/Tess-4-27B-EAGLE3

SGLang: --speculative-algorithm EAGLE3 --speculative-draft-model-path migtissera/Tess-4-27B-EAGLE3
vLLM: --speculative-config '{"method":"eagle3","model":"migtissera/Tess-4-27B-EAGLE3","num_speculative_tokens":4}'

migtissera changed discussion status to closed 4 days ago

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment