g1nie/broken-model-fixed

broken-model (fixed)

This repository is a corrected version of yunmorning/broken-model. The original repository could not serve a working /chat/completions endpoint. This README documents exactly which files were modified, what the change was, and why it was necessary.

TL;DR

File	Change	Reason
tokenizer_config.json	Added the chat_template field (canonical Qwen3 Jinja template).	/chat/completions cannot format role-based messages without a chat template; the inference server (vLLM, SGLang, Friendli Engine) errors out or falls back to a wrong default. This is the root cause.
README.md	Changed base_model: meta-llama/Meta-Llama-3.1-8B to Qwen/Qwen3-8B.	The weights, architecture, tokenizer, and vocabulary all belong to Qwen3-8B, not Llama-3.1. The wrong metadata misleads consumers and can break downstream tooling that inspects base_model.

No other file was modified. config.json, generation_config.json, model.safetensors.index.json, the weight shards, tokenizer.json, vocab.json, and merges.txt are identical to the original because they were already correct.

Root-cause analysis

Evidence that the model is Qwen3-8B (not Llama-3.1-8B)

The original README declares base_model: meta-llama/Meta-Llama-3.1-8B, but the artifacts on disk tell a different story:

Signal	Value in repo	What it means
config.json -> architectures	["Qwen3ForCausalLM"]	Loaded by the Qwen3 model class. Llama would be LlamaForCausalLM.
config.json -> model_type	"qwen3"	Routes to the Qwen3 model registry in transformers.
config.json -> vocab_size	151936	Qwen tokenizer vocab. Llama-3.1 vocab is 128256.
tokenizer_config.json -> tokenizer_class	"Qwen2Tokenizer"	BPE tokenizer with vocab.json and merges.txt (Qwen family).
added_tokens_decoder	<|im_start|>, <|im_end|>, <tool_call>, <think>, etc.	ChatML-style markers used by Qwen, not by Llama.
model.safetensors.index.json	Contains model.layers.N.self_attn.k_norm.weight and q_norm.weight	Qwen3-specific per-head Q/K RMSNorm. Llama-3.x has no such layers.
total_size	16.38 GB in BF16, roughly 8.19 B params	Matches Qwen3-8B (around 8.2 B). Llama-3.1-8B is around 8.03 B and has a smaller intermediate size of 14336, not 12288.
config.json -> head_dim, num_hidden_layers, intermediate_size	128 / 36 / 12288	Exactly Qwen3-8B. Llama-3.1-8B is 128 / 32 / 14336.

So the base_model tag in the original card was simply wrong. This is metadata only and does not affect inference; we correct it for hygiene, but did not consider it a "bug" in the inference sense.

Why /chat/completions was broken

The /chat/completions endpoint (whether served by vLLM, SGLang, TGI, or Friendli Engine) follows the OpenAI Chat Completions contract. It receives a list of {"role": ..., "content": ...} messages and must convert them to a single token stream before sending it to the model. That conversion is performed by the Jinja template stored as tokenizer.chat_template, exposed in tokenizer_config.json.

In the original repo, tokenizer_config.json did not contain a chat_template field at all. When the chat template is missing:

• vLLM raises ValueError: As of transformers v4.44, default chat template is no longer allowed, so you must provide a chat template … and the request fails with HTTP 400.
• SGLang and TGI behave similarly. Either they refuse to start the OpenAI route, or they fall back to a generic template that does not emit the <|im_start|> / <|im_end|> markers Qwen3 was trained on, producing nonsense output.
• Friendli Engine likewise requires chat_template for its OpenAI-compatible route.

The other artifacts (text completions endpoint, raw generate) would still work because they bypass templating, which is consistent with a "completions doesn't work" report rather than a "model doesn't load" report.

Why the fix is only a chat_template insertion

The model was clearly stripped of its template but otherwise left intact. Restoring the canonical Qwen3-8B chat template (taken verbatim from Qwen/Qwen3-8B) is the minimal change that resolves the failure. Every other field in tokenizer_config.json is already correct for Qwen3: eos_token is <|im_end|>, pad_token is <|endoftext|>, special tokens 151643 through 151668 (including <think> and </think>) are all properly registered.

The restored template:

• Emits ChatML structure (<|im_start|>role\ncontent<|im_end|>).
• Supports the tools=[...] parameter for function calling, embedding tool definitions in the system message and wrapping calls in <tool_call>…</tool_call> (the same tokens already declared in added_tokens_decoder).
• Supports the enable_thinking boolean used by Qwen3 to gate <think>…</think> blocks (which feeds directly into Problem (b)).
• Honors add_generation_prompt so streaming endpoints know where to start the assistant turn.

Things that look suspicious but are not bugs

I considered and rejected several other candidates so the fix stays minimal:

• config.json: max_position_embeddings = 40960 vs. tokenizer_config.json: model_max_length = 131072. This looks like a contradiction but it matches the upstream Qwen/Qwen3-8B repo exactly. 40960 is the native pre-trained context; 131072 is the YaRN-extended cap once rope_scaling is enabled. Not an inference bug.
• bos_token = null, add_bos_token = false. Correct for the Qwen family; the model was not trained with a BOS token.
• Both vocab.json plus merges.txt and tokenizer.json present. Redundant but harmless; the fast tokenizer.json is preferred, the legacy files exist for backward compatibility. Not a bug.
• eos_token_id = 151645 in config.json but [151645, 151643] in generation_config.json. Intentional. Stopping on either <|im_end|> (turn boundary) or <|endoftext|> (document boundary) is the standard Qwen3 behavior.

Reproduction

After applying the fix:

vllm serve g1nie/broken-model-fixed \
  --served-model-name broken-model

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "broken-model",
        "messages": [{"role":"user","content":"Hello"}]
      }'

This returns a valid Chat Completion response. Before the fix, the same command fails with a missing-chat-template error.