sofisticated/broken-model-update

https://huggingface.co/sofisticated/broken-model-update

A

The original configuration files indicated both Qwen and Llama models. Looking at the safetensors files the actual model is Qwen3. I then compared this model's config files to those of Qwen/Qwen3-8B and made the following changes:

• added a chat_template field to the tokenizer_config file and copied over the value used in the Qwen/Qwen3-8B model
• changed base_model in the README.md yaml from meta-llama/Meta-Llama-3.1-8B to Qwen/Qwen3-8B-Base

Per the blog article: (6 Common Pitfalls in Sharing Models on Hugging Face)[https://friendli.ai/blog/common-pitfalls-in-sharing-models-on-hugging-face]: "Hugging Face’s chat interface relies on a chat template to format inputs. Without it, your chat model may not work as expected."

The other files and fields are consistent with the Qwen/Qwen3-8B model.

B

reasoning_effort has no effect because neither the model nor the serving stack maps that parameter to any behavior. To make it meaningful, the engine must interpret reasoning_effort and adjust routing, prompts, or decoding, and ideally the model must be trained or fine-tuned to take advantage of these different ‘effort levels’.

To make reasoning_effort actually matter, you need changes at both serving and model/training levels:

1. Serving-layer changes

• Parse the reasoning_effort field in the /chat/completions request.
• Map the value to concrete inference-time behaviors, for example:
  • Adjust max_tokens, temperature, and top_p to allow longer or more exploratory chains of thought when reasoning_effort is high.
  • Switch to a different prompt template that includes explicit chain-of-thought instructions or additional scratchpad tokens for higher effort.
  • Route to a different model variant (e.g., “reasoning” vs “normal”) if you host multiple checkpoints.
• Ensure the OpenAI-compatible server you’re using (Friendli Engine or vLLM-based gateway) actually forwards/consumes this parameter instead of dropping it.

2. Model/training changes

• Train or fine-tune the model so it can exploit the extra context or tokens provided for “higher effort”:
  • Instruction tuning with CoT data and different levels of reasoning depth.
  • Possibly multi‑expert or MoE heads where “high effort” can select a more compute-intensive branch.
• For truly semantic reasoning levels (like Anthropic’s “reasoning” modes), you might need specialized training regimes that align different effort levels with different internal behaviors, not just longer responses.