Not thinking even when `chat_template_kwargs` is set?
chat_template_kwargs was set to:
{"enable_thinking":true}
startup script:
export CUTE_DSL_ARCH=sm_121a # arch string for FP4 kernel JIT
export PATH=/usr/local/cuda/bin:$PATH # nvcc for JIT
export MAX_JOBS=4 # cap JIT fan-out; see warning below
vllm serve /data/models/Laguna-S-2.1-NVFP4 \
--speculative-config '{"model":"/data/models/Laguna-S-2.1-DFlash-NVFP4","num_speculative_tokens":5}' \
--enable-auto-tool-choice \
--tool-call-parser poolside_v1 \
--reasoning-parser poolside_v1 \
--override-generation-config '{"temperature":0.7,"top_p":0.95}' \
--max-num-seqs 4 \
--max-model-len 262144 \
--gpu-memory-utilization 0.93
The model only thinks when its needed.
Request some complex task and it will think.
This is not Qwen which always thinks ;)
The model only thinks when its needed.
Request some complex task and it will think.This is not Qwen which always thinks ;)
Thanks for the tip, however I found that vllm seemed not picking up </think> instead it just spit out in its output.
Is that chat template issue then or vLLM?
To make it think like Qwen3.6, fix the chat template.
https://huggingface.co/Qwen/Qwen3.6-27B/blob/main/chat_template.jinja#L150 - use <think>\n.
I am having the same issue where every output starts with inside of OpenWebUI 0.10.2.
My attempt at a SparkRun recipe using a locally built version of spark-vllm-docker:
recipe_version: "2"
model: poolside/Laguna-S-2.1-NVFP4
runtime: vllm
container: vllm-node # attempting to use a local image instead of: eugr/spark-vllm:latest
max_nodes: 1
metadata:
description: "Laguna S 2.1 NVFP4 (117.6B MoE, 8.5B active) + DFlash speculative decoding — solo DGX Spark"
category: agent
model_params: 117.6B
quantization: nvfp4
quant_bits: 4
kv_dtype: fp8
num_layers: 48
env:
CUTE_DSL_ARCH: sm_121a
MAX_JOBS: "4"
defaults:
port: 8000
host: 0.0.0.0
tensor_parallel: 1
gpu_memory_utilization: 0.85
max_model_len: 262144
max_num_seqs: 32
served_model_name: laguna-s-2.1
command: |
vllm serve
{model}
--served-model-name {served_model_name}
--host {host} --port {port}
-tp {tensor_parallel}
--gpu-memory-utilization {gpu_memory_utilization}
--max-model-len {max_model_len}
--max-num-seqs {max_num_seqs}
--speculative-config '{"model":"poolside/Laguna-S-2.1-DFlash-NVFP4","num_speculative_tokens":15,"method":"dflash"}'
--enable-auto-tool-choice
--tool-call-parser poolside_v1
--reasoning-parser poolside_v1
--override-generation-config '{"temperature":0.7,"top_p":0.95}'