Hugging Face's logo

moonshotai
/
Kimi-Linear-48B-A3B-Instruct

Text Generation
Transformers
Safetensors
kimi_linear
conversational
custom_code
Model card Files
xet
 Community
25

Kimi-Linear-48B-A3B-Instruct vLLM Issues

#24
by pangyh2001 - opened
Discussion
pangyh2001
 

## Summary

We are testing `Kimi-Linear-48B-A3B-Instruct` with vLLM `0.11.2`.

We observe three major issues:

1. Short-input requests can degenerate into repeated `!` tokens and timeout.
2. Long-context requests under high concurrency mostly timeout.
3. vLLM repeatedly logs xgrammar FSM errors during the failure period.

## Environment

- Model: `Kimi-Linear-48B-A3B-Instruct`
- Backend: vLLM `0.11.2`
- GPU: single NVIDIA H200
- Serving mode: OpenAI-compatible `/v1/chat/completions`
- Context length: `32768`
- dtype: `bfloat16`

## Launch Command

```bash
CUDA_VISIBLE_DEVICES=6 \
PYTHONPATH=/tmp/vllm_triton_allocator \
/tmp/vllm-0.11.2/bin/python -u -m vllm.entrypoints.cli.main serve \
  /upfs/models/Kimi-Linear-48B-A3B-Instruct/Kimi-Linear-48B-A3B-Instruct \
  --host 127.0.0.1 \
  --port 8003 \
  --served-model-name Kimi-Linear-48B-A3B-Instruct \
  --tensor-parallel-size 1 \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --trust-remote-code \
  --enforce-eager

Request Parameters

We call /v1/chat/completions with:

{
  "temperature": 0.1,
  "top_p": 0.4,
  "max_tokens": 8192,
  "stream": true,
  "response_format": {
    "type": "json_object"
  }
}

The prompt asks the model to return exactly one valid JSON object.

Issue 1: Short Input Degenerates Into ! Loop

Test Setup

  • Input length: about 256 words
  • Requests: 32
  • Concurrency: 8
  • max_tokens: 8192
  • Timeout: 300 seconds

Result 

completed: 20
total_timeout: 12
json_ok: 20
parse_failed: 12

Observed Failure Pattern

The failed requests did not fail because of a low output-token limit. They ran until the 300-second timeout and produced mostly repeated exclamation marks:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
...

Questions

  • Is this a known issue with Kimi-Linear-48B-A3B-Instruct under vLLM?
  • Could this be caused by tokenizer behavior, logits processing, or the model’s custom remote-code implementation?
  • Are there recommended sampling parameters, stop tokens, or serving flags to prevent this repeated ! loop?
  • Is response_format={"type":"json_object"} recommended for this model?

Issue 2: Long Context High-Concurrency Requests Mostly Timeout

Test Setup

We used a real long-context financial analysis prompt.

  • Requests: 192
  • Concurrency: 192
  • max_tokens: 8192
  • Timeout: 300 seconds
  • Same prompt repeated 192 times

Result 

total requests: 192
completed with finish_reason=stop: 19
total_timeout: 173
json_ok: 29
parse_failed: 163
latency_min: 181.8s
latency_p50: 312.6s
latency_p95: 331.1s
latency_max: 334.8s

Many timeout requests returned zero characters. Some requests produced partial JSON and then degenerated into repeated !.

Example:

{"role": null, "evidence_assessment": {"main_positive...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

vLLM Runtime Metrics

During the test, vLLM showed heavy queueing:

Running: 26 reqs
Waiting: 147 reqs
GPU KV cache usage: 96.9%

Questions

  • Is concurrency 192 unrealistic for this model on a single H200 with long-context prompts?
  • What maximum concurrency would you recommend for this model on one H200?
  • Does this model require any special vLLM flags for stable high-concurrency serving?
  • Is the timeout mainly due to queueing/KV cache pressure, model generation degeneration, or both?

Issue 3: xgrammar FSM Errors

During the long-context high-concurrency test, vLLM repeatedly logged:

backend_xgrammar.py:158 Failed to advance FSM for request ... for tokens 0. Please file an issue.

This appeared many times around the timeout period.

Questions For vLLM Developers

  • Does response_format={"type":"json_object"} use xgrammar in vLLM 0.11.2?
  • What does Failed to advance FSM ... for tokens 0 usually mean?
  • Could xgrammar be interacting with token generation in a way that causes the repeated ! loop?
  • Is there a recommended workaround?
    • Disable structured decoding?
    • Use a different guided decoding backend?
    • Avoid response_format and rely on prompt-only JSON formatting?
    • Change tokenizer or guided decoding settings?

Low-Concurrency Observation

With the same long-context prompt but only 4 repeated requests at concurrency 2, all requests completed successfully:

requests: 4
completed: 4
finish_reason=stop: 4
json_ok: 4
latency range: about 51.8s to 156.5s

So vLLM 0.11.2 appears better than the previous deployment at low concurrency, but high concurrency remains unstable.

Main Questions

We would like to know whether this is more likely caused by:

  1. The model checkpoint itself.
  2. The custom Kimi model implementation.
  3. Tokenizer behavior.
  4. vLLM structured decoding / xgrammar.
  5. Excessive concurrency on a single H200.
  6. An interaction between these factors.

What We Need

We need the model to reliably produce one valid JSON object for long-context financial role-agent prompts.

Could you please provide guidance on:

  1. The recommended vLLM version for Kimi-Linear-48B-A3B-Instruct.
  2. The recommended launch command and serving flags.
  3. Whether response_format={"type":"json_object"} is supported/recommended.
  4. The recommended maximum concurrency on one H200.
  5. How to avoid repeated ! generation loops.
  6. How to handle or avoid the xgrammar FSM errors.

Sign up or log in to comment