Has anyone got this same CUDA error: an illegal memory access was encountered while using this model in vllm:v0.25.0?
RTX 6000d SM120
settings:
kv_cache: fp8
backend: flashinfer
max_total_tokens: 262144
enable-prefix-caching
enable-chunk-prefill
trust-remote-code
mtp: num_spec:3
while some conversation is decoding, it suddently causes these error：
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line
3664, in synchronize_input_prep
yield
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line
4167, in execute_model
self._build_attention_metadata(
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line
2442, in _build_attention_metadata
build_attn_group_metadata(kv_cache_gid, attn_gid, cm)
File "/usr/local/lib/python3.12/dist-packages/vl1m/v1/worker/gpu_model_runner.py", line
2380, in _build_attn_group_metadata
attn_metadata_i = builder.build(
line 984, in build
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/flashinfer.py",
seq lens cpu = common attn metadata.seq lens cpu if needs seq lens cpu else None
return arg(*args, **kwargs)
File "/usr/local/lib/python3.12/dist-packages/typing extensions.py", line 3004, in wrappe seq_lens_cpu
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backend.py", line 437, in self._seq_lens_cpu = self.seq_lens.to("cpu")
torch.AcceleratorError: CUDA error: an illegal memory access was encountered
etc....
RTX 5090 here. Cant say I did as 0.24.0 has been the stable version so far. Perhaps that's your best bet unless you're tinkering with 0.25.0. gl
RTX 5090 here. Cant say I did as 0.24.0 has been the stable version so far. Perhaps that's your best bet unless you're tinkering with 0.25.0. gl
thanks for the info!Im using the FP8 model right now in 0.25.0 and havent run into that issue anymore. Though I still suspect some bad requests might be triggering this kernel crash.