zhiminy/EvalEng huggingface dataset:
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: CharXiv: multiprocessing .map crashes with TypeError: cannot pickle 'SSLContext' object
Root cause: Resource Mishandling

Here’s a minimal GitHub issue draft you could file:

**Description**
Running the CharXiv task fails when `datasets.map` is called with `num_proc>1` in [`[lmms_eval/tasks/charxiv/utils.py](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/lmms_eval/tasks/charxiv/utils.py)`](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/lmms_eval/tasks/charxiv/utils.py).
Error:

```
TypeError: cannot pickle 'SSLContext' object
...
tenacity.RetryError: RetryError[<Future ... raised TypeError>]
```

With `num_proc=1`, the task works.

**Steps to Reproduce**

```python
from datasets import load_dataset
from lmms_eval.tasks.charxiv.utils import charxiv_descriptive_process_docs

ds = load_dataset("princeton-nlp/CharXiv", split="test")
charxiv_descriptive_process_docs(ds)  # crashes with num_proc>1
```

**Expected Behavior**
CharXiv preprocessing should succeed with multiprocessing enabled.

**Workaround**
Run with `num_proc=1` (or `HF_DATASETS_DISABLE_MULTIPROCESSING=1`).

Oh yeah, I forget to revise this for a fix. Thank you!

The issue persists in my trial, but setting `num_proc=None` resolves it.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Error while testing Qwen  on POPE
Root cause: Validation Gap

`
accelerate launch --num_machines 1 --num_processes=$NUMSGPU -m lmms_eval --model qwen_vl --model_args pretrained=Qwen-VL --tasks $TASK --batch_size 1 --log_samples --log_samples_suffix qwen-vl-chat_$TASK --output_path ./logs/ 
`

结果
![image](https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/71424268/d3fb08da-cd97-4f2d-8a72-b38617e4005c)



Have you check the log of Qwen? It might possibly because it output some answers that can not be parsed correctly
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Lacking of the dependency of OCRBench_V2
Root cause: Environment Incompatibility

Dear Maintainers,

I use the main branch and install the lmms_eval with `uv pip install -e ./lmms_eval[qwen,metrics]`

When I evaluate the qwen2_vl by using `ocrbench_v2`, it raised a lot of messages about cannot fine the modules like
- ipdb
- jieba
- distance
- editdistance
- apted
and so on.


After I installed the packages mentioned above, the evaluation process can start to launch successfully, but It may take around 7+ hours in my computer. Could you check the package dependency?

Thank you,
Best Regards,
Eli Chen
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: get result 0 on iconqa benchmark
Root cause: Validation Gap

running
```
accelerate launch --num_processes=2 --main_process_port 12227 -m lmms_eval \
        --model llava_hf \
        --model_args pretrained="model/llava_rationale_bs125lr2e-5" \
        --tasks iconqa_val\
        --batch_size 1 \
        --output_path "/lmms-eval/evaluation_results" \
        --log_samples
```
and get result 
|  Tasks   |Version|Filter|n-shot|Metric|   |Value|   |Stderr|
|----------|-------|------|-----:|------|---|----:|---|-----:|
|iconqa_val|Yaml   |none  |     0|anls  |↑  |    0|±  |     0|

I found that use the base model llava_hf also get 0 result, how to solve it?

You might need to check your evaluation log results for further information and see if it is because of the format or other issue in training
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [Feature Request] Evaluating quantized models
Root cause: Unimplemented Feature Gap

Hello,

As the title suggested, I'm wondering if lmms-eval has plan for enabling evaluation with quantized lmms, e.g. those with [AWQ](https://github.com/mit-han-lab/llm-awq).

**Why this is necessary?** The inference cost would definitely keep increasing as we have 1) more or larger benchmarks and 2) larger models (for example LLaVA-Next has 72B and 110B version already). Therefore quantized models are always of great interests to the broad community (including and beyond researchers). Being able to evaluate quantized models is therefore important.

I'm opening this issue mainly to see if this would be something lmms-eval will support. If so, I'm interested in contributing to this feature in some ways.

Thanks

Hi, I don't think it is supported in lmms-eval currently and definitely you are welcome to raise a PR for this feaure

If you wish to do large models inference, I strongly suggest using `llava_sglang`, in my test, that's even close or faster than quantization.

I see. Thank you both. Closing now.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Padding Side Option Missing in Qwen2_5_VL
Root cause: Configuration Error

I am trying to evaluate Qwen2_5_VL on MathVista to reproduce results closer to what Qwen team report. I am seeing padding_side being right by default and can give bad results. I did evaluate earlier directly from MathVista Github repo based code and saw similar warning which ended in bad results but passing left padding in tokenizer fixed and got closer to 64% reported for Qwen2_5_VL_7B_Instruct 

Upon investigating Qwen files in lmms-eval repo I can see qwen_vl version had default padding option of left but 
qwen2_5_vl doesn't have option only to update from command line. Before making any PR I wanted to check if I am missing some trick to fix this or we should change this behaviour via PR

<img width="974" height="411" alt="Image" src="https://github.com/user-attachments/assets/113012a9-c83b-469f-8f30-1e7809799281" />

Hello, I’ve encountered the same issue. Could you please share how you solved it?

same ask, otherwise https://github.com/QwenLM/Qwen3-VL/issues/759?timeline_page=1
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Any benchmark doesn't requires OPENAI key?
Root cause: Configuration Error

I have tried  : MMMU-val | Flickr30K | AI2D | Math-Vision | MathVista-mini

script is like 
```
accelerate launch --num_processes=${NGPU} --main_process_port=12345 -m lmms_eval \
    --model mmada \
    --model_args=pretrained=Gen-Verse/MMaDA-8B-MixCoT,gen_method=${GEN_METHOD},gen_length=${GEN_LENGTH},diff_step=${DIFF_STEP},block_length=${BLOCK_LENGTH},reasoning=True \
    --tasks mathvision_test_mmada \
    --batch_size 1 \
    --log_samples \
    --output_path "./results"
```
But all shows 
Error on attempt 1: Error code: 401 - {'error': {'message': 'Incorrect API key provided: YOUR_API_KEY. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}




mme, pope, qbench, seedbench, mmvet
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Request for Meta-Llama-3-8B-Instruct when evaluating LLaVA-Video-7B with conv_template=qwen_1_5
Root cause: Configuration Error

I am using the following script to evaluate **llava-video-7B**:

```bash
accelerate launch --num_processes=4 \
-m lmms_eval \
--model llava_vid \
--model_args pretrained=/data/gys/models/LLaVA-Video-7B-Qwen2,conv_template=qwen_1_5,max_frames_num=64,mm_spatial_pool_mode=average \
--tasks mme \
--batch_size 1 \
--log_samples \
--log_samples_suffix llava_vid \
--output_path ./logs/
```
But I encountered the following error:
```
'(MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /meta-llama/Meta-Llama-3-8B-Instruct/resolve/main/tokenizer_config.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f32141fb7f0>, 'Connection to huggingface.co timed out. (connect timeout=10)'))"), '(Request ID: 9d6f8820-9bec-4a91-ae0c-b189630c8e26)')' thrown while requesting HEAD https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct/resolve/main/tokenizer_config.json
```
However, I have already specified ` conv_template=qwen_1_5 `. Why is `Meta-Llama-3-8B-Instruct` still appearing here?

same issue

same issue

I solve this promble by add HF_ENDPOINT and HUGGINGFACE_TOKEN in my scripts, but it always stuck(or too slow), the log is：

`Rank 0:  Loaded LLaVA model: /node2/disk2/model_weight/llava-hf/llava-onevision-qwen2-7b-ov-hf
You are using a model of type llava_onevision to instantiate a model of type llava_qwen. This is not supported for all configurations of models and can yield errors.
You are using a model of type llava_onevision to instantiate a model of type llava_qwen. This is not supported for all configurations of models and can yield errors.
You are using a model of type llava_onevision to instantiate a model of type llava_qwen. This is not supported for all configurations of models and can yield errors.
Rank 0:  Overwriting config with {'mm_spatial_pool_stride': 2, 'mm_spatial_pool_mode': 'bilinear'}
You are using a model of type llava_onevision to instantiate a model of type llava_qwen. This is not supported for all configurations of models and can yield errors.
`
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Support for Qwen3-VL-30B-A3B

Dear authors, do you have any plan to support for Qwen3-VL-30B-A3B model?

as long as it's supported by vllm, it can be used in lmms-eval.

Thank you for your reply!
I wonder if there is any documentation about how to use lmms-eval for sglang/vllm deployed models?

you can check this PR's newly added doc:

https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/849

Thank you! That helps a lot.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: how to get three pope results [rad,pop,adv] with lmms-eval?
Root cause: Documentation Deficiency

I find the pope results reported in the paper are like this.
<img width="463" alt="image" src="https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/20677791/d10339a1-8400-47d4-b6b7-9523a4effe03">
how to use lmms-eval to get such three results? I only get results as below, there is no such metric.
<img width="82" alt="image" src="https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/20677791/85bd1f96-5b66-412e-87c5-6c1bea72fc4a">


Hi, thank you for the issue. I have created new tasks group to output the score separately. You can pull the main branch again and run with `--tasks pope_full`. Here are my results using llava_hf llava-1.5-7b.

<img width="825" alt="image" src="https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/92624596/6a4c923a-262d-4638-93c8-eff4d039a2f0">

Feel free to check whether my implementation is correct or not


I run command and find it work well. By now I think it is ok. thanks
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: What's the difference between `LMMS_EVAL_USE_CACHE` and the `--use_cache` argument?

Hi, I noticed there seem to be two different ways to enable caching in version 0.5, and I'm a bit confused about their differences.

The first method uses environment variables:


```bash
export LMMS_EVAL_USE_CACHE=True
export LMMS_EVAL_HOME="/path/to/cache_root"  # optional

python -m lmms_eval \
  --model async_openai \
  --model_args model_version=gpt-4o-2024-11-20,base_url=$OPENAI_API_BASE \
  --tasks mmmu_val \
  --batch_size 1 \
  --output_path ./logs/
```

The second method uses a command-line argument:
```bash
--use_cache "$CACHE_PATH"
```
According to the docstring:
> :param use_cache: str, optional  
> A path to a sqlite db file for caching model responses. `None` if not caching.

Could you please clarify the distinction between these two caching mechanisms? Specifically:
- What does `LMMS_EVAL_USE_CACHE` control, and how does it interact with `--use_cache`?
- Is `LMMS_EVAL_HOME` related to the path provided in `--use_cache`, or are they independent?
- Which one should be preferred for caching model responses?

Thanks for your help!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Does current code support parallel evaluation of model outputs?
Root cause: Unimplemented Feature Gap

Hi, thanks for sharing this great framework!

I noticed that in the current evaluation loop (in `lmms_eval/evaluator.py`), the processing of model outputs is done **sequentially per document**, as shown in this snippet:

```python
for doc_id, doc in doc_iterator:
    requests = instances_by_doc_id[doc_id]
    metrics = task.process_results(doc, [req.filtered_resps[filter_key] for req in requests])
    ...
```

While model inference itself may be batched, the **post-processing and evaluation** (especially `task.process_results`) — which in many cases involves calling an LLM to compute metrics (e.g., for open-ended generation, visual reasoning, etc.) — is performed **synchronously and one document at a time**.

This becomes a significant bottleneck when evaluation requires additional LLM calls (e.g., using GPT-4 as a judge), resulting in an extremely slow overall evaluation.

Does the current codebase support **parallel or multi-threaded evaluation** during the post-processing stage?  

Thank you so much for your attention and participation.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: How to Resume and Download Only Missing Samples Without Re-downloading the Entire Dataset
Root cause: Documentation Deficiency

When evaluating **VideoMME**, since the dataset is quite large and the network is unstable, some samples failed to download. How can I **resume and download only the missing samples** to avoid deleting the already downloaded data and re-downloading the entire dataset?

`Starting from v4.46, the `logits` model output will have the same type as the model (except at train time, where it will always be FP32) 
Model Responding:   1%|▍                                           | 6/676 [00:42<1:17:26,  6.94s/it]video path:/data/huggingface/videomme/data/YPaLGP_xr-w.mp4 does not exist, please check
video path:/data/huggingface/videomme/data/YPaLGP_xr-w.mp4 does not exist, please check`

You can download the video dataset first (retry or resume) from the hub. Then extract them to the cache_dir. Then in evaluation, the model will skip the extraction part
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: AttributeError: 'NoneType' object has no attribute 'get'
Root cause: Validation Gap

When I evaluated **LLaVA-Video** on the **mlvu_test** benchmark, the following error occurred:
> Traceback (most recent call last):
  File "/home/hunterj/gys/xxx/lmms-eval/lmms_eval/__main__.py", line 353, in cli_evaluate
    results, samples = cli_evaluate_single(args)
  File "/home/hunterj/gys/xxx/lmms-eval/lmms_eval/__main__.py", line 488, in cli_evaluate_single
    results = evaluator.simple_evaluate(
  File "/home/hunterj/gys/xxx/lmms-eval/lmms_eval/utils.py", line 536, in _wrapper
    return fn(*args, **kwargs)
  File "/home/hunterj/gys/xxx/lmms-eval/lmms_eval/evaluator.py", line 204, in simple_evaluate
    task_dict = get_task_dict(tasks, task_manager, task_type)
  File "/home/hunterj/gys/xxx/lmms-eval/lmms_eval/tasks/__init__.py", line 565, in get_task_dict
    task_name_from_string_dict = task_manager.load_task_or_group(
  File "/home/hunterj/gys/xxx/lmms-eval/lmms_eval/tasks/__init__.py", line 378, in load_task_or_group
    all_loaded_tasks = dict(collections.ChainMap(*map(load_fn, task_list)))
  File "/home/hunterj/gys/xxx/lmms-eval/lmms_eval/tasks/__init__.py", line 297, in _load_individual_task_or_group
    return _load_task(task_config, task=name_or_config)
  File "/home/hunterj/gys/xxx/lmms-eval/lmms_eval/tasks/__init__.py", line 267, in _load_task
    task_object = TaskObj(config=config, model_name=self.model_name)
  File "/home/hunterj/gys/xxx/lmms-eval/lmms_eval/api/task.py", line 755, in __init__
    test_text = self.doc_to_text(test_doc)
  File "/home/hunterj/gys/xxx/lmms-eval/lmms_eval/api/task.py", line 1321, in doc_to_text
    else doc_to_text(
  File "/home/hunterj/gys/xxx/lmms-eval/lmms_eval/tasks/mlvu/utils.py", line 66, in mlvu_doc_to_text
    pre_prompt = lmms_eval_specific_kwargs.get("pre_prompt", "")
AttributeError: 'NoneType' object has no attribute 'get'
2025-10-09 09:38:08 | ERROR    | __main__:cli_evaluate:375 - Error during evaluation: 'NoneType' object has no attribute 'get'. Please set `--verbosity=DEBUG` to get more information.


the same question, have u solved it?


maybe u can move the mlvu_dev.yml's "lmms_eval_specific_kwargs" to the mlvu_test.yml，i did this, and it works 

> maybe u can move the mlvu_dev.yml's "lmms_eval_specific_kwargs" to the mlvu_test.yml，i did this, and it works

thank you very much, it works!!!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Question on evaluation across multiple tasks
Root cause: Unimplemented Feature Gap

Hello. Thanks for your great effort on this comprehensive codebase!

I'm now trying to integrate our MM-UPD Bench into this codebase.
https://arxiv.org/abs/2403.20331

I have a question about whether we can evaluate across multiple tasks.
Specifically, we need to evaluate across multiple tasks (e.g., datasets on AAD ver. and Standard ver.) when calculating UPD accuracy.
If we were to replace the question with the use of MMBench, the question would be whether it is possible to evaluate with both MMBench_en and MMBench_cn datasets simultaneously.

For example, if there are questions with common indices in both MMBench_en and MMBench_cn, I want to consider them correct if both predictions for the same index are answered correctly.

If there is any other way to achieve the above, please let me know🙇

Thanks for your cooperation.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Error while runnig lava-onevision-qwen2-7b example
Root cause: External Dependency Breakage


```
export HF_HOME="~/.cache/huggingface"

accelerate launch --num_processes=8 --main_process_port 12399 -m lmms_eval \
    --model=llava_onevision \
    --model_args=pretrained=lmms-lab/llava-onevision-qwen2-7b-ov,conv_template=qwen_1_5,device_map=auto,model_name=llava_qwen \
    --tasks=longvideobench_val_v \
    --batch_size=1 \
    --verbosity=DEBUG
```

When I try to run the code above, I got the error above:

```
Failed to import llava_llama from llava.language_model.llava_llama. Error: cannot import name 'apply_chunking_to_forward' from 'transformers.modeling_utils' (~/anaconda3/envs/lmms-eval/lib/python3.12/site-packages/transformers/modeling_utils.py)
Failed to import llava_qwen from llava.language_model.llava_qwen. Error: cannot import name 'apply_chunking_to_forward' from 'transformers.modeling_utils' (~/anaconda3/envs/lmms-eval/lib/python3.12/site-packages/transformers/modeling_utils.py)
Failed to import llava_mistral from llava.language_model.llava_mistral. Error: cannot import name 'apply_chunking_to_forward' from 'transformers.modeling_utils' (~/anaconda3/envs/lmms-eval/lib/python3.12/site-packages/transformers/modeling_utils.py)
Failed to import llava_mixtral from llava.language_model.llava_mixtral. Error: cannot import name 'apply_chunking_to_forward' from 'transformers.modeling_utils' (~/anaconda3/envs/lmms-eval/lib/python3.12/site-packages/transformers/modeling_utils.py)
```


Can try to use llava_hf instead of llava_onevision as you may need to install it from a specific commit from LLaVA-NeXT to make it actually works

I fix this issue by replace the code in [LLaVA-NeXT](https://github.com/LLaVA-VL/LLaVA-NeXT/blob/e9835311c6f515a13702eb7a7750fcd936f65ed8/llava/model/multimodal_resampler/qformer.py#L39-L44):
remove the code in `llava/model/multimodal_resampler/qformer.py`
```
    apply_chunking_to_forward,
    find_pruneable_heads_and_indices,
    prune_linear_layer,
```
and add new code to import them by 
```
from transformers.pytorch_utils import apply_chunking_to_forward, find_pruneable_heads_and_indices, prune_linear_layer # change modeling_utils to pytorch_utils
```
Thanks to the discussion in https://github.com/modelscope/evalscope/issues/864


@morninghut I replaced this code and then another exception came out:
```python
Failed to import llava_mixtral from llava.language_model.llava_mixtral. Error: cannot import name 'PreTrainedModel' from 'transformers.pytorch_utils'
```
I use transformers-4.57.3, torch2.8+cu128
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Bug: TypeError: unhashable type: 'dict'
Root cause: Validation Gap

https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/861

When I run it, I also encounter an "unhashable type: dict" error when calling cached_get_processor.

and `self` is 

```python
[{'fps': 29.97002997002997, 'frames_indices': [0, 72, 144, 215, 287, 359, 431, 503, 574, 646, 718, 790, 862, 933, 1005, 1077, 1149, 1221, 1293, 1364, 1436, 1508, 1580, 1652, 1723, 1795, 1867, 1939, 2011, 2082, 2154, 2226], 'total_num_frames': 2227, 'video_backend': 'decord'}]

```


Hi, I think the error has been fixed

Thanks for the timely reply and fix.

hi, @kcz358 
For models prior to qwen3-vl, what is the difference between `chat` and `generate`? Especially in the video scenario, I noticed that the annotation states that the Qwen3 series can process video correctly. Is there a difference in speed between the two for videos? Is there a more detailed explanation for `chat` and `generate`? 

So because the input ids for video inference in qwen3-vl is a bit difference then the others and time information is being inserted explicitly, we need to acquire video metadata and prepare the prompt before the actual inference. Basically chat use a openai like chat messages and the preprocess in controlled in the vllm. For generate, we use transformers processor to prepare the prompt first before inference.

Thanks for your explanation.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: RuntimeError: cu_seqlens_q must be on CUDA
Root cause: External Dependency Breakage

when I run `examples/models/vllm_qwen2vl.sh`, I meet the error above. and my `vllm==0.11.0` and `flash_attn==2.8.2`.

details:

```
File "/mnt/bn/lhx-usw/conda/envs/vllm/lib/python3.12/site-packages/vllm/model_executor/models/qwen2_vl.py", line 398, in forward
    output = flash_attn_varlen_func(q,
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/bn/lhx-usw/conda/envs/vllm/lib/python3.12/site-packages/flash_attn/flash_attn_interface.py", line 1443, in flash_attn_varlen_func
    return FlashAttnVarlenFunc.apply(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/bn/lhx-usw/conda/envs/vllm/lib/python3.12/site-packages/torch/autograd/function.py", line 576, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/bn/lhx-usw/conda/envs/vllm/lib/python3.12/site-packages/flash_attn/flash_attn_interface.py", line 925, in forward
    out_padded, softmax_lse, S_dmask, rng_state = _wrapped_flash_attn_varlen_forward(
                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/bn/lhx-usw/conda/envs/vllm/lib/python3.12/site-packages/torch/_ops.py", line 1243, in __call__
    return self._op(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/bn/lhx-usw/conda/envs/vllm/lib/python3.12/site-packages/torch/_library/custom_ops.py", line 344, in backend_impl
    result = self._backend_fns[device_type](*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/bn/lhx-usw/conda/envs/vllm/lib/python3.12/site-packages/torch/_compile.py", line 53, in inner
    return disable_fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/bn/lhx-usw/conda/envs/vllm/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 929, in _fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/mnt/bn/lhx-usw/conda/envs/vllm/lib/python3.12/site-packages/torch/_library/custom_ops.py", line 377, in wrapped_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/mnt/bn/lhx-usw/conda/envs/vllm/lib/python3.12/site-packages/flash_attn/flash_attn_interface.py", line 165, in _flash_attn_varlen_forward
    out, softmax_lse, S_dmask, rng_state = flash_attn_gpu.varlen_fwd(
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: cu_seqlens_q must be on CUDA
```


Solved by upgrading `vllm` to 0.11.1rc2