zhiminy/EvalEng huggingface dataset:
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: llava-interleave-bench
Root cause: Documentation Deficiency

为什么我用llava-interleave-7b模型跑interleave-bench的时候性能对不上，有没有好心人帮忙一下，或者给一下lmms-eval llava-interleave-7b运行的模板
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Bug using local dataset
Root cause: Algorithmic Error

In the yaml file, when video is set to True in dataset_kwargs, lmms-eval.api.task.ConfigurableTask.download will download the dataset from hugging face before judging load_from_disk, which makes the use of the local dataset invalid.

Glad to know it, would you kindly provide a fix here? Thanks!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Can't debug with only one process on one GPU.
Root cause: Configuration Error

I used the following script:

TASK=refcoco  
MODEL_NAME=qwen2_5_vl  
CKPT_PATH=./lmms-eval/Qwen2.5-VL-3B-Instruct

echo $TASK  
TASK_SUFFIX="${TASK//,/_}"  
echo $TASK_SUFFIX

MASTER_PORT=$((18000 + $RANDOM % 100))  
NUM_GPUS=1  
NUM_CPU=16

accelerate launch \  
    --num_processes ${NUM_GPUS} \  
    --main_process_port ${MASTER_PORT} \  
    -m lmms_eval \  
    --model ${MODEL_NAME} \  
    --model_args pretrained=$CKPT_PATH \  
    --tasks $TASK \  
    --batch_size 1 \  
    --log_samples \  
    --log_samples_suffix $TASK_SUFFIX \  
    --output_path ./logs/${JOB_NAME}_${MODEL_NAME}

However, it still uses 8 processes on a single GPU, which causes a CUDA out-of-memory error. How can I modify the configuration to avoid this?
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: token: False  does not work in gqa.yaml
Root cause: Configuration Error

 I set token: False in gqa.yaml.

When running 
`CUDA_VISIBLE_DEVICES=3 accelerate launch --num_processes=1 -m lmms_eval --model llava   --model_args pretrained="/home/xxx/huggingface/liuhaotian/llava-v1.5-7b,device_map=auto,use_flash_attention_2=False"   --tasks gqa  --batch_size 1 --log_samples --log_samples_suffix reproduce --output_path ./logs/`

I get an error:
```
huggingface_hub.utils._headers.LocalTokenNotFoundError: Token is required (`token=True`), but no token found. You need to provide a token or be logged in to Hugging Face with `huggingface-cli login` or `huggingface_hub.login`. See https://huggingface.co/settings/tokens.
04-24 20:50:35 [lmms-eval/lmms_eval/__main__.py:213] ERROR Error during evaluation: Token is required (`token=True`), but no token found. You need to provide a token or be logged in to Hugging Face with `huggingface-cli login` or `huggingface_hub.login`. See https://huggingface.co/settings/tokens.
```

hi you need to set it to `True` 

And you may need to use `huggingface-cli login` to connect your local environment to huggingface, it needs to download the `gqa` dataset. 

Or another way is to manually download the `gqa` dataset (in repo form, in a folder), and change the `lmms-lab/gqa` to your local folder path.

Ok, I chose to manually download the `gqa `dataset, and load it from my huggingface cache dir.  
I also set token=False to avoid connecting the huggingface because there is no need to download the `gqa `dataset.
I find that it still throws error: `huggingface_hub.utils._headers.LocalTokenNotFoundError: Token is required (`token=True`)`.
It shows error info
 `File "/home/yutingbai/test/lmms-eval/lmms_eval/tasks/gqa/utils.py", line 11, in gqa_doc_to_visual
    GQA_RAW_IMAGE_DATASET = load_dataset("lmms-lab/GQA", "testdev_balanced_images", split="testdev", token=True)`
So, I just set `token=False` and it is ok.

Besides, I test a model on `gqa `dataset and get the following result:
<img width="327" alt="image" src="https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/20677791/67df7904-93ed-46ac-aa67-145d279df4ae">
So, does it mean that the accuracy is **`1.7729%`**?

For llava-1.5-7B (lmms-eval), the value and Stderr in [table](https://docs.google.com/spreadsheets/d/1AvaEmuG4csSmXaHjgu4ei1KBMmNNW8wflOD_kkTDdv8/edit#gid=1250451976) is 0.6197328669 and 0.0043, respectively. So, the log should be like the following?
|Tasks|Version|Filter|n-shot|  Metric   |Value |   |Stderr|
|-----|-------|------|-----:|-----------|-----:|---|-----:|
|gqa  |Yaml   |none  |     0|exact_match|61.97328669|±  |0.0043|

I want to figure it out.







It means 1.7729% accuracy. You can check the log to see how your model response. It is worth noting that exact match actually means for `exact match`. Any extra characters such as string or new line will causing the mark to be 0. We will add in some filter class in the next release for this.

Ok
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Qwen2.5_Omni Models fails with ImportError: cannot import name 'Qwen2_5OmniModel' from 'transformers'
Root cause: External Dependency Breakage

Python Version:

Python 3.11.11

System Info:

Linux 755dae8474ae 6.1.85+ Thu Jun 27 21:05:47 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Command to run:
```
python -m lmms_eval \
    --model qwen2_5_omni \
    --model_args max_pixels=1000000 \
    --tasks mmbench_en_dev,mmbench_cn_dev \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix vllm \
    --output_path ./logs \
    --limit=640 \
    --verbosity=DEBUG
```

Error Message:

```
2025-04-08 03:04:36.783 | ERROR    | lmms_eval.models:get_model:90 - Failed to import Qwen2_5_Omni from qwen2_5_omni: cannot import name 'Qwen2_5OmniModel' from 'transformers' (/usr/local/lib/python3.11/dist-packages/transformers/__init__.py)
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/local/lib/python3.11/dist-packages/lmms_eval/__main__.py", line 533, in <module>
    cli_evaluate()
  File "/usr/local/lib/python3.11/dist-packages/lmms_eval/__main__.py", line 346, in cli_evaluate
    raise e
  File "/usr/local/lib/python3.11/dist-packages/lmms_eval/__main__.py", line 330, in cli_evaluate
    results, samples = cli_evaluate_single(args)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lmms_eval/__main__.py", line 471, in cli_evaluate_single
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lmms_eval/utils.py", line 533, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lmms_eval/evaluator.py", line 176, in simple_evaluate
    lm = lmms_eval.models.get_model(model).create_from_arg_string(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lmms_eval/models/__init__.py", line 87, in get_model
    module = __import__(model_module, fromlist=[model_class])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/lmms_eval/models/qwen2_5_omni.py", line 16, in <module>
    from transformers import Qwen2_5OmniModel, Qwen2_5OmniProcessor
ImportError: cannot import name 'Qwen2_5OmniModel' from 'transformers' (/usr/local/lib/python3.11/dist-packages/transformers/__init__.py)

```

The merging fork of transformers is [here](https://github.com/huggingface/transformers/pull/36752/files#diff-7723156f6b075b1bf1525f769d90a7cc0b5f233becdcbe28707aaa753960d897R733), indeed it does not expose Qwen2_5OmniModel.





I use following cmd to install the pull request and solved the issue:

```
pip install git+https://github.com/huggingface/transformers@3a1ead0aabed473eafe527915eea8c197d424356
```
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: about llava-interleave model inconsistent performance
Root cause: Configuration Error

Why is there a significant performance difference when I run tasks including but not limited to llava-interleave-benchmark using the llava-interleave-7b model? When setting up the model, should it be set as llava, llava-ov, or something else? Or are there other configurations that need to be set?
this is my setting
    python3 -m accelerate.commands.launch --num_processes=1 \
    -m lmms_eval \
    --model llava \
    --model_args pretrained="llava-next-interleave-qwen-7b",conv_template=qwen_1_5,device_map=auto,model_name=llava_qwen \
    --tasks llava_interleave_bench \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix llava_interleave_bench \
    --output_path LLaVA-NeXT-main/run/logs

I have the same problem, how can I solve it? Thank you very much！！！


Same problem!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: openai_compatible model not complatible with chat/completions endpoint due to max_output_tokens
Root cause: Configuration Error

After this PR https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/614 lmms-eval is not compatible with `chat/completions` endpoint.
There is sent `max_output_tokens` parameter which is not compatible. Based on specification, it is part of v1/responses endpoint only.
Could it be configurable based on the endpoint?
Right now, it is not possible to run the test with vLLM and other servings compatible with `chat/completions`.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: VILA error: 'get_model_name_from_path' is not defined
Root cause: Validation Gap

I have tried installing VILA then run the command
```
git clone https://github.com/NVlabs/VILA.git
cd VILA
bash ./environment_setup.sh vila
conda activate vila

# # also tried 
# pip install -e .
```

Command:
```
python3 -m accelerate.commands.launch \
--num_processes=1 \
--mixed_precision=fp16 \
--main_process_port=29501 \
-m lmms_eval \
--model vila \
--model_args "pretrained=Efficient-Large-Model/VILA1.5-40b" \
--tasks mme \
--batch_size 1 \
--limit 10 \
--log_samples \
--output_path logs \
--verbosity=DEBUG
```

Error:

```
.../lmms-eval/lmms_eval/models/vila.py", line 85, in __init__
[rank0]:     self.model_name = get_model_name_from_path(pretrained)
[rank0]: NameError: name 'get_model_name_from_path' is not defined
```

Hi! I have the same issues,did you solved it?

Error: NameError: name 'get_model_name_from_path' is not defined
Try this:


Create and run this bash script to install VILA (and llava). DO NOT install llava from the official llava repo.
```bash
#!/bin/bash
set -e  # Exit on error

# Clone the repository if it doesn't exist
if [ ! -d "VILA" ]; then
    git clone https://github.com/NVlabs/VILA.git
else
    echo "VILA directory already exists, skipping clone"
fi

cd VILA

# Path to the target file
target_file="llava/constants.py"

# Append the content only if the file exists and doesn't already have the content
if [ -f "$target_file" ] && ! grep -q "DEFAULT_IM_END_TOKEN" "$target_file"; then
    cat << 'EOF' >> "$target_file"
# --------
# NOTE: This is from the original LLaVA codebase:
# https://github.com/haotian-liu/LLaVA/blob/main/llava/constants.py
# --------
IGNORE_INDEX = -100
IMAGE_TOKEN_INDEX = -200
DEFAULT_IMAGE_TOKEN = "<image>"
DEFAULT_IMAGE_PATCH_TOKEN = "<im_patch>"
DEFAULT_IM_START_TOKEN = "<im_start>"
DEFAULT_IM_END_TOKEN = "<im_end>"
# --------
EOF
    echo "Added constants to $target_file"
else
    echo "File already contains the constants or does not exist"
fi

pip install .

```
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: API calls are Sync
Root cause: Architectural Constraint

API calls are run synchronously, instead of starting as many as possible at once while staying under API provider rate limits. This makes many evaluations, eg gemini-2.0-flash, take >100x longer than they need to.

We are working on put the judge in a unified interface and it provides the async API call for LLM judge.

https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/681
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Unreasonably high scores in llava_in_the_wild benchmark
Root cause: Validation Gap



![image](https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/72066971/af580d7c-60fb-4e0f-a818-988a1734ff2b)

I tested Llava-1.5-hf under the llava_in_the_wild benchmark and got unreasonably high scores (96.1) compared to that reported in the paper. Any idea of the cause of it?


Somehow we notice that you got 100 for both detail and complex. This is likely that you query gpt4 and get an error so we auto fill it with `[-1, -1]`. Otherwise it is very hard to explain why the model only get 74.9 in `conv` if it is really so good.

Also to be noticed that it is almost impossible to replicate the exact score from the report now as gpt-0314 is deprecated and different version will have different taste. If you want to make sure your reponse is almost the same with the origin one instead of the hf one, we recommend you to use the origin one since they have some slightly different implementation detail.

Thanks for the quick reply! I believe it's because of the error in calling openai API.

Hi, I have failed to call openai API correctly when testing llava_in_the_wild. Even though I have specified the token for OPENAI_API_KEY, it still reports Cannot connect to proxy. May I ask if I am missing something in the code?
![Image](https://github.com/user-attachments/assets/3077b05d-b2af-4625-9bc9-6cd837251137)
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [Bug] Qwen2.5VL multi-gpu bug:  Expected all tensors to be on the same device, but found at least two devices, cuda:4 and cpu
Root cause: Resource Mishandling

Hi, when I evaluate Qwen2.5VL, I met a bug as follows: 

![Image](https://github.com/user-attachments/assets/734b31ec-5b3d-4f18-bfac-52c938764c68) Do you know how to fix it?

I try image benchmark is fine, and Qwen2VL is also work, but on video benchmarks, it happens.

dear author,
when I evaluate on single GPU, I have similar errors, could you tell me how to solve this ?
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Big results gap for qwenvl2.5-7b
Root cause: Configuration Error

here is the results：
1 mme rleave_visuals=True), gen_kwargs: (), limit: None, num_fewshot: None, batch_size: 1
|Tasks|Version|Filter|n-shot|       Metric       |   | Value  |   |Stderr|
|-----|-------|------|-----:|--------------------|---|-------:|---|------|
|mme  |Yaml   |none  |     0|mme_cognition_score |↑  |261.7857|±  |   N/A|
|mme  |Yaml   |none  |     0|mme_perception_score|↑  |598.3680|±  |   N/A|
total less than 900，half the reported results
2 mmu   36  3point less than the reported results

| Tasks |Version|Filter|n-shot|        Metric         |   |Value |   |Stderr|
|-------|-------|------|-----:|-----------------------|---|-----:|---|-----:|
|chartqa|Yaml   |none  |     0|relaxed_augmented_split|↑  |0.0808|±  |0.0077|
|chartqa|Yaml   |none  |     0|relaxed_human_split    |↑  |0.0928|±  |0.0082|
|chartqa|Yaml   |none  |     0|relaxed_overall        |↑  |0.0868|±  |0.0056|       chartqa

|    Tasks     |Version|Filter|n-shot|  Metric  |   |Value |   |Stderr|
|--------------|-------|------|-----:|----------|---|------|---|------|
|docvqa        |    N/A|      |      |          |   |      |   |      |
| - docvqa_test|Yaml   |none  |     0|submission|↑  |N/A   |±  |   N/A|
| - docvqa_val |Yaml   |none  |     0|anls      |↑  |0.0393|±  |0.0024|

I did find out some results gap for mmmu for qwenvl 2.5 7b when I developing, but I haven't found such a huge gap. May I ask what commands you are using?

> I did find out some results gap for mmmu for qwenvl 2.5 7b when I developing, but I haven't found such a huge gap. May I ask what commands you are using?

like this:accelerate launch --num_processes=8 --main_process_port=12346 -m lmms_eval \
    --model qwen2_5_vl \
    --model_args=pretrained=Qwen/Qwen2.5-VL-7B-Instruct,max_pixels=12845056,use_flash_attention_2=True,interleave_visuals=True \
    --tasks mmmu_pro \
    --batch_size 1.    for mmmu, i did found 2 points gap, however, for datasets like chatrqa, there is a huge gap. 

@Blissy-32 For charvqa, you need to set `interleave_visuals=False`. Here is my commands and corresponding result of charvqa:
```
accelerate launch --num_processes=2 --main_process_port=12346 --gpu_ids 2,3 -m lmms_eval \
    --model qwen2_vl \
    --model_args=pretrained=Qwen/Qwen2-VL-2B-Instruct,max_pixels=12845056,use_flash_attention_2=True,interleave_visuals=False \
    --gen_kwargs=max_new_tokens=100 \
    --tasks chartqa \
    --batch_size 1 \
    --output_path="./tmp"
```
| Tasks |Version|Filter|n-shot|        Metric         |   |Value |   |Stderr|
|-------|-------|------|-----:|-----------------------|---|-----:|---|-----:|
|chartqa|Yaml   |none  |     0|relaxed_augmented_split|↑  |0.8944|±  |0.0087|
|chartqa|Yaml   |none  |     0|relaxed_human_split    |↑  |0.5520|±  |0.0141|
|chartqa|Yaml   |none  |     0|relaxed_overall        |↑  |0.7232|±  |0.0090|
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: How can I become the contributor？

I have deployed the kimi-vl on this framework. I want to contribute it.

May I ask how many frames you used? When I try 128 frames, the video memory will be depleted. 

Moreover, when I test the model on the temporal grounding task, it completely ignores the instructions and simply outputs dense captions of the video. Do you have similar issues in your evaluation?

Hi, you are welcome to contribute. Just clone the repo and raise the PR and we will review your code.

> Hi, you are welcome to contribute. Just clone the repo and raise the PR and we will review your code.

Thanks a lot
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Bug: The images blending within the same batch leads to the performance degradation of Qwen2.5_VL when the batch size increases
Root cause: Algorithmic Error

# Environment
```
lmms-eval=0.3.3
transformers=4.51.3
```

# Context

I used the following commands to test the performance of the Qwen2.5VL-3B model on the `vqav2_val_lite` task, with batch sizes of 1, 2, and 4 respectively.

```shell
task="vqav2_val_lite"

CUDA_VISIBLE_DEVICES=0 \
accelerate launch --num_processes=1 --main_process_port=12345 -m lmms_eval \
    --model qwen2_5_vl \
    --model_args=pretrained=./Qwen/Qwen2.5-VL-3B-Instruct,use_flash_attention_2=True \
    --tasks $task \
    --batch_size 1 \
    --output_path ./result/Qwen2.5-VL-3B-Instruct/lmms_eval_$task \
    --log_samples

CUDA_VISIBLE_DEVICES=0 \
accelerate launch --num_processes=1 --main_process_port=12345 -m lmms_eval \
    --model qwen2_5_vl \
    --model_args=pretrained=./Qwen/Qwen2.5-VL-3B-Instruct,use_flash_attention_2=True \
    --tasks $task \
    --batch_size 2 \
    --output_path ./result/Qwen2.5-VL-3B-Instruct/lmms_eval_${task}_bs2 \
    --log_samples


CUDA_VISIBLE_DEVICES=0 \
accelerate launch --num_processes=1 --main_process_port=12345 -m lmms_eval \
    --model qwen2_5_vl \
    --model_args=pretrained=./Qwen/Qwen2.5-VL-3B-Instruct,use_flash_attention_2=True \
    --tasks $task \
    --batch_size 4 \
    --output_path ./result/Qwen2.5-VL-3B-Instruct/lmms_eval_${task}_bs4 \
    --log_samples
```

The results are as follows:

|Batch size| Performance |
|:---:|:---:|
| 1 | 0.7478 |
| 2 | 0.7234 |
| 4 | 0.6810 |



# Observation

Based on my current observations, the issue might be located [here](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/514082ea326d903f7dfed9ec04bdbc70b7018015/lmms_eval/models/qwen2_5_vl.py#L192-L278)

Simply put, the situation in the code seems to be like this:

```python
# `contexts` is a batch of context messages.
# `visual_list` is a batch of image list.

visual_list = self.flatten(visual_list)  # Why do this?

# ...

batched_messages = []
for context in contexts:
    # ...
    processed_visuals = []
    for visual in visual_list:
        # ...
        processed_visuals.append(...)  # This seems to cause every message within the batch to include all images from that entire batch.
```



I hope to get clarification from the contributors or the community.

If you print out the results of a QWen 2.5 3B batch inference, you'll see that all results are weird except one. That's because if it's weird padding. And if you add `padding_side="left"` parameter when initializing the processor and tokenizer, batch inference becomes normal.

That might be the cause of your problem. Not sure though.

You can test it by modifying here:
https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/4b4049da17af9140442067aad5a1e2ddc1715c42/lmms_eval/models/simple/qwen2_5_vl.py#L104
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: incomplete tasks problem
Root cause: Documentation Deficiency

After installing the project, I evaluated its performance using the MME benchmark. However, it appears that the tasks I expected to see were not included in the evaluation. When I used the '--tasks list' command to display the available tasks, it became evident that only a subset of tasks had been installed. Can you assist me in resolving this issue?

[lmms-eval/lmms_eval/__main__.py:240] INFO Available Tasks:
 - mathverse
 - mathverse_testmini
 - mathverse_testmini_text_dominant
 - mathverse_testmini_text_lite
 - mathverse_testmini_text_only
 - mathverse_testmini_vision_dominant
 - mathverse_testmini_vision_intensive
 - mathverse_testmini_vision_only
 - mathvista
 - mathvista_test
 - mathvista_testmini
 - ocrbench

If you are sure everything is under the folder and you install our packages, you might want to use `lmms-eval --task list --verbosity DEBUG` to see what is preventing you from loading the task.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: How to add keywords stopping criterion during evaluation?
Root cause: Unimplemented Feature Gap

Hi, thank you for your great work!

I was trying to use your toolkit to evaluate my trained llava-llama3. My llama3 was designed to stop generation after the `
` token. In the original LLaVA repo, I simply added these lines before `model.generate` and it works fine:
```python
conv = conv_templates[args.conv_mode].copy()
stop_str = conv.sep if conv.sep_style != SeparatorStyle.TWO else conv.sep2
keywords = [stop_str]
stopping_criteria = KeywordsStoppingCriteria(keywords, tokenizer, input_ids)
```
If these lines were not added, model would continue to produce texts that are just gibberish. For example, it took two hours to get evaluation reults for MMMU because of the undesired continued generation problem. So I wonder if it is possible to custom-configure `stopping_criteria` during evaluation when using lmms-eval

For future reference, adding the desired stop tokens in the generation config should resolve this issue.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: No chat template is set for this processor
Root cause: Validation Gap

bash examples/models/llama_vision.sh

ValueError: No chat template is set for this processor. Please either set the `chat_template` attribute, or provide a chat template as an argument. See https://huggingface.co/docs/transformers/main/en/chat_templating for more information.

**Where can I set this chat template?**

Same issue. Encountered when using base model instead of instruct model.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Qwen2.5-Omni-7B CUDA OOM
Root cause: Resource Mishandling

I was easily able to do SFT on 4 *48GB GPUs, but running evaluation on video benchmarks like video_mmmu or vinoground (very short videos) gives OOM. Only using 1 GPU and splitting the model across the 4 GPUs (device_map="auto"), the issue still persists.

```
Token indices sequence length is longer than the specified maximum sequence length for this model (115108 > 32768). Running this sequence through the model will result in indexing errors
2025-05-03 13:40:54.819 | ERROR    | lmms_eval.models.qwen2_5_omni:generate_until:302 - Error CUDA out of memory. Tried to allocate 334.76 GiB. GPU 0 has a total capacity of 44.42 GiB of which 27.48 GiB is free. Including non-PyTorch memory, this process has 16.93 GiB memory in use. Of the allocated memory 12.10 GiB is allocated by PyTorch, and 4.32 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) in generating
```

I met the same issue

https://github.com/QwenLM/Qwen2.5-Omni/issues/257#issuecomment-2852311027
I fixed it by installing those specific versions.
Update lmms-eval qwen_omni.py file to use flash_attention_2