zhiminy/EvalEng huggingface dataset:
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: How to debug in vscode

```
accelerate launch --num_processes=8 -m lmms_eval --model llava   --model_args pretrained="liuhaotian/llava-v1.5-7b"   --tasks mme,mmbench_en --batch_size 1 --log_samples --log_samples_suffix llava_v1.5_mme_mmbenchen --output_path ./logs/ #
```

Since "lmms_eval" is a folder in your script, I would like to know how to debug in vscode. What should the launch.json of vscode be like?

It will be like the following

```
{
    "name": "LMMs-Eval-ICONQA",
    "type": "python",
    "request": "launch",
    "module": "accelerate.commands.launch",
    "python": "/home/tiger/miniconda3/envs/lmms-eval/bin/python",
    "env": {
        "CUDA_VISIBLE_DEVICES": "0,1,2,3,4,5,6,7",
        "PYTHONWARNINGS": "ignore",
        "TOKENIZERS_PARALLELISM": "false",
        "ACCELERATE_DEBUG_MODE": "1"
    },
    "cwd": "${workspaceFolder}",
    "justMyCode": false,
    "args": [
        "--main_process_port=12566",
        "--num_processes=1",
        "lmms_eval",
        "--model=llava",
        "--model_args=pretrained=/home/tiger/checkpoints/llava-v1.5-7b",
        "--tasks=iconqa_val",
        "--batch_size=1",
        "--limit=8",
        "--log_samples",
        "--log_samples_suffix=debug",
        "--output_path=./logs/",
        "--verbosity=DEBUG"
    ],
    "console": "integratedTerminal"
},
```

> It will be like the following
> 
> ```
> {
>     "name": "LMMs-Eval-ICONQA",
>     "type": "python",
>     "request": "launch",
>     "module": "accelerate.commands.launch",
>     "python": "/home/tiger/miniconda3/envs/lmms-eval/bin/python",
>     "env": {
>         "CUDA_VISIBLE_DEVICES": "0,1,2,3,4,5,6,7",
>         "PYTHONWARNINGS": "ignore",
>         "TOKENIZERS_PARALLELISM": "false",
>         "ACCELERATE_DEBUG_MODE": "1"
>     },
>     "cwd": "${workspaceFolder}",
>     "justMyCode": false,
>     "args": [
>         "--main_process_port=12566",
>         "--num_processes=1",
>         "lmms_eval",
>         "--model=llava",
>         "--model_args=pretrained=/home/tiger/checkpoints/llava-v1.5-7b",
>         "--tasks=iconqa_val",
>         "--batch_size=1",
>         "--limit=8",
>         "--log_samples",
>         "--log_samples_suffix=debug",
>         "--output_path=./logs/",
>         "--verbosity=DEBUG"
>     ],
>     "console": "integratedTerminal"
> },
> ```

![Image](https://github.com/user-attachments/assets/1c91fbfa-0939-4eee-b9af-2bd6e70d4df3) The program is stuck here. How to solve this?
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: No such file or directory & TypeError: expected str, bytes or os.PathLike object, not NoneType
Root cause: Validation Gap

<img width="1584" alt="image" src="https://github.com/user-attachments/assets/345a0802-6f19-40bc-96ca-9a365ec3df67">
<img width="844" alt="image" src="https://github.com/user-attachments/assets/dbc70fda-ebe8-4583-8376-03c169f20ad6">
How to solve the above bugs?
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [logging] The program is not responding during multi-gpu evaluation
Root cause: Resource Mishandling

[logging] The program is not responding during multi-gpu evaluation.
The program froze and did not respond during multi-gpu evaluation.

After checking the code, I've noticed that multi-gpu evaluation will create a "rank0_metric_eval_done.txt" under the result output-dir. (from [New Updates] LLaVA OneVision Release; MVBench, InternVL2, IXC2.5 Interleave-Bench integration. [#182]). However, due to the gap among the logging dir creation (process 0 is often slower). There are 2 folders created :
![image](https://github.com/user-attachments/assets/4fd0f941-ef2c-47f4-a8c4-f4a0c040b7b1)
The logging procedure is done in 2 different minutes. So, the "txt" files are saved in different dirs. Subsequently, the program will run into a while loop to check whether the "txt" from all GPUs already exist, which will cause a dead loop.

FIX:my way is simple, I just cancelled the minutes display during the creation of logging dir:
(lmms-eval/lmms_eval/utils.py) 
![image](https://github.com/user-attachments/assets/b8d31bfc-b787-4bb5-80ef-3b596cd46610)
Maybe there exist more appropriate methods.



that's interesting, thanks for your PR, let me see if there's better to improve.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [BUG]: gen_kwargs in requests are shallow copy?
Root cause: Resource Mishandling

I have used the repo to evaluate LLaVA and InternVL2 model, both met the same problem, if I `gen_kwargs.pop("some_key", None)` in previous requests, then in latter requests **before executing that line of code**, that key in gen_kwargs will **already** been popped.  In general, the first two or three or more requests work fine, popping the key out of previous gen_kwargs won't affect the latter ones, but then at some point, all the gen_kwargs will have that key already been popped before exectuing that line of code.
To be pointed, **the key is specified through cli commands**, not written in the yaml file. *generation_kwargs specified through cli, these settings will be used over set parameters in yaml tasks.*
I don't know the logic of generating the requests, but I think this is a bug. Currently I use copy.deepcopy to fix this.
![image](https://github.com/user-attachments/assets/65d80a21-a478-47f0-8097-aa0ec1c09169)



Hello! We have fixed the bug in #231 by `deepcopy`. Thank you so much!!!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: How to check detailed results for benchmarks, like VideoMME?
Root cause: Documentation Deficiency

Before you open an issue, please check if a similar issue already exists or has been closed before.

- [x] A descriptive title:
- [x] A detailed description

Thank you for your contributions!

Benchmarks, like VideoMME, have many categories of tasks. How could we check the results for each task and results for their predefined video length? I find that such results are not in the result.json and videomme.json files. 

Thank you!

Hello,
I have the same concern. Could you tell me how did you solve this problem?
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: How to load dataset in offline mode?
Root cause: Documentation Deficiency

Since my server can't access huggingface.co, how can I load dataset from local direction?
Thank you for your contributions!


Hi, you can refer to

#15 and #21 

Thanks, I solve this problem by defining `local_dir` in `lmms-eval/lmms_eval/api/task.py line.528`
```python
self.DATASET_PATH = os.path.join(local_dir, self.config.dataset_path)
```
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Cannot find a valid ref in  kc/worldqa
Root cause: Documentation Deficiency

Before you open an issue, please check if a similar issue already exists or has been closed before.

### When you open an issue, please be sure to include the following

- [ ] A descriptive title: [xxx] XXXX
- [ ] A detailed description

Thank you for your contributions!


Hi, I think now the worldqa has been officially integrated in to the `lmms-eval`. There is no need to switch to a different branch to evaluate worldqa. You can evaluate using main branch
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: task_manager.list_all_tasks() is not showing outputs properly.
Root cause: Validation Gap

1. `task_manager.list_all_tasks()` function in line [409](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/d309fa17633af8bee739697a1247e21427892b8c/lmms_eval/__main__.py#L407-L409) needs to be written as follows to prevent improper display of all tasks in terminal. I am running in docker and there `\n` character isn't properly executing. So need to add `flush=True` in print statement.
2. Since return of `task_manager.list_all_tasks()` is string and adding `"\n - ".join()` along with `sorted()` functionality is making it worse.
3. Correction of line 409; `eval_logger.info("Available Tasks:\n - {}".format(task_manager.list_all_tasks()), flush=True)`


Having the same issue!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Encountering task missing issue even when specifying task from list mentioned at `current_tasks.md`
Root cause: Validation Gap

As per the code snippet at line [446-461](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/d309fa17633af8bee739697a1247e21427892b8c/lmms_eval/__main__.py#L446-L461), `args.tasks.split()` create an empty string in list if comma (`,`) is used at end (just before line continuation character `\`) while mentioning multiple task in `--tasks` argument in CLI. Example shown below.

```bash
python3 -m accelerate.commands.launch \
    --num_processes=8 \
    -m lmms_eval \
    --model llava \
    --model_args pretrained="/dockerx/code/AMD-LLaVA/checkpoints/llava-v1.5-13b-lora-test-zero3-merged,conv_template=vicuna_v1" \
    --tasks ok_vqa,pope,ai2d, \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix llava_v1.5_ok_vqa_pope_ai2d \
    --output_path /dockerx/code/lmms-eval/logs/ \
    --verbosity='DEBUG'
```

Fix: Need to remove the comma at end.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Is multi-node evaluation possible?

Hi,

Is multi-node evaluation possible? If yes, how?

Best,
Orr

possible, but we are still experimenting and wish to support it soon.

@choiszt has already done the trial experiment, may explain it in detail.

Hi Orr,

Yes, multi-node evaluation is possible, but there is a small modification needed to ensure the correct rank is used when working with multi-node setups.

To enable proper multi-node evaluation, you should update the _rank assignment in the init function within the lmmseval model, such as in lmms_eval/models/llava_onevision.py. Currently, the code uses [self._rank = self.accelerator.local_process_index](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/9515cb4442199f0dec2b1634341b2b97b11b12ab/lmms_eval/models/llava_onevision.py#L179), which works fine for single-node setups but may not properly handle multi-node cases.

You should change to `self._rank = self.accelerator.process_index` and use appropriate slurm/torchrun commands to launch processes across multiple nodes.

Let me know if you have any questions or need further assistance!

Best,

Hi @choiszt,
Thank you for your help. 
I was able to run multi-node evaluation with your proposed changes.
Best,
Orr

@choiszt 
One thing I did notice: sometimes, the multi-node hangs. I do see that the evaluation completed + printed the results of all the eval. 
Any reason why?
Best,
Orr

Hi all,
Updating this:
The cause was that different nodes sometime get different log suffix (via the datetime):
https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/e1f9adefcc4b92a39edb2450fbadf659a84364fb/lmms_eval/__main__.py#L467

This causes the reports that the job is finished to sit in different dirs, ultimately messing up the end-of-run collection. I suggest you move the definition of this parameter under the `is_main_process=True' setup.
To solve this, I added another command line input that to overwrites the 'datetime'.

Hi Orr,
Thank you for the update, and we appreciate your investigation into the issue！🚀

Just to clarify, it seems like the evaluation scores are being generated correctly, and the issue is with the storage of the reports due to the datetime handling, correct? Please let us know if there are any other aspects we should address.

Best,
Shuai

Hi Shuai,

Yes what happens is that when torchrun is run near the 'minute' mark, two nodes call `utils.get_datetime_str(timezone=args.timezone)`, but one gets XX_XX_{rest of str} and the other XX_X(X+1)_{rest of str}. 

lmmseval produces these `rank{}_metric_eval_done.txt` files - but when the above happens, some are in one dir and the other in the other dir. This is the source of the hanging/etc. 

My suggestion is calling the datetime_str generator in `cli_evaluate` and not `cli_evaluate_single`. You can use the accelerator object to communicate the value you get from the main thread to the rest of the threads, and this should solve this issue. 

When I do this on my codebase, multinode works just fine (with the changes you suggested earlier).

Best,
Orr
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Issues with LMMs-EVAL for LLaVA-1.5-7B on ScienceQA and SeedBench
Root cause: Validation Gap

Hi,

Thank you for your excellent work! I've encountered two issues while using LMMs-EVAL to assess LLaVA-1.5-7B on ScienceQA (Image) and SeedBench:

<img width="677" alt="WechatIMG465" src="https://github.com/user-attachments/assets/f4289880-ba40-4b5b-86f8-5cdced247b66">
<img width="674" alt="WechatIMG464" src="https://github.com/user-attachments/assets/3f47d52e-795e-49a8-9e46-a524e7dcac2b">

1. **ScienceQA:** I noticed that changing the batch_size from 1 to 2 led to a performance drop from 69.56 to 58.70. Is such a significant variation expected?
2. **SeedBench:** For batch sizes of 1 and 2, LMMs-EVAL reported performances of 53.58 and 50.99, respectively. These results do not align with the official number of 60.49 listed in your [spreadsheet](https://docs.google.com/spreadsheets/d/1a5ImfdKATDI8T7Cwh6eH-bEsnQFzanFraFUgcS9KHWc/edit?gid=0#gid=0).

I'm running these evaluations in a different setup (torch 2.1.2 + cuda-11.2). Could this be affecting the outcomes, or might there be another issue?

Thank you for looking into this.

We do not recommend you to run llava with batch_size larger than 1 because it has some problem with batch inferencing and batch inferencing may be inaccurate in this case. If you are seeking batch inference, you may check srt model or llava sglang that use sglang as a server to perform inferencing.

For seedbench, the results are not aligned because we slightly changes the logic of the image tokens. Previously we only append one image token for llava in seedbench even when evaluating videos but now we append one image token for each frames and causing the score of video to decrease.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: No llavavid module in the latest LLaVA-NeXT
Root cause: Documentation Deficiency

I have installed LLaVA-NeXT and upgraded my dependency using something like
```
cd /path/to/LLaVA-NeXT;
pip install -e ".[train]"
``` 
But my code showed the **No module named 'llavavid'** error.

After searching I found this [issue](https://github.com/LLaVA-VL/LLaVA-NeXT/issues/124) tells that llavavid module has merged into llava. So what should I do at this step?

Hi, I figured out this tmp solution and it works for me after manually fix some unmatched argment.
Hopefully, there will be some update on this repo to catch up the latest change from llava side later on.
https://github.com/EvolvingLMMs-Lab/lmms-eval/issues/197#issuecomment-2337089080

@floatingbigcat Thank you very much!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [BUG] Commit #0d02bad may break `srt_api` by mis-use of asyncio
Root cause: Algorithmic Error

### When you open an issue, please be sure to include the following

- [x] A descriptive title: [xxx] XXXX
- [x] A detailed description


# Where's the problem

At `generate_until` in `srt_api.py`, at [here](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/8f9f18edaca0134d44ba9a544b87076539e86e0d/lmms_eval/models/srt_api.py#L217)

The author @kcz358 uses `asyncio.as_completed` in an order to send multiple requests to the server at the same time for faster evaluation, committed as 0d02bad.  

# What's happening
`asyncio.as_completed` is non-blocking, which means this iterator yield whatever request that has a response, regardless of their order of creation.  
So, the order of responses in `res` may not be the order of the `requests`. In fact I should use "probably", because that probability is well above 50% whenever the CPU has more than 4 threads resulting in a `num_processes` of more than 2.
This make `srt_api` more like a rolling dice rather a valid evaluation, giving my colleage a evaluation score of non-sense.

# How to fix
I'm no where near a professional python programmer, but I suggest using `asyncio.gather` instead, which is blocking thus preserves the order.  
Or, if I mis-understood your code? Feel free to correct me and please forgive me if so.


Looking forward for your reply ;)

Yeah I think you are correct. I will try to fix this.

Hi @Maxwell-Lyu , I have created a PR in #244 to fix this issue. The fix is quite simple but there seems somehow some issues in current sglang that might causing the evaluation result to be inconsistent. I have tested that our result can be reproduced by the commit hash `2f1d92834f41df42e266ed6d7036b4add906d21f`. I believe some changes after we PR have changed the behaviour of the model. I will try to check and fix it with the sgl team.


Thank you for your enthusiastic contribution to this project. Have a great day!

Hi @Maxwell-Lyu , I have fixed the bugs in the sglang in https://github.com/sgl-project/sglang/pull/1402. Since this is already got merged, you no longer need to checkout from that specific commit hash.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: download wildvision dataset error
Root cause: External Dependency Breakage

when eval wildvision bench, an error occurs:

ConnectionError: Couldn't reach https://huggingface.co/datasets/WildVision/wildvision-arena-data/resolve/5d8238e04e319f95b89addc9d697dd808df6f3cf/dataset_infos.json (error 401)

seems there is a problem about downloading this dataset, how to fix it?
Thanks~
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [Result] Can you provide some results on benchmarks? 

Hi,

Thank you for the excellent work, when I evaluate the model on different benchmarks, I find some results are unreasonable, so can u provide your results to help me figure out the correct config? Thank you. 


Thanks for this suggestion. Providing a comprehensive table is challenging, but we are trying to make it. 

Can you first point out which result is unreasonable? 

We can help you to figure out.

Hi, I run llava1.5-7b on ICON-QA but get the result is 0

I also run llava1.5-7b on MMVet and get 30.367, but the official result is 43.9
![image](https://github.com/user-attachments/assets/3c5f0303-cd47-45c8-bf36-15ba9b4e7ac2)


The evaluation on hallusion_bench_image has a bug
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: lmms_eval/evaluator.py", line 83, in simple_evaluate     assert tasks != [], "No tasks specified, or no tasks found. Please verify the task names." AssertionError: No tasks specified, or no tasks found. Please verify the task names.
Root cause: Validation Gap

lmms_eval/evaluator.py", line 83, in simple_evaluate
    assert tasks != [], "No tasks specified, or no tasks found. Please verify the task names."
AssertionError: No tasks specified, or no tasks found. Please verify the task names.

I dont understand, why?

```
03-22 20:22:48 [lewisjin/mllm/llms_eval.py:240] INFO Available Tasks:
 - ai2d
 - chartqa
 - cmmmu
 - cmmmu_test
 - cmmmu_val
 - docvqa
 - docvqa_test
 - docvqa_val
 - gqa
 - hallusion_bench_image
 - iconqa
 - iconqa_test
 - iconqa_val
 - infovqa
 - infovqa_test
 - infovqa_val
 - mmbench
 - mmbench_cn
 - mmbench_cn_cc
 - mmbench_cn_dev
 - mmbench_cn_test
 - mmbench_en
 - mmbench_en_dev
 - mmbench_en_test
 - mmmu
 - mmmu_test
 - mmmu_val
 - mmvet
 - multidocvqa
 - multidocvqa_test
 - multidocvqa_val
 - ok_vqa
 - ok_vqa_val2014
 - pope
```

And how to specific mmbench_cn custom local dataset path?

May I ask what is your command, you should use task names from the task list

For your second question, you may refer to #15 , #23 

Hi， looks like even set local envrioment variable, the script still try to read from hf remote, how to fix this issue?

> lmms_eval/evaluator.py", line 83, in simple_evaluate assert tasks != [], "No tasks specified, or no tasks found. Please verify the task names." AssertionError: No tasks specified, or no tasks found. Please verify the task names.

have you solved this problem? i met the same when using a machine that doesn't have internet access...

I am move to VLMEvalKit.....

For me, I have a local huggingface dataset builder which can be loaded with `load_dataset('/path/to/my/dataset/dataset.py') but throw this error when I put `dataset_path: /path/to/my/dataset` in the yaml.
Step through the debugger, I found the error lies in TaskConfig's post_init method in `api/task.py`.

Remove the line 99-103 solves the problem, which fail on `import_module('/path/to/my/dataset/')`:
```python
   def __post_init__(self):
        #if self.dataset_path and os.path.exists(os.path.dirname(self.dataset_path)):
        #    import inspect
         #   from importlib import import_module

        #    self.dataset_path = inspect.getfile(import_module(self.dataset_path))
```
Not sure what's the purpose of this code but thing runs fine if I remove it.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: video-mme (short/medium/long)
Root cause: Unimplemented Feature Gap

Hi,

Currently, video-mme only outputs an overall score, without the short/medium/long video breakdown.
How can I get this?

Best,
Orr

@choiszt Hi Shuai, can you help to check this?

Hi Orr,

Regarding the breakdown of short, medium, and long video types (VIDEO_TYPE), as well as the CATEGORIES and SUB_CATEGORIES, we currently output these directly by printing them to the terminal.

If you’re facing issues with terminal output due to multi-node setups, you might consider redirecting the terminal output to a .txt file, which would allow you to check specific fields more easily.

For more details on how we handle the evaluation output, you can refer to our implementation [here](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/e1f9adefcc4b92a39edb2450fbadf659a84364fb/lmms_eval/tasks/videomme/utils.py#L346).

Best,
Shuai

Hi @choiszt,

I am doing a lot of evals (1000s), so am using the results.json as opposed to the terminal printouts to automatically collect the evaluation results.

Is there an easy way I can change the write out to have all these in the results.json? could you point me to where I would need to make the changes?

Best,
Orr

Hi Orr, here's code related to saving `results.json`

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/e1f9adefcc4b92a39edb2450fbadf659a84364fb/lmms_eval/__main__.py#L511-L537

I was considering making a PR to fix the output_path issue, since it's mentioned many times.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Issue of GPT4o evaluation with dense frame sampling strategy
Root cause: Validation Gap

Thanks for open-sourcing this great evaluation framework!

I've encountered an issue while using GPT-4o on Video-MME and was wondering if anyone has already found a solution for it.

When attempting to evaluate GPT-4o with a dense frame sampling strategy (e.g., 512 frames) in low-resolution mode, I got an invalid request error. I’ve verified that the number of tokens is within the 128K context length limit, and I've tried both normal API and Batch API requests, but the error happens.

Specifically, I’m prompting GPT-4o with 338 frames in low-resolution mode, interleaved with some text prompts between the frames. The following error occurs:
`openai.BadRequestError: Error code: 400 - {'error': {'message': 'Too many images in request: 251, maximum allowed: 250.', 'type': 'invalid_request_error', 'param': None, 'code': None}}`

@HYUNJS Where can I find the evaluation files for GPT-4O? I only see files for [4V](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/lmms_eval/models/simple/gpt4v.py)