zhiminy/EvalEng huggingface dataset:
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Bug in qwen2-vl ?
Root cause: Validation Gap

```python
  for i in range(len(contexts)):
      if "<image>" in contexts[i]:
          contexts[i] = contexts[i].replace("<image>", "")

  messages = []
  processed_visuals = []
  for i, context in enumerate(contexts):
      if "<image>" in context:
          context = context.replace("<image>", "")
```

Code above is from the Qwen2_VL's generate_until function,. It replaces all the "\<image\>" token to empty string. When I evaluate the longvideobench with subtitles, it interleaves <image> and subtitle text. In my opinion， it seems to evaluate in this format, but the implement of qwen2vl makes it only include text. Is this a bug?

Additionally, I see that the Qwen2_5_VL delete these code.

Because this is a dummy image token in LLaVA, so I don't think it would make any effect for Qwen2-VL. To truly interleave, the inputs are packed with messages format

> Because this is a dummy image token in LLaVA, so I don't think it would make any effect for Qwen2-VL. To truly interleave, the inputs are packed with messages format

Thank you for your answer.
Another question, the option to w/wo subtitiles is false default.

Does this meet the requirements of the benchmark?

I think there are 2 splits, one is interleave one is v right? Does this correspond to with and without subtitle?

> I think there are 2 splits, one is interleave one is v right? Does this correspond to with and without subtitle?

```python
  if lmms_eval_specific_kwargs.get("insert_interleave_subtitles", False):
      with open(Path(__file__).parent / "longvideobench_val_i.yaml", "r") as f:
          raw_data = f.readlines()
          safe_data = []
          for i, line in enumerate(raw_data):
              # remove function definition since yaml load cannot handle it
              if "!function" not in line:
                  safe_data.append(line)
      cache_name = yaml.safe_load("".join(safe_data))["dataset_kwargs"]["cache_dir"]
      subtitle_subdir_name = yaml.safe_load("".join(safe_data))["dataset_kwargs"].get("subtitle_subdir", "subtitles")
      cache_dir = os.path.join(base_cache_dir, cache_name, subtitle_subdir_name)
      with open(os.path.join(cache_dir, doc["subtitle_path"])) as f:
          subtitles = json.load(f)

      max_num_frames = yaml.safe_load("".join(safe_data))["dataset_kwargs"].get("max_num_frames", 16)

      frame_timestamps = compute_frame_timestamps(doc["duration"], max_num_frames)
      interleaved_prefix = insert_subtitles_into_frames(frame_timestamps, subtitles, doc["starting_timestamp_for_subtitles"], doc["duration"])
      return f"{pre_prompt}{interleaved_prefix}\n{question}\n{post_prompt}"
  else:
      return f"{pre_prompt}{question}\n{post_prompt}"
```
This is the code segment to construct the prompt of longvideobench.
If the insert_interleaved_subtitles is false(default is false), there is no subtitles.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: The version does not match the llava version.
Root cause: Environment Incompatibility

pip install lmms-eval，
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
llava 1.2.2.post1 requires transformers==4.37.2, but you have transformers 4.39.2 which is incompatible.

How can I solve this problem? Thank you

Hi, I think you can ignore this warning. This is just a warning from pip that it is not resolving the dep conflicts. If it is not effecting the evalution, it is okay.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Unrecognized AriaConfig
Root cause: External Dependency Breakage

Hi, I got this error when testing the rhymes-ai/Aria
The error is:

`    accelerate launch --num_processes=8 --main_process_port 12345 -m lmms_eval \
        --model aria \
        --model_args pretrained=rhymes-ai/Aria \
        --tasks seedbench \
        --batch_size 1 \
        --output_path ./logs/ \
        --log_samples`


 File "/anaconda3/envs/lmms-eval/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 567, in from_pretrained
    raise ValueError(
ValueError: Unrecognized configuration class <class 'transformers.models.aria.configuration_aria.AriaConfig'> for this kind of AutoModel: AutoModelForCausalLM.
Model type should be one of AriaTextConfig, BambaConfig, BartConfig, BertConfig, BertGenerationConfig, BigBirdConfig, BigBirdPegasusConfig, BioGptConfig, BlenderbotConfig, BlenderbotSmallConfig, BloomConfig, CamembertConfig, LlamaConfig, Code
GenConfig, CohereConfig, Cohere2Config, CpmAntConfig, CTRLConfig, Data2VecTextConfig, DbrxConfig, DiffLlamaConfig, ElectraConfig, Emu3Config, ErnieConfig, FalconConfig, FalconMambaConfig, FuyuConfig, GemmaConfig, Gemma2Config, GitConfig, GlmC
onfig, GotOcr2Config, GPT2Config, GPT2Config, GPTBigCodeConfig, GPTNeoConfig, GPTNeoXConfig, GPTNeoXJapaneseConfig, GPTJConfig, GraniteConfig, GraniteMoeConfig, GraniteMoeSharedConfig, HeliumConfig, JambaConfig, JetMoeConfig, LlamaConfig, Mam
baConfig, Mamba2Config, MarianConfig, MBartConfig, MegaConfig, MegatronBertConfig, MistralConfig, MixtralConfig, MllamaConfig, MoshiConfig, MptConfig, MusicgenConfig, MusicgenMelodyConfig, MvpConfig, NemotronConfig, OlmoConfig, Olmo2Config, O
lmoeConfig, OpenLlamaConfig, OpenAIGPTConfig, OPTConfig, PegasusConfig, PersimmonConfig, PhiConfig, Phi3Config, PhimoeConfig, PLBartConfig, ProphetNetConfig, QDQBertConfig, Qwen2Config, Qwen2MoeConfig, RecurrentGemmaConfig, ReformerConfig, Re
mBertConfig, RobertaConfig, RobertaPreLayerNormConfig, RoCBertConfig, RoFormerConfig, RwkvConfig, Speech2Text2Config, StableLmConfig, Starcoder2Config, TransfoXLConfig, TrOCRConfig, WhisperConfig, XGLMConfig, XLMConfig, XLMProphetNetConfig, X
LMRobertaConfig, XLMRobertaXLConfig, XLNetConfig, XmodConfig, ZambaConfig, Zamba2Config

Can `pip install -U transformers` solve this issue?
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [BUG] NameError: name 'random' is not defined in qbench/utils.py causes failure in qbench task.
Root cause: Validation Gap

**Description**
During the execution of the Q-Bench evaluation task, a NameError: name 'random' is not defined error occurs when parsing multi-choice answers from the model. This is caused by the use of random.choice() in utils.py without importing the random module.
Trigger Condition
When the parse_multi_choice_response function attempts to randomly select an answer (e.g., when the model output cannot be parsed).
Relevant code location: lmms_eval/tasks/qbench/utils.py, line 81:
``` python
   pred_index = random.choice(all_choices) 
```
**Proposed Fix**
Add the following import statement at the top of lmms_eval/tasks/qbench/utils.py:

Thanks, I have fixed it
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Results differ when running multiple tasks
Root cause: Algorithmic Error

Hello, thank you for your work! I'm encountering a problem when evaluating tinyllava on multiple tasks, where the results differ compared to running each task on its own. The commands are exactly the same between the two runs, only differing in the amount of tasks ran. I have also attached the results from each as an example.

Commands:
Two tasks command:
```python
python3 -m accelerate.commands.launch \
    --num_processes=4 \
    -m lmms_eval \
    --model tinyllava \
    --model_args pretrained="$PATH,conv_mode=$conv_mode" \
    --tasks mme,mmstar \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix $suffix \
    --output_path outputs/ \
    --verbosity=DEBUG
```

Multiple tasks command:
```python
python3 -m accelerate.commands.launch \
    --num_processes=4 \
    -m lmms_eval \
    --model tinyllava \
    --model_args pretrained="$PATH,conv_mode=$conv_mode" \
    --tasks mme,mmstar,mmvet,pope,scienceqa,textvqa,gqa,vizwiz_vqa,seedbench_2_plus \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix $suffix \
    --output_path outputs/ \
    --verbosity=DEBUG
```
Results:
Two tasks:

<img width="588" alt="Image" src="https://github.com/user-attachments/assets/a92faafc-d04f-4a2f-8d4f-4c667adc50eb" />

Multiple tasks:
<img width="715" alt="Image" src="https://github.com/user-attachments/assets/631d8302-9362-4068-a496-2a33f63a546a" />
Here we see that the mme and mmstar results are completely different compared to the earlier image when we run just two tasks (mme and mmstar) with mmstar getting 0% in the multiple tasks command compared to 23% when we run just mme and mmstar (the same story happens for mme).


Hi @pumetu , thank you for the  report. This seems to be an important issue. Have you checked the logs that the result are consistent for mme and mmstar under different settings? I will try to reproduce this result recently.

Hi @kcz358, I've found that running them individually is fine but I forgot to log other combinations (some were fine some failed) I can test a few more and get back to you. A side note, I found that some benchmarks received a very low score compared to other implementations (ScienceQA and textVQA), I tested the official tinyllava implementation for scienceQA and got around 60-80% but with the same model on lmms it falls to 20-35%.

Thanks and sorry for the late reply, I forget to reproduce this error recently. I will assign this to myself and check if I can find out the source of error.

I know this is old, but this might be an issue related to tinyllava in particular, especially this [line](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/8daf4457d757488e6e2d79d50ebddd4aa5720d13/lmms_eval/models/simple/tinyllava.py#L316). 

If the `gen_kwargs` differ from one task to another, tinyllava will take the wrong `gen_kwargs` for some tasks (as it uses the same `gen_kwargs` no matter the task)
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Qwen2_5-vl frames control
Root cause: Unimplemented Feature Gap

`lmms_eval/models/qwen2_5_vl.py` line 242
```python
message.append({"role": "user", "content": [{"type": "video", "video": visual, "max_pixels": 360 * 420}, {"type": "text", "text": context}]})
```
Would it be better to change to the following to control the number of frames?
```python
message.append({"role": "user", "content": [{"type": "video", "video": visual, "max_pixels": 360 * 420,"nframes":self.max_num_frames}, {"type": "text", "text": context}]})
```
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [BUG] Long Idle Time Between Batches in vLLM, GPU Utilization Drops to 0
Root cause: Architectural Constraint

**Description:**  
When using vLLM, I noticed that there is a long idle time between each batch, during which GPU utilization drops to 0. This significantly affects performance.  

### **Steps to Reproduce**  
Env setup:

```bash
conda create -n lmms_eval python=3.12 -y
conda activate lmms_eval
pip install torch==2.5.1 transformers==4.49.0 peft==0.12.0 vllm==0.7.3 accelerate==1.4.0
cd src/lmms-eval
pip install -e .
```

Run the following commands:  

```bash
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export NCCL_BLOCKING_WAIT=1
export NCCL_TIMEOUT=18000000
export NCCL_DEBUG=DEBUG

python3 -m lmms_eval \
    --model vllm \
    --model_args model_version=llava-hf/llava-1.5-7b-hf,tensor_parallel_size=1 \
    --tasks gqa \
    --batch_size 64 \
    --log_samples \
    --log_samples_suffix vllm \
    --output_path ./logs
```

### **Expected Behavior**  
The batches should be processed efficiently without long idle times, ensuring continuous GPU utilization.  

### **Actual Behavior**  

Between each batch, there is a significant idle period where GPU utilization drops to 0. 

For example, the actual inference time for the first two batches was 02:06 + 01:49, but the total time taken was 12:52.

<img width="541" alt="Image" src="https://github.com/user-attachments/assets/738b3689-d0dd-4d7c-bc29-1c5defbeb6f0" />

I am guessing this is the time taken for processing the images for each batches, but I am not totally sure for this. May I ask are there any similar issue for offline inference for vllm?

I haven't observed this issue during pure text inference, so if it's related to image processing time, would it be possible to handle it asynchronously or through preprocessing?

Yeah then I think this could be possibly the reason. I do observe that the image processing time for vllm sometimes unusually takes too much. I think another solution is to first host a vllm service then use the openai compatible model to post the request. But I remember currently that one is only multiprocessing, possibly need to do the same thing as in the srt model with async openai
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: MLVU Bug
Root cause: Validation Gap

There is a bug for mlvu.

key error of "needle"

same

Can you provide more error logs? For example how to reproduce this error?

> Can you provide more error logs? For example how to reproduce this error?
I put my error log here
https://github.com/EvolvingLMMs-Lab/lmms-eval/issues/595

same. Anyone figured it out?

Hi, could you try to update the lmms-eval to the latest version?

I can't reproduce the error with this commands. Could you share your env and scripts?

```bash
accelerate launch --num_processes=8 --main_process_port 12345 -m lmms_eval \
        --model qwen2_5_vl \
        --model_args pretrained=Qwen/Qwen2.5-VL-7B-Instruct,use_flash_attention_2=True \
        --tasks mlvu_dev,mlvu_test \
        --batch_size 1 \
        --output_path ./logs/ \
        --limit 16 --verbosity DEBUG
```

I also get the same error. I use this command. @kcz358 

```
accelerate launch --num_processes=1 --main_process_port 12345 \
-m lmms_eval \
--model llava_vid \
--model_args pretrained=lmms-lab/LLaVA-Video-7B-Qwen2,conv_template=qwen_1_5,max_frames_num=64,mm_spatial_pool_mode=average \
--tasks mlvu \
--batch_size 1 \
--log_samples \
--log_samples_suffix llava_vid
```




> I can't reproduce the error with this commands. Could you share your env and scripts?
> 
> accelerate launch --num_processes=8 --main_process_port 12345 -m lmms_eval \
>         --model qwen2_5_vl \
>         --model_args pretrained=Qwen/Qwen2.5-VL-7B-Instruct,use_flash_attention_2=True \
>         --tasks mlvu_dev,mlvu_test \
>         --batch_size 1 \
>         --output_path ./logs/ \
>         --limit 16 --verbosity DEBUG

Do you mind explain what is the difference be between mlvu_dev,mlvu_test and mlvu? Which one should I use to reproduce LLaVA mlvu result

> > I can't reproduce the error with this commands. Could you share your env and scripts?
> > accelerate launch --num_processes=8 --main_process_port 12345 -m lmms_eval 
> > --model qwen2_5_vl 
> > --model_args pretrained=Qwen/Qwen2.5-VL-7B-Instruct,use_flash_attention_2=True 
> > --tasks mlvu_dev,mlvu_test 
> > --batch_size 1 
> > --output_path ./logs/ 
> > --limit 16 --verbosity DEBUG
> 
> Do you mind explain what is the difference be between mlvu_dev,mlvu_test and mlvu? Which one should I use to reproduce LLaVA mlvu result

@kcz358  Do you mind explain it ?


I think the version used in LLaVA should be mlvu_dev. I remember mlvu_test is being added later. Can @shuyansy double check on this? Thanks
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: How to add the new dataset?
Root cause: Documentation Deficiency

i've create the yaml file under the guidance. but it still not appear in available task list when calling `lmms-eval --tasks list`, what should I do?

You might want to set `lmms-eval --tasks list --verbosity DEBUG` to see why the task is not added into the list.

it works thank you!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Local docvqa dataset loading speed slow
Root cause: Configuration Error

program stuck in `task.py` :
```python
self.dataset = datasets.load_dataset(
                path=self.DATASET_PATH,
                name=self.DATASET_NAME,
                download_mode=datasets.DownloadMode.REUSE_DATASET_IF_EXISTS,
                download_config=download_config,
                **dataset_kwargs if dataset_kwargs is not None else {},
            )
```
already set `local_files_only: True`

while infovqa dataset runs file.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Recommended Practices for Video Benchmark Submission + Broken Link
Root cause: Documentation Deficiency

Thanks for maintaining this fantastic repository!

I am planning to submit a video evaluation dataset - what is the recommended way to submit video datasets?

In HF, I can either upload a zip file of the videos or upload them as is, in a folder- as my benchmark is not too large.

I looked at the `utils` file of some other video benchmarks - some assume that the videos are present in the cache dir of the dataset, while others unzip the downloaded zip files in the code.

What is preferred?

Are users of lmms-eval expected to prepare the datasets themselves before evaluating, o is the contributor of the dataset expected to write a script that does all of this in the utils?


Also, 
I was checking the documentation for dataset format [here](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/docs/README.md) but
```
If you need to upload your datasets into correct HF format with viewer supported, please refer to tools
```
but the link to `tools` - https://github.com/EvolvingLMMs-Lab/lmms-eval/tree/pufanyi/hf_dataset_docs/tools) - is broken (404)

Is there an updated guideline I could check out?

Thanks!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Significant performance drop on MMMU (val) when evaluating LLaVA-OV
Root cause: Algorithmic Error

Hi, I am trying to use this package to reproduce `lmms-lab/llava-onevision-qwen2-7b-ov`. However, I found significant performance drop on MMMU (val).
1. LLaVA-OV reports MMMU=48.3
2. lmms-eval gets MMMU=43.5

BTW, I get reproduce the number on AI2D, ChartQA, MMBench (en-dev), and SeedBench (image) by using lmms-eval.
Do you know what could be the potential reason of the performance drop on MMMU? Thanks.

resolved by pulling the latest code.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Metadata not implemented for LLaVA Onevision + Loglikelihoods Outputs
Root cause: Unimplemented Feature Gap

I am trying to do video evaluation using my own task with `llava_onevision`, however at [line](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/fbda82f020833b31b416f0b5b5dc7470ee5bba62/lmms_eval/models/llava_onevision.py#L268)

```python
if "task_type" in self.metadata and self.metadata["task_type"] == "video" and "sample_frames" in self.metadata:

```

when run with the command;

```bash
 python3 -m accelerate.commands.launch \                                                                                                                                                                                       _  lmms _  varungupta@jatayu
    --num_processes=1 \
    -m lmms_eval \
    --model llava_onevision \
    --model_args pretrained="lmms-lab/llava-onevision-qwen2-0.5b-ov" \
    --tasks velociti \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix llava_onevision_lmms_run \
    --output_path ./logs/ \
    --verbosity=DEBUG
```

gives an error,

```bash
    if "task_type" in self.metadata and self.metadata["task_type"] == "video" and "sample_frames" in self.metadata:
AttributeError: 'Llava_OneVision' object has no attribute 'metadata'
```

Also, I am trying to submit a new task for video evaluation - but my benchmark requires logits score from the model of "Yes" and "No" token - how can I do this? 

I couldn't find any task with `output_type: loglikelihood` in its YAML configuration.
I require either logits from the model, and then the token ID from its processor of 'Yes' and 'No' tokens - how do I fetch these values inside my `utils.py` code for my new task?

This is my benchmark's scoring logic in my `utils.py` - how do I get this running?

```python
def get_score(logits, token_id_yes, token_id_no):
    logits = F.softmax(logits, dim=-1)
    score_yes = logits[token_id_yes] / (logits[token_id_yes] + logits[token_id_no])
    score_no = logits[token_id_no] / (logits[token_id_yes] + logits[token_id_no])
    return score_yes.item(), score_no.item()
```


Lastly, I see `loglikehood` not implemented for most video models - is that in the immediate plans to complete?


(@kcz358 - can you please help, thanks)
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: quetions about the evals
Root cause: Validation Gap

Thanks for your great work!

I have some questions about the evaluation results.
1. TextVQA, results of  llava-1.5-13B get score of 61.3 in the paper, but in the sheet https://docs.google.com/spreadsheets/d/1a5ImfdKATDI8T7Cwh6eH-bEsnQFzanFraFUgcS9KHWc/edit#gid=0, the result is only 48.73, why?
2. After executing the command as README, the generated submission file of mmbench seems wrong. When I upload it to the evaluation server, I got the logs "Your excel file should have a column named `A`, please double check and submit again"

hope to get your response, thanks ~


I think it's because llava reports the `test` result meanwhile we are reporting `val` split.

Running textvqa will get both val metric result and a submission file for test split.

And users may submit the file to https://eval.ai/web/challenges/challenge-page/874/ to get the results.

Maybe @pufanyi could address this more clearly. 

Hello! Thank you for your interest in our work! Regarding question 1, as referenced in the LLaVA evaluation code [here](https://github.com/haotian-liu/LLaVA/blob/7440ec9ee37b0374c6b5548818e89878e38f3353/llava/eval/eval_textvqa.py#L17), the OCR token was utilized as a prompt for evaluation.

However, in our evaluation of LLaVA, we did not incorporate the OCR token:

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/cbb387441fd039252a30580d40ee31491be7aaf5/lmms_eval/tasks/textvqa/_default_template_textvqa_yaml#L14

To achieve results consistent with the LLaVA paper, you can set this to `true`.

@Luodian @pufanyi thanks for your reply, as for the question 2, do you have any ideas?

As for MMBench, We are updating it in a new PR #7 .  

But the we are waiting the run to finish to get a eval result and then decide to merge it. You can also help to check if it's correct.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Way to do model response and post-processing seprately.
Root cause: Unimplemented Feature Gap

Hello, I would like to test models on the captions datasets such as activitynetqa, mlvu, etc, which requires long running time for both the model response and post-processing part. However, it seems that those two parts are currently coupled together. When I got some errors from the post-processing step, I need to re-run the model response part again to test if the post-processing work. 
I wonder if there is an option to cache the responses from the first model response phase and then apply to the post-processing phase.

It's not directly supported, but I think it's valid to add this feature in current pipeline.

The relevant code is here:

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/4abf677d90a1f85f69aa929a81f3eba069e390e5/lmms_eval/evaluator.py#L444-L465
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: How to use cached datasets from huggingface
Root cause: Documentation Deficiency

I have cached the refcoco dataset from huggingface, now the dataset is in ".cache/huggingface/datasets/lmms-lab___ref_coco", however, when I changed the dataset_path in "lmms-eval/lmms_eval/tasks/refcoco/default*" from "lmms-lab/RefCOCO" to ".cache/huggingface/datasets/lmms-lab___ref_coco", I'm facing a bug KeyError: 'testB', is there anything that I should modify further?

![Image](https://github.com/user-attachments/assets/4ca407d7-19b4-43b5-b3cc-ba226f105812)

Download the refcoco dataset from huggingface and set the path to local can solve this.

Hi @Creator404 , I'm facing the same issues with vinoground. Can you please share exactly which path to modify? 

> Hi [@Creator404](https://github.com/Creator404) , I'm facing the same issues with vinoground. Can you please share exactly which path to modify?

You can try changing the "dataset_path" to point to your local directory. However, I don’t quite remember if there are other files that also need modification. I've made quite a few changes while evaluating different tasks, so it's possible that some bugs remain. You might need to debug a bit as you go.