zhiminy/EvalEng huggingface dataset:
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Allow loading models and tasks from external packages
Root cause: Unimplemented Feature Gap

At the current stage there's a somehow simple way to include new tasks using the --include_external flag, yet there's no way to include external models except from cloning the lmms-eval repository and doing modifications on it, which might not be the ideal scenario in many cases (for example when developing new models internally).

In my scenario I would like to work on an external package (let's say it's an lmms-eval plugin) without the necessity of cloning this repository (using it as a package)

Right now I've done a fork of the repository which allows loading external tasks and models using an environment variable called `LMMS_EVAL_PLUGINS` to include external repositories (like LMMS_EVAL_PLUGINS=package1,package2), particularly in the `__main__.py` I'm able to integrate new tasks in this way

```python
if os.environ.get("LMMS_EVAL_PLUGINS", None):
    for plugin in os.environ["LMMS_EVAL_PLUGINS"].split(","):
        package_tasks_location = importlib.util.find_spec(f"{plugin}.tasks").submodule_search_locations[0]
        eval_logger.info(f"Including path: {args.include_path}")
        include_path(package_tasks_location)
```
Assuming that there's a package installed which follows the same structure as lmms-eval, this piece of code retrieves all the content of {package_name}.tasks and includes them as tasks.

In a similar way I've modified the `__init__.py` of models
```python
if os.environ.get("LMMS_EVAL_PLUGINS", None):
    # Allow specifying other packages to import models from
    for plugin in os.environ["LMMS_EVAL_PLUGINS"].split(","):
        m = importlib.import_module(f"{plugin}.models")
        for model_name, model_class in getattr(m, "AVAILABLE_MODELS").items():
            try:
                exec(f"from {plugin}.models.{model_name} import {model_class}")
            except ImportError:
                pass
```

In a similar way assuming that there's an external package that follows the same structure as lmms-eval, this piece of code will read the AVAILABLE_MODELS dict from the external package and register new models to be used by lmms-eval.

Does it seem something that you can consider to integrate? I could partially contribute with a PR

Hi, I think it is a cool feature. We didn't implement it because the original repo `lm-eval` also does not contain this feature. But feel free to implement and PR one if you have some thoughts on it and we will try to look into it and work on it.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Unreasonable data in AI2D dataset used for evaluation
Root cause: Validation Gap

Hello! I examined the AI2D dataset used for evaluation and found that a portion of them are unreasonable, suggesting errors in the replacement of options (A, B, ...).

Could you fix this and share your replacement strategy?

Thank you.

Examples:

![img_v3_02bj_e9ad2a78-4a85-4f04-8c9b-0a8315f417fg](https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/55196500/3d12fe3d-1f58-4cb9-8bff-291c014e9caf)
![img_v3_02bj_84214837-58ac-49c2-8179-866b715c22bg](https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/55196500/2a031193-917b-480d-a1fe-442233a9d1f2)
![img_v3_02bj_0a102be3-d612-46c1-ac38-7e0af91dadag](https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/55196500/53da56c9-9f7c-4417-9844-6a92fe54d9e2)
![img_v3_02bj_18323e14-892b-4c5e-96ef-fb2f309a6ffg](https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/55196500/ee78fe36-7592-4458-8cc5-62254100cb33)
![img_v3_02bj_411b5d9a-3ae7-4d40-9005-34d5e41c7f4g](https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/55196500/c6f74c75-f567-4b55-b5fa-6a3c97daafb7)


Hi, I remember that there is an image replacement issue for ai2d during our development raised by the community. I think at the end we follow the pipeline of the ai2d and perform text replacement.

I took into deep look and got confused on AI2D's data. 

Here's a check from another evaluation suite and widely used in the development of InternVL-1.5. 

https://github.com/OpenGVLab/InternVL/tree/main/internvl_chat#ai2d-test

I downloaded their AI2D_TEST data and found the image is the same as ours. 

<img width="2553" alt="image" src="https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/15847405/5d95a3fa-40f1-43d1-ab88-e279055c1ea7">

<img width="620" alt="image" src="https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/15847405/8e0c6d1a-c99b-4e93-9ca1-dbb0b67aca6d">

confusing +1

i think for this task, since the options are A-D, whether can we change A-D to 0-3, and ask the model just answer the index(0, 1, 2, 3)?
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Why cot not supported for Mathverse
Root cause: Unimplemented Feature Gap

Hi! I'm looking the evaluation config of mathverse in `lmms-eval/lmms_eval/tasks/mathverse/mathverse_testmini.yaml` 

```yaml
model_specific_prompt_kwargs:
  default:
    shot_type: "format-prompt" # can also be "custom-prompt"
    query_type: "query_wo" # now only support query_wo
```
What will happen if query_type is specified as ”query_cot"? When I read `MathVerseEvaluator.create_one_query`, it seems that change `query_type` will just change the input prompt.

```python
        # question
        if shot_type == "format-prompt":
            question_text = f"{problem[query_type]}"
```

I guess that maybe the difficulty is to implement the evaluation strategy for cot? Thanks for any helpful comment.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: AssertionError: No tasks specified, or no tasks found. Please verify the task names when using local dataset path
Root cause: Validation Gap

### When I trying to use local dataset path instead of `lmms-lab/MME` in` lmms_eval/tasks/mme/mme.yaml`, `No tasks specified, or no tasks found`  ERROR occurred.


**This is the only one change in `mme.yaml` file**
`dataset_path: D:/xxxx/hf_cache/MME 
dataset_kwargs:
  token: false`


**This is the full traceback:**
  warnings.warn(
SGLang is not installed. If you want to use llava_sglang, please install it using pip install 'sglang[all]' 
06-11 00:01:07 [lmms-eval\lmms_eval\__main__.py:165] INFO Verbosity set to INFO
06-11 00:01:07 [lmms-eval\lmms_eval\__main__.py:226] INFO Verbosity set to INFO
06-11 00:01:09 [lmms-eval\lmms_eval\__main__.py:261] INFO Evaluating on 1 tasks.
06-11 00:01:09 [lmms-eval\lmms_eval\__main__.py:267] ERROR Tasks were not found: mme. Try `lmms-eval --tasks list` for list of available tasks
06-11 00:01:09 [lmms-eval\lmms_eval\__main__.py:272] INFO Selected Tasks: []
Traceback (most recent call last):
  File "C:\Users\Administrator\PycharmProjects\lmms-eval\lmms_eval\__main__.py", line 199, in cli_evaluate
    results, samples = cli_evaluate_single(args)
  File "C:\Users\Administrator\PycharmProjects\lmms-eval\lmms_eval\__main__.py", line 286, in cli_evaluate_single
    results = evaluator.simple_evaluate(
  File "C:\Users\Administrator\PycharmProjects\lmms-eval\lmms_eval\utils.py", line 450, in _wrapper
    return fn(*args, **kwargs)
  File "C:\Users\Administrator\PycharmProjects\lmms-eval\lmms_eval\evaluator.py", line 83, in simple_evaluate
    assert tasks != [], "No tasks specified, or no tasks found. Please verify the task names."
AssertionError: No tasks specified, or no tasks found. Please verify the task names.
06-11 00:01:09 [lmms-eval\lmms_eval\__main__.py:213] ERROR Error during evaluation: No tasks specified, or no tasks found. Please verify the task names.
Traceback (most recent call last):
  File "C:\Users\Administrator\PycharmProjects\lmms-eval\lmms_eval\__main__.py", line 199, in cli_evaluate
    results, samples = cli_evaluate_single(args)
  File "C:\Users\Administrator\PycharmProjects\lmms-eval\lmms_eval\__main__.py", line 286, in cli_evaluate_single
    results = evaluator.simple_evaluate(
  File "C:\Users\Administrator\PycharmProjects\lmms-eval\lmms_eval\utils.py", line 450, in _wrapper
    return fn(*args, **kwargs)
  File "C:\Users\Administrator\PycharmProjects\lmms-eval\lmms_eval\evaluator.py", line 83, in simple_evaluate
    assert tasks != [], "No tasks specified, or no tasks found. Please verify the task names."
AssertionError: No tasks specified, or no tasks found. Please verify the task names.

But I momentarily solve it by delete one line code in `api/task.py` ,delete this line will fix this problem `self.dataset_path = inspect.getfile(import_module(self.dataset_path))`
![image](https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/44715654/d32a1c37-4e79-4ab3-9752-e53379c6dc53)

Thank you for your contributions!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Error during videomme
Root cause: Unimplemented Feature Gap

Hi,

When testing models on llava model and videomme benchmarks, there is one error when loading videos. The `process_images` function can not read mp4 files.

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/2797987f5b88b87bd172714b678a75a1d8051826/lmms_eval/models/llava.py#L326


@choiszt Can you help to take into look at this issue? Thanks!

Hi @yukang2017,

Thank you for your interest in the LMMS-Eval and Videomme benchmarks! 

Regarding your question, the LLaVA model is specifically designed for image benchmark evaluations. For supported video models, I would recommend referring to the following resources:
- [LLaVA-NeXT-Video](https://github.com/LLaVA-VL/LLaVA-NeXT/tree/inference/llavavid)
- [Video-LLaVA](https://github.com/PKU-YuanGroup/Video-LLaVA)
- [LLaMA-VID](https://github.com/dvlab-research/LLaMA-VID)
- [Video-ChatGPT](https://github.com/mbzuai-oryx/Video-ChatGPT)
- [MPLUG-OWL](https://github.com/X-PLUG/mPLUG-Owl)

I hope you find these resources helpful for your work on video benchmarks.

Hello, I got the same error while testing llava-onevision, anyone did find a solution ?


Hi @Mohamed-Dhouib：

Could you provide a more detailed error message? It would help to better understand the issue and offer a more specific solution.

Best,

Hi @choiszt,
Thanks for the offred help, I figured out the cause of the error. I was using model type llava instead of llava_onevision which works well actually for all datasets except for video based ones.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Unable to reproduce SQA results for llava-1.5
Root cause: Algorithmic Error

I was attempting to reproduce llava-1.5's results in ScienceQA but was not able to reproduce the results reported.
Command: 
```
python -m accelerate.commands.launch --num_processes=1 -m lmms_eval --config ./configs/eval_scienceqa_llava1.5.yaml
```

Config:
```
- model: llava
  model_args: pretrained=liuhaotian/llava-v1.5-7b,use_flash_attention_2=False,model_name=llava
  tasks: scienceqa_full
  batch_size: 1
  log_samples: true
  log_samples_suffix: llava1.5_sqa
  output_path: "./logs/"
```
The results I got:
```
|     Tasks      |Version|Filter|n-shot|  Metric   |Value |   |Stderr|
|----------------|-------|------|-----:|-----------|-----:|---|-----:|
|scienceqa_full  |N/A    |none  |     0|exact_match|0.3699|±  |0.0097|
| - scienceqa    |Yaml   |none  |     0|exact_match|0.3744|±  |0.0074|
| - scienceqa_img|Yaml   |none  |     0|exact_match|0.3604|±  |0.0107|

|    Groups    |Version|Filter|n-shot|  Metric   |Value |   |Stderr|
|--------------|-------|------|-----:|-----------|-----:|---|-----:|
|scienceqa_full|N/A    |none  |     0|exact_match|0.3699|±  |0.0097|
```
which is far from what's reported in the paper, for example, SQA-IMG is reported to have 71.6 in the llava-1.5 paper and SQA in general is reported to be around 70.4 in the [excel sheet](https://docs.google.com/spreadsheets/d/1a5ImfdKATDI8T7Cwh6eH-bEsnQFzanFraFUgcS9KHWc/edit?gid=0#gid=0)
What could be wrong?

Thank you for reporting the issue. I will try to look into this error later.

I I encountered the same problem when reproduce llava-1.6-mistral-7b results in ScienceQA. I found the reason maybe the following lines in `models/llava.py`.

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/efb529552c5e4ba039a4cba8e9aa5cb7ba65bf90/lmms_eval/models/llava.py#L361-L371

Although the annotation says “The above for loop has bug” when input has no visuals, but actualy, the above loop run normally and add a `prompt_question` to the `question_input` list, and then these line add a `prompt_question` again. As the result, these no visual inputs generate 2 answers, leading to order mismatch of questions and answers. 

After remove these line code, the scienceqa-full result changes from 36.3 to 76.8.

Hi @GoGoJoestar , I think your fix is correct. We previously use flattened visuals instead of batched visuals in the previous loop, resulting error when handling none visuals. I will remove these lines
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: There is an inconsistency in the number of mmbench images
Root cause: External Dependency Breakage

According to the images downloaded by the LLaVA 1.5 evaluation tool, the number of images in mmbench en dev is 4377, while that of `lmms-lab/MMBench` is only 4329, which may be the cause of the discrepancy between the two evaluation results.

I want to know why there are some pictures missing in huggingface` lmms-lab/MMBench` 

Thank you for your contributions!


Thanks! I am aware of that MMBench has been updated by the authors during the rebbutal and resubmission from CVPR to ECCV. So I am not sure if we used version is newer than LLaVA 1.5's

I think the MMBench evaluation set from old to new is:

- LLaVA 1.5 used version
- LMMs-Eval used version
- MMBench-v0.2

I will take deep look into this issue.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Can not load llava_hf models since the new updates!
Root cause: Configuration Error

I am getting this error:

ValueError: Attempted to load model 'llava_hf', but no model for this name found! Supported model names: llava, qwen_vl, fuyu, batch_gpt4, gpt4v, instructblip, minicpm_v, claude, qwen-vl-api, llava_sglang, idefics2, internvl, gemini_api, reka, from_log, phi3v

When I execute the following:

python3 -m accelerate.commands.launch \
    --num_processes=1 \
    -m lmms_eval \
    --model llava_hf \
    --model_args pretrained="llava-hf/llava-1.5-7b-hf" \
    --tasks mmmu \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix llava_v1.5_mmmu \
    --output_path ./logs/
    
It looks like only the llava_hf model is not registered in lmms-eval. Possible fix?
  
  Thanks

Hi, can you check what error occur when you run `from lmms_eval.models.llava_hf import LlavaHf`. This usually relates to environment problems

>>> from lmms_eval.models.llava_hf import LlavaHf
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Error importing reka: No module named 'reka'
Error importing flash_attn in mplug_owl. Please install flash-attn first.

Installing reka-api resolved the first error. However, the second error is still there. I already install flash-attn!

Hi, can you pull again the main branch again? I don't know why the llava_hf was being removed from the registry. I have added it back.

For the flash_attn in mplug_owl, you don't need to take care of it. It does not affect the inference of the whole pipeline using flash-attn nor the mplug_owl. I think it is some kind of version problem for mplug_owl
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [Feature] Evaluate model with 2gpu per process.
Root cause: Unimplemented Feature Gap

Thank you for your great work.
I want to evaluate llava-next 34b, but I only have A100 40B which is not enough for inference 34b in only 1gpu.
Can you give me some advices?


Hi, @tzjtatata 

Thank you for your interest in our work.

To evaluate a 34b model, you can use tensor parallel to achieve this on 2 A100 40G.

Use llava 1.6 34b as an example. You can pass `--model_args "pretrained=liuhaotian/llava-v1.6-34b,conv_template=mistral_direct,device_map=auto` and start your job without using `accelerate launch`.

Thank you for careful reply. Can I use accelerate with 4 process x 2gpu？



---Original---
From: ***@***.***&gt;
Date: Fri, Mar 15, 2024 20:42 PM
To: ***@***.***&gt;;
Cc: ***@***.******@***.***&gt;;
Subject: Re: [EvolvingLMMs-Lab/lmms-eval] [Feature] Evaluate model with 2gpuper process. (Issue #12)




 
Hi, @tzjtatata
 
Thank you for your interest in our work.
 
To evaluate a 34b model, you can use tensor parallel to achieve this on 2 A100 40G.
 
Use llava 1.6 34b as an example. You can pass --model_args "pretrained=liuhaotian/llava-v1.6-34b,conv_template=mistral_direct,device_map=auto and start your job without using accelerate launch.
 
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***&gt;

@tzjtatata , unfortunately it might not be possible to do so since currently you have to make sure num process to be 1 do activate tensor parallel. We will try to work on features that can support batch size is larger than 1 in the future.

Thank you!



---Original---
From: ***@***.***&gt;
Date: Sat, Mar 16, 2024 10:17 AM
To: ***@***.***&gt;;
Cc: ***@***.******@***.***&gt;;
Subject: Re: [EvolvingLMMs-Lab/lmms-eval] [Feature] Evaluate model with 2gpuper process. (Issue #12)




 
@tzjtatata , unfortunately it might not be possible to do so since currently you have to make sure num process to be 1 do activate tensor parallel. We will try to work on features that can support batch size is larger than 1 in the future.
 
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***&gt;

> Hi, @tzjtatata
> 
> Thank you for your interest in our work.
> 
> To evaluate a 34b model, you can use tensor parallel to achieve this on 2 A100 40G.
> 
> Use llava 1.6 34b as an example. You can pass `--model_args "pretrained=liuhaotian/llava-v1.6-34b,conv_template=mistral_direct,device_map=auto` and start your job without using `accelerate launch`.

https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/4

Yes, it's addressed in this PR.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Cannot find any code / link to worldqa dataset
Root cause: Documentation Deficiency

Hi, 
The link provided in the paper (https://arxiv.org/pdf/2405.03272) seems to be invalid. Can you please provide a link to download the dataset? 

Thank you in advance. 


Hi, you can use this https://huggingface.co/datasets/lmms-lab/worldqa to download dataset and use our current pipeline to evaluate on worldqa

> Hi, you can use this https://huggingface.co/datasets/lmms-lab/worldqa to download dataset and use our current pipeline to evaluate on worldqa

Hi, the access to worldqa dataset needs your permission, and I have submitted requests to access this repository two days ago.

Could you help approve my request? Thanks!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [Reproduce] Unable to reproduce AI2D, ChartQA and InfoVQA results for llava-1.6-mistral-7b
Root cause: Configuration Error

I am tring to reproduce some tasks' result on llava-1.6-mistral-7b, but found large gap on AI2D, ChartQA and InfoVQA. The lmms-eval version I ues is `0.2.0`.

My script:
```shell
python3 -m accelerate.commands.launch \
    --num_processes=3 \
    -m lmms_eval \
    --model llava \
    --model_args pretrained=/My/Downloaded/llava-v1.6-mistral-7b,conv_template=mistral_instruct,attn_implementation=flash_attention_2 \
    --tasks ai2d,chartqa,docvqa_val,scienceqa_full,infovqa_val \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix llava_mistral \
    --output_path ./logs/
```

The results from my running, lmms-eval 0.1 [results excel](https://docs.google.com/spreadsheets/d/1a5ImfdKATDI8T7Cwh6eH-bEsnQFzanFraFUgcS9KHWc/edit?gid=0#gid=0) and llava official blogs are shown below.

||AI2D|ChartQA|infoVQA|DocVQA|ScienceQA-full|ScienceQA-img|
|:--|:--:|:--:|:--:|:--:|:--:|:--:|
|llava official|60.8|38.8| - |72.2| - |72.8|
|lmms-eval 0.1 blog|60.75|38.76|43.77|72.16|0.23|0|
|my reproduce|67.42|52.92|36.74|70.18|76.80|72.83|

The results of DocVQA and ScienceQA-img are similar, but results of AI2D, ChartQA and InfoVQA gap from 7 to 14 point. 
What reason could lead to such gap? Does any task config change between lmms-eval `0.1.0` and `0.2.0`?


I got the same thing for ChartQA, which is also 52.92, I'll try if lmms-eval version 0.1.0 works. But, if we all get this, I think this is a bug. Give it a [bug] tag and someone please fix it.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Propose to print error information when import models
Root cause: Validation Gap

In the file `/root/lmms-eval/lmms_eval/models/__init__.py`, no output is printed if an ImportError occurs. This can lead to confusion for users who encounter a ValueError stating 'Attempted to load model '...', but no model for this name found!' in the file `/root/lmms-eval/lmms_eval/api/registry.py`, specifically at line 32, within the `get_model` function. This error might actually be triggered by a previous ImportError from `{model}.py`, but because the import error doesn’t display any associated messages, users remain unaware of the underlying issue.

recommend:
```
for model_name, model_class in AVAILABLE_MODELS.items():
    try:
        exec(f"from .{model_name} import {model_class}")
    except ImportError as e:
        print(f"Failed to import {model_class} from {model_name}: {e}")
        pass
```

Thanks for this proposal, I will integrate it in a new PR.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [FAQ] about caching the zipped file from HF to local folder
GitHub labels: faq

If you encountered connection issue when downloading the video zipped files to your local cache dir, please consider adding the following kwargs to your yaml file to forcefully download and zip the videos correctly to your local folder.

For example,

```yaml
# lmms_eval/tasks/videomme/videomme.yaml
dataset_kwargs:
  token: True
  cache_dir: videomme
  video: True
  force_download: True
  force_unzip: True
```
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Adding MobileCaptureVQA to the benchmark
Root cause: Unimplemented Feature Gap

Hi team,
I'd be interested to see whether we could add the [MobileCaptureVQA](https://huggingface.co/datasets/arnaudstiegler/mobile_capture_vqa) dataset on this benchmark.

This VQA dataset focused on mobile capture (i.e. images taken from a phone), that aims at assessing models on extraction capabilities specifically for mobile capture. 
Contrarily to existing VQA benchmarks (DocVQA, ChartVQA), it puts the emphasis on mobile-capture-specific noise such as bad lighting, document skew, and provides a much higher variability of text in the wild (can be a receipt, a bottle of wine, food packaging, etc..). Similarly to other VQA datasets, it is meant to be purely extractive, i.e. the answer to the question is written somewhere in the image (which allows for easy scoring).

The dataset is already available on HuggingFace: https://huggingface.co/datasets/arnaudstiegler/mobile_capture_vqa
It contains ~850 questions for ~120 unique images.

I'd be happy to contribute the code to add the dataset if there's any interest!

Here's one sample from the dataset (question/answers is at the top)
![Screenshot 2024-06-19 at 3 58 53 PM](https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/26485052/16d32839-eac2-47ea-820a-8170dcd3432a)



Hi, feel free to contribute dataset and benchmarks into our pipeline. Once you create a PR, we will try to review your code and working on it for a merge