zhiminy/EvalEng huggingface dataset:
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: latency and throughput (tokens/sec) in evaluation 
Root cause: Unimplemented Feature Gap

Do you have a plan to include metrics like latency of generation and throughput (tokens/sec) in the evaluation? I think this would be a good addition. Having these system evaluations will surely help the development and research on efficiency vs. performance

Was wondering the same!

Oh, that's a good suggestion, thank you for proposing that haha.

We will list this feature in our dev plan.

thanks!!

@Luodian is the efficiency evaluation code implemented already or not? Thanks!

We dont have the bandwidth to implement this last few days, and wish to have some contributors can do this! lol. 

But I will look into this if no one has the willing.

> We dont have the bandwidth to implement this last few days, and wish to have some contributors can do this! lol.
> 
> But I will look into this if no one has the willing.

I saw some people are referring to LLM-Viewer: https://github.com/hahnyuan/LLM-Viewer?tab=readme-ov-file#citation. I haven't checked it carefully but some inspiration could be get. Thank you so much!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: The score is zero for `seedbench/seed_video` [Bug]
Root cause: Architectural Constraint

I evaluated LLaVA-1.5 model (BOTH the official checkpoint, and the checkpoint trained by myself), on SeedBench. Here are the results:
- seedbench/seed_image 0.66163
- seedbench/seed_video **0.0**
- seedbench/seed_all **0.52346**

The `seed_image` score is consistent with the LLaVA Paper and the [Google Sheet](https://docs.google.com/spreadsheets/d/1a5ImfdKATDI8T7Cwh6eH-bEsnQFzanFraFUgcS9KHWc/edit?gid=0#gid=0) you provided, but the `seed_video` score is ZERO, resulting in a low `seed_all` score as well. How can I get the expected results on SEED_VIDEO?


Hi, I think that this is because right now our implementation of llava append multi image tokens for llava instead of just one image token. In v0.1.0, we just append one image token to the question for seedbench.

If you changed `batched_visuals` to `flattened_visuals` here then you would get the same result as v0.1.0 of seedbench

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/7c208b76640c986cfe94233dce735c3ca4ad4319/lmms_eval/models/llava.py#L338-L348

Here are my result after the changes

<img width="796" alt="image" src="https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/92624596/9dcf748f-dcc4-4c22-b89c-3e76fa2d4edf">


Thanks！I read the code you provided. So in v0.2.0, all 8 images from a video in seed-bench-video are fed as input. LLaVA-NeXT, which is trained with `anyres` technique, can handle this multi-image input. But LLaVA-1.5 which doesn't adopt `anyres` training, cannot handle multi-image sequence. Am I understanding this correctly?

I think we can add a `if` statement here about the model version and then adopt the appropriate preprocessing(single/multi-image) here, which would be compatible with different LLaVA versions.
Maybe like this:
<img width="678" alt="image" src="https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/65840607/bd9dfb83-6780-4a30-ac82-2aaefb7e12dc">


> Thanks！I read the code you provided. So in v0.2.0, all 8 images from a video in seed-bench-video are fed as input. LLaVA-NeXT, which is trained with `anyres` technique, can handle this multi-image input. But LLaVA-1.5 which doesn't adopt `anyres` training, cannot handle multi-image sequence. Am I understanding this correctly?

Yes, I think this is why LLaVA-1.5 can not handle multi-images correctly.

> I think we can add a `if` statement here about the model version and then adopt the appropriate 

I feel current implementation is the most correct way to reflect the ability of the model on video evaluation. In early version, only the first image was passed to the model because there is only one image token. I think in official repo, the author concat all frames of video into one image. Here we opt not to do so and choose to test the model in multi-images way.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Error when evaluate llava-v1.5-7b[importing reka,flashattention error]
Root cause: Validation Gap

[Error importing reka,flashattention when evaluate llava]

when I try to evaluate the llava-v1.5-7b, I use this command:
 python -m accelerate.commands.launch \
    --num_processes=1 \
    -m lmms_eval \
    --model llava \
    --model_args pretrained="xxx/checkpoints/llava-v1.5-7b" \
    --tasks mme,mmbench_en \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix llava_v1.5_mme_mmbenchen \
    --output_path ./logs/
 
but i get this error, the this program get stuck:
Error importing reka: No module named 'reka.client'
Error importing flash_attn in mplug_owl. Please install flash-attn first

I have install this two module
(lmms-eval) ➜  llava pip show reka
Name: reka
Version: 0.1
Summary: REKA file parser
Home-page:
Author: David Balogh
Author-email: balogh.david@gmail.com
License: MIT
Location: /xxx/.conda/envs/lmms-eval/lib/python3.9/site-packages
Requires:
Required-by:
(lmms-eval) ➜  llava pip show flash-attn
Name: flash-attn
Version: 2.5.9.post1
Summary: Flash Attention: Fast and Memory-Efficient Exact Attention
Home-page: https://github.com/Dao-AILab/flash-attention
Author: Tri Dao
Author-email: trid@cs.stanford.edu
License:
Location: /xxx/.conda/envs/lmms-eval/lib/python3.9/site-packages
Requires: einops, torch
Required-by:

This is just an error for registering `reka.py`. If you could run llava-1.5 then dont worry about this error message.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Reproduction results of llava v1.5 in MMBench CN
Root cause: Configuration Error

Hello, I tried to replicate the experimental results of LLaVA-1.5-7B on MMBench CN, and the final test result was only 55.756, which is slightly different from the [document's ](https://docs.google.com/spreadsheets/d/1a5ImfdKATDI8T7Cwh6eH-bEsnQFzanFraFUgcS9KHWc/edit?gid=0#gid=0)(58.3 and 57.62). What modifications do I need to make?
![image](https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/57285067/ad31fb09-193c-4a76-b2e6-2d197552230d)


I also encountered the same problem, and my result was 55.756. How to solve it?

I also encountered the same problem, and my result was also aournd 55.7. How to solve it?
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Reproduction results of videollava in VideoMMe
Root cause: Configuration Error

Hi, while conducting my experiments, I noticed a discrepancy in the performance results. Specifically, the performance of the videoLlava model in my evaluation is approximately 4% lower than what is reported in the video-mme benchmark [35.0 Vs 39.0].

I have taken the following steps to ensure the accuracy of my evaluation:

Used the LanguageBind/Video-LLaVA-7B-hf model as provided.
Employed the same frame extraction settings and functions that you provided.
Conducted multiple runs to account for any variations due to randomness or external factors.

Despite these efforts, I am unable to account for the 4% performance drop. Thanks for your help.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: BUG: when I test mme  on internvl2, an error will occur：'InternVL2' object has no attribute 'accelerator'
Root cause: Validation Gap

```
CUDA_VISIBLE_DEVICES="7"  accelerate launch \
    --num_processes=1 \
    --main_process_port 12348 \
    -m lmms_eval \
    --model internvl2 \
    --model_args pretrained="OpenGVLab/InternVL2-8B,device_map=auto" \
    --tasks mme \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix llava_mme \
    --output_path ./logs/
```
https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/bd9908fb714552a00385d662f2b9b43c752f46dc/lmms_eval/models/internvl2.py#L180
lm model have't  **self.accelerator = accelerato**r, but **lm.accelerator.num_processes** attribute will appear in  evaluator.py,
https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/bd9908fb714552a00385d662f2b9b43c752f46dc/lmms_eval/evaluator.py#L639






Oh, I think we forgot to add the `self.accelerator` when num_process is equal to 1. I will fix it later

I think we can close this issue since @kcz358 fixed it.

https://github.com/EvolvingLMMs-Lab/lmms-eval/commit/1703b02ed66e06636929f213156c82e392ea3fd6

> I think we can close this issue since @kcz358 fixed it.
> 
> [1703b02](https://github.com/EvolvingLMMs-Lab/lmms-eval/commit/1703b02ed66e06636929f213156c82e392ea3fd6)

yes
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: ConnectionError: Couldn't reach 'lmms-lab/ScienceQA' on the Hub (ProxyError)
Root cause: External Dependency Breakage

Before you open an issue, please check if a similar issue already exists or has been closed before.

### When you open an issue, please be sure to include the following

When I run lmms-eval for scienceqa-img, I face such issue:
_ConnectionError: Couldn't reach 'lmms-lab/ScienceQA' on the Hub (ProxyError)_

Thank you for your contributions!


I'm running into an issue downloading `ScienceQA` as well. Specifically, I'm getting a 502 Server Error. This happens when I try and download using the `huggingface-cli` directly too, so it definitely seems like some files might be missing/corrupted on the huggingface side. 

Just tested: `huggingface-cli download --repo-type dataset lmms-lab/ScienceQA` which is working fine. They had some issues with `datasets` about 1-2 weeks ago, might be related
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Failed to Load data 
Root cause: Documentation Deficiency

Hi,
I tried to load local datasets following https://github.com/EvolvingLMMs-Lab/lmms-eval/issues/23, but I met the following errors
<img width="1385" alt="image" src="https://github.com/user-attachments/assets/6c0237a3-8bf2-4a11-bbfe-7000856d75e5">

This issue has been solved in https://github.com/EvolvingLMMs-Lab/lmms-eval/issues/18
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [ERROR] 'sgl' is not defined
Root cause: Validation Gap

Hi,

I followed your code to evaluate the `llava-next` model. But I met the following errors (2nd figure).

<img width="928" alt="image" src="https://github.com/user-attachments/assets/03c511fa-2426-44cc-824c-4173d32246b1">

<img width="1113" alt="image" src="https://github.com/user-attachments/assets/952ebf45-243e-466a-9aaa-f0cdd641ad1b">







The reason is the missing of corresponding packages. But, interestingly, the message is not shown in my terminal.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: How to evaluate the offline video datasets.
Root cause: Architectural Constraint

Hi,

I find that the default evaluation for videos are online. But my machine is not online, so how to evaluate the offline video datasets

I think current our code logic will call `snapshot_download` to check download cache every time you evaluate so I am not sure whether this is possible. If you want to evaluate offline video currently, the most possible way is likely you prepare your video files in our format under your `HF_HOME` folder and then comment out our video download code in `api/task.py`
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Adding InfiniBench: A Comprehensive Benchmark for Large Multimodal Models in Very Long Video Understanding
Root cause: Documentation Deficiency

Hello team, 
Thank you for this great work for video evaluation, could you add my new benchmark to the evaluation benchmarks
<font size=3><div align='center' > [[Project Page](https://vision-cair.github.io/InfiniBench/)] [[Code](https://github.com/Vision-CAIR/InfiniBench)] [[📝 arXiv Paper](https://arxiv.org/abs/2406.19875)] [[🤗 Dataset](https://huggingface.co/datasets/Vision-CAIR/InfiniBench/tree/main)]</div></font>


Let me know if you need any help from my side. 

Thanks in advance  

Hi Kirolos congrats on the release! You can open a PR and we can then review & merge.

Is there any tutorial for how to add new benchmark to your framework 

Hi @KerolosAtef , you can refer to the [docs](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/docs/task_guide.md) for more detail. If you are doing video tasks, you can check tasks such as `worldqa,videomme,nextqa`
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [BUG] Get weird response from `llava_hf` on `pope` 
Root cause: Environment Incompatibility

command:
```
accelerate launch --num_processes=4 --main_process_port 12345 -m lmms_eval --model llava_hf --model_args pretrained=llava-hf/llava-1.5-7b-hf --tasks pope --batch_size 1 --output_path ./logs/ --log_samples --limit 4
```

result:
[pope.json](https://github.com/user-attachments/files/16403201/pope.json)

version:
lmms_eval                     0.2.0

fixed by remove the whole repository and re-clone from GitHub.
maybe caused by my wrong operation:
I `git pull` the newest code on the `main` branch, and merge the `main` to my custom branch, which is far behind the main
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: LLAVA-next-vid errorThe size of tensor a (2) must match the size of tensor b (7) at non-singleton dimension 0
Root cause: Validation Gap

When I try to run the script on LLAVA-next-vid 7B :

```
if (output_ids[0, -keyword_id.shape[0]:] == keyword_id).all():
RuntimeError: The size of tensor a (2) must match the size of tensor b (7) at non-singleton dimension 0
Model Responding:   0%|                                                                                                                              | 0/1350 [00:17<?, ?it/s]
notebook-c67eb8db5d27-worker-0:126679:135960 [0] NCCL INFO [Service thread] Connection closed by localRank 1
notebook-c67eb8db5d27-worker-0:126680:135961 [1] NCCL INFO [Service thread] Connection closed by localRank 1
notebook-c67eb8db5d27-worker-0:126679:135960 [0] NCCL INFO [Service thread] Connection closed by localRank 0
notebook-c67eb8db5d27-worker-0:126680:135961 [1] NCCL INFO [Service thread] Connection closed by localRank 0
```

after debug，I found the source in `/lmms-eval/LLaVA-NeXT/llavavid/mm_utils.py`:
```
def call_for_batch(self, output_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:
        offset = min(output_ids.shape[1] - self.start_len, self.max_keyword_len)
        self.keyword_ids = [keyword_id.to(output_ids.device) for keyword_id in self.keyword_ids]
        for keyword_id in self.keyword_ids:
             if (output_ids[0, -keyword_id.shape[0]:] == keyword_id).all():
                 return True
        outputs = self.tokenizer.batch_decode(output_ids[:, -offset:], skip_special_tokens=True)[0]
        for keyword in self.keywords:
            if keyword in outputs:
                return True
        return False
```
the shape of output_ids mismatch what the code needs it,for every loop,the shape is:
```
output_ids shape: torch.Size([1, 1])
keyword_id shape: torch.Size([7])
output_ids shape: torch.Size([1, 2])
keyword_id shape: torch.Size([7])
output_ids shape: torch.Size([1, 3])
keyword_id shape: torch.Size([7])
output_ids shape: torch.Size([1, 4])
keyword_id shape: torch.Size([7])
output_ids shape: torch.Size([1, 5])
keyword_id shape: torch.Size([7])
........
```
can this problem be solved?

Hi @guoyanan1g , this is probably because you are using the wrong `conv_template` for the model

> Hi @guoyanan1g , this is probably because you are using the wrong `conv_template` for the model

thanks!This actually the right solve!☺️

I'm also using this model and running into a similar issue.. what's the correct conv_template? Is there a list o these maintained anywhere?
Repository: EvolvingLMMs-Lab/lmms-eval
Issue:  Where should this function `loglikelihood` be called?
Root cause: Documentation Deficiency

Hi,

I tried to apply your framework for my model, but I don't know  where should this function `loglikelihood` be called?

Hi @liguopeng0923 , `loglikelihood` is a function to be used for calculating scores such as perplexity or used for answering multi-choice questions. You can check tasks such as seedbench, worldqa, and perception_test that use this forward function. Most of the tasks do not require this forward function and are generation based. So if you are not going to test your model on these kind of tasks then would be fine to use a dummy function for it.

Thanks! I got it!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: How to use local dataset(avoid downloading from huggingface)
Root cause: Unimplemented Feature Gap

Thanks for your contribution.
I have already downloaded the videochatgpt dataset to a directory by `huggingface-cli download lmms-lab/VideoChatGPT --repo-type dataset --local-dir .`, and I have `export HF_HOME=that_directory` and change the `dataset_path` to that directory of `_default_template_yaml` under videochatgpt task dir as well.
But the program encounters an error when getting the task object (`return TASK_REGISTRY[task_name](model_name=model_name)`)
Here is my logging info:

![image](https://github.com/user-attachments/assets/8977393a-08be-4375-a07c-f89c68345997)
![image](https://github.com/user-attachments/assets/d04ce78f-807c-42c2-ae0a-d0ca997943d3)


I also set `token=False` inside the _default_template_yaml

I think you can explore some HF settings, like `export HF_DATASETS_OFFLINE=1` to set the local dataset path. 

Here's the actual download & load process, and if you can mock the offline load in your local environment using a few lines of code. You would also sucessfully load it with lmms-eval.

Hope you can resolve this and provide your insights to us!

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/3d4884ae16ff3189a5c1dd6bac44265d05ef6a97/lmms_eval/api/task.py#L853

Thanks for your response!
After commenting this line of code,
https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/3d4884ae16ff3189a5c1dd6bac44265d05ef6a97/lmms_eval/api/task.py#L779
and manually mange the `cache_path`, I can successfully loading the local dataset.
It appears that the code will always execute snapshot_download, attempting to download the dataset. I humbly suggest considering the addition of another option(by exporting env or adding cli_args) to load the data from a local folder.

> Thanks for your response! After commenting this line of code,
> 
> https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/3d4884ae16ff3189a5c1dd6bac44265d05ef6a97/lmms_eval/api/task.py#L779
> 
> 
> and manually mange the `cache_path`, I can successfully loading the local dataset.
> It appears that the code will always execute snapshot_download, attempting to download the dataset. I humbly suggest considering the addition of another option(by exporting env or adding cli_args) to load the data from a local folder.


Hi, I am facing the same problem but I am not clear on how to manage the cache_path to the local directory, could you share your change? Thanks a lot!

@ecoli-hit Try this: For a task, set `dataset_path` of `_default_template_yaml` to the local directory where the dataset is downloaded, then replace line `cache_path = snapshot_download(repo_id=self.DATASET_PATH, repo_type="dataset", force_download=force_download, etag_timeout=60)` with `cache_path = self.DATASET_PATH` in the `lmms-eval/lmms_eval/api/task.py`
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [Enhancement] custom logits_processings
Root cause: Unimplemented Feature Gap

Is there any plan to add custom logits processing
during evaluation like: https://github.com/huggingface/transformers/pull/12219#issue-923570044

Hi, I don't think there is such option to do so currently. If you want to do so, I think you have to manually change the logits processing in the generate until function for the model you want to change.

Thanks. If not the custom logit processor, it would be nice to support all the `gen_kwargs` supported by the HF model. 
E.g. https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/lmms_eval/models/llava_hf.py#L363

```
cont = self.model.generate(**inputs, 
    **gen_kwargs, 
    use_cache=self.use_cache, 
    pad_token_id=self.tokenizer.eos_token_id,
    eos_token_id=self.specified_eot_token_id,)
```

user can specify the appropriate `gen_kwargs`.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: pip install broken on Mac because of `decord`
Root cause: External Dependency Breakage

```
pip install lmms-eval

[...]
ERROR: Could not find a version that satisfies the requirement decord (from lmms-eval) (from versions: none)
ERROR: No matching distribution found for decord
```

This is probably due to `decord` not installing properly on Mac Apple Silicon: https://github.com/dmlc/decord/issues/240

Upping this, make decord an optional dep in the pyproject.toml?
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [BUG] science_qa dataset evaluation might be wrong
Root cause: Algorithmic Error

in the function `sqa_process_results` of science_qa/utils.py`,  
```
  target = sqa_doc_to_target(doc).strip().lower()
  pred = results[0].strip().lower()
  if pred == target:
      return {"exact_match": 1.0}
  # pattern: ^[A-Z]\. .*
  if len(pred) >= 2 and pred[0].isupper() and pred[1] == ".":
```
you have made `pred` to be lower,  the `pred[0].isupper()` will always be `False`, which is wrong.  
Maybe the following fix the bug:  
```
  target = sqa_doc_to_target(doc).strip().lower()
  pred = results[0].strip().lower()
  if pred == target:
      return {"exact_match": 1.0}
  # pattern: ^[A-Z]\. .*
  if len(pred) >= 2 and pred[0].isalpha() and pred[1] == ".":
```
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [Common Issue] Common issues you might encounter when using lmms-eval

# Common issues or Feature Requests

Before you raise a question, you can check if your questions can be solved here or read the [docs](https://github.com/EvolvingLMMs-Lab/lmms-eval/tree/main/docs)

More issues will be updated.

Q : How to use local dataset?
A : Though might need to edit some code inside the repo, there are multiple ways to achieve this. You can refer to these issues #335 #179 #124 #23 #21 #25

Q : Exact match is not found
A : This is also related to the network issue. You can refer to this issue #75 

Q : How to change the default settings of the model?
A : You can change the pass in parameters in the `--model_args` flag. A simple example would be `param1=xxx,param2=xxx`.

Q : How to evaluate on big models?
A : For most of the models, you can try setting `device_map=auto` in the model args. For llava, you can try the srt api model that use sglang for faster speed.

Q : About the link for worldqa
A : The old branch for worldqa has been removed as the dataset has been integrated into the main branch. You can directly evaluate using the main branch.

Q : Task not found error
A : This is likely due to some errors occur when trying to register the tasks. The errors are most likely to occur in the `utils.py` or other python scripts written for the tasks. You might check if you have the dependencies needed. For video tasks, you are recommended to set `HF_HOME` explicitly in your envs


# New Feature Requests

Q : New Datasets?
A : Sorry but we won't adding new datasets request from the communities unless we actually need to evaluate on then as we don't have time or member to do so. If you actually needs them on the `lmms-eval`, we welcome the PR and we will review your code as quickly as possible.

Q : New models?
A : This is same as the datasets. You are welcome to raise the PR.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: How to run lmms-eval locally?
Root cause: Documentation Deficiency

Thank you for your excellent work. Unfortunately, my server is unable to connect to the network. Could you please provide me with instructions on how to run lmms-eval locally?

Can this solve your issue?

https://github.com/EvolvingLMMs-Lab/lmms-eval/issues/179