Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Questions About SEED-Bench 
Root cause: Documentation Deficiency

Hi, sorry to bother you.

I have several questions regarding the SEED-Bench evaluation:

1. When evaluating SEED-Bench, the preparation process is quite slow. Specifically:  
   Every time I run the evaluation, the `Building contexts` step is always required, and its speed is noticeably slow. Other benchmarks works well and speed is fine, except seedbench. Do I use it correctly? Or is there any method to speedup?

![image](https://github.com/user-attachments/assets/fd2c5a34-a81a-499b-b216-322493a2e104)

---

2. In [`./lmms_eval/models/llava_onevision.py`](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/lmms_eval/models/llava_onevision.py), which branch is used when evaluating SEED-Bench?  
   It seems that for **seedbench-image**, the `elif type(visual[0]) == PIL.Image.Image:` branch is utilized. However, I’m uncertain about **seedbench-video**. I assume the following branch should be used:  
   `if "task_type" in metadata and metadata["task_type"] == "video" and "sample_frames" in metadata:`  
   But I noticed that `metadata["task_type"]` and `metadata["sample_frames"]` don’t seem to exist (though I’m not entirely sure).
Actually, it seems that the `elif type(visual[0]) == PIL.Image.Image:` will be used for both **seedbench-image** and **seedbench-video**, causing the `task_type` wrong for **seedbench-video** which should be `task_type = "video"`.


![image](https://github.com/user-attachments/assets/85eccda8-b6c2-49dd-9151-693e5684d2b7)

---

3. Additionally, what are the main differences in the evaluation methods between [`./lmms_eval/models/llava_onevision.py`](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/lmms_eval/models/llava_onevision.py) and [`./lmms_eval/models/llava_vid.py`](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/lmms_eval/models/llava_vid.py)? I noticed that `llava_vid.py` includes `add_time_instruction`, whereas `llava_onevision.py` does not. Could you clarify the reason for this difference?

---

4. Also, it seems that SEED-Bench does not work well with `llava_vid.py`. Specifically, there’s a bug preventing it from entering the branch `if task == "seedbench":` (line 430). This happens because `visuals` is like `[<PIL.Image.Image image mode=RGB size=1300x956 at 0xFFFECFAA9070>]` for **seedbench-image** whose length is one while its `self.video_decode_backend` is `decord`. Perhaps it could be modified as follows (not very sure):

![image](https://github.com/user-attachments/assets/ae783d51-d719-494f-8df0-c9e14c71fbd9)



1. If most of time is spending on load datasets, this is mostly the performance from the huggingface load_datasets which we can't actually help. If it is not, can you send me the tqdm bar when building the context. I believe we removed the images when building the context and should be faster.
2. I believe it is being treated as multi-images because the video are stored in frames
3. These are two different models. Refered to https://arxiv.org/abs/2408.03326 and https://arxiv.org/abs/2410.02713
4. I believe in this case you should pass in video_decode_backend=image in model_args

Thank you for your response!

1. To avoid the time-consuming operation at line 433 `doc_id_docs = list(self.doc_iterator(rank=rank, limit=limit, world_size=world_size))` in [`./lmms_eval/api/task.py`](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/lmms_eval/api/task.py), I manually saved the context files of seedbench and read them directly. It works in my current environment though it's not elegant. The underlying issue seems to be related to the `.parquet` files from seedbench (which I manually downloaded and placed in `/cache/huggingface`) containing images that are not removed by my code when building the contexts.

2. All other questions have been resolved.


> Thank you for your response!
> 
> 1. To avoid the time-consuming operation at line 433 `doc_id_docs = list(self.doc_iterator(rank=rank, limit=limit, world_size=world_size))` in [`./lmms_eval/api/task.py`](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/lmms_eval/api/task.py), I manually saved the context files of seedbench and read them directly. It works in my current environment though it's not elegant. The underlying issue seems to be related to the `.parquet` files from seedbench (which I manually downloaded and placed in `/cache/huggingface`) containing images that are not removed by my code when building the contexts.
> 2. All other questions have been resolved.

Hi, I also met the same problem, building context of seed-bench may consume more than 10 minutes. Could you tell me how to speed it up? Many thanks!

I manually saved the contexts into `.pkl` files in the data directory for the first time and then read them repeatedly during the evaluation process. You can refer to my implementation for more details:

![image](https://github.com/user-attachments/assets/591644f3-e519-49e8-a39f-ca089f3f9d31)
 

> I manually saved the contexts into `.pkl` files in the data directory for the first time and then read them repeatedly during the evaluation process. You can refer to my implementation for more details:
> 
> ![image](https://private-user-images.githubusercontent.com/31790772/391873878-591644f3-e519-49e8-a39f-ca089f3f9d31.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzMyMTU0MTYsIm5iZiI6MTczMzIxNTExNiwicGF0aCI6Ii8zMTc5MDc3Mi8zOTE4NzM4NzgtNTkxNjQ0ZjMtZTUxOS00OWU4LWEzOWYtY2EwODlmM2Y5ZDMxLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDEyMDMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQxMjAzVDA4MzgzNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTcxNzk2MTJhY2JkN2I1MzNkZWE3OTgxYWU0MTdkNmZiYjMzNjBlMzU4MTA3MzcyZmIzZjcxMDliMjk3MmExZmImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.FFMF6uF9hKu4gQEx9-w_bUmh3ymiaAZi-yrYJUQKWnA)

Got it. Thany you, Bro! Salute.