Repository: EvolvingLMMs-Lab/lmms-eval
Issue: The evaluation of videoMME is very slow
Root cause: Configuration Error

- The evaluation of videoMME is very slow; it seems that the CPU is reading the video samples serially. Can this be optimized?

@choiszt @ZhangYuanhan-AI Do you have any idea about it? 😜

@xiaofei05 

Please kindly give us more information:
1. the command you used
2. The model you used

From our side, the evaluation time for a 7B model on video-mme should be around 15 minutes with 8 GPU.

I met the same problem. I found that when I set the `num_process` as `1`, the pre-process is very slow.
<img width="1152" alt="image" src="https://github.com/user-attachments/assets/9d4ae187-ee25-417c-b58c-cb72e9df020a">


When I use num_process with `2`, the speed is normal. But the `videomme` is not registered in All_tasks.

<img width="1201" alt="image" src="https://github.com/user-attachments/assets/ae9ae464-5922-42a5-a9c9-ee8d6a94645c">

<img width="1388" alt="image" src="https://github.com/user-attachments/assets/16ab90df-1856-4a47-b46a-59e6280f35b9">


> When I use num_process with `2`, the speed is normal. But the `videomme` is not registered in All_tasks.
> 
> <img alt="image" width="1201" src="https://private-user-images.githubusercontent.com/61257834/351589203-ae9ae464-5922-42a5-a9c9-ee8d6a94645c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjE4MTMzMzUsIm5iZiI6MTcyMTgxMzAzNSwicGF0aCI6Ii82MTI1NzgzNC8zNTE1ODkyMDMtYWU5YWU0NjQtNTkyMi00MmE1LWE5YzktZWU4ZDZhOTQ2NDVjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MjQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzI0VDA5MjM1NVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWI1MTA3ZDdkYTY2YWFiZWRlMzIyZDExMDM3YmY3MWJkMTA2ZTRlZjc1N2UxNGNmYzM2MzMxOGFlNmEwMTg1N2ImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.u2qj2JVshYB5oBz0j5TbuzqjrM33S_yJmVN1Z3PtEeM"> <img alt="image" width="1388" src="https://private-user-images.githubusercontent.com/61257834/351589163-16ab90df-1856-4a47-b46a-59e6280f35b9.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjE4MTMzMzUsIm5iZiI6MTcyMTgxMzAzNSwicGF0aCI6Ii82MTI1NzgzNC8zNTE1ODkxNjMtMTZhYjkwZGYtMTg1Ni00YTQ3LWI0NmEtNTllNjI4MGYzNWI5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MjQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzI0VDA5MjM1NVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTUzZDEyOTg2MWU5YzJhOTJkYjc0NGU5MDkzNmVkZjQ0YzcyMmM1OGMzMzVmNzdjMTczNDNiYmIzYjZjNzJiOTMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.xBFtMc-cfn9mY_IiCVGpcm4oFsiFNaDlCwm2r_KiJrY">

I solve it by `export HF_HOME="~/.cache/huggingface/"
`

I've re-cloned the latest version of the code, and the issue is now resolved.

@liguopeng0923 Hi. How to solve the promble? 
I alse came cross the videomme is not registered in All_tasks.

> [@xiaofei05](https://github.com/xiaofei05)
> 
> Please kindly give us more information:
> 
> 1. the command you used
> 2. The model you used
> 
> From our side, the evaluation time for a 7B model on video-mme should be around 15 minutes with 8 GPU.

But for me ,the evaluation time for a 7B model on video-mme is around 50 minutes also with 8GPU. Is it because the model i used is LLaVA_Video_7B_Qwen2 ? 

> I've re-cloned the latest version of the code, and the issue is now resolved.

How many time did you cost for videomme?