Repository: EvolvingLMMs-Lab/lmms-eval
Issue: How to evaluate the VILA-8B on the video, there are many conflicts of dependencies.
Root cause: Environment Incompatibility

the dependencies of lmms-eval are conflict with VILA, I can not evaluate vila-8b on the videomme and other video benchmark.

I have solved it.

> I have solved it.

Hi, I encountered a problem when evaluating vila-8B，here is my evaluation command:
CUDA_VISIBLE_DEVICES=1 python3 -m accelerate.commands.launch \
    --num_processes=1 \
    -m lmms_eval \
    --model vila \
    --model_args pretrained="Efficient-Large-Model/VILA-7b",attn_implementation="flash_attention_2" \
    --tasks mme \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix VILA-7B \
    --output_path ./logs/outputs_full_pixel

Here is the error message:
![image](https://github.com/user-attachments/assets/8c694cbc-53b8-432f-9fa2-f9e611028ba2)
If I set the num_video_frames to a value, then it will result in another error.

Here is the environment I have:
![image](https://github.com/user-attachments/assets/97063732-adce-4b39-a108-8cfaeb480484)

Can you please tell me what the problem is and how can I correct it?

Thks a lot!



> > I have solved it.
> 
> Hi, I encountered a problem when evaluating vila-8B，here is my evaluation command: CUDA_VISIBLE_DEVICES=1 python3 -m accelerate.commands.launch --num_processes=1 -m lmms_eval --model vila --model_args pretrained="Efficient-Large-Model/VILA-7b",attn_implementation="flash_attention_2" --tasks mme --batch_size 1 --log_samples --log_samples_suffix VILA-7B --output_path ./logs/outputs_full_pixel
> 
> Here is the error message: ![image](https://github.com/user-attachments/assets/8c694cbc-53b8-432f-9fa2-f9e611028ba2) If I set the num_video_frames to a value, then it will result in another error.
> 
> Here is the environment I have: ![image](https://github.com/user-attachments/assets/97063732-adce-4b39-a108-8cfaeb480484)
> 
> Can you please tell me what the problem is and how can I correct it?
> 
> Thks a lot!

Have you solved it? I meets the same problem.

> > > I have solved it.
> > 
> > 
> > Hi, I encountered a problem when evaluating vila-8B，here is my evaluation command: CUDA_VISIBLE_DEVICES=1 python3 -m accelerate.commands.launch --num_processes=1 -m lmms_eval --model vila --model_args pretrained="Efficient-Large-Model/VILA-7b",attn_implementation="flash_attention_2" --tasks mme --batch_size 1 --log_samples --log_samples_suffix VILA-7B --output_path ./logs/outputs_full_pixel
> > Here is the error message: ![image](https://github.com/user-attachments/assets/8c694cbc-53b8-432f-9fa2-f9e611028ba2) If I set the num_video_frames to a value, then it will result in another error.
> > Here is the environment I have: ![image](https://github.com/user-attachments/assets/97063732-adce-4b39-a108-8cfaeb480484)
> > Can you please tell me what the problem is and how can I correct it?
> > Thks a lot!
> 
> Have you solved it? I meets the same problem.


 @huayicong23 @yusalei  It is my configure.ymal:

``` yaml

- model: vila
  model_args: pretrained=Efficient-Large-Model/Llama-3-VILA1.5-8B,max_frames_num=128,conv_template=llama_3,video_decode_backend=decord,attn_implementation=eager
  tasks: videomme,longvideobench_val_v,mlvu
  batch_size: 1
  log_samples: true
  log_samples_suffix: vila_8b
  output_path: "./otuput/"

```