Repository: open-compass/VLMEvalKit
Issue: LLaVA-onevision-qwen2-7b-ov and Qwen2-VL-7B-Instruct can't inference on two 24G 3090 GPUs
Root cause: External Dependency Breakage

对于LLaVA-onevision-qwen2-7b-ov，我将LLaVA-NEXT仓库下的llava文件夹导入至VLMEvalKit下后运行
```
CUDA_VISIBLE_DEVICES=0,1 python run.py --data Q-Bench1_VAL Q-Bench1_TEST --model llava_onevision_qwen2_7b_ov --verbose --work-dir llava-ov_Q-Bench
```
报错：
```
decoder_config_dict = decoder_config.to_dict() AttributeError: 'dict' object has no attribute 'to_dict'
```
模型无法导入进来，和 [LLaVA-VL/LLaVA-NeXT#284 ](https://github.com/LLaVA-VL/LLaVA-NeXT/issues/284)问题一样。

对于Qwen2-VL-7B-Instruct,运行
```
CUDA_VISIBLE_DEVICES=0,1 python run.py --data Q-Bench1_TEST --model Qwen2-VL-7B-Instruct --verbose --work-dir qwen2-vl_Q-Bench_TEST
```
观察到一开始两张卡显存占用在11G左右，推理两个问题后不再显示结果，显存占用飞速上升到OOM。

>   0%|                                                               | 0/1475 [00:00<?, ?it/s][{'role': 'user', 'content': [{'type': 'image', 'image': 'file:///home/liaowenjie/LMUData/images/Q-Bench1_TEST/11474376103.jpg.jpg', 'min_pixels': 1003520, 'max_pixels': 12845056}, {'type': 'text', 'text': 'Question: How is the brightness level of this starry sky?\nOptions:\nA. High\nB. Average\nC. Low\nPlease select the correct answer from the options above.'}]}]
> C. Low
> C. Low
>   0%|                                                       | 1/1475 [00:01<31:20,  1.28s/it][{'role': 'user', 'content': [{'type': 'image', 'image': 'file:///home/liaowenjie/LMUData/images/Q-Bench1_TEST/AttnGAN_normal_152.jpg.jpg', 'min_pixels': 1003520, 'max_pixels': 12845056}, {'type': 'text', 'text': 'Question: What kind of visual impression does the image give?\nOptions:\nA. Vibrant\nB. Dark\nC. Fresh\nD. Plain\nPlease select the correct answer from the options above.'}]}]
> B. Dark
> B. Dark
>   0%|                                                       | 2/1475 [00:02<25:34,  1.04s/it][{'role': 'user', 'content': [{'type': 'image', 'image': 'file:///home/liaowenjie/LMUData/images/Q-Bench1_TEST/00100.jpg.jpg', 'min_pixels': 1003520, 'max_pixels': 12845056}, {'type': 'text', 'text': 'Question: How bright is this image?\nOptions:\nA. Normal\nB. Dim\nC. Bright\nPlease select the correct answer from the options above.'}]}]
>   0%|                                                     | 2/1475 [00:21<4:20:28, 10.61s/it]
> [2024-10-16 22:55:30,982] ERROR - RUN - run.py: main - 229: Model Qwen2-VL-7B-Instruct x Dataset Q-Bench1_TEST combination failed: CUDA out of memory. Tried to allocate 8.70 GiB. GPU 0 has a total capacty of 23.69 GiB of which 4.93 GiB is free. Including non-PyTorch memory, this process has 17.56 GiB memory in use. Of the allocated memory 16.36 GiB is allocated by PyTorch, and 903.39 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF, skipping this combination.

我的transformers版本为4.45.2

For **LLaVA-onevision-qwen2-7b-ov**, after importing the `llava` folder from the **LLaVA-NEXT** repository into the **VLMEvalKit** and running the following command:

```
CUDA_VISIBLE_DEVICES=0,1 python run.py --data Q-Bench1_VAL Q-Bench1_TEST --model llava_onevision_qwen2_7b_ov --verbose --work-dir llava-ov_Q-Bench
```

I encountered the error: 
```
decoder_config_dict = decoder_config.to_dict() AttributeError: 'dict' object has no attribute 'to_dict'
```

The model could not be loaded, which is the same issue as described in [LLaVA-VL/LLaVA-NeXT#284](https://github.com/LLaVA-VL/LLaVA-NeXT/issues/284).

For **Qwen2-VL-7B-Instruct**, after running:

```
CUDA_VISIBLE_DEVICES=0,1 python run.py --data Q-Bench1_TEST --model Qwen2-VL-7B-Instruct --verbose --work-dir qwen2-vl_Q-Bench_TEST
```

I observed that the memory usage on both GPUs started at around 11GB, but after inferring two questions, no further results were displayed, and the memory usage rapidly increased until an OOM (Out of Memory) error occurred.
my transformers version: 4.45.2



> 对于LLaVA-onevision-qwen2-7b-ov，我将LLaVA-NEXT仓库下的llava文件夹导入至VLMEvalKit下后运行
> 
> ```
> CUDA_VISIBLE_DEVICES=0,1 python run.py --data Q-Bench1_VAL Q-Bench1_TEST --model llava_onevision_qwen2_7b_ov --verbose --work-dir llava-ov_Q-Bench
> ```
> 
> 报错：
> 
> ```
> decoder_config_dict = decoder_config.to_dict() AttributeError: 'dict' object has no attribute 'to_dict'
> ```
> 
> 模型无法导入进来，和 [LLaVA-VL/LLaVA-NeXT#284 ](https://github.com/LLaVA-VL/LLaVA-NeXT/issues/284)问题一样。
> 
> 对于Qwen2-VL-7B-Instruct,运行
> 
> ```
> CUDA_VISIBLE_DEVICES=0,1 python run.py --data Q-Bench1_TEST --model Qwen2-VL-7B-Instruct --verbose --work-dir qwen2-vl_Q-Bench_TEST
> ```
> 
> 观察到一开始两张卡显存占用在11G左右，推理两个问题后不再显示结果，显存占用飞速上升到OOM。
> 
> > 0%|                                                               | 0/1475 [00:00<?, ?it/s][{'role': 'user', 'content': [{'type': 'image', 'image': 'file:///home/liaowenjie/LMUData/images/Q-Bench1_TEST/11474376103.jpg.jpg', 'min_pixels': 1003520, 'max_pixels': 12845056}, {'type': 'text', 'text': 'Question: How is the brightness level of this starry sky?\nOptions:\nA. High\nB. Average\nC. Low\nPlease select the correct answer from the options above.'}]}]
> > C. Low
> > C. Low
> > 0%|                                                       | 1/1475 [00:01<31:20,  1.28s/it][{'role': 'user', 'content': [{'type': 'image', 'image': 'file:///home/liaowenjie/LMUData/images/Q-Bench1_TEST/AttnGAN_normal_152.jpg.jpg', 'min_pixels': 1003520, 'max_pixels': 12845056}, {'type': 'text', 'text': 'Question: What kind of visual impression does the image give?\nOptions:\nA. Vibrant\nB. Dark\nC. Fresh\nD. Plain\nPlease select the correct answer from the options above.'}]}]
> > B. Dark
> > B. Dark
> > 0%|                                                       | 2/1475 [00:02<25:34,  1.04s/it][{'role': 'user', 'content': [{'type': 'image', 'image': 'file:///home/liaowenjie/LMUData/images/Q-Bench1_TEST/00100.jpg.jpg', 'min_pixels': 1003520, 'max_pixels': 12845056}, {'type': 'text', 'text': 'Question: How bright is this image?\nOptions:\nA. Normal\nB. Dim\nC. Bright\nPlease select the correct answer from the options above.'}]}]
> > 0%|                                                     | 2/1475 [00:21<4:20:28, 10.61s/it]
> > [2024-10-16 22:55:30,982] ERROR - RUN - run.py: main - 229: Model Qwen2-VL-7B-Instruct x Dataset Q-Bench1_TEST combination failed: CUDA out of memory. Tried to allocate 8.70 GiB. GPU 0 has a total capacty of 23.69 GiB of which 4.93 GiB is free. Including non-PyTorch memory, this process has 17.56 GiB memory in use. Of the allocated memory 16.36 GiB is allocated by PyTorch, and 903.39 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF, skipping this combination.
> 
> 我的transformers版本为4.45.2

对于Qwen2-VL-7B-Instruct我和你遇到了一样的问题，重装transformers后解决
https://github.com/open-compass/VLMEvalKit/issues/492#issuecomment-2380573490

> > 对于LLaVA-onevision-qwen2-7b-ov，我将LLaVA-NEXT仓库下的llava文件夹导入至VLMEvalKit下后运行
> > ```
> > CUDA_VISIBLE_DEVICES=0,1 python run.py --data Q-Bench1_VAL Q-Bench1_TEST --model llava_onevision_qwen2_7b_ov --verbose --work-dir llava-ov_Q-Bench
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > 报错：
> > ```
> > decoder_config_dict = decoder_config.to_dict() AttributeError: 'dict' object has no attribute 'to_dict'
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > 模型无法导入进来，和 [LLaVA-VL/LLaVA-NeXT#284 ](https://github.com/LLaVA-VL/LLaVA-NeXT/issues/284)问题一样。
> > 对于Qwen2-VL-7B-Instruct,运行
> > ```
> > CUDA_VISIBLE_DEVICES=0,1 python run.py --data Q-Bench1_TEST --model Qwen2-VL-7B-Instruct --verbose --work-dir qwen2-vl_Q-Bench_TEST
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > 观察到一开始两张卡显存占用在11G左右，推理两个问题后不再显示结果，显存占用飞速上升到OOM。
> > > 0%|                                                               | 0/1475 [00:00<?, ?it/s][{'role': 'user', 'content': [{'type': 'image', 'image': 'file:///home/liaowenjie/LMUData/images/Q-Bench1_TEST/11474376103.jpg.jpg', 'min_pixels': 1003520, 'max_pixels': 12845056}, {'type': 'text', 'text': 'Question: How is the brightness level of this starry sky?\nOptions:\nA. High\nB. Average\nC. Low\nPlease select the correct answer from the options above.'}]}]
> > > C. Low
> > > C. Low
> > > 0%|                                                       | 1/1475 [00:01<31:20,  1.28s/it][{'role': 'user', 'content': [{'type': 'image', 'image': 'file:///home/liaowenjie/LMUData/images/Q-Bench1_TEST/AttnGAN_normal_152.jpg.jpg', 'min_pixels': 1003520, 'max_pixels': 12845056}, {'type': 'text', 'text': 'Question: What kind of visual impression does the image give?\nOptions:\nA. Vibrant\nB. Dark\nC. Fresh\nD. Plain\nPlease select the correct answer from the options above.'}]}]
> > > B. Dark
> > > B. Dark
> > > 0%|                                                       | 2/1475 [00:02<25:34,  1.04s/it][{'role': 'user', 'content': [{'type': 'image', 'image': 'file:///home/liaowenjie/LMUData/images/Q-Bench1_TEST/00100.jpg.jpg', 'min_pixels': 1003520, 'max_pixels': 12845056}, {'type': 'text', 'text': 'Question: How bright is this image?\nOptions:\nA. Normal\nB. Dim\nC. Bright\nPlease select the correct answer from the options above.'}]}]
> > > 0%|                                                     | 2/1475 [00:21<4:20:28, 10.61s/it]
> > > [2024-10-16 22:55:30,982] ERROR - RUN - run.py: main - 229: Model Qwen2-VL-7B-Instruct x Dataset Q-Bench1_TEST combination failed: CUDA out of memory. Tried to allocate 8.70 GiB. GPU 0 has a total capacty of 23.69 GiB of which 4.93 GiB is free. Including non-PyTorch memory, this process has 17.56 GiB memory in use. Of the allocated memory 16.36 GiB is allocated by PyTorch, and 903.39 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF, skipping this combination.
> > 
> > 
> > 我的transformers版本为4.45.2
> 
> 对于Qwen2-VL-7B-Instruct我和你遇到了一样的问题，重装transformers后解决 [#492 (comment)](https://github.com/open-compass/VLMEvalKit/issues/492#issuecomment-2380573490)

我重装了transformers后还是一样，能否给一下您的环境配置呢，谢谢！还有就是我用8b的internvl2进行推理都不会爆显存，我感觉可能是qwen2代码没有写好

正常安装vlmeval后`pip install einops transformers_stream_generator`
`pip install git+https://github.com/huggingface/transformers@21fac7abba2a37fae86106f87fcf9974fd1e3830 accelerate`

Hi, @lily410 @whycantfindaname 

You can now set the environment variable `AUTO_SPLIT=1` for evaluating on 3090. 

Related resources: #566 , #546 , #533 

感谢🙏