Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Failure with Running vqav2_test on llava-v1.5-7b Model
Root cause: Resource Mishandling

I encountered an issue while running the vqav2_test task on the liuhaotian/llava-v1.5-7b model. The command was executed on a setup with 32 CPUs and 7 RTX A6000 GPUs, but it failed with a subprocess.CalledProcessError due to a SIGKILL (signal 9). The error traceback is provided below.

python3 -m accelerate.commands.launch \
    --num_processes=1 \
    -m lmms_eval \
    --model llava \
    --model_args pretrained="liuhaotian/llava-v1.5-7b" \
    --tasks vqav2_test \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix llava-v1.5_vqav2_test \
    --output_path ./logs/

Traceback (most recent call last):
  File "/fs/nexus-scratch/xijun/junyun/llama3.2-v1/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None)
  File "/fs/nexus-scratch/xijun/junyun/llama3.2-v1/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/fs/nexus-scratch/xijun/junyun/llama3.2-v1/lib/python3.9/site-packages/accelerate/commands/launch.py", line 1178, in <module>
    main()
  File "/fs/nexus-scratch/xijun/junyun/llama3.2-v1/lib/python3.9/site-packages/accelerate/commands/launch.py", line 1174, in main
    launch_command(args)
  File "/fs/nexus-scratch/xijun/junyun/llama3.2-v1/lib/python3.9/site-packages/accelerate/commands/launch.py", line 1168, in launch_command
    simple_launcher(args)
  File "/fs/nexus-scratch/xijun/junyun/llama3.2-v1/lib/python3.9/site-packages/accelerate/commands/launch.py", line 763, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/fs/nexus-scratch/xijun/junyun/llama3.2-v1/bin/python3', '-m', 'lmms_eval', '--model', 'llava', '--model_args', 'pretrained=liuhaotian/llava-v1.5-7b', '--tasks', 'vqav2_test', '--batch_size', '1', '--log_samples', '--log_samples_suffix', 'llava-v1.5_vqav2_test', '--output_path', './logs/']' died with <Signals.SIGKILL: 9>.

The system appears to have sufficient resources, and there were no apparent signs of resource exhaustion.
If further debugging or additional logs are needed, please let me know.


Can you successfully download and load the dataset? VQA requires a lot memory to load the dataset

I was able to successfully download and load the lmms-lab/VQAv2 dataset for the test split, and there were no issues during the loading process.

May I ask what is your full log? Say in which step of the logging this error occurs?

(/fs/nexus-scratch/xijun/junyun/llama3.2-v1) bash-4.4$ python3 -m accelerate.commands.launch --num_processes=1 -m lmms_eval --model llava --model_args
 pretrained="liuhaotian/llava-v1.5-7b" --tasks vqav2_test --batch_size 1 --log_samples --log_samples_suffix llava-v1.5_vqav2_test --output_path ./logs
/                                                                                                                                                     
                                                                                                                                                      
/fs/nexus-scratch/xijun/junyun/llama3.2-v1/lib/python3.9/site-packages/transformers/utils/hub.py:128: FutureWarning: Using `TRANSFORMERS_CACHE` is de$
recated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
  warnings.warn(
2024-11-23 04:33:22.693 | INFO     | __main__:cli_evaluate:294 - Verbosity set to INFO
Detected kernel version 4.18.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the
kernel to the minimum version or higher.
2024-11-23 04:33:26.476 | INFO     | __main__:cli_evaluate_single:377 - Evaluation tracker args: {'output_path': './logs/'}
2024-11-23 04:33:26.477 | INFO     | __main__:cli_evaluate_single:466 - Selected Tasks: ['vqav2_test']
2024-11-23 04:33:26.491 | INFO     | lmms_eval.evaluator:simple_evaluate:155 - Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch $
anual seed to 1234
Resolving data files: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 68/68 [00:00<00:00, 866.37it/s$
Resolving data files: 100%|███████████████████████████████████████████████████████████████████████████████████████| 36/36 [00:00<00:00, 249991.63it/s$
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████| 143/143 [00:00<00:00, 157838.28it/s$
Resolving data files: 100%|███████████████████████████████████████████████████████████████████████████████████████| 68/68 [00:00<00:00, 198754.48it/s$
Resolving data files: 100%|███████████████████████████████████████████████████████████████████████████████████████| 36/36 [00:00<00:00, 245920.10it/s$
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████| 143/143 [00:00<00:00, 412223.69it/s$
OpenCLIP not installed
Detected kernel version 4.18.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the
kernel to the minimum version or higher.
Loaded LLaVA model: liuhaotian/llava-v1.5-7b
You are using a model of type llava to instantiate a model of type llava_llama. This is not supported for all configurations of models and can yield $
rrors.
Downloading shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 676.94it/s$
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:59<00:00, 29.83s/it$
Model Class: LlavaLlamaForCausalLM
2024-11-23 04:34:39.971 | INFO     | lmms_eval.models.llava:__init__:142 - Using single device: cuda:0
2024-11-23 04:34:39.976 | INFO     | lmms_eval.evaluator_utils:from_taskdict:91 - No metadata found in task config for vqav2_test, using default n_sh$
t=0
2024-11-23 04:34:39.977 | INFO     | lmms_eval.api.task:build_all_requests:425 - Building contexts for vqav2_test on rank 0...
Traceback (most recent call last):
  File "/fs/nexus-scratch/xijun/junyun/llama3.2-v1/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/fs/nexus-scratch/xijun/junyun/llama3.2-v1/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/fs/nexus-scratch/xijun/junyun/llama3.2-v1/lib/python3.9/site-packages/accelerate/commands/launch.py", line 1178, in <module>
    main()
  File "/fs/nexus-scratch/xijun/junyun/llama3.2-v1/lib/python3.9/site-packages/accelerate/commands/launch.py", line 1174, in main
    launch_command(args)
  File "/fs/nexus-scratch/xijun/junyun/llama3.2-v1/lib/python3.9/site-packages/accelerate/commands/launch.py", line 1168, in launch_command
    simple_launcher(args)
  File "/fs/nexus-scratch/xijun/junyun/llama3.2-v1/lib/python3.9/site-packages/accelerate/commands/launch.py", line 763, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/fs/nexus-scratch/xijun/junyun/llama3.2-v1/bin/python3', '-m', 'lmms_eval', '--model', 'llava', '--model_ar$
s', 'pretrained=liuhaotian/llava-v1.5-7b', '--tasks', 'vqav2_test', '--batch_size', '1', '--log_samples', '--log_samples_suffix', 'llava-v1.5_vqav2_t$
st', '--output_path', './logs/']' died with <Signals.SIGKILL: 9>.

Ok thanks, I will try this later to see if I can reproduce this error

Hi, I try the vqav2 but I can not reproduce this error. I believe this could be some memory issue on your machine. Seems like when trying to create the iterator, the machine fail and kill the process.



Thanks for trying to reproduce the issue. Could you please share the details of the resources you used for testing? Specifically, it would be helpful to know:

The number and type of GPUs
The amount of CPU cores and RAM
Any specific configurations or environment settings (e.g., CUDA version, driver version)
This information would help narrow down potential resource-related issues on my end.

Thanks in advance!

Hi, I also encounter this problem when I try to evaluate LLaVA-v1.5-7b on VQAv2(no matter _val or _test) on a single RTX4090 or ADA6000. Other datasets such as MME and COCO can be successfully evaluated, but only VQAv2 results in OOM Error. Have you solved this problem? Thanks!

> Hi, I also encounter this problem when I try to evaluate LLaVA-v1.5-7b on VQAv2(no matter _val or _test) on a single RTX4090 or ADA6000. Other datasets such as MME and COCO can be successfully evaluated, but only VQAv2 results in OOM Error. Have you solved this problem? Thanks!

Having the same issue with RTX4090, and a different (smaller) model

Hi everyone, I optimize the build request flow in the newest PR #479. This should prevent loading the entire VQA dataset into the pipeline at the beginning. Can you guys try to pull again from the main and retry the commands?

> Hi everyone, I optimize the build request flow in the newest PR #479. This should prevent loading the entire VQA dataset into the pipeline at the beginning. Can you guys try to pull again from the main and retry the commands?

Thanks! I have successfully evaluated LLaVA-v1.5-7b on vqav2_val and the result is correct according to your given sheet.