Repository: open-compass/VLMEvalKit
Issue: Error at the end of eval
Root cause: Validation Gap

Hello, big fan of your work! Also it was great meeting you at ECCV @kennymckormick 🫶

Trying to run TextVQA evals on PaliGemma, all eval went well but then I came across this error. It's a bit cryptic. I ran the evals on a single GPU (when looking up at the error I thought it was related to distributed setups but it isn't). I took PaliGemma to cuda as opposed to your default setup, do running evals on GPU have such side effects? Should I just run on CPU? 

Here are the logs: 
```
[2024-10-31 19:32:52,220] ERROR - RUN - run.py: main - 243: Model paligemma-3b-mix-448 x Dataset TextVQA_VAL combination failed: Default process group has not been initialized, please make sure to call init_process_group., skipping this combination.
Traceback (most recent call last):
  File "/home/merve/vlmevalkit/VLMEvalKit/run.py", line 222, in main
    dist.barrier()
  File "/home/merve/anaconda3/envs/python311/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 79, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/merve/anaconda3/envs/python311/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 3925, in barrier
    opts.device = _get_pg_default_device(group)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/merve/anaconda3/envs/python311/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 669, in _get_pg_default_device
    group = group or _get_default_group()
                     ^^^^^^^^^^^^^^^^^^^^
  File "/home/merve/anaconda3/envs/python311/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 1025, in _get_default_group
    raise ValueError(
ValueError: Default process group has not been initialized, please make sure to call init_process_group.
[2024-10-31 19:32:52] ERROR - run.py: main - 243: Model paligemma-3b-mix-448 x Dataset TextVQA_VAL combination failed: Default process group has not been initialized, please make sure to call init_process_group., skipping this combination.
Traceback (most recent call last):
  File "/home/merve/vlmevalkit/VLMEvalKit/run.py", line 222, in main
    dist.barrier()
  File "/home/merve/anaconda3/envs/python311/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 79, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/merve/anaconda3/envs/python311/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 3925, in barrier
    opts.device = _get_pg_default_device(group)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/merve/anaconda3/envs/python311/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 669, in _get_pg_default_device
    group = group or _get_default_group()
                     ^^^^^^^^^^^^^^^^^^^^
  File "/home/merve/anaconda3/envs/python311/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 1025, in _get_default_group
    raise ValueError(
ValueError: Default process group has not been initialized, please make sure to call init_process_group.
Traceback (most recent call last):
  File "/home/merve/vlmevalkit/VLMEvalKit/run.py", line 255, in <module>
    main()
  File "/home/merve/vlmevalkit/VLMEvalKit/run.py", line 250, in main
    dist.destroy_process_group()
  File "/home/merve/anaconda3/envs/python311/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 1721, in destroy_process_group
    assert pg is not None
           ^^^^^^^^^^^^^^
AssertionError
```


Seems run.py is not handling the non-distributed setting properly, let me create a PR to fix

OK now. You can have another try~

@kennymckormick running evals again, thank you!

it has directly given me eval results now, seems to work, thank you!