Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [Performance] Too slow MMBench evaluation
Root cause: Documentation Deficiency

I’m running an evaluation on the MMBench-en dataset. The evaluation on the MME benchmark went smoothly, but when I switched to MMBench-en, the evaluation speed significantly slowed down.

I’m using an A40 GPU, and each iteration now takes 35 seconds. I’m wondering if this is the expected speed for the MMBench-en dataset or if something might be wrong.

Below are my command and the result of `pip list`.
Let me know if you need any other information. Thank you.

```bash
accelerate launch -m lmms_eval \
        --model llava_hf \
        --model_args pretrained="llava-hf/llava-1.5-7b-hf,device_map=" \
        --tasks mmbench_en \
        --batch_size 1 \
        --output_path ./logs/ \
        --log_samples
```

```
Package            Version
------------------ -----------
accelerate         0.33.0
aiohappyeyeballs   2.3.5
aiohttp            3.10.3
aiosignal          1.3.1
attrs              24.2.0
Brotli             1.0.9
certifi            2024.7.4
chardet            5.2.0
charset-normalizer 3.3.2
colorama           0.4.6
DataProperty       1.0.1
datasets           2.21.0
dill               0.3.8
evaluate           0.4.2
filelock           3.13.1
frozenlist         1.4.1
fsspec             2024.6.1
gmpy2              2.1.2
hf_transfer        0.1.8
huggingface-hub    0.24.5
idna               3.7
Jinja2             3.1.4
joblib             1.4.2
loguru             0.7.2
lxml               5.3.0
MarkupSafe         2.1.3
mbstrdecoder       1.1.3
mkl-fft            1.3.8
mkl-random         1.2.4
mkl-service        2.4.0
mpmath             1.3.0
multidict          6.0.5
multiprocess       0.70.16
networkx           3.3
numpy              1.26.4
packaging          24.1
pandas             2.2.2
pathvalidate       3.2.0
pillow             10.4.0
pip                24.2
portalocker        2.10.1
psutil             6.0.0
pyarrow            17.0.0
PySocks            1.7.1
pytablewriter      1.2.0
python-dateutil    2.9.0.post0
pytz               2024.1
PyYAML             6.0.1
regex              2024.7.24
requests           2.32.3
sacrebleu          2.4.2
safetensors        0.4.4
scikit-learn       1.5.1
scipy              1.14.0
setuptools         72.1.0
six                1.16.0
sqlitedict         2.1.0
sympy              1.12
tabledata          1.3.3
tabulate           0.9.0
tcolorpy           0.1.6
tenacity           9.0.0
threadpoolctl      3.5.0
tokenizers         0.19.1
torch              2.4.0
torchvision        0.19.0
tqdm               4.66.5
transformers       4.44.0
triton             3.0.0
typepy             1.3.2
typing_extensions  4.11.0
tzdata             2024.1
urllib3            2.2.2
wheel              0.43.0
xxhash             3.4.1
yarl               1.9.4
```

It is probably because the pipeline is calling gpt for evaluation. You might wanna check your openai token

It's indeed true that the slowness is caused by the API requests.   
In fact, you can see in the latter part that it continues without the GPU utilization going up.
If you want to save on cloud costs, consider storing only the results and using GPT batch requests.