Repository: open-compass/opencompass
Issue: [Bug] query_per_second does not work
Root cause: Configuration Error

### Prerequisite

- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

{'CUDA available': False,
 'GCC': 'n/a',
 'MMEngine': '0.10.4',
 'MUSA available': False,
 'OpenCV': '4.10.0',
 'PyTorch': '2.4.0',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - C++ Version: 201703\n'
                              '  - MSVC 192930154\n'
                              '  - Intel(R) oneAPI Math Kernel Library Version '
                              '2023.1-Product Build 20230303 for Intel(R) 64 '
                              'architecture applications\n'
                              '  - Intel(R) MKL-DNN v3.4.2 (Git Hash '
                              '1137e04ec0b5251ca2b4400a4fd3c667ce843d67)\n'
                              '  - OpenMP 2019\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - CPU capability usage: AVX2\n'
                              '  - Build settings: BLAS_INFO=mkl, '
                              'BUILD_TYPE=Release, CUDA_VERSION=12.4, '
                              'CUDNN_VERSION=9.1.0, '
                              'CXX_COMPILER=C:/cb/pytorch_1000000000000/work/tmp_bin/sccache-cl.exe, '
                              'CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc '
                              '/Zc:__cplusplus /bigobj /FS /utf-8 '
                              '-DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO '
                              '-DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER '
                              '-DUSE_FBGEMM -DUSE_XNNPACK '
                              '-DSYMBOLICATE_MOBILE_DEBUG_HANDLE /wd4624 '
                              '/wd4068 /wd4067 /wd4267 /wd4661 /wd4717 /wd4244 '
                              '/wd4804 /wd4273, LAPACK_INFO=mkl, '
                              'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
                              'PERF_WITH_AVX512=1, TORCH_VERSION=2.4.0, '
                              'USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=OFF, '
                              'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
                              'USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, '
                              'USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, '
                              'USE_NNPACK=OFF, USE_OPENMP=ON, USE_ROCM=OFF, '
                              'USE_ROCM_KERNEL_ASSERT=OFF, \n',
 'Python': '3.10.14 | packaged by Anaconda, Inc. | (main, May  6 2024, '
           '19:44:50) [MSC v.1916 64 bit (AMD64)]',
 'TorchVision': '0.19.0',
 'numpy_random_seed': 2147483648,
 'opencompass': '0.2.6+',
 'sys.platform': 'win32'}


### Reproduces the problem - code/configuration sample

from opencompass.models import VLLMwithChatTemplate
from mmengine.config import read_base
from opencompass.models import Gemini
from opencompass.partitioners import NaivePartitioner
from opencompass.runners.local_api import LocalAPIRunner
from opencompass.tasks import OpenICLInferTask
with read_base():
    # from .lvevalcmrc_mixup.lveval_cmrc_mixup_gen import (
    #     LVEval_cmrc_mixup_datasets,
    # )
    # from .lvevaldureader_mixup.lveval_dureader_mixup_gen import (
    #     LVEval_dureader_mixup_datasets,
    # )
    # from .lvevalfactrecall_en.lveval_factrecall_en_gen import (
    #     LVEval_factrecall_en_datasets,
    # )
    # from .lvevalfactrecall_zh.lveval_factrecall_zh_gen import (
    #     LVEval_factrecall_zh_datasets,
    # )
    # from .lvevalhotpotwikiqa_mixup.lveval_hotpotwikiqa_mixup_gen import (
    #     LVEval_hotpotwikiqa_mixup_datasets,
    # )
    # from .lvevallic_mixup.lveval_lic_mixup_gen import LVEval_lic_mixup_datasets
    # from .lvevalloogle_CR_mixup.lveval_loogle_CR_mixup_gen import (
    #     LVEval_loogle_CR_mixup_datasets,
    # )
    # from .lvevalloogle_MIR_mixup.lveval_loogle_MIR_mixup_gen import (
    #     LVEval_loogle_MIR_mixup_datasets,
    # )
    # from .lvevalloogle_SD_mixup.lveval_loogle_SD_mixup_gen import (
    #     LVEval_loogle_SD_mixup_datasets,
    # )
    # from .lvevalmultifieldqa_en_mixup.lveval_multifieldqa_en_mixup_gen import (
    #     LVEval_multifieldqa_en_mixup_datasets,
    # )
    from .datasets.lveval.lvevalmultifieldqa_zh_mixup.lveval_multifieldqa_zh_mixup_gen import (
        LVEval_multifieldqa_zh_mixup_datasets,
    )

datasets = sum(
    (v for k, v in locals().items() if k.endswith('_datasets')), []
)



api_meta_template = dict(round=[
    dict(role='HUMAN', api_role='HUMAN'),
    dict(role='BOT', api_role='BOT', generate=True),
], )

models = [
    dict(
        abbr='gemini-1.5-flash',
        type=Gemini,
        path='gemini-1.5-flash',
        key=
        '',  # The key will be obtained from $GEMINI_API_KEY, but you can write down your key here as well
        meta_template=api_meta_template,
        query_per_second=0.1,
        # max_out_len=100,
        # max_seq_len=2048,
        batch_size=1,
        temperature=1,
    )
]

infer = dict(
    partitioner=dict(type=NaivePartitioner),
    runner=dict(
        type=LocalAPIRunner,
        max_num_workers=1,
        concurrent_users=1,
        task=dict(type=OpenICLInferTask)),
)



### Reproduces the problem - command or script

configs/eval_gemini_1_5_lveval.py --debug

### Reproduces the problem - error message

signal.SIGALRM is not available on this platform
signal.SIGALRM is not available on this platform
08/13 11:17:55 - OpenCompass - INFO - Current exp folder: outputs\default\20240813_111755
08/13 11:17:55 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored.
08/13 11:17:55 - OpenCompass - DEBUG - Modules of opencompass's partitioner registry have been automatically imported from opencompass.partitioners
08/13 11:17:55 - OpenCompass - DEBUG - Get class `NaivePartitioner` from "partitioner" registry in "opencompass"
08/13 11:17:55 - OpenCompass - DEBUG - An `NaivePartitioner` instance is built from registry, and its implementation can be found in opencompass.partitioners.naive
08/13 11:17:55 - OpenCompass - DEBUG - Key eval.runner.task.judge_cfg not found in config, ignored.
08/13 11:17:55 - OpenCompass - DEBUG - Key eval.runner.task.dump_details not found in config, ignored.
08/13 11:17:55 - OpenCompass - DEBUG - Key eval.given_pred not found in config, ignored.
08/13 11:17:55 - OpenCompass - DEBUG - Additional config: {}
08/13 11:17:55 - OpenCompass - INFO - Partitioned into 1 tasks.
08/13 11:17:55 - OpenCompass - DEBUG - Task 0: [gemini-1.5-flash/LVEval_multifieldqa_zh_mixup_32k]
08/13 11:17:55 - OpenCompass - DEBUG - Modules of opencompass's runner registry have been automatically imported from opencompass.runners
08/13 11:17:55 - OpenCompass - DEBUG - Get class `LocalAPIRunner` from "runner" registry in "opencompass"
08/13 11:17:55 - OpenCompass - DEBUG - An `LocalAPIRunner` instance is built from registry, and its implementation can be found in opencompass.runners.local_api
08/13 11:17:55 - OpenCompass - DEBUG - Modules of opencompass's task registry have been automatically imported from opencompass.tasks
08/13 11:17:55 - OpenCompass - DEBUG - Get class `OpenICLInferTask` from "task" registry in "opencompass"
08/13 11:17:55 - OpenCompass - DEBUG - An `OpenICLInferTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_infer
08/13 11:17:59 - OpenCompass - INFO - Task [gemini-1.5-flash/LVEval_multifieldqa_zh_mixup_32k]
signal.SIGALRM is not available on this platform
signal.SIGALRM is not available on this platform
C:\Users\PC\anaconda3\envs\opencompass\lib\site-packages\datasets\load.py:1486: FutureWarning: The repository for Infinigence/LVEval contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/Infinigence/LVEval
You can avoid this message in future by passing the argument `trust_remote_code=True`.
Passing `trust_remote_code=True` will be mandatory to load this dataset from the next major release of `datasets`.
  warnings.warn(
08/13 11:18:17 - OpenCompass - INFO - Start inferencing [gemini-1.5-flash/LVEval_multifieldqa_zh_mixup_32k]
[2024-08-13 11:18:17,031] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
  2%|▏         | 2/133 [00:15<17:44,  8.12s/it]08/13 11:18:38 - OpenCompass - ERROR - C:\code\opencompass-main\opencompass\models\gemini_api.py - _generate - 197 - API key expired. Please renew the API key.
08/13 11:18:38 - OpenCompass - ERROR - C:\code\opencompass-main\opencompass\models\gemini_api.py - _generate - 200 - {'error': {'code': 400, 'message': 'API key expired. Please renew the API key.', 'status': 'INVALID_ARGUMENT', 'details': [{'@type': 'type.googleapis.com/google.rpc.ErrorInfo', 'reason': 'API_KEY_INVALID', 'domain': 'googleapis.com', 'metadata': {'service': 'generativelanguage.googleapis.com'}}]}}
  5%|▌         | 7/133 [01:16<22:28, 10.70s/it]08/13 11:19:40 - OpenCompass - ERROR - C:\code\opencompass-main\opencompass\models\gemini_api.py - _generate - 197 - An internal error has occurred. Please retry or report in https://developers.generativeai.google/guide/troubleshooting
08/13 11:19:40 - OpenCompass - ERROR - C:\code\opencompass-main\opencompass\models\gemini_api.py - _generate - 200 - {'error': {'code': 500, 'message': 'An internal error has occurred. Please retry or report in https://developers.generativeai.google/guide/troubleshooting', 'status': 'INTERNAL'}}
08/13 11:19:50 - OpenCompass - ERROR - C:\code\opencompass-main\opencompass\models\gemini_api.py - _generate - 197 - An internal error has occurred. Please retry or report in https://developers.generativeai.google/guide/troubleshooting
08/13 11:19:50 - OpenCompass - ERROR - C:\code\opencompass-main\opencompass\models\gemini_api.py - _generate - 200 - {'error': {'code': 500, 'message': 'An internal error has occurred. Please retry or report in https://developers.generativeai.google/guide/troubleshooting', 'status': 'INTERNAL'}}
  5%|▌         | 7/133 [01:34<28:18, 13.48s/it]

### Other information

I changed the query_per_second value but it doesn't work

To improve the speed, you can increase the batch_size.